Add sample proof bundle configurations and verification script
Some checks failed
AOC Guard CI / aoc-guard (push) Has been cancelled
AOC Guard CI / aoc-verify (push) Has been cancelled
Concelier Attestation Tests / attestation-tests (push) Has been cancelled
Console CI / console-ci (push) Has been cancelled
Docs CI / lint-and-preview (push) Has been cancelled
Export Center CI / export-ci (push) Has been cancelled
VEX Proof Bundles / verify-bundles (push) Has been cancelled

- Introduced sample proof bundle configuration files for testing, including `sample-proof-bundle-config.dsse.json`, `sample-proof-bundle.dsse.json`, and `sample-proof-bundle.json`.
- Implemented a verification script `test_verify_sample.sh` to validate proof bundles against specified schemas and catalogs.
- Updated existing proof bundle configurations with new metadata, including versioning, created timestamps, and justification details.
- Enhanced evidence entries with expiration dates and hashes for better integrity checks.
- Ensured all new configurations adhere to the defined schema for consistency and reliability in testing.
This commit is contained in:
StellaOps Bot
2025-12-04 08:54:32 +02:00
parent e1262eb916
commit 4dc7cf834a
76 changed files with 3051 additions and 355 deletions

View File

@@ -44,6 +44,8 @@ Upcoming EB1EB10 remediation (Sprint 0161; advisory `docs/product-advisories/
- Ship an offline verifier script and golden bundles/replay fixtures to prove determinism.
- Add incident-mode activation/exit records and redaction/tenant isolation guidance for portable bundles.
Canonical schemas now live in `docs/modules/evidence-locker/schemas/` (EB1, EB2). Offline verification steps and the embeddable script are documented in `docs/modules/evidence-locker/verify-offline.md` (EB9); use the computed Merkle root as the DSSE subject for sealed and portable bundles.
### Merkle recipe (example)
```bash
cd bundle

View File

@@ -0,0 +1,32 @@
# EB1EB10 Gap Closure Plan (EVID-GAPS-161-007)
Purpose: track remediation items from the 28-Nov-2025 advisory so Evidence Locker bundles, replay payloads, and portable exports are provably deterministic and verifiable offline.
Working directory: `docs/implplan` (sprint coordination) with artefacts in `docs/modules/evidence-locker` and `tests/EvidenceLocker`.
## Scope Items
| ID | Deliverable | Artifact / Path | Owner(s) | Acceptance / Notes | Status |
| --- | --- | --- | --- | --- | --- |
| EB1 | Publish canonical manifest schema | `docs/modules/evidence-locker/schemas/bundle.manifest.schema.json` | Evidence Locker Guild | JSON Schema matches EvidenceBundleManifest (bundleId, tenantId, kind, metadata, entries) and captures replay/incident/redaction hooks. | Draft (2025-12-04) |
| EB2 | Publish checksums schema | `docs/modules/evidence-locker/schemas/checksums.schema.json` | Evidence Locker Guild | Canonical map for `checksums.txt`; Merkle root + chunking metadata; sorted entry rule recorded. | Draft (2025-12-04) |
| EB3 | Hash/Merkle recipe doc | `docs/modules/evidence-locker/bundle-packaging.md` (new section) | Evidence Locker Guild | Normative steps for Merkle root + DSSE subject; clarifies gzip/tar invariants and CAS compatibility. | TODO |
| EB4 | Mandatory DSSE predicate/log policy | `docs/modules/evidence-locker/attestation-contract.md` | Evidence Locker Guild · Security Guild | Required claims + signing profiles; Rekor/log policy (optional vs required); aligns with crypto registry defaults. | TODO |
| EB5 | Replay provenance block | `docs/modules/evidence-locker/replay-payload-contract.md` + manifest schema | Evidence Locker Guild · Replay Delivery Guild | Replay digest + DSSE envelope recorded; ordering rules match `DETERMINISTIC_REPLAY.md`; portable bundle retains linkage. | TODO |
| EB6 | Chunking/CAS rules | `checksums.schema.json` + `bundle-packaging.md` | Evidence Locker Guild · Storage/DevOps | Defines chunk sizing, CAS digest, and stability guarantees; CI test to catch ordering changes. | TODO |
| EB7 | Incident-mode signed activation/exit | `docs/modules/evidence-locker/incident-mode.md` | Evidence Locker Guild · Security Guild | Manifest/DSSE captures activation + deactivation events with signer identity; API/CLI steps documented. | TODO |
| EB8 | Tenant isolation + redaction manifest | `bundle-packaging.md` + portable bundle guidance | Evidence Locker Guild · Privacy Guild | Portable bundles omit tenant identifiers; redaction map recorded; verifier asserts redacted fields absent. | TODO |
| EB9 | Offline verifier script | `docs/modules/evidence-locker/verify-offline.md` | Evidence Locker Guild | POSIX script included; no network dependencies; emits Merkle root used by DSSE subject. | DONE (2025-12-04) |
| EB10 | Golden bundles/replay fixtures + SemVer/changelog | `tests/EvidenceLocker/Bundles/Golden/` + release notes (TBD) | Evidence Locker Guild · CLI Guild | Golden sealed + portable bundles and replay NDJSON with expected roots; changelog bump covering EB1EB9. | TODO |
## Near-Term Actions (to move EB1EB10 to DONE)
- Wire schemas into EvidenceLocker CI (manifest + checksums validation) and surface in API/CLI OpenAPI/Help.
- Update `attestation-contract.md` and `incident-mode.md` with DSSE predicate/log policy and signed incident toggles (EB4, EB7).
- Extend replay contract with provenance block and ordering example, and mirror in manifest schema (EB5).
- Add normative Merkle/CAS section to `bundle-packaging.md`, ensuring DSSE subject references the root hash (EB3, EB6).
- Create golden fixtures under `tests/EvidenceLocker/Bundles/Golden/` with recorded expected hashes and replay traces; hook into xUnit tests (EB10).
- Bump Evidence Locker and CLI SemVer and changelog once above artefacts are wired (EB10).
## Dependencies and Links
- Advisory: `docs/product-advisories/archived/27-Nov-2025-superseded/28-Nov-2025 - Evidence Bundle and Replay Contracts.md`
- Replay rules: `docs/replay/DETERMINISTIC_REPLAY.md`
- Sprint tracking: `docs/implplan/SPRINT_0161_0001_0001_evidencelocker.md` (EVID-GAPS-161-007)

View File

@@ -0,0 +1,142 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://stellaops.local/schemas/evidence/bundle.manifest.schema.json",
"title": "StellaOps Evidence Bundle Manifest (EB1)",
"description": "Canonical manifest for deterministic evidence bundles; aligns with EvidenceLocker build models and EB1EB10 advisory gaps.",
"type": "object",
"additionalProperties": false,
"required": [
"bundleId",
"tenantId",
"kind",
"createdAt",
"metadata",
"entries"
],
"properties": {
"bundleId": {
"type": "string",
"description": "Bundle identifier in UUID v4 N-format (no dashes).",
"pattern": "^[0-9a-fA-F]{32}$"
},
"tenantId": {
"type": "string",
"description": "Tenant identifier in UUID v4 N-format (no dashes).",
"pattern": "^[0-9a-fA-F]{32}$"
},
"kind": {
"description": "Bundle category; numeric values mirror EvidenceBundleKind enum.",
"oneOf": [
{ "type": "string", "enum": ["evaluation", "job", "export"] },
{ "type": "integer", "enum": [1, 2, 3] }
]
},
"createdAt": {
"type": "string",
"format": "date-time",
"description": "Bundle creation timestamp (UTC, RFC3339)."
},
"metadata": {
"type": "object",
"description": "Arbitrary key/value metadata captured at bundle creation.",
"additionalProperties": { "type": "string" }
},
"entries": {
"type": "array",
"description": "Canonical file inventory used to derive checksums and Merkle root.",
"minItems": 1,
"items": { "$ref": "#/$defs/manifestEntry" }
},
"hashSummary": {
"type": "object",
"description": "Optional Merkle root summary that binds the manifest to checksums.txt.",
"additionalProperties": false,
"required": ["algorithm", "merkleRoot"],
"properties": {
"algorithm": { "type": "string", "enum": ["sha256"] },
"merkleRoot": { "type": "string", "pattern": "^[0-9a-f]{64}$" },
"checksumsPath": {
"type": "string",
"description": "Relative path to canonical checksums file inside the bundle.",
"default": "checksums.txt"
}
}
},
"replayProvenance": {
"type": "object",
"description": "Optional replay linkage proving how the bundle was produced for deterministic re-run.",
"additionalProperties": false,
"required": ["recordDigest"],
"properties": {
"recordDigest": { "type": "string", "pattern": "^sha256:[0-9a-f]{64}$" },
"sequence": { "type": "integer", "minimum": 0 },
"ledgerUri": { "type": "string", "format": "uri" },
"dsseEnvelope": {
"type": "string",
"description": "Base64-encoded DSSE envelope for replay record provenance.",
"contentEncoding": "base64"
},
"transparencyLog": {
"type": "object",
"additionalProperties": false,
"properties": {
"rekorUuid": { "type": "string" },
"logIndex": { "type": "integer", "minimum": 0 },
"inclusionProof": { "type": "string" }
}
}
}
},
"incident": {
"type": "object",
"description": "Incident-mode activation/exit records captured at bundle time.",
"additionalProperties": false,
"properties": {
"activatedAt": { "type": "string", "format": "date-time" },
"activatedBy": { "type": "string" },
"reason": { "type": "string" },
"deactivatedAt": { "type": "string", "format": "date-time" },
"deactivatedBy": { "type": "string" }
}
},
"redaction": {
"type": "object",
"description": "Portable-bundle redaction details to prove tenant isolation.",
"additionalProperties": false,
"properties": {
"portable": { "type": "boolean", "default": false },
"maskedFields": {
"type": "array",
"items": { "type": "string" }
},
"tenantToken": {
"type": "string",
"description": "Opaque token replacing tenantId in portable bundles."
}
}
}
},
"$defs": {
"manifestEntry": {
"type": "object",
"additionalProperties": false,
"required": ["section", "canonicalPath", "sha256", "sizeBytes", "mediaType"],
"properties": {
"section": { "type": "string", "minLength": 1 },
"canonicalPath": {
"type": "string",
"description": "Deterministic path within the bundle using '/' separators.",
"pattern": "^(?:[A-Za-z0-9_.-]+/)*[A-Za-z0-9_.-]+$"
},
"sha256": { "type": "string", "pattern": "^[0-9a-f]{64}$" },
"sizeBytes": { "type": "integer", "minimum": 0 },
"mediaType": { "type": "string" },
"attributes": {
"type": "object",
"description": "Section-specific attributes (e.g., sbom format, dsse predicate).",
"additionalProperties": { "type": "string" }
}
}
}
}
}

View File

@@ -0,0 +1,47 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://stellaops.local/schemas/evidence/checksums.schema.json",
"title": "StellaOps Evidence Bundle Checksums (EB2)",
"description": "Canonical checksum map used to derive the Merkle root and DSSE subject for evidence bundles.",
"type": "object",
"additionalProperties": false,
"required": ["algorithm", "root", "entries"],
"properties": {
"algorithm": { "type": "string", "enum": ["sha256"] },
"root": { "type": "string", "pattern": "^[0-9a-f]{64}$" },
"generatedAt": { "type": "string", "format": "date-time" },
"bundleId": { "type": "string", "pattern": "^[0-9a-fA-F]{32}$" },
"tenantId": { "type": "string", "pattern": "^[0-9a-fA-F]{32}$" },
"entries": {
"type": "array",
"minItems": 1,
"description": "Sorted list of entry hashes; order must be lexicographic on canonicalPath.",
"items": { "$ref": "#/$defs/checksumEntry" }
},
"chunking": {
"type": "object",
"description": "Optional chunked/CAS hashing strategy for large payloads.",
"additionalProperties": false,
"properties": {
"strategy": { "type": "string", "enum": ["none", "fixed", "buzhash"] },
"chunkSizeBytes": { "type": "integer", "minimum": 1024 },
"casDigestAlgorithm": { "type": "string", "enum": ["sha256"] }
}
}
},
"$defs": {
"checksumEntry": {
"type": "object",
"additionalProperties": false,
"required": ["canonicalPath", "sha256", "sizeBytes"],
"properties": {
"canonicalPath": {
"type": "string",
"pattern": "^(?:[A-Za-z0-9_.-]+/)*[A-Za-z0-9_.-]+$"
},
"sha256": { "type": "string", "pattern": "^[0-9a-f]{64}$" },
"sizeBytes": { "type": "integer", "minimum": 0 }
}
}
}
}

View File

@@ -0,0 +1,51 @@
# Offline Verification Playbook (EB9)
Purpose: allow auditors to validate Evidence Locker bundles without network access, using only POSIX tools. Applies to both sealed `bundle.tgz` and portable `portable-bundle-v1.tgz`.
## Prerequisites
- `tar`, `sha256sum` (or `shasum`), `awk`, `base64`.
- Optional: `jq` for schema validation; `cosign` or `stella` CLI for DSSE verification if pre-loaded.
## Quick steps (sealed bundle)
1) `tar -xzf bundle.tgz -C /tmp/bundle`
2) `cd /tmp/bundle`
3) Validate checksums: `sha256sum -c checksums.txt`
4) Derive Merkle root (matches DSSE subject): `sha256sum checksums.txt | awk '{print $1}'`
5) Validate manifest against schema (if `jq` present): `jq -e 'input | type=="object"' manifest.json >/dev/null`
6) Verify DSSE envelope (optional but recommended):
- `cat manifest.json | base64 | cosign verify-blob --key cosign.pub --bundle signature.json --bundleType dsse`
- or `stella evidence verify --bundle ../bundle.tgz --offline` once CLI supports offline mode.
## Quick steps (portable bundle)
Same as sealed, plus confirm redaction:
- `jq -e 'has(\"redaction\") and .redaction.portable==true' manifest.json >/dev/null` (if `jq` available)
- Confirm no tenant identifiers in `bundle.json` and `manifest.json`.
## Embeddable verifier script
Place the following script into `verify-offline.sh` when assembling portable bundles. It exits non-zero on any mismatch and prints the Merkle root used as DSSE subject.
```bash
#!/usr/bin/env bash
set -euo pipefail
BUNDLE="${1:-bundle.tgz}"
WORKDIR="$(mktemp -d)"
cleanup() { rm -rf "$WORKDIR"; }
trap cleanup EXIT
tar -xzf "$BUNDLE" -C "$WORKDIR"
cd "$WORKDIR"
sha256sum -c checksums.txt
MERKLE=$(sha256sum checksums.txt | awk '{print $1}')
printf "merkle_root=%s\n" "$MERKLE"
if command -v jq >/dev/null; then
jq -e 'type=="object" and has("entries")' manifest.json >/dev/null
fi
```
## Fixtures
- Golden bundles and replay records live under `tests/EvidenceLocker/Bundles/Golden/`.
- Expected Merkle roots and DSSE payload digests should be recorded alongside each fixture to keep CI deterministic.
## References
- Manifest schema: `docs/modules/evidence-locker/schemas/bundle.manifest.schema.json`
- Checksums schema: `docs/modules/evidence-locker/schemas/checksums.schema.json`
- Merkle recipe: see `docs/modules/evidence-locker/bundle-packaging.md`

View File

@@ -1,34 +1,58 @@
# Export Center Determinism & Rerun Hash Guide
Advisory: `docs/product-advisories/28-Nov-2025 - Export Center and Reporting Strategy.md` (EC1EC10).
Advisory anchor: `docs/product-advisories/archived/27-Nov-2025-superseded/28-Nov-2025 - Export Center and Reporting Strategy.md` (EC1EC10).
## Adapter settings (runnable example)
- JSON adapters: `--compression zstd --compression-level 19 --deterministic-order`
- Mirror adapter: sort descriptors by digest, emit annotations in lexicographic order, disable mtime in tar (`--mtime 0`).
- Delta adapter: include `baseManifestHash` and sorted `added`/`removed` lists; tombstones must be explicit.
## EC1 — Signed schemas
- Export profile schema: `docs/modules/export-center/schemas/export-profile.schema.json` (selectors, approvals, quotas).
- Export manifest schema: `docs/modules/export-center/schemas/export-manifest.schema.json` (rerunHash, integrity headers, attestations, quotas/backpressure).
- Both schemas must be signed (DSSE) alongside publication; DSSE envelopes live next to the schema files when generated in CI.
## Rerun-hash check
## EC2 — Per-adapter determinism and rerun hash
- JSON adapters: canonical JSONL, sorted keys, zstd level 19; filenames stable (`advisories-<shard>.jsonl.zst`); gzip forbidden.
- Trivy adapters: pin schema version (see `trivy-adapter.md`), normalize namespaces, ordered records by `(namespace, package, vulnerabilityId)`.
- Mirror full: tar with `--sort=name --mtime=@0 --owner=0 --group=0 --numeric-owner`; manifest entries sorted by path; indexes stable.
- Mirror delta: include `baseManifestDigest`, sorted `added`/`removed`, explicit `tombstones`; reject deltas without tombstones for removed entries.
- Rerun hash algorithm: SHA-256 over newline-joined, sorted `contents[*].digest` values; stored in `manifest.rerunHash` and asserted in CI.
- CI harness: `docs/modules/export-center/operations/verify-export-kit.sh` recomputes rerun hash and schema-consistent integrity hints.
## EC3 — DSSE + SLSA attestation with log metadata
- All manifests and provenance files carry DSSE envelopes; provenance must include SLSA v1 builder metadata plus log proof (`kind`, `logId`, `logIndex`, `entryDigest`, `timestamp`).
- Provenance subjects list both `manifests/export.json` and bundle tar/OCI digest; log metadata is mandatory even when transparency uploads are deferred.
## EC4 — Cross-tenant approval flow
- `selectors.tenants` must contain the profile tenant; when selectors include additional tenants or wildcards, `approval.required=true` with `approvedBy` and `ticket` is mandatory (validated by the verify script).
## EC5 — Distribution integrity headers and OCI annotations
- HTTP: `Digest: sha-256=<base64>` derived from bundle digest; `X-Stella-Signature: dsse-b64:<envelope>`; `X-Stella-Immutability: true` for immutable responses.
- OCI: annotations must include `io.stellaops.export.profile`, `io.stellaops.export.run`, `io.stellaops.export.manifest-digest`, `io.stellaops.export.provenance-ref`, and `org.opencontainers.image.ref.name`.
## EC6 — Trivy schema pinning
- Schema compatibility is pinned in `trivy-adapter.md`; CI rejects versions above the pinned set and emits `ERR_EXPORT_UNSUPPORTED_SCHEMA`.
- Mirror/export manifests must record the targeted `schemaVersion` so rerun-hash and consumers can enforce deterministic decoding.
## EC7 — Mirror delta/tombstone rules
- Deltas MUST include tombstones for all removals and a `baseManifestDigest` that matches the referenced baseline; omitted tombstones fail verification.
- `delta.added/removed` are sorted, and `resetBaseline=false` unless explicitly set; consumers apply deltas in order and refuse out-of-order manifests.
## EC8 — Encryption/recipient policy
- Only `age` or `aes-gcm` envelopes; recipients enumerated with `fingerprint` and optional `wrappedKey` in manifest and provenance.
- `strict=true` encrypts everything except manifest/provenance; defaults to `false` to keep discovery metadata plaintext.
## EC9 — Quotas and backpressure
- Manifest `quotas` block captures `maxActiveRuns`, `maxQueuedRuns`, `backpressureMode` (`reject`|`defer`|`throttle`), and optional `cpuThrottlePercent`.
- CI verifies presence of quotas; operators surface `429` with `X-Stella-Quota-*` hints when limits engage.
## EC10 — Offline export kit + verify script
- Fixtures: `src/ExportCenter/__fixtures/export-kit/*` (manifest, manifest.sha256, manifest.dsse, provenance).
- Verifier: `docs/modules/export-center/operations/verify-export-kit.sh`
- Validates manifest hash against `manifest.sha256`.
- Recomputes rerun hash.
- Confirms integrity headers align with OCI annotations.
- Enforces approval + quota presence for cross-tenant selectors.
- Confirms provenance references manifest digest and carries log metadata.
- Tar flags for offline kit assembly: `tar --sort=name --mtime=@0 --owner=0 --group=0 --numeric-owner`.
## Quick rerun-hash smoke (uses fixtures)
```bash
set -euo pipefail
run_id=$(uuidgen)
stella export run --profile demo --run-id "$run_id" --out /tmp/export1
sha256sum /tmp/export1/manifest.json > /tmp/export1/manifest.sha256
# second run
run_id2=$(uuidgen)
stella export run --profile demo --run-id "$run_id2" --out /tmp/export2
sha256sum /tmp/export2/manifest.json > /tmp/export2/manifest.sha256
diff -u /tmp/export1/manifest.sha256 /tmp/export2/manifest.sha256
./docs/modules/export-center/operations/verify-export-kit.sh src/ExportCenter/__fixtures/export-kit
```
## Integrity headers (HTTP example)
- `Digest: sha-256=<base64>`
- `X-Stella-Signature: dsse-b64=<payload>`
- `X-Stella-Immutability: true`
## Offline kit packaging
- Tar flags: `tar --sort=name --mtime=@0 --owner=0 --group=0 --numeric-owner`
- Include `export-kit/manifest.json` + `manifest.dsse`; add `verify-export-kit.sh` to check hashes and signatures.
## Where to place fixtures
- `src/ExportCenter/__fixtures/` for deterministic manifests/outputs used by tests.
- Add rerun-hash CI to compare fixture hash against regenerated outputs.

View File

@@ -92,9 +92,11 @@ delta/
manifest.diff.json # summary of counts, hashes, base export metadata
```
- **Base lookup:** The worker verifies that the base export is reachable (download path or OCI reference). If missing, the run fails with `ERR_EXPORT_BASE_MISSING`.
- **Change detection:** Uses deterministic hashing of normalized records to compute additions/updates. Indexes are regenerated only for affected subjects.
- **Application order:** Consumers apply deltas sequentially. A `resetBaseline=true` flag instructs them to drop cached state and apply the bundle as a full refresh.
- **Base lookup:** The worker verifies that the base export is reachable (download path or OCI reference). If missing, the run fails with `ERR_EXPORT_BASE_MISSING`.
- **Change detection:** Uses deterministic hashing of normalized records to compute additions/updates. Indexes are regenerated only for affected subjects.
- **Application order:** Consumers apply deltas sequentially. A `resetBaseline=true` flag instructs them to drop cached state and apply the bundle as a full refresh.
- **Tombstones required:** Every removal must emit a tombstone entry plus the `removed` list; deltas without tombstones fail verification (`verify-export-kit.sh`).
- **Integrity headers:** Each delta bundle exports `Digest`, `X-Stella-Signature`, and `X-Stella-Immutability` derived from the OCI annotation `io.stellaops.export.manifest-digest`. Consumers must validate before applying.
Example `manifest.diff.json` (delta):
@@ -195,6 +197,7 @@ sequenceDiagram
- Post-import checks:
- Recompute SHA256 for `manifest.yaml` and a sample data file; compare to manifest.
- Run `mirror verify` (Offline Kit) and confirm zero mismatches.
- Confirm OCI annotations `io.stellaops.export.profile/run/manifest-digest/provenance-ref` match the bundle being applied.
## 8. Troubleshooting

View File

@@ -0,0 +1,99 @@
#!/usr/bin/env bash
set -euo pipefail
# Offline verifier for Export Center bundles (EC10)
# Usage: VERIFY_FIXTURE=1 ./verify-export-kit.sh [kit_dir]
kit_dir="${1:-src/ExportCenter/__fixtures/export-kit}"
for f in manifest.json manifest.sha256 provenance.json manifest.dsse; do
if [[ ! -f "${kit_dir}/${f}" ]]; then
echo "missing ${kit_dir}/${f}" >&2
exit 1
fi
done
manifest_sha="$(sha256sum "${kit_dir}/manifest.json" | awk '{print $1}')"
expected_sha="$(awk '{print $1}' "${kit_dir}/manifest.sha256")"
if [[ "${manifest_sha}" != "${expected_sha}" ]]; then
echo "manifest hash mismatch: expected ${expected_sha} got ${manifest_sha}" >&2
exit 1
fi
python - <<'PY' "${kit_dir}" "${manifest_sha}"
import base64, hashlib, json, sys, pathlib
kit = pathlib.Path(sys.argv[1])
manifest_sha = sys.argv[2]
manifest = json.loads(kit.joinpath("manifest.json").read_text())
errors = []
selectors = manifest.get("selectors", {})
tenants = selectors.get("tenants", [])
tenant = manifest.get("tenant")
if tenant and tenants and tenant not in tenants and "*" not in tenants[0]:
errors.append(f"tenant {tenant} not included in selectors.tenants {tenants}")
digests = []
for item in manifest.get("contents", []):
digest = item.get("digest", "")
if not digest.startswith("sha256:"):
errors.append(f"invalid digest format for {item.get('path')}")
continue
digests.append(digest.split("sha256:")[1])
rerun_calc = hashlib.sha256("\n".join(sorted(digests)).encode()).hexdigest()
rerun_expected = manifest.get("rerunHash")
if rerun_expected != f"sha256:{rerun_calc}":
errors.append(f"rerunHash mismatch: expected sha256:{rerun_calc} got {rerun_expected}")
annotations = manifest["integrity"]["oci"]["annotations"]
manifest_digest_hex = annotations["io.stellaops.export.manifest-digest"].split("sha256:")[1]
digest_hdr = manifest["integrity"]["httpHeaders"]["Digest"]
expected_hdr = "sha-256=" + base64.b64encode(bytes.fromhex(manifest_digest_hex)).decode()
if digest_hdr != expected_hdr:
errors.append("Digest header does not match manifest digest annotation")
log_meta = manifest.get("attestations", {}).get("log")
if not log_meta:
errors.append("attestation log metadata missing")
if not manifest.get("quotas"):
errors.append("quotas/backpressure block missing")
if len(set(tenants)) > 1 and not manifest.get("approval", {}).get("required"):
errors.append("cross-tenant approval required but not present")
if errors:
for err in errors:
print(err)
sys.exit(1)
print("manifest checks ok")
PY
python - <<'PY' "${kit_dir}" "${manifest_sha}"
import json, sys, pathlib
kit = pathlib.Path(sys.argv[1])
manifest_sha = sys.argv[2]
prov = json.loads(kit.joinpath("provenance.json").read_text())
manifest_subject = prov["subject"][0]["digest"]["sha256"]
if manifest_subject != manifest_sha:
print(f"provenance manifest digest mismatch: {manifest_subject} vs {manifest_sha}")
sys.exit(1)
log = prov["predicate"]["environment"]["logs"]
required = ["kind", "logId", "logIndex", "entryDigest", "timestamp"]
missing = [k for k in required if k not in log]
if missing:
print(f"provenance log metadata missing keys: {missing}")
sys.exit(1)
print("provenance checks ok")
PY
echo "verify-export-kit: PASS"

View File

@@ -102,12 +102,12 @@ Selectors (time windows, tenants, products, SBOM subjects, ecosystems) are suppl
- **Constraints:** Requires the base manifest to exist in object storage or artifact registry accessible to the worker. Fails with `ERR_EXPORT_BASE_MISSING` otherwise.
- **Workflow:** Ideal for frequent updates to mirrored environments with limited bandwidth.
## Compatibility and guardrails
- **Aggregation-Only Contract:** All profiles respect AOC boundaries: raw evidence is never mutated. Policy outputs are appended separately with clear provenance.
- **Tenant scoping:** Profiles are tenant-specific. Cross-tenant exports require explicit administrative approval and signed justification.
- **Retriable runs:** Re-running a profile with identical selectors yields matching manifests and hashes, facilitating verify-on-download workflows.
- **Offline operation:** JSON and mirror profiles function in offline mode without additional configuration. Trivy profiles require pre-seeded schema metadata shipped via Offline Kit.
- **Quota integration:** Profiles can define run quotas (per tenant per day). Quota exhaustion surfaces as `429 Too Many Requests` with `X-Stella-Quota-*` hints.
## Compatibility and guardrails
- **Aggregation-Only Contract:** All profiles respect AOC boundaries: raw evidence is never mutated. Policy outputs are appended separately with clear provenance.
- **Tenant scoping + approvals:** Profiles are tenant-specific. When `selectors.tenants` includes additional tenants or wildcards, `approval.required=true` plus `approvedBy` and `ticket` must be present (validated by `verify-export-kit.sh` and schema).
- **Retriable runs:** Re-running a profile with identical selectors yields matching manifests and hashes, facilitating verify-on-download workflows.
- **Offline operation:** JSON and mirror profiles function in offline mode without additional configuration. Trivy profiles require pre-seeded schema metadata shipped via Offline Kit.
- **Quota integration and backpressure:** Profiles declare `limits.maxActiveRuns`, `limits.maxQueuedRuns`, and `backpressureMode` (`reject`|`defer`|`throttle`). When limits trigger, exporters emit `429` with `X-Stella-Quota-*` plus `Retry-After` to keep retries deterministic.
## Example profile definition (CLI)

View File

@@ -12,6 +12,7 @@ Export Center runs emit deterministic manifests, provenance records, and signatu
- **Traceability.** Provenance links each bundle to the inputs that produced it: tenant, findings ledger queries, policy snapshots, SBOM identifiers, adapter versions, and encryption recipients.
- **Determinism.** Canonical JSON (sorted keys, RFC3339 UTC timestamps, normalized numbers) guarantees byte-for-byte stability across reruns with identical input.
- **Portability.** Signatures and attestations travel with filesystem bundles, OCI artefacts, and Offline Kit staging trees. Verification does not require online Authority access when the bundle includes the cosign public key.
- **Transparency metadata.** DSSE/SLSA artefacts must embed log metadata (hashedrekord/rekor-style `logId`, `logIndex`, `entryDigest`, `timestamp`) so offline kits can prove submission intent even without online verification.
---
@@ -38,8 +39,9 @@ All digests use lowercase hex SHA-256 (`sha256:<digest>`). When bundle encryptio
- Provenance `subjects[]` contains both manifest hash and bundle/archive hash.
3. **Key retrieval.** Worker obtains a short-lived signing token from Authoritys KMS client using tenant-scoped credentials (`export.sign` scope). Keys live in Authority or tenant-specific HSMs depending on deployment.
4. **Signature emission.** Cosign generates detached signatures (`*.sig`). If DSSE is enabled, cosign wraps payload bytes in a DSSE envelope (`*.dsse`). Attestations follow the SLSA Level2 provenance template; Level3 requires builder metadata (`EXPORT-SVC-37-002` optional feature flag).
5. **Storage & distribution.** Signatures and attestations are written alongside manifests in object storage, included in filesystem bundles, and attached as OCI artefact layers/annotations.
6. **Audit trail.** Run metadata captures signer identity (`signing_key_id`), cosign certificate serial, signature timestamps, and verification hints. Console/CLI surface these details for downstream automation.
5. **Log metadata.** DSSE/SLSA outputs record log hints: `{kind: "hashedrekord", logId, logIndex, entryDigest, timestamp}`. For air-gap deployments the hints ride inside the attestation; when online, the same values come from Rekor receipts.
6. **Storage & distribution.** Signatures and attestations are written alongside manifests in object storage, included in filesystem bundles, and attached as OCI artefact layers/annotations.
7. **Audit trail.** Run metadata captures signer identity (`signing_key_id`), cosign certificate serial, signature timestamps, log hints, and verification outcomes. Console/CLI surface these details for downstream automation.
> **Key management.** Secrets and key references are configured per tenant via `export.signing`, pointing to Authority clients or external HSM aliases. Offline deployments pre-load cosign public keys into the bundle (`signatures/pubkeys/{tenant}.pem`).
@@ -61,6 +63,7 @@ All digests use lowercase hex SHA-256 (`sha256:<digest>`). When bundle encryptio
| `predicate.metadata.reproducible` | Always `true`—workers guarantee determinism. |
| `predicate.environment.encryption` | Records encryption recipients, wrapped keys, algorithm (`age` or `aes-gcm`). |
| `predicate.environment.kms` | Signing key identifier (`authority://tenant/export-signing-key`) and certificate chain fingerprints. |
| `predicate.environment.logs` | Transparency metadata `{kind,logId,logIndex,entryDigest,timestamp}` required by EC3 to keep DSSE verifiable offline. |
Sample (abridged):

View File

@@ -0,0 +1,254 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://stellaops.io/schemas/export-center/export-manifest.schema.json",
"title": "StellaOps Export Manifest",
"description": "Schema for run manifests, attestations, integrity headers, and quota/backpressure metadata (EC2EC9).",
"type": "object",
"required": [
"schema",
"version",
"exportId",
"profile",
"tenant",
"selectors",
"generatedAt",
"contents"
],
"properties": {
"schema": { "type": "string", "const": "https://stellaops.io/export-center/manifest/v1alpha2" },
"version": { "type": "string", "pattern": "^1\\.1\\.[0-9]+$" },
"exportId": { "type": "string", "pattern": "^[a-z0-9-]{6,64}$" },
"runId": { "type": "string", "pattern": "^[a-z0-9-]{6,64}$" },
"profile": {
"type": "object",
"required": ["kind", "variant", "name"],
"properties": {
"kind": { "type": "string", "enum": ["json", "trivy", "mirror", "devportal", "attestation"] },
"variant": {
"type": "string",
"enum": ["raw", "policy", "db", "java-db", "full", "delta", "offline", "bundle"]
},
"name": { "type": "string", "pattern": "^[a-z0-9]([a-z0-9-]*[a-z0-9])?$" },
"revision": { "type": "string", "pattern": "^r[0-9]+$" }
},
"additionalProperties": false
},
"tenant": { "type": "string", "pattern": "^[a-z0-9]([a-z0-9-]*[a-z0-9])?$" },
"selectors": { "$ref": "#/$defs/selectors" },
"generatedAt": { "type": "string", "format": "date-time" },
"rerunHash": { "type": "string", "pattern": "^sha256:[a-f0-9]{64}$" },
"contents": {
"type": "array",
"items": {
"type": "object",
"required": ["path", "digest", "bytes"],
"properties": {
"path": { "type": "string", "pattern": "^[A-Za-z0-9._/-]+$" },
"digest": { "$ref": "#/$defs/digest" },
"bytes": { "type": "integer", "minimum": 0 },
"records": { "type": "integer", "minimum": 0 },
"contentType": { "type": "string" }
},
"additionalProperties": false
}
},
"delta": {
"type": "object",
"required": ["baseExportId", "baseManifestDigest", "tombstones"],
"properties": {
"baseExportId": { "type": "string", "pattern": "^[a-z0-9-]{6,64}$" },
"baseManifestDigest": { "$ref": "#/$defs/digest" },
"tombstones": {
"type": "array",
"items": { "type": "string", "pattern": "^[A-Za-z0-9._/-]+$" }
},
"added": {
"type": "array",
"items": { "type": "string", "pattern": "^[A-Za-z0-9._/-]+$" }
},
"removed": {
"type": "array",
"items": { "type": "string", "pattern": "^[A-Za-z0-9._/-]+$" }
}
},
"additionalProperties": false
},
"integrity": {
"type": "object",
"required": ["httpHeaders", "oci"],
"properties": {
"httpHeaders": {
"type": "object",
"required": ["Digest", "X-Stella-Signature"],
"properties": {
"Digest": { "type": "string", "pattern": "^sha-256=[A-Za-z0-9+/=]+$" },
"X-Stella-Signature": { "type": "string" },
"X-Stella-Immutability": { "type": "string" }
},
"additionalProperties": false
},
"oci": {
"type": "object",
"required": ["annotations"],
"properties": {
"annotations": {
"type": "object",
"required": [
"io.stellaops.export.profile",
"io.stellaops.export.run",
"io.stellaops.export.manifest-digest",
"io.stellaops.export.provenance-ref"
],
"properties": {
"io.stellaops.export.profile": { "type": "string" },
"io.stellaops.export.run": { "type": "string" },
"io.stellaops.export.manifest-digest": { "$ref": "#/$defs/digest" },
"io.stellaops.export.provenance-ref": { "type": "string" },
"org.opencontainers.image.ref.name": { "type": "string" }
}
}
},
"additionalProperties": false
}
},
"additionalProperties": false
},
"attestations": {
"type": "object",
"required": ["provenanceRef", "dsseEnvelope", "slsaLevel", "log"],
"properties": {
"provenanceRef": { "type": "string" },
"dsseEnvelope": { "type": "string" },
"slsaLevel": { "type": "string" },
"log": {
"type": "object",
"required": ["kind", "logId", "logIndex", "entryDigest", "timestamp"],
"properties": {
"kind": { "type": "string", "enum": ["hashedrekord", "rekor"] },
"logId": { "type": "string" },
"logIndex": { "type": "integer", "minimum": 0 },
"entryDigest": { "$ref": "#/$defs/digest" },
"timestamp": { "type": "string", "format": "date-time" }
},
"additionalProperties": false
}
},
"additionalProperties": false
},
"distribution": {
"type": "object",
"properties": {
"http": {
"type": "object",
"properties": {
"enabled": { "type": "boolean" },
"retentionDays": { "type": "integer", "minimum": 1, "maximum": 3650 },
"etag": { "type": "string" },
"rangeRequests": { "type": "boolean" }
},
"additionalProperties": false
},
"oci": {
"type": "object",
"properties": {
"enabled": { "type": "boolean" },
"reference": { "type": "string" }
},
"additionalProperties": false
},
"object": {
"type": "object",
"properties": {
"enabled": { "type": "boolean" },
"bucket": { "type": "string" },
"prefix": { "type": "string" }
},
"additionalProperties": false
}
},
"additionalProperties": false
},
"encryption": {
"type": "object",
"properties": {
"mode": { "type": "string", "enum": ["age", "aes-gcm"] },
"recipients": {
"type": "array",
"items": {
"type": "object",
"required": ["keyId", "fingerprint"],
"properties": {
"keyId": { "type": "string" },
"fingerprint": { "type": "string" },
"wrappedKey": { "type": "string" }
},
"additionalProperties": false
}
},
"strict": { "type": "boolean" }
},
"additionalProperties": false
},
"approval": {
"type": "object",
"properties": {
"required": { "type": "boolean" },
"reason": { "type": "string" },
"approvedBy": { "type": "string" },
"ticket": { "type": "string" }
},
"additionalProperties": false
},
"quotas": {
"type": "object",
"properties": {
"maxActiveRuns": { "type": "integer", "minimum": 1, "maximum": 32 },
"maxQueuedRuns": { "type": "integer", "minimum": 1, "maximum": 512 },
"backpressureMode": {
"type": "string",
"enum": ["reject", "defer", "throttle"]
},
"cpuThrottlePercent": { "type": "integer", "minimum": 1, "maximum": 100 }
},
"additionalProperties": false
}
},
"additionalProperties": false,
"$defs": {
"digest": { "type": "string", "pattern": "^sha256:[a-f0-9]{64}$" },
"selectors": {
"type": "object",
"properties": {
"tenants": {
"type": "array",
"items": { "type": "string", "pattern": "^[a-z0-9*.-]+$" },
"uniqueItems": true
},
"products": {
"type": "array",
"items": { "type": "string", "pattern": "^pkg:[A-Za-z0-9.+\\-_/:@*]+$" }
},
"timeWindow": {
"oneOf": [
{ "type": "string", "pattern": "^[0-9]+d$" },
{ "type": "string", "pattern": "^[0-9]{4}-[0-9]{2}-[0-9]{2}/[0-9]{4}-[0-9]{2}-[0-9]{2}$" }
]
},
"severities": {
"type": "array",
"items": { "type": "string", "enum": ["critical", "high", "medium", "low", "info"] },
"uniqueItems": true
},
"ecosystems": {
"type": "array",
"items": {
"type": "string",
"enum": ["npm", "maven", "pypi", "nuget", "go", "cargo", "rpm", "deb", "apk", "java"]
},
"uniqueItems": true
}
},
"additionalProperties": false
}
}
}

View File

@@ -0,0 +1,206 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://stellaops.io/schemas/export-center/export-profile.schema.json",
"title": "StellaOps ExportProfile",
"description": "Canonical schema for Export Center profile definitions with selector and approval guardrails (EC1, EC4, EC9).",
"type": "object",
"required": ["apiVersion", "kind", "metadata", "spec"],
"properties": {
"apiVersion": {
"type": "string",
"const": "stellaops.io/export.v1"
},
"kind": {
"type": "string",
"const": "ExportProfile"
},
"metadata": {
"type": "object",
"required": ["name", "tenant"],
"properties": {
"name": {
"type": "string",
"minLength": 3,
"maxLength": 64,
"pattern": "^[a-z0-9]([a-z0-9-]*[a-z0-9])?$"
},
"tenant": {
"type": "string",
"pattern": "^[a-z0-9]([a-z0-9-]*[a-z0-9])?$"
},
"revision": {
"type": "string",
"pattern": "^r[0-9]+$"
},
"labels": {
"type": "object",
"additionalProperties": {
"type": "string",
"maxLength": 128
}
}
},
"additionalProperties": false
},
"spec": {
"type": "object",
"required": ["kind", "variant", "distribution"],
"properties": {
"kind": {
"type": "string",
"enum": ["json", "trivy", "mirror", "devportal", "attestation"]
},
"variant": {
"type": "string",
"enum": [
"raw",
"policy",
"db",
"java-db",
"full",
"delta",
"offline",
"bundle"
]
},
"distribution": {
"type": "array",
"items": {
"type": "string",
"enum": ["http", "oci", "object"]
},
"uniqueItems": true,
"minItems": 1
},
"compression": {
"type": "object",
"properties": {
"codec": {
"type": "string",
"enum": ["zstd", "gzip", "none"]
},
"level": {
"type": "integer",
"minimum": 1,
"maximum": 22
}
},
"additionalProperties": false
},
"encryption": {
"type": "object",
"properties": {
"enabled": { "type": "boolean" },
"mode": { "type": "string", "enum": ["age", "aes-gcm"] },
"recipientKeys": {
"type": "array",
"items": {
"type": "string",
"pattern": "^(age1|kms://)"
}
},
"strict": { "type": "boolean" }
},
"additionalProperties": false
},
"retention": {
"type": "object",
"properties": {
"mode": { "type": "string", "enum": ["days", "never"] },
"value": { "type": "integer", "minimum": 1, "maximum": 3650 }
},
"additionalProperties": false
},
"limits": {
"type": "object",
"properties": {
"maxActiveRuns": { "type": "integer", "minimum": 1, "maximum": 32 },
"maxQueuedRuns": { "type": "integer", "minimum": 1, "maximum": 512 },
"backpressureMode": {
"type": "string",
"enum": ["reject", "defer", "throttle"]
}
},
"additionalProperties": false
},
"selectors": { "$ref": "#/$defs/selectors" },
"approval": {
"type": "object",
"properties": {
"required": { "type": "boolean" },
"reason": { "type": "string", "maxLength": 256 },
"ticket": { "type": "string", "maxLength": 64 },
"approver": { "type": "string", "maxLength": 64 }
},
"additionalProperties": false
},
"schemaVersion": {
"type": "string",
"enum": ["1.1.0"],
"default": "1.1.0"
}
},
"additionalProperties": false
}
},
"additionalProperties": false,
"$defs": {
"selectors": {
"type": "object",
"properties": {
"tenants": {
"type": "array",
"items": {
"type": "string",
"pattern": "^[a-z0-9*.-]+$"
},
"uniqueItems": true
},
"products": {
"type": "array",
"items": {
"type": "string",
"pattern": "^pkg:[A-Za-z0-9.+\\-_/:@*]+$"
}
},
"ecosystems": {
"type": "array",
"items": {
"type": "string",
"enum": [
"npm",
"maven",
"pypi",
"nuget",
"go",
"cargo",
"rpm",
"deb",
"apk",
"java"
]
},
"uniqueItems": true
},
"timeWindow": {
"oneOf": [
{ "type": "string", "pattern": "^[0-9]+d$" },
{
"type": "string",
"pattern": "^[0-9]{4}-[0-9]{2}-[0-9]{2}/[0-9]{4}-[0-9]{2}-[0-9]{2}$"
}
]
},
"severities": {
"type": "array",
"items": {
"type": "string",
"enum": ["critical", "high", "medium", "low", "info"]
},
"uniqueItems": true
}
},
"additionalProperties": false
}
}
}

View File

@@ -152,11 +152,11 @@ The Java supplement only includes ecosystems `maven`, `gradle`, `sbt`. Additiona
## 4. Compatibility matrix
| Trivy version | Schema version | Supported by adapter | Notes |
|---------------|----------------|----------------------|-------|
| 0.46.x | 2 | Yes | Baseline compatibility target. |
| 0.50.x | 2 | Yes | Default validation target in CI. |
| 0.51.x+ | 3 | Pending | Adapter throws `ERR_EXPORT_UNSUPPORTED_SCHEMA` until implemented. |
| Trivy version | Schema version | Supported by adapter | Notes |
|---------------|----------------|----------------------|-------|
| 0.46.x | 2 (pinned) | Yes | Baseline compatibility target. |
| 0.50.x | 2 (pinned) | Yes | Default validation target in CI and fixtures. |
| 0.51.x+ | 3 | Pending | Adapter throws `ERR_EXPORT_UNSUPPORTED_SCHEMA` until implemented or explicitly overridden. |
Schema mismatches emit `adapter.trivy.unsupported_schema_version` and abort the run. Operators can pin the schema via `ExportCenter:Adapters:Trivy:SchemaVersion`.
@@ -169,10 +169,13 @@ Schema mismatches emit `adapter.trivy.unsupported_schema_version` and abort the
- Generate bundle from fixture dataset.
- Run `trivy module db import <bundle>` (Trivy CLI) to ensure the bundle is accepted.
- For Java DB, run `trivy java-repo --db <bundle>` against sample repository.
3. **CI smoke (`DEVOPS-EXPORT-36-001`)**:
- Validate metadata fields using `jq`.
- Ensure signatures verify with `cosign`.
- Check runtime by invoking `trivy fs --cache-dir <temp> --skip-update --custom-db <bundle> fixtures/image`.
3. **CI smoke (`DEVOPS-EXPORT-36-001`)**:
- Validate metadata fields using `jq`.
- Ensure signatures verify with `cosign`.
- Check runtime by invoking `trivy fs --cache-dir <temp> --skip-update --custom-db <bundle> fixtures/image`.
4. **Schema pinning (EC6)**:
- CI enforces `ExportCenter:Adapters:Trivy:SchemaVersion=2`; higher versions fail fast with `adapter.trivy.unsupported_schema_version`.
- Export manifests/OCI annotations record the pinned schema for rerun-hash stability.
Failures set the run status to `failed` with `errorCode="adapter-trivy"` so Console/CLI expose the reason.

View File

@@ -15,6 +15,8 @@
- `sbom_snapshots` (immutable versions; tenant + artifact + digest + createdAt)
- `sbom_projections` (materialised views keyed by snapshotId, entrypoint/service node flags)
- `sbom_assets` (asset metadata, criticality/owner/env/exposure; append-only history)
- `sbom_catalog` (console catalog surface; indexed by artifact, scope, license, assetTags.*, createdAt for deterministic pagination)
- `sbom_component_neighbors` (component lookup graph edges; indexed by purl+artifact for cursor pagination)
- `sbom_paths` (resolved dependency paths with runtime flags, blast-radius hints)
- `sbom_events` (outbox for event delivery + watermark/backfill tracking)
@@ -84,7 +86,12 @@ Operational rules:
- Logs: structured, include tenant + artifact digest + sbomVersion; classify ingest failures (schema, storage, orchestrator, validation).
- Alerts: backlog thresholds for outbox/event delivery; high latency on path/timeline endpoints.
## 9) Open questions / dependencies
## 9) Configuration (Mongo-backed catalog & lookup)
- Enable Mongo storage for `/console/sboms` and `/components/lookup` by setting `SbomService:Mongo:ConnectionString` (env: `SBOM_SbomService__Mongo__ConnectionString`).
- Optional overrides: `SbomService:Mongo:Database`, `SbomService:Mongo:CatalogCollection`, `SbomService:Mongo:ComponentLookupCollection`; defaults are `sbom_service`, `sbom_catalog`, `sbom_component_neighbors`.
- When the connection string is absent the service falls back to fixture JSON or deterministic in-memory seeds to keep air-gapped workflows alive.
## 10) Open questions / dependencies
- Confirm orchestrator pause/backfill contract (shared with Runtime & Signals 140-series).
- Finalise storage collection names and indexes (compound on tenant+artifactDigest+version, TTL for transient staging).
- Publish canonical LNM v1 fixtures and JSON schemas for projections and asset metadata.