docs consolidation and others
This commit is contained in:
master
355
docs/modules/provenance/guides/inline-dsse.md
Normal file
355
docs/modules/provenance/guides/inline-dsse.md
Normal file
@@ -0,0 +1,355 @@
|
||||
# Inline DSSE Provenance
|
||||
|
||||
> **Status:** Draft – aligns with the November 2025 advisory “store DSSE attestation refs inline on every SBOM/VEX event node.”
|
||||
> **Owners:** Authority Guild · Feedser Guild · Platform Guild · Docs Guild.
|
||||
|
||||
This document defines how Stella Ops records provenance for SBOM, VEX, scan, and derived events: every event node in the PostgreSQL event store includes DSSE + Rekor references and verification metadata so audits and replay become first-class queries.
|
||||
|
||||
---
|
||||
|
||||
## 1. Event patch (PostgreSQL schema)
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"_id": "evt_...",
|
||||
"kind": "SBOM|VEX|SCAN|INGEST|DERIVED",
|
||||
"subject": {
|
||||
"purl": "pkg:nuget/example@1.2.3",
|
||||
"digest": { "sha256": "..." },
|
||||
"version": "1.2.3"
|
||||
},
|
||||
"provenance": {
|
||||
"dsse": {
|
||||
"envelopeDigest": "sha256:...",
|
||||
"payloadType": "application/vnd.in-toto+json",
|
||||
"key": {
|
||||
"keyId": "cosign:SHA256-PKIX:ABC...",
|
||||
"issuer": "fulcio",
|
||||
"algo": "ECDSA"
|
||||
},
|
||||
"rekor": {
|
||||
"logIndex": 1234567,
|
||||
"uuid": "b3f0...",
|
||||
"integratedTime": 1731081600,
|
||||
"mirrorSeq": 987654 // optional
|
||||
},
|
||||
"chain": [
|
||||
{ "type": "build", "id": "att:build#...", "digest": "sha256:..." },
|
||||
{ "type": "sbom", "id": "att:sbom#...", "digest": "sha256:..." }
|
||||
]
|
||||
}
|
||||
},
|
||||
"trust": {
|
||||
"verified": true,
|
||||
"verifier": "Authority@stella",
|
||||
"witnesses": 1,
|
||||
"policyScore": 0.92
|
||||
},
|
||||
"ts": "2025-11-11T12:00:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
### Key fields
|
||||
|
||||
| Field | Description |
|
||||
|-------|-------------|
|
||||
| `provenance.dsse.envelopeDigest` | SHA-256 of the DSSE envelope (not payload). |
|
||||
| `provenance.dsse.payloadType` | Usually `application/vnd.in-toto+json`. |
|
||||
| `provenance.dsse.key` | Key fingerprint / issuer / algorithm. |
|
||||
| `provenance.dsse.rekor` | Rekor transparency log metadata (index, UUID, integrated time). |
|
||||
| `provenance.dsse.chain` | Optional chain of dependent attestations (build → sbom → scan). |
|
||||
| `trust.*` | Result of local verification (DSSE signature, Rekor proof, policy). |
|
||||
|
||||
---
|
||||
|
||||
## 2. Write path (ingest flow)
|
||||
|
||||
1. **Obtain provenance metadata** for each attested artifact (build, SBOM, VEX, scan). The CI script (`scripts/publish_attestation_with_provenance.sh`) captures `envelopeDigest`, Rekor `logIndex`/`uuid`, and key info.
|
||||
2. **Authority/Feedser** verify the DSSE + Rekor proof (local cosign/rekor libs or the Signer service) and set `trust.verified = true`, `trust.verifier = "Authority@stella"`, `trust.witnesses = 1`.
|
||||
3. **Attach** the provenance block before appending the event to PostgreSQL, using `StellaOps.Provenance.Postgres` helpers.
|
||||
4. **Backfill** historical events by resolving known subjects → attestation digests and running an update script.
|
||||
|
||||
### 2.1 Supplying metadata from Concelier statements
|
||||
|
||||
Concelier ingestion jobs can now inline provenance when they create advisory statements. Add an `AdvisoryProvenance` entry with `kind = "dsse"` (or `dsse-metadata` / `attestation-dsse`) and set `value` to the same JSON emitted by the CI snippet. `AdvisoryEventLog` and `AdvisoryMergeService` automatically parse that entry, hydrate `AdvisoryStatementInput.Provenance/Trust`, and persist the metadata alongside the statement.
|
||||
|
||||
```json
|
||||
{
|
||||
"source": "attestor",
|
||||
"kind": "dsse",
|
||||
"value": "{ \"dsse\": { \"envelopeDigest\": \"sha256:…\", \"payloadType\": \"application/vnd.in-toto+json\" }, \"trust\": { \"verified\": true, \"verifier\": \"Authority@stella\" } }",
|
||||
"recordedAt": "2025-11-10T00:00:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
Providing the metadata during ingestion keeps new statements self-contained and reduces the surface that the `/events/statements/{statementId}/provenance` endpoint needs to backfill later.
|
||||
|
||||
Reference helper: `src/__Libraries/StellaOps.Provenance.Postgres/ProvenancePostgresExtensions.cs`.
|
||||
|
||||
---
|
||||
|
||||
### 2.2 Advisory AI structured chunk schema (GHSA/Cisco parity)
|
||||
|
||||
Advisory AI consumes the canonical `Advisory` aggregate and emits structured chunks that mirror GHSA GraphQL and Cisco PSIRT provenance anchors. The response contract is:
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"advisoryKey": "CVE-2025-0001",
|
||||
"fingerprint": "<sha256 of canonical advisory>",
|
||||
"total": 3,
|
||||
"truncated": false,
|
||||
"entries": [
|
||||
{
|
||||
"type": "workaround", // sorted by (type, observationPath, documentId)
|
||||
"chunkId": "c0ffee12", // sha256(advisory.observationId + observationPath)[:16]
|
||||
"content": { /* structured field */ },
|
||||
"provenance": {
|
||||
"documentId": "tenant-a:chunk:newest", // PostgreSQL id of backing observation
|
||||
"observationPath": "/references/0", // JSON Pointer into the observation
|
||||
"source": "nvd",
|
||||
"kind": "workaround",
|
||||
"value": "tenant-a:chunk:newest",
|
||||
"recordedAt": "2025-01-07T00:00:00Z",
|
||||
"fieldMask": ["/references/0"]
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Determinism requirements:
|
||||
|
||||
- Order entries by `(type, observationPath, documentId)` to keep cache keys stable across nodes.
|
||||
- Always include the advisory `fingerprint` in cache keys and responses.
|
||||
- Preserve observation-level provenance by emitting both `documentId` and `observationPath` under `provenance`.
|
||||
|
||||
These anchors let Attestor/Console deep-link evidence and allow offline mirrors to prove origin without merging transforms.
|
||||
|
||||
---
|
||||
|
||||
## 3. CI/CD snippet
|
||||
|
||||
See `scripts/publish_attestation_with_provenance.sh`:
|
||||
|
||||
```bash
|
||||
rekor-cli upload --rekor_server "$REKOR_URL" \
|
||||
--artifact "$ATTEST_PATH" --type dsse --format json > rekor-upload.json
|
||||
LOG_INDEX=$(jq '.LogIndex' rekor-upload.json)
|
||||
UUID=$(jq -r '.UUID' rekor-upload.json)
|
||||
ENVELOPE_SHA256=$(sha256sum "$ATTEST_PATH" | awk '{print $1}')
|
||||
cat > provenance-meta.json <<EOF
|
||||
{
|
||||
"subject": { "imageRef": "$IMAGE_REF", "digest": { "sha256": "$IMAGE_DIGEST" } },
|
||||
"dsse": {
|
||||
"envelopeDigest": "sha256:$ENVELOPE_SHA256",
|
||||
"payloadType": "application/vnd.in-toto+json",
|
||||
"key": { "keyId": "$KEY_ID", "issuer": "$KEY_ISSUER", "algo": "$KEY_ALGO" },
|
||||
"rekor": { "logIndex": $LOG_INDEX, "uuid": "$UUID", "integratedTime": $(jq '.IntegratedTime' rekor-upload.json) }
|
||||
}
|
||||
}
|
||||
EOF
|
||||
```
|
||||
|
||||
Feedser ingests this JSON and maps it to `DsseProvenance` + `TrustInfo`.
|
||||
|
||||
---
|
||||
|
||||
## 4. PostgreSQL indexes
|
||||
|
||||
Create indexes to keep provenance queries fast (PostgreSQL DDL):
|
||||
|
||||
```sql
|
||||
-- events_by_subject_kind_provenance
|
||||
CREATE INDEX events_by_subject_kind_provenance
|
||||
ON events (subject_digest_sha256, kind, provenance_dsse_rekor_log_index);
|
||||
|
||||
-- events_unproven_by_kind
|
||||
CREATE INDEX events_unproven_by_kind
|
||||
ON events (kind, trust_verified, provenance_dsse_rekor_log_index);
|
||||
|
||||
-- events_by_rekor_logindex
|
||||
CREATE INDEX events_by_rekor_logindex
|
||||
ON events (provenance_dsse_rekor_log_index);
|
||||
|
||||
-- events_by_envelope_digest (partial index for non-null values)
|
||||
CREATE INDEX events_by_envelope_digest
|
||||
ON events (provenance_dsse_envelope_digest)
|
||||
WHERE provenance_dsse_envelope_digest IS NOT NULL;
|
||||
|
||||
-- events_by_ts_kind_verified
|
||||
CREATE INDEX events_by_ts_kind_verified
|
||||
ON events (ts DESC, kind, trust_verified);
|
||||
```
|
||||
|
||||
Deployment options:
|
||||
- **Ops script:** `psql -d stellaops_db -f ops/postgres/indices/events_provenance_indices.sql`
|
||||
- **C# helper:** `PostgresIndexes.EnsureEventIndexesAsync(connection, ct)`
|
||||
|
||||
This section was updated as part of `PROV-INDEX-401-030` (completed 2025-11-27).
|
||||
|
||||
---
|
||||
|
||||
## 5. Query recipes
|
||||
|
||||
* **All proven VEX for an image digest:**
|
||||
|
||||
```sql
|
||||
SELECT * FROM events
|
||||
WHERE kind = 'VEX'
|
||||
AND subject_digest_sha256 = '<digest>'
|
||||
AND provenance_dsse_rekor_log_index IS NOT NULL
|
||||
AND trust_verified = true;
|
||||
```
|
||||
|
||||
* **Compliance gap (unverified data used for decisions):**
|
||||
|
||||
```sql
|
||||
SELECT kind, COUNT(*) as count
|
||||
FROM events
|
||||
WHERE kind IN ('VEX', 'SBOM', 'SCAN')
|
||||
AND (trust_verified IS NOT TRUE
|
||||
OR provenance_dsse_rekor_log_index IS NULL)
|
||||
GROUP BY kind;
|
||||
```
|
||||
|
||||
* **Replay slice:** filter for events where `provenance.dsse.chain` covers build → sbom → scan and export referenced attestation digests.
|
||||
|
||||
---
|
||||
|
||||
## 6. Policy gates
|
||||
|
||||
Examples:
|
||||
|
||||
```yaml
|
||||
rules:
|
||||
- id: GATE-PROVEN-VEX
|
||||
when:
|
||||
all:
|
||||
- kind: "VEX"
|
||||
- trust.verified: true
|
||||
- key.keyId in VendorAllowlist
|
||||
- rekor.integratedTime <= releaseFreeze
|
||||
then:
|
||||
decision: ALLOW
|
||||
|
||||
- id: BLOCK-UNPROVEN
|
||||
when:
|
||||
any:
|
||||
- trust.verified != true
|
||||
- provenance.dsse.rekor.logIndex missing
|
||||
then:
|
||||
decision: FAIL
|
||||
reason: "Unproven evidence influences decision; require Rekor-backed attestation."
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7. UI nudges
|
||||
|
||||
* **Provenance chip** on findings/events: `Verified • Rekor#1234567 • KeyID:cosign:...` (click → inclusion proof & DSSE preview).
|
||||
* Facet filter: `Provenance = Verified / Missing / Key-Policy-Mismatch`.
|
||||
|
||||
---
|
||||
|
||||
## 8. Implementation tasks
|
||||
|
||||
| Task ID | Scope |
|
||||
|---------|-------|
|
||||
| `PROV-INLINE-401-028` | Extend Authority/Feedser write-paths to attach `provenance.dsse` + `trust` blocks using `StellaOps.Provenance.Postgres`. |
|
||||
| `PROV-BACKFILL-401-029` | Backfill historical events with DSSE/Rekor refs based on existing attestation digests. |
|
||||
| `PROV-INDEX-401-030` | Create PostgreSQL indexes and expose helper queries for audits. |
|
||||
|
||||
Keep this document updated when new attestation types or mirror/witness policies land.
|
||||
|
||||
---
|
||||
|
||||
## 9. Feedser API for provenance updates
|
||||
|
||||
Feedser exposes a lightweight endpoint for attaching provenance after an event is recorded:
|
||||
|
||||
```
|
||||
POST /events/statements/{statementId}/provenance
|
||||
Headers: X-Stella-Tenant, Authorization (if Authority is enabled)
|
||||
Body: { "dsse": { ... }, "trust": { ... } }
|
||||
```
|
||||
|
||||
The body matches the JSON emitted by `publish_attestation_with_provenance.sh`. Feedser validates the payload, ensures `trust.verified = true`, and then calls `AttachStatementProvenanceAsync` so the DSSE metadata lands inline on the target statement. Clients receive HTTP 202 on success, 400 on malformed input, and 404 if the statement id is unknown.
|
||||
|
||||
---
|
||||
|
||||
## 10. Backfill service
|
||||
|
||||
`EventProvenanceBackfillService` (`src/StellaOps.Events.Postgres/EventProvenanceBackfillService.cs`) orchestrates backfilling historical events with DSSE provenance metadata.
|
||||
|
||||
### 10.1 Components
|
||||
|
||||
| Class | Purpose |
|
||||
|-------|---------|
|
||||
| `IAttestationResolver` | Interface for resolving attestation metadata by subject digest. |
|
||||
| `EventProvenanceBackfillService` | Queries unproven events, resolves attestations, updates events. |
|
||||
| `StubAttestationResolver` | Test/development stub implementation. |
|
||||
|
||||
### 10.2 Usage
|
||||
|
||||
```csharp
|
||||
var resolver = new MyAttestationResolver(rekorClient, attestationRepo);
|
||||
var backfillService = new EventProvenanceBackfillService(postgresConnection, resolver);
|
||||
|
||||
// Count unproven events
|
||||
var count = await backfillService.CountUnprovenEventsAsync(
|
||||
new[] { "SBOM", "VEX", "SCAN" });
|
||||
|
||||
// Backfill with progress reporting
|
||||
var progress = new Progress<BackfillResult>(r =>
|
||||
Console.WriteLine($"{r.EventId}: {r.Status}"));
|
||||
|
||||
var summary = await backfillService.BackfillAllAsync(
|
||||
kinds: new[] { "SBOM", "VEX", "SCAN" },
|
||||
limit: 1000,
|
||||
progress: progress);
|
||||
|
||||
Console.WriteLine($"Processed: {summary.TotalProcessed}");
|
||||
Console.WriteLine($"Success: {summary.SuccessCount}");
|
||||
Console.WriteLine($"Not found: {summary.NotFoundCount}");
|
||||
Console.WriteLine($"Errors: {summary.ErrorCount}");
|
||||
```
|
||||
|
||||
### 10.3 Implementing IAttestationResolver
|
||||
|
||||
Implementations should query the attestation store (Rekor, CAS, or local PostgreSQL) by subject digest:
|
||||
|
||||
```csharp
|
||||
public class RekorAttestationResolver : IAttestationResolver
|
||||
{
|
||||
private readonly IRekorClient _rekor;
|
||||
private readonly IAttestationRepository _attestations;
|
||||
|
||||
public async Task<AttestationResolution?> ResolveAsync(
|
||||
string subjectDigestSha256,
|
||||
string eventKind,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
// Look up attestation by subject digest
|
||||
var record = await _attestations.GetAsync(subjectDigestSha256, eventKind, cancellationToken);
|
||||
if (record is null) return null;
|
||||
|
||||
// Fetch Rekor proof if available
|
||||
var proof = await _rekor.GetProofAsync(record.RekorUuid, RekorBackend.Sigstore, cancellationToken);
|
||||
|
||||
return new AttestationResolution
|
||||
{
|
||||
Dsse = new DsseProvenance { /* ... */ },
|
||||
Trust = new TrustInfo { Verified = true, Verifier = "Authority@stella" },
|
||||
AttestationId = record.Id
|
||||
};
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 10.4 Reference files
|
||||
|
||||
- `src/StellaOps.Events.Postgres/IAttestationResolver.cs`
|
||||
- `src/StellaOps.Events.Postgres/EventProvenanceBackfillService.cs`
|
||||
- `src/StellaOps.Events.Postgres/StubAttestationResolver.cs`
|
||||
|
||||
This section was added as part of `PROV-BACKFILL-401-029` (completed 2025-11-27).
|
||||
16
docs/modules/provenance/guides/prov-backfill-plan.md
Normal file
16
docs/modules/provenance/guides/prov-backfill-plan.md
Normal file
@@ -0,0 +1,16 @@
|
||||
# Provenance Backfill Plan (Sprint 401)
|
||||
|
||||
Artifacts available
|
||||
- Attestation inventory: `docs/modules/provenance/guides/attestation-inventory-2025-11-18.ndjson`
|
||||
- Subject→Rekor map: `docs/modules/provenance/guides/subject-rekor-map-2025-11-18.json`
|
||||
|
||||
Procedure (deterministic)
|
||||
1) Load inventory NDJSON; validate UUID/ULID and digest formats.
|
||||
2) For each record, resolve Rekor entry via the subject→Rekor map; if missing, record gap and skip write.
|
||||
3) Emit backfilled events to the provenance store using `scripts/publish_attestation_with_provenance.sh --mode backfill` (add `--subject` and `--rekor` arguments) with sorted input to guarantee stable ordering.
|
||||
4) Log every backfilled subject + Rekor digest pair to `logs/provenance-backfill-2025-11-18.ndjson` (UTC timestamps, ISO-8601).
|
||||
5) Rerun until gaps are zero; then mark PROV-BACKFILL-401-029 DONE.
|
||||
|
||||
Determinism
|
||||
- Sort by subject, then rekorEntry before processing.
|
||||
- Use canonical JSON writer for outputs; timestamps in UTC `O` format.
|
||||
76
docs/modules/provenance/guides/provenance-attestation.md
Normal file
76
docs/modules/provenance/guides/provenance-attestation.md
Normal file
@@ -0,0 +1,76 @@
|
||||
# Provenance & Attestation Reference
|
||||
|
||||
> **Imposed rule:** All exported evidence must ship with DSSE + transparency proof bundles; unsigned or proof-less artifacts are rejected at ingress and may not be stored in the Evidence Locker.
|
||||
|
||||
This guide explains how StellaOps generates, signs, verifies, and distributes DSSE attestations for SBOMs, policy evaluations, and runtime evidence.
|
||||
|
||||
## 1. Attestation Workflow (online and offline)
|
||||
|
||||
1. **Producer** (Scanner, Policy Engine, runtime probes) emits a payload and a request to sign.
|
||||
2. **Signer** authenticates the caller, validates supply-chain policy (release integrity, image pinning), then signs using keyless or tenant KMS keys.
|
||||
3. **Attestor** wraps the payload in DSSE, records it in Rekor v2 (when online), persists the bundle plus inclusion proof, and exposes a verification package API.
|
||||
4. **Export Center** and **Evidence Locker** embed the bundle and proof into export artifacts for offline replay; CLI retrieves the same package via `stella attest fetch`.
|
||||
5. **Verifiers** (CLI, Policy Engine, auditors) validate signature roots, Rekor proof, and optional transparency witness endorsements.
|
||||
|
||||
## 2. DSSE Payload Types & Schemas
|
||||
|
||||
Supported payload types (all versioned and protobuf/JSON dual-encoded):
|
||||
|
||||
- `StellaOps.BuildProvenance@1`
|
||||
- `StellaOps.SBOMAttestation@1`
|
||||
- `StellaOps.ScanResults@1`
|
||||
- `StellaOps.PolicyEvaluation@1`
|
||||
- `StellaOps.VEXAttestation@1`
|
||||
- `StellaOps.RiskProfileEvidence@1`
|
||||
- `StellaOps.PromotionAttestation@1` (predicate `stella.ops/promotion@v1`, see `docs/release/promotion-attestations.md`)
|
||||
|
||||
Schema sources: `src/Attestor/StellaOps.Attestor.Types` and module dossiers. All payloads include:
|
||||
|
||||
- `subject` (digest + PURL/NEVRA coordinates)
|
||||
- `timestamp` (UTC, ISO-8601)
|
||||
- `producer` (service + version)
|
||||
- `critical` block (policy version, scanner defs, reachability context)
|
||||
- `materials` (SBOM/VEX references) and optional `auxiliary_proofs`
|
||||
|
||||
## 3. Signing & storage controls
|
||||
|
||||
- **Key policy:** Short-lived OIDC keyless by default; tenant KMS allowed; Ed25519 and ECDSA P-256 supported.
|
||||
- **Inclusion:** Rekor v2 UUID + log index cached; when offline, the Attestor stamps a `transparency_pending` marker to be replayed later.
|
||||
- **WORM:** Evidence Locker keeps immutable copies; retention and legal hold are enforced per tenant and surfaced in `docs/modules/evidence-locker/guides/evidence-locker.md`.
|
||||
- **Redaction:** Sensitive fields (secrets, PII) must be excluded at payload creation; the signer refuses payloads marked `pii=true` without a redaction ticket.
|
||||
|
||||
## 4. Verification workflow
|
||||
|
||||
Command-line (online or offline bundle):
|
||||
|
||||
```sh
|
||||
stella attest verify \
|
||||
--bundle path/to/bundle.dsse.json \
|
||||
--rekor-root pubkeys/rekor.pub \
|
||||
--fulcio-root pubkeys/fulcio.pub \
|
||||
--certificate-chain pubkeys/issuer-chain.pem
|
||||
```
|
||||
|
||||
Verification steps performed by services and CLI:
|
||||
|
||||
- Validate DSSE signature against Fulcio/tenant roots and certificate policies.
|
||||
- Confirm subject digest matches expected container/image/SBOM digest.
|
||||
- Check Rekor inclusion proof and (if present) transparency witness signatures.
|
||||
- Enforce freshness: reject bundles older than `attestation.max_age_days` (tenant policy).
|
||||
- Record verification result into Timeline events for auditability.
|
||||
|
||||
## 5. Offline / air-gap posture
|
||||
|
||||
- Export Center emits self-contained bundles (`*.dsse.json`, `rekor-proof.json`, `cert-chain.pem`) plus a verification manifest for deterministic replay.
|
||||
- CLI `stella attest verify --bundle bundle.dsse.json --offline` skips Rekor lookups and relies on embedded proofs.
|
||||
- When connectivity returns, the Attestor replays pending `transparency_pending` entries and updates Evidence Locker indexes; Timeline events capture the replay.
|
||||
|
||||
## 6. References
|
||||
|
||||
- `docs/modules/signer/architecture.md`
|
||||
- `docs/modules/attestor/architecture.md`
|
||||
- `docs/modules/export-center/architecture.md`
|
||||
- `docs/modules/policy/architecture.md`
|
||||
- `docs/modules/telemetry/architecture.md`
|
||||
- `docs/modules/evidence-locker/guides/evidence-locker.md`
|
||||
- `src/Provenance/StellaOps.Provenance.Attestation`
|
||||
Reference in New Issue
Block a user