8bbfe4d2d2
- Add RateLimitConfig for configuration management with YAML binding support. - Introduce RateLimitDecision to encapsulate the result of rate limit checks. - Implement RateLimitMetrics for OpenTelemetry metrics tracking. - Create RateLimitMiddleware for enforcing rate limits on incoming requests. - Develop RateLimitService to orchestrate instance and environment rate limit checks. - Add RateLimitServiceCollectionExtensions for dependency injection registration.
356 lines
12 KiB
Markdown
356 lines
12 KiB
Markdown
# Inline DSSE Provenance
|
||
|
||
> **Status:** Draft – aligns with the November 2025 advisory “store DSSE attestation refs inline on every SBOM/VEX event node.”
|
||
> **Owners:** Authority Guild · Feedser Guild · Platform Guild · Docs Guild.
|
||
|
||
This document defines how Stella Ops records provenance for SBOM, VEX, scan, and derived events: every event node in the PostgreSQL event store includes DSSE + Rekor references and verification metadata so audits and replay become first-class queries.
|
||
|
||
---
|
||
|
||
## 1. Event patch (PostgreSQL schema)
|
||
|
||
```jsonc
|
||
{
|
||
"_id": "evt_...",
|
||
"kind": "SBOM|VEX|SCAN|INGEST|DERIVED",
|
||
"subject": {
|
||
"purl": "pkg:nuget/example@1.2.3",
|
||
"digest": { "sha256": "..." },
|
||
"version": "1.2.3"
|
||
},
|
||
"provenance": {
|
||
"dsse": {
|
||
"envelopeDigest": "sha256:...",
|
||
"payloadType": "application/vnd.in-toto+json",
|
||
"key": {
|
||
"keyId": "cosign:SHA256-PKIX:ABC...",
|
||
"issuer": "fulcio",
|
||
"algo": "ECDSA"
|
||
},
|
||
"rekor": {
|
||
"logIndex": 1234567,
|
||
"uuid": "b3f0...",
|
||
"integratedTime": 1731081600,
|
||
"mirrorSeq": 987654 // optional
|
||
},
|
||
"chain": [
|
||
{ "type": "build", "id": "att:build#...", "digest": "sha256:..." },
|
||
{ "type": "sbom", "id": "att:sbom#...", "digest": "sha256:..." }
|
||
]
|
||
}
|
||
},
|
||
"trust": {
|
||
"verified": true,
|
||
"verifier": "Authority@stella",
|
||
"witnesses": 1,
|
||
"policyScore": 0.92
|
||
},
|
||
"ts": "2025-11-11T12:00:00Z"
|
||
}
|
||
```
|
||
|
||
### Key fields
|
||
|
||
| Field | Description |
|
||
|-------|-------------|
|
||
| `provenance.dsse.envelopeDigest` | SHA-256 of the DSSE envelope (not payload). |
|
||
| `provenance.dsse.payloadType` | Usually `application/vnd.in-toto+json`. |
|
||
| `provenance.dsse.key` | Key fingerprint / issuer / algorithm. |
|
||
| `provenance.dsse.rekor` | Rekor transparency log metadata (index, UUID, integrated time). |
|
||
| `provenance.dsse.chain` | Optional chain of dependent attestations (build → sbom → scan). |
|
||
| `trust.*` | Result of local verification (DSSE signature, Rekor proof, policy). |
|
||
|
||
---
|
||
|
||
## 2. Write path (ingest flow)
|
||
|
||
1. **Obtain provenance metadata** for each attested artifact (build, SBOM, VEX, scan). The CI script (`scripts/publish_attestation_with_provenance.sh`) captures `envelopeDigest`, Rekor `logIndex`/`uuid`, and key info.
|
||
2. **Authority/Feedser** verify the DSSE + Rekor proof (local cosign/rekor libs or the Signer service) and set `trust.verified = true`, `trust.verifier = "Authority@stella"`, `trust.witnesses = 1`.
|
||
3. **Attach** the provenance block before appending the event to PostgreSQL, using `StellaOps.Provenance.Postgres` helpers.
|
||
4. **Backfill** historical events by resolving known subjects → attestation digests and running an update script.
|
||
|
||
### 2.1 Supplying metadata from Concelier statements
|
||
|
||
Concelier ingestion jobs can now inline provenance when they create advisory statements. Add an `AdvisoryProvenance` entry with `kind = "dsse"` (or `dsse-metadata` / `attestation-dsse`) and set `value` to the same JSON emitted by the CI snippet. `AdvisoryEventLog` and `AdvisoryMergeService` automatically parse that entry, hydrate `AdvisoryStatementInput.Provenance/Trust`, and persist the metadata alongside the statement.
|
||
|
||
```json
|
||
{
|
||
"source": "attestor",
|
||
"kind": "dsse",
|
||
"value": "{ \"dsse\": { \"envelopeDigest\": \"sha256:…\", \"payloadType\": \"application/vnd.in-toto+json\" }, \"trust\": { \"verified\": true, \"verifier\": \"Authority@stella\" } }",
|
||
"recordedAt": "2025-11-10T00:00:00Z"
|
||
}
|
||
```
|
||
|
||
Providing the metadata during ingestion keeps new statements self-contained and reduces the surface that the `/events/statements/{statementId}/provenance` endpoint needs to backfill later.
|
||
|
||
Reference helper: `src/__Libraries/StellaOps.Provenance.Postgres/ProvenancePostgresExtensions.cs`.
|
||
|
||
---
|
||
|
||
### 2.2 Advisory AI structured chunk schema (GHSA/Cisco parity)
|
||
|
||
Advisory AI consumes the canonical `Advisory` aggregate and emits structured chunks that mirror GHSA GraphQL and Cisco PSIRT provenance anchors. The response contract is:
|
||
|
||
```jsonc
|
||
{
|
||
"advisoryKey": "CVE-2025-0001",
|
||
"fingerprint": "<sha256 of canonical advisory>",
|
||
"total": 3,
|
||
"truncated": false,
|
||
"entries": [
|
||
{
|
||
"type": "workaround", // sorted by (type, observationPath, documentId)
|
||
"chunkId": "c0ffee12", // sha256(advisory.observationId + observationPath)[:16]
|
||
"content": { /* structured field */ },
|
||
"provenance": {
|
||
"documentId": "tenant-a:chunk:newest", // PostgreSQL id of backing observation
|
||
"observationPath": "/references/0", // JSON Pointer into the observation
|
||
"source": "nvd",
|
||
"kind": "workaround",
|
||
"value": "tenant-a:chunk:newest",
|
||
"recordedAt": "2025-01-07T00:00:00Z",
|
||
"fieldMask": ["/references/0"]
|
||
}
|
||
}
|
||
]
|
||
}
|
||
```
|
||
|
||
Determinism requirements:
|
||
|
||
- Order entries by `(type, observationPath, documentId)` to keep cache keys stable across nodes.
|
||
- Always include the advisory `fingerprint` in cache keys and responses.
|
||
- Preserve observation-level provenance by emitting both `documentId` and `observationPath` under `provenance`.
|
||
|
||
These anchors let Attestor/Console deep-link evidence and allow offline mirrors to prove origin without merging transforms.
|
||
|
||
---
|
||
|
||
## 3. CI/CD snippet
|
||
|
||
See `scripts/publish_attestation_with_provenance.sh`:
|
||
|
||
```bash
|
||
rekor-cli upload --rekor_server "$REKOR_URL" \
|
||
--artifact "$ATTEST_PATH" --type dsse --format json > rekor-upload.json
|
||
LOG_INDEX=$(jq '.LogIndex' rekor-upload.json)
|
||
UUID=$(jq -r '.UUID' rekor-upload.json)
|
||
ENVELOPE_SHA256=$(sha256sum "$ATTEST_PATH" | awk '{print $1}')
|
||
cat > provenance-meta.json <<EOF
|
||
{
|
||
"subject": { "imageRef": "$IMAGE_REF", "digest": { "sha256": "$IMAGE_DIGEST" } },
|
||
"dsse": {
|
||
"envelopeDigest": "sha256:$ENVELOPE_SHA256",
|
||
"payloadType": "application/vnd.in-toto+json",
|
||
"key": { "keyId": "$KEY_ID", "issuer": "$KEY_ISSUER", "algo": "$KEY_ALGO" },
|
||
"rekor": { "logIndex": $LOG_INDEX, "uuid": "$UUID", "integratedTime": $(jq '.IntegratedTime' rekor-upload.json) }
|
||
}
|
||
}
|
||
EOF
|
||
```
|
||
|
||
Feedser ingests this JSON and maps it to `DsseProvenance` + `TrustInfo`.
|
||
|
||
---
|
||
|
||
## 4. PostgreSQL indexes
|
||
|
||
Create indexes to keep provenance queries fast (PostgreSQL DDL):
|
||
|
||
```sql
|
||
-- events_by_subject_kind_provenance
|
||
CREATE INDEX events_by_subject_kind_provenance
|
||
ON events (subject_digest_sha256, kind, provenance_dsse_rekor_log_index);
|
||
|
||
-- events_unproven_by_kind
|
||
CREATE INDEX events_unproven_by_kind
|
||
ON events (kind, trust_verified, provenance_dsse_rekor_log_index);
|
||
|
||
-- events_by_rekor_logindex
|
||
CREATE INDEX events_by_rekor_logindex
|
||
ON events (provenance_dsse_rekor_log_index);
|
||
|
||
-- events_by_envelope_digest (partial index for non-null values)
|
||
CREATE INDEX events_by_envelope_digest
|
||
ON events (provenance_dsse_envelope_digest)
|
||
WHERE provenance_dsse_envelope_digest IS NOT NULL;
|
||
|
||
-- events_by_ts_kind_verified
|
||
CREATE INDEX events_by_ts_kind_verified
|
||
ON events (ts DESC, kind, trust_verified);
|
||
```
|
||
|
||
Deployment options:
|
||
- **Ops script:** `psql -d stellaops_db -f ops/postgres/indices/events_provenance_indices.sql`
|
||
- **C# helper:** `PostgresIndexes.EnsureEventIndexesAsync(connection, ct)`
|
||
|
||
This section was updated as part of `PROV-INDEX-401-030` (completed 2025-11-27).
|
||
|
||
---
|
||
|
||
## 5. Query recipes
|
||
|
||
* **All proven VEX for an image digest:**
|
||
|
||
```sql
|
||
SELECT * FROM events
|
||
WHERE kind = 'VEX'
|
||
AND subject_digest_sha256 = '<digest>'
|
||
AND provenance_dsse_rekor_log_index IS NOT NULL
|
||
AND trust_verified = true;
|
||
```
|
||
|
||
* **Compliance gap (unverified data used for decisions):**
|
||
|
||
```sql
|
||
SELECT kind, COUNT(*) as count
|
||
FROM events
|
||
WHERE kind IN ('VEX', 'SBOM', 'SCAN')
|
||
AND (trust_verified IS NOT TRUE
|
||
OR provenance_dsse_rekor_log_index IS NULL)
|
||
GROUP BY kind;
|
||
```
|
||
|
||
* **Replay slice:** filter for events where `provenance.dsse.chain` covers build → sbom → scan and export referenced attestation digests.
|
||
|
||
---
|
||
|
||
## 6. Policy gates
|
||
|
||
Examples:
|
||
|
||
```yaml
|
||
rules:
|
||
- id: GATE-PROVEN-VEX
|
||
when:
|
||
all:
|
||
- kind: "VEX"
|
||
- trust.verified: true
|
||
- key.keyId in VendorAllowlist
|
||
- rekor.integratedTime <= releaseFreeze
|
||
then:
|
||
decision: ALLOW
|
||
|
||
- id: BLOCK-UNPROVEN
|
||
when:
|
||
any:
|
||
- trust.verified != true
|
||
- provenance.dsse.rekor.logIndex missing
|
||
then:
|
||
decision: FAIL
|
||
reason: "Unproven evidence influences decision; require Rekor-backed attestation."
|
||
```
|
||
|
||
---
|
||
|
||
## 7. UI nudges
|
||
|
||
* **Provenance chip** on findings/events: `Verified • Rekor#1234567 • KeyID:cosign:...` (click → inclusion proof & DSSE preview).
|
||
* Facet filter: `Provenance = Verified / Missing / Key-Policy-Mismatch`.
|
||
|
||
---
|
||
|
||
## 8. Implementation tasks
|
||
|
||
| Task ID | Scope |
|
||
|---------|-------|
|
||
| `PROV-INLINE-401-028` | Extend Authority/Feedser write-paths to attach `provenance.dsse` + `trust` blocks using `StellaOps.Provenance.Postgres`. |
|
||
| `PROV-BACKFILL-401-029` | Backfill historical events with DSSE/Rekor refs based on existing attestation digests. |
|
||
| `PROV-INDEX-401-030` | Create PostgreSQL indexes and expose helper queries for audits. |
|
||
|
||
Keep this document updated when new attestation types or mirror/witness policies land.
|
||
|
||
---
|
||
|
||
## 9. Feedser API for provenance updates
|
||
|
||
Feedser exposes a lightweight endpoint for attaching provenance after an event is recorded:
|
||
|
||
```
|
||
POST /events/statements/{statementId}/provenance
|
||
Headers: X-Stella-Tenant, Authorization (if Authority is enabled)
|
||
Body: { "dsse": { ... }, "trust": { ... } }
|
||
```
|
||
|
||
The body matches the JSON emitted by `publish_attestation_with_provenance.sh`. Feedser validates the payload, ensures `trust.verified = true`, and then calls `AttachStatementProvenanceAsync` so the DSSE metadata lands inline on the target statement. Clients receive HTTP 202 on success, 400 on malformed input, and 404 if the statement id is unknown.
|
||
|
||
---
|
||
|
||
## 10. Backfill service
|
||
|
||
`EventProvenanceBackfillService` (`src/StellaOps.Events.Postgres/EventProvenanceBackfillService.cs`) orchestrates backfilling historical events with DSSE provenance metadata.
|
||
|
||
### 10.1 Components
|
||
|
||
| Class | Purpose |
|
||
|-------|---------|
|
||
| `IAttestationResolver` | Interface for resolving attestation metadata by subject digest. |
|
||
| `EventProvenanceBackfillService` | Queries unproven events, resolves attestations, updates events. |
|
||
| `StubAttestationResolver` | Test/development stub implementation. |
|
||
|
||
### 10.2 Usage
|
||
|
||
```csharp
|
||
var resolver = new MyAttestationResolver(rekorClient, attestationRepo);
|
||
var backfillService = new EventProvenanceBackfillService(postgresConnection, resolver);
|
||
|
||
// Count unproven events
|
||
var count = await backfillService.CountUnprovenEventsAsync(
|
||
new[] { "SBOM", "VEX", "SCAN" });
|
||
|
||
// Backfill with progress reporting
|
||
var progress = new Progress<BackfillResult>(r =>
|
||
Console.WriteLine($"{r.EventId}: {r.Status}"));
|
||
|
||
var summary = await backfillService.BackfillAllAsync(
|
||
kinds: new[] { "SBOM", "VEX", "SCAN" },
|
||
limit: 1000,
|
||
progress: progress);
|
||
|
||
Console.WriteLine($"Processed: {summary.TotalProcessed}");
|
||
Console.WriteLine($"Success: {summary.SuccessCount}");
|
||
Console.WriteLine($"Not found: {summary.NotFoundCount}");
|
||
Console.WriteLine($"Errors: {summary.ErrorCount}");
|
||
```
|
||
|
||
### 10.3 Implementing IAttestationResolver
|
||
|
||
Implementations should query the attestation store (Rekor, CAS, or local PostgreSQL) by subject digest:
|
||
|
||
```csharp
|
||
public class RekorAttestationResolver : IAttestationResolver
|
||
{
|
||
private readonly IRekorClient _rekor;
|
||
private readonly IAttestationRepository _attestations;
|
||
|
||
public async Task<AttestationResolution?> ResolveAsync(
|
||
string subjectDigestSha256,
|
||
string eventKind,
|
||
CancellationToken cancellationToken)
|
||
{
|
||
// Look up attestation by subject digest
|
||
var record = await _attestations.GetAsync(subjectDigestSha256, eventKind, cancellationToken);
|
||
if (record is null) return null;
|
||
|
||
// Fetch Rekor proof if available
|
||
var proof = await _rekor.GetProofAsync(record.RekorUuid, RekorBackend.Sigstore, cancellationToken);
|
||
|
||
return new AttestationResolution
|
||
{
|
||
Dsse = new DsseProvenance { /* ... */ },
|
||
Trust = new TrustInfo { Verified = true, Verifier = "Authority@stella" },
|
||
AttestationId = record.Id
|
||
};
|
||
}
|
||
}
|
||
```
|
||
|
||
### 10.4 Reference files
|
||
|
||
- `src/StellaOps.Events.Postgres/IAttestationResolver.cs`
|
||
- `src/StellaOps.Events.Postgres/EventProvenanceBackfillService.cs`
|
||
- `src/StellaOps.Events.Postgres/StubAttestationResolver.cs`
|
||
|
||
This section was added as part of `PROV-BACKFILL-401-029` (completed 2025-11-27).
|