feat(rate-limiting): Implement core rate limiting functionality with configuration, decision-making, metrics, middleware, and service registration
- Add RateLimitConfig for configuration management with YAML binding support. - Introduce RateLimitDecision to encapsulate the result of rate limit checks. - Implement RateLimitMetrics for OpenTelemetry metrics tracking. - Create RateLimitMiddleware for enforcing rate limits on incoming requests. - Develop RateLimitService to orchestrate instance and environment rate limit checks. - Add RateLimitServiceCollectionExtensions for dependency injection registration.
This commit is contained in:
master
@@ -3,11 +3,11 @@
|
||||
> **Status:** Draft – aligns with the November 2025 advisory “store DSSE attestation refs inline on every SBOM/VEX event node.”
|
||||
> **Owners:** Authority Guild · Feedser Guild · Platform Guild · Docs Guild.
|
||||
|
||||
This document defines how Stella Ops records provenance for SBOM, VEX, scan, and derived events: every event node in the Mongo event graph includes DSSE + Rekor references and verification metadata so audits and replay become first-class queries.
|
||||
This document defines how Stella Ops records provenance for SBOM, VEX, scan, and derived events: every event node in the PostgreSQL event store includes DSSE + Rekor references and verification metadata so audits and replay become first-class queries.
|
||||
|
||||
---
|
||||
|
||||
## 1. Event patch (Mongo schema)
|
||||
## 1. Event patch (PostgreSQL schema)
|
||||
|
||||
```jsonc
|
||||
{
|
||||
@@ -66,7 +66,7 @@ This document defines how Stella Ops records provenance for SBOM, VEX, scan, a
|
||||
|
||||
1. **Obtain provenance metadata** for each attested artifact (build, SBOM, VEX, scan). The CI script (`scripts/publish_attestation_with_provenance.sh`) captures `envelopeDigest`, Rekor `logIndex`/`uuid`, and key info.
|
||||
2. **Authority/Feedser** verify the DSSE + Rekor proof (local cosign/rekor libs or the Signer service) and set `trust.verified = true`, `trust.verifier = "Authority@stella"`, `trust.witnesses = 1`.
|
||||
3. **Attach** the provenance block before appending the event to Mongo, using `StellaOps.Provenance.Mongo` helpers.
|
||||
3. **Attach** the provenance block before appending the event to PostgreSQL, using `StellaOps.Provenance.Postgres` helpers.
|
||||
4. **Backfill** historical events by resolving known subjects → attestation digests and running an update script.
|
||||
|
||||
### 2.1 Supplying metadata from Concelier statements
|
||||
@@ -84,7 +84,7 @@ Concelier ingestion jobs can now inline provenance when they create advisory sta
|
||||
|
||||
Providing the metadata during ingestion keeps new statements self-contained and reduces the surface that the `/events/statements/{statementId}/provenance` endpoint needs to backfill later.
|
||||
|
||||
Reference helper: `src/__Libraries/StellaOps.Provenance.Mongo/ProvenanceMongoExtensions.cs`.
|
||||
Reference helper: `src/__Libraries/StellaOps.Provenance.Postgres/ProvenancePostgresExtensions.cs`.
|
||||
|
||||
---
|
||||
|
||||
@@ -104,7 +104,7 @@ Advisory AI consumes the canonical `Advisory` aggregate and emits structured chu
|
||||
"chunkId": "c0ffee12", // sha256(advisory.observationId + observationPath)[:16]
|
||||
"content": { /* structured field */ },
|
||||
"provenance": {
|
||||
"documentId": "tenant-a:chunk:newest", // Mongo _id of backing observation
|
||||
"documentId": "tenant-a:chunk:newest", // PostgreSQL id of backing observation
|
||||
"observationPath": "/references/0", // JSON Pointer into the observation
|
||||
"source": "nvd",
|
||||
"kind": "workaround",
|
||||
@@ -154,40 +154,36 @@ Feedser ingests this JSON and maps it to `DsseProvenance` + `TrustInfo`.
|
||||
|
||||
---
|
||||
|
||||
## 4. Mongo indexes
|
||||
## 4. PostgreSQL indexes
|
||||
|
||||
Create indexes to keep provenance queries fast (`mongosh`):
|
||||
Create indexes to keep provenance queries fast (PostgreSQL DDL):
|
||||
|
||||
```javascript
|
||||
db.events.createIndex(
|
||||
{ "subject.digest.sha256": 1, "kind": 1, "provenance.dsse.rekor.logIndex": 1 },
|
||||
{ name: "events_by_subject_kind_provenance" }
|
||||
);
|
||||
```sql
|
||||
-- events_by_subject_kind_provenance
|
||||
CREATE INDEX events_by_subject_kind_provenance
|
||||
ON events (subject_digest_sha256, kind, provenance_dsse_rekor_log_index);
|
||||
|
||||
db.events.createIndex(
|
||||
{ "kind": 1, "trust.verified": 1, "provenance.dsse.rekor.logIndex": 1 },
|
||||
{ name: "events_unproven_by_kind" }
|
||||
);
|
||||
-- events_unproven_by_kind
|
||||
CREATE INDEX events_unproven_by_kind
|
||||
ON events (kind, trust_verified, provenance_dsse_rekor_log_index);
|
||||
|
||||
db.events.createIndex(
|
||||
{ "provenance.dsse.rekor.logIndex": 1 },
|
||||
{ name: "events_by_rekor_logindex" }
|
||||
);
|
||||
-- events_by_rekor_logindex
|
||||
CREATE INDEX events_by_rekor_logindex
|
||||
ON events (provenance_dsse_rekor_log_index);
|
||||
|
||||
db.events.createIndex(
|
||||
{ "provenance.dsse.envelopeDigest": 1 },
|
||||
{ name: "events_by_envelope_digest", sparse: true }
|
||||
);
|
||||
-- events_by_envelope_digest (partial index for non-null values)
|
||||
CREATE INDEX events_by_envelope_digest
|
||||
ON events (provenance_dsse_envelope_digest)
|
||||
WHERE provenance_dsse_envelope_digest IS NOT NULL;
|
||||
|
||||
db.events.createIndex(
|
||||
{ "ts": -1, "kind": 1, "trust.verified": 1 },
|
||||
{ name: "events_by_ts_kind_verified" }
|
||||
);
|
||||
-- events_by_ts_kind_verified
|
||||
CREATE INDEX events_by_ts_kind_verified
|
||||
ON events (ts DESC, kind, trust_verified);
|
||||
```
|
||||
|
||||
Deployment options:
|
||||
- **Ops script:** `mongosh stellaops_db < ops/mongo/indices/events_provenance_indices.js`
|
||||
- **C# helper:** `MongoIndexes.EnsureEventIndexesAsync(database, ct)`
|
||||
- **Ops script:** `psql -d stellaops_db -f ops/postgres/indices/events_provenance_indices.sql`
|
||||
- **C# helper:** `PostgresIndexes.EnsureEventIndexesAsync(connection, ct)`
|
||||
|
||||
This section was updated as part of `PROV-INDEX-401-030` (completed 2025-11-27).
|
||||
|
||||
@@ -197,29 +193,23 @@ This section was updated as part of `PROV-INDEX-401-030` (completed 2025-11-27).
|
||||
|
||||
* **All proven VEX for an image digest:**
|
||||
|
||||
```javascript
|
||||
db.events.find({
|
||||
kind: "VEX",
|
||||
"subject.digest.sha256": "<digest>",
|
||||
"provenance.dsse.rekor.logIndex": { $exists: true },
|
||||
"trust.verified": true
|
||||
})
|
||||
```sql
|
||||
SELECT * FROM events
|
||||
WHERE kind = 'VEX'
|
||||
AND subject_digest_sha256 = '<digest>'
|
||||
AND provenance_dsse_rekor_log_index IS NOT NULL
|
||||
AND trust_verified = true;
|
||||
```
|
||||
|
||||
* **Compliance gap (unverified data used for decisions):**
|
||||
|
||||
```javascript
|
||||
db.events.aggregate([
|
||||
{ $match: { kind: { $in: ["VEX","SBOM","SCAN"] } } },
|
||||
{ $match: {
|
||||
$or: [
|
||||
{ "trust.verified": { $ne: true } },
|
||||
{ "provenance.dsse.rekor.logIndex": { $exists: false } }
|
||||
]
|
||||
}
|
||||
},
|
||||
{ $group: { _id: "$kind", count: { $sum: 1 } } }
|
||||
])
|
||||
```sql
|
||||
SELECT kind, COUNT(*) as count
|
||||
FROM events
|
||||
WHERE kind IN ('VEX', 'SBOM', 'SCAN')
|
||||
AND (trust_verified IS NOT TRUE
|
||||
OR provenance_dsse_rekor_log_index IS NULL)
|
||||
GROUP BY kind;
|
||||
```
|
||||
|
||||
* **Replay slice:** filter for events where `provenance.dsse.chain` covers build → sbom → scan and export referenced attestation digests.
|
||||
@@ -265,9 +255,9 @@ rules:
|
||||
|
||||
| Task ID | Scope |
|
||||
|---------|-------|
|
||||
| `PROV-INLINE-401-028` | Extend Authority/Feedser write-paths to attach `provenance.dsse` + `trust` blocks using `StellaOps.Provenance.Mongo`. |
|
||||
| `PROV-INLINE-401-028` | Extend Authority/Feedser write-paths to attach `provenance.dsse` + `trust` blocks using `StellaOps.Provenance.Postgres`. |
|
||||
| `PROV-BACKFILL-401-029` | Backfill historical events with DSSE/Rekor refs based on existing attestation digests. |
|
||||
| `PROV-INDEX-401-030` | Create Mongo indexes and expose helper queries for audits. |
|
||||
| `PROV-INDEX-401-030` | Create PostgreSQL indexes and expose helper queries for audits. |
|
||||
|
||||
Keep this document updated when new attestation types or mirror/witness policies land.
|
||||
|
||||
@@ -289,7 +279,7 @@ The body matches the JSON emitted by `publish_attestation_with_provenance.sh`. F
|
||||
|
||||
## 10. Backfill service
|
||||
|
||||
`EventProvenanceBackfillService` (`src/StellaOps.Events.Mongo/EventProvenanceBackfillService.cs`) orchestrates backfilling historical events with DSSE provenance metadata.
|
||||
`EventProvenanceBackfillService` (`src/StellaOps.Events.Postgres/EventProvenanceBackfillService.cs`) orchestrates backfilling historical events with DSSE provenance metadata.
|
||||
|
||||
### 10.1 Components
|
||||
|
||||
@@ -303,7 +293,7 @@ The body matches the JSON emitted by `publish_attestation_with_provenance.sh`. F
|
||||
|
||||
```csharp
|
||||
var resolver = new MyAttestationResolver(rekorClient, attestationRepo);
|
||||
var backfillService = new EventProvenanceBackfillService(mongoDatabase, resolver);
|
||||
var backfillService = new EventProvenanceBackfillService(postgresConnection, resolver);
|
||||
|
||||
// Count unproven events
|
||||
var count = await backfillService.CountUnprovenEventsAsync(
|
||||
@@ -326,7 +316,7 @@ Console.WriteLine($"Errors: {summary.ErrorCount}");
|
||||
|
||||
### 10.3 Implementing IAttestationResolver
|
||||
|
||||
Implementations should query the attestation store (Rekor, CAS, or local Mongo) by subject digest:
|
||||
Implementations should query the attestation store (Rekor, CAS, or local PostgreSQL) by subject digest:
|
||||
|
||||
```csharp
|
||||
public class RekorAttestationResolver : IAttestationResolver
|
||||
@@ -358,8 +348,8 @@ public class RekorAttestationResolver : IAttestationResolver
|
||||
|
||||
### 10.4 Reference files
|
||||
|
||||
- `src/StellaOps.Events.Mongo/IAttestationResolver.cs`
|
||||
- `src/StellaOps.Events.Mongo/EventProvenanceBackfillService.cs`
|
||||
- `src/StellaOps.Events.Mongo/StubAttestationResolver.cs`
|
||||
- `src/StellaOps.Events.Postgres/IAttestationResolver.cs`
|
||||
- `src/StellaOps.Events.Postgres/EventProvenanceBackfillService.cs`
|
||||
- `src/StellaOps.Events.Postgres/StubAttestationResolver.cs`
|
||||
|
||||
This section was added as part of `PROV-BACKFILL-401-029` (completed 2025-11-27).
|
||||
|
||||
Reference in New Issue
Block a user