feat(rate-limiting): Implement core rate limiting functionality with configuration, decision-making, metrics, middleware, and service registration
- Add RateLimitConfig for configuration management with YAML binding support. - Introduce RateLimitDecision to encapsulate the result of rate limit checks. - Implement RateLimitMetrics for OpenTelemetry metrics tracking. - Create RateLimitMiddleware for enforcing rate limits on incoming requests. - Develop RateLimitService to orchestrate instance and environment rate limit checks. - Add RateLimitServiceCollectionExtensions for dependency injection registration.
This commit is contained in:
master
@@ -27,7 +27,7 @@
|
||||
* **Signer** (caller) — authenticated via **mTLS** and **Authority** OpToks.
|
||||
* **Rekor v2** — tile‑backed transparency log endpoint(s).
|
||||
* **MinIO (S3)** — optional archive store for DSSE envelopes & verification bundles.
|
||||
* **MongoDB** — local cache of `{uuid, index, proof, artifactSha256, bundleSha256}`; job state; audit.
|
||||
* **PostgreSQL** — local cache of `{uuid, index, proof, artifactSha256, bundleSha256}`; job state; audit.
|
||||
* **Redis** — dedupe/idempotency keys and short‑lived rate‑limit buckets.
|
||||
* **Licensing Service (optional)** — “endorse” call for cross‑log publishing when customer opts‑in.
|
||||
|
||||
@@ -109,48 +109,70 @@ The Attestor implements RFC 6962-compliant Merkle inclusion proof verification f
|
||||
|
||||
---
|
||||
|
||||
## 2) Data model (Mongo)
|
||||
## 2) Data model (PostgreSQL)
|
||||
|
||||
Database: `attestor`
|
||||
|
||||
**Collections & schemas**
|
||||
**Tables & schemas**
|
||||
|
||||
* `entries`
|
||||
* `entries` table
|
||||
|
||||
```
|
||||
{ _id: "<rekor-uuid>",
|
||||
artifact: { sha256: "<sha256>", kind: "sbom|report|vex-export", imageDigest?, subjectUri? },
|
||||
bundleSha256: "<sha256>", // canonicalized DSSE
|
||||
index: <int>, // log index/sequence if provided by backend
|
||||
proof: { // inclusion proof
|
||||
checkpoint: { origin, size, rootHash, timestamp },
|
||||
inclusion: { leafHash, path[] } // Merkle path (tiles)
|
||||
},
|
||||
log: { url, logId? },
|
||||
createdAt, status: "included|pending|failed",
|
||||
signerIdentity: { mode: "keyless|kms", issuer, san?, kid? }
|
||||
}
|
||||
```sql
|
||||
CREATE TABLE attestor.entries (
|
||||
id UUID PRIMARY KEY, -- rekor-uuid
|
||||
artifact_sha256 TEXT NOT NULL,
|
||||
artifact_kind TEXT NOT NULL, -- sbom|report|vex-export
|
||||
artifact_image_digest TEXT,
|
||||
artifact_subject_uri TEXT,
|
||||
bundle_sha256 TEXT NOT NULL, -- canonicalized DSSE
|
||||
log_index INTEGER, -- log index/sequence if provided by backend
|
||||
proof_checkpoint JSONB, -- { origin, size, rootHash, timestamp }
|
||||
proof_inclusion JSONB, -- { leafHash, path[] } Merkle path (tiles)
|
||||
log_url TEXT,
|
||||
log_id TEXT,
|
||||
created_at TIMESTAMPTZ DEFAULT NOW(),
|
||||
status TEXT NOT NULL, -- included|pending|failed
|
||||
signer_identity JSONB -- { mode, issuer, san?, kid? }
|
||||
);
|
||||
```
|
||||
|
||||
* `dedupe`
|
||||
* `dedupe` table
|
||||
|
||||
```
|
||||
{ key: "bundle:<sha256>", rekorUuid, createdAt, ttlAt } // idempotency key
|
||||
```sql
|
||||
CREATE TABLE attestor.dedupe (
|
||||
key TEXT PRIMARY KEY, -- bundle:<sha256> idempotency key
|
||||
rekor_uuid UUID NOT NULL,
|
||||
created_at TIMESTAMPTZ DEFAULT NOW(),
|
||||
ttl_at TIMESTAMPTZ NOT NULL -- for scheduled cleanup
|
||||
);
|
||||
```
|
||||
|
||||
* `audit`
|
||||
* `audit` table
|
||||
|
||||
```
|
||||
{ _id, ts, caller: { cn, mTLSThumbprint, sub, aud }, // from mTLS + OpTok
|
||||
action: "submit|verify|fetch",
|
||||
artifactSha256, bundleSha256, rekorUuid?, index?, result, latencyMs, backend }
|
||||
```sql
|
||||
CREATE TABLE attestor.audit (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
ts TIMESTAMPTZ DEFAULT NOW(),
|
||||
caller_cn TEXT,
|
||||
caller_mtls_thumbprint TEXT,
|
||||
caller_sub TEXT,
|
||||
caller_aud TEXT,
|
||||
action TEXT NOT NULL, -- submit|verify|fetch
|
||||
artifact_sha256 TEXT,
|
||||
bundle_sha256 TEXT,
|
||||
rekor_uuid UUID,
|
||||
log_index INTEGER,
|
||||
result TEXT NOT NULL,
|
||||
latency_ms INTEGER,
|
||||
backend TEXT
|
||||
);
|
||||
```
|
||||
|
||||
Indexes:
|
||||
|
||||
* `entries` on `artifact.sha256`, `bundleSha256`, `createdAt`, and `{status:1, createdAt:-1}`.
|
||||
* `dedupe.key` unique (TTL 24–48h).
|
||||
* `audit.ts` for time‑range queries.
|
||||
* `entries`: indexes on `artifact_sha256`, `bundle_sha256`, `created_at`, and composite `(status, created_at DESC)`.
|
||||
* `dedupe`: unique index on `key`; scheduled job cleans rows where `ttl_at < NOW()` (24–48h retention).
|
||||
* `audit`: index on `ts` for time‑range queries.
|
||||
|
||||
---
|
||||
|
||||
@@ -207,16 +229,100 @@ public interface IContentAddressedIdGenerator
|
||||
|
||||
### Predicate Types
|
||||
|
||||
The ProofChain library defines DSSE predicates for each attestation type:
|
||||
The ProofChain library defines DSSE predicates for proof chain attestations. All predicates follow the in-toto Statement/v1 format.
|
||||
|
||||
| Predicate | Type URI | Purpose |
|
||||
|-----------|----------|---------|
|
||||
| `EvidencePredicate` | `stellaops.org/evidence/v1` | Scan evidence (findings, reachability) |
|
||||
| `ReasoningPredicate` | `stellaops.org/reasoning/v1` | Exploitability reasoning |
|
||||
| `VexPredicate` | `stellaops.org/vex-verdict/v1` | VEX status determination |
|
||||
| `ProofSpinePredicate` | `stellaops.org/proof-spine/v1` | Complete proof bundle |
|
||||
#### Predicate Type Registry
|
||||
|
||||
**Reference:** `src/Attestor/__Libraries/StellaOps.Attestor.ProofChain/`
|
||||
| Predicate | Type URI | Purpose | Signer Role |
|
||||
|-----------|----------|---------|-------------|
|
||||
| **Evidence** | `evidence.stella/v1` | Raw evidence from scanner/ingestor (findings, reachability data) | Scanner/Ingestor key |
|
||||
| **Reasoning** | `reasoning.stella/v1` | Policy evaluation trace with inputs and intermediate findings | Policy/Authority key |
|
||||
| **VEX Verdict** | `cdx-vex.stella/v1` | VEX verdict with status, justification, and provenance | VEXer/Vendor key |
|
||||
| **Proof Spine** | `proofspine.stella/v1` | Merkle-aggregated proof spine linking evidence to verdict | Authority key |
|
||||
| **Verdict Receipt** | `verdict.stella/v1` | Final surfaced decision receipt with policy rule reference | Authority key |
|
||||
| **SBOM Linkage** | `https://stella-ops.org/predicates/sbom-linkage/v1` | SBOM-to-component linkage metadata | Generator key |
|
||||
|
||||
#### Evidence Statement (`evidence.stella/v1`)
|
||||
|
||||
Captures raw evidence collected from scanners or vulnerability feeds.
|
||||
|
||||
| Field | Type | Description |
|
||||
|-------|------|-------------|
|
||||
| `source` | string | Scanner or feed name that produced this evidence |
|
||||
| `sourceVersion` | string | Version of the source tool |
|
||||
| `collectionTime` | DateTimeOffset | UTC timestamp when evidence was collected |
|
||||
| `sbomEntryId` | string | Reference to the SBOM entry this evidence relates to |
|
||||
| `vulnerabilityId` | string? | CVE or vulnerability identifier if applicable |
|
||||
| `rawFinding` | object | Pointer to or inline representation of raw finding data |
|
||||
| `evidenceId` | string | Content-addressed ID (sha256:<hash>) |
|
||||
|
||||
#### Reasoning Statement (`reasoning.stella/v1`)
|
||||
|
||||
Captures policy evaluation traces linking evidence to decisions.
|
||||
|
||||
| Field | Type | Description |
|
||||
|-------|------|-------------|
|
||||
| `sbomEntryId` | string | SBOM entry this reasoning applies to |
|
||||
| `evidenceIds` | string[] | Evidence IDs considered in this reasoning |
|
||||
| `policyVersion` | string | Version of the policy used for evaluation |
|
||||
| `inputs` | object | Inputs to the reasoning process (evaluation time, thresholds, lattice rules) |
|
||||
| `intermediateFindings` | object? | Intermediate findings from the evaluation |
|
||||
| `reasoningId` | string | Content-addressed ID (sha256:<hash>) |
|
||||
|
||||
#### VEX Verdict Statement (`cdx-vex.stella/v1`)
|
||||
|
||||
Captures VEX status determinations with provenance.
|
||||
|
||||
| Field | Type | Description |
|
||||
|-------|------|-------------|
|
||||
| `sbomEntryId` | string | SBOM entry this verdict applies to |
|
||||
| `vulnerabilityId` | string | CVE, GHSA, or other vulnerability identifier |
|
||||
| `status` | string | VEX status: `not_affected`, `affected`, `fixed`, `under_investigation` |
|
||||
| `justification` | string | Justification for the VEX status |
|
||||
| `policyVersion` | string | Version of the policy used |
|
||||
| `reasoningId` | string | Reference to the reasoning that led to this verdict |
|
||||
| `vexVerdictId` | string | Content-addressed ID (sha256:<hash>) |
|
||||
|
||||
#### Proof Spine Statement (`proofspine.stella/v1`)
|
||||
|
||||
Merkle-aggregated proof bundle linking all chain components.
|
||||
|
||||
| Field | Type | Description |
|
||||
|-------|------|-------------|
|
||||
| `sbomEntryId` | string | SBOM entry this proof spine covers |
|
||||
| `evidenceIds` | string[] | Sorted list of evidence IDs included in this proof bundle |
|
||||
| `reasoningId` | string | Reasoning ID linking evidence to verdict |
|
||||
| `vexVerdictId` | string | VEX verdict ID for this entry |
|
||||
| `policyVersion` | string | Version of the policy used |
|
||||
| `proofBundleId` | string | Content-addressed ID (sha256:<merkle_root>) |
|
||||
|
||||
#### Verdict Receipt Statement (`verdict.stella/v1`)
|
||||
|
||||
Final surfaced decision receipt with full provenance.
|
||||
|
||||
| Field | Type | Description |
|
||||
|-------|------|-------------|
|
||||
| `graphRevisionId` | string | Graph revision ID this verdict was computed from |
|
||||
| `findingKey` | object | Finding key (sbomEntryId + vulnerabilityId) |
|
||||
| `rule` | object | Policy rule that produced this verdict |
|
||||
| `decision` | object | Decision made by the rule |
|
||||
| `inputs` | object | Inputs used to compute this verdict |
|
||||
| `outputs` | object | Outputs/references from this verdict |
|
||||
| `createdAt` | DateTimeOffset | UTC timestamp when verdict was created |
|
||||
|
||||
#### SBOM Linkage Statement (`sbom-linkage/v1`)
|
||||
|
||||
SBOM-to-component linkage metadata.
|
||||
|
||||
| Field | Type | Description |
|
||||
|-------|------|-------------|
|
||||
| `sbom` | object | SBOM descriptor (id, format, specVersion, mediaType, sha256, location) |
|
||||
| `generator` | object | Generator tool descriptor |
|
||||
| `generatedAt` | DateTimeOffset | UTC timestamp when linkage was generated |
|
||||
| `incompleteSubjects` | object[]? | Subjects that could not be fully resolved |
|
||||
| `tags` | object? | Arbitrary tags for classification or filtering |
|
||||
|
||||
**Reference:** `src/Attestor/__Libraries/StellaOps.Attestor.ProofChain/Statements/`
|
||||
|
||||
---
|
||||
|
||||
@@ -354,7 +460,7 @@ The ProofChain library defines DSSE predicates for each attestation type:
|
||||
|
||||
### 4.5 Bulk verification
|
||||
|
||||
`POST /api/v1/rekor/verify:bulk` enqueues a verification job containing up to `quotas.bulk.maxItemsPerJob` items. Each item mirrors the single verification payload (uuid | artifactSha256 | subject+envelopeId, optional policyVersion/refreshProof). The handler persists a MongoDB job document (`bulk_jobs` collection) and returns `202 Accepted` with a job descriptor and polling URL.
|
||||
`POST /api/v1/rekor/verify:bulk` enqueues a verification job containing up to `quotas.bulk.maxItemsPerJob` items. Each item mirrors the single verification payload (uuid | artifactSha256 | subject+envelopeId, optional policyVersion/refreshProof). The handler persists a PostgreSQL job record (`bulk_jobs` table) and returns `202 Accepted` with a job descriptor and polling URL.
|
||||
|
||||
`GET /api/v1/rekor/verify:bulk/{jobId}` returns progress and per-item results (subject/uuid, status, issues, cached verification report if available). Jobs are tenant- and subject-scoped; only the initiating principal can read their progress.
|
||||
|
||||
@@ -405,7 +511,7 @@ The worker honours `bulkVerification.itemDelayMilliseconds` for throttling and r
|
||||
|
||||
## 7) Storage & archival
|
||||
|
||||
* **Entries** in Mongo provide a local ledger keyed by `rekorUuid` and **artifact sha256** for quick reverse lookups.
|
||||
* **Entries** in PostgreSQL provide a local ledger keyed by `rekorUuid` and **artifact sha256** for quick reverse lookups.
|
||||
* **S3 archival** (if enabled):
|
||||
|
||||
```
|
||||
@@ -505,8 +611,8 @@ attestor:
|
||||
mirror:
|
||||
enabled: false
|
||||
url: "https://rekor-v2.mirror"
|
||||
mongo:
|
||||
uri: "mongodb://mongo/attestor"
|
||||
postgres:
|
||||
connectionString: "Host=postgres;Port=5432;Database=attestor;Username=stellaops;Password=secret"
|
||||
s3:
|
||||
enabled: true
|
||||
endpoint: "http://minio:9000"
|
||||
|
||||
Reference in New Issue
Block a user