feat(rate-limiting): Implement core rate limiting functionality with configuration, decision-making, metrics, middleware, and service registration

- Add RateLimitConfig for configuration management with YAML binding support.
- Introduce RateLimitDecision to encapsulate the result of rate limit checks.
- Implement RateLimitMetrics for OpenTelemetry metrics tracking.
- Create RateLimitMiddleware for enforcing rate limits on incoming requests.
- Develop RateLimitService to orchestrate instance and environment rate limit checks.
- Add RateLimitServiceCollectionExtensions for dependency injection registration.
This commit is contained in:
master
2025-12-17 18:02:37 +02:00
parent 394b57f6bf
commit 8bbfe4d2d2
211 changed files with 47179 additions and 1590 deletions

View File

@@ -22,7 +22,7 @@
Excititor enforces the same ingestion covenant as Concelier, tailored to VEX payloads:
1. **Immutable `vex_raw` documents.** Upstream OpenVEX/CSAF/CycloneDX files are stored verbatim (`content.raw`) with provenance (`issuer`, `statement_id`, timestamps, signatures). Revisions append new versions linked by `supersedes`.
1. **Immutable `vex_raw` rows.** Upstream OpenVEX/CSAF/CycloneDX files are stored verbatim (`content.raw`) with provenance (`issuer`, `statement_id`, timestamps, signatures). Revisions append new versions linked by `supersedes`.
2. **No derived consensus at ingest time.** Fields such as `effective_status`, `merged_state`, `severity`, or reachability are forbidden. Roslyn analyzers and runtime guards block violations before writes.
3. **Linkset-only joins.** Product aliases, CVE keys, SBOM hints, and references live under `linkset`; ingestion must never mutate the underlying statement.
@@ -330,11 +330,11 @@ All exports remain deterministic and, when configured, attested via DSSE + Rekor
---
## 4) Storage schema (MongoDB)
## 4) Storage schema (PostgreSQL)
Database: `excititor`
### 3.1 Collections
### 3.1 Tables
**`vex.providers`**
@@ -357,7 +357,7 @@ uri
ingestedAt
contentType
sig: { verified: bool, method: pgp|cosign|x509|none, keyId|certSubject, bundle? }
payload: GridFS pointer (if large)
payload: object storage pointer (if large)
disposition: kept|replaced|superseded
correlation: { replaces?: sha256, replacedBy?: sha256 }
```
@@ -620,7 +620,8 @@ GET /providers/{id}/status → last fetch, doc counts, signature stats
```yaml
excititor:
mongo: { uri: "mongodb://mongo/excititor" }
postgres:
connectionString: "Host=postgres;Port=5432;Database=excititor;Username=stellaops;Password=stellaops"
s3:
endpoint: http://minio:9000
bucket: stellaops
@@ -703,7 +704,7 @@ Run the ingestion endpoint once after applying migration `20251019-consensus-sig
* **Scaling:**
* WebService handles control APIs; **Worker** background services (same image) execute fetch/normalize in parallel with ratelimits; Mongo writes batched; upserts by natural keys.
* WebService handles control APIs; **Worker** background services (same image) execute fetch/normalize in parallel with ratelimits; PostgreSQL writes batched; upserts by natural keys.
* Exports stream straight to S3 (MinIO) with rolling buffers.
* **Caching:**
@@ -760,7 +761,7 @@ Excititor.Worker ships with a background refresh service that re-evaluates stale
* **Dashboards:** provider staleness, linkset conflict hot spots, signature posture, export cache hit-rate.
* **Telemetry configuration:** `Excititor:Telemetry` toggles OpenTelemetry for the host (`Enabled`, `EnableTracing`, `EnableMetrics`, `ServiceName`, `OtlpEndpoint`, optional `OtlpHeaders` and `ResourceAttributes`). Point it at the collector profile listed in `docs/observability/observability.md` so Excititors `ingestion_*` metrics land in the same Grafana dashboards as Concelier.
* **Health endpoint:** `/obs/excititor/health` (scope `vex.admin`) surfaces ingest/link/signature/conflict SLOs for Console + Grafana. Thresholds are configurable via `Excititor:Observability:*` (see `docs/observability/observability.md`).
* **Local replica set:** `tools/mongodb/local-mongo.sh start` downloads the vetted MongoDB binaries (6.0.x), boots a `rs0` single-node replica set, and prints the `EXCITITOR_TEST_MONGO_URI` export line so storage/integration tests can bypass Mongo2Go. `restart` restarts in-place, `clean` wipes the managed data/logs for deterministic runs, and `stop/status/logs` cover teardown/inspection.
* **Local database:** Use Docker Compose or `tools/postgres/local-postgres.sh start` to boot a PostgreSQL instance for storage/integration tests. `restart` restarts in-place, `clean` wipes the managed data/logs for deterministic runs, and `stop/status/logs` cover teardown/inspection.
* **API headers:** responses echo `X-Stella-TraceId` and `X-Stella-CorrelationId` to keep Console/Loki links deterministic; inbound correlation headers are preserved when present.
---