feat(rate-limiting): Implement core rate limiting functionality with configuration, decision-making, metrics, middleware, and service registration
- Add RateLimitConfig for configuration management with YAML binding support. - Introduce RateLimitDecision to encapsulate the result of rate limit checks. - Implement RateLimitMetrics for OpenTelemetry metrics tracking. - Create RateLimitMiddleware for enforcing rate limits on incoming requests. - Develop RateLimitService to orchestrate instance and environment rate limit checks. - Add RateLimitServiceCollectionExtensions for dependency injection registration.
This commit is contained in:
@@ -22,7 +22,7 @@
|
||||
|
||||
Excititor enforces the same ingestion covenant as Concelier, tailored to VEX payloads:
|
||||
|
||||
1. **Immutable `vex_raw` documents.** Upstream OpenVEX/CSAF/CycloneDX files are stored verbatim (`content.raw`) with provenance (`issuer`, `statement_id`, timestamps, signatures). Revisions append new versions linked by `supersedes`.
|
||||
1. **Immutable `vex_raw` rows.** Upstream OpenVEX/CSAF/CycloneDX files are stored verbatim (`content.raw`) with provenance (`issuer`, `statement_id`, timestamps, signatures). Revisions append new versions linked by `supersedes`.
|
||||
2. **No derived consensus at ingest time.** Fields such as `effective_status`, `merged_state`, `severity`, or reachability are forbidden. Roslyn analyzers and runtime guards block violations before writes.
|
||||
3. **Linkset-only joins.** Product aliases, CVE keys, SBOM hints, and references live under `linkset`; ingestion must never mutate the underlying statement.
|
||||
|
||||
@@ -330,11 +330,11 @@ All exports remain deterministic and, when configured, attested via DSSE + Rekor
|
||||
|
||||
---
|
||||
|
||||
## 4) Storage schema (MongoDB)
|
||||
## 4) Storage schema (PostgreSQL)
|
||||
|
||||
Database: `excititor`
|
||||
|
||||
### 3.1 Collections
|
||||
### 3.1 Tables
|
||||
|
||||
**`vex.providers`**
|
||||
|
||||
@@ -357,7 +357,7 @@ uri
|
||||
ingestedAt
|
||||
contentType
|
||||
sig: { verified: bool, method: pgp|cosign|x509|none, keyId|certSubject, bundle? }
|
||||
payload: GridFS pointer (if large)
|
||||
payload: object storage pointer (if large)
|
||||
disposition: kept|replaced|superseded
|
||||
correlation: { replaces?: sha256, replacedBy?: sha256 }
|
||||
```
|
||||
@@ -620,7 +620,8 @@ GET /providers/{id}/status → last fetch, doc counts, signature stats
|
||||
|
||||
```yaml
|
||||
excititor:
|
||||
mongo: { uri: "mongodb://mongo/excititor" }
|
||||
postgres:
|
||||
connectionString: "Host=postgres;Port=5432;Database=excititor;Username=stellaops;Password=stellaops"
|
||||
s3:
|
||||
endpoint: http://minio:9000
|
||||
bucket: stellaops
|
||||
@@ -703,7 +704,7 @@ Run the ingestion endpoint once after applying migration `20251019-consensus-sig
|
||||
|
||||
* **Scaling:**
|
||||
|
||||
* WebService handles control APIs; **Worker** background services (same image) execute fetch/normalize in parallel with rate‑limits; Mongo writes batched; upserts by natural keys.
|
||||
* WebService handles control APIs; **Worker** background services (same image) execute fetch/normalize in parallel with rate‑limits; PostgreSQL writes batched; upserts by natural keys.
|
||||
* Exports stream straight to S3 (MinIO) with rolling buffers.
|
||||
|
||||
* **Caching:**
|
||||
@@ -760,7 +761,7 @@ Excititor.Worker ships with a background refresh service that re-evaluates stale
|
||||
* **Dashboards:** provider staleness, linkset conflict hot spots, signature posture, export cache hit-rate.
|
||||
* **Telemetry configuration:** `Excititor:Telemetry` toggles OpenTelemetry for the host (`Enabled`, `EnableTracing`, `EnableMetrics`, `ServiceName`, `OtlpEndpoint`, optional `OtlpHeaders` and `ResourceAttributes`). Point it at the collector profile listed in `docs/observability/observability.md` so Excititor’s `ingestion_*` metrics land in the same Grafana dashboards as Concelier.
|
||||
* **Health endpoint:** `/obs/excititor/health` (scope `vex.admin`) surfaces ingest/link/signature/conflict SLOs for Console + Grafana. Thresholds are configurable via `Excititor:Observability:*` (see `docs/observability/observability.md`).
|
||||
* **Local replica set:** `tools/mongodb/local-mongo.sh start` downloads the vetted MongoDB binaries (6.0.x), boots a `rs0` single-node replica set, and prints the `EXCITITOR_TEST_MONGO_URI` export line so storage/integration tests can bypass Mongo2Go. `restart` restarts in-place, `clean` wipes the managed data/logs for deterministic runs, and `stop/status/logs` cover teardown/inspection.
|
||||
* **Local database:** Use Docker Compose or `tools/postgres/local-postgres.sh start` to boot a PostgreSQL instance for storage/integration tests. `restart` restarts in-place, `clean` wipes the managed data/logs for deterministic runs, and `stop/status/logs` cover teardown/inspection.
|
||||
* **API headers:** responses echo `X-Stella-TraceId` and `X-Stella-CorrelationId` to keep Console/Loki links deterministic; inbound correlation headers are preserved when present.
|
||||
|
||||
---
|
||||
|
||||
Reference in New Issue
Block a user