feat(rate-limiting): Implement core rate limiting functionality with configuration, decision-making, metrics, middleware, and service registration
- Add RateLimitConfig for configuration management with YAML binding support. - Introduce RateLimitDecision to encapsulate the result of rate limit checks. - Implement RateLimitMetrics for OpenTelemetry metrics tracking. - Create RateLimitMiddleware for enforcing rate limits on incoming requests. - Develop RateLimitService to orchestrate instance and environment rate limit checks. - Add RateLimitServiceCollectionExtensions for dependency injection registration.
This commit is contained in:
master
@@ -49,13 +49,13 @@ graph TD
|
||||
Materializer[Effective Findings Writer]
|
||||
end
|
||||
subgraph RawStores["Raw Stores (AOC)"]
|
||||
AdvisoryRaw[(MongoDB<br/>advisory_raw)]
|
||||
VexRaw[(MongoDB<br/>vex_raw)]
|
||||
AdvisoryRaw[(PostgreSQL<br/>advisory_raw)]
|
||||
VexRaw[(PostgreSQL<br/>vex_raw)]
|
||||
end
|
||||
subgraph Derived["Derived Stores"]
|
||||
Mongo[(MongoDB<br/>policies / policy_runs / effective_finding_*)]
|
||||
PG[(PostgreSQL<br/>policies / policy_runs / effective_finding_*)]
|
||||
Blob[(Object Store / Evidence Locker)]
|
||||
Queue[(Mongo Queue / NATS)]
|
||||
Queue[(PostgreSQL Queue / NATS)]
|
||||
end
|
||||
Concelier[(Concelier APIs)]
|
||||
Excititor[(Excititor APIs)]
|
||||
@@ -75,12 +75,12 @@ graph TD
|
||||
WorkerPool --> VexRaw
|
||||
WorkerPool --> SBOM
|
||||
WorkerPool --> Materializer
|
||||
Materializer --> Mongo
|
||||
Materializer --> PG
|
||||
WorkerPool --> Blob
|
||||
API --> Mongo
|
||||
API --> PG
|
||||
API --> Blob
|
||||
API --> Authority
|
||||
Orchestrator --> Mongo
|
||||
Orchestrator --> PG
|
||||
Authority --> API
|
||||
```
|
||||
|
||||
@@ -88,14 +88,14 @@ Key notes:
|
||||
|
||||
- API host exposes lifecycle, run, simulate, findings endpoints with DPoP-bound OAuth enforcement.
|
||||
- Orchestrator manages run scheduling/fairness; writes run tickets to queue, leases jobs to worker pool.
|
||||
- Workers evaluate policies using cached IR; join external services via tenant-scoped clients; pull immutable advisories/VEX from the raw stores; write derived overlays to Mongo and optional explain bundles to blob storage.
|
||||
- Workers evaluate policies using cached IR; join external services via tenant-scoped clients; pull immutable advisories/VEX from the raw stores; write derived overlays to PostgreSQL and optional explain bundles to blob storage.
|
||||
- Observability (metrics/traces/logs) integrated via OpenTelemetry (not shown).
|
||||
|
||||
---
|
||||
|
||||
### 2.1 · AOC inputs & immutability
|
||||
|
||||
- **Raw-only reads.** Evaluation workers access `advisory_raw` / `vex_raw` via tenant-scoped Mongo clients or the Concelier/Excititor raw APIs. No Policy Engine component is permitted to mutate these collections.
|
||||
- **Raw-only reads.** Evaluation workers access `advisory_raw` / `vex_raw` via tenant-scoped PostgreSQL clients or the Concelier/Excititor raw APIs. No Policy Engine component is permitted to mutate these tables.
|
||||
- **Guarded ingestion.** `AOCWriteGuard` rejects forbidden fields before data reaches the raw stores. Policy tests replay known `ERR_AOC_00x` violations to confirm ingestion compliance.
|
||||
- **Change streams as contract.** Run orchestration stores resumable cursors for raw change streams. Replays of these cursors (e.g., after failover) must yield identical materialisation outcomes.
|
||||
- **Derived stores only.** All severity, consensus, and suppression state lives in `effective_finding_*` collections and explain bundles owned by Policy Engine. Provenance fields link back to raw document IDs so auditors can trace every verdict.
|
||||
@@ -107,13 +107,13 @@ Key notes:
|
||||
|
||||
| Module | Responsibility | Notes |
|
||||
|--------|----------------|-------|
|
||||
| **Configuration** (`Configuration/`) | Bind settings (Mongo URIs, queue options, service URLs, sealed mode), validate on start. | Strict schema; fails fast on missing secrets. |
|
||||
| **Configuration** (`Configuration/`) | Bind settings (PostgreSQL connection strings, queue options, service URLs, sealed mode), validate on start. | Strict schema; fails fast on missing secrets. |
|
||||
| **Authority Client** (`Authority/`) | Acquire tokens, enforce scopes, perform DPoP key rotation. | Only service identity uses `effective:write`. |
|
||||
| **DSL Compiler** (`Dsl/`) | Parse, canonicalise, IR generation, checksum caching. | Uses Roslyn-like pipeline; caches by `policyId+version+hash`. |
|
||||
| **Selection Layer** (`Selection/`) | Batch SBOM ↔ advisory ↔ VEX joiners; apply equivalence tables; support incremental cursors. | Deterministic ordering (SBOM → advisory → VEX). |
|
||||
| **Evaluator** (`Evaluation/`) | Execute IR with first-match semantics, compute severity/trust/reachability weights, record rule hits. | Stateless; all inputs provided by selection layer. |
|
||||
| **Signals** (`Signals/`) | Normalizes reachability, trust, entropy, uncertainty, runtime hits into a single dictionary passed to Evaluator; supplies default `unknown` values when signals missing. Entropy penalties are derived from Scanner `layer_summary.json`/`entropy.report.json` (K=0.5, cap=0.3, block at image opaque ratio > 0.15 w/ unknown provenance) and exported via `policy_entropy_penalty_value` / `policy_entropy_image_opaque_ratio`; SPL scope `entropy.*` exposes `penalty`, `image_opaque_ratio`, `blocked`, `warned`, `capped`, `top_file_opaque_ratio`. | Aligns with `signals.*` namespace in DSL. |
|
||||
| **Materialiser** (`Materialization/`) | Upsert effective findings, append history, manage explain bundle exports. | Mongo transactions per SBOM chunk. |
|
||||
| **Materialiser** (`Materialization/`) | Upsert effective findings, append history, manage explain bundle exports. | PostgreSQL transactions per SBOM chunk. |
|
||||
| **Orchestrator** (`Runs/`) | Change-stream ingestion, fairness, retry/backoff, queue writer. | Works with Scheduler Models DTOs. |
|
||||
| **API** (`Api/`) | Minimal API endpoints, DTO validation, problem responses, idempotency. | Generated clients for CLI/UI. |
|
||||
| **Observability** (`Telemetry/`) | Metrics (`policy_run_seconds`, `rules_fired_total`), traces, structured logs. | Sampled rule-hit logs with redaction. |
|
||||
@@ -183,7 +183,7 @@ Determinism guard instrumentation wraps the evaluator, rejecting access to forbi
|
||||
|
||||
- **Change streams:** Concelier and Excititor publish document changes to the scheduler queue (`policy.trigger.delta`). Payload includes `tenant`, `source`, `linkset digests`, `cursor`.
|
||||
- **Orchestrator:** Maintains per-tenant backlog; merges deltas until time/size thresholds met, then enqueues `PolicyRunRequest`.
|
||||
- **Queue:** Mongo queue with lease; each job assigned `leaseDuration`, `maxAttempts`.
|
||||
- **Queue:** PostgreSQL queue with lease; each job assigned `leaseDuration`, `maxAttempts`.
|
||||
- **Workers:** Lease jobs, execute evaluation pipeline, report status (success/failure/canceled). Failures with recoverable errors requeue with backoff; determinism or schema violations mark job `failed` and raise incident event.
|
||||
- **Fairness:** Round-robin per `{tenant, policyId}`; emergency jobs (`priority=emergency`) jump queue but limited via circuit breaker.
|
||||
- **Replay:** On demand, orchestrator rehydrates run via stored cursors and exports sealed bundle for audit/CI determinism checks.
|
||||
|
||||
Reference in New Issue
Block a user