feat(rate-limiting): Implement core rate limiting functionality with configuration, decision-making, metrics, middleware, and service registration
- Add RateLimitConfig for configuration management with YAML binding support. - Introduce RateLimitDecision to encapsulate the result of rate limit checks. - Implement RateLimitMetrics for OpenTelemetry metrics tracking. - Create RateLimitMiddleware for enforcing rate limits on incoming requests. - Develop RateLimitService to orchestrate instance and environment rate limit checks. - Add RateLimitServiceCollectionExtensions for dependency injection registration.
This commit is contained in:
master
@@ -4,11 +4,11 @@
|
||||
|
||||
The Export Center is the dedicated service layer that packages StellaOps evidence and policy overlays into reproducible bundles. It runs as a multi-surface API backed by asynchronous workers and format adapters, enforcing Aggregation-Only Contract (AOC) guardrails while providing deterministic manifests, signing, and distribution paths.
|
||||
|
||||
## Runtime topology
|
||||
## Runtime topology
|
||||
- **Export Center API (`StellaOps.ExportCenter.WebService`).** Receives profile CRUD, export run requests, status queries, and download streams through the unified Web API gateway. Enforces tenant scopes, RBAC, quotas, and concurrency guards.
|
||||
- **Export Center Worker (`StellaOps.ExportCenter.Worker`).** Dequeues export jobs from the Orchestrator, resolves selectors, invokes adapters, and writes manifests and bundle artefacts. Stateless; scales horizontally.
|
||||
- **Backing stores.**
|
||||
- MongoDB collections: `export_profiles`, `export_runs`, `export_inputs`, `export_distributions`, `export_events`.
|
||||
- PostgreSQL tables: `export_profiles`, `export_runs`, `export_inputs`, `export_distributions`, `export_events`.
|
||||
- Object storage bucket or filesystem for staging bundle payloads.
|
||||
- Optional registry/object storage credentials injected via Authority-scoped secrets.
|
||||
- **Integration peers.**
|
||||
@@ -16,16 +16,16 @@ The Export Center is the dedicated service layer that packages StellaOps evidenc
|
||||
- **Policy Engine** for deterministic policy snapshots and evaluated findings.
|
||||
- **Orchestrator** for job scheduling, quotas, and telemetry fan-out.
|
||||
- **Authority** for tenant-aware access tokens and KMS key references.
|
||||
- **Console & CLI** as presentation surfaces consuming the API.
|
||||
|
||||
## Gap remediation (EC1–EC10)
|
||||
- Schemas: publish signed `ExportProfile` + manifest schemas with selector validation; keep in repo alongside OpenAPI docs.
|
||||
- Determinism: per-adapter ordering/compression rules with rerun-hash CI; pin Trivy DB schema versions.
|
||||
- Provenance: DSSE/SLSA attestations with log metadata for every export run; include tenant IDs in predicates.
|
||||
- Integrity: require checksum/signature headers and OCI annotations; mirror delta/tombstone rules documented for adapters.
|
||||
- Security: cross-tenant exports denied by default; enforce approval tokens and encryption recipient validation.
|
||||
- Offline parity: provide export-kit packaging + verify script for air-gap consumers; include fixtures under `src/ExportCenter/__fixtures`.
|
||||
- Advisory link: see `docs/product-advisories/28-Nov-2025 - Export Center and Reporting Strategy.md` (EC1–EC10) for original requirements and keep it alongside sprint tasks for implementers.
|
||||
- **Console & CLI** as presentation surfaces consuming the API.
|
||||
|
||||
## Gap remediation (EC1–EC10)
|
||||
- Schemas: publish signed `ExportProfile` + manifest schemas with selector validation; keep in repo alongside OpenAPI docs.
|
||||
- Determinism: per-adapter ordering/compression rules with rerun-hash CI; pin Trivy DB schema versions.
|
||||
- Provenance: DSSE/SLSA attestations with log metadata for every export run; include tenant IDs in predicates.
|
||||
- Integrity: require checksum/signature headers and OCI annotations; mirror delta/tombstone rules documented for adapters.
|
||||
- Security: cross-tenant exports denied by default; enforce approval tokens and encryption recipient validation.
|
||||
- Offline parity: provide export-kit packaging + verify script for air-gap consumers; include fixtures under `src/ExportCenter/__fixtures`.
|
||||
- Advisory link: see `docs/product-advisories/28-Nov-2025 - Export Center and Reporting Strategy.md` (EC1–EC10) for original requirements and keep it alongside sprint tasks for implementers.
|
||||
|
||||
## Job lifecycle
|
||||
1. **Profile selection.** Operator or automation picks a profile (`json:raw`, `json:policy`, `trivy:db`, `trivy:java-db`, `mirror:full`, `mirror:delta`) and submits scope selectors (tenant, time window, products, SBOM subjects, ecosystems). See `docs/modules/export-center/profiles.md` for profile definitions and configuration fields.
|
||||
@@ -58,7 +58,7 @@ Cancellation requests mark runs as `aborted` and cause workers to stop iterating
|
||||
All endpoints require Authority-issued JWT + DPoP tokens with scopes `export:run`, `export:read`, and tenant claim alignment. Rate-limiting and quotas surface via `X-Stella-Quota-*` headers.
|
||||
|
||||
### Worker pipeline
|
||||
- **Input resolvers.** Query Findings Ledger and Policy Engine using stable pagination (Mongo `_id` ascending, or resume tokens for change streams). Selector expressions compile into Mongo filter fragments and/or API query parameters.
|
||||
- **Input resolvers.** Query Findings Ledger and Policy Engine using stable pagination (PostgreSQL `id` ascending, or cursor-based pagination). Selector expressions compile into PostgreSQL WHERE clauses and/or API query parameters.
|
||||
- **Adapter host.** Adapter plugin loader (restart-time only) resolves profile variant to adapter implementation. Adapters present a deterministic `RunAsync(context)` contract with streaming writers and telemetry instrumentation.
|
||||
- **Content writers.**
|
||||
- JSON adapters emit `.jsonl.zst` files with canonical ordering (tenant, subject, document id).
|
||||
@@ -75,40 +75,40 @@ All endpoints require Authority-issued JWT + DPoP tokens with scopes `export:run
|
||||
| `export_profiles` | Profile definitions (kind, variant, config). | `_id`, `tenant`, `name`, `kind`, `variant`, `config_json`, `created_by`, `created_at`. | Config includes adapter parameters (included record types, compression, encryption). |
|
||||
| `export_runs` | Run state machine and audit info. | `_id`, `profile_id`, `tenant`, `status`, `requested_by`, `selectors`, `policy_snapshot_id`, `started_at`, `completed_at`, `duration_ms`, `error_code`. | Immutable selectors; status transitions recorded in `export_events`. |
|
||||
| `export_inputs` | Resolved input ranges. | `run_id`, `source`, `cursor`, `count`, `hash`. | Enables resumable retries and audit. |
|
||||
| `export_distributions` | Distribution artefacts. | `run_id`, `type` (`http`, `oci`, `object`), `location`, `sha256`, `size_bytes`, `expires_at`. | `expires_at` used for retention policies and automatic pruning. |
|
||||
| `export_events` | Timeline of state transitions and metrics. | `run_id`, `event_type`, `message`, `at`, `metrics`. | Feeds SSE stream and audit trails. |
|
||||
|
||||
## Audit bundles (immutable triage exports)
|
||||
|
||||
Audit bundles are a specialized Export Center output: a deterministic, immutable evidence pack for a single subject (and optional time window) suitable for audits and incident response.
|
||||
|
||||
- **Schema**: `docs/schemas/audit-bundle-index.schema.json` (bundle index/manifest with integrity hashes and referenced artefacts).
|
||||
- **Core APIs**:
|
||||
- `POST /v1/audit-bundles` - Create a new bundle (async generation).
|
||||
- `GET /v1/audit-bundles` - List previously created bundles.
|
||||
- `GET /v1/audit-bundles/{bundleId}` - Returns job metadata (`Accept: application/json`) or streams bundle bytes (`Accept: application/octet-stream`).
|
||||
- **Typical contents**: vuln reports, SBOM(s), VEX decisions, policy evaluations, and DSSE attestations, plus an integrity root hash and optional OCI reference.
|
||||
- **Reference**: `docs/product-advisories/archived/27-Nov-2025-superseded/28-Nov-2025 - Vulnerability Triage UX & VEX-First Decisioning.md`.
|
||||
|
||||
## Adapter responsibilities
|
||||
- **JSON (`json:raw`, `json:policy`).**
|
||||
- Ensures canonical casing, timezone normalization, and linkset preservation.
|
||||
- Policy variant embeds policy snapshot metadata (`policy_version`, `inputs_hash`, `decision_trace` fingerprint) and emits evaluated findings as separate files.
|
||||
| `export_distributions` | Distribution artefacts. | `run_id`, `type` (`http`, `oci`, `object`), `location`, `sha256`, `size_bytes`, `expires_at`. | `expires_at` used for retention policies and automatic pruning. |
|
||||
| `export_events` | Timeline of state transitions and metrics. | `run_id`, `event_type`, `message`, `at`, `metrics`. | Feeds SSE stream and audit trails. |
|
||||
|
||||
## Audit bundles (immutable triage exports)
|
||||
|
||||
Audit bundles are a specialized Export Center output: a deterministic, immutable evidence pack for a single subject (and optional time window) suitable for audits and incident response.
|
||||
|
||||
- **Schema**: `docs/schemas/audit-bundle-index.schema.json` (bundle index/manifest with integrity hashes and referenced artefacts).
|
||||
- **Core APIs**:
|
||||
- `POST /v1/audit-bundles` - Create a new bundle (async generation).
|
||||
- `GET /v1/audit-bundles` - List previously created bundles.
|
||||
- `GET /v1/audit-bundles/{bundleId}` - Returns job metadata (`Accept: application/json`) or streams bundle bytes (`Accept: application/octet-stream`).
|
||||
- **Typical contents**: vuln reports, SBOM(s), VEX decisions, policy evaluations, and DSSE attestations, plus an integrity root hash and optional OCI reference.
|
||||
- **Reference**: `docs/product-advisories/archived/27-Nov-2025-superseded/28-Nov-2025 - Vulnerability Triage UX & VEX-First Decisioning.md`.
|
||||
|
||||
## Adapter responsibilities
|
||||
- **JSON (`json:raw`, `json:policy`).**
|
||||
- Ensures canonical casing, timezone normalization, and linkset preservation.
|
||||
- Policy variant embeds policy snapshot metadata (`policy_version`, `inputs_hash`, `decision_trace` fingerprint) and emits evaluated findings as separate files.
|
||||
- Enforces AOC guardrails: no derived modifications to raw evidence fields.
|
||||
- **Trivy (`trivy:db`, `trivy:java-db`).**
|
||||
- Maps StellaOps advisory schema to Trivy DB format, handling namespace collisions and ecosystem-specific ranges.
|
||||
- Validates compatibility against supported Trivy schema versions; run fails fast if mismatch.
|
||||
- Emits optional manifest summarising package counts and severity distribution.
|
||||
- **Mirror (`mirror:full`, `mirror:delta`).**
|
||||
- Builds self-contained filesystem layout (`/manifests`, `/data/raw`, `/data/policy`, `/indexes`).
|
||||
- Delta variant compares against base manifest (`base_export_id`) to write only changed artefacts; records `removed` entries for cleanup.
|
||||
- Supports optional encryption of `/data` subtree (age/AES-GCM) with key wrapping stored in `provenance.json`.
|
||||
- **DevPortal (`devportal:offline`).**
|
||||
- Packages developer portal static assets, OpenAPI specs, SDK releases, and changelog content into a reproducible archive with manifest/checksum pairs.
|
||||
- Emits `manifest.json`, `checksums.txt`, helper scripts, and a DSSE signature document (`manifest.dsse.json`) as described in [DevPortal Offline Bundle Specification](devportal-offline.md).
|
||||
- Stores artefacts under `<storagePrefix>/<bundleId>/` and signs manifests via the Export Center signing adapter (HMAC-SHA256 v1, tenant scoped).
|
||||
|
||||
Adapters expose structured telemetry events (`adapter.start`, `adapter.chunk`, `adapter.complete`) with record counts and byte totals per chunk. Failures emit `adapter.error` with reason codes.
|
||||
- **Mirror (`mirror:full`, `mirror:delta`).**
|
||||
- Builds self-contained filesystem layout (`/manifests`, `/data/raw`, `/data/policy`, `/indexes`).
|
||||
- Delta variant compares against base manifest (`base_export_id`) to write only changed artefacts; records `removed` entries for cleanup.
|
||||
- Supports optional encryption of `/data` subtree (age/AES-GCM) with key wrapping stored in `provenance.json`.
|
||||
- **DevPortal (`devportal:offline`).**
|
||||
- Packages developer portal static assets, OpenAPI specs, SDK releases, and changelog content into a reproducible archive with manifest/checksum pairs.
|
||||
- Emits `manifest.json`, `checksums.txt`, helper scripts, and a DSSE signature document (`manifest.dsse.json`) as described in [DevPortal Offline Bundle Specification](devportal-offline.md).
|
||||
- Stores artefacts under `<storagePrefix>/<bundleId>/` and signs manifests via the Export Center signing adapter (HMAC-SHA256 v1, tenant scoped).
|
||||
|
||||
Adapters expose structured telemetry events (`adapter.start`, `adapter.chunk`, `adapter.complete`) with record counts and byte totals per chunk. Failures emit `adapter.error` with reason codes.
|
||||
|
||||
## Signing and provenance
|
||||
- **Manifest schema.** `export.json` contains run metadata, profile descriptor, selector summary, counts, SHA-256 digests, compression hints, and distribution list. Deterministic field ordering and normalized timestamps.
|
||||
@@ -122,11 +122,11 @@ Adapters expose structured telemetry events (`adapter.start`, `adapter.chunk`, `
|
||||
- **Object storage.** Writes to tenant-prefixed paths (`s3://stella-exports/{tenant}/{run-id}/...`) with immutable retention policies. Retention scheduler purges expired runs based on profile configuration.
|
||||
- **Offline Kit seeding.** Mirror bundles optionally staged into Offline Kit assembly pipelines, inheriting the same manifests and signatures.
|
||||
|
||||
## Observability
|
||||
- **Metrics.** Emits `exporter_run_duration_seconds`, `exporter_run_bytes_total{profile}`, `exporter_run_failures_total{error_code}`, `exporter_active_runs{tenant}`, `exporter_distribution_push_seconds{type}`.
|
||||
- **Logs.** Structured logs with fields `run_id`, `tenant`, `profile_kind`, `adapter`, `phase`, `correlation_id`, `error_code`. Phases include `plan`, `resolve`, `adapter`, `manifest`, `sign`, `distribute`.
|
||||
- **Traces.** Optional OpenTelemetry spans (`export.plan`, `export.fetch`, `export.write`, `export.sign`, `export.distribute`) for cross-service correlation.
|
||||
- **Dashboards & alerts.** DevOps pipeline seeds Grafana dashboards summarising throughput, size, failure ratios, and distribution latency. Alert thresholds: failure rate >5% per profile, median run duration >p95 baseline, signature verification failures >0. Runbook + dashboard stub for offline import: `operations/observability.md`, `operations/dashboards/export-center-observability.json`.
|
||||
## Observability
|
||||
- **Metrics.** Emits `exporter_run_duration_seconds`, `exporter_run_bytes_total{profile}`, `exporter_run_failures_total{error_code}`, `exporter_active_runs{tenant}`, `exporter_distribution_push_seconds{type}`.
|
||||
- **Logs.** Structured logs with fields `run_id`, `tenant`, `profile_kind`, `adapter`, `phase`, `correlation_id`, `error_code`. Phases include `plan`, `resolve`, `adapter`, `manifest`, `sign`, `distribute`.
|
||||
- **Traces.** Optional OpenTelemetry spans (`export.plan`, `export.fetch`, `export.write`, `export.sign`, `export.distribute`) for cross-service correlation.
|
||||
- **Dashboards & alerts.** DevOps pipeline seeds Grafana dashboards summarising throughput, size, failure ratios, and distribution latency. Alert thresholds: failure rate >5% per profile, median run duration >p95 baseline, signature verification failures >0. Runbook + dashboard stub for offline import: `operations/observability.md`, `operations/dashboards/export-center-observability.json`.
|
||||
|
||||
## Security posture
|
||||
- Tenant claim enforced at every query and distribution path; cross-tenant selectors rejected unless explicit cross-tenant mirror feature toggled with signed approval.
|
||||
@@ -139,7 +139,7 @@ Adapters expose structured telemetry events (`adapter.start`, `adapter.chunk`, `
|
||||
- Packaged as separate API and worker containers. Helm chart and compose overlays define horizontal scaling, worker concurrency, queue leases, and object storage credentials.
|
||||
- Requires Authority client credentials for KMS and optional registry credentials stored via sealed secrets.
|
||||
- Offline-first deployments disable OCI distribution by default and provide local object storage endpoints; HTTP downloads served via internal gateway.
|
||||
- Health endpoints: `/health/ready` validates Mongo connectivity, object storage access, adapter registry integrity, and KMS signer readiness.
|
||||
- Health endpoints: `/health/ready` validates PostgreSQL connectivity, object storage access, adapter registry integrity, and KMS signer readiness.
|
||||
|
||||
## Compliance checklist
|
||||
- [ ] Profiles and runs enforce tenant scoping; cross-tenant exports disabled unless approved.
|
||||
|
||||
Reference in New Issue
Block a user