git.stella-ops.org/docs/export-center/architecture.md

# Export Center Architecture

The Export Center is the dedicated service layer that packages StellaOps evidence and policy overlays into reproducible bundles. It runs as a multi-surface API backed by asynchronous workers and format adapters, enforcing Aggregation-Only Contract (AOC) guardrails while providing deterministic manifests, signing, and distribution paths.

## Runtime topology
- **Export Center API (`StellaOps.ExportCenter.WebService`).** Receives profile CRUD, export run requests, status queries, and download streams through the unified Web API gateway. Enforces tenant scopes, RBAC, quotas, and concurrency guards.
- **Export Center Worker (`StellaOps.ExportCenter.Worker`).** Dequeues export jobs from the Orchestrator, resolves selectors, invokes adapters, and writes manifests and bundle artefacts. Stateless; scales horizontally.
- **Backing stores.**
  - MongoDB collections: `export_profiles`, `export_runs`, `export_inputs`, `export_distributions`, `export_events`.
  - Object storage bucket or filesystem for staging bundle payloads.
  - Optional registry/object storage credentials injected via Authority-scoped secrets.
- **Integration peers.**
  - **Findings Ledger** for advisory, VEX, SBOM payload streaming.
  - **Policy Engine** for deterministic policy snapshots and evaluated findings.
  - **Orchestrator** for job scheduling, quotas, and telemetry fan-out.
  - **Authority** for tenant-aware access tokens and KMS key references.
  - **Console & CLI** as presentation surfaces consuming the API.

## Job lifecycle
1. **Profile selection.** Operator or automation picks a profile (`json:raw`, `json:policy`, `trivy:db`, `trivy:java-db`, `mirror:full`, `mirror:delta`) and submits scope selectors (tenant, time window, products, SBOM subjects, ecosystems). See `docs/export-center/profiles.md` for profile definitions and configuration fields.
2. **Planner resolution.** API validates selectors, expands include/exclude lists, and writes a pending `export_run` with immutable parameters and deterministic ordering hints.
3. **Orchestrator dispatch.** `export_run` triggers a job lease via Orchestrator with quotas per tenant/profile and concurrency caps (default 4 active per tenant).
4. **Worker execution.** Worker streams data from Findings Ledger and Policy Engine using pagination cursors. Adapters write canonical payloads to staging storage, compute checksums, and emit streaming progress events (SSE).
5. **Manifest and provenance emission.** Worker writes `export.json` and `provenance.json`, signs them with configured KMS keys (cosign-compatible), and uploads signatures alongside content.
6. **Distribution registration.** Worker records available distribution methods (download URL, OCI reference, object storage path), raises completion/failure events, and exposes metrics/logs.
7. **Download & verification.** Clients download bundles or pull OCI artefacts, verify signatures, and consume provenance to trace source artefacts.

Cancellation requests mark runs as `aborted` and cause workers to stop iterating sources; partially written files are destroyed and the run is marked with an audit entry.

## Core components
### API surface
- Detailed request and response payloads are catalogued in `docs/export-center/api.md`.
- **Profiles API.**
  - `GET /api/export/profiles`: list tenant-scoped profiles.
  - `POST /api/export/profiles`: create custom profiles (variants of JSON, Trivy, mirror) with validated configuration schema.
  - `PATCH /api/export/profiles/{id}`: update metadata; config changes clone new revision to preserve determinism.
- **Runs API.**
  - `POST /api/export/runs`: submit export run for a profile with selectors and options (policy snapshot id, mirror base manifest).
  - `GET /api/export/runs/{id}`: status, progress counters, provenance summary.
  - `GET /api/export/runs/{id}/events`: server-sent events with state transitions, adapter milestones, signing status.
  - `POST /api/export/runs/{id}/cancel`: cooperative cancellation with audit logging.
- **Downloads API.**
  - `GET /api/export/runs/{id}/download`: streaming download with range support and checksum trailers.
  - `GET /api/export/runs/{id}/manifest`: signed `export.json`.
  - `GET /api/export/runs/{id}/provenance`: signed `provenance.json`.

All endpoints require Authority-issued JWT + DPoP tokens with scopes `export:run`, `export:read`, and tenant claim alignment. Rate-limiting and quotas surface via `X-Stella-Quota-*` headers.

### Worker pipeline
- **Input resolvers.** Query Findings Ledger and Policy Engine using stable pagination (Mongo `_id` ascending, or resume tokens for change streams). Selector expressions compile into Mongo filter fragments and/or API query parameters.
- **Adapter host.** Adapter plugin loader (restart-time only) resolves profile variant to adapter implementation. Adapters present a deterministic `RunAsync(context)` contract with streaming writers and telemetry instrumentation.
- **Content writers.**
  - JSON adapters emit `.jsonl.zst` files with canonical ordering (tenant, subject, document id).
  - Trivy adapters materialise SQLite databases or tar archives matching Trivy DB expectations; schema version gates prevent unsupported outputs.
  - Mirror adapters assemble deterministic filesystem trees (manifests, indexes, payload subtrees) and, when configured, OCI artefact layers.
- **Manifest generator.** Aggregates counts, bytes, hash digests (SHA-256), profile metadata, and input references. Writes `export.json` and `provenance.json` using canonical JSON (sorted keys, RFC3339 UTC timestamps).
- **Signing service.** Integrates with platform KMS via Authority (default cosign signer). Produces in-toto SLSA attestations when configured. Supports detached signatures and optional in-bundle signatures.
- **Distribution drivers.** `dist-http` exposes staged files via download endpoint; `dist-oci` pushes artefacts to registries using ORAS with digest pinning; `dist-objstore` uploads to tenant-specific prefixes with immutability flags.

## Data model snapshots

| Collection | Purpose | Key fields | Notes |
|------------|---------|------------|-------|
| `export_profiles` | Profile definitions (kind, variant, config). | `_id`, `tenant`, `name`, `kind`, `variant`, `config_json`, `created_by`, `created_at`. | Config includes adapter parameters (included record types, compression, encryption). |
| `export_runs` | Run state machine and audit info. | `_id`, `profile_id`, `tenant`, `status`, `requested_by`, `selectors`, `policy_snapshot_id`, `started_at`, `completed_at`, `duration_ms`, `error_code`. | Immutable selectors; status transitions recorded in `export_events`. |
| `export_inputs` | Resolved input ranges. | `run_id`, `source`, `cursor`, `count`, `hash`. | Enables resumable retries and audit. |
| `export_distributions` | Distribution artefacts. | `run_id`, `type` (`http`, `oci`, `object`), `location`, `sha256`, `size_bytes`, `expires_at`. | `expires_at` used for retention policies and automatic pruning. |
| `export_events` | Timeline of state transitions and metrics. | `run_id`, `event_type`, `message`, `at`, `metrics`. | Feeds SSE stream and audit trails. |

## Adapter responsibilities
- **JSON (`json:raw`, `json:policy`).**
  - Ensures canonical casing, timezone normalization, and linkset preservation.
  - Policy variant embeds policy snapshot metadata (`policy_version`, `inputs_hash`, `decision_trace` fingerprint) and emits evaluated findings as separate files.
  - Enforces AOC guardrails: no derived modifications to raw evidence fields.
- **Trivy (`trivy:db`, `trivy:java-db`).**
  - Maps StellaOps advisory schema to Trivy DB format, handling namespace collisions and ecosystem-specific ranges.
  - Validates compatibility against supported Trivy schema versions; run fails fast if mismatch.
  - Emits optional manifest summarising package counts and severity distribution.
- **Mirror (`mirror:full`, `mirror:delta`).**
  - Builds self-contained filesystem layout (`/manifests`, `/data/raw`, `/data/policy`, `/indexes`).
  - Delta variant compares against base manifest (`base_export_id`) to write only changed artefacts; records `removed` entries for cleanup.
  - Supports optional encryption of `/data` subtree (age/AES-GCM) with key wrapping stored in `provenance.json`.

Adapters expose structured telemetry events (`adapter.start`, `adapter.chunk`, `adapter.complete`) with record counts and byte totals per chunk. Failures emit `adapter.error` with reason codes.

## Signing and provenance
- **Manifest schema.** `export.json` contains run metadata, profile descriptor, selector summary, counts, SHA-256 digests, compression hints, and distribution list. Deterministic field ordering and normalized timestamps.
- **Provenance schema.** `provenance.json` captures in-toto subject listing (bundle digest, manifest digest), referenced inputs (findings ledger queries, policy snapshot ids, SBOM identifiers), tool version (`exporter_version`, adapter versions), and KMS key identifiers.
- **Attestation.** Cosign SLSA Level 2 template by default; optional SLSA Level 3 when supply chain attestations are enabled. Detached signatures stored alongside manifests; CLI/Console encourage `cosign verify --key <tenant-key>` workflow.
- **Audit trail.** Each run stores success/failure status, signature identifiers, and verification hints for downstream automation (CI pipelines, offline verification scripts).

## Distribution flows
- **HTTP download.** Console and CLI stream bundles via chunked transfer; supports range requests and resumable downloads. Response includes `X-Export-Digest`, `X-Export-Length`, and optional encryption metadata.
- **OCI push.** Worker uses ORAS to publish bundles as OCI artefacts with annotations describing profile, tenant, manifest digest, and provenance reference. Supports multi-tenant registries with `repository-per-tenant` naming.
- **Object storage.** Writes to tenant-prefixed paths (`s3://stella-exports/{tenant}/{run-id}/...`) with immutable retention policies. Retention scheduler purges expired runs based on profile configuration.
- **Offline Kit seeding.** Mirror bundles optionally staged into Offline Kit assembly pipelines, inheriting the same manifests and signatures.

## Observability
- **Metrics.** Emits `exporter_run_duration_seconds`, `exporter_run_bytes_total{profile}`, `exporter_run_failures_total{error_code}`, `exporter_active_runs{tenant}`, `exporter_distribution_push_seconds{type}`.
- **Logs.** Structured logs with fields `run_id`, `tenant`, `profile_kind`, `adapter`, `phase`, `correlation_id`, `error_code`. Phases include `plan`, `resolve`, `adapter`, `manifest`, `sign`, `distribute`.
- **Traces.** Optional OpenTelemetry spans (`export.plan`, `export.fetch`, `export.write`, `export.sign`, `export.distribute`) for cross-service correlation.
- **Dashboards & alerts.** DevOps pipeline seeds Grafana dashboards summarising throughput, size, failure ratios, and distribution latency. Alert thresholds: failure rate >5% per profile, median run duration >p95 baseline, signature verification failures >0.

## Security posture
- Tenant claim enforced at every query and distribution path; cross-tenant selectors rejected unless explicit cross-tenant mirror feature toggled with signed approval.
- RBAC scopes: `export:profile:manage`, `export:run`, `export:read`, `export:download`. Console hides actions without scope; CLI returns `401/403`.
- Encryption options configurable per profile; keys derived from Authority-managed KMS. Mirror encryption uses tenant-specific recipients; JSON/Trivy rely on transport security plus optional encryption at rest.
- Restart-only plugin loading ensures adapters and distribution drivers are vetted at deployment time, reducing runtime injection risks.
- Deterministic output ensures tamper detection via content hashes; provenance links to source runs and policy snapshots to maintain auditability.

## Deployment considerations
- Packaged as separate API and worker containers. Helm chart and compose overlays define horizontal scaling, worker concurrency, queue leases, and object storage credentials.
- Requires Authority client credentials for KMS and optional registry credentials stored via sealed secrets.
- Offline-first deployments disable OCI distribution by default and provide local object storage endpoints; HTTP downloads served via internal gateway.
- Health endpoints: `/health/ready` validates Mongo connectivity, object storage access, adapter registry integrity, and KMS signer readiness.

## Compliance checklist
- [ ] Profiles and runs enforce tenant scoping; cross-tenant exports disabled unless approved.
- [ ] Manifests and provenance files are generated with deterministic hashes and signed via configured KMS.
- [ ] Adapters run with restart-time registration only; no runtime plugin loading.
- [ ] Distribution drivers respect allowlist; OCI push disabled when offline mode is active.
- [ ] Metrics, logs, and traces follow observability guidelines; dashboards and alerts configured.
- [ ] Retention policies and pruning jobs configured for staged bundles.

> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.