git.stella-ops.org/EPIC_10.md

Fine. Here’s your next brick of “maximum documentation.” Try not to drop it on your foot.

Imposed rule: Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.

---

Epic 10: Export Center (JSON, Trivy DB, Mirror bundles)

Short name: Export Center Primary service: exporter Surfaces: Console (Web UI), CLI, Web API Touches: Conseiller (Feedser), Excitator (Vexer), SBOM Service, Policy Engine, VEX Consensus Lens, Findings Ledger, Authority (authN/Z), Object Storage, Orchestrator, Signing/Attestation, Telemetry

AOC ground rule: Conseiller and Excitator aggregate but never merge. The Export Center serializes evidence and policy results; it does not rewrite or “improve” your data in-flight.

---

1) What it is

The Export Center is the unified system for packaging StellaOps data into portable, verifiable bundles:

• Canonical JSON exports for advisories, VEX, SBOMs, findings, and policy-evaluation snapshots.
• Trivy DB compatible bundles so downstream scanners can use Stella’s curated vulnerability knowledge without custom integrations.
• Mirror bundles for air‑gapped or disconnected environments containing raw evidence, normalized records, indexes, and policy snapshots, with provenance, signatures, and optional encryption.

It centralizes format adapters, compliance and provenance, scheduling, versioning, and distribution (download, OCI push, file share). Every export is reproducible by run_id, cryptographically signed, and audit‑traceable back to source artifacts.

---

2) Why (brief)

Teams need to move results into other scanners, CI systems, and isolated networks without babysitting ten different scripts. The Export Center gives a single, policy‑aware, verifiable exit point that doesn’t surprise compliance or set your ops team on fire.

---

3) How it should work (maximum detail)

3.1 Capabilities

• Profiles

  • json:raw canonical JSON lines for advisories, VEX, SBOMs.
  • json:policy adds policy evaluation results (allow/deny/risk, rationales).
  • trivy:db Trivy DB‑compatible export (core vulnerability DB).
  • trivy:java-db optional Java ecosystem DB export if enabled.
  • mirror:full air‑gap bundle with raw + normalized + indexes + policy + VEX consensus + SBOMs.
  • mirror:delta incremental bundle based on a previous export manifest.

• Scope selectors

  • By time window, product, ecosystem, package, image digest, repository, tenant, tag.
  • Include/exclude SBOMs, advisories, VEX, findings, policy snapshots.

• Policy awareness

  • Optionally bake a policy snapshot into the bundle, including the policy version, inputs and decision traces.
  • Can export raw evidence only (AOC) or raw + evaluated.

• Provenance & signing

  • Generate attestation metadata with source URIs, artifact hashes, schema versions, export profile and filters.
  • Sign manifests and bundle using configured KMS; support detached and in‑bundle signatures.

• Distribution

  • Download via Console or API (streaming).
  • Push to OCI registry as an artifact/image with annotations.
  • Write to object storage prefix for batch pickup.

• Scheduling & automation

  • One‑off, cron, and event‑triggered exports (e.g., after a “VEX consensus recompute” run).
  • Retention policies and automatic expiry for old bundles.

• Observability

  • Export metrics, throughput, size, downstream pull counts (when pushed to registry with report backs).

3.2 Architecture

• exporter (service)

  • Orchestrates export runs, gathers records from the ledger and indexes, calls format adapters, writes bundles, signs, and publishes distribution tasks.
  • Stateless workers pull “export.jobs” from the orchestrator, stream data, and write manifests into object storage.

• format adapters

  • Pluggable adapters:

    • adapter-json: canonicalized JSONL writers per record type.
    • adapter-trivy: translates Stella’s normalized advisory model into Trivy DB format (and Java DB if enabled).
    • adapter-mirror: constructs a portable filesystem/OCI layout with manifests, indexes, and data subtrees.

• manifesting & provenance

  • export.json (export manifest): profile, filters, counts, schema versions, content checksums, start/finish times, inputs list (artifact ids + hashes).
  • provenance.json: full chain back to source runs and artifacts; linked signatures.

• distribution engines

  • dist-http streaming for downloads.
  • dist-oci layer writer with descriptors and annotations.
  • dist-objstore for staging to buckets.

• security

  • Tenant scoping, RBAC on export creation and retrieval.
  • Optional in‑bundle encryption (age/AES‑GCM) with key wrapping using KMS.

3.3 Data model (selected tables)

• export_profiles

  • id, name, kind (json|trivy|mirror), variant (raw|policy|db|java-db|full|delta), config_json, created_by, created_at.

• export_runs

  • id, profile_id, trigger (manual|schedule|event|api), state, filters_json, policy_version, started_at, finished_at, artifact_uri, size_bytes, sig_uri, provenance_uri, tenant_id, error_class, error_message.

• export_inputs

  • Link table between export_runs and source artifacts. export_run_id, artifact_id, hash.

• export_distributions

  • export_run_id, type (download|oci|objstore), target, state, meta_json, created_at, updated_at.

3.4 Canonical file layouts

JSON profile output Directory layout under export root:

/export/
  export.json                 # export manifest
  provenance.json             # provenance and source artifact chain
  signatures/
    export.sig                # detached signature for export.json
  advisories/
    normalized.jsonl          # normalized advisory records
  vex/
    normalized.jsonl          # normalized VEX records
  sboms/
    <subject>/sbom.spdx.json  # one per subject; SPDX JSON or CycloneDX JSON
  findings/
    policy_evaluated.jsonl    # if profile=json:policy

Trivy DB profile output Produced as a compressed artifact:

/export/
  export.json
  provenance.json
  trivy/
    db.bundle                 # Trivy DB compatible archive
    java-db.bundle            # optional Java DB bundle (if enabled)
  signatures/
    trivy-db.sig

Notes:

• The adapter keeps an internal mapping of Stella normalized fields to Trivy’s expected fields and namespaces. The mapping is versioned to track upstream schema evolution.

Mirror bundle (filesystem layout)

/mirror/
  manifest.yaml               # high-level bundle manifest (profile, filters, counts)
  export.json                 # same as JSON profile
  provenance.json
  indexes/
    advisories.index.json     # quick lookups (pkg -> advisory ids)
    vex.index.json
    sbom.index.json
  advisories/raw/...
  advisories/normalized/...
  vex/raw/...
  vex/normalized/...
  sboms/raw/...
  sboms/graph/...
  policy/
    snapshot.yaml             # full policy set used for evaluation
    evaluations.jsonl         # decision outputs if requested
  consensus/
    vex_consensus.jsonl
  signatures/
    manifest.sig
    export.sig
  README.md

Mirror bundle (OCI layout) Following standard OCI image artifact layout with annotations (org.opencontainers.artifact.description, com.stella.export.profile, com.stella.export.filters), and manifest lists for large bundles.

3.5 Export workflow

1. Plan Exporter computes candidates based on filters. For mirror:delta, compares with previous manifest to compute changes.

2. Stream & write Records are streamed from the Findings Ledger and stores. Writers are forward‑only, emitting JSONL or adapter‑specific structures, chunked for memory safety.

3. Sign & attest Once all content hashes are stable, Export Center writes export.json, provenance.json, and signs using KMS. Optional encryption wraps data layers.

4. Distribute Depending on profile settings, it exposes a download URL, pushes an OCI artifact, or writes to object storage. Distribution metadata is recorded.

5. Audit & retention Run, manifest, and signatures are immutable. Retention policy prunes large data folders after N days with manifests retained longer.

3.6 APIs

POST   /export/profiles
GET    /export/profiles?kind=&variant=
GET    /export/profiles/{id}
PATCH  /export/profiles/{id}
DELETE /export/profiles/{id}

POST   /export/runs
GET    /export/runs?state=&profile_id=&from=&to=&tenant_id=
GET    /export/runs/{run_id}
POST   /export/runs/{run_id}/cancel

GET    /export/runs/{run_id}/download                # presigned URL or streaming
POST   /export/runs/{run_id}/distribute              # { "type":"oci|objstore", "target":"..." }
GET    /export/runs/{run_id}/manifest                # export.json
GET    /export/runs/{run_id}/provenance
GET    /export/runs/{run_id}/signatures

GET    /export/metrics/overview
WS     /export/streams/updates

Request example (create run):

{
  "profile_id": "prof_json_policy",
  "filters": {
    "time_from": "2025-01-01T00:00:00Z",
    "time_to": "2025-01-31T23:59:59Z",
    "ecosystems": ["pypi", "npm"],
    "include": ["advisories", "vex", "sboms", "findings"]
  },
  "distribution": { "type": "download" },
  "policy_version": "pol-v1.8.2"
}

3.7 CLI

stella export profiles list --kind mirror
stella export profiles create --file profile.yaml
stella export run create --profile prof_json_policy --from 2025-01-01 --to 2025-01-31 --include advisories,vex,sboms --download
stella export run status <run-id>
stella export run cancel <run-id>
stella export run get <run-id> --manifest
stella export run download <run-id> --out export-jan.tar.zst
stella export distribute <run-id> --oci ghcr.io/org/stella-export:jan2025
stella export verify <bundle> --manifest export.json --sig signatures/export.sig

Exit codes: 0 ok, 2 bad args, 4 not found, 5 denied, 6 integrity failed, 8 export error.

3.8 RBAC & security

• Roles:

  • Export.Viewer: list runs, download completed bundles.
  • Export.Operator: create runs, cancel, schedule, distribute.
  • Export.Admin: manage profiles, set retention, manage signing keys.

• Tenancy:

  • Every run and artifact scoped by tenant; cross‑tenant export is disallowed.

• Secrets:

  • KMS references for signing and encryption; never store private keys.

• PII & redaction:

  • Exporters must not include secrets or credentials. Redaction rules enforced at writer level with schema‑based allowlists.

3.9 Observability

• Metrics:

  • export_bytes_total{profile,tenant}
  • export_records_total{type}
  • export_duration_ms{profile}
  • export_failures_total{error_class}
  • export_distributions_total{type}
  • export_verify_fail_total

• Traces:

  • Spans per export phase: plan, stream, write, sign, distribute; baggage includes export_run_id.

• Logs:

  • Structured JSON with counts, sizes, hashes, and redaction hints.

3.10 Performance targets

• Stream throughput ≥ 25k records/sec per worker for JSONL writing with compression.
• Trivy bundle generation for 1M advisories ≤ 8 minutes on a standard worker.
• Mirror delta export for 5% change set ≤ 2 minutes.

3.11 Edge cases & behavior

• Schema drift: adapter refuses to emit unknown fields without explicit mapping; run fails with error_class=schema_mismatch.
• Oversized bundles: automatic sharding by time or content type; mirror OCI uses multi‑manifest indices.
• Missing policy snapshot: profile json:policy will auto‑pull latest version unless pinned; pinning is recommended for reproducibility.
• Duplicate evidence: writers dedupe by artifact hash and advisory id; AOC forbids merging.
• Air‑gap encryption: if encrypt=true, mirror bundles require recipient public key material; decryption tooling documented.

---

4) Implementation plan

4.1 Modules

• New service: src/StellaOps.ExportCenter

  • api/ REST + WS

  • planner/ scope planning, delta computation, sampling

  • adapters/

    • json/ canonical writers
    • trivy/ db builders and schema mapping
    • mirror/ fs/OCI builders, sharding, delta logic

  • signing/ KMS clients, attestations

  • dist/ download streaming, OCI push, object storage writer

  • state/ repositories, migrations

  • metrics/, audit/, security/

• SDK/CLI

  • src/StellaOps.Cli subcommands with streaming download and verification.

• Console

  • console/apps/export-center/ pages:

    • Overview, Profiles, Runs, Run Detail, Distributions, Settings.

  • Components: ExportPlanPreview, ProfileEditor, RunDiff, VerifyPanel.

• Existing services updates

  • Findings Ledger: new paginated streaming endpoints for advisories/VEX/SBOM/findings by filters and snapshots.
  • Policy Engine: “policy snapshot” exportable endpoint.
  • VEX Lens: consensus snapshot endpoint.

4.2 Packaging & deployment

• Containers:

  • stella/exporter:<ver>
  • stella/exporter-worker:<ver> (optional separated worker pool)

• Helm:

  • WS replicas, concurrent export limits, default compression (zstd), default retention, KMS settings, OCI creds secrets.

• DB migrations:

  • Create export_* tables with proper indices (tenant, time, state).

4.3 Rollout

• Phase 1: JSON (raw, policy) and Mirror (full) as download only.
• Phase 2: Trivy DB adapters, OCI distribution.
• Phase 3: Mirror deltas, encryption, verification tooling, scheduling.

---

5) Documentation changes

Create/update the following docs; each page must end with the imposed rule statement.

1. /docs/export-center/overview.md Purpose, profiles, supported targets, AOC alignment, security model.

2. /docs/export-center/architecture.md Service components, adapters, manifests, signing, distribution flows.

3. /docs/export-center/profiles.md Profile schemas, examples, versioning, compatibility notes.

4. /docs/export-center/api.md All endpoints with request/response examples and error codes.

5. /docs/export-center/cli.md Commands with examples, scripts for CI/CD, verification.

6. /docs/export-center/mirror-bundles.md Filesystem and OCI layouts, delta exports, encryption, air‑gap import guide.

7. /docs/export-center/trivy-adapter.md Field mapping, supported ecosystems, compatibility and test matrix.

8. /docs/export-center/provenance-and-signing.md Manifest format, attestation details, verification process.

9. /docs/operations/export-runbook.md Common failures, recovery, tuning, capacity planning.

10. /docs/security/export-hardening.md RBAC, tenant isolation, secret redaction, encryption keys.

Imposed rule: Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.

---

6) Engineering tasks

Backend: exporter

• Migrations for export_profiles, export_runs, export_inputs, export_distributions.
• Planner to resolve filters to iterators over advisory/VEX/SBOM/findings datasets with pagination.
• JSON adapter: canonical JSONL writers with schema normalization and redaction enforcement.
• Policy snapshot embedder: pull policy version and evaluation outputs when requested.
• Trivy adapter: implement schema mapping, writer, integrity validation, and compatibility version flag.
• Mirror adapter: filesystem and OCI writer, sharding, manifest creation, delta computation.
• Signing/attestation using KMS; detached and embedded options.
• Distribution engines: download streaming, OCI push, object storage staging.
• API layer with async export run handling and WebSocket updates.
• Rate limit and concurrency controls per tenant/profile.
• Audit logging for all create/cancel/distribute/verify actions.

Integrations

• Findings Ledger streaming APIs for each content type.
• Policy Engine endpoint to return deterministic policy snapshot and decision set by run.
• VEX Lens endpoint to expose consensus snapshot.

Console

• Profiles CRUD with validation and test preview.
• Create Run wizard with live count estimates and storage footprint prediction.
• Runs list + detail page with manifest, provenance, and quick verify.
• Download and distribution actions with progress and logs.
• Verification panel to check signatures and hashes client‑side.

CLI

• stella export commands as defined; include verify that checks signatures and hashes.
• Auto‑resume of interrupted downloads with range requests.
• Friendly error messages for schema mismatch and verification failure.

Observability

• Metrics and traces per §3.9; dashboards for throughput, durations, failures, sizes.
• Alerts for export failure rate and verify failures.

Security & RBAC

• Enforce tenant scoping at query level; fuzz tests for leakage.
• Role matrix checks on each API; Console hides forbidden actions.
• Encryption test vectors and key rotation procedure.

Docs

• Author all files in §5 with concrete examples and diagrams.
• Cross‑link from Orchestrator, Policy Studio, VEX Lens, and SBOM docs to the Export Center pages.
• Append imposed rule line at the end of each page.

Imposed rule: Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.

---

7) Implementation notes per profile

7.1 JSON: raw

• Content: advisories.normalized, vex.normalized, sboms (SPDX/CycloneDX), optional findings.raw.
• Normalization: enforce Stella field casing, timestamps in RFC3339, unicode NFC.
• Compression: .jsonl.zst per file to allow split/merge.

7.2 JSON: policy

• Adds: policy_snapshot and findings.policy_evaluated.jsonl with decision, rule_id, rationale, inputs fingerprint.
• Determinism: include policy_version and inputs_hash; replays should match exactly.

7.3 Trivy DB

• Mapping:

  • Package name, ecosystem, version ranges, CVE/CWE/aliases, severity mapping, vendor statements, fixed versions.
  • Ensure namespace mapping avoids collisions (e.g., distro vs ecosystem).

• Compatibility: version flag in manifest; adapter throws if upstream schema version not supported.

• Validation: run post‑build sanity checks (counts, required indexes).

7.4 Mirror: full/delta

• Full: everything needed to spin up an isolated read‑only Stella mirror with local search.
• Delta: compute changed/added/removed advisory ids, VEX statements, SBOM subjects; update indexes and manifest with base_export_id.
• Encryption: if enabled, encrypt data subtrees; leave manifest.yaml unencrypted for discoverability unless strict=true.

---

8) Acceptance criteria

• Operators can create an export with filters, download it, verify signatures, and trace back to source artifacts via provenance.
• Trivy adapter produces a bundle consumable by Trivy without custom flags (basic validation in CI).
• Mirror bundle imports successfully in an air‑gapped “mirror‑reader” sample app and serves queries from indexes.
• Policy‑aware exports include deterministic decisions matching the specified policy_version.
• RBAC prevents a Viewer from creating or canceling exports; tenancy prevents cross‑tenant leakage.
• Metrics and dashboards show per‑profile throughput and error classes; alerts trigger on failure spikes.
• Export retries are idempotent and do not duplicate content; hashes stable across re‑runs with identical inputs.

---

9) Risks & mitigations

• Upstream schema changes break Trivy export. Mitigation: versioned adapter with compatibility gate; integration tests against known fixtures; fail early with clear remediation.

• Bundle size explosion. Mitigation: zstd compression, sharding, delta exports, content‑addressed storage reuse for mirror OCI.

• Data leakage via exports. Mitigation: strict allowlist schemas, redaction filters, RBAC, tenant scoping tests, encryption for mirror.

• Non‑deterministic policy outputs. Mitigation: pin policy version and inputs hash; snapshot embedded rules; deterministic evaluation mode only.

• Slow downloads/timeouts. Mitigation: streaming with range support, resumable downloads, CDN integration if needed.

---

10) Test plan

• Unit Schema normalization; Trivy mapping; mirror delta computation; manifest hashing; signing.

• Integration End‑to‑end export with each profile; verify bundles; replay determinism of policy exports; OCI push/pull.

• Compatibility Validate Trivy bundle against a matrix of versions; import mirror bundle into a reference reader and run queries.

• Security Tenant isolation fuzzing; redaction checks; encryption round‑trip; signature verification with rotated keys.

• Performance Large dataset generation; parallel writer stress; OCI multi‑manifest publishing; download resume under packet loss.

• Chaos Kill exporter mid‑write; ensure resume or clean failure without partial corrupt bundles.

---

11) Philosophy

• Ports, not prisons. Exports should free your data to move with integrity and context, not trap it in a proprietary maze.
• Reproducible or it didn’t happen. Every bit derived from known inputs, signed, traceable.
• Air‑gap is a first‑class citizen. Mirror bundles are not an afterthought; they’re how serious orgs actually run.

Final reminder: Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.