feat(docs): Add comprehensive documentation for Vexer, Vulnerability Explorer, and Zastava modules

- Introduced AGENTS.md, README.md, TASKS.md, and implementation_plan.md for Vexer, detailing mission, responsibilities, key components, and operational notes. - Established similar documentation structure for Vulnerability Explorer and Zastava modules, including their respective workflows, integrations, and observability notes. - Created risk scoring profiles documentation outlining the core workflow, factor model, governance, and deliverables. - Ensured all modules adhere to the Aggregation-Only Contract and maintain determinism and provenance in outputs.
2025-10-30 00:09:39 +02:00
parent 3154c67978
commit 7b5bdcf4d3
503 changed files with 16136 additions and 54638 deletions
--- a/docs/modules/export-center/profiles.md
+++ b/docs/modules/export-center/profiles.md
@@ -0,0 +1,139 @@
+# Export Center Profiles
+
+Export Center profiles define what data is collected, how it is encoded, and which distribution paths are enabled for a run. Profiles are tenant-scoped and deterministic: identical selectors and source data produce identical bundles. This guide summarises built-in profiles, configuration fields, schema conventions, and compatibility notes.
+
+## Profile catalogue
+
+| Profile | Kind / Variant | Output artefacts | Primary use cases |
+|---------|----------------|------------------|-------------------|
+| `json:raw` | `json` / `raw` | Canonical JSONL archives of advisories, VEX, SBOMs | Evidence escrow, analytics pipelines |
+| `json:policy` | `json` / `policy` | `json:raw` artefacts plus policy snapshot + evaluated findings | Audit, compliance attestations |
+| `trivy:db` | `trivy` / `db` | Trivy-compatible vulnerability database | Feeding external scanners / CI |
+| `trivy:java-db` | `trivy` / `java-db` | Java ecosystem supplement for Trivy | Supply Java CVE data to Trivy |
+| `mirror:full` | `mirror` / `full` | Complete mirror bundle (raw, policy, indexes, provenance) | Air-gap deployments, disaster recovery |
+| `mirror:delta` | `mirror` / `delta` | Incremental changes relative to a prior manifest | Efficient mirror updates |
+
+Profiles can be cloned and customised; configuration is immutable per revision to keep runs reproducible.
+
+## Common configuration fields
+
+| Field | Description | Applies to | Notes |
+|-------|-------------|------------|-------|
+| `name` | Human-readable identifier displayed in Console/CLI | All | Unique per tenant. |
+| `kind` | Logical family (`json`, `trivy`, `mirror`) | All | Determines eligible adapters. |
+| `variant` | Specific export flavour (see table above) | All | Controls adapter behaviour. |
+| `include` | Record types to include (`advisories`, `vex`, `sboms`, `findings`) | JSON, mirror | Defaults depend on variant. |
+| `policySnapshotMode` | `required`, `optional`, or `none` | JSON policy, mirror | `required` forces a policy snapshot id when creating runs. |
+| `distribution` | Enabled distribution drivers (`http`, `oci`, `object`) | All | Offline installs typically disable `oci`. |
+| `compression` | Compression settings (`zstd`, level) | JSON, mirror | Trivy adapters manage compression internally. |
+| `encryption` | Mirror encryption options (`enabled`, `recipientKeys`, `strict`) | Mirror | When enabled, only `/data` subtree is encrypted; manifests remain plaintext. |
+| `retention` | Retention policy (days or `never`) | All | Drives pruning jobs for staged bundles. |
+
+Selectors (time windows, tenants, products, SBOM subjects, ecosystems) are supplied per run, not stored in the profile.
+
+## JSON profiles
+
+### `json:raw`
+- **Content:** Exports raw advisories, VEX statements, and SBOMs as newline-delimited JSON (`.jsonl.zst`).
+- **Schema:** Follows canonical StellaOps schema with casing and timestamps normalised. Each record includes `tenant`, `source`, `linkset`, and `content` fields.
+- **Options:**
+  - `include` defaults to `["advisories", "vex", "sboms"]`.
+  - `compression` defaults to `zstd` level 9.
+  - `maxRecordsPerFile` (optional) splits outputs for large datasets.
+- **Compatibility:** Intended for analytics platforms, data escrow, or feeding downstream normalisation pipelines.
+- **Sample manifest excerpt:**
+
+```json
+{
+  "profile": { "kind": "json", "variant": "raw" },
+  "outputs": [
+    { "type": "advisories.jsonl.zst", "sha256": "...", "count": 15234 },
+    { "type": "vex.jsonl.zst", "sha256": "...", "count": 3045 },
+    { "type": "sboms.jsonl.zst", "sha256": "...", "count": 872 }
+  ],
+  "selectors": { "tenant": "acme", "products": ["registry.example/app"] }
+}
+```
+
+### `json:policy`
+- **Content:** Everything from `json:raw` plus:
+  - `policy_snapshot.json` (policy metadata, version, hash).
+  - `findings.policy.jsonl.zst` (evaluated findings with decision, rationale, rule id, inputs hash).
+- **Determinism:** Requires a policy snapshot id; runs fail if snapshot is missing or non-deterministic mode is active.
+- **Use cases:** Compliance exports, auditor packages, policy attestation archives.
+- **Guardrails:** AOC boundaries preserved: policy outputs are clearly partitioned from raw evidence.
+
+## Trivy profiles
+
+### `trivy:db`
+- Detailed adapter behaviour is documented in `docs/modules/export-center/trivy-adapter.md`.
+- **Content:** Produces a Trivy DB-compatible bundle (SQLite database or tarball as required by Trivy version).
+- **Mapping rules:**
+  - Advisory namespaces mapped to Trivy vendor IDs (e.g., `ubuntu`, `debian`, `npm`).
+  - Version ranges translated into Trivy's semantic version syntax.
+  - Severity mapped to Trivy standard (`UNKNOWN`, `LOW`, `MEDIUM`, `HIGH`, `CRITICAL`).
+- **Validation:** Adapter enforces supported Trivy schema versions; configuration includes `targetSchemaVersion`.
+- **Distribution:** Typically pushed to OCI or object storage for downstream scanners; Console download remains available.
+
+### `trivy:java-db`
+- Refer to `docs/modules/export-center/trivy-adapter.md` for ecosystem-specific notes.
+- **Content:** Optional Java ecosystem supplement for Trivy (matching Trivy's separate Java DB).
+- **Dependencies:** Requires Java advisories in Findings Ledger; run fails with `ERR_EXPORT_EMPTY` if no Java data present and `allowEmpty=false`.
+- **Compatibility:** Intended for organisations using Trivy's Java plugin or hardened pipelines that split general and Java feeds.
+
+## Mirror profiles
+
+### `mirror:full`
+- Bundle structure and delta strategy are covered in `docs/modules/export-center/mirror-bundles.md`.
+- **Content:** Complete export with:
+  - Raw advisories, VEX, SBOMs (`/data/raw`).
+  - Policy overlays (`/data/policy`), including evaluated findings and policy snapshots.
+  - Indexes for fast lookup (`/indexes/advisories.pb`, `/indexes/sboms.pb`).
+  - Manifests and provenance (`/manifests/export.json`, `/manifests/provenance.json`).
+- **Layout:** Deterministic directory structure with hashed filenames to reduce duplication.
+- **Encryption:** Optional `encryption` block enables age/AES-GCM encryption of `/data`. `strict=true` encrypts everything except `export.json`.
+- **Use cases:** Air-gap replication, disaster recovery drills, Offline Kit seeding.
+
+### `mirror:delta`
+- See `docs/modules/export-center/mirror-bundles.md` for delta mechanics and application order.
+- **Content:** Includes only changes relative to a base manifest (specified by `baseExportId` when running).
+  - `changed`, `added`, `removed` lists in `manifests/delta.json`.
+  - Incremental indexes capturing only updated subjects.
+- **Constraints:** Requires the base manifest to exist in object storage or artifact registry accessible to the worker. Fails with `ERR_EXPORT_BASE_MISSING` otherwise.
+- **Workflow:** Ideal for frequent updates to mirrored environments with limited bandwidth.
+
+## Compatibility and guardrails
+- **Aggregation-Only Contract:** All profiles respect AOC boundaries: raw evidence is never mutated. Policy outputs are appended separately with clear provenance.
+- **Tenant scoping:** Profiles are tenant-specific. Cross-tenant exports require explicit administrative approval and signed justification.
+- **Retriable runs:** Re-running a profile with identical selectors yields matching manifests and hashes, facilitating verify-on-download workflows.
+- **Offline operation:** JSON and mirror profiles function in offline mode without additional configuration. Trivy profiles require pre-seeded schema metadata shipped via Offline Kit.
+- **Quota integration:** Profiles can define run quotas (per tenant per day). Quota exhaustion surfaces as `429 Too Many Requests` with `X-Stella-Quota-*` hints.
+
+## Example profile definition (CLI)
+
+```jsonc
+{
+  "name": "daily-json-raw",
+  "kind": "json",
+  "variant": "raw",
+  "include": ["advisories", "vex", "sboms"],
+  "distribution": ["http", "object"],
+  "compression": { "codec": "zstd", "level": 9 },
+  "retention": { "mode": "days", "value": 14 }
+}
+```
+
+Create via `stella export profile create --file profile.json` (CLI command documented separately).
+
+## Verification workflow
+- Download bundle via Console/CLI and extract `export.json` and `provenance.json`.
+- Run `cosign verify --key <tenant-key> export.json` (for detatched signatures use `--signature export.json.sig`).
+- Validate Trivy bundles with `trivy --cache-dir <temp> --debug --db-repository <oci-ref>` or local `trivy module db import`.
+- For mirror bundles, run internal `mirror verify` script (bundled in Offline Kit) to ensure directory layout and digests match manifest.
+
+## Extending profiles
+- Use API/CLI to clone an existing profile and adjust `include`, `distribution`, or retention policies.
+- Adapter plug-ins can introduce new variants (e.g., `json:raw-lite`, `mirror:policy-only`). Plug-ins must be registered at service restart and documented under `/docs/modules/export-center/profiles.md`.
+- Any new profile must append the imposed rule line and follow determinism and guardrail requirements.
+
+> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.