feat(docs): Add comprehensive documentation for Vexer, Vulnerability Explorer, and Zastava modules

- Introduced AGENTS.md, README.md, TASKS.md, and implementation_plan.md for Vexer, detailing mission, responsibilities, key components, and operational notes. - Established similar documentation structure for Vulnerability Explorer and Zastava modules, including their respective workflows, integrations, and observability notes. - Created risk scoring profiles documentation outlining the core workflow, factor model, governance, and deliverables. - Ensured all modules adhere to the Aggregation-Only Contract and maintain determinism and provenance in outputs.
2025-10-30 00:09:39 +02:00
parent 3154c67978
commit 7b5bdcf4d3
503 changed files with 16136 additions and 54638 deletions
--- a/docs/modules/export-center/mirror-bundles.md
+++ b/docs/modules/export-center/mirror-bundles.md
@@ -0,0 +1,202 @@
+# Export Center Mirror Bundles
+
+Mirror bundles package StellaOps evidence, policy overlays, and indexes for air-gapped or bandwidth-constrained environments. They are produced by the `mirror:full` and `mirror:delta` profiles described in Epic 10 (Export Center) and implemented across Sprints 35-37 (`EXPORT-SVC-35-004`, `EXPORT-SVC-37-001`, `EXPORT-SVC-37-002`). This guide details bundle layouts, delta mechanics, encryption workflow, import procedures, and operational best practices.
+
+> Export Center workers are being wired while this document is written. Treat the content as the target contract for adapter development and update specifics as the implementation lands.
+
+## 1. Bundle overview
+
+| Profile | Contents | Typical use cases | Distribution |
+|---------|----------|-------------------|--------------|
+| `mirror:full` | Complete snapshot of raw evidence, normalized records, indexes, policy snapshots, provenance, signatures. | Initial seeding of an air-gapped mirror, disaster recovery drills. | Download bundle, optional OCI artifact. |
+| `mirror:delta` | Changes since a specified base export: added/updated/removed advisories, VEX statements, SBOMs, indexes, manifests. | Incremental updates, bandwidth reduction, nightly refreshes. | Download bundle, optional OCI artifact. |
+
+Both profiles respect AOC boundaries: raw ingestion data remains untouched, and policy outputs live under their own directory with explicit provenance.
+
+## 2. Filesystem layout
+
+Directory structure inside the extracted bundle:
+
+```
+mirror/
+  manifest.yaml
+  export.json
+  provenance.json
+  README.md
+  indexes/
+    advisories.index.json
+    vex.index.json
+    sbom.index.json
+    findings.index.json
+  data/
+    raw/
+      advisories/*.jsonl.zst
+      vex/*.jsonl.zst
+      sboms/<subject>/sbom.json
+    normalized/
+      advisories/*.jsonl.zst
+      vex/*.jsonl.zst
+    policy/
+      snapshot.json
+      evaluations.jsonl.zst
+    consensus/
+      vex_consensus.jsonl.zst
+  signatures/
+    export.sig
+    manifest.sig
+```
+
+`manifest.yaml` summarises profile metadata, selectors, counts, sizes, and SHA-256 digests. `export.json` and `provenance.json` mirror the JSON profile manifests and are signed using the configured KMS key.
+
+Example `manifest.yaml`:
+
+```yaml
+profile: mirror:full
+runId: run-20251029-01
+tenant: acme
+selectors:
+  products:
+    - registry.example.com/app:*
+  timeWindow:
+    from: 2025-10-01T00:00:00Z
+    to: 2025-10-29T00:00:00Z
+counts:
+  advisories: 15234
+  vex: 3045
+  sboms: 872
+  policyEvaluations: 19876
+artifacts:
+  - path: data/raw/advisories/a0.jsonl.zst
+    sha256: 9f4b...
+    bytes: 7340021
+encryption:
+  mode: age
+  strict: false
+  recipients:
+    - age1tenantkey...
+```
+
+## 3. Delta mechanics
+
+Delta bundles reference a previous full or delta run via `baseExportId` and `baseManifestDigest`. They contain:
+
+```
+delta/
+  changed/
+    data/raw/advisories/*.jsonl.zst
+    ...
+  removed/
+    advisories.jsonl        # list of advisory IDs removed
+    vex.jsonl
+    sboms.jsonl
+  manifest.diff.json        # summary of counts, hashes, base export metadata
+```
+
+- **Base lookup:** The worker verifies that the base export is reachable (download path or OCI reference). If missing, the run fails with `ERR_EXPORT_BASE_MISSING`.
+- **Change detection:** Uses deterministic hashing of normalized records to compute additions/updates. Indexes are regenerated only for affected subjects.
+- **Application order:** Consumers apply deltas sequentially. A `resetBaseline=true` flag instructs them to drop cached state and apply the bundle as a full refresh.
+
+Example `manifest.diff.json` (delta):
+
+```json
+{
+  "baseExportId": "run-20251025-01",
+  "baseManifestDigest": "sha256:aa11...",
+  "resetBaseline": false,
+  "added": {
+    "advisories": 43,
+    "vex": 12,
+    "sboms": 5
+  },
+  "changed": {
+    "advisories": 18,
+    "vex": 7
+  },
+  "removed": {
+    "advisories": 2,
+    "vex": 0,
+    "sboms": 0
+  }
+}
+```
+
+## 4. Encryption workflow
+
+Mirror bundles support optional encryption of the `data/` subtree:
+
+- **Algorithm:** Age (X25519) or AES-GCM (256-bit) based on profile configuration.
+- **Key wrapping:** Keys fetched from Authority-managed KMS through Export Center. Wrapped data keys stored in `provenance.json` under `encryption.recipients[]`.
+- **Metadata:** `manifest.yaml` records `encryption.mode`, `recipients`, and `encryptedPaths`.
+- **Strict mode:** `strict=true` encrypts everything except `manifest.yaml` and `export.json`. Default (`false`) leaves manifests unencrypted to simplify discovery.
+- **Verification:** CLI (`stella export verify`) and Offline Kit scripts perform signature checks prior to decryption.
+
+Operators must distribute recipient keys out of band. Export Center does not transmit private keys.
+
+## 5. Import workflow
+
+### 5.1 Offline Kit
+
+Offline Kit bundles reference the latest full mirror export plus the last `N` deltas. Administrators run:
+
+```
+./offline-kit/bin/mirror import /path/to/mirror-20251029-full.tar.zst
+./offline-kit/bin/mirror import /path/to/mirror-20251030-delta.tar.zst
+```
+
+The tool verifies signatures, applies deltas, and updates the mirror index served by the local gateway.
+
+### 5.2 Custom automation
+
+1. Download bundle (`stella export download`) and verify signatures (`stella export verify`).
+2. Extract archive into a staging directory.
+3. For encrypted bundles, decrypt using the provided age/AES key.
+4. Sync `mirror/data` onto the target mirror store (object storage, NFS, etc.).
+5. Republish indexes or reload services that depend on the mirror.
+
+Delta consumers must track `appliedExportIds` to ensure ordering.
+
+Sequence diagram of download/import:
+
+```mermaid
+sequenceDiagram
+  participant CLI as stella CLI
+  participant Mirror as Mirror Store
+  participant Verify as Verification Tool
+  CLI->>CLI: stella export download run-20251029-01
+  CLI->>Verify: stella export verify run-20251029-01
+  CLI->>Mirror: mirror import mirror-20251029-full.tar.zst
+  CLI->>Mirror: mirror import mirror-20251030-delta.tar.zst
+  Mirror-->>CLI: import complete (run-20251030-02)
+```
+
+## 6. Operational guidance
+
+- **Retention:** Keep at least one full bundle plus the deltas required for disaster recovery. Configure `ExportCenter:Retention:Mirror` to prune older bundles automatically.
+- **Storage footprint:** Full bundles can exceed tens of gigabytes. Plan object storage or NAS capacity accordingly and enable compression (`compression.codec=zstd`).
+- **Scheduling:** For high-churn environments, run daily full exports and hourly deltas. Record cadence in `manifest.yaml` (`schedule.cron`).
+- **Incident recovery:** To rebuild a mirror:
+  1. Apply the most recent full bundle.
+  2. Apply deltas in order of `createdAt`.
+  3. Re-run integrity checks (`mirror verify <path>`).
+- **Audit logging:** Export Center logs `mirror.bundle.created`, `mirror.delta.applied`, and `mirror.encryption.enabled` events. Consume them in the central observability pipeline.
+
+## 7. Troubleshooting
+
+| Symptom | Meaning | Action |
+|---------|---------|--------|
+| `ERR_EXPORT_BASE_MISSING` | Base export not available | Republish base bundle or rebuild as full export. |
+| Delta applies but mirror misses entries | Deltas applied out of order | Rebuild from last full bundle and reapply in sequence. |
+| Decryption fails | Recipient key mismatch or corrupted bundle | Confirm key distribution and re-download bundle. |
+| Verification errors | Signature mismatch | Do not import; regenerate bundle and investigate signing pipeline. |
+| Manifest hash mismatch | Files changed after extraction | Re-extract bundle and re-run verification; check storage tampering. |
+
+## 8. References
+
+- [Export Center Overview](overview.md)
+- [Export Center Architecture](architecture.md)
+- [Export Center API reference](api.md)
+- [Export Center CLI Guide](cli.md)
+- [Concelier mirror runbook](../concelier/operations/mirror.md)
+- [Aggregation-Only Contract reference](../../ingestion/aggregation-only-contract.md)
+
+> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.