stella-ops.org/git.stella-ops.org

Fork 0

Files

root 68da90a11a

Docs CI / lint-and-preview (push) Has been cancelled

Details

Restructure solution layout by module

2025-10-28 15:10:40 +02:00

8.1 KiB

Raw Blame History

Export Center Mirror Bundles

Mirror bundles package StellaOps evidence, policy overlays, and indexes for air-gapped or bandwidth-constrained environments. They are produced by the mirror:full and mirror:delta profiles described in Epic 10 (Export Center) and implemented across Sprints 35-37 (EXPORT-SVC-35-004, EXPORT-SVC-37-001, EXPORT-SVC-37-002). This guide details bundle layouts, delta mechanics, encryption workflow, import procedures, and operational best practices.

Export Center workers are being wired while this document is written. Treat the content as the target contract for adapter development and update specifics as the implementation lands.

1. Bundle overview

Profile	Contents	Typical use cases	Distribution
`mirror:full`	Complete snapshot of raw evidence, normalized records, indexes, policy snapshots, provenance, signatures.	Initial seeding of an air-gapped mirror, disaster recovery drills.	Download bundle, optional OCI artifact.
`mirror:delta`	Changes since a specified base export: added/updated/removed advisories, VEX statements, SBOMs, indexes, manifests.	Incremental updates, bandwidth reduction, nightly refreshes.	Download bundle, optional OCI artifact.

Both profiles respect AOC boundaries: raw ingestion data remains untouched, and policy outputs live under their own directory with explicit provenance.

2. Filesystem layout

Directory structure inside the extracted bundle:

mirror/
  manifest.yaml
  export.json
  provenance.json
  README.md
  indexes/
    advisories.index.json
    vex.index.json
    sbom.index.json
    findings.index.json
  data/
    raw/
      advisories/*.jsonl.zst
      vex/*.jsonl.zst
      sboms/<subject>/sbom.json
    normalized/
      advisories/*.jsonl.zst
      vex/*.jsonl.zst
    policy/
      snapshot.json
      evaluations.jsonl.zst
    consensus/
      vex_consensus.jsonl.zst
  signatures/
    export.sig
    manifest.sig

manifest.yaml summarises profile metadata, selectors, counts, sizes, and SHA-256 digests. export.json and provenance.json mirror the JSON profile manifests and are signed using the configured KMS key.

Example manifest.yaml:

profile: mirror:full
runId: run-20251029-01
tenant: acme
selectors:
  products:
    - registry.example.com/app:*
  timeWindow:
    from: 2025-10-01T00:00:00Z
    to: 2025-10-29T00:00:00Z
counts:
  advisories: 15234
  vex: 3045
  sboms: 872
  policyEvaluations: 19876
artifacts:
  - path: data/raw/advisories/a0.jsonl.zst
    sha256: 9f4b...
    bytes: 7340021
encryption:
  mode: age
  strict: false
  recipients:
    - age1tenantkey...

3. Delta mechanics

Delta bundles reference a previous full or delta run via baseExportId and baseManifestDigest. They contain:

delta/
  changed/
    data/raw/advisories/*.jsonl.zst
    ...
  removed/
    advisories.jsonl        # list of advisory IDs removed
    vex.jsonl
    sboms.jsonl
  manifest.diff.json        # summary of counts, hashes, base export metadata

Base lookup: The worker verifies that the base export is reachable (download path or OCI reference). If missing, the run fails with ERR_EXPORT_BASE_MISSING.
Change detection: Uses deterministic hashing of normalized records to compute additions/updates. Indexes are regenerated only for affected subjects.
Application order: Consumers apply deltas sequentially. A resetBaseline=true flag instructs them to drop cached state and apply the bundle as a full refresh.

Example manifest.diff.json (delta):

{
  "baseExportId": "run-20251025-01",
  "baseManifestDigest": "sha256:aa11...",
  "resetBaseline": false,
  "added": {
    "advisories": 43,
    "vex": 12,
    "sboms": 5
  },
  "changed": {
    "advisories": 18,
    "vex": 7
  },
  "removed": {
    "advisories": 2,
    "vex": 0,
    "sboms": 0
  }
}

4. Encryption workflow

Mirror bundles support optional encryption of the data/ subtree:

Algorithm: Age (X25519) or AES-GCM (256-bit) based on profile configuration.
Key wrapping: Keys fetched from Authority-managed KMS through Export Center. Wrapped data keys stored in provenance.json under encryption.recipients[].
Metadata: manifest.yaml records encryption.mode, recipients, and encryptedPaths.
Strict mode: strict=true encrypts everything except manifest.yaml and export.json. Default (false) leaves manifests unencrypted to simplify discovery.
Verification: CLI (stella export verify) and Offline Kit scripts perform signature checks prior to decryption.

Operators must distribute recipient keys out of band. Export Center does not transmit private keys.

5. Import workflow

5.1 Offline Kit

Offline Kit bundles reference the latest full mirror export plus the last N deltas. Administrators run:

./offline-kit/bin/mirror import /path/to/mirror-20251029-full.tar.zst
./offline-kit/bin/mirror import /path/to/mirror-20251030-delta.tar.zst

The tool verifies signatures, applies deltas, and updates the mirror index served by the local gateway.

5.2 Custom automation

Download bundle (stella export download) and verify signatures (stella export verify).
Extract archive into a staging directory.
For encrypted bundles, decrypt using the provided age/AES key.
Sync mirror/data onto the target mirror store (object storage, NFS, etc.).
Republish indexes or reload services that depend on the mirror.

Delta consumers must track appliedExportIds to ensure ordering.

Sequence diagram of download/import:

sequenceDiagram
  participant CLI as stella CLI
  participant Mirror as Mirror Store
  participant Verify as Verification Tool
  CLI->>CLI: stella export download run-20251029-01
  CLI->>Verify: stella export verify run-20251029-01
  CLI->>Mirror: mirror import mirror-20251029-full.tar.zst
  CLI->>Mirror: mirror import mirror-20251030-delta.tar.zst
  Mirror-->>CLI: import complete (run-20251030-02)

6. Operational guidance

Retention: Keep at least one full bundle plus the deltas required for disaster recovery. Configure ExportCenter:Retention:Mirror to prune older bundles automatically.
Storage footprint: Full bundles can exceed tens of gigabytes. Plan object storage or NAS capacity accordingly and enable compression (compression.codec=zstd).
Scheduling: For high-churn environments, run daily full exports and hourly deltas. Record cadence in manifest.yaml (schedule.cron).
Incident recovery: To rebuild a mirror:
1. Apply the most recent full bundle.
2. Apply deltas in order of createdAt.
3. Re-run integrity checks (mirror verify <path>).
Audit logging: Export Center logs mirror.bundle.created, mirror.delta.applied, and mirror.encryption.enabled events. Consume them in the central observability pipeline.

7. Troubleshooting

Symptom	Meaning	Action
`ERR_EXPORT_BASE_MISSING`	Base export not available	Republish base bundle or rebuild as full export.
Delta applies but mirror misses entries	Deltas applied out of order	Rebuild from last full bundle and reapply in sequence.
Decryption fails	Recipient key mismatch or corrupted bundle	Confirm key distribution and re-download bundle.
Verification errors	Signature mismatch	Do not import; regenerate bundle and investigate signing pipeline.
Manifest hash mismatch	Files changed after extraction	Re-extract bundle and re-run verification; check storage tampering.

8. References

Imposed rule: Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.

8.1 KiB Raw Blame History