This commit is contained in:
202
docs/export-center/mirror-bundles.md
Normal file
202
docs/export-center/mirror-bundles.md
Normal file
@@ -0,0 +1,202 @@
|
||||
# Export Center Mirror Bundles
|
||||
|
||||
Mirror bundles package StellaOps evidence, policy overlays, and indexes for air-gapped or bandwidth-constrained environments. They are produced by the `mirror:full` and `mirror:delta` profiles described in Epic 10 (Export Center) and implemented across Sprints 35-37 (`EXPORT-SVC-35-004`, `EXPORT-SVC-37-001`, `EXPORT-SVC-37-002`). This guide details bundle layouts, delta mechanics, encryption workflow, import procedures, and operational best practices.
|
||||
|
||||
> Export Center workers are being wired while this document is written. Treat the content as the target contract for adapter development and update specifics as the implementation lands.
|
||||
|
||||
## 1. Bundle overview
|
||||
|
||||
| Profile | Contents | Typical use cases | Distribution |
|
||||
|---------|----------|-------------------|--------------|
|
||||
| `mirror:full` | Complete snapshot of raw evidence, normalized records, indexes, policy snapshots, provenance, signatures. | Initial seeding of an air-gapped mirror, disaster recovery drills. | Download bundle, optional OCI artifact. |
|
||||
| `mirror:delta` | Changes since a specified base export: added/updated/removed advisories, VEX statements, SBOMs, indexes, manifests. | Incremental updates, bandwidth reduction, nightly refreshes. | Download bundle, optional OCI artifact. |
|
||||
|
||||
Both profiles respect AOC boundaries: raw ingestion data remains untouched, and policy outputs live under their own directory with explicit provenance.
|
||||
|
||||
## 2. Filesystem layout
|
||||
|
||||
Directory structure inside the extracted bundle:
|
||||
|
||||
```
|
||||
mirror/
|
||||
manifest.yaml
|
||||
export.json
|
||||
provenance.json
|
||||
README.md
|
||||
indexes/
|
||||
advisories.index.json
|
||||
vex.index.json
|
||||
sbom.index.json
|
||||
findings.index.json
|
||||
data/
|
||||
raw/
|
||||
advisories/*.jsonl.zst
|
||||
vex/*.jsonl.zst
|
||||
sboms/<subject>/sbom.json
|
||||
normalized/
|
||||
advisories/*.jsonl.zst
|
||||
vex/*.jsonl.zst
|
||||
policy/
|
||||
snapshot.json
|
||||
evaluations.jsonl.zst
|
||||
consensus/
|
||||
vex_consensus.jsonl.zst
|
||||
signatures/
|
||||
export.sig
|
||||
manifest.sig
|
||||
```
|
||||
|
||||
`manifest.yaml` summarises profile metadata, selectors, counts, sizes, and SHA-256 digests. `export.json` and `provenance.json` mirror the JSON profile manifests and are signed using the configured KMS key.
|
||||
|
||||
Example `manifest.yaml`:
|
||||
|
||||
```yaml
|
||||
profile: mirror:full
|
||||
runId: run-20251029-01
|
||||
tenant: acme
|
||||
selectors:
|
||||
products:
|
||||
- registry.example.com/app:*
|
||||
timeWindow:
|
||||
from: 2025-10-01T00:00:00Z
|
||||
to: 2025-10-29T00:00:00Z
|
||||
counts:
|
||||
advisories: 15234
|
||||
vex: 3045
|
||||
sboms: 872
|
||||
policyEvaluations: 19876
|
||||
artifacts:
|
||||
- path: data/raw/advisories/a0.jsonl.zst
|
||||
sha256: 9f4b...
|
||||
bytes: 7340021
|
||||
encryption:
|
||||
mode: age
|
||||
strict: false
|
||||
recipients:
|
||||
- age1tenantkey...
|
||||
```
|
||||
|
||||
## 3. Delta mechanics
|
||||
|
||||
Delta bundles reference a previous full or delta run via `baseExportId` and `baseManifestDigest`. They contain:
|
||||
|
||||
```
|
||||
delta/
|
||||
changed/
|
||||
data/raw/advisories/*.jsonl.zst
|
||||
...
|
||||
removed/
|
||||
advisories.jsonl # list of advisory IDs removed
|
||||
vex.jsonl
|
||||
sboms.jsonl
|
||||
manifest.diff.json # summary of counts, hashes, base export metadata
|
||||
```
|
||||
|
||||
- **Base lookup:** The worker verifies that the base export is reachable (download path or OCI reference). If missing, the run fails with `ERR_EXPORT_BASE_MISSING`.
|
||||
- **Change detection:** Uses deterministic hashing of normalized records to compute additions/updates. Indexes are regenerated only for affected subjects.
|
||||
- **Application order:** Consumers apply deltas sequentially. A `resetBaseline=true` flag instructs them to drop cached state and apply the bundle as a full refresh.
|
||||
|
||||
Example `manifest.diff.json` (delta):
|
||||
|
||||
```json
|
||||
{
|
||||
"baseExportId": "run-20251025-01",
|
||||
"baseManifestDigest": "sha256:aa11...",
|
||||
"resetBaseline": false,
|
||||
"added": {
|
||||
"advisories": 43,
|
||||
"vex": 12,
|
||||
"sboms": 5
|
||||
},
|
||||
"changed": {
|
||||
"advisories": 18,
|
||||
"vex": 7
|
||||
},
|
||||
"removed": {
|
||||
"advisories": 2,
|
||||
"vex": 0,
|
||||
"sboms": 0
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## 4. Encryption workflow
|
||||
|
||||
Mirror bundles support optional encryption of the `data/` subtree:
|
||||
|
||||
- **Algorithm:** Age (X25519) or AES-GCM (256-bit) based on profile configuration.
|
||||
- **Key wrapping:** Keys fetched from Authority-managed KMS through Export Center. Wrapped data keys stored in `provenance.json` under `encryption.recipients[]`.
|
||||
- **Metadata:** `manifest.yaml` records `encryption.mode`, `recipients`, and `encryptedPaths`.
|
||||
- **Strict mode:** `strict=true` encrypts everything except `manifest.yaml` and `export.json`. Default (`false`) leaves manifests unencrypted to simplify discovery.
|
||||
- **Verification:** CLI (`stella export verify`) and Offline Kit scripts perform signature checks prior to decryption.
|
||||
|
||||
Operators must distribute recipient keys out of band. Export Center does not transmit private keys.
|
||||
|
||||
## 5. Import workflow
|
||||
|
||||
### 5.1 Offline Kit
|
||||
|
||||
Offline Kit bundles reference the latest full mirror export plus the last `N` deltas. Administrators run:
|
||||
|
||||
```
|
||||
./offline-kit/bin/mirror import /path/to/mirror-20251029-full.tar.zst
|
||||
./offline-kit/bin/mirror import /path/to/mirror-20251030-delta.tar.zst
|
||||
```
|
||||
|
||||
The tool verifies signatures, applies deltas, and updates the mirror index served by the local gateway.
|
||||
|
||||
### 5.2 Custom automation
|
||||
|
||||
1. Download bundle (`stella export download`) and verify signatures (`stella export verify`).
|
||||
2. Extract archive into a staging directory.
|
||||
3. For encrypted bundles, decrypt using the provided age/AES key.
|
||||
4. Sync `mirror/data` onto the target mirror store (object storage, NFS, etc.).
|
||||
5. Republish indexes or reload services that depend on the mirror.
|
||||
|
||||
Delta consumers must track `appliedExportIds` to ensure ordering.
|
||||
|
||||
Sequence diagram of download/import:
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant CLI as stella CLI
|
||||
participant Mirror as Mirror Store
|
||||
participant Verify as Verification Tool
|
||||
CLI->>CLI: stella export download run-20251029-01
|
||||
CLI->>Verify: stella export verify run-20251029-01
|
||||
CLI->>Mirror: mirror import mirror-20251029-full.tar.zst
|
||||
CLI->>Mirror: mirror import mirror-20251030-delta.tar.zst
|
||||
Mirror-->>CLI: import complete (run-20251030-02)
|
||||
```
|
||||
|
||||
## 6. Operational guidance
|
||||
|
||||
- **Retention:** Keep at least one full bundle plus the deltas required for disaster recovery. Configure `ExportCenter:Retention:Mirror` to prune older bundles automatically.
|
||||
- **Storage footprint:** Full bundles can exceed tens of gigabytes. Plan object storage or NAS capacity accordingly and enable compression (`compression.codec=zstd`).
|
||||
- **Scheduling:** For high-churn environments, run daily full exports and hourly deltas. Record cadence in `manifest.yaml` (`schedule.cron`).
|
||||
- **Incident recovery:** To rebuild a mirror:
|
||||
1. Apply the most recent full bundle.
|
||||
2. Apply deltas in order of `createdAt`.
|
||||
3. Re-run integrity checks (`mirror verify <path>`).
|
||||
- **Audit logging:** Export Center logs `mirror.bundle.created`, `mirror.delta.applied`, and `mirror.encryption.enabled` events. Consume them in the central observability pipeline.
|
||||
|
||||
## 7. Troubleshooting
|
||||
|
||||
| Symptom | Meaning | Action |
|
||||
|---------|---------|--------|
|
||||
| `ERR_EXPORT_BASE_MISSING` | Base export not available | Republish base bundle or rebuild as full export. |
|
||||
| Delta applies but mirror misses entries | Deltas applied out of order | Rebuild from last full bundle and reapply in sequence. |
|
||||
| Decryption fails | Recipient key mismatch or corrupted bundle | Confirm key distribution and re-download bundle. |
|
||||
| Verification errors | Signature mismatch | Do not import; regenerate bundle and investigate signing pipeline. |
|
||||
| Manifest hash mismatch | Files changed after extraction | Re-extract bundle and re-run verification; check storage tampering. |
|
||||
|
||||
## 8. References
|
||||
|
||||
- [Export Center Overview](overview.md)
|
||||
- [Export Center Architecture](architecture.md)
|
||||
- [Export Center API reference](api.md)
|
||||
- [Export Center CLI Guide](cli.md)
|
||||
- [Concelier mirror runbook](../ops/concelier-mirror-operations.md)
|
||||
- [Aggregation-Only Contract reference](../ingestion/aggregation-only-contract.md)
|
||||
|
||||
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
Reference in New Issue
Block a user