203 lines
8.1 KiB
Markdown
203 lines
8.1 KiB
Markdown
# Export Center Mirror Bundles
|
|
|
|
Mirror bundles package StellaOps evidence, policy overlays, and indexes for air-gapped or bandwidth-constrained environments. They are produced by the `mirror:full` and `mirror:delta` profiles described in Epic 10 (Export Center) and implemented across Sprints 35-37 (`EXPORT-SVC-35-004`, `EXPORT-SVC-37-001`, `EXPORT-SVC-37-002`). This guide details bundle layouts, delta mechanics, encryption workflow, import procedures, and operational best practices.
|
|
|
|
> Export Center workers are being wired while this document is written. Treat the content as the target contract for adapter development and update specifics as the implementation lands.
|
|
|
|
## 1. Bundle overview
|
|
|
|
| Profile | Contents | Typical use cases | Distribution |
|
|
|---------|----------|-------------------|--------------|
|
|
| `mirror:full` | Complete snapshot of raw evidence, normalized records, indexes, policy snapshots, provenance, signatures. | Initial seeding of an air-gapped mirror, disaster recovery drills. | Download bundle, optional OCI artifact. |
|
|
| `mirror:delta` | Changes since a specified base export: added/updated/removed advisories, VEX statements, SBOMs, indexes, manifests. | Incremental updates, bandwidth reduction, nightly refreshes. | Download bundle, optional OCI artifact. |
|
|
|
|
Both profiles respect AOC boundaries: raw ingestion data remains untouched, and policy outputs live under their own directory with explicit provenance.
|
|
|
|
## 2. Filesystem layout
|
|
|
|
Directory structure inside the extracted bundle:
|
|
|
|
```
|
|
mirror/
|
|
manifest.yaml
|
|
export.json
|
|
provenance.json
|
|
README.md
|
|
indexes/
|
|
advisories.index.json
|
|
vex.index.json
|
|
sbom.index.json
|
|
findings.index.json
|
|
data/
|
|
raw/
|
|
advisories/*.jsonl.zst
|
|
vex/*.jsonl.zst
|
|
sboms/<subject>/sbom.json
|
|
normalized/
|
|
advisories/*.jsonl.zst
|
|
vex/*.jsonl.zst
|
|
policy/
|
|
snapshot.json
|
|
evaluations.jsonl.zst
|
|
consensus/
|
|
vex_consensus.jsonl.zst
|
|
signatures/
|
|
export.sig
|
|
manifest.sig
|
|
```
|
|
|
|
`manifest.yaml` summarises profile metadata, selectors, counts, sizes, and SHA-256 digests. `export.json` and `provenance.json` mirror the JSON profile manifests and are signed using the configured KMS key.
|
|
|
|
Example `manifest.yaml`:
|
|
|
|
```yaml
|
|
profile: mirror:full
|
|
runId: run-20251029-01
|
|
tenant: acme
|
|
selectors:
|
|
products:
|
|
- registry.example.com/app:*
|
|
timeWindow:
|
|
from: 2025-10-01T00:00:00Z
|
|
to: 2025-10-29T00:00:00Z
|
|
counts:
|
|
advisories: 15234
|
|
vex: 3045
|
|
sboms: 872
|
|
policyEvaluations: 19876
|
|
artifacts:
|
|
- path: data/raw/advisories/a0.jsonl.zst
|
|
sha256: 9f4b...
|
|
bytes: 7340021
|
|
encryption:
|
|
mode: age
|
|
strict: false
|
|
recipients:
|
|
- age1tenantkey...
|
|
```
|
|
|
|
## 3. Delta mechanics
|
|
|
|
Delta bundles reference a previous full or delta run via `baseExportId` and `baseManifestDigest`. They contain:
|
|
|
|
```
|
|
delta/
|
|
changed/
|
|
data/raw/advisories/*.jsonl.zst
|
|
...
|
|
removed/
|
|
advisories.jsonl # list of advisory IDs removed
|
|
vex.jsonl
|
|
sboms.jsonl
|
|
manifest.diff.json # summary of counts, hashes, base export metadata
|
|
```
|
|
|
|
- **Base lookup:** The worker verifies that the base export is reachable (download path or OCI reference). If missing, the run fails with `ERR_EXPORT_BASE_MISSING`.
|
|
- **Change detection:** Uses deterministic hashing of normalized records to compute additions/updates. Indexes are regenerated only for affected subjects.
|
|
- **Application order:** Consumers apply deltas sequentially. A `resetBaseline=true` flag instructs them to drop cached state and apply the bundle as a full refresh.
|
|
|
|
Example `manifest.diff.json` (delta):
|
|
|
|
```json
|
|
{
|
|
"baseExportId": "run-20251025-01",
|
|
"baseManifestDigest": "sha256:aa11...",
|
|
"resetBaseline": false,
|
|
"added": {
|
|
"advisories": 43,
|
|
"vex": 12,
|
|
"sboms": 5
|
|
},
|
|
"changed": {
|
|
"advisories": 18,
|
|
"vex": 7
|
|
},
|
|
"removed": {
|
|
"advisories": 2,
|
|
"vex": 0,
|
|
"sboms": 0
|
|
}
|
|
}
|
|
```
|
|
|
|
## 4. Encryption workflow
|
|
|
|
Mirror bundles support optional encryption of the `data/` subtree:
|
|
|
|
- **Algorithm:** Age (X25519) or AES-GCM (256-bit) based on profile configuration.
|
|
- **Key wrapping:** Keys fetched from Authority-managed KMS through Export Center. Wrapped data keys stored in `provenance.json` under `encryption.recipients[]`.
|
|
- **Metadata:** `manifest.yaml` records `encryption.mode`, `recipients`, and `encryptedPaths`.
|
|
- **Strict mode:** `strict=true` encrypts everything except `manifest.yaml` and `export.json`. Default (`false`) leaves manifests unencrypted to simplify discovery.
|
|
- **Verification:** CLI (`stella export verify`) and Offline Kit scripts perform signature checks prior to decryption.
|
|
|
|
Operators must distribute recipient keys out of band. Export Center does not transmit private keys.
|
|
|
|
## 5. Import workflow
|
|
|
|
### 5.1 Offline Kit
|
|
|
|
Offline Kit bundles reference the latest full mirror export plus the last `N` deltas. Administrators run:
|
|
|
|
```
|
|
./offline-kit/bin/mirror import /path/to/mirror-20251029-full.tar.zst
|
|
./offline-kit/bin/mirror import /path/to/mirror-20251030-delta.tar.zst
|
|
```
|
|
|
|
The tool verifies signatures, applies deltas, and updates the mirror index served by the local gateway.
|
|
|
|
### 5.2 Custom automation
|
|
|
|
1. Download bundle (`stella export download`) and verify signatures (`stella export verify`).
|
|
2. Extract archive into a staging directory.
|
|
3. For encrypted bundles, decrypt using the provided age/AES key.
|
|
4. Sync `mirror/data` onto the target mirror store (object storage, NFS, etc.).
|
|
5. Republish indexes or reload services that depend on the mirror.
|
|
|
|
Delta consumers must track `appliedExportIds` to ensure ordering.
|
|
|
|
Sequence diagram of download/import:
|
|
|
|
```mermaid
|
|
sequenceDiagram
|
|
participant CLI as stella CLI
|
|
participant Mirror as Mirror Store
|
|
participant Verify as Verification Tool
|
|
CLI->>CLI: stella export download run-20251029-01
|
|
CLI->>Verify: stella export verify run-20251029-01
|
|
CLI->>Mirror: mirror import mirror-20251029-full.tar.zst
|
|
CLI->>Mirror: mirror import mirror-20251030-delta.tar.zst
|
|
Mirror-->>CLI: import complete (run-20251030-02)
|
|
```
|
|
|
|
## 6. Operational guidance
|
|
|
|
- **Retention:** Keep at least one full bundle plus the deltas required for disaster recovery. Configure `ExportCenter:Retention:Mirror` to prune older bundles automatically.
|
|
- **Storage footprint:** Full bundles can exceed tens of gigabytes. Plan object storage or NAS capacity accordingly and enable compression (`compression.codec=zstd`).
|
|
- **Scheduling:** For high-churn environments, run daily full exports and hourly deltas. Record cadence in `manifest.yaml` (`schedule.cron`).
|
|
- **Incident recovery:** To rebuild a mirror:
|
|
1. Apply the most recent full bundle.
|
|
2. Apply deltas in order of `createdAt`.
|
|
3. Re-run integrity checks (`mirror verify <path>`).
|
|
- **Audit logging:** Export Center logs `mirror.bundle.created`, `mirror.delta.applied`, and `mirror.encryption.enabled` events. Consume them in the central observability pipeline.
|
|
|
|
## 7. Troubleshooting
|
|
|
|
| Symptom | Meaning | Action |
|
|
|---------|---------|--------|
|
|
| `ERR_EXPORT_BASE_MISSING` | Base export not available | Republish base bundle or rebuild as full export. |
|
|
| Delta applies but mirror misses entries | Deltas applied out of order | Rebuild from last full bundle and reapply in sequence. |
|
|
| Decryption fails | Recipient key mismatch or corrupted bundle | Confirm key distribution and re-download bundle. |
|
|
| Verification errors | Signature mismatch | Do not import; regenerate bundle and investigate signing pipeline. |
|
|
| Manifest hash mismatch | Files changed after extraction | Re-extract bundle and re-run verification; check storage tampering. |
|
|
|
|
## 8. References
|
|
|
|
- [Export Center Overview](overview.md)
|
|
- [Export Center Architecture](architecture.md)
|
|
- [Export Center API reference](api.md)
|
|
- [Export Center CLI Guide](cli.md)
|
|
- [Concelier mirror runbook](../ops/concelier-mirror-operations.md)
|
|
- [Aggregation-Only Contract reference](../ingestion/aggregation-only-contract.md)
|
|
|
|
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|