stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 1 Releases Wiki Activity

Restructure solution layout by module

Browse Source
This commit is contained in:
master
2025-10-28 15:10:40 +02:00
parent 95daa159c4
commit d870da18ce
4103 changed files with 192899 additions and 187024 deletions
Download Patch File Download Diff File Expand all files Collapse all files

268
docs/operations/cli-release-and-packaging.md
View File

@@ -1,134 +1,134 @@
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
# CLI Release & Packaging Runbook
This runbook describes how to build, sign, package, and distribute the StellaOps CLI with Task Pack support. It covers connected and air-gapped workflows, SBOM generation, parity gating, and distribution artifacts required by Sprint 43 (`DEVOPS-CLI-43-001`, `DEPLOY-PACKS-43-001`).
---
## 1 · Release Artifacts
| Artifact | Description | Notes |
|----------|-------------|-------|
| `stella-<version>-linux-x64.tar.gz` | Linux binary + completions | Includes man pages, localization files. |
| `stella-<version>-macos-universal.tar.gz` | macOS universal binary | Signed/notarized where applicable. |
| `stella-<version>-windows-x64.zip` | Windows binary + PowerShell modules | Code-signed. |
| `stella-cli-container:<version>` | OCI image with CLI + pack runtime | Deterministic rootfs (scratch/distroless). |
| SBOM (`.cdx.json`) | CycloneDX SBOM per artifact | Generated via `stella sbom generate` or `syft`. |
| Checksums (`SHA256SUMS`) | Aggregated digest list | Signed with cosign. |
| Provenance (`.intoto.jsonl`) | DSSE attestation (SLSA L2) | Contains build metadata. |
| Release notes | Markdown summary | Links to task packs docs, parity matrix. |
---
## 2 · Build Pipeline
1. **Source checkout** – pinned commit, reproducible environment (Docker).
2. **Dependency lock** – `dotnet restore`, `npm ci` (for CLI frontends), ensure deterministic build flags.
3. **Build binaries** – cross-platform targets with reproducible timestamps.
4. **Run tests** – unit + integration; include `stella pack` commands (plan/run/verify) in CI.
5. **Generate SBOM** – `syft packages dist/stella-linux-x64 --output cyclonedx-json`.
6. **Bundle** – compress artifacts, include completions (`bash`, `zsh`, `fish`, PowerShell).
7. **Sign** – cosign signatures for binaries, checksums, container image.
8. **Publish** – upload to `downloads.stella-ops.org`, container registry, Packs Registry (for CLI container).
9. **Parity gating** – run CLI parity matrix tests vs Console features (automation in `DEVOPS-CLI-43-001`).
CI must run in isolated environment (no network beyond allowlist). Cache dependencies for offline bundling.
---
## 3 · Versioning & Channels
- Semantic versioning (`YYYY.MM.patch`), e.g., `2025.10.0`.
- Channels:
- `edge` – nightly builds, limited support.
- `beta` – pre-release candidates.
- `stable` – production-ready, after parity gating.
- Release promotions mirror Task Pack channels; update downloads manifest (`deploy/downloads/manifest.json`).
---
## 4 · Signing & Verification
- Binaries signed with cosign (`cosign sign-blob`).
- Container image signed (`cosign sign stella-cli-container:<version>`).
- DSSE provenance includes:
- Build pipeline ID.
- Source commit and repo.
- Dependencies SBOM digest.
- Test results summary.
- Verification command for operators:
```bash
cosign verify-blob \
--certificate-identity https://ci.stella-ops.org \
--certificate-oidc-issuer https://fulcio.sigstore.dev \
--signature stella-2025.10.0-linux-x64.sig \
stella-2025.10.0-linux-x64.tar.gz
```
---
## 5 · Distribution
### 5.1 Online
- Publish artifacts to Downloads service; update manifest with digests, SBOM URLs, attestations.
- Update CLI parity docs (`docs/cli-vs-ui-parity.md`) and release notes.
- Push container image to registry with SBOM + attestations referenced as OCI referrers.
- Notify stakeholders via `#release-cli` channel and release mailing list.
### 5.2 Offline / Air-Gap
- Bundle CLI artifacts, Task Pack samples, and registry mirror:
```bash
stella pack bundle export \
--packs "sbom-remediation:1.3.0" \
--output offline/packs-bundle-2025.10.0.tgz
stella cli bundle export \
--output offline/cli-2025.10.0.tgz \
--include-container \
--include-sbom
```
- Update Offline Kit manifest with new CLI version and pack bundle entries.
- Provide import scripts (`ouk import`) for sealed sites.
---
## 6 · Parity Gating
- `stella cli parity check` compares CLI commands vs parity matrix.
- CI fails release if any required command flagged `🟥` or `🟡` with severity > threshold.
- Parity report uploaded to Downloads workspace and linked in docs.
- Manual review required for new commands (ensure `man` pages and help text localized).
---
## 7 · Localization & Documentation
- CLI includes localization bundles; ensure `i18n.txz` packaged.
- Update man pages (`man/stella-pack.1`) and HTML docs.
- Sync docs: `docs/cli/overview.md`, pack authoring guide, release notes.
- Document new flags/commands in `docs/cli/commands/pack.md` (tracked in Sprint 42 tasks).
---
## 8 · Release Checklist
- [ ] All binaries built reproducibly (CI logs archived).
- [ ] Tests + parity matrix passing.
- [ ] SBOM + provenance generated and published.
- [ ] Cosign signatures created and verified.
- [ ] Downloads manifest updated (edge/beta/stable).
- [ ] Offline bundle exported and validated.
- [ ] Release notes + documentation updates merged.
- [ ] Notifications sent (chat/email).
- [ ] Imposed rule reminder present at top of document.
---
*Last updated: 2025-10-27 (Sprint 43).*
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
# CLI Release & Packaging Runbook
This runbook describes how to build, sign, package, and distribute the StellaOps CLI with Task Pack support. It covers connected and air-gapped workflows, SBOM generation, parity gating, and distribution artifacts required by Sprint 43 (`DEVOPS-CLI-43-001`, `DEPLOY-PACKS-43-001`).
---
## 1 · Release Artifacts
| Artifact | Description | Notes |
|----------|-------------|-------|
| `stella-<version>-linux-x64.tar.gz` | Linux binary + completions | Includes man pages, localization files. |
| `stella-<version>-macos-universal.tar.gz` | macOS universal binary | Signed/notarized where applicable. |
| `stella-<version>-windows-x64.zip` | Windows binary + PowerShell modules | Code-signed. |
| `stella-cli-container:<version>` | OCI image with CLI + pack runtime | Deterministic rootfs (scratch/distroless). |
| SBOM (`.cdx.json`) | CycloneDX SBOM per artifact | Generated via `stella sbom generate` or `syft`. |
| Checksums (`SHA256SUMS`) | Aggregated digest list | Signed with cosign. |
| Provenance (`.intoto.jsonl`) | DSSE attestation (SLSA L2) | Contains build metadata. |
| Release notes | Markdown summary | Links to task packs docs, parity matrix. |
---
## 2 · Build Pipeline
1. **Source checkout** – pinned commit, reproducible environment (Docker).
2. **Dependency lock** – `dotnet restore`, `npm ci` (for CLI frontends), ensure deterministic build flags.
3. **Build binaries** – cross-platform targets with reproducible timestamps.
4. **Run tests** – unit + integration; include `stella pack` commands (plan/run/verify) in CI.
5. **Generate SBOM** – `syft packages dist/stella-linux-x64 --output cyclonedx-json`.
6. **Bundle** – compress artifacts, include completions (`bash`, `zsh`, `fish`, PowerShell).
7. **Sign** – cosign signatures for binaries, checksums, container image.
8. **Publish** – upload to `downloads.stella-ops.org`, container registry, Packs Registry (for CLI container).
9. **Parity gating** – run CLI parity matrix tests vs Console features (automation in `DEVOPS-CLI-43-001`).
CI must run in isolated environment (no network beyond allowlist). Cache dependencies for offline bundling.
---
## 3 · Versioning & Channels
- Semantic versioning (`YYYY.MM.patch`), e.g., `2025.10.0`.
- Channels:
- `edge` – nightly builds, limited support.
- `beta` – pre-release candidates.
- `stable` – production-ready, after parity gating.
- Release promotions mirror Task Pack channels; update downloads manifest (`deploy/downloads/manifest.json`).
---
## 4 · Signing & Verification
- Binaries signed with cosign (`cosign sign-blob`).
- Container image signed (`cosign sign stella-cli-container:<version>`).
- DSSE provenance includes:
- Build pipeline ID.
- Source commit and repo.
- Dependencies SBOM digest.
- Test results summary.
- Verification command for operators:
```bash
cosign verify-blob \
--certificate-identity https://ci.stella-ops.org \
--certificate-oidc-issuer https://fulcio.sigstore.dev \
--signature stella-2025.10.0-linux-x64.sig \
stella-2025.10.0-linux-x64.tar.gz
```
---
## 5 · Distribution
### 5.1 Online
- Publish artifacts to Downloads service; update manifest with digests, SBOM URLs, attestations.
- Update CLI parity docs (`docs/cli-vs-ui-parity.md`) and release notes.
- Push container image to registry with SBOM + attestations referenced as OCI referrers.
- Notify stakeholders via `#release-cli` channel and release mailing list.
### 5.2 Offline / Air-Gap
- Bundle CLI artifacts, Task Pack samples, and registry mirror:
```bash
stella pack bundle export \
--packs "sbom-remediation:1.3.0" \
--output offline/packs-bundle-2025.10.0.tgz
stella cli bundle export \
--output offline/cli-2025.10.0.tgz \
--include-container \
--include-sbom
```
- Update Offline Kit manifest with new CLI version and pack bundle entries.
- Provide import scripts (`ouk import`) for sealed sites.
---
## 6 · Parity Gating
- `stella cli parity check` compares CLI commands vs parity matrix.
- CI fails release if any required command flagged `🟥` or `🟡` with severity > threshold.
- Parity report uploaded to Downloads workspace and linked in docs.
- Manual review required for new commands (ensure `man` pages and help text localized).
---
## 7 · Localization & Documentation
- CLI includes localization bundles; ensure `i18n.txz` packaged.
- Update man pages (`man/stella-pack.1`) and HTML docs.
- Sync docs: `docs/cli/overview.md`, pack authoring guide, release notes.
- Document new flags/commands in `docs/cli/commands/pack.md` (tracked in Sprint 42 tasks).
---
## 8 · Release Checklist
- [ ] All binaries built reproducibly (CI logs archived).
- [ ] Tests + parity matrix passing.
- [ ] SBOM + provenance generated and published.
- [ ] Cosign signatures created and verified.
- [ ] Downloads manifest updated (edge/beta/stable).
- [ ] Offline bundle exported and validated.
- [ ] Release notes + documentation updates merged.
- [ ] Notifications sent (chat/email).
- [ ] Imposed rule reminder present at top of document.
---
*Last updated: 2025-10-27 (Sprint 43).*

406
docs/operations/export-runbook.md
View File

@@ -1,203 +1,203 @@
# Export Center Operations Runbook
> Export Center workers and API are landing across Sprints 35-37. This runbook captures the target operational procedures so DevOps can validate them as each milestone goes live. Update specific commands once `EXPORT-SVC-35-006`, `EXPORT-SVC-36-001..004`, and related CLI tasks ship.
## 1. Service scope
The Export Center packages StellaOps evidence and policy overlays into reproducible bundles (JSON, Trivy DB, mirror). Operations owns:
- Worker scaling, queue management, and distribution storage.
- Monitoring and alerts for run throughput, failures, and verification issues.
- Runbook execution for recovery, retention, and compliance.
- Coordination with DevOps validation (cosign + `trivy module db import` smoke tests).
Related documentation:
- `docs/export-center/overview.md`
- `docs/export-center/architecture.md`
- `docs/export-center/profiles.md`
- `docs/export-center/trivy-adapter.md`
- `docs/export-center/mirror-bundles.md`
- `docs/export-center/api.md`
- `docs/export-center/cli.md`
## 2. Contacts & tooling
| Area | Owner(s) | Escalation |
|------|----------|------------|
| Export Center service | Exporter Service Guild | `#export-center-ops`, on-call rotation |
| Distribution & CI smoke | DevOps Guild | CI channel, PagerDuty `devops-export` |
| KMS / encryption | Authority Core | `#authority-core` |
| Offline Kit dissemination | Offline Kit Guild | `#offline-kit` |
Primary tooling:
- `stella export` CLI (submit, watch, download, verify).
- Export Center API (`/api/export/*`) for automation.
- Grafana dashboards (`Export Center / Run Health`, `Export Center / Distribution`).
- Alertmanager routes (`Export.Center.Failures`, `Export.Center.Verify`).
## 3. Monitoring & SLOs
Key metrics (exposed by workers and API):
| Metric | SLO / Alert | Notes |
|--------|-------------|-------|
| `exporter_run_duration_seconds` | p95 < 300 s (full), < 120 s (delta) | Break down by profile (`profile_kind`). |
| `exporter_run_failures_total` | Alert when > 3 failures/15 min per profile | Include `error_code` label. |
| `exporter_run_bytes_total` | Track growth trends | Helps with storage planning. |
| `exporter_distribution_push_seconds` | p95 < 60 s | Covers OCI/object storage. |
| `exporter_verify_failures_total` | Alert on any non-zero | Raised when cosign/Trivy smoke tests fail. |
| `exporter_retention_pruned_total` | Should increase nightly | Confirms retention job success. |
Dashboards must include:
- Run throughput by profile.
- Failure breakdown (adapter, signing, distribution).
- Queue depth and worker concurrency (via Orchestrator metrics).
- Storage consumption (object storage buckets, local staging).
Alerts (Alertmanager):
- `ExportCenterRunFailureSpike` - `exporter_run_failures_total` increase rate > 3/15 min.
- `ExportCenterVerifyFailure` - any entry in `exporter_verify_failures_total` > 0.
- `ExportCenterWorkerLag` - queue backlog > threshold for 10 minutes.
- `ExportCenterRetentionStale` - no pruning events in 24 hours.
## 4. Routine operations
### 4.1 Daily checklist
- Review dashboard for run throughput and error classes.
- Confirm CI smoke job (cosign + `trivy module db import`) passed.
- Check storage usage against capacity thresholds.
- Verify retention job executed (look for `exporter_retention_pruned_total` increment).
- Scan logs for `adapter.trivy.unsupported_schema_version` or `mirror.delta.apply_failed`.
### 4.2 Weekly tasks
- Rotate Download/OCI API tokens if configured with short-lived credentials.
- Review upcoming profile changes (new tenants, profile updates).
- Test `stella export verify` against a recent run for each profile.
- Exercise failover of workers (scale to zero one replica, ensure others pick up).
### 4.3 Pre-release
- Ensure bundles generated for release candidates pass cosign verification.
- Capture sample manifests (`export.json`, `manifest.yaml`) for documentation archives.
- Validate Offline Kit packaging includes latest full + delta mirror bundles.
## 5. Capacity & scaling
### 5.1 Worker sizing
- Default workers handle ~2 full runs or 6 delta runs concurrently per 4 vCPU.
- Scale out when:
- Queue depth (`exporter_jobs_ready`) > 10 for 10 minutes.
- p95 durations exceed SLO for multiple runs without failures.
- Use Orchestrator quotas: ensure per-tenant concurrency (`max_active_runs`) is tuned.
### 5.2 Storage planning
- Staging storage (object store or filesystem) must hold at least:
- Latest full bundle per tenant per profile.
- Last `N` deltas (default N=5).
- Set retention policy via configuration:
```yaml
ExportCenter:
Retention:
Mirror:
Mode: days
Value: 30
Trivy:
Mode: count
Value: 10
```
- Monitor `exporter_storage_bytes_total` (if available) or use bucket metrics from storage provider.
## 6. Failure response
| Symptom | Likely cause | Immediate action | Follow-up |
|---------|--------------|------------------|-----------|
| `ERR_EXPORT_UNSUPPORTED_SCHEMA` | Trivy schema mismatch | Pin `SchemaVersion` to previous value; rerun export | Coordinate with Exporter Guild to add new mapping |
| `ERR_EXPORT_BASE_MISSING` | Base manifest unavailable | Trigger full export (`mirror:full`), notify tenant | Investigate storage retention settings |
| Run stuck in `pending` | Worker unavailable / queue paused | Check worker pods / Orchestrator status | Scale workers or fix queue |
| Signing failure (`errorCode=signing`) | KMS outage or permission change | Verify KMS health; retry run; escalate to Authority | Document incident, review key rotation schedule |
| Distribution failure (`errorCode=distribution`) | OCI/object store outage | Switch profile distribution to download-only (`distribution: ["http"]`) | Restore distribution backend, resume normal config |
| CLI verification failure in CI | New bundle did not pass cosign or Trivy import | Inspect pipeline logs; download bundle; rerun verification manually | Engage Exporter Guild if data quality issue |
| Retention job skipped | Scheduler failure or misconfiguration | Run retention job manually (`stella export retention run`) | Audit scheduler configuration |
Log locations: `exporter` service emits structured logs with `runId`, `profile`, `errorCode`. For Kubernetes deployments, check `kubectl logs deployment/export-center-worker`.
## 7. Recovery playbooks
### 7.1 Replaying a failed run
1. Identify run (`runId`) and root cause via `GET /api/export/runs/{id}`.
2. If configuration changed, clone profile and adjust settings.
3. Resubmit run (`stella export run submit` or API) with `--allow-empty` if intentionally empty.
4. Monitor SSE stream or `stella export run watch`.
5. After success, prune failed run data if necessary.
### 7.2 Restoring from previous full bundle
1. Locate last successful full bundle (`mirror:full`) and associated manifest.
2. Download and verify signatures.
3. Extract into mirror staging area.
4. Apply subsequent delta bundles in order.
5. Trigger mirror verification script (`mirror verify <path>`).
### 7.3 KMS outage response
1. Disable new export submissions temporarily (set per-tenant quota to 0).
2. Coordinate with Authority Core to restore KMS.
3. Once KMS back, run `stella export run submit --profile <id> --selectors ... --priority catch-up` for affected tenants.
## 8. Verification workflow
All bundles must pass both signature and content verification.
### 8.1 Trivy bundle validation (CI job)
```bash
cosign verify-blob \
--key tenants/acme/export-center.pub \
--signature signatures/trivy-db.sig \
trivy/db.bundle
trivy module db import trivy/db.bundle --cache-dir /tmp/trivy-cache
```
Automation: `DEVOPS-EXPORT-36-001` ensures this runs on every pipeline.
### 8.2 Mirror bundle validation
```bash
cosign verify-blob \
--key tenants/acme/export-center.pub \
--signature signatures/export.sig \
mirror/export.json
./offline-kit/bin/mirror verify mirror-20251029-full.tar.zst
```
If encryption enabled, decrypt using age or AES key before verification.
## 9. Change management
- Profile changes require change record referencing tenant impact and expected bundle size.
- Distribution configuration updates (`OCI` vs `HTTP`) must be tested in staging.
- Schema upgrades (e.g., Trivy schema v3) need coordination with DevOps, Exporter, and Docs.
- Update runbook and related docs when processes change (tie updates to `DOCS-EXPORT-37-005`).
## 10. References
- `docs/export-center/trivy-adapter.md`
- `docs/export-center/mirror-bundles.md`
- `ops/devops/TASKS.md` (`DEVOPS-EXPORT-36-001`, `DEVOPS-EXPORT-37-001`)
- `docs/ingestion/aggregation-only-contract.md`
- `docs/24_OFFLINE_KIT.md`
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
# Export Center Operations Runbook
> Export Center workers and API are landing across Sprints 35-37. This runbook captures the target operational procedures so DevOps can validate them as each milestone goes live. Update specific commands once `EXPORT-SVC-35-006`, `EXPORT-SVC-36-001..004`, and related CLI tasks ship.
## 1. Service scope
The Export Center packages StellaOps evidence and policy overlays into reproducible bundles (JSON, Trivy DB, mirror). Operations owns:
- Worker scaling, queue management, and distribution storage.
- Monitoring and alerts for run throughput, failures, and verification issues.
- Runbook execution for recovery, retention, and compliance.
- Coordination with DevOps validation (cosign + `trivy module db import` smoke tests).
Related documentation:
- `docs/export-center/overview.md`
- `docs/export-center/architecture.md`
- `docs/export-center/profiles.md`
- `docs/export-center/trivy-adapter.md`
- `docs/export-center/mirror-bundles.md`
- `docs/export-center/api.md`
- `docs/export-center/cli.md`
## 2. Contacts & tooling
| Area | Owner(s) | Escalation |
|------|----------|------------|
| Export Center service | Exporter Service Guild | `#export-center-ops`, on-call rotation |
| Distribution & CI smoke | DevOps Guild | CI channel, PagerDuty `devops-export` |
| KMS / encryption | Authority Core | `#authority-core` |
| Offline Kit dissemination | Offline Kit Guild | `#offline-kit` |
Primary tooling:
- `stella export` CLI (submit, watch, download, verify).
- Export Center API (`/api/export/*`) for automation.
- Grafana dashboards (`Export Center / Run Health`, `Export Center / Distribution`).
- Alertmanager routes (`Export.Center.Failures`, `Export.Center.Verify`).
## 3. Monitoring & SLOs
Key metrics (exposed by workers and API):
| Metric | SLO / Alert | Notes |
|--------|-------------|-------|
| `exporter_run_duration_seconds` | p95 < 300 s (full), < 120 s (delta) | Break down by profile (`profile_kind`). |
| `exporter_run_failures_total` | Alert when > 3 failures/15 min per profile | Include `error_code` label. |
| `exporter_run_bytes_total` | Track growth trends | Helps with storage planning. |
| `exporter_distribution_push_seconds` | p95 < 60 s | Covers OCI/object storage. |
| `exporter_verify_failures_total` | Alert on any non-zero | Raised when cosign/Trivy smoke tests fail. |
| `exporter_retention_pruned_total` | Should increase nightly | Confirms retention job success. |
Dashboards must include:
- Run throughput by profile.
- Failure breakdown (adapter, signing, distribution).
- Queue depth and worker concurrency (via Orchestrator metrics).
- Storage consumption (object storage buckets, local staging).
Alerts (Alertmanager):
- `ExportCenterRunFailureSpike` - `exporter_run_failures_total` increase rate > 3/15 min.
- `ExportCenterVerifyFailure` - any entry in `exporter_verify_failures_total` > 0.
- `ExportCenterWorkerLag` - queue backlog > threshold for 10 minutes.
- `ExportCenterRetentionStale` - no pruning events in 24 hours.
## 4. Routine operations
### 4.1 Daily checklist
- Review dashboard for run throughput and error classes.
- Confirm CI smoke job (cosign + `trivy module db import`) passed.
- Check storage usage against capacity thresholds.
- Verify retention job executed (look for `exporter_retention_pruned_total` increment).
- Scan logs for `adapter.trivy.unsupported_schema_version` or `mirror.delta.apply_failed`.
### 4.2 Weekly tasks
- Rotate Download/OCI API tokens if configured with short-lived credentials.
- Review upcoming profile changes (new tenants, profile updates).
- Test `stella export verify` against a recent run for each profile.
- Exercise failover of workers (scale to zero one replica, ensure others pick up).
### 4.3 Pre-release
- Ensure bundles generated for release candidates pass cosign verification.
- Capture sample manifests (`export.json`, `manifest.yaml`) for documentation archives.
- Validate Offline Kit packaging includes latest full + delta mirror bundles.
## 5. Capacity & scaling
### 5.1 Worker sizing
- Default workers handle ~2 full runs or 6 delta runs concurrently per 4 vCPU.
- Scale out when:
- Queue depth (`exporter_jobs_ready`) > 10 for 10 minutes.
- p95 durations exceed SLO for multiple runs without failures.
- Use Orchestrator quotas: ensure per-tenant concurrency (`max_active_runs`) is tuned.
### 5.2 Storage planning
- Staging storage (object store or filesystem) must hold at least:
- Latest full bundle per tenant per profile.
- Last `N` deltas (default N=5).
- Set retention policy via configuration:
```yaml
ExportCenter:
Retention:
Mirror:
Mode: days
Value: 30
Trivy:
Mode: count
Value: 10
```
- Monitor `exporter_storage_bytes_total` (if available) or use bucket metrics from storage provider.
## 6. Failure response
| Symptom | Likely cause | Immediate action | Follow-up |
|---------|--------------|------------------|-----------|
| `ERR_EXPORT_UNSUPPORTED_SCHEMA` | Trivy schema mismatch | Pin `SchemaVersion` to previous value; rerun export | Coordinate with Exporter Guild to add new mapping |
| `ERR_EXPORT_BASE_MISSING` | Base manifest unavailable | Trigger full export (`mirror:full`), notify tenant | Investigate storage retention settings |
| Run stuck in `pending` | Worker unavailable / queue paused | Check worker pods / Orchestrator status | Scale workers or fix queue |
| Signing failure (`errorCode=signing`) | KMS outage or permission change | Verify KMS health; retry run; escalate to Authority | Document incident, review key rotation schedule |
| Distribution failure (`errorCode=distribution`) | OCI/object store outage | Switch profile distribution to download-only (`distribution: ["http"]`) | Restore distribution backend, resume normal config |
| CLI verification failure in CI | New bundle did not pass cosign or Trivy import | Inspect pipeline logs; download bundle; rerun verification manually | Engage Exporter Guild if data quality issue |
| Retention job skipped | Scheduler failure or misconfiguration | Run retention job manually (`stella export retention run`) | Audit scheduler configuration |
Log locations: `exporter` service emits structured logs with `runId`, `profile`, `errorCode`. For Kubernetes deployments, check `kubectl logs deployment/export-center-worker`.
## 7. Recovery playbooks
### 7.1 Replaying a failed run
1. Identify run (`runId`) and root cause via `GET /api/export/runs/{id}`.
2. If configuration changed, clone profile and adjust settings.
3. Resubmit run (`stella export run submit` or API) with `--allow-empty` if intentionally empty.
4. Monitor SSE stream or `stella export run watch`.
5. After success, prune failed run data if necessary.
### 7.2 Restoring from previous full bundle
1. Locate last successful full bundle (`mirror:full`) and associated manifest.
2. Download and verify signatures.
3. Extract into mirror staging area.
4. Apply subsequent delta bundles in order.
5. Trigger mirror verification script (`mirror verify <path>`).
### 7.3 KMS outage response
1. Disable new export submissions temporarily (set per-tenant quota to 0).
2. Coordinate with Authority Core to restore KMS.
3. Once KMS back, run `stella export run submit --profile <id> --selectors ... --priority catch-up` for affected tenants.
## 8. Verification workflow
All bundles must pass both signature and content verification.
### 8.1 Trivy bundle validation (CI job)
```bash
cosign verify-blob \
--key tenants/acme/export-center.pub \
--signature signatures/trivy-db.sig \
trivy/db.bundle
trivy module db import trivy/db.bundle --cache-dir /tmp/trivy-cache
```
Automation: `DEVOPS-EXPORT-36-001` ensures this runs on every pipeline.
### 8.2 Mirror bundle validation
```bash
cosign verify-blob \
--key tenants/acme/export-center.pub \
--signature signatures/export.sig \
mirror/export.json
./offline-kit/bin/mirror verify mirror-20251029-full.tar.zst
```
If encryption enabled, decrypt using age or AES key before verification.
## 9. Change management
- Profile changes require change record referencing tenant impact and expected bundle size.
- Distribution configuration updates (`OCI` vs `HTTP`) must be tested in staging.
- Schema upgrades (e.g., Trivy schema v3) need coordination with DevOps, Exporter, and Docs.
- Update runbook and related docs when processes change (tie updates to `DOCS-EXPORT-37-005`).
## 10. References
- `docs/export-center/trivy-adapter.md`
- `docs/export-center/mirror-bundles.md`
- `ops/devops/TASKS.md` (`DEVOPS-EXPORT-36-001`, `DEVOPS-EXPORT-37-001`)
- `docs/ingestion/aggregation-only-contract.md`
- `docs/24_OFFLINE_KIT.md`
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.