# Sprint 506 · Ops DevOps IV (Ops & Offline 190.B)

## Topic & Scope
- Ops & Offline focus on DevOps phase IV: incident automation, orchestrator observability, policy CI, signing/SDK pipelines, and mirror signing.
- Consolidate CI helpers, dashboards, and signing assets; document runbooks and cleanup paths.
- **Working directory:** ops/devops (and associated ops/devops/* subfolders).

## Dependencies & Concurrency
- Depends on Sprint 190.B Ops DevOps III artifacts.
- Mirror signing follow-ons depend on AIRGAP-TIME-57-001 and thin bundle v1 outputs.
- Tenant chaos/tests depend on Authority tenancy harness availability.

## Documentation Prerequisites
- docs/README.md
- docs/07_HIGH_LEVEL_ARCHITECTURE.md
- docs/modules/devops/architecture.md
- ops/devops/README.md

> **BLOCKED Tasks:** Before working on BLOCKED tasks, review [BLOCKED_DEPENDENCY_TREE.md](./BLOCKED_DEPENDENCY_TREE.md) for root blockers and dependencies.

## Delivery Tracker
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
| --- | --- | --- | --- | --- | --- |
| 1 | DEVOPS-OBS-55-001 | DONE (2025-11-25) | Depends on DEVOPS-OBS-54-001 | DevOps Guild · Ops Guild | Incident mode automation: feature flag service, burn-rate trigger, retention overrides, reset job. |
| 2 | DEVOPS-ORCH-32-001 | DONE (2025-11-25) | Bootstrap orchestrator infra | DevOps Guild · Orchestrator Guild | Provision orchestrator Postgres/message bus, CI smoke deploy, dashboards, bootstrap docs. |
| 3 | DEVOPS-ORCH-33-001 | DONE (2025-11-25) | Depends on 32-001 | DevOps Guild · Observability Guild | Grafana dashboards/alerts for rate limiter, backpressure, error clustering, DLQ depth. |
| 4 | DEVOPS-ORCH-34-001 | DONE (2025-11-25) | Depends on 33-001 | DevOps Guild · Orchestrator Guild | Harden production monitoring: synthetic probes, burn-rate alerts, replay smoke, GA readiness checklist. |
| 5 | DEVOPS-POLICY-27-001 | DONE (2025-11-25) | None | DevOps Guild · DevEx/CLI Guild | Add CI stage to run `stella policy lint`. |
| 6 | DEVOPS-POLICY-27-002 | DONE (2025-11-25) | Depends on 27-001 | DevOps Guild · Policy Registry Guild | Batch simulation CI job, threshold enforcement, PR markdown summary. |
| 7 | DEVOPS-POLICY-27-003 | DONE (2025-11-25) | Depends on 27-002 | DevOps Guild · Security Guild | Manage signing keys (OIDC + cosign), rotate keys, verify attestations. |
| 8 | DEVOPS-POLICY-27-004 | DONE (2025-11-25) | Depends on 27-003 | DevOps Guild · Observability Guild | Dashboards/alerts for policy compile latency, simulation queue depth, approval latency, promotion outcomes. |
| 9 | DEVOPS-REL-17-004 | DONE (2025-11-23) | None | DevOps Guild | Release workflow uploads `out/release/debug` and fails when symbols missing. |
| 10 | DEVOPS-RULES-33-001 | DONE (2025-11-25) | None | DevOps Guild · Platform Leads | Contracts & Rules anchor (gateway proxies, AOC no-merge, graph platform consolidation). |
| 11 | DEVOPS-SDK-63-001 | DONE (2025-11-25) | None | DevOps Guild · SDK Release Guild | Provision registry creds, signing keys, secure storage for SDK publishing pipelines. |
| 12 | DEVOPS-SIG-26-001 | DONE (2025-11-25) | None | DevOps Guild · Signals Guild | Provision CI/CD, Helm/Compose manifests for Signals service with artifact storage + Redis. |
| 13 | DEVOPS-SIG-26-002 | DONE (2025-11-25) | Depends on 26-001 | DevOps Guild · Observability Guild | Dashboards/alerts for reachability scoring latency, cache hit rates, sensor staleness. |
| 14 | DEVOPS-TEN-47-001 | BLOCKED (2025-11-25) | Needs Authority tenancy harness | DevOps Guild | JWKS cache monitoring, signature verification regression tests, token expiration chaos tests in CI. |
| 15 | DEVOPS-TEN-48-001 | BLOCKED (2025-11-25) | Depends on 47-001 | DevOps Guild | Integration tests for RLS enforcement, tenant-prefixed object storage, audit events; lint to prevent raw SQL bypass. |
| 16 | DEVOPS-CI-110-001 | DONE (2025-11-25) | None | DevOps Guild · Concelier Guild · Excititor Guild | CI helper + TRX slices at `ops/devops/ci-110-runner/`; warm restore + health smokes. |
| 17 | MIRROR-CRT-56-CI-001 | DONE (2025-11-25) | None | Mirror Creator Guild · DevOps Guild | Move `make-thin-v1.sh` into CI assembler, enforce DSSE/TUF/time-anchor, publish milestone hashes. |
| 18 | MIRROR-CRT-56-002 | DONE (2025-11-25) | Depends on 56-CI-001 | Mirror Creator Guild · Security Guild | Release signing for thin bundle v1 using `MIRROR_SIGN_KEY_B64`; run `.gitea/workflows/mirror-sign.yml`. |
| 19 | MIRROR-CRT-57-001/002 | BLOCKED | Wait on 56-002 + AIRGAP-TIME-57-001 | Mirror Creator Guild · AirGap Time Guild | OCI/time-anchor signing follow-ons. |
| 20 | MIRROR-CRT-58-001/002 | BLOCKED | Depends on 56-002 | Mirror Creator · CLI · Exporter Guilds | CLI/Export signing follow-ons. |
| 21 | EXPORT-OBS-51-001 / 54-001 / AIRGAP-TIME-57-001 / CLI-AIRGAP-56-001 / PROV-OBS-53-001 | BLOCKED | Need signed thin bundle + time anchors | Exporter · AirGap Time · CLI Guild | Export/airgap provenance chain work. |
| 22 | DEVOPS-LEDGER-29-009-REL | BLOCKED (2025-11-25) | Needs LEDGER-29-009 dev outputs | DevOps Guild · Findings Ledger Guild | Release/offline-kit packaging for ledger manifests/backups. |
| 23 | DEVOPS-LEDGER-TEN-48-001-REL | BLOCKED (2025-11-25) | Needs ledger tenant partition work | DevOps Guild · Findings Ledger Guild | Apply RLS/partition migrations in release pipelines; publish manifests/offline-kit artefacts. |
| 24 | DEVOPS-SCANNER-JAVA-21-011-REL | BLOCKED (2025-11-25) | Needs SCANNER-ANALYZERS-JAVA-21-011 outputs | DevOps Guild · Java Analyzer Guild | Package/sign Java analyzer plug-in for release/offline kits. |

## Execution Log
| Date (UTC) | Update | Owner |
| --- | --- | --- |
| 2025-12-03 | Normalised sprint file to standard template; preserved all tasks/logs; no status changes. | Planning |
| 2025-11-25 | DEVOPS-CI-110-001 runner published at `ops/devops/ci-110-runner/`; initial TRX slices stored under `ops/devops/artifacts/ci-110/20251125T030557Z/`. | DevOps |
| 2025-11-25 | MIRROR-CRT-56-CI-001 completed: CI signing script emits milestone hash summary, enforces DSSE/TUF/time-anchor steps, uploads `milestone.json` via `mirror-sign.yml`. | DevOps |
| 2025-11-25 | DEVOPS-OBS-55-001 completed: added incident-mode automation script (`scripts/observability/incident-mode.sh`) and runbook (`ops/devops/observability/incident-mode.md`). | DevOps |
| 2025-11-25 | DEVOPS-ORCH-32-001 completed: orchestrator infra compose stack, smoke script, alerts, Grafana dashboard, bootstrap README under `ops/devops/orchestrator/`. | DevOps |
| 2025-11-25 | DEVOPS-ORCH-33-001 completed: DLQ/backpressure/error panels + alerts (`ops/devops/orchestrator/alerts.yaml`); dashboard `ops/devops/orchestrator/grafana/orchestrator-overview.json`. | DevOps |
| 2025-11-25 | DEVOPS-POLICY-27-003 completed: cosign key rotation/signing/attestation scripts added; CI attestation verification stage wired into `.gitea/workflows/policy-simulate.yml`; runbook `ops/devops/policy-signing.md`. | DevOps |
| 2025-11-25 | DEVOPS-POLICY-27-004 completed: policy pipeline alerts/dashboard/playbook added. | DevOps |
| 2025-11-25 | DEVOPS-POLICY-27-001 completed: `policy-lint` workflow added; caches nugets; publishes lint artifacts. | DevOps |
| 2025-11-25 | DEVOPS-POLICY-27-002 completed: batch simulation harness + CI workflow enforcing violation thresholds, uploads summaries. | DevOps |
| 2025-11-25 | DEVOPS-ORCH-34-001 completed: synthetic probe, replay smoke wrapper, burn-rate alert, README, incident/GA readiness playbook. | DevOps |
| 2025-11-25 | MIRROR-CRT-56-002 completed: mirror-sign workflow enforces prod signing for thin bundle v1 (`REQUIRE_PROD_SIGNING=1`). | DevOps |
| 2025-11-25 | DEVOPS-SDK-63-001 completed: SDK signing/publishing toolchain, secrets guidance, CI workflow, offline/local feed config. | DevOps |
| 2025-11-25 | DEVOPS-TEN-47-001 marked BLOCKED: requires Authority tenancy harness and tenant fixture. | DevOps |
| 2025-11-25 | DEVOPS-TEN-48-001 marked BLOCKED: RLS/object-store/audit tests depend on TEN-47 harness. | DevOps |
| 2025-11-25 | DEVOPS-LEDGER-29-009-REL marked BLOCKED: waiting on LEDGER-29-009 dev outputs. | DevOps |
| 2025-11-25 | DEVOPS-LEDGER-TEN-48-001-REL marked BLOCKED: RLS migrations/artefacts depend on ledger tenant partition work. | DevOps |
| 2025-11-25 | DEVOPS-SCANNER-JAVA-21-011-REL marked BLOCKED: Java analyzer plugin artefacts unavailable. | DevOps |
| 2025-11-25 | Work paused: `No space left on device`; added cleanup helper `scripts/devops/cleanup-workspace.sh` and doc `ops/devops/README-space.md`. | DevOps |
| 2025-11-25 | DEVOPS-SIG-26-001 completed: Signals Dockerfile/compose, Helm values, CI workflow, image export helper with Mongo/Redis deps. | DevOps |
| 2025-11-25 | DEVOPS-SIG-26-002 completed: Signals alerts, dashboard, playbook for latency/cache/staleness. | DevOps |
| 2025-11-23 | DEVOPS-REL-17-004 completed: release workflow uploads debug artefacts and fails on missing symbols. | DevOps |
| 2025-11-08 | Archived completed/historic work to `docs/implplan/archived/tasks.md` (updated 2025-11-08). | Planning |

## Decisions & Risks
- Hardened Docker/CI artefacts rely on available disk; keep cleanup script in runner docs.
- Cosign key management supports keyless; offline/air-gap paths require mirrored registry + secrets provided to `sbom_attest.sh`.
- Tenant chaos drill requires iptables/root; run only on isolated agents; monitor JWKS cache TTL to avoid auth outages.
- Surface.Env: ZASTAVA_* fallback to SCANNER_* in Helm/Compose; keep docs aligned if prefixes/fields change.
- Surface.Secrets: provisioning playbook published; ensure Helm/Compose env stays in sync; offline kit bundles encrypted secrets—unpack path must match `*_SURFACE_SECRETS_ROOT`.

## Next Checkpoints
- Unblock DEVOPS-TEN-47-001/48-001 by landing Authority tenancy harness and tenant fixtures.
- Deliver AIRGAP-TIME-57-001 to unblock mirror signing follow-ons (MIRROR-CRT-57/58) and export provenance chain.
- Free runner disk space routinely using `scripts/devops/cleanup-workspace.sh` and docker prune to keep CI green.