75f6942769
Some checks failed
Docs CI / lint-and-preview (push) Has been cancelled
Policy Lint & Smoke / policy-lint (push) Has been cancelled
Concelier Attestation Tests / attestation-tests (push) Has been cancelled
AOC Guard CI / aoc-guard (push) Has been cancelled
AOC Guard CI / aoc-verify (push) Has been cancelled
- Implemented MigrationCategoryTests to validate migration categorization for startup, release, seed, and data migrations. - Added tests for edge cases, including null, empty, and whitespace migration names. - Created StartupMigrationHostTests to verify the behavior of the migration host with real PostgreSQL instances using Testcontainers. - Included tests for migration execution, schema creation, and handling of pending release migrations. - Added SQL migration files for testing: creating a test table, adding a column, a release migration, and seeding data.
9.9 KiB
9.9 KiB
Sprint 506 · Ops DevOps IV (Ops & Offline 190.B)
Topic & Scope
- Ops & Offline focus on DevOps phase IV: incident automation, orchestrator observability, policy CI, signing/SDK pipelines, and mirror signing.
- Consolidate CI helpers, dashboards, and signing assets; document runbooks and cleanup paths.
- Working directory: ops/devops (and associated ops/devops/* subfolders).
Dependencies & Concurrency
- Depends on Sprint 190.B Ops DevOps III artifacts.
- Mirror signing follow-ons depend on AIRGAP-TIME-57-001 and thin bundle v1 outputs.
- Tenant chaos/tests depend on Authority tenancy harness availability.
Documentation Prerequisites
- docs/README.md
- docs/07_HIGH_LEVEL_ARCHITECTURE.md
- docs/modules/devops/architecture.md
- ops/devops/README.md
BLOCKED Tasks: Before working on BLOCKED tasks, review BLOCKED_DEPENDENCY_TREE.md for root blockers and dependencies.
Delivery Tracker
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
|---|---|---|---|---|---|
| 1 | DEVOPS-OBS-55-001 | DONE (2025-11-25) | Depends on DEVOPS-OBS-54-001 | DevOps Guild · Ops Guild | Incident mode automation: feature flag service, burn-rate trigger, retention overrides, reset job. |
| 2 | DEVOPS-ORCH-32-001 | DONE (2025-11-25) | Bootstrap orchestrator infra | DevOps Guild · Orchestrator Guild | Provision orchestrator Postgres/message bus, CI smoke deploy, dashboards, bootstrap docs. |
| 3 | DEVOPS-ORCH-33-001 | DONE (2025-11-25) | Depends on 32-001 | DevOps Guild · Observability Guild | Grafana dashboards/alerts for rate limiter, backpressure, error clustering, DLQ depth. |
| 4 | DEVOPS-ORCH-34-001 | DONE (2025-11-25) | Depends on 33-001 | DevOps Guild · Orchestrator Guild | Harden production monitoring: synthetic probes, burn-rate alerts, replay smoke, GA readiness checklist. |
| 5 | DEVOPS-POLICY-27-001 | DONE (2025-11-25) | None | DevOps Guild · DevEx/CLI Guild | Add CI stage to run stella policy lint. |
| 6 | DEVOPS-POLICY-27-002 | DONE (2025-11-25) | Depends on 27-001 | DevOps Guild · Policy Registry Guild | Batch simulation CI job, threshold enforcement, PR markdown summary. |
| 7 | DEVOPS-POLICY-27-003 | DONE (2025-11-25) | Depends on 27-002 | DevOps Guild · Security Guild | Manage signing keys (OIDC + cosign), rotate keys, verify attestations. |
| 8 | DEVOPS-POLICY-27-004 | DONE (2025-11-25) | Depends on 27-003 | DevOps Guild · Observability Guild | Dashboards/alerts for policy compile latency, simulation queue depth, approval latency, promotion outcomes. |
| 9 | DEVOPS-REL-17-004 | DONE (2025-11-23) | None | DevOps Guild | Release workflow uploads out/release/debug and fails when symbols missing. |
| 10 | DEVOPS-RULES-33-001 | DONE (2025-11-25) | None | DevOps Guild · Platform Leads | Contracts & Rules anchor (gateway proxies, AOC no-merge, graph platform consolidation). |
| 11 | DEVOPS-SDK-63-001 | DONE (2025-11-25) | None | DevOps Guild · SDK Release Guild | Provision registry creds, signing keys, secure storage for SDK publishing pipelines. |
| 12 | DEVOPS-SIG-26-001 | DONE (2025-11-25) | None | DevOps Guild · Signals Guild | Provision CI/CD, Helm/Compose manifests for Signals service with artifact storage + Redis. |
| 13 | DEVOPS-SIG-26-002 | DONE (2025-11-25) | Depends on 26-001 | DevOps Guild · Observability Guild | Dashboards/alerts for reachability scoring latency, cache hit rates, sensor staleness. |
| 14 | DEVOPS-TEN-47-001 | BLOCKED (2025-11-25) | Needs Authority tenancy harness | DevOps Guild | JWKS cache monitoring, signature verification regression tests, token expiration chaos tests in CI. |
| 15 | DEVOPS-TEN-48-001 | BLOCKED (2025-11-25) | Depends on 47-001 | DevOps Guild | Integration tests for RLS enforcement, tenant-prefixed object storage, audit events; lint to prevent raw SQL bypass. |
| 16 | DEVOPS-CI-110-001 | DONE (2025-11-25) | None | DevOps Guild · Concelier Guild · Excititor Guild | CI helper + TRX slices at ops/devops/ci-110-runner/; warm restore + health smokes. |
| 17 | MIRROR-CRT-56-CI-001 | DONE (2025-11-25) | None | Mirror Creator Guild · DevOps Guild | Move make-thin-v1.sh into CI assembler, enforce DSSE/TUF/time-anchor, publish milestone hashes. |
| 18 | MIRROR-CRT-56-002 | DONE (2025-11-25) | Depends on 56-CI-001 | Mirror Creator Guild · Security Guild | Release signing for thin bundle v1 using MIRROR_SIGN_KEY_B64; run .gitea/workflows/mirror-sign.yml. |
| 19 | MIRROR-CRT-57-001/002 | BLOCKED | Wait on 56-002 + AIRGAP-TIME-57-001 | Mirror Creator Guild · AirGap Time Guild | OCI/time-anchor signing follow-ons. |
| 20 | MIRROR-CRT-58-001/002 | BLOCKED | Depends on 56-002 | Mirror Creator · CLI · Exporter Guilds | CLI/Export signing follow-ons. |
| 21 | EXPORT-OBS-51-001 / 54-001 / AIRGAP-TIME-57-001 / CLI-AIRGAP-56-001 / PROV-OBS-53-001 | BLOCKED | Need signed thin bundle + time anchors | Exporter · AirGap Time · CLI Guild | Export/airgap provenance chain work. |
| 22 | DEVOPS-LEDGER-29-009-REL | BLOCKED (2025-11-25) | Needs LEDGER-29-009 dev outputs | DevOps Guild · Findings Ledger Guild | Release/offline-kit packaging for ledger manifests/backups. |
| 23 | DEVOPS-LEDGER-TEN-48-001-REL | BLOCKED (2025-11-25) | Needs ledger tenant partition work | DevOps Guild · Findings Ledger Guild | Apply RLS/partition migrations in release pipelines; publish manifests/offline-kit artefacts. |
| 24 | DEVOPS-SCANNER-JAVA-21-011-REL | BLOCKED (2025-11-25) | Needs SCANNER-ANALYZERS-JAVA-21-011 outputs | DevOps Guild · Java Analyzer Guild | Package/sign Java analyzer plug-in for release/offline kits. |
Execution Log
| Date (UTC) | Update | Owner |
|---|---|---|
| 2025-12-03 | Normalised sprint file to standard template; preserved all tasks/logs; no status changes. | Planning |
| 2025-11-25 | DEVOPS-CI-110-001 runner published at ops/devops/ci-110-runner/; initial TRX slices stored under ops/devops/artifacts/ci-110/20251125T030557Z/. |
DevOps |
| 2025-11-25 | MIRROR-CRT-56-CI-001 completed: CI signing script emits milestone hash summary, enforces DSSE/TUF/time-anchor steps, uploads milestone.json via mirror-sign.yml. |
DevOps |
| 2025-11-25 | DEVOPS-OBS-55-001 completed: added incident-mode automation script (scripts/observability/incident-mode.sh) and runbook (ops/devops/observability/incident-mode.md). |
DevOps |
| 2025-11-25 | DEVOPS-ORCH-32-001 completed: orchestrator infra compose stack, smoke script, alerts, Grafana dashboard, bootstrap README under ops/devops/orchestrator/. |
DevOps |
| 2025-11-25 | DEVOPS-ORCH-33-001 completed: DLQ/backpressure/error panels + alerts (ops/devops/orchestrator/alerts.yaml); dashboard ops/devops/orchestrator/grafana/orchestrator-overview.json. |
DevOps |
| 2025-11-25 | DEVOPS-POLICY-27-003 completed: cosign key rotation/signing/attestation scripts added; CI attestation verification stage wired into .gitea/workflows/policy-simulate.yml; runbook ops/devops/policy-signing.md. |
DevOps |
| 2025-11-25 | DEVOPS-POLICY-27-004 completed: policy pipeline alerts/dashboard/playbook added. | DevOps |
| 2025-11-25 | DEVOPS-POLICY-27-001 completed: policy-lint workflow added; caches nugets; publishes lint artifacts. |
DevOps |
| 2025-11-25 | DEVOPS-POLICY-27-002 completed: batch simulation harness + CI workflow enforcing violation thresholds, uploads summaries. | DevOps |
| 2025-11-25 | DEVOPS-ORCH-34-001 completed: synthetic probe, replay smoke wrapper, burn-rate alert, README, incident/GA readiness playbook. | DevOps |
| 2025-11-25 | MIRROR-CRT-56-002 completed: mirror-sign workflow enforces prod signing for thin bundle v1 (REQUIRE_PROD_SIGNING=1). |
DevOps |
| 2025-11-25 | DEVOPS-SDK-63-001 completed: SDK signing/publishing toolchain, secrets guidance, CI workflow, offline/local feed config. | DevOps |
| 2025-11-25 | DEVOPS-TEN-47-001 marked BLOCKED: requires Authority tenancy harness and tenant fixture. | DevOps |
| 2025-11-25 | DEVOPS-TEN-48-001 marked BLOCKED: RLS/object-store/audit tests depend on TEN-47 harness. | DevOps |
| 2025-11-25 | DEVOPS-LEDGER-29-009-REL marked BLOCKED: waiting on LEDGER-29-009 dev outputs. | DevOps |
| 2025-11-25 | DEVOPS-LEDGER-TEN-48-001-REL marked BLOCKED: RLS migrations/artefacts depend on ledger tenant partition work. | DevOps |
| 2025-11-25 | DEVOPS-SCANNER-JAVA-21-011-REL marked BLOCKED: Java analyzer plugin artefacts unavailable. | DevOps |
| 2025-11-25 | Work paused: No space left on device; added cleanup helper scripts/devops/cleanup-workspace.sh and doc ops/devops/README-space.md. |
DevOps |
| 2025-11-25 | DEVOPS-SIG-26-001 completed: Signals Dockerfile/compose, Helm values, CI workflow, image export helper with Mongo/Redis deps. | DevOps |
| 2025-11-25 | DEVOPS-SIG-26-002 completed: Signals alerts, dashboard, playbook for latency/cache/staleness. | DevOps |
| 2025-11-23 | DEVOPS-REL-17-004 completed: release workflow uploads debug artefacts and fails on missing symbols. | DevOps |
| 2025-11-08 | Archived completed/historic work to docs/implplan/archived/tasks.md (updated 2025-11-08). |
Planning |
Decisions & Risks
- Hardened Docker/CI artefacts rely on available disk; keep cleanup script in runner docs.
- Cosign key management supports keyless; offline/air-gap paths require mirrored registry + secrets provided to
sbom_attest.sh. - Tenant chaos drill requires iptables/root; run only on isolated agents; monitor JWKS cache TTL to avoid auth outages.
- Surface.Env: ZASTAVA_* fallback to SCANNER_* in Helm/Compose; keep docs aligned if prefixes/fields change.
- Surface.Secrets: provisioning playbook published; ensure Helm/Compose env stays in sync; offline kit bundles encrypted secrets—unpack path must match
*_SURFACE_SECRETS_ROOT.
Next Checkpoints
- Unblock DEVOPS-TEN-47-001/48-001 by landing Authority tenancy harness and tenant fixtures.
- Deliver AIRGAP-TIME-57-001 to unblock mirror signing follow-ons (MIRROR-CRT-57/58) and export provenance chain.
- Free runner disk space routinely using
scripts/devops/cleanup-workspace.shand docker prune to keep CI green.