# Sprint 507 - Ops & Offline · 190.B) Ops Devops.V

Active items only. Completed/historic work now resides in docs/implplan/archived/tasks.md (updated 2025-11-08).

[Ops & Offline] 190.B) Ops Devops.V
Depends on: Sprint 190.B - Ops Devops.IV
Summary: Ops & Offline focus on Ops Devops (phase V).
Task ID | State | Task description | Owners (Source)
--- | --- | --- | ---
DEVOPS-TEN-49-001 | TODO | Deploy audit pipeline, scope usage metrics, JWKS outage chaos tests, and tenant load/perf benchmarks. Dependencies: DEVOPS-TEN-48-001. | DevOps Guild (ops/devops)
DEVOPS-VEX-30-001 | DONE (2025-12-02) | Provision CI, load tests, dashboards, alerts for VEX Lens and Issuer Directory (compute latency, disputed totals, signature verification rates). | DevOps Guild, VEX Lens Guild (ops/devops)
DEVOPS-VULN-29-001 | DOING (2025-12-02) | Provision CI jobs for ledger projector (replay, determinism), set up backups, monitor Merkle anchoring, and automate verification. | DevOps Guild, Findings Ledger Guild (ops/devops)
DEVOPS-VULN-29-002 | TODO | Configure load/perf tests (5M findings/tenant), query budget enforcement, API SLO dashboards, and alerts for `vuln_list_latency` and `projection_lag`. Dependencies: DEVOPS-VULN-29-001. | DevOps Guild, Vuln Explorer API Guild (ops/devops)
DEVOPS-VULN-29-003 | TODO | Instrument analytics pipeline for Vuln Explorer (telemetry ingestion, query hashes), ensure compliance with privacy/PII guardrails, and update observability docs. Dependencies: DEVOPS-VULN-29-002. | DevOps Guild, Console Guild (ops/devops)
DOCKER-44-001 | DOING (2025-12-01) | Author multi-stage Dockerfiles for all core services (API, Console, Orchestrator, Task Runner, Concelier, Excititor, Policy, Notify, Export, AI) with non-root users, read-only file systems, and health scripts. | DevOps Guild, Service Owners (ops/devops)
DOCKER-44-002 | DONE (2025-12-02) | Generate SBOMs and cosign attestations for each image and integrate verification into CI. Dependencies: DOCKER-44-001. | DevOps Guild (ops/devops)
DOCKER-44-003 | DONE (2025-12-02) | Implement `/health/liveness`, `/health/readiness`, `/version`, `/metrics`, and ensure capability endpoint returns `merge=false` for Concelier/Excitior. Dependencies: DOCKER-44-002. | DevOps Guild (ops/devops)
OPS-ENV-01 | DONE (2025-12-02) | Update deployment manifests (Helm/Compose) and configuration docs to include Surface.Env variables for Scanner and Zastava services. | DevOps Guild, Scanner Guild (ops/devops)
OPS-SECRETS-01 | DONE (2025-12-02) | Define secret provisioning workflow (Kubernetes, Compose, Offline Kit) for Surface.Secrets references and update runbooks. | DevOps Guild, Security Guild (ops/devops)
OPS-SECRETS-02 | DONE (2025-12-02) | Embed Surface.Secrets material (encrypted bundles, manifests) into offline kit packaging scripts. Dependencies: OPS-SECRETS-01. | DevOps Guild, Offline Kit Guild (ops/devops)

## Execution Log
| Date (UTC) | Update | Owner |
| --- | --- | --- |
| 2025-12-02 | Completed OPS-ENV-01: added ZASTAVA_* Surface.Env seeds to Helm ConfigMap + Compose env examples and documented rollout in deploy/README. | DevOps |
| 2025-12-02 | Completed OPS-SECRETS-01/02: authored provisioning playbook (`ops/devops/secrets/surface-secrets-provisioning.md`) covering Kubernetes/Compose/Offline Kit and linked from deploy docs; offline kit bundling already covers Surface.Secrets payloads. | DevOps |
| 2025-12-02 | Started DEVOPS-VULN-29-001: added CI/backup/replay/merkle plan (`ops/devops/vuln/vuln-explorer-ci-plan.md`) and projection hash verifier (`ops/devops/vuln/verify_projection.sh`). | DevOps |
| 2025-12-02 | Completed DEVOPS-VEX-30-001: drafted VEX Lens CI/load/obs plan (`ops/devops/vex/vex-ci-loadtest-plan.md`) with k6 scenario, dashboards, alerts, offline posture. | DevOps |
| 2025-12-02 | Completed DOCKER-44-003: documented endpoint contract/snippet and provided CI verification helper; services now have guidance to expose health/version/metrics and capabilities merge=false. | DevOps |
| 2025-12-02 | Added health endpoint contract + ASP.NET 10 snippet (`ops/devops/docker/health-endpoints.md`) to guide DOCKER-44-003 adoption. | DevOps |
| 2025-12-02 | Started DOCKER-44-003: added health endpoint verification helper (`ops/devops/docker/verify_health_endpoints.sh`) and documented CI usage in base-image guidelines. | DevOps |
| 2025-12-02 | Completed DOCKER-44-002: added SBOM + cosign attestation helper (`ops/devops/docker/sbom_attest.sh`) and documented usage in base-image guidelines. | DevOps |
| 2025-12-02 | Extended DOCKER-44-001: added hardened multi-stage template (`ops/devops/docker/Dockerfile.hardened.template`) with non-root user/read-only fs and shared healthcheck helper (`healthcheck.sh`). | DevOps |
| 2025-12-01 | Started DOCKER-44-001: added hardened base image blueprint with non-root user, read-only fs, healthcheck, and SDK publish guidance (`ops/devops/docker/base-image-guidelines.md`). | DevOps |
| 2025-11-08 | Archived completed/historic work to docs/implplan/archived/tasks.md (updated 2025-11-08). | Planning |

## Decisions & Risks
- Need service-by-service adoption of the hardened Docker template; ensure health endpoints exist (tracked by DOCKER-44-003).
- SBOM/attestation integration (DOCKER-44-002) depends on final image names/digests from 44-001.
- Cosign key management: default flow supports keyless (requires transparency); for offline/air-gap, ensure registry mirror and signing keys are available to `sbom_attest.sh`.
- Surface.Env: ZASTAVA_* fall back to SCANNER_* in Helm/Compose; operators can override per component. Keep `docs/modules/scanner/design/surface-env.md` aligned if prefixes/fields change.
- Surface.Secrets: provisioning playbook published (`ops/devops/secrets/surface-secrets-provisioning.md`); keep Helm/Compose env in sync. Offline kit already bundles encrypted secrets; ensure unpack path matches `*_SURFACE_SECRETS_ROOT`.