# Sprint 507 · Ops DevOps V (Ops & Offline 190.B)

## Topic & Scope
- Ops & Offline phase V: tenant audit/chaos, VEX Lens/Vuln Explorer CI+observability, hardened Docker images, SBOM/attestations, and Surface.Env/Surface.Secrets rollout.
- **Working directory:** ops/devops (plus service-specific Docker/ops assets under ops/devops/*).

## Dependencies & Concurrency
- Depends on Sprint 506 (Ops DevOps IV) outputs and TEN-48 harness for tenant tests.
- Docker hardening (DOCKER-44-001) underpins SBOM/health endpoints tasks.

## Documentation Prerequisites
- docs/modules/devops/architecture.md
- ops/devops/README.md
- ops/devops/docker/base-image-guidelines.md

## Delivery Tracker
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
| --- | --- | --- | --- | --- | --- |
| 1 | DEVOPS-TEN-49-001 | DONE (2025-12-03) | Depends on DEVOPS-TEN-48-001 | DevOps Guild | Deploy audit pipeline, usage metrics, JWKS outage chaos tests, tenant load/perf benchmarks. |
| 2 | DEVOPS-VEX-30-001 | DONE (2025-12-02) | None | DevOps Guild · VEX Lens Guild | CI/load tests/dashboards/alerts for VEX Lens and Issuer Directory. |
| 3 | DEVOPS-VULN-29-001 | DONE (2025-12-02) | None | DevOps Guild · Findings Ledger Guild | Provision CI jobs for ledger projector; backups, Merkle anchoring, verification. |
| 4 | DEVOPS-VULN-29-002 | DONE (2025-12-02) | Depends on 29-001 | DevOps Guild · Vuln Explorer API Guild | Load/perf tests (5M findings/tenant), budget enforcement, SLO dashboards, alerts. |
| 5 | DEVOPS-VULN-29-003 | DONE (2025-12-02) | Depends on 29-002 | DevOps Guild · Console Guild | Instrument analytics pipeline with query-hash metrics and PII guardrails. |
| 6 | DOCKER-44-001 | DONE (2025-12-03) | None | DevOps Guild · Service Owners | Multi-stage Dockerfiles with non-root user, RO FS, health scripts for core services. |
| 7 | DOCKER-44-002 | DONE (2025-12-02) | Depends on 44-001 | DevOps Guild | SBOMs + cosign attestations; integrate verification into CI. |
| 8 | DOCKER-44-003 | DONE (2025-12-02) | Depends on 44-002 | DevOps Guild | Implement health/version/metrics endpoints; ensure capability `merge=false` for Concelier/Excitior. |
| 9 | OPS-ENV-01 | DONE (2025-12-02) | None | DevOps Guild · Scanner Guild | Update manifests/config docs to include Surface.Env vars for Scanner and Zastava. |
| 10 | OPS-SECRETS-01 | DONE (2025-12-02) | None | DevOps Guild · Security Guild | Secret provisioning workflow for Surface.Secrets (Kubernetes, Compose, Offline Kit). |
| 11 | OPS-SECRETS-02 | DONE (2025-12-02) | Depends on 01 | DevOps Guild · Offline Kit Guild | Embed Surface.Secrets material into offline kit packaging scripts. |

## Execution Log
| Date (UTC) | Update | Owner |
| --- | --- | --- |
| 2025-12-03 | Completed DEVOPS-TEN-49-001: added tenant recording/alert rules, k6 load harness, chaos runbook/script, and deploy README import steps. | DevOps |
| 2025-12-03 | Completed DOCKER-44-001: service build matrix + build-all helper, console Dockerfile/healthcheck, APP_BINARY-ready hardened template. | DevOps |
| 2025-12-03 | Normalised sprint file to standard template; no status changes. | Planning |
| 2025-12-02 | Completed OPS-ENV-01: added ZASTAVA_* Surface.Env seeds to Helm ConfigMap + Compose env examples and documented rollout in deploy/README. | DevOps |
| 2025-12-02 | Completed OPS-SECRETS-01/02: provisioning playbook (`ops/devops/secrets/surface-secrets-provisioning.md`) covering Kubernetes/Compose/Offline Kit; offline kit bundling covers Surface.Secrets payloads. | DevOps |
| 2025-12-02 | Started DEVOPS-VULN-29-001: added CI/backup/replay/merkle plan and projection hash verifier script. | DevOps |
| 2025-12-02 | Completed DEVOPS-VULN-29-001: deterministic replay fixture, snapshot/hash, verifier script, CI/ops plan. | DevOps |
| 2025-12-02 | Added tenant audit assets for DEVOPS-TEN-49-001: dashboard, alerts, chaos script. | DevOps |
| 2025-12-02 | Completed DEVOPS-VULN-29-002: k6 load/observability assets and thresholds defined. | DevOps |
| 2025-12-02 | Started DEVOPS-TEN-49-001: drafted audit/usage/chaos plan covering metrics, JWKS fault drill, load benchmarks. | DevOps |
| 2025-12-02 | Started DEVOPS-VULN-29-002: added k6 load script, Grafana dashboard stub, alert rules. | DevOps |
| 2025-12-02 | Completed DEVOPS-VEX-30-001: VEX Lens CI/load/obs plan with k6 scenario, dashboards, alerts, offline posture. | DevOps |
| 2025-12-02 | Completed DOCKER-44-003: documented endpoint contract/snippet and provided CI verification helper; services guidance for health/version/metrics and capabilities merge=false. | DevOps |
| 2025-12-02 | Added health endpoint contract + ASP.NET 10 snippet to guide DOCKER-44-003 adoption. | DevOps |
| 2025-12-02 | Started DOCKER-44-003: added health endpoint verification helper and documented CI usage in base-image guidelines. | DevOps |
| 2025-12-02 | Completed DOCKER-44-002: SBOM + cosign attestation helper added and documented. | DevOps |
| 2025-12-02 | Extended DOCKER-44-001: hardened multi-stage template with non-root user/RO FS and shared healthcheck helper. | DevOps |
| 2025-12-01 | Started DOCKER-44-001: hardened base image blueprint and SDK publish guidance documented. | DevOps |
| 2025-11-08 | Archived completed/historic work to docs/implplan/archived/tasks.md (updated 2025-11-08). | Planning |

## Decisions & Risks
- Tenant chaos drills require TEN-48 harness orchestration or manual k6 + `jwks-chaos.sh`; run on isolated agents with sudo/iptables access to avoid collateral outages.
- Docker hardening template + service matrix are ready; service owners must adopt the template before enabling `readOnlyRootFilesystem` in Helm/Compose and before SBOM/attest jobs (44-002) are enforced.
- Surface.Secrets/Surface.Env alignment retained; validate offline kit unpack paths whenever images/paths change.

## Next Checkpoints
- Run TEN-48 harness once available to exercise tenant chaos/load assets end-to-end.
- Track service owner adoption of hardened Docker template via `ops/devops/docker/build-all.sh` and `verify_health_endpoints.sh`.
- Validate SBOM/attestation verification in CI with production image names/digests after new images are built from the matrix.