# Sprint 503 - Ops & Offline · 190.B) Ops Devops.I

Active items only. Completed/historic work now resides in docs/implplan/archived/tasks.md (updated 2025-11-08).

[Ops & Offline] 190.B) Ops Devops.I
Depends on: Sprint 100.A - Attestor, Sprint 110.A - AdvisoryAI, Sprint 120.A - AirGap, Sprint 130.A - Scanner, Sprint 140.A - Graph, Sprint 150.A - Orchestrator, Sprint 160.A - EvidenceLocker, Sprint 170.A - Notifier, Sprint 180.A - Cli

## Topic & Scope
- Stand up CI, signing, and offline pipelines that unblock module sprints without embedding DevOps work in dev backlogs.
- Provide sealed/airgap bootstrap artefacts and mirrors required by downstream airgap/attestation tasks.
- Ensure AOC/guard rails are enforced in CI across ingestion-heavy modules.

## Dependencies & Concurrency
- Upstream artefacts: mirror bundle automation (DEVOPS-AIRGAP-57-001), AOC analyzers, module-specific prep notes referenced per task.
- Runs in parallel with module sprints; deliverables are CI/pipeline assets, not code changes inside module working dirs.

## Documentation Prerequisites
- docs/modules/devops/architecture.md
- docs/modules/ci/architecture.md
- docs/airgap/** (for sealed-mode tasks)

## Delivery Tracker
| Task ID | State | Task description | Owners (Source) |
| --- | --- | --- | --- |
| DEVOPS-AIAI-31-001 | DONE (2025-11-30) | Stand up CI pipelines, inference monitoring, privacy logging review, and perf dashboards for Advisory AI (summaries/conflicts/remediation). | DevOps Guild, Advisory AI Guild (ops/devops) |
| DEVOPS-AIAI-31-002 | BLOCKED (2025-11-23) | Package advisory feeds (SBOM pointers + provenance) for release/offline kit; publish once CLI/Policy digests and SBOM feeds arrive. | DevOps Guild, Advisory AI Release (ops/devops) |
| DEVOPS-SPANSINK-31-003 | DONE (2025-11-30) | Deploy span sink/Signals pipeline for Excititor evidence APIs (31-003) and publish dashboards; unblock traces for `/v1/vex/observations/**`. | DevOps Guild · Observability Guild (ops/devops) |
| DEVOPS-AIRGAP-56-001 | DONE (2025-11-30) | Ship deny-all egress policies for Kubernetes (NetworkPolicy/eBPF) and docker-compose firewall rules; provide verification script for sealed mode. | DevOps Guild (ops/devops) |
| DEVOPS-AIRGAP-56-002 | DONE (2025-11-30) | Provide import tooling for bundle staging: checksum validation, offline object-store loader scripts, removable media guidance. Dependencies: DEVOPS-AIRGAP-56-001. | DevOps Guild, AirGap Importer Guild (ops/devops) |
| DEVOPS-AIRGAP-56-003 | DONE (2025-11-30) | Build Bootstrap Pack pipeline bundling images/charts, generating checksums, and publishing manifest for offline transfer. Dependencies: DEVOPS-AIRGAP-56-002. | DevOps Guild, Container Distribution Guild (ops/devops) |
| DEVOPS-AIRGAP-57-001 | DONE (2025-11-30) | Automate Mirror Bundle creation jobs with dual-control approvals, artifact signing, and checksum publication. Dependencies: DEVOPS-AIRGAP-56-003. | DevOps Guild, Mirror Creator Guild (ops/devops) |
| DEVOPS-AIRGAP-57-002 | BLOCKED (2025-11-18) | Waiting on upstream DEVOPS-AIRGAP-57-001 (mirror bundle automation) to provide artifacts/endpoints for sealed-mode CI; no sealed fixtures available to exercise tests. | DevOps Guild, Authority Guild (ops/devops) |
| DEVOPS-AIRGAP-58-001 | DONE (2025-11-30) | Provide local SMTP/syslog container templates and health checks for sealed environments; integrate into Bootstrap Pack. Dependencies: DEVOPS-AIRGAP-57-002. | DevOps Guild, Notifications Guild (ops/devops) |
| DEVOPS-AIRGAP-58-002 | DONE (2025-11-30) | Ship sealed-mode observability stack (Prometheus/Grafana/Tempo/Loki) pre-configured with offline dashboards and no remote exporters. Dependencies: DEVOPS-AIRGAP-58-001. | DevOps Guild, Observability Guild (ops/devops) |
| DEVOPS-AOC-19-001 | BLOCKED (2025-10-26) | Integrate the AOC Roslyn analyzer and guard tests into CI, failing builds when ingestion projects attempt banned writes. | DevOps Guild, Platform Guild (ops/devops) |
| DEVOPS-AOC-19-002 | BLOCKED (2025-10-26) | Add pipeline stage executing `stella aoc verify --since` against seeded Mongo snapshots for Concelier + Excititor, publishing violation report artefacts. Dependencies: DEVOPS-AOC-19-001. | DevOps Guild (ops/devops) |
| DEVOPS-AOC-19-003 | BLOCKED (2025-10-26) | Enforce unit test coverage thresholds for AOC guard suites and ensure coverage exported to dashboards. Dependencies: DEVOPS-AOC-19-002. | DevOps Guild, QA Guild (ops/devops) |
| DEVOPS-AOC-19-101 | DONE (2025-12-01) | Draft supersedes backfill rollout (freeze window, dry-run steps, rollback) once advisory_raw idempotency index passes staging verification. Dependencies: DEVOPS-AOC-19-003. | DevOps Guild, Concelier Storage Guild (ops/devops) |
| DEVOPS-ATTEST-73-001 | DONE (2025-11-30) | Provision CI pipelines for attestor service (lint/test/security scan, seed data) and manage secrets for KMS drivers. | DevOps Guild, Attestor Service Guild (ops/devops) |
| DEVOPS-ATTEST-73-002 | DONE (2025-11-30) | Establish secure storage for signing keys (vault integration, rotation schedule) and audit logging. Dependencies: DEVOPS-ATTEST-73-001. | DevOps Guild, KMS Guild (ops/devops) |
| DEVOPS-ATTEST-74-001 | DONE (2025-12-01) | Deploy transparency log witness infrastructure and monitoring. Dependencies: DEVOPS-ATTEST-73-002. | DevOps Guild, Transparency Guild (ops/devops) |
| DEVOPS-GRAPH-INDEX-28-010-REL | DONE (2025-12-01) | Publish signed Helm/Compose/offline bundles for Graph Indexer; depends on GRAPH-INDEX-28-010 dev artefacts. | DevOps Guild, Graph Indexer Guild (ops/devops) |
| DEVOPS-LNM-21-101-REL | DONE (2025-12-01) | Run/apply shard/index migrations (Concelier LNM) in release pipelines; capture artefacts and rollback scripts. | DevOps Guild, Concelier Storage Guild (ops/devops) |
| DEVOPS-LNM-21-102-REL | DONE (2025-12-01) | Package/publish LNM backfill/rollback bundles for release/offline kit; depends on 21-102 dev outputs. | DevOps Guild, Concelier Storage Guild (ops/devops) |
| DEVOPS-LNM-21-103-REL | DONE (2025-12-01) | Publish/rotate object-store seeds and offline bootstraps with provenance hashes; depends on 21-103 dev outputs. | DevOps Guild, Concelier Storage Guild (ops/devops) |
| DEVOPS-STORE-AOC-19-005-REL | BLOCKED | Release/offline-kit packaging for Concelier backfill; waiting on dataset hash + dev rehearsal. | DevOps Guild, Concelier Storage Guild (ops/devops) |
| DEVOPS-CONCELIER-CI-24-101 | DONE (2025-11-25) | Provide clean CI runner + warmed NuGet cache + vstest harness for Concelier WebService & Storage; deliver TRX/binlogs and unblock CONCELIER-GRAPH-24-101/28-102 and LNM-21-004..203. | DevOps Guild, Concelier Core Guild (ops/devops) |
| DEVOPS-SCANNER-CI-11-001 | DONE (2025-11-30) | Supply warmed cache/diag runner for Scanner analyzers (LANG-11-001, JAVA 21-005/008) with binlogs + TRX; unblock restore/test hangs. | DevOps Guild, Scanner EPDR Guild (ops/devops) |
| DEVOPS-SCANNER-JAVA-21-011-REL | DONE (2025-12-01) | Package/sign Java analyzer plug-in once dev task 21-011 delivers; publish to Offline Kit/CLI release pipelines with provenance. | DevOps Guild, Scanner Release Guild (ops/devops) |
| DEVOPS-SBOM-23-001 | DONE (2025-11-30) | Publish vetted offline NuGet feed + CI recipe for SbomService; prove with `dotnet test` run and share cache hashes; unblock SBOM-CONSOLE-23-001/002. | DevOps Guild, SBOM Service Guild (ops/devops) |
| FEED-REMEDIATION-1001 | BLOCKED (2025-11-24) | Define remediation scope and runbook for overdue feeds (CCCS/CERTBUND); schedule refresh; depends on PREP-FEEDCONN-ICS-KISA-PLAN. | Concelier Feed Owners (ops/devops) |
| FEEDCONN-ICSCISA-02-012 / FEEDCONN-KISA-02-008 | BLOCKED (2025-11-24) | Publish provenance refresh/connector schedule for ICSCISA/KISA feeds; execute remediation per runbook once owners provide plan. | Concelier Feed Owners (ops/devops) |

## Execution Log
| Date (UTC) | Update | Owner |
| --- | --- | --- |
| 2025-11-30 | Completed DEVOPS-AIRGAP-58-002: added sealed-mode observability compose stack (Prometheus/Grafana/Tempo/Loki) with offline configs plus health script under `ops/devops/airgap/`; ready for sealed-mode bootstrap. | DevOps |
| 2025-11-30 | Completed DEVOPS-SBOM-23-001: added SBOM CI runner (`ops/devops/sbom-ci-runner/run-sbom-ci.sh`) with warmed-cache restore, binlog/TRX outputs, and NuGet cache hash evidence; documented in runner README. | DevOps |
| 2025-11-30 | Completed DEVOPS-SCANNER-CI-11-001: added offline-friendly Scanner CI runner (`ops/devops/scanner-ci-runner/run-scanner-ci.sh`) and README; produces build binlog + TRX outputs from key test projects with warmed NuGet cache. | DevOps |
| 2025-11-30 | Completed DEVOPS-ATTEST-73-001/73-002: added attestor CI stub (`ops/devops/attestation/ci.yml`) and secrets/rotation plan in `ops/devops/attestation/README.md`; pending mirror into `.gitea/workflows/attestor-ci.yml` for live runs. | DevOps |
| 2025-11-30 | Completed DEVOPS-SPANSINK-31-003: added OTLP span sink compose stack + collector config (`docker-compose.spansink.yml`, `otel-spansink.yaml`), run script, and Grafana dashboard stub (`ops/devops/signals/dashboards/excititor-vex-traces.json`). | DevOps |
| 2025-11-30 | Completed DEVOPS-AIRGAP-57-001: added mirror bundle manifest/signing tooling (`build_mirror_bundle.py`) with dual-approval support and optional cosign, documented in `ops/devops/airgap/README.md`. | DevOps |
| 2025-11-30 | Completed DEVOPS-AIRGAP-56-003: added Bootstrap Pack builder scripts (`build_bootstrap_pack.py`, `build_bootstrap_pack.sh`) producing manifest and checksums for images/charts/extras; docs updated in `ops/devops/airgap/README.md`. | DevOps |
| 2025-11-30 | Completed DEVOPS-AIRGAP-56-002: added bundle staging/import tooling (`bundle_stage_import.py`, `stage-bundle.sh`, README) under `ops/devops/airgap/` with checksum validation and evidence report output. | DevOps |
| 2025-11-30 | Completed DEVOPS-AIRGAP-56-001: added K8s deny-all egress NetworkPolicy, compose DOCKER-USER guard script, and verification harness for Docker/Kubernetes under `ops/devops/airgap/`. | DevOps |
| 2025-11-25 | Delivered Concelier CI runner harness (`ops/devops/concelier-ci-runner/run-concelier-ci.sh`) with warmed NuGet cache + TRX/binlogs; artefacts land under `ops/devops/artifacts/concelier-ci/<ts>`. | DevOps |
| 2025-11-25 | Local execution of the runner still hits MSBuild worker shutdown on this host (MSB4242); script is ready, but a clean CI agent should be used to produce TRX/binlogs. | DevOps |
| 2025-11-23 | Normalised sprint toward template (sections added); added DEVOPS-CONCELIER-CI-24-101, DEVOPS-SCANNER-CI-11-001, DEVOPS-SBOM-23-001 to absorb CI/restore blockers from module sprints. | Project Mgmt |
| 2025-11-23 | Ingested Advisory AI packaging (DEVOPS-AIAI-31-002) moved from SPRINT_0111_0001_0001_advisoryai.md to keep ops work out of dev sprint. | Project Mgmt |
| 2025-11-24 | Added DEVOPS-SCANNER-JAVA-21-011-REL (moved from SPRINT_0131_0001_0001_scanner_surface.md) to keep DevOps release packaging in ops track. | Project Mgmt |
| 2025-11-24 | Added DEVOPS-SPANSINK-31-003 (Excititor span sink for 31-003 traces) moved from SPRINT_0119_0001_0001_excititor_i per ops-only directive. | Project Mgmt |
| 2025-11-24 | Imported Concelier feed ops items FEED-REMEDIATION-1001 and FEEDCONN-ICSCISA/KISA from Sprint 110; keeping feed remediation in ops track. | Project Mgmt |
| 2025-12-01 | Completed DEVOPS-AIRGAP-58-001: added syslog/SMTP compose stack (`ops/devops/airgap/compose-syslog-smtp.yaml`) and health script (`health_syslog_smtp.sh`); documented in airgap README for sealed environments. | DevOps |
| 2025-12-01 | Completed DEVOPS-AIRGAP-58-002: added sealed-mode observability compose (Prometheus/Grafana/Tempo/Loki) with offline configs and `health_observability.sh`; updated airgap README. | DevOps |
| 2025-12-01 | Marked DEVOPS-SPANSINK-31-003 to DOING; span sink/Signals pipeline setup underway. | DevOps |
| 2025-11-30 | Completed DEVOPS-AIRGAP-58-001: added syslog/SMTP compose stack (`ops/devops/airgap/compose-syslog-smtp.yaml`) and health script (`health_syslog_smtp.sh`); documented in airgap README for sealed environments. | DevOps |
| 2025-11-30 | DEVOPS-AIAI-31-001 DONE: added Advisory AI CI harness (`ops/devops/advisoryai-ci-runner/run-advisoryai-ci.sh`) producing binlog/TRX/summary; warmed local NuGet cache for offline runs; docs in runner README. | DevOps |
| 2025-12-01 | Completed DEVOPS-AOC-19-101: authored supersedes backfill rollout plan (`ops/devops/aoc/supersedes-rollout.md`) covering freeze window, dry-run, validation, rollback, evidence capture, and monitoring. | DevOps |
| 2025-12-01 | Completed DEVOPS-ATTEST-74-001: published transparency log witness deployment plan (`ops/devops/attestation/witness-plan.md`) with security hardening, CI tests, monitoring/alerts, and air-gap mode guidance. | DevOps |
| 2025-12-01 | Completed DEVOPS-GRAPH-INDEX-28-010-REL: documented signed Helm/Compose/offline bundle plan for Graph Indexer (`ops/devops/graph-indexer/release-plan.md`) including SBOMs, cosign attestations, air-gap bundle layout, and verification steps. | DevOps |
| 2025-12-01 | Completed DEVOPS-SCANNER-JAVA-21-011-REL: added Java analyzer release/offline plan (`ops/devops/scanner-java/release-plan.md`) covering SBOMs, cosign attestations, offline bundle packaging, and verification. | DevOps |
| 2025-12-01 | Completed DEVOPS-LNM-21-101/102/103-REL: added Concelier LNM release/offline plan (`ops/devops/concelier/lnm-release-plan.md`) covering shard/index migrations, backfill/rollback bundles, object-store seeds, offline tarball layout, signatures, and rollback. | DevOps |

## Decisions & Risks
- Mirror bundle automation (DEVOPS-AIRGAP-57-001) and AOC guardrails remain gating risks; several downstream tasks inherit these.
- New CI-runner tasks must produce reproducible binlogs/TRX and cache hashes to keep offline posture intact.

## Next Checkpoints
- 2025-11-25: CI runner provisioning check for Concelier/Scanner/SBOM cache jobs.
- 2025-11-27: Sealed-mode fixture availability review (DEVOPS-AIRGAP-57-002).