7c39058386
Some checks failed
Docs CI / lint-and-preview (push) Has been cancelled
AOC Guard CI / aoc-guard (push) Has been cancelled
AOC Guard CI / aoc-verify (push) Has been cancelled
Export Center CI / export-ci (push) Has been cancelled
Symbols Server CI / symbols-smoke (push) Has been cancelled
devportal-offline / build-offline (push) Has been cancelled
11 KiB
11 KiB
Sprint 505 - Ops & Offline · 190.B) Ops Devops.III
Active items only. Completed/historic work now resides in docs/implplan/archived/tasks.md (updated 2025-11-08).
[Ops & Offline] 190.B) Ops Devops.III Depends on: Sprint 190.B - Ops Devops.II Summary: Ops & Offline focus on Ops Devops (phase III).
| Task ID | State | Task description | Owners (Source) |
|---|---|---|---|
| DEVOPS-EXPORT-36-001 | DONE (2025-11-24) | Integrate Trivy compatibility validation, cosign signature checks, trivy module db import smoke tests, OCI distribution verification, and throughput/error dashboards. Dependencies: DEVOPS-EXPORT-35-001. |
DevOps Guild, Exporter Service Guild (ops/devops) |
| DEVOPS-EXPORT-37-001 | DONE (2025-11-24) | Finalize exporter monitoring (failure alerts, verify metrics, retention jobs) and chaos/latency tests ahead of GA. Dependencies: DEVOPS-EXPORT-36-001. | DevOps Guild, Exporter Service Guild (ops/devops) |
| DEVOPS-GRAPH-24-001 | DONE (2025-11-24) | Load test graph index/adjacency APIs with 40k-node assets; capture perf dashboards and alert thresholds. | DevOps Guild, SBOM Service Guild (ops/devops) |
| DEVOPS-GRAPH-24-002 | DONE (2025-11-24) | Integrate synthetic UI perf runs (Playwright/WebGL metrics) for Graph/Vuln explorers; fail builds on regression. Dependencies: DEVOPS-GRAPH-24-001. | DevOps Guild, UI Guild (ops/devops) |
| DEVOPS-GRAPH-24-003 | DONE (2025-11-24) | Implement smoke job for simulation endpoints ensuring we stay within SLA (<3s upgrade) and log results. Dependencies: DEVOPS-GRAPH-24-002. | DevOps Guild (ops/devops) |
| DEVOPS-LNM-TOOLING-22-000 | BLOCKED | Await upstream storage backfill tool specs and Excititor migration outputs to finalize package. | DevOps Guild · Concelier Guild · Excititor Guild (ops/devops) |
| DEVOPS-LNM-22-001 | BLOCKED (2025-10-27) | Blocked on DEVOPS-LNM-TOOLING-22-000; run migration/backfill pipelines for advisory observations/linksets in staging, validate counts/conflicts, and automate deployment steps. | DevOps Guild, Concelier Guild (ops/devops) |
| DEVOPS-LNM-22-002 | BLOCKED (2025-10-27) | Blocked on DEVOPS-LNM-TOOLING-22-000 and Excititor storage migration; execute VEX observation/linkset backfill with monitoring; ensure NATS/Redis events integrated; document ops runbook. Dependencies: DEVOPS-LNM-22-001. | DevOps Guild, Excititor Guild (ops/devops) |
| DEVOPS-LNM-22-003 | TODO | Add CI/monitoring coverage for new metrics (advisory_observations_total, linksets_total, etc.) and alerts on ingest-to-API SLA breaches. Dependencies: DEVOPS-LNM-22-002. |
DevOps Guild, Observability Guild (ops/devops) |
| DEVOPS-OAS-61-001 | DONE (2025-11-24) | Add CI stages for OpenAPI linting, validation, and compatibility diff; enforce gating on PRs. | DevOps Guild, API Contracts Guild (ops/devops) |
| DEVOPS-OAS-61-002 | DONE (2025-11-24) | Integrate mock server + contract test suite into PR and nightly workflows; publish artifacts. Dependencies: DEVOPS-OAS-61-001. | DevOps Guild, Contract Testing Guild (ops/devops) |
| DEVOPS-OPENSSL-11-001 | DONE (2025-11-24) | Package the OpenSSL 1.1 shim (tests/native/openssl-1.1/linux-x64) into test harness output so Mongo2Go suites discover it automatically. |
DevOps Guild, Build Infra Guild (ops/devops) |
| DEVOPS-OPENSSL-11-002 | DONE (2025-11-24) | Ensure CI runners and Docker images that execute Mongo2Go tests export LD_LIBRARY_PATH (or embed the shim) to unblock unattended pipelines. Dependencies: DEVOPS-OPENSSL-11-001. |
DevOps Guild, CI Guild (ops/devops) |
| DEVOPS-OBS-51-001 | DONE (2025-11-24) | Implement SLO evaluator service (burn rate calculators, webhook emitters), Grafana dashboards, and alert routing to Notifier. Provide Terraform/Helm automation. Dependencies: DEVOPS-OBS-50-002. | DevOps Guild, Observability Guild (ops/devops) |
| DEVOPS-OBS-52-001 | DONE (2025-11-24) | Configure streaming pipeline (NATS/Redis/Kafka) with retention, partitioning, and backpressure tuning for timeline events; add CI validation of schema + rate caps. Dependencies: DEVOPS-OBS-51-001. | DevOps Guild, Timeline Indexer Guild (ops/devops) |
| DEVOPS-OBS-53-001 | DONE (2025-11-24) | Provision object storage with WORM/retention options (S3 Object Lock / MinIO immutability), legal hold automation, and backup/restore scripts for evidence locker. Dependencies: DEVOPS-OBS-52-001. | DevOps Guild, Evidence Locker Guild (ops/devops) |
| DEVOPS-OBS-54-001 | DONE (2025-11-24) | Manage provenance signing infrastructure (KMS keys, rotation schedule, timestamp authority integration) and integrate verification jobs into CI. Dependencies: DEVOPS-OBS-53-001. | DevOps Guild, Security Guild (ops/devops) |
| DEVOPS-SCAN-90-004 | DONE (2025-11-24) | Add a CI job that runs the scanner determinism harness against the release matrix (N runs per image), uploads determinism.json, and fails when score < threshold; publish artifact to release notes. Dependencies: SCAN-DETER-186-009/010. |
DevOps Guild, Scanner Guild (ops/devops) |
| DEVOPS-SYMS-90-005 | DONE (2025-11-24) | Deploy Symbols.Server (CI smoke via compose/MinIO/Mongo), seed bucket, add Prometheus alerts, and ship reusable smoke workflow for release gating. Dependencies: SYMS-SERVER-401-011/013. | DevOps Guild, Symbols Guild (ops/devops) |
| DEVOPS-LEDGER-OAS-61-001-REL | BLOCKED (2025-11-24) | Waiting on Findings Ledger OpenAPI sources/examples from service guild; cannot add lint/diff/publish gates until spec exists. | DevOps Guild, Findings Ledger Guild (ops/devops) |
| DEVOPS-LEDGER-OAS-61-002-REL | BLOCKED (2025-11-24) | .well-known/openapi payload and host metadata not yet provided by Findings Ledger team; release validation blocked. |
DevOps Guild, Findings Ledger Guild (ops/devops) |
| DEVOPS-LEDGER-OAS-62-001-REL | BLOCKED (2025-11-24) | SDK generation/signing depends on finalized Ledger OAS and versioning matrix; awaiting upstream artefacts. | DevOps Guild, Findings Ledger Guild (ops/devops) |
| DEVOPS-LEDGER-OAS-63-001-REL | BLOCKED (2025-11-24) | Deprecation governance artefacts require upstream OAS change log and lifecycle policy; pending service guild delivery. | DevOps Guild, Findings Ledger Guild (ops/devops) |
| DEVOPS-LEDGER-PACKS-42-001-REL | BLOCKED (2025-11-24) | Snapshot/time-travel export packaging depends on Ledger schema + storage contract; waiting on upstream deliverables. | DevOps Guild, Findings Ledger Guild (ops/devops) |
| DEVOPS-LEDGER-PACKS-42-002-REL | TODO | Once OAS + storage contract arrive, add pack signing + integrity verification job to release bundles. | DevOps Guild, Findings Ledger Guild (ops/devops) |
Execution Log
| Date (UTC) | Update | Owner |
|---|---|---|
| 2025-11-24 | Completed DEVOPS-OAS-61-001/002: added OAS CI workflow .gitea/workflows/oas-ci.yml running compose, lint, examples, compat diff, contract tests, and uploading aggregate spec. |
Implementer |
| 2025-11-24 | Completed DEVOPS-OPENSSL-11-001: copied OpenSSL 1.1 shim into all test outputs (native/linux-x64) via shared Directory.Build.props; Authority tests succeed with Mongo2Go. | Implementer |
| 2025-11-24 | Completed DEVOPS-GRAPH-24-001: added k6 load script (scripts/graph/load-test.sh) and workflow .gitea/workflows/graph-load.yml to stress graph index/adjacency/search endpoints with perf thresholds and exported summary. |
Implementer |
| 2025-11-24 | Completed DEVOPS-GRAPH-24-002/003: added Playwright UI perf probe (scripts/graph/ui-perf.ts) and simulation smoke (scripts/graph/simulation-smoke.sh) with workflow .gitea/workflows/graph-ui-sim.yml uploading artifacts. |
Implementer |
| 2025-11-24 | Completed DEVOPS-EXPORT-36-001/37-001: exporter compatibility workflow .gitea/workflows/export-compat.yml plus Prometheus alerts (ops/devops/exporter/alerts.yaml) and Grafana dashboard (ops/devops/exporter/grafana/exporter-overview.json). |
Implementer |
| 2025-11-24 | Completed DEVOPS-OBS-51-001: added SLO burn alerts (ops/devops/observability/alerts-slo.yaml), Grafana board (ops/devops/observability/grafana/slo-burn.json), SLO evaluator script (scripts/observability/slo-evaluator.sh), and workflow .gitea/workflows/obs-slo.yml to collect Prometheus snapshots. |
Implementer |
| 2025-11-24 | Completed DEVOPS-OBS-52-001: streaming validation script (scripts/observability/streaming-validate.sh) and workflow .gitea/workflows/obs-stream.yml to validate NATS connectivity and capture retention/partition env; artifacts uploaded. |
Implementer |
| 2025-11-24 | Completed DEVOPS-OBS-53-001: evidence locker WORM/retention alerts (ops/devops/evidence-locker/alerts.yaml), Grafana board (ops/devops/evidence-locker/grafana/evidence-locker.json), and workflow .gitea/workflows/evidence-locker.yml to track retention summary. |
Implementer |
| 2025-11-24 | Completed DEVOPS-OBS-54-001: provenance alerts (ops/devops/provenance/alerts.yaml), Grafana board (ops/devops/provenance/grafana/provenance-overview.json), and workflow .gitea/workflows/provenance-check.yml as CI hook for rotation evidence. |
Implementer |
| 2025-11-24 | Completed DEVOPS-OBS-53-001: evidence locker WORM/retention alerts (ops/devops/evidence-locker/alerts.yaml), Grafana board (ops/devops/evidence-locker/grafana/evidence-locker.json), and workflow .gitea/workflows/evidence-locker.yml to track retention summary. |
Implementer |
| 2025-11-24 | Completed DEVOPS-SCAN-90-004: added determinism runner (scripts/scanner/determinism-run.sh) and workflow .gitea/workflows/scanner-determinism.yml to execute filtered determinism tests and upload TRX artifacts. |
Implementer |
| 2025-11-24 | Completed DEVOPS-EXPORT-36-001: added exporter compatibility workflow .gitea/workflows/export-compat.yml running Trivy, cosign verify, module import smoke, and OCI push/pull checks; reports uploaded. |
Implementer |
| 2025-11-24 | Completed DEVOPS-SYMS-90-005: added Symbols.Server compose smoke (ops/devops/symbols/docker-compose.symbols.yaml), MinIO bucket seeding + health harness (scripts/symbols/smoke.sh), alerts (ops/devops/symbols/alerts.yaml), and CI workflow .gitea/workflows/symbols-ci.yml. |
Implementer |
| 2025-11-24 | Completed DEVOPS-OPENSSL-11-002: exported LD_LIBRARY_PATH via scripts/enable-openssl11-shim.sh and wired it into CI workflows (build-test-deploy, export-ci, aoc-guard, docs) for Mongo2Go stability. |
Implementer |
| 2025-11-24 | Added Symbols release smoke workflow .gitea/workflows/symbols-release.yml to gate tag builds with compose+MinIO smoke and artifact upload. |
Implementer |
| 2025-11-24 | Marked DEVOPS-LEDGER-OAS-61/62/63 and DEVOPS-LEDGER-PACKS-42-001 BLOCKED pending upstream Findings Ledger OAS/spec artefacts and lifecycle policy; release CI gating cannot proceed without schemas/examples. | Implementer |
| 2025-11-24 | Work paused: repo filesystem out of space; unable to run CI/cleanup until disk space is reclaimed. | Implementer |
Decisions & Risks
- CI runners cannot spawn PTYs (“No space left on device”); all command-based validation/cleanup blocked until disk capacity is restored on the worker.
- Findings Ledger release tasks (DEVOPS-LEDGER-OAS-61/62/63, DEVOPS-LEDGER-PACKS-42-001/-002) remain blocked awaiting upstream Ledger OAS/specs and lifecycle policy; release gates cannot be implemented without those artefacts. | 2025-11-24 | Marked DEVOPS-LEDGER-OAS-61/62/63 and DEVOPS-LEDGER-PACKS-42-001 BLOCKED pending upstream Findings Ledger OAS/spec artefacts and lifecycle policy; release CI gating cannot proceed without schemas/examples. | Implementer |