stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 1 Releases Wiki Activity

feat(docs): Add comprehensive documentation for Vexer, Vulnerability Explorer, and Zastava modules

Browse Source 
- Introduced AGENTS.md, README.md, TASKS.md, and implementation_plan.md for Vexer, detailing mission, responsibilities, key components, and operational notes.
- Established similar documentation structure for Vulnerability Explorer and Zastava modules, including their respective workflows, integrations, and observability notes.
- Created risk scoring profiles documentation outlining the core workflow, factor model, governance, and deliverables.
- Ensured all modules adhere to the Aggregation-Only Contract and maintain determinism and provenance in outputs.
This commit is contained in:
master
2025-10-30 00:09:39 +02:00
parent 3154c67978
commit 7b5bdcf4d3
503 changed files with 16136 additions and 54638 deletions
Download Patch File Download Diff File Expand all files Collapse all files

22
ops/devops/AGENTS.md
View File

@@ -1,11 +1,11 @@
# DevOps & Release — Agent Charter
## Mission
Execute deterministic build/release pipeline per `docs/ARCHITECTURE_DEVOPS.md`:
- Reproducible builds with SBOM/provenance, cosign signing, transparency logging.
- Channel manifests (LTS/Stable/Edge) with digests, Helm/Compose profiles.
- Performance guard jobs ensuring budgets.
## Expectations
- Coordinate with Scanner/Scheduler/Notify teams for artifact availability.
- Maintain CI reliability; update `TASKS.md` as states change.
# DevOps & Release — Agent Charter
## Mission
Execute deterministic build/release pipeline per `docs/modules/devops/ARCHITECTURE.md`:
- Reproducible builds with SBOM/provenance, cosign signing, transparency logging.
- Channel manifests (LTS/Stable/Edge) with digests, Helm/Compose profiles.
- Performance guard jobs ensuring budgets.
## Expectations
- Coordinate with Scanner/Scheduler/Notify teams for artifact availability.
- Maintain CI reliability; update `TASKS.md` as states change.

184
ops/devops/README.md
View File

@@ -1,92 +1,92 @@
# DevOps Release Automation
The **release** workflow builds and signs the StellaOps service containers,
generates SBOM + provenance attestations, and emits a canonical
`release.yaml`. The logic lives under `ops/devops/release/` and is invoked
by the new `.gitea/workflows/release.yml` pipeline.
## Local dry run
```bash
./ops/devops/release/build_release.py \
--version 2025.10.0-edge \
--channel edge \
--dry-run
```
Outputs land under `out/release/`. Use `--no-push` to run full builds without
pushing to the registry.
After the build completes, run the verifier to validate recorded hashes and artefact
presence:
```bash
python ops/devops/release/verify_release.py --release-dir out/release
```
## Python analyzer smoke & signing
`dotnet run --project tools/LanguageAnalyzerSmoke` exercises the Python language
analyzer plug-in against the golden fixtures (cold/warm timings, determinism). The
release workflow runs this harness automatically and then produces Cosign
signatures + SHA-256 sidecars for `StellaOps.Scanner.Analyzers.Lang.Python.dll`
and its `manifest.json`. Keep `COSIGN_KEY_REF`/`COSIGN_IDENTITY_TOKEN` populated so
the step can sign the artefacts; the generated `.sig`/`.sha256` files ship with the
Offline Kit bundle.
## Required tooling
- Docker 25+ with Buildx
- .NET 10 preview SDK (builds container stages and the SBOM generator)
- Node.js 20 (Angular UI build)
- Helm 3.16+
- Cosign 2.2+
Supply signing material via environment variables:
- `COSIGN_KEY_REF` – e.g. `file:./keys/cosign.key` or `azurekms://…`
- `COSIGN_PASSWORD` – password protecting the above key
The workflow defaults to multi-arch (`linux/amd64,linux/arm64`), SBOM in
CycloneDX, and SLSA provenance (`https://slsa.dev/provenance/v1`).
## Debug store extraction
`build_release.py` now exports stripped debug artefacts for every ELF discovered in the published images. The files land under `out/release/debug/.build-id/<aa>/<rest>.debug`, with metadata captured in `debug/debug-manifest.json` (and a `.sha256` sidecar). Use `jq` to inspect the manifest or `readelf -n` to spot-check a build-id. Offline Kit packaging should reuse the `debug/` directory as-is.
## UI auth smoke (Playwright)
As part of **DEVOPS-UI-13-006** the pipelines will execute the UI auth smoke
tests (`npm run test:e2e`) after building the Angular bundle. See
`docs/ops/ui-auth-smoke.md` for the job design, environment stubs, and
offline runner considerations.
## NuGet preview bootstrap
`.NET 10` preview packages (Microsoft.Extensions.*, JwtBearer 10.0 RC, Sqlite 9 RC)
ship from the public `dotnet-public` Azure DevOps feed. We mirror them into
`./local-nuget` so restores succeed inside Offline Kit.
1. Run `./ops/devops/sync-preview-nuget.sh` whenever you update the manifest.
2. The script now understands the optional `SourceBase` column (V3 flat container)
and writes packages alongside their SHA-256 checks.
3. `NuGet.config` registers the mirror (`local`), dotnet-public, and nuget.org.
Use `python3 ops/devops/validate_restore_sources.py` to prove the repo still
prefers the local mirror and that `Directory.Build.props` enforces the same order.
The validator now runs automatically in the `build-test-deploy` and `release`
workflows so CI fails fast when a feed priority regression slips in.
Detailed operator instructions live in `docs/ops/nuget-preview-bootstrap.md`.
## Telemetry collector tooling (DEVOPS-OBS-50-001)
- `ops/devops/telemetry/generate_dev_tls.sh` – generates a development CA and
client/server certificates for the OpenTelemetry collector overlay (mutual TLS).
- `ops/devops/telemetry/smoke_otel_collector.py` – sends OTLP traces/metrics/logs
over TLS and validates that the collector increments its receiver counters.
- `ops/devops/telemetry/package_offline_bundle.py` – re-packages collector assets for the Offline Kit.
- `deploy/compose/docker-compose.telemetry-storage.yaml` – Prometheus/Tempo/Loki stack for staging validation.
Combine these helpers with `deploy/compose/docker-compose.telemetry.yaml` to run
a secured collector locally before rolling out the Helm-based deployment.
# DevOps Release Automation
The **release** workflow builds and signs the StellaOps service containers,
generates SBOM + provenance attestations, and emits a canonical
`release.yaml`. The logic lives under `ops/devops/release/` and is invoked
by the new `.gitea/workflows/release.yml` pipeline.
## Local dry run
```bash
./ops/devops/release/build_release.py \
--version 2025.10.0-edge \
--channel edge \
--dry-run
```
Outputs land under `out/release/`. Use `--no-push` to run full builds without
pushing to the registry.
After the build completes, run the verifier to validate recorded hashes and artefact
presence:
```bash
python ops/devops/release/verify_release.py --release-dir out/release
```
## Python analyzer smoke & signing
`dotnet run --project src/Tools/LanguageAnalyzerSmoke` exercises the Python language
analyzer plug-in against the golden fixtures (cold/warm timings, determinism). The
release workflow runs this harness automatically and then produces Cosign
signatures + SHA-256 sidecars for `StellaOps.Scanner.Analyzers.Lang.Python.dll`
and its `manifest.json`. Keep `COSIGN_KEY_REF`/`COSIGN_IDENTITY_TOKEN` populated so
the step can sign the artefacts; the generated `.sig`/`.sha256` files ship with the
Offline Kit bundle.
## Required tooling
- Docker 25+ with Buildx
- .NET 10 preview SDK (builds container stages and the SBOM generator)
- Node.js 20 (Angular UI build)
- Helm 3.16+
- Cosign 2.2+
Supply signing material via environment variables:
- `COSIGN_KEY_REF` – e.g. `file:./keys/cosign.key` or `azurekms://…`
- `COSIGN_PASSWORD` – password protecting the above key
The workflow defaults to multi-arch (`linux/amd64,linux/arm64`), SBOM in
CycloneDX, and SLSA provenance (`https://slsa.dev/provenance/v1`).
## Debug store extraction
`build_release.py` now exports stripped debug artefacts for every ELF discovered in the published images. The files land under `out/release/debug/.build-id/<aa>/<rest>.debug`, with metadata captured in `debug/debug-manifest.json` (and a `.sha256` sidecar). Use `jq` to inspect the manifest or `readelf -n` to spot-check a build-id. Offline Kit packaging should reuse the `debug/` directory as-is.
## UI auth smoke (Playwright)
As part of **DEVOPS-UI-13-006** the pipelines will execute the UI auth smoke
tests (`npm run test:e2e`) after building the Angular bundle. See
`docs/modules/ui/operations/auth-smoke.md` for the job design, environment stubs, and
offline runner considerations.
## NuGet preview bootstrap
`.NET 10` preview packages (Microsoft.Extensions.*, JwtBearer 10.0 RC, Sqlite 9 RC)
ship from the public `dotnet-public` Azure DevOps feed. We mirror them into
`./local-nuget` so restores succeed inside Offline Kit.
1. Run `./ops/devops/sync-preview-nuget.sh` whenever you update the manifest.
2. The script now understands the optional `SourceBase` column (V3 flat container)
and writes packages alongside their SHA-256 checks.
3. `NuGet.config` registers the mirror (`local`), dotnet-public, and nuget.org.
Use `python3 ops/devops/validate_restore_sources.py` to prove the repo still
prefers the local mirror and that `Directory.Build.props` enforces the same order.
The validator now runs automatically in the `build-test-deploy` and `release`
workflows so CI fails fast when a feed priority regression slips in.
Detailed operator instructions live in `docs/modules/devops/runbooks/nuget-preview-bootstrap.md`.
## Telemetry collector tooling (DEVOPS-OBS-50-001)
- `ops/devops/telemetry/generate_dev_tls.sh` – generates a development CA and
client/server certificates for the OpenTelemetry collector overlay (mutual TLS).
- `ops/devops/telemetry/smoke_otel_collector.py` – sends OTLP traces/metrics/logs
over TLS and validates that the collector increments its receiver counters.
- `ops/devops/telemetry/package_offline_bundle.py` – re-packages collector assets for the Offline Kit.
- `deploy/compose/docker-compose.telemetry-storage.yaml` – Prometheus/Tempo/Loki stack for staging validation.
Combine these helpers with `deploy/compose/docker-compose.telemetry.yaml` to run
a secured collector locally before rolling out the Helm-based deployment.

344
ops/devops/TASKS.md
View File

@@ -1,172 +1,172 @@
# DevOps Task Board
## Governance & Rules
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|----|--------|----------|------------|-------------|---------------|
| DEVOPS-RULES-33-001 | DOING (2025-10-26) | DevOps Guild, Platform Leads | — | Contracts & Rules anchor:<br>• Gateway proxies only; Policy Engine composes overlays/simulations.<br>• AOC ingestion cannot merge; only lossless canonicalization.<br>• One graph platform: Graph Indexer + Graph API. Cartographer retired. | Rules posted in SPRINTS/TASKS; duplicates cleaned per guidance; reviewers acknowledge in changelog. |
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|----|--------|----------|------------|-------------|---------------|
| DEVOPS-HELM-09-001 | DONE | DevOps Guild | SCANNER-WEB-09-101 | Create Helm/Compose environment profiles (dev, staging, airgap) with deterministic digests. | Profiles committed under `deploy/`; docs updated; CI smoke deploy passes. |
| DEVOPS-SCANNER-09-204 | DONE (2025-10-21) | DevOps Guild, Scanner WebService Guild | SCANNER-EVENTS-15-201 | Surface `SCANNER__EVENTS__*` environment variables across docker-compose (dev/stage/airgap) and Helm values, defaulting to share the Redis queue DSN. | Compose/Helm configs ship enabled Redis event publishing with documented overrides; lint jobs updated; docs cross-link to new knobs. |
| DEVOPS-SCANNER-09-205 | DONE (2025-10-21) | DevOps Guild, Notify Guild | DEVOPS-SCANNER-09-204 | Add Notify smoke stage that tails the Redis stream and asserts `scanner.report.ready`/`scanner.scan.completed` reach Notify WebService in staging. | CI job reads Redis stream during scanner smoke deploy, confirms Notify ingestion via API, alerts on failure. |
| DEVOPS-PERF-10-001 | DONE | DevOps Guild | BENCH-SCANNER-10-001 | Add perf smoke job (SBOM compose <5 s target) to CI. | CI job runs sample build verifying <5 s; alerts configured. |
| DEVOPS-PERF-10-002 | DONE (2025-10-23) | DevOps Guild | BENCH-SCANNER-10-002 | Publish analyzer bench metrics to Grafana/perf workbook and alarm on ≥20 % regressions. | CI exports JSON for dashboards; Grafana panel wired; Ops on-call doc updated with alert hook. |
| DEVOPS-AOC-19-001 | BLOCKED (2025-10-26) | DevOps Guild, Platform Guild | WEB-AOC-19-003 | Integrate the AOC Roslyn analyzer and guard tests into CI, failing builds when ingestion projects attempt banned writes. | Analyzer runs in PR/CI pipelines, results surfaced in build summary, docs updated under `docs/ops/ci-aoc.md`. |
> Docs hand-off (2025-10-26): see `docs/ingestion/aggregation-only-contract.md` §5, `docs/architecture/overview.md`, and `docs/cli/cli-reference.md` for guard + verifier expectations.
| DEVOPS-AOC-19-002 | BLOCKED (2025-10-26) | DevOps Guild | CLI-AOC-19-002, CONCELIER-WEB-AOC-19-004, EXCITITOR-WEB-AOC-19-004 | Add pipeline stage executing `stella aoc verify --since` against seeded Mongo snapshots for Concelier + Excititor, publishing violation report artefacts. | Stage runs on main/nightly, fails on violations, artifacts retained, runbook documented. |
> Blocked: waiting on CLI verifier command and Concelier/Excititor guard endpoints to land (CLI-AOC-19-002, CONCELIER-WEB-AOC-19-004, EXCITITOR-WEB-AOC-19-004).
| DEVOPS-AOC-19-003 | BLOCKED (2025-10-26) | DevOps Guild, QA Guild | CONCELIER-WEB-AOC-19-003, EXCITITOR-WEB-AOC-19-003 | Enforce unit test coverage thresholds for AOC guard suites and ensure coverage exported to dashboards. | Coverage report includes guard projects, threshold gate passes/fails as expected, dashboards refreshed with new metrics. |
> Blocked: guard coverage suites and exporter hooks pending in Concelier/Excititor (CONCELIER-WEB-AOC-19-003, EXCITITOR-WEB-AOC-19-003).
| DEVOPS-AOC-19-101 | TODO (2025-10-28) | DevOps Guild, Concelier Storage Guild | CONCELIER-STORE-AOC-19-002 | Draft supersedes backfill rollout (freeze window, dry-run steps, rollback) once advisory_raw idempotency index passes staging verification. | Runbook committed in `docs/deploy/containers.md` + Offline Kit notes, staging rehearsal scheduled with dependencies captured in SPRINTS. |
| DEVOPS-OBS-50-001 | DONE (2025-10-26) | DevOps Guild, Observability Guild | TELEMETRY-OBS-50-001 | Deliver default OpenTelemetry collector deployment (Compose/Helm manifests), OTLP ingestion endpoints, and secure pipeline (authN, mTLS, tenant partitioning). Provide smoke test verifying traces/logs/metrics ingestion. | Collector manifests committed; smoke test green; docs updated; imposed rule banner reminder noted. |
| DEVOPS-OBS-50-002 | DOING (2025-10-26) | DevOps Guild, Security Guild | DEVOPS-OBS-50-001, TELEMETRY-OBS-51-002 | Stand up multi-tenant storage backends (Prometheus, Tempo/Jaeger, Loki) with retention policies, tenant isolation, and redaction guard rails. Integrate with Authority scopes for read paths. | Storage stack deployed with auth; retention configured; integration tests verify tenant isolation; runbook drafted. |
> Coordination started with Observability Guild (2025-10-26) to schedule staging rollout and provision service accounts. Staging bootstrap commands and secret names documented in `docs/ops/telemetry-storage.md`.
| DEVOPS-OBS-50-003 | DONE (2025-10-26) | DevOps Guild, Offline Kit Guild | DEVOPS-OBS-50-001 | Package telemetry stack configs for air-gapped installs (Offline Kit bundle, documented overrides, sample values) and automate checksum/signature generation. | Offline bundle includes collector+storage configs; checksums published; docs cross-linked; imposed rule annotation recorded. |
| DEVOPS-OBS-51-001 | TODO | DevOps Guild, Observability Guild | WEB-OBS-51-001, DEVOPS-OBS-50-001 | Implement SLO evaluator service (burn rate calculators, webhook emitters), Grafana dashboards, and alert routing to Notifier. Provide Terraform/Helm automation. | Dashboards live; evaluator emits webhooks; alert runbook referenced; staging alert fired in test. |
| DEVOPS-OBS-52-001 | TODO | DevOps Guild, Timeline Indexer Guild | TIMELINE-OBS-52-002 | Configure streaming pipeline (NATS/Redis/Kafka) with retention, partitioning, and backpressure tuning for timeline events; add CI validation of schema + rate caps. | Pipeline deployed; load test meets SLA; schema validation job passes; documentation updated. |
| DEVOPS-OBS-53-001 | TODO | DevOps Guild, Evidence Locker Guild | EVID-OBS-53-001 | Provision object storage with WORM/retention options (S3 Object Lock / MinIO immutability), legal hold automation, and backup/restore scripts for evidence locker. | Storage configured with WORM; legal hold script documented; backup test performed; runbook updated. |
| DEVOPS-OBS-54-001 | TODO | DevOps Guild, Security Guild | PROV-OBS-53-002, EVID-OBS-54-001 | Manage provenance signing infrastructure (KMS keys, rotation schedule, timestamp authority integration) and integrate verification jobs into CI. | Keys provisioned with rotation policy; timestamp authority configured; CI verifies sample bundles; audit trail stored. |
| DEVOPS-OBS-55-001 | TODO | DevOps Guild, Ops Guild | DEVOPS-OBS-51-001, WEB-OBS-55-001 | Implement incident mode automation: feature flag service, auto-activation via SLO burn-rate, retention override management, and post-incident reset job. | Incident mode toggles via API/CLI; automation tested in staging; reset job verified; runbook referenced. |
## Air-Gapped Mode (Epic 16)
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|----|--------|----------|------------|-------------|---------------|
| DEVOPS-AIRGAP-56-001 | TODO | DevOps Guild | AIRGAP-CTL-56-001 | Ship deny-all egress policies for Kubernetes (NetworkPolicy/eBPF) and docker-compose firewall rules; provide verification script for sealed mode. | Policies committed with tests; verification script passes/fails as expected; docs cross-linked. |
| DEVOPS-AIRGAP-56-002 | TODO | DevOps Guild, AirGap Importer Guild | AIRGAP-IMP-57-002 | Provide import tooling for bundle staging: checksum validation, offline object-store loader scripts, removable media guidance. | Scripts documented; smoke tests validate import; runbook updated. |
| DEVOPS-AIRGAP-56-003 | TODO | DevOps Guild, Container Distribution Guild | EXPORT-AIRGAP-56-002 | Build Bootstrap Pack pipeline bundling images/charts, generating checksums, and publishing manifest for offline transfer. | Pipeline runs in connected env; pack verified in air-gap smoke test; manifest recorded. |
| DEVOPS-AIRGAP-57-001 | TODO | DevOps Guild, Mirror Creator Guild | MIRROR-CRT-56-002 | Automate Mirror Bundle creation jobs with dual-control approvals, artifact signing, and checksum publication. | Approval workflow enforced; CI artifact includes DSSE/TUF metadata; audit logs stored. |
| DEVOPS-AIRGAP-57-002 | TODO | DevOps Guild, Authority Guild | AUTH-OBS-50-001 | Configure sealed-mode CI tests that run services with sealed flag and ensure no egress occurs (iptables + mock DNS). | CI suite fails on attempted egress; reports remediation; documentation updated. |
| DEVOPS-AIRGAP-58-001 | TODO | DevOps Guild, Notifications Guild | NOTIFY-AIRGAP-56-002 | Provide local SMTP/syslog container templates and health checks for sealed environments; integrate into Bootstrap Pack. | Templates deployed successfully; health checks in CI; docs updated. |
| DEVOPS-AIRGAP-58-002 | TODO | DevOps Guild, Observability Guild | DEVOPS-AIRGAP-56-001, DEVOPS-OBS-51-001 | Ship sealed-mode observability stack (Prometheus/Grafana/Tempo/Loki) pre-configured with offline dashboards and no remote exporters. | Stack boots offline; dashboards available; verification script confirms zero egress. |
| DEVOPS-REL-14-001 | DONE (2025-10-26) | DevOps Guild | SIGNER-API-11-101, ATTESTOR-API-11-201 | Deterministic build/release pipeline with SBOM/provenance, signing, manifest generation. | CI pipeline produces signed images + SBOM/attestations, manifests published with verified hashes, docs updated. |
| DEVOPS-REL-14-004 | DONE (2025-10-26) | DevOps Guild, Scanner Guild | DEVOPS-REL-14-001, SCANNER-ANALYZERS-LANG-10-309P | Extend release/offline smoke jobs to exercise the Python analyzer plug-in (warm/cold scans, determinism, signature checks). | Release/Offline pipelines run Python analyzer smoke suite; alerts hooked; docs updated with new coverage matrix. |
| DEVOPS-REL-17-002 | DONE (2025-10-26) | DevOps Guild | DEVOPS-REL-14-001, SCANNER-EMIT-17-701 | Persist stripped-debug artifacts organised by GNU build-id and bundle them into release/offline kits with checksum manifests. | CI job writes `.debug` files under `artifacts/debug/.build-id/`, manifest + checksums published, offline kit includes cache, smoke job proves symbol lookup via build-id. |
| DEVOPS-REL-17-004 | BLOCKED (2025-10-26) | DevOps Guild | DEVOPS-REL-17-002 | Ensure release workflow publishes `out/release/debug` (build-id tree + manifest) and fails when symbols are missing. | Release job emits debug artefacts, `mirror_debug_store.py` summary committed, warning cleared from build logs, docs updated. |
| DEVOPS-MIRROR-08-001 | DONE (2025-10-19) | DevOps Guild | DEVOPS-REL-14-001 | Stand up managed mirror profiles for `*.stella-ops.org` (Concelier/Excititor), including Helm/Compose overlays, multi-tenant secrets, CDN caching, and sync documentation. | Infra overlays committed, CI smoke deploy hits mirror endpoints, runbooks published for downstream sync and quota management. |
> Note (2025-10-26, BLOCKED): IdentityModel.Tokens patched for logging 9.x, but release bundle still fails because Docker cannot stream multi-arch build context (`unix:///var/run/docker.sock` unavailable, EOF during copy). Retry once docker daemon/socket is healthy; until then `out/release/debug` cannot be generated.
| DEVOPS-CONSOLE-23-001 | BLOCKED (2025-10-26) | DevOps Guild, Console Guild | CONSOLE-CORE-23-001 | Add console CI workflow (pnpm cache, lint, type-check, unit, Storybook a11y, Playwright, Lighthouse) with offline runners and artifact retention for screenshots/reports. | Workflow runs on PR & main, caches reduce install time, failing checks block merges, artifacts uploaded for triage, docs updated. |
> Blocked: Console workspace and package scripts (CONSOLE-CORE-23-001..005) are not yet present; CI cannot execute pnpm/Playwright/Lighthouse until the Next.js app lands.
| DEVOPS-CONSOLE-23-002 | TODO | DevOps Guild, Console Guild | DEVOPS-CONSOLE-23-001, CONSOLE-REL-23-301 | Produce `stella-console` container build + Helm chart overlays with deterministic digests, SBOM/provenance artefacts, and offline bundle packaging scripts. | Container published to registry mirror, Helm values committed, SBOM/attestations generated, offline kit job passes smoke test, docs updated. |
| DEVOPS-LAUNCH-18-100 | DONE (2025-10-26) | DevOps Guild | - | Finalise production environment footprint (clusters, secrets, network overlays) for full-platform go-live. | IaC/compose overlays committed, secrets placeholders documented, dry-run deploy succeeds in staging. |
| DEVOPS-LAUNCH-18-900 | DONE (2025-10-26) | DevOps Guild, Module Leads | Wave 0 completion | Collect “full implementation” sign-off from module owners and consolidate launch readiness checklist. | Sign-off record stored under `docs/ops/launch-readiness.md`; outstanding gaps triaged; checklist approved. |
| DEVOPS-LAUNCH-18-001 | DONE (2025-10-26) | DevOps Guild | DEVOPS-LAUNCH-18-100, DEVOPS-LAUNCH-18-900 | Production launch cutover rehearsal and runbook publication. | `docs/ops/launch-cutover.md` drafted, rehearsal executed with rollback drill, approvals captured. |
| DEVOPS-NUGET-13-001 | DONE (2025-10-25) | DevOps Guild, Platform Leads | DEVOPS-REL-14-001 | Add .NET 10 preview feeds / local mirrors so `Microsoft.Extensions.*` 10.0 preview packages restore offline; refresh restore docs. | NuGet.config maps preview feeds (or local mirrored packages), `dotnet restore` succeeds for Excititor/Concelier solutions without ad-hoc feed edits, docs updated for offline bootstrap. |
| DEVOPS-NUGET-13-002 | DONE (2025-10-26) | DevOps Guild | DEVOPS-NUGET-13-001 | Ensure all solutions/projects prefer `local-nuget` before public sources and document restore order validation. | `NuGet.config` and solution-level configs resolve from `local-nuget` first; automated check verifies priority; docs updated for restore ordering. |
| DEVOPS-NUGET-13-003 | DONE (2025-10-26) | DevOps Guild, Platform Leads | DEVOPS-NUGET-13-002 | Sweep `Microsoft.*` NuGet dependencies pinned to 8.* and upgrade to latest .NET 10 equivalents (or .NET 9 when 10 unavailable), updating restore guidance. | Dependency audit shows no 8.* `Microsoft.*` packages remaining; CI builds green; changelog/doc sections capture upgrade rationale. |
## Policy Engine v2
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|----|--------|----------|------------|-------------|---------------|
| DEVOPS-POLICY-20-001 | DONE (2025-10-26) | DevOps Guild, Policy Guild | POLICY-ENGINE-20-001 | Integrate DSL linting in CI (parser/compile) to block invalid policies; add pipeline step compiling sample policies. | CI fails on syntax errors; lint logs surfaced; docs updated with pipeline instructions. |
| DEVOPS-POLICY-20-003 | DONE (2025-10-26) | DevOps Guild, QA Guild | DEVOPS-POLICY-20-001, POLICY-ENGINE-20-005 | Determinism CI: run Policy Engine twice with identical inputs and diff outputs to guard non-determinism. | CI job compares outputs, fails on differences, logs stored; documentation updated. |
| DEVOPS-POLICY-20-004 | DONE (2025-10-27) | DevOps Guild, Scheduler Guild, CLI Guild | SCHED-MODELS-20-001, CLI-POLICY-20-002 | Automate policy schema exports: generate JSON Schema from `PolicyRun*` DTOs during CI, publish artefacts, and emit change alerts for CLI consumers (Slack + changelog). | CI stage outputs versioned schema files, uploads artefacts, notifies #policy-engine channel on change; docs/CLI references updated. |
> 2025-10-27: `.gitea/workflows/build-test-deploy.yml` publishes the `policy-schema-exports` artefact under `artifacts/policy-schemas/<commit>/` and posts Slack diffs via `POLICY_ENGINE_SCHEMA_WEBHOOK`; diff stored as `policy-schema-diff.patch`.
## Graph Explorer v1
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|----|--------|----------|------------|-------------|---------------|
## Orchestrator Dashboard
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|----|--------|----------|------------|-------------|---------------|
| DEVOPS-ORCH-32-001 | TODO | DevOps Guild, Orchestrator Service Guild | ORCH-SVC-32-001 | Provision orchestrator Postgres/message-bus infrastructure, add CI smoke deploy, seed Grafana dashboards (queue depth, inflight jobs), and document bootstrap. | Helm/Compose profiles committed; CI smoke deploy runs; dashboards live with metrics; runbook updated. |
| DEVOPS-ORCH-33-001 | TODO | DevOps Guild, Observability Guild | DEVOPS-ORCH-32-001, ORCH-SVC-33-001..003 | Publish Grafana dashboards/alerts for rate limiter, backpressure, error clustering, and DLQ depth; integrate with on-call rotations. | Dashboards and alerts configured; synthetic tests validate thresholds; on-call playbook updated. |
| DEVOPS-ORCH-34-001 | TODO | DevOps Guild, Orchestrator Service Guild | DEVOPS-ORCH-33-001, ORCH-SVC-34-001..003 | Harden production monitoring (synthetic probes, burn-rate alerts, replay smoke), document incident response, and prep GA readiness checklist. | Synthetic probes created; burn-rate alerts firing on test scenario; GA checklist approved; runbook linked. |
## Link-Not-Merge v1
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|----|--------|----------|------------|-------------|---------------|
| DEVOPS-LNM-22-001 | BLOCKED (2025-10-27) | DevOps Guild, Concelier Guild | CONCELIER-LNM-21-102 | Run migration/backfill pipelines for advisory observations/linksets in staging, validate counts/conflicts, and automate deployment steps. Awaiting storage backfill tooling. |
| DEVOPS-LNM-22-002 | BLOCKED (2025-10-27) | DevOps Guild, Excititor Guild | EXCITITOR-LNM-21-102 | Execute VEX observation/linkset backfill with monitoring; ensure NATS/Redis events integrated; document ops runbook. Blocked until Excititor storage migration lands. |
| DEVOPS-LNM-22-003 | TODO | DevOps Guild, Observability Guild | CONCELIER-LNM-21-005, EXCITITOR-LNM-21-005 | Add CI/monitoring coverage for new metrics (`advisory_observations_total`, `linksets_total`, etc.) and alerts on ingest-to-API SLA breaches. | Metrics scraped into Grafana; alert thresholds set; CI job verifies metric emission. |
## Graph & Vuln Explorer v1
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|----|--------|----------|------------|-------------|---------------|
| DEVOPS-GRAPH-24-001 | TODO | DevOps Guild, SBOM Service Guild | SBOM-GRAPH-24-002 | Load test graph index/adjacency APIs with 40k-node assets; capture perf dashboards and alert thresholds. | Perf suite added; dashboards live; alerts configured. |
| DEVOPS-GRAPH-24-002 | TODO | DevOps Guild, UI Guild | UI-GRAPH-24-001..005 | Integrate synthetic UI perf runs (Playwright/WebGL metrics) for Graph/Vuln explorers; fail builds on regression. | CI job runs UI perf tests; baseline stored; documentation updated. |
| DEVOPS-GRAPH-24-003 | TODO | DevOps Guild | WEB-GRAPH-24-002 | Implement smoke job for simulation endpoints ensuring we stay within SLA (<3s upgrade) and log results. | Smoke job in CI; alerts when SLA breached; runbook documented. |
| DEVOPS-POLICY-27-001 | TODO | DevOps Guild, DevEx/CLI Guild | CLI-POLICY-27-001, REGISTRY-API-27-001 | Add CI pipeline stages to run `stella policy lint|compile|test` with secret scanning on policy sources for PRs touching `/policies/**`; publish diagnostics artifacts. | Pipeline executes on PR/main, failures block merges, secret scan summary uploaded, docs updated. |
| DEVOPS-POLICY-27-002 | TODO | DevOps Guild, Policy Registry Guild | REGISTRY-API-27-005, SCHED-WORKER-27-301 | Provide optional batch simulation CI job (staging inventory) that triggers Registry run, polls results, and posts markdown summary to PR; enforce drift thresholds. | Job configurable via label, summary comment generated, drift threshold gates merges, runbook documented. |
| DEVOPS-POLICY-27-003 | TODO | DevOps Guild, Security Guild | AUTH-POLICY-27-002, REGISTRY-API-27-007 | Manage signing key material for policy publish pipeline (OIDC workload identity + cosign), rotate keys, and document verification steps; integrate attestation verification stage. | Keys stored in secure vault, rotation procedure documented, CI verifies attestations, audit logs recorded. |
| DEVOPS-POLICY-27-004 | TODO | DevOps Guild, Observability Guild | WEB-POLICY-27-005, TELEMETRY-CONSOLE-27-001 | Create dashboards/alerts for policy compile latency, simulation queue depth, approval latency, and promotion outcomes; integrate with on-call playbooks. | Grafana dashboards live, alerts tuned, runbooks updated, observability tests verify metric ingestion. |
> Remark (2025-10-20): Repacked `Mongo2Go` local feed to require MongoDB.Driver 3.5.0 + SharpCompress 0.41.0; cache regression tests green and NU1902/NU1903 suppressed.
> Remark (2025-10-21): Compose/Helm profiles now surface `SCANNER__EVENTS__*` toggles with docs pointing at new `.env` placeholders.
## Reachability v1
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|----|--------|----------|------------|-------------|---------------|
| DEVOPS-SIG-26-001 | TODO | DevOps Guild, Signals Guild | SIGNALS-24-001 | Provision CI/CD pipelines, Helm/Compose manifests for Signals service, including artifact storage and Redis dependencies. | Pipelines ship Signals service; deployment docs updated; smoke tests green. |
| DEVOPS-SIG-26-002 | TODO | DevOps Guild, Observability Guild | SIGNALS-24-004 | Create dashboards/alerts for reachability scoring latency, cache hit rates, sensor staleness. | Dashboards live; alert thresholds configured; documentation updated. |
| DEVOPS-VULN-29-001 | TODO | DevOps Guild, Findings Ledger Guild | LEDGER-29-002..009 | Provision CI jobs for ledger projector (replay, determinism), set up backups, monitor Merkle anchoring, and automate verification. | CI job verifies hash chains; backups documented; alerts for anchoring failures configured. |
| DEVOPS-VULN-29-002 | TODO | DevOps Guild, Vuln Explorer API Guild | VULN-API-29-002..009 | Configure load/perf tests (5M findings/tenant), query budget enforcement, API SLO dashboards, and alerts for `vuln_list_latency` and `projection_lag`. | Perf suite integrated; dashboards live; alerts firing; runbooks updated. |
| DEVOPS-VULN-29-003 | TODO | DevOps Guild, Console Guild | WEB-VULN-29-004, CONSOLE-VULN-29-007 | Instrument analytics pipeline for Vuln Explorer (telemetry ingestion, query hashes), ensure compliance with privacy/PII guardrails, and update observability docs. | Telemetry pipeline operational; PII redaction verified; docs updated with checklist. |
| DEVOPS-VEX-30-001 | TODO | DevOps Guild, VEX Lens Guild | VEXLENS-30-009, ISSUER-30-005 | Provision CI, load tests, dashboards, alerts for VEX Lens and Issuer Directory (compute latency, disputed totals, signature verification rates). | CI/perf suites running; dashboards live; alerts configured; docs updated. |
| DEVOPS-AIAI-31-001 | TODO | DevOps Guild, Advisory AI Guild | AIAI-31-006..007 | Stand up CI pipelines, inference monitoring, privacy logging review, and perf dashboards for Advisory AI (summaries/conflicts/remediation). | CI covers golden outputs, telemetry dashboards live, privacy controls reviewed, alerts configured. |
## Export Center
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|----|--------|----------|------------|-------------|---------------|
| DEVOPS-EXPORT-35-001 | BLOCKED (2025-10-29) | DevOps Guild, Exporter Service Guild | EXPORT-SVC-35-001..006 | Establish exporter CI pipeline (lint/test/perf smoke), configure object storage fixtures, seed Grafana dashboards, and document bootstrap steps. | CI pipeline running; smoke export job seeded; dashboards live; runbook updated. |
| DEVOPS-EXPORT-36-001 | TODO | DevOps Guild, Exporter Service Guild | DEVOPS-EXPORT-35-001, EXPORT-SVC-36-001..004 | Integrate Trivy compatibility validation, cosign signature checks, `trivy module db import` smoke tests, OCI distribution verification, and throughput/error dashboards. | CI executes cosign + Trivy import validation; OCI push smoke passes; dashboards/alerts configured. |
| DEVOPS-EXPORT-37-001 | TODO | DevOps Guild, Exporter Service Guild | DEVOPS-EXPORT-36-001, EXPORT-SVC-37-001..004 | Finalize exporter monitoring (failure alerts, verify metrics, retention jobs) and chaos/latency tests ahead of GA. | Alerts tuned; chaos tests documented; retention monitoring active; runbook updated. |
## CLI Parity & Task Packs
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|----|--------|----------|------------|-------------|---------------|
| DEVOPS-CLI-41-001 | TODO | DevOps Guild, DevEx/CLI Guild | CLI-CORE-41-001 | Establish CLI build pipeline (multi-platform binaries, SBOM, checksums), parity matrix CI enforcement, and release artifact signing. | Build pipeline operational; SBOM/checksums published; parity gate failing on drift; docs updated. |
| DEVOPS-CLI-42-001 | TODO | DevOps Guild | DEVOPS-CLI-41-001, CLI-PARITY-41-001 | Add CLI golden output tests, parity diff automation, pack run CI harness, and artifact cache for remote mode. | Golden tests running; parity diff automation in CI; pack run harness executes sample packs; documentation updated. |
| DEVOPS-CLI-43-001 | DOING (2025-10-27) | DevOps Guild | DEVOPS-CLI-42-001, TASKRUN-42-001 | Finalize multi-platform release automation, SBOM signing, parity gate enforcement, and Task Pack chaos tests. | Release automation verified; SBOM signed; parity gate enforced; chaos tests documented. |
> 2025-10-27: Release pipeline now packages CLI multi-platform artefacts with SBOM/signature coverage and enforces the CLI parity gate (`ops/devops/check_cli_parity.py`). Task Pack chaos smoke still pending CLI pack command delivery.
| DEVOPS-CLI-43-002 | TODO | DevOps Guild, Task Runner Guild | CLI-PACKS-43-001, TASKRUN-43-001 | Implement Task Pack chaos smoke in CI (random failure injection, resume, sealed-mode toggle) and publish evidence bundles for review. | Chaos smoke job runs nightly; failures alert Slack; evidence stored in `out/pack-chaos`; runbook updated. |
| DEVOPS-CLI-43-003 | TODO | DevOps Guild, DevEx/CLI Guild | CLI-PARITY-41-001, CLI-PACKS-42-001 | Integrate CLI golden output/parity diff automation into release gating; export parity report artifact consumed by Console Downloads workspace. | `check_cli_parity.py` wired to compare parity matrix and CLI outputs; artifact uploaded; release fails on regressions.
## Containerized Distribution (Epic 13)
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|----|--------|----------|------------|-------------|---------------|
| DEVOPS-CONTAINERS-44-001 | TODO | DevOps Guild | DOCKER-44-001..003 | Automate multi-arch image builds with buildx, SBOM generation, cosign signing, and signature verification in CI. | Pipeline builds amd64/arm64; SBOMs pushed as referrers; cosign verify job passes. |
| DEVOPS-CONTAINERS-45-001 | TODO | DevOps Guild | HELM-45-001 | Add Compose and Helm smoke tests (fresh VM + kind cluster) to CI; publish test artifacts and logs. | CI jobs running; failures block releases; documentation updated. |
| DEVOPS-CONTAINERS-46-001 | TODO | DevOps Guild | DEPLOY-PACKS-43-001 | Build air-gap bundle generator (`tools/make-airgap-bundle.sh`), produce signed bundle, and verify in CI using private registry. | Bundle artifact produced with signatures/checksums; verification job passes; instructions documented. |
### Container Images (Epic 13)
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|----|--------|----------|------------|-------------|---------------|
| DOCKER-44-001 | TODO | DevOps Guild, Service Owners | DEVOPS-CLI-41-001 | Author multi-stage Dockerfiles for all core services (API, Console, Orchestrator, Task Runner, Conseiller, Excitator, Policy, Notify, Export, AI) with non-root users, read-only file systems, and health scripts. | Dockerfiles committed; images build successfully; container security scans clean; health endpoints reachable. |
| DOCKER-44-002 | TODO | DevOps Guild | DOCKER-44-001 | Generate SBOMs and cosign attestations for each image and integrate verification into CI. | SBOMs attached as OCI artifacts; cosign signatures published; CI verifies signatures prior to release. |
| DOCKER-44-003 | TODO | DevOps Guild | DOCKER-44-001 | Implement `/health/liveness`, `/health/readiness`, `/version`, `/metrics`, and ensure capability endpoint returns `merge=false` for Conseiller/Excitator. | Endpoints available across services; automated tests confirm responses; documentation updated with imposed rule reminder. |
## Authority-Backed Scopes & Tenancy (Epic 14)
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|----|--------|----------|------------|-------------|---------------|
| DEVOPS-TEN-47-001 | TODO | DevOps Guild | AUTH-TEN-47-001 | Add JWKS cache monitoring, signature verification regression tests, and token expiration chaos tests to CI. | CI verifies tokens using cached keys; chaos test for expired keys passes; documentation updated. |
| DEVOPS-TEN-48-001 | TODO | DevOps Guild | WEB-TEN-48-001 | Build integration tests to assert RLS enforcement, tenant-prefixed object storage, and audit event emission; set up lint to prevent raw SQL bypass. | Tests fail on cross-tenant access; lint enforced; dashboards capture audit events. |
| DEVOPS-TEN-49-001 | TODO | DevOps Guild | AUTH-TEN-49-001 | Deploy audit pipeline, scope usage metrics, JWKS outage chaos tests, and tenant load/perf benchmarks. | Audit pipeline live; metrics dashboards updated; chaos tests documented; perf benchmarks recorded. |
## SDKs & OpenAPI (Epic 17)
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|----|--------|----------|------------|-------------|---------------|
| DEVOPS-OAS-61-001 | TODO | DevOps Guild, API Contracts Guild | OAS-61-002 | Add CI stages for OpenAPI linting, validation, and compatibility diff; enforce gating on PRs. | Pipeline active; merge blocked on failures; documentation updated. |
| DEVOPS-OAS-61-002 | TODO | DevOps Guild, Contract Testing Guild | CONTR-62-002 | Integrate mock server + contract test suite into PR and nightly workflows; publish artifacts. | Tests run in CI; artifacts stored; failures alert. |
| DEVOPS-SDK-63-001 | TODO | DevOps Guild, SDK Release Guild | SDKREL-63-001 | Provision registry credentials, signing keys, and secure storage for SDK publishing pipelines. | Keys stored/rotated; publish pipeline authenticated; audit logs recorded. |
| DEVOPS-DEVPORT-63-001 | TODO | DevOps Guild, Developer Portal Guild | DEVPORT-62-001 | Automate developer portal build pipeline with caching, link & accessibility checks, performance budgets. | Pipeline enforced; reports archived; failures gate merges. |
| DEVOPS-DEVPORT-64-001 | TODO | DevOps Guild, DevPortal Offline Guild | DVOFF-64-001 | Schedule `devportal --offline` nightly builds with checksum validation and artifact retention policies. | Nightly job running; checksums published; retention policy documented. |
## Attestor Console (Epic 19)
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|----|--------|----------|------------|-------------|---------------|
| DEVOPS-ATTEST-73-001 | TODO | DevOps Guild, Attestor Service Guild | ATTESTOR-72-002 | Provision CI pipelines for attestor service (lint/test/security scan, seed data) and manage secrets for KMS drivers. | CI pipeline running; secrets stored securely; docs updated. |
| DEVOPS-ATTEST-73-002 | TODO | DevOps Guild, KMS Guild | KMS-72-001 | Establish secure storage for signing keys (vault integration, rotation schedule) and audit logging. | Key storage configured; rotation documented; audit logs verified. |
| DEVOPS-ATTEST-74-001 | TODO | DevOps Guild, Transparency Guild | TRANSP-74-001 | Deploy transparency log witness infrastructure and monitoring. | Witness service deployed; dashboards/alerts live. |
| DEVOPS-ATTEST-74-002 | TODO | DevOps Guild, Export Attestation Guild | EXPORT-ATTEST-74-001 | Integrate attestation bundle builds into release/offline pipelines with checksum verification. | Bundle job in CI; checksum verification passes; docs updated. |
| DEVOPS-ATTEST-75-001 | TODO | DevOps Guild, Observability Guild | ATTEST-VERIFY-74-001 | Add dashboards/alerts for signing latency, verification failures, key rotation events. | Dashboards live; alerts configured. |
# DevOps Task Board
## Governance & Rules
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|----|--------|----------|------------|-------------|---------------|
| DEVOPS-RULES-33-001 | DOING (2025-10-26) | DevOps Guild, Platform Leads | — | Contracts & Rules anchor:<br>• Gateway proxies only; Policy Engine composes overlays/simulations.<br>• AOC ingestion cannot merge; only lossless canonicalization.<br>• One graph platform: Graph Indexer + Graph API. Cartographer retired. | Rules posted in SPRINTS/TASKS; duplicates cleaned per guidance; reviewers acknowledge in changelog. |
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|----|--------|----------|------------|-------------|---------------|
| DEVOPS-HELM-09-001 | DONE | DevOps Guild | SCANNER-WEB-09-101 | Create Helm/Compose environment profiles (dev, staging, airgap) with deterministic digests. | Profiles committed under `deploy/`; docs updated; CI smoke deploy passes. |
| DEVOPS-SCANNER-09-204 | DONE (2025-10-21) | DevOps Guild, Scanner WebService Guild | SCANNER-EVENTS-15-201 | Surface `SCANNER__EVENTS__*` environment variables across docker-compose (dev/stage/airgap) and Helm values, defaulting to share the Redis queue DSN. | Compose/Helm configs ship enabled Redis event publishing with documented overrides; lint jobs updated; docs cross-link to new knobs. |
| DEVOPS-SCANNER-09-205 | DONE (2025-10-21) | DevOps Guild, Notify Guild | DEVOPS-SCANNER-09-204 | Add Notify smoke stage that tails the Redis stream and asserts `scanner.report.ready`/`scanner.scan.completed` reach Notify WebService in staging. | CI job reads Redis stream during scanner smoke deploy, confirms Notify ingestion via API, alerts on failure. |
| DEVOPS-PERF-10-001 | DONE | DevOps Guild | BENCH-SCANNER-10-001 | Add perf smoke job (SBOM compose <5 s target) to CI. | CI job runs sample build verifying <5 s; alerts configured. |
| DEVOPS-PERF-10-002 | DONE (2025-10-23) | DevOps Guild | BENCH-SCANNER-10-002 | Publish analyzer bench metrics to Grafana/perf workbook and alarm on ≥20 % regressions. | CI exports JSON for dashboards; Grafana panel wired; Ops on-call doc updated with alert hook. |
| DEVOPS-AOC-19-001 | BLOCKED (2025-10-26) | DevOps Guild, Platform Guild | WEB-AOC-19-003 | Integrate the AOC Roslyn analyzer and guard tests into CI, failing builds when ingestion projects attempt banned writes. | Analyzer runs in PR/CI pipelines, results surfaced in build summary, docs updated under `docs/modules/devops/runbooks/ci-aoc.md`. |
> Docs hand-off (2025-10-26): see `docs/ingestion/aggregation-only-contract.md` §5, `docs/modules/platform/architecture-overview.md`, and `docs/modules/cli/guides/cli-reference.md` for guard + verifier expectations.
| DEVOPS-AOC-19-002 | BLOCKED (2025-10-26) | DevOps Guild | CLI-AOC-19-002, CONCELIER-WEB-AOC-19-004, EXCITITOR-WEB-AOC-19-004 | Add pipeline stage executing `stella aoc verify --since` against seeded Mongo snapshots for Concelier + Excititor, publishing violation report artefacts. | Stage runs on main/nightly, fails on violations, artifacts retained, runbook documented. |
> Blocked: waiting on CLI verifier command and Concelier/Excititor guard endpoints to land (CLI-AOC-19-002, CONCELIER-WEB-AOC-19-004, EXCITITOR-WEB-AOC-19-004).
| DEVOPS-AOC-19-003 | BLOCKED (2025-10-26) | DevOps Guild, QA Guild | CONCELIER-WEB-AOC-19-003, EXCITITOR-WEB-AOC-19-003 | Enforce unit test coverage thresholds for AOC guard suites and ensure coverage exported to dashboards. | Coverage report includes guard projects, threshold gate passes/fails as expected, dashboards refreshed with new metrics. |
> Blocked: guard coverage suites and exporter hooks pending in Concelier/Excititor (CONCELIER-WEB-AOC-19-003, EXCITITOR-WEB-AOC-19-003).
| DEVOPS-AOC-19-101 | TODO (2025-10-28) | DevOps Guild, Concelier Storage Guild | CONCELIER-STORE-AOC-19-002 | Draft supersedes backfill rollout (freeze window, dry-run steps, rollback) once advisory_raw idempotency index passes staging verification. | Runbook committed in `docs/deploy/containers.md` + Offline Kit notes, staging rehearsal scheduled with dependencies captured in SPRINTS. |
| DEVOPS-OBS-50-001 | DONE (2025-10-26) | DevOps Guild, Observability Guild | TELEMETRY-OBS-50-001 | Deliver default OpenTelemetry collector deployment (Compose/Helm manifests), OTLP ingestion endpoints, and secure pipeline (authN, mTLS, tenant partitioning). Provide smoke test verifying traces/logs/metrics ingestion. | Collector manifests committed; smoke test green; docs updated; imposed rule banner reminder noted. |
| DEVOPS-OBS-50-002 | DOING (2025-10-26) | DevOps Guild, Security Guild | DEVOPS-OBS-50-001, TELEMETRY-OBS-51-002 | Stand up multi-tenant storage backends (Prometheus, Tempo/Jaeger, Loki) with retention policies, tenant isolation, and redaction guard rails. Integrate with Authority scopes for read paths. | Storage stack deployed with auth; retention configured; integration tests verify tenant isolation; runbook drafted. |
> Coordination started with Observability Guild (2025-10-26) to schedule staging rollout and provision service accounts. Staging bootstrap commands and secret names documented in `docs/modules/telemetry/operations/storage.md`.
| DEVOPS-OBS-50-003 | DONE (2025-10-26) | DevOps Guild, Offline Kit Guild | DEVOPS-OBS-50-001 | Package telemetry stack configs for air-gapped installs (Offline Kit bundle, documented overrides, sample values) and automate checksum/signature generation. | Offline bundle includes collector+storage configs; checksums published; docs cross-linked; imposed rule annotation recorded. |
| DEVOPS-OBS-51-001 | TODO | DevOps Guild, Observability Guild | WEB-OBS-51-001, DEVOPS-OBS-50-001 | Implement SLO evaluator service (burn rate calculators, webhook emitters), Grafana dashboards, and alert routing to Notifier. Provide Terraform/Helm automation. | Dashboards live; evaluator emits webhooks; alert runbook referenced; staging alert fired in test. |
| DEVOPS-OBS-52-001 | TODO | DevOps Guild, Timeline Indexer Guild | TIMELINE-OBS-52-002 | Configure streaming pipeline (NATS/Redis/Kafka) with retention, partitioning, and backpressure tuning for timeline events; add CI validation of schema + rate caps. | Pipeline deployed; load test meets SLA; schema validation job passes; documentation updated. |
| DEVOPS-OBS-53-001 | TODO | DevOps Guild, Evidence Locker Guild | EVID-OBS-53-001 | Provision object storage with WORM/retention options (S3 Object Lock / MinIO immutability), legal hold automation, and backup/restore scripts for evidence locker. | Storage configured with WORM; legal hold script documented; backup test performed; runbook updated. |
| DEVOPS-OBS-54-001 | TODO | DevOps Guild, Security Guild | PROV-OBS-53-002, EVID-OBS-54-001 | Manage provenance signing infrastructure (KMS keys, rotation schedule, timestamp authority integration) and integrate verification jobs into CI. | Keys provisioned with rotation policy; timestamp authority configured; CI verifies sample bundles; audit trail stored. |
| DEVOPS-OBS-55-001 | TODO | DevOps Guild, Ops Guild | DEVOPS-OBS-51-001, WEB-OBS-55-001 | Implement incident mode automation: feature flag service, auto-activation via SLO burn-rate, retention override management, and post-incident reset job. | Incident mode toggles via API/CLI; automation tested in staging; reset job verified; runbook referenced. |
## Air-Gapped Mode (Epic 16)
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|----|--------|----------|------------|-------------|---------------|
| DEVOPS-AIRGAP-56-001 | TODO | DevOps Guild | AIRGAP-CTL-56-001 | Ship deny-all egress policies for Kubernetes (NetworkPolicy/eBPF) and docker-compose firewall rules; provide verification script for sealed mode. | Policies committed with tests; verification script passes/fails as expected; docs cross-linked. |
| DEVOPS-AIRGAP-56-002 | TODO | DevOps Guild, AirGap Importer Guild | AIRGAP-IMP-57-002 | Provide import tooling for bundle staging: checksum validation, offline object-store loader scripts, removable media guidance. | Scripts documented; smoke tests validate import; runbook updated. |
| DEVOPS-AIRGAP-56-003 | TODO | DevOps Guild, Container Distribution Guild | EXPORT-AIRGAP-56-002 | Build Bootstrap Pack pipeline bundling images/charts, generating checksums, and publishing manifest for offline transfer. | Pipeline runs in connected env; pack verified in air-gap smoke test; manifest recorded. |
| DEVOPS-AIRGAP-57-001 | TODO | DevOps Guild, Mirror Creator Guild | MIRROR-CRT-56-002 | Automate Mirror Bundle creation jobs with dual-control approvals, artifact signing, and checksum publication. | Approval workflow enforced; CI artifact includes DSSE/TUF metadata; audit logs stored. |
| DEVOPS-AIRGAP-57-002 | TODO | DevOps Guild, Authority Guild | AUTH-OBS-50-001 | Configure sealed-mode CI tests that run services with sealed flag and ensure no egress occurs (iptables + mock DNS). | CI suite fails on attempted egress; reports remediation; documentation updated. |
| DEVOPS-AIRGAP-58-001 | TODO | DevOps Guild, Notifications Guild | NOTIFY-AIRGAP-56-002 | Provide local SMTP/syslog container templates and health checks for sealed environments; integrate into Bootstrap Pack. | Templates deployed successfully; health checks in CI; docs updated. |
| DEVOPS-AIRGAP-58-002 | TODO | DevOps Guild, Observability Guild | DEVOPS-AIRGAP-56-001, DEVOPS-OBS-51-001 | Ship sealed-mode observability stack (Prometheus/Grafana/Tempo/Loki) pre-configured with offline dashboards and no remote exporters. | Stack boots offline; dashboards available; verification script confirms zero egress. |
| DEVOPS-REL-14-001 | DONE (2025-10-26) | DevOps Guild | SIGNER-API-11-101, ATTESTOR-API-11-201 | Deterministic build/release pipeline with SBOM/provenance, signing, manifest generation. | CI pipeline produces signed images + SBOM/attestations, manifests published with verified hashes, docs updated. |
| DEVOPS-REL-14-004 | DONE (2025-10-26) | DevOps Guild, Scanner Guild | DEVOPS-REL-14-001, SCANNER-ANALYZERS-LANG-10-309P | Extend release/offline smoke jobs to exercise the Python analyzer plug-in (warm/cold scans, determinism, signature checks). | Release/Offline pipelines run Python analyzer smoke suite; alerts hooked; docs updated with new coverage matrix. |
| DEVOPS-REL-17-002 | DONE (2025-10-26) | DevOps Guild | DEVOPS-REL-14-001, SCANNER-EMIT-17-701 | Persist stripped-debug artifacts organised by GNU build-id and bundle them into release/offline kits with checksum manifests. | CI job writes `.debug` files under `artifacts/debug/.build-id/`, manifest + checksums published, offline kit includes cache, smoke job proves symbol lookup via build-id. |
| DEVOPS-REL-17-004 | BLOCKED (2025-10-26) | DevOps Guild | DEVOPS-REL-17-002 | Ensure release workflow publishes `out/release/debug` (build-id tree + manifest) and fails when symbols are missing. | Release job emits debug artefacts, `mirror_debug_store.py` summary committed, warning cleared from build logs, docs updated. |
| DEVOPS-MIRROR-08-001 | DONE (2025-10-19) | DevOps Guild | DEVOPS-REL-14-001 | Stand up managed mirror profiles for `*.stella-ops.org` (Concelier/Excititor), including Helm/Compose overlays, multi-tenant secrets, CDN caching, and sync documentation. | Infra overlays committed, CI smoke deploy hits mirror endpoints, runbooks published for downstream sync and quota management. |
> Note (2025-10-26, BLOCKED): IdentityModel.Tokens patched for logging 9.x, but release bundle still fails because Docker cannot stream multi-arch build context (`unix:///var/run/docker.sock` unavailable, EOF during copy). Retry once docker daemon/socket is healthy; until then `out/release/debug` cannot be generated.
| DEVOPS-CONSOLE-23-001 | BLOCKED (2025-10-26) | DevOps Guild, Console Guild | CONSOLE-CORE-23-001 | Add console CI workflow (pnpm cache, lint, type-check, unit, Storybook a11y, Playwright, Lighthouse) with offline runners and artifact retention for screenshots/reports. | Workflow runs on PR & main, caches reduce install time, failing checks block merges, artifacts uploaded for triage, docs updated. |
> Blocked: Console workspace and package scripts (CONSOLE-CORE-23-001..005) are not yet present; CI cannot execute pnpm/Playwright/Lighthouse until the Next.js app lands.
| DEVOPS-CONSOLE-23-002 | TODO | DevOps Guild, Console Guild | DEVOPS-CONSOLE-23-001, CONSOLE-REL-23-301 | Produce `stella-console` container build + Helm chart overlays with deterministic digests, SBOM/provenance artefacts, and offline bundle packaging scripts. | Container published to registry mirror, Helm values committed, SBOM/attestations generated, offline kit job passes smoke test, docs updated. |
| DEVOPS-LAUNCH-18-100 | DONE (2025-10-26) | DevOps Guild | - | Finalise production environment footprint (clusters, secrets, network overlays) for full-platform go-live. | IaC/compose overlays committed, secrets placeholders documented, dry-run deploy succeeds in staging. |
| DEVOPS-LAUNCH-18-900 | DONE (2025-10-26) | DevOps Guild, Module Leads | Wave 0 completion | Collect “full implementation” sign-off from module owners and consolidate launch readiness checklist. | Sign-off record stored under `docs/modules/devops/runbooks/launch-readiness.md`; outstanding gaps triaged; checklist approved. |
| DEVOPS-LAUNCH-18-001 | DONE (2025-10-26) | DevOps Guild | DEVOPS-LAUNCH-18-100, DEVOPS-LAUNCH-18-900 | Production launch cutover rehearsal and runbook publication. | `docs/modules/devops/runbooks/launch-cutover.md` drafted, rehearsal executed with rollback drill, approvals captured. |
| DEVOPS-NUGET-13-001 | DONE (2025-10-25) | DevOps Guild, Platform Leads | DEVOPS-REL-14-001 | Add .NET 10 preview feeds / local mirrors so `Microsoft.Extensions.*` 10.0 preview packages restore offline; refresh restore docs. | NuGet.config maps preview feeds (or local mirrored packages), `dotnet restore` succeeds for Excititor/Concelier solutions without ad-hoc feed edits, docs updated for offline bootstrap. |
| DEVOPS-NUGET-13-002 | DONE (2025-10-26) | DevOps Guild | DEVOPS-NUGET-13-001 | Ensure all solutions/projects prefer `local-nuget` before public sources and document restore order validation. | `NuGet.config` and solution-level configs resolve from `local-nuget` first; automated check verifies priority; docs updated for restore ordering. |
| DEVOPS-NUGET-13-003 | DONE (2025-10-26) | DevOps Guild, Platform Leads | DEVOPS-NUGET-13-002 | Sweep `Microsoft.*` NuGet dependencies pinned to 8.* and upgrade to latest .NET 10 equivalents (or .NET 9 when 10 unavailable), updating restore guidance. | Dependency audit shows no 8.* `Microsoft.*` packages remaining; CI builds green; changelog/doc sections capture upgrade rationale. |
## Policy Engine v2
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|----|--------|----------|------------|-------------|---------------|
| DEVOPS-POLICY-20-001 | DONE (2025-10-26) | DevOps Guild, Policy Guild | POLICY-ENGINE-20-001 | Integrate DSL linting in CI (parser/compile) to block invalid policies; add pipeline step compiling sample policies. | CI fails on syntax errors; lint logs surfaced; docs updated with pipeline instructions. |
| DEVOPS-POLICY-20-003 | DONE (2025-10-26) | DevOps Guild, QA Guild | DEVOPS-POLICY-20-001, POLICY-ENGINE-20-005 | Determinism CI: run Policy Engine twice with identical inputs and diff outputs to guard non-determinism. | CI job compares outputs, fails on differences, logs stored; documentation updated. |
| DEVOPS-POLICY-20-004 | DONE (2025-10-27) | DevOps Guild, Scheduler Guild, CLI Guild | SCHED-MODELS-20-001, CLI-POLICY-20-002 | Automate policy schema exports: generate JSON Schema from `PolicyRun*` DTOs during CI, publish artefacts, and emit change alerts for CLI consumers (Slack + changelog). | CI stage outputs versioned schema files, uploads artefacts, notifies #policy-engine channel on change; docs/CLI references updated. |
> 2025-10-27: `.gitea/workflows/build-test-deploy.yml` publishes the `policy-schema-exports` artefact under `artifacts/policy-schemas/<commit>/` and posts Slack diffs via `POLICY_ENGINE_SCHEMA_WEBHOOK`; diff stored as `policy-schema-diff.patch`.
## Graph Explorer v1
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|----|--------|----------|------------|-------------|---------------|
## Orchestrator Dashboard
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|----|--------|----------|------------|-------------|---------------|
| DEVOPS-ORCH-32-001 | TODO | DevOps Guild, Orchestrator Service Guild | ORCH-SVC-32-001 | Provision orchestrator Postgres/message-bus infrastructure, add CI smoke deploy, seed Grafana dashboards (queue depth, inflight jobs), and document bootstrap. | Helm/Compose profiles committed; CI smoke deploy runs; dashboards live with metrics; runbook updated. |
| DEVOPS-ORCH-33-001 | TODO | DevOps Guild, Observability Guild | DEVOPS-ORCH-32-001, ORCH-SVC-33-001..003 | Publish Grafana dashboards/alerts for rate limiter, backpressure, error clustering, and DLQ depth; integrate with on-call rotations. | Dashboards and alerts configured; synthetic tests validate thresholds; on-call playbook updated. |
| DEVOPS-ORCH-34-001 | TODO | DevOps Guild, Orchestrator Service Guild | DEVOPS-ORCH-33-001, ORCH-SVC-34-001..003 | Harden production monitoring (synthetic probes, burn-rate alerts, replay smoke), document incident response, and prep GA readiness checklist. | Synthetic probes created; burn-rate alerts firing on test scenario; GA checklist approved; runbook linked. |
## Link-Not-Merge v1
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|----|--------|----------|------------|-------------|---------------|
| DEVOPS-LNM-22-001 | BLOCKED (2025-10-27) | DevOps Guild, Concelier Guild | CONCELIER-LNM-21-102 | Run migration/backfill pipelines for advisory observations/linksets in staging, validate counts/conflicts, and automate deployment steps. Awaiting storage backfill tooling. |
| DEVOPS-LNM-22-002 | BLOCKED (2025-10-27) | DevOps Guild, Excititor Guild | EXCITITOR-LNM-21-102 | Execute VEX observation/linkset backfill with monitoring; ensure NATS/Redis events integrated; document ops runbook. Blocked until Excititor storage migration lands. |
| DEVOPS-LNM-22-003 | TODO | DevOps Guild, Observability Guild | CONCELIER-LNM-21-005, EXCITITOR-LNM-21-005 | Add CI/monitoring coverage for new metrics (`advisory_observations_total`, `linksets_total`, etc.) and alerts on ingest-to-API SLA breaches. | Metrics scraped into Grafana; alert thresholds set; CI job verifies metric emission. |
## Graph & Vuln Explorer v1
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|----|--------|----------|------------|-------------|---------------|
| DEVOPS-GRAPH-24-001 | TODO | DevOps Guild, SBOM Service Guild | SBOM-GRAPH-24-002 | Load test graph index/adjacency APIs with 40k-node assets; capture perf dashboards and alert thresholds. | Perf suite added; dashboards live; alerts configured. |
| DEVOPS-GRAPH-24-002 | TODO | DevOps Guild, UI Guild | UI-GRAPH-24-001..005 | Integrate synthetic UI perf runs (Playwright/WebGL metrics) for Graph/Vuln explorers; fail builds on regression. | CI job runs UI perf tests; baseline stored; documentation updated. |
| DEVOPS-GRAPH-24-003 | TODO | DevOps Guild | WEB-GRAPH-24-002 | Implement smoke job for simulation endpoints ensuring we stay within SLA (<3s upgrade) and log results. | Smoke job in CI; alerts when SLA breached; runbook documented. |
| DEVOPS-POLICY-27-001 | TODO | DevOps Guild, DevEx/CLI Guild | CLI-POLICY-27-001, REGISTRY-API-27-001 | Add CI pipeline stages to run `stella policy lint|compile|test` with secret scanning on policy sources for PRs touching `/policies/**`; publish diagnostics artifacts. | Pipeline executes on PR/main, failures block merges, secret scan summary uploaded, docs updated. |
| DEVOPS-POLICY-27-002 | TODO | DevOps Guild, Policy Registry Guild | REGISTRY-API-27-005, SCHED-WORKER-27-301 | Provide optional batch simulation CI job (staging inventory) that triggers Registry run, polls results, and posts markdown summary to PR; enforce drift thresholds. | Job configurable via label, summary comment generated, drift threshold gates merges, runbook documented. |
| DEVOPS-POLICY-27-003 | TODO | DevOps Guild, Security Guild | AUTH-POLICY-27-002, REGISTRY-API-27-007 | Manage signing key material for policy publish pipeline (OIDC workload identity + cosign), rotate keys, and document verification steps; integrate attestation verification stage. | Keys stored in secure vault, rotation procedure documented, CI verifies attestations, audit logs recorded. |
| DEVOPS-POLICY-27-004 | TODO | DevOps Guild, Observability Guild | WEB-POLICY-27-005, TELEMETRY-CONSOLE-27-001 | Create dashboards/alerts for policy compile latency, simulation queue depth, approval latency, and promotion outcomes; integrate with on-call playbooks. | Grafana dashboards live, alerts tuned, runbooks updated, observability tests verify metric ingestion. |
> Remark (2025-10-20): Repacked `Mongo2Go` local feed to require MongoDB.Driver 3.5.0 + SharpCompress 0.41.0; cache regression tests green and NU1902/NU1903 suppressed.
> Remark (2025-10-21): Compose/Helm profiles now surface `SCANNER__EVENTS__*` toggles with docs pointing at new `.env` placeholders.
## Reachability v1
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|----|--------|----------|------------|-------------|---------------|
| DEVOPS-SIG-26-001 | TODO | DevOps Guild, Signals Guild | SIGNALS-24-001 | Provision CI/CD pipelines, Helm/Compose manifests for Signals service, including artifact storage and Redis dependencies. | Pipelines ship Signals service; deployment docs updated; smoke tests green. |
| DEVOPS-SIG-26-002 | TODO | DevOps Guild, Observability Guild | SIGNALS-24-004 | Create dashboards/alerts for reachability scoring latency, cache hit rates, sensor staleness. | Dashboards live; alert thresholds configured; documentation updated. |
| DEVOPS-VULN-29-001 | TODO | DevOps Guild, Findings Ledger Guild | LEDGER-29-002..009 | Provision CI jobs for ledger projector (replay, determinism), set up backups, monitor Merkle anchoring, and automate verification. | CI job verifies hash chains; backups documented; alerts for anchoring failures configured. |
| DEVOPS-VULN-29-002 | TODO | DevOps Guild, Vuln Explorer API Guild | VULN-API-29-002..009 | Configure load/perf tests (5M findings/tenant), query budget enforcement, API SLO dashboards, and alerts for `vuln_list_latency` and `projection_lag`. | Perf suite integrated; dashboards live; alerts firing; runbooks updated. |
| DEVOPS-VULN-29-003 | TODO | DevOps Guild, Console Guild | WEB-VULN-29-004, CONSOLE-VULN-29-007 | Instrument analytics pipeline for Vuln Explorer (telemetry ingestion, query hashes), ensure compliance with privacy/PII guardrails, and update observability docs. | Telemetry pipeline operational; PII redaction verified; docs updated with checklist. |
| DEVOPS-VEX-30-001 | TODO | DevOps Guild, VEX Lens Guild | VEXLENS-30-009, ISSUER-30-005 | Provision CI, load tests, dashboards, alerts for VEX Lens and Issuer Directory (compute latency, disputed totals, signature verification rates). | CI/perf suites running; dashboards live; alerts configured; docs updated. |
| DEVOPS-AIAI-31-001 | TODO | DevOps Guild, Advisory AI Guild | AIAI-31-006..007 | Stand up CI pipelines, inference monitoring, privacy logging review, and perf dashboards for Advisory AI (summaries/conflicts/remediation). | CI covers golden outputs, telemetry dashboards live, privacy controls reviewed, alerts configured. |
## Export Center
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|----|--------|----------|------------|-------------|---------------|
| DEVOPS-EXPORT-35-001 | BLOCKED (2025-10-29) | DevOps Guild, Exporter Service Guild | EXPORT-SVC-35-001..006 | Establish exporter CI pipeline (lint/test/perf smoke), configure object storage fixtures, seed Grafana dashboards, and document bootstrap steps. | CI pipeline running; smoke export job seeded; dashboards live; runbook updated. |
| DEVOPS-EXPORT-36-001 | TODO | DevOps Guild, Exporter Service Guild | DEVOPS-EXPORT-35-001, EXPORT-SVC-36-001..004 | Integrate Trivy compatibility validation, cosign signature checks, `trivy module db import` smoke tests, OCI distribution verification, and throughput/error dashboards. | CI executes cosign + Trivy import validation; OCI push smoke passes; dashboards/alerts configured. |
| DEVOPS-EXPORT-37-001 | TODO | DevOps Guild, Exporter Service Guild | DEVOPS-EXPORT-36-001, EXPORT-SVC-37-001..004 | Finalize exporter monitoring (failure alerts, verify metrics, retention jobs) and chaos/latency tests ahead of GA. | Alerts tuned; chaos tests documented; retention monitoring active; runbook updated. |
## CLI Parity & Task Packs
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|----|--------|----------|------------|-------------|---------------|
| DEVOPS-CLI-41-001 | TODO | DevOps Guild, DevEx/CLI Guild | CLI-CORE-41-001 | Establish CLI build pipeline (multi-platform binaries, SBOM, checksums), parity matrix CI enforcement, and release artifact signing. | Build pipeline operational; SBOM/checksums published; parity gate failing on drift; docs updated. |
| DEVOPS-CLI-42-001 | TODO | DevOps Guild | DEVOPS-CLI-41-001, CLI-PARITY-41-001 | Add CLI golden output tests, parity diff automation, pack run CI harness, and artifact cache for remote mode. | Golden tests running; parity diff automation in CI; pack run harness executes sample packs; documentation updated. |
| DEVOPS-CLI-43-001 | DOING (2025-10-27) | DevOps Guild | DEVOPS-CLI-42-001, TASKRUN-42-001 | Finalize multi-platform release automation, SBOM signing, parity gate enforcement, and Task Pack chaos tests. | Release automation verified; SBOM signed; parity gate enforced; chaos tests documented. |
> 2025-10-27: Release pipeline now packages CLI multi-platform artefacts with SBOM/signature coverage and enforces the CLI parity gate (`ops/devops/check_cli_parity.py`). Task Pack chaos smoke still pending CLI pack command delivery.
| DEVOPS-CLI-43-002 | TODO | DevOps Guild, Task Runner Guild | CLI-PACKS-43-001, TASKRUN-43-001 | Implement Task Pack chaos smoke in CI (random failure injection, resume, sealed-mode toggle) and publish evidence bundles for review. | Chaos smoke job runs nightly; failures alert Slack; evidence stored in `out/pack-chaos`; runbook updated. |
| DEVOPS-CLI-43-003 | TODO | DevOps Guild, DevEx/CLI Guild | CLI-PARITY-41-001, CLI-PACKS-42-001 | Integrate CLI golden output/parity diff automation into release gating; export parity report artifact consumed by Console Downloads workspace. | `check_cli_parity.py` wired to compare parity matrix and CLI outputs; artifact uploaded; release fails on regressions.
## Containerized Distribution (Epic 13)
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|----|--------|----------|------------|-------------|---------------|
| DEVOPS-CONTAINERS-44-001 | TODO | DevOps Guild | DOCKER-44-001..003 | Automate multi-arch image builds with buildx, SBOM generation, cosign signing, and signature verification in CI. | Pipeline builds amd64/arm64; SBOMs pushed as referrers; cosign verify job passes. |
| DEVOPS-CONTAINERS-45-001 | TODO | DevOps Guild | HELM-45-001 | Add Compose and Helm smoke tests (fresh VM + kind cluster) to CI; publish test artifacts and logs. | CI jobs running; failures block releases; documentation updated. |
| DEVOPS-CONTAINERS-46-001 | TODO | DevOps Guild | DEPLOY-PACKS-43-001 | Build air-gap bundle generator (`src/Tools/make-airgap-bundle.sh`), produce signed bundle, and verify in CI using private registry. | Bundle artifact produced with signatures/checksums; verification job passes; instructions documented. |
### Container Images (Epic 13)
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|----|--------|----------|------------|-------------|---------------|
| DOCKER-44-001 | TODO | DevOps Guild, Service Owners | DEVOPS-CLI-41-001 | Author multi-stage Dockerfiles for all core services (API, Console, Orchestrator, Task Runner, Conseiller, Excitator, Policy, Notify, Export, AI) with non-root users, read-only file systems, and health scripts. | Dockerfiles committed; images build successfully; container security scans clean; health endpoints reachable. |
| DOCKER-44-002 | TODO | DevOps Guild | DOCKER-44-001 | Generate SBOMs and cosign attestations for each image and integrate verification into CI. | SBOMs attached as OCI artifacts; cosign signatures published; CI verifies signatures prior to release. |
| DOCKER-44-003 | TODO | DevOps Guild | DOCKER-44-001 | Implement `/health/liveness`, `/health/readiness`, `/version`, `/metrics`, and ensure capability endpoint returns `merge=false` for Conseiller/Excitator. | Endpoints available across services; automated tests confirm responses; documentation updated with imposed rule reminder. |
## Authority-Backed Scopes & Tenancy (Epic 14)
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|----|--------|----------|------------|-------------|---------------|
| DEVOPS-TEN-47-001 | TODO | DevOps Guild | AUTH-TEN-47-001 | Add JWKS cache monitoring, signature verification regression tests, and token expiration chaos tests to CI. | CI verifies tokens using cached keys; chaos test for expired keys passes; documentation updated. |
| DEVOPS-TEN-48-001 | TODO | DevOps Guild | WEB-TEN-48-001 | Build integration tests to assert RLS enforcement, tenant-prefixed object storage, and audit event emission; set up lint to prevent raw SQL bypass. | Tests fail on cross-tenant access; lint enforced; dashboards capture audit events. |
| DEVOPS-TEN-49-001 | TODO | DevOps Guild | AUTH-TEN-49-001 | Deploy audit pipeline, scope usage metrics, JWKS outage chaos tests, and tenant load/perf benchmarks. | Audit pipeline live; metrics dashboards updated; chaos tests documented; perf benchmarks recorded. |
## SDKs & OpenAPI (Epic 17)
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|----|--------|----------|------------|-------------|---------------|
| DEVOPS-OAS-61-001 | TODO | DevOps Guild, API Contracts Guild | OAS-61-002 | Add CI stages for OpenAPI linting, validation, and compatibility diff; enforce gating on PRs. | Pipeline active; merge blocked on failures; documentation updated. |
| DEVOPS-OAS-61-002 | TODO | DevOps Guild, Contract Testing Guild | CONTR-62-002 | Integrate mock server + contract test suite into PR and nightly workflows; publish artifacts. | Tests run in CI; artifacts stored; failures alert. |
| DEVOPS-SDK-63-001 | TODO | DevOps Guild, SDK Release Guild | SDKREL-63-001 | Provision registry credentials, signing keys, and secure storage for SDK publishing pipelines. | Keys stored/rotated; publish pipeline authenticated; audit logs recorded. |
| DEVOPS-DEVPORT-63-001 | TODO | DevOps Guild, Developer Portal Guild | DEVPORT-62-001 | Automate developer portal build pipeline with caching, link & accessibility checks, performance budgets. | Pipeline enforced; reports archived; failures gate merges. |
| DEVOPS-DEVPORT-64-001 | TODO | DevOps Guild, DevPortal Offline Guild | DVOFF-64-001 | Schedule `devportal --offline` nightly builds with checksum validation and artifact retention policies. | Nightly job running; checksums published; retention policy documented. |
## Attestor Console (Epic 19)
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|----|--------|----------|------------|-------------|---------------|
| DEVOPS-ATTEST-73-001 | TODO | DevOps Guild, Attestor Service Guild | ATTESTOR-72-002 | Provision CI pipelines for attestor service (lint/test/security scan, seed data) and manage secrets for KMS drivers. | CI pipeline running; secrets stored securely; docs updated. |
| DEVOPS-ATTEST-73-002 | TODO | DevOps Guild, KMS Guild | KMS-72-001 | Establish secure storage for signing keys (vault integration, rotation schedule) and audit logging. | Key storage configured; rotation documented; audit logs verified. |
| DEVOPS-ATTEST-74-001 | TODO | DevOps Guild, Transparency Guild | TRANSP-74-001 | Deploy transparency log witness infrastructure and monitoring. | Witness service deployed; dashboards/alerts live. |
| DEVOPS-ATTEST-74-002 | TODO | DevOps Guild, Export Attestation Guild | EXPORT-ATTEST-74-001 | Integrate attestation bundle builds into release/offline pipelines with checksum verification. | Bundle job in CI; checksum verification passes; docs updated. |
| DEVOPS-ATTEST-75-001 | TODO | DevOps Guild, Observability Guild | ATTEST-VERIFY-74-001 | Add dashboards/alerts for signing latency, verification failures, key rotation events. | Dashboards live; alerts configured. |

272
ops/devops/telemetry/package_offline_bundle.py
View File

@@ -1,136 +1,136 @@
#!/usr/bin/env python3
"""Package telemetry collector assets for offline/air-gapped installs.
Outputs a tarball containing the collector configuration, Compose overlay,
Helm defaults, and operator README. A SHA-256 checksum sidecar is emitted, and
optional Cosign signing can be enabled with --sign.
"""
from __future__ import annotations
import argparse
import hashlib
import os
import subprocess
import sys
import tarfile
from pathlib import Path
from typing import Iterable
REPO_ROOT = Path(__file__).resolve().parents[3]
DEFAULT_OUTPUT = REPO_ROOT / "out" / "telemetry" / "telemetry-offline-bundle.tar.gz"
BUNDLE_CONTENTS: tuple[Path, ...] = (
Path("deploy/telemetry/README.md"),
Path("deploy/telemetry/otel-collector-config.yaml"),
Path("deploy/telemetry/storage/README.md"),
Path("deploy/telemetry/storage/prometheus.yaml"),
Path("deploy/telemetry/storage/tempo.yaml"),
Path("deploy/telemetry/storage/loki.yaml"),
Path("deploy/telemetry/storage/tenants/tempo-overrides.yaml"),
Path("deploy/telemetry/storage/tenants/loki-overrides.yaml"),
Path("deploy/helm/stellaops/files/otel-collector-config.yaml"),
Path("deploy/helm/stellaops/values.yaml"),
Path("deploy/helm/stellaops/templates/otel-collector.yaml"),
Path("deploy/compose/docker-compose.telemetry.yaml"),
Path("deploy/compose/docker-compose.telemetry-storage.yaml"),
Path("docs/ops/telemetry-collector.md"),
Path("docs/ops/telemetry-storage.md"),
)
def compute_sha256(path: Path) -> str:
sha = hashlib.sha256()
with path.open("rb") as handle:
for chunk in iter(lambda: handle.read(1024 * 1024), b""):
sha.update(chunk)
return sha.hexdigest()
def validate_files(paths: Iterable[Path]) -> None:
missing = [str(p) for p in paths if not (REPO_ROOT / p).exists()]
if missing:
raise FileNotFoundError(f"Missing bundle artefacts: {', '.join(missing)}")
def create_bundle(output_path: Path) -> Path:
output_path.parent.mkdir(parents=True, exist_ok=True)
with tarfile.open(output_path, "w:gz") as tar:
for rel_path in BUNDLE_CONTENTS:
abs_path = REPO_ROOT / rel_path
tar.add(abs_path, arcname=str(rel_path))
return output_path
def write_checksum(bundle_path: Path) -> Path:
digest = compute_sha256(bundle_path)
sha_path = bundle_path.with_suffix(bundle_path.suffix + ".sha256")
sha_path.write_text(f"{digest} {bundle_path.name}\n", encoding="utf-8")
return sha_path
def cosign_sign(bundle_path: Path, key_ref: str | None, identity_token: str | None) -> None:
cmd = ["cosign", "sign-blob", "--yes", str(bundle_path)]
if key_ref:
cmd.extend(["--key", key_ref])
env = os.environ.copy()
if identity_token:
env["COSIGN_IDENTITY_TOKEN"] = identity_token
try:
subprocess.run(cmd, check=True, env=env)
except FileNotFoundError as exc:
raise RuntimeError("cosign not found on PATH; install cosign or omit --sign") from exc
except subprocess.CalledProcessError as exc:
raise RuntimeError(f"cosign sign-blob failed: {exc}") from exc
def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument(
"--output",
type=Path,
default=DEFAULT_OUTPUT,
help=f"Output bundle path (default: {DEFAULT_OUTPUT})",
)
parser.add_argument(
"--sign",
action="store_true",
help="Sign the bundle using cosign (requires cosign on PATH)",
)
parser.add_argument(
"--cosign-key",
type=str,
default=os.environ.get("COSIGN_KEY_REF"),
help="Cosign key reference (file:..., azurekms://..., etc.)",
)
parser.add_argument(
"--identity-token",
type=str,
default=os.environ.get("COSIGN_IDENTITY_TOKEN"),
help="OIDC identity token for keyless signing",
)
return parser.parse_args(argv)
def main(argv: list[str] | None = None) -> int:
args = parse_args(argv)
validate_files(BUNDLE_CONTENTS)
bundle_path = args.output.resolve()
print(f"[*] Creating telemetry bundle at {bundle_path}")
create_bundle(bundle_path)
sha_path = write_checksum(bundle_path)
print(f"[✓] SHA-256 written to {sha_path}")
if args.sign:
print("[*] Signing bundle with cosign")
cosign_sign(bundle_path, args.cosign_key, args.identity_token)
sig_path = bundle_path.with_suffix(bundle_path.suffix + ".sig")
if sig_path.exists():
print(f"[✓] Cosign signature written to {sig_path}")
else:
print("[!] Cosign completed but signature file not found (ensure cosign version >= 2.2)")
return 0
if __name__ == "__main__":
sys.exit(main())
#!/usr/bin/env python3
"""Package telemetry collector assets for offline/air-gapped installs.
Outputs a tarball containing the collector configuration, Compose overlay,
Helm defaults, and operator README. A SHA-256 checksum sidecar is emitted, and
optional Cosign signing can be enabled with --sign.
"""
from __future__ import annotations
import argparse
import hashlib
import os
import subprocess
import sys
import tarfile
from pathlib import Path
from typing import Iterable
REPO_ROOT = Path(__file__).resolve().parents[3]
DEFAULT_OUTPUT = REPO_ROOT / "out" / "telemetry" / "telemetry-offline-bundle.tar.gz"
BUNDLE_CONTENTS: tuple[Path, ...] = (
Path("deploy/telemetry/README.md"),
Path("deploy/telemetry/otel-collector-config.yaml"),
Path("deploy/telemetry/storage/README.md"),
Path("deploy/telemetry/storage/prometheus.yaml"),
Path("deploy/telemetry/storage/tempo.yaml"),
Path("deploy/telemetry/storage/loki.yaml"),
Path("deploy/telemetry/storage/tenants/tempo-overrides.yaml"),
Path("deploy/telemetry/storage/tenants/loki-overrides.yaml"),
Path("deploy/helm/stellaops/files/otel-collector-config.yaml"),
Path("deploy/helm/stellaops/values.yaml"),
Path("deploy/helm/stellaops/templates/otel-collector.yaml"),
Path("deploy/compose/docker-compose.telemetry.yaml"),
Path("deploy/compose/docker-compose.telemetry-storage.yaml"),
Path("docs/modules/telemetry/operations/collector.md"),
Path("docs/modules/telemetry/operations/storage.md"),
)
def compute_sha256(path: Path) -> str:
sha = hashlib.sha256()
with path.open("rb") as handle:
for chunk in iter(lambda: handle.read(1024 * 1024), b""):
sha.update(chunk)
return sha.hexdigest()
def validate_files(paths: Iterable[Path]) -> None:
missing = [str(p) for p in paths if not (REPO_ROOT / p).exists()]
if missing:
raise FileNotFoundError(f"Missing bundle artefacts: {', '.join(missing)}")
def create_bundle(output_path: Path) -> Path:
output_path.parent.mkdir(parents=True, exist_ok=True)
with tarfile.open(output_path, "w:gz") as tar:
for rel_path in BUNDLE_CONTENTS:
abs_path = REPO_ROOT / rel_path
tar.add(abs_path, arcname=str(rel_path))
return output_path
def write_checksum(bundle_path: Path) -> Path:
digest = compute_sha256(bundle_path)
sha_path = bundle_path.with_suffix(bundle_path.suffix + ".sha256")
sha_path.write_text(f"{digest} {bundle_path.name}\n", encoding="utf-8")
return sha_path
def cosign_sign(bundle_path: Path, key_ref: str | None, identity_token: str | None) -> None:
cmd = ["cosign", "sign-blob", "--yes", str(bundle_path)]
if key_ref:
cmd.extend(["--key", key_ref])
env = os.environ.copy()
if identity_token:
env["COSIGN_IDENTITY_TOKEN"] = identity_token
try:
subprocess.run(cmd, check=True, env=env)
except FileNotFoundError as exc:
raise RuntimeError("cosign not found on PATH; install cosign or omit --sign") from exc
except subprocess.CalledProcessError as exc:
raise RuntimeError(f"cosign sign-blob failed: {exc}") from exc
def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument(
"--output",
type=Path,
default=DEFAULT_OUTPUT,
help=f"Output bundle path (default: {DEFAULT_OUTPUT})",
)
parser.add_argument(
"--sign",
action="store_true",
help="Sign the bundle using cosign (requires cosign on PATH)",
)
parser.add_argument(
"--cosign-key",
type=str,
default=os.environ.get("COSIGN_KEY_REF"),
help="Cosign key reference (file:..., azurekms://..., etc.)",
)
parser.add_argument(
"--identity-token",
type=str,
default=os.environ.get("COSIGN_IDENTITY_TOKEN"),
help="OIDC identity token for keyless signing",
)
return parser.parse_args(argv)
def main(argv: list[str] | None = None) -> int:
args = parse_args(argv)
validate_files(BUNDLE_CONTENTS)
bundle_path = args.output.resolve()
print(f"[*] Creating telemetry bundle at {bundle_path}")
create_bundle(bundle_path)
sha_path = write_checksum(bundle_path)
print(f"[✓] SHA-256 written to {sha_path}")
if args.sign:
print("[*] Signing bundle with cosign")
cosign_sign(bundle_path, args.cosign_key, args.identity_token)
sig_path = bundle_path.with_suffix(bundle_path.suffix + ".sig")
if sig_path.exists():
print(f"[✓] Cosign signature written to {sig_path}")
else:
print("[!] Cosign completed but signature file not found (ensure cosign version >= 2.2)")
return 0
if __name__ == "__main__":
sys.exit(main())