From 8a72779c16f6c58282f35e1cfe2d8efcf934634b Mon Sep 17 00:00:00 2001 From: StellaOps Bot Date: Sun, 7 Dec 2025 00:09:24 +0000 Subject: [PATCH] ops: add mock-ready VEX/Vuln runbooks --- ...SPRINT_0502_0001_0001_ops_deployment_ii.md | 1 + docs/implplan/tasks-all.md | 16 +++--- docs/runbooks/vex-ops.md | 38 ++++++++++---- docs/runbooks/vuln-ops.md | 50 +++++++++++++------ .../lang/bun/multi-workspace/package.json | 8 +++ 5 files changed, 80 insertions(+), 33 deletions(-) create mode 100644 src/Scanner/__Tests/StellaOps.Scanner.Analyzers.Lang.Bun.Tests/Fixtures/lang/bun/multi-workspace/package.json diff --git a/docs/implplan/SPRINT_0502_0001_0001_ops_deployment_ii.md b/docs/implplan/SPRINT_0502_0001_0001_ops_deployment_ii.md index 5785e9402..bacc386ce 100644 --- a/docs/implplan/SPRINT_0502_0001_0001_ops_deployment_ii.md +++ b/docs/implplan/SPRINT_0502_0001_0001_ops_deployment_ii.md @@ -33,6 +33,7 @@ ## Execution Log | Date (UTC) | Update | Owner | | --- | --- | --- | +| 2025-12-06 | Added mock-ready VEX/Vuln ops runbooks (`docs/runbooks/vex-ops.md`, `docs/runbooks/vuln-ops.md`); tasks remain DOING until production digests/schemas land. | Deployment Guild | | 2025-12-06 | Drafted policy incident runbook (`docs/runbooks/policy-incident.md`); set DEPLOY-POLICY-27-002 to DOING pending policy overlay/digests. | Deployment Guild | | 2025-12-06 | Header normalised to standard template; no content/status changes. | Project Mgmt | | 2025-12-06 | Seeded mock dev release manifest (`deploy/releases/2025.09-mock-dev.yaml`) covering VEX Lens and Findings/Vuln stacks; tasks moved to DOING (dev-mock) for development packaging. Production release still awaits real digests. | Deployment Guild | diff --git a/docs/implplan/tasks-all.md b/docs/implplan/tasks-all.md index 8ed1e5063..0cc638c96 100644 --- a/docs/implplan/tasks-all.md +++ b/docs/implplan/tasks-all.md @@ -540,10 +540,10 @@ | DEPLOY-PACKS-43-001 | DOING (dev-mock 2025-12-06) | 2025-12-06 | SPRINT_0501_0001_0001_ops_deployment_i | Deployment Guild · Task Runner Guild | ops/deployment | Ship remote Task Runner worker profiles, object storage bootstrap, approval workflow integration, and Offline Kit packaging instructions. Dependencies: DEPLOY-PACKS-42-001. | Needs #7 artifacts | AGDP0101 | | DEPLOY-POLICY-27-001 | DOING (dev-mock 2025-12-06) | 2025-12-05 | SPRINT_0501_0001_0001_ops_deployment_i | Deployment Guild · Policy Registry Guild | ops/deployment | Produce Helm/Compose overlays for Policy Registry + simulation workers (migrations, buckets, signing keys, tenancy defaults). | WEPO0101 | DVPL0105 | | DEPLOY-POLICY-27-002 | DOING (draft 2025-12-06) | 2025-12-06 | SPRINT_0502_0001_0001_ops_deployment_ii | Deployment Guild · Policy Guild | ops/deployment | Drafted `docs/runbooks/policy-incident.md` (publish/promote, freeze, evidence); awaiting policy overlay schema/digests from DEPLOY-POLICY-27-001. | DEPLOY-POLICY-27-001 | DVPL0105 | -| DEPLOY-VEX-30-001 | DOING (dev-mock 2025-12-06) | 2025-12-06 | SPRINT_0502_0001_0001_ops_deployment_ii | Deployment + VEX Lens Guild | ops/deployment | Provide Helm/Compose overlays, scaling defaults, and offline kit instructions for VEX Lens service. | Wait for CCWO0101 schema | DVPL0101 | -| DEPLOY-VEX-30-002 | DOING (dev-mock 2025-12-06) | 2025-12-06 | SPRINT_0502_0001_0001_ops_deployment_ii | Deployment Guild | ops/deployment | Package Issuer Directory deployment manifests, backups, and security hardening guidance. Dependencies: DEPLOY-VEX-30-001. | Depends on #5 | DVPL0101 | -| DEPLOY-VULN-29-001 | DOING (dev-mock 2025-12-06) | 2025-12-06 | SPRINT_0502_0001_0001_ops_deployment_ii | Deployment + Vuln Guild | ops/deployment | Produce Helm/Compose overlays for Findings Ledger + projector, including DB migrations, Merkle anchor jobs, and scaling guidance. | Needs CCWO0101 | DVPL0101 | -| DEPLOY-VULN-29-002 | DOING (dev-mock 2025-12-06) | 2025-12-06 | SPRINT_0502_0001_0001_ops_deployment_ii | Deployment Guild | ops/deployment | Package `stella-vuln-explorer-api` deployment manifests, health checks, autoscaling policies, and offline kit instructions with signed images. Dependencies: DEPLOY-VULN-29-001. | Depends on #7 | DVPL0101 | +| DEPLOY-VEX-30-001 | DOING (dev-mock 2025-12-06) | 2025-12-06 | SPRINT_0502_0001_0001_ops_deployment_ii | Deployment + VEX Lens Guild | ops/deployment | Mock-ready runbook added (`docs/runbooks/vex-ops.md`); awaiting schema/digests for final Helm/Compose overlays. | Wait for CCWO0101 schema | DVPL0101 | +| DEPLOY-VEX-30-002 | DOING (dev-mock 2025-12-06) | 2025-12-06 | SPRINT_0502_0001_0001_ops_deployment_ii | Deployment Guild | ops/deployment | Issuer Directory guidance covered in `docs/runbooks/vex-ops.md`; finalize once DEPLOY-VEX-30-001 pins production values. | Depends on #5 | DVPL0101 | +| DEPLOY-VULN-29-001 | DOING (dev-mock 2025-12-06) | 2025-12-06 | SPRINT_0502_0001_0001_ops_deployment_ii | Deployment + Vuln Guild | ops/deployment | Mock-ready runbook added (`docs/runbooks/vuln-ops.md`); production overlays pending schema/digests. | Needs CCWO0101 | DVPL0101 | +| DEPLOY-VULN-29-002 | DOING (dev-mock 2025-12-06) | 2025-12-06 | SPRINT_0502_0001_0001_ops_deployment_ii | Deployment Guild | ops/deployment | Vuln Explorer API steps captured in `docs/runbooks/vuln-ops.md`; finalize with real pins after DEPLOY-VULN-29-001. | Depends on #7 | DVPL0101 | | DETER-186-008 | TODO | | SPRINT_0186_0001_0001_record_deterministic_execution | Scanner Guild | `src/Scanner/StellaOps.Scanner.WebService`, `src/Scanner/StellaOps.Scanner.Worker` | Wait for RLRC0101 fixture | Wait for RLRC0101 fixture | SCDT0101 | | DETER-186-009 | TODO | | SPRINT_0186_0001_0001_record_deterministic_execution | Scanner Guild · QA Guild | `src/Scanner/StellaOps.Scanner.Replay`, `src/Scanner/__Tests` | Depends on #1 | Depends on #1 | SCDT0101 | | DETER-186-010 | TODO | | SPRINT_0186_0001_0001_record_deterministic_execution | Scanner Guild · Export Center Guild | `src/Scanner/StellaOps.Scanner.WebService`, `docs/modules/scanner/operations/release.md` | Depends on #2 | Depends on #2 | SCDT0101 | @@ -2754,10 +2754,10 @@ | DEPLOY-PACKS-43-001 | DOING (dev-mock 2025-12-06) | 2025-12-06 | SPRINT_0501_0001_0001_ops_deployment_i | Deployment Guild · Task Runner Guild | ops/deployment | Ship remote Task Runner worker profiles, object storage bootstrap, approval workflow integration, and Offline Kit packaging instructions. Dependencies: DEPLOY-PACKS-42-001. | Needs #7 artifacts | AGDP0101 | | DEPLOY-POLICY-27-001 | DOING (dev-mock 2025-12-06) | 2025-12-05 | SPRINT_0501_0001_0001_ops_deployment_i | Deployment Guild · Policy Registry Guild | ops/deployment | Produce Helm/Compose overlays for Policy Registry + simulation workers, including Mongo migrations, object storage buckets, signing key secrets, and tenancy defaults. | Needs registry schema + secrets | AGDP0101 | | DEPLOY-POLICY-27-002 | DOING (draft 2025-12-06) | 2025-12-06 | SPRINT_0502_0001_0001_ops_deployment_ii | Deployment Guild · Policy Guild | ops/deployment | Drafted `docs/runbooks/policy-incident.md` (publish/promote, freeze, evidence); finalize once DEPLOY-POLICY-27-001 ships schema/digests. | Depends on 27-001 | AGDP0101 | -| DEPLOY-VEX-30-001 | DOING (dev-mock 2025-12-06) | 2025-12-06 | SPRINT_0502_0001_0001_ops_deployment_ii | Deployment + VEX Lens Guild | ops/deployment | Provide Helm/Compose overlays, scaling defaults, and offline kit instructions for VEX Lens service. | Wait for CCWO0101 schema | DVPL0101 | -| DEPLOY-VEX-30-002 | DOING (dev-mock 2025-12-06) | 2025-12-06 | SPRINT_0502_0001_0001_ops_deployment_ii | Deployment Guild | ops/deployment | Package Issuer Directory deployment manifests, backups, and security hardening guidance. Dependencies: DEPLOY-VEX-30-001. | Depends on #5 | DVPL0101 | -| DEPLOY-VULN-29-001 | DOING (dev-mock 2025-12-06) | 2025-12-06 | SPRINT_0502_0001_0001_ops_deployment_ii | Deployment + Vuln Guild | ops/deployment | Produce Helm/Compose overlays for Findings Ledger + projector, including DB migrations, Merkle anchor jobs, and scaling guidance. | Needs CCWO0101 | DVPL0101 | -| DEPLOY-VULN-29-002 | DOING (dev-mock 2025-12-06) | 2025-12-06 | SPRINT_0502_0001_0001_ops_deployment_ii | Deployment Guild | ops/deployment | Package `stella-vuln-explorer-api` deployment manifests, health checks, autoscaling policies, and offline kit instructions with signed images. Dependencies: DEPLOY-VULN-29-001. | Depends on #7 | DVPL0101 | +| DEPLOY-VEX-30-001 | DOING (dev-mock 2025-12-06) | 2025-12-06 | SPRINT_0502_0001_0001_ops_deployment_ii | Deployment + VEX Lens Guild | ops/deployment | Mock-ready runbook added (`docs/runbooks/vex-ops.md`); awaiting schema/digests for final Helm/Compose overlays. | Wait for CCWO0101 schema | DVPL0101 | +| DEPLOY-VEX-30-002 | DOING (dev-mock 2025-12-06) | 2025-12-06 | SPRINT_0502_0001_0001_ops_deployment_ii | Deployment Guild | ops/deployment | Issuer Directory guidance covered in `docs/runbooks/vex-ops.md`; finalize once DEPLOY-VEX-30-001 pins production values. | Depends on #5 | DVPL0101 | +| DEPLOY-VULN-29-001 | DOING (dev-mock 2025-12-06) | 2025-12-06 | SPRINT_0502_0001_0001_ops_deployment_ii | Deployment + Vuln Guild | ops/deployment | Mock-ready runbook added (`docs/runbooks/vuln-ops.md`); production overlays pending schema/digests. | Needs CCWO0101 | DVPL0101 | +| DEPLOY-VULN-29-002 | DOING (dev-mock 2025-12-06) | 2025-12-06 | SPRINT_0502_0001_0001_ops_deployment_ii | Deployment Guild | ops/deployment | Vuln Explorer API steps captured in `docs/runbooks/vuln-ops.md`; finalize with real pins after DEPLOY-VULN-29-001. | Depends on #7 | DVPL0101 | | DETER-186-008 | TODO | | SPRINT_0186_0001_0001_record_deterministic_execution | Scanner Guild | `src/Scanner/StellaOps.Scanner.WebService`, `src/Scanner/StellaOps.Scanner.Worker` | Wait for RLRC0101 fixture | Wait for RLRC0101 fixture | SCDT0101 | | DETER-186-009 | TODO | | SPRINT_0186_0001_0001_record_deterministic_execution | Scanner Guild · QA Guild | `src/Scanner/StellaOps.Scanner.Replay`, `src/Scanner/__Tests` | Depends on #1 | Depends on #1 | SCDT0101 | | DETER-186-010 | TODO | | SPRINT_0186_0001_0001_record_deterministic_execution | Scanner Guild · Export Center Guild | `src/Scanner/StellaOps.Scanner.WebService`, `docs/modules/scanner/operations/release.md` | Depends on #2 | Depends on #2 | SCDT0101 | diff --git a/docs/runbooks/vex-ops.md b/docs/runbooks/vex-ops.md index 4da11539c..432682aa6 100644 --- a/docs/runbooks/vex-ops.md +++ b/docs/runbooks/vex-ops.md @@ -1,15 +1,35 @@ -# VEX Ops Runbook — Draft Skeleton (2025-12-05 UTC) +# VEX Ops Runbook (dev-mock ready) -Status: draft placeholder. Inputs pending: DevOps rollout plan for signatures/ops. +Status: DRAFT (2025-12-06 UTC). Safe for dev/mock exercises; production rollouts wait on policy/VEX final digests. -## Recompute Storms -- Steps to mitigate; throttling knobs (to fill). +## Pre-flight (dev vs. prod) +1) Release manifest guard + - Dev/mock: `python ops/devops/release/check_release_manifest.py deploy/releases/2025.09-mock-dev.yaml --downloads deploy/downloads/manifest.json` + - Prod: rerun against `deploy/releases/2025.09-stable.yaml` once VEX digests land. +2) Render plan + - Helm (mock overlay): `helm template vex-mock ./deploy/helm/stellaops -f deploy/helm/stellaops/values-mock.yaml --debug --validate > /tmp/vex-mock.yaml` + - Compose (dev with overlay): `USE_MOCK=1 deploy/compose/scripts/quickstart.sh env/dev.env.example && docker compose --env-file env/dev.env.example -f deploy/compose/docker-compose.dev.yaml -f deploy/compose/docker-compose.mock.yaml config > /tmp/vex-compose.yaml` +3) Backups (when touching prod data) — not required for mock, but in prod take Mongo snapshots for issuer-directory and VEX state before rollout. -## Mapping Failures -- Triage steps; retry/backfill guidance. +## Deploy (mock path) +- Helm dry-run already covers structural checks. To apply in a dev cluster: `helm upgrade --install stellaops ./deploy/helm/stellaops -f deploy/helm/stellaops/values-mock.yaml --atomic --timeout 10m`. +- Observe VEX Lens pod logs: `kubectl logs deploy/vex-lens -n stellaops --tail=200 -f`. +- Issuer Directory seed: ensure `issuer-directory-config` ConfigMap includes `csaf-publishers.json`; mock overlay already mounts default seed. -## Signature Errors -- Diagnosis workflow; key rotation checks. +## Rollback +- Helm: `helm rollback stellaops 1` (choose previous revision). Mock overlay uses `stellaops.dev/mock: "true"` annotations; safe to tear down after tests. +- Compose: `docker compose --env-file env/dev.env.example -f docker-compose.dev.yaml -f docker-compose.mock.yaml down`. + +## Troubleshooting +- Recompute storms: throttle via `VEX_LENS__MAX_PARALLELISM` env (set in values once schema lands); for now scale deployment down to 1 replica to reduce concurrency. +- Mapping failures: capture request/response with `kubectl logs ... --since=10m`; rerun after clearing queue. +- Signature errors: confirm Authority token audience/issuer; mock overlay uses the same auth settings as dev compose. + +## Evidence capture +- Save `/tmp/vex-mock.yaml` and `/tmp/vex-compose.yaml` with the manifest used. +- `kubectl get deploy/pod,svc -n stellaops -l app=vex-lens -o yaml > /tmp/vex-live.yaml`. +- Tarball: `tar -czf vex-evidence-$(date -u +%Y%m%dT%H%M%SZ).tar.gz /tmp/vex-*`. ## Open TODOs -- Add concrete commands and dashboards once rollout plan is delivered. +- Replace mock digests with production pins and add env/schema knobs for VEX Lens once published. +- Add Grafana panels for recompute throughput and mapping failure rate after metrics are exposed. diff --git a/docs/runbooks/vuln-ops.md b/docs/runbooks/vuln-ops.md index af2083893..131761021 100644 --- a/docs/runbooks/vuln-ops.md +++ b/docs/runbooks/vuln-ops.md @@ -1,22 +1,40 @@ -# Vuln Ops Runbook (Md.XI draft) +# Vuln / Findings Ops Runbook (dev-mock ready) -> Status: DRAFT — pending policy overlay outputs and Ops scenarios. Keep TODO. +Status: DRAFT (2025-12-06 UTC). Safe for dev/mock exercises; production steps need final digests and schema from DEPLOY-VULN-29-001. ## Scope -- Operational responses: projector lag, resolver storms, export failures, policy activation steps. +- Findings Ledger + projector + Vuln Explorer API deployment/rollback, plus common incident drills (lag, storms, export failures). -## Dependencies -- Policy overlay outputs; GRAP0101 identifiers; export bundle spec. +## Pre-flight (dev vs. prod) +1) Release manifest guard + - Dev/mock: `python ops/devops/release/check_release_manifest.py deploy/releases/2025.09-mock-dev.yaml --downloads deploy/downloads/manifest.json` + - Prod: rerun against `deploy/releases/2025.09-stable.yaml` once ledger/api digests land. +2) Render plan + - Helm (mock overlay): `helm template vuln-mock ./deploy/helm/stellaops -f deploy/helm/stellaops/values-mock.yaml --debug --validate > /tmp/vuln-mock.yaml` + - Compose (dev with overlay): `USE_MOCK=1 deploy/compose/scripts/quickstart.sh env/dev.env.example && docker compose --env-file env/dev.env.example -f docker-compose.dev.yaml -f docker-compose.mock.yaml config > /tmp/vuln-compose.yaml` +3) Backups (prod only) + - Postgres dump for Findings Ledger DB; Mongo dump if projector uses Mongo cache; copy object-store buckets tied to projector anchors. -## Outline -- Projector lag: detection, remediation, replay steps. -- Resolver storms: rate limits, backpressure, queue drains. -- Export failures: bundle retry, manifest verification, hash checks. -- Policy activation: rollout checklist and rollback. +## Deploy (mock path) +- Helm apply (dev): `helm upgrade --install stellaops ./deploy/helm/stellaops -f deploy/helm/stellaops/values-mock.yaml --atomic --timeout 10m`. +- Compose: quickstart already starts ledger + vuln API with mock pins; validate health at `https://localhost:8443/swagger` (dev certs). -### Hash Capture Checklist (when scenarios scripted) -- `assets/vuln-explorer/runbook-projector-lag.md` -- `assets/vuln-explorer/runbook-resolver-storm.json` -- `assets/vuln-explorer/runbook-export-failure.json` -- `assets/vuln-explorer/runbook-policy-activation.md` -_Last updated: 2025-12-05 (UTC)_ +## Incident drills +- Projector lag: scale projector worker up (`kubectl scale deploy/findings-ledger -n stellaops --replicas=2`) then back down; monitor queue length (metric hook pending). +- Resolver storms: temporarily set `ASPNETCORE_THREADPOOL_MINTHREADS` higher or scale API horizontally; in compose, use `docker compose restart vuln-explorer-api` after bumping `VULNEXPLORER__MAX_CONCURRENCY` env once schema lands. +- Export failures: re-run export job after verifying hashes in `deploy/releases/*`; mock path skips signing but still exercises checksum validation via `ops/devops/release/check_release_manifest.py`. + +## Rollback +- Helm: `helm rollback stellaops 1` to previous revision. +- Compose: `docker compose --env-file env/dev.env.example -f docker-compose.dev.yaml -f docker-compose.mock.yaml down`. + +## Evidence capture +- Keep `/tmp/vuln-mock.yaml`, `/tmp/vuln-compose.yaml`, and the release manifest used. +- `kubectl logs deployment/findings-ledger -n stellaops --since=30m > /tmp/ledger-logs.txt` +- DB snapshot checksums if taken; bundle into `vuln-evidence-$(date -u +%Y%m%dT%H%M%SZ).tar.gz`. + +## Open TODOs +- Replace mock digests with production pins; add concrete env knobs for projector and API when schemas publish. +- Hook Prometheus counters for projector lag and resolver storm dashboards once metrics are exported. + +_Last updated: 2025-12-06 (UTC)_ diff --git a/src/Scanner/__Tests/StellaOps.Scanner.Analyzers.Lang.Bun.Tests/Fixtures/lang/bun/multi-workspace/package.json b/src/Scanner/__Tests/StellaOps.Scanner.Analyzers.Lang.Bun.Tests/Fixtures/lang/bun/multi-workspace/package.json new file mode 100644 index 000000000..c75a64717 --- /dev/null +++ b/src/Scanner/__Tests/StellaOps.Scanner.Analyzers.Lang.Bun.Tests/Fixtures/lang/bun/multi-workspace/package.json @@ -0,0 +1,8 @@ +{ + "name": "multi-workspace-fixture", + "version": "1.0.0", + "workspaces": ["packages/*"], + "dependencies": { + "lodash": "^4.17.21" + } +}