ops: add mock-ready VEX/Vuln runbooks

docs/implplan/SPRINT_0502_0001_0001_ops_deployment_ii.md

@@ -33,6 +33,7 @@
 ## Execution Log
 | Date (UTC) | Update | Owner |
 | --- | --- | --- |
+| 2025-12-06 | Added mock-ready VEX/Vuln ops runbooks (`docs/runbooks/vex-ops.md`, `docs/runbooks/vuln-ops.md`); tasks remain DOING until production digests/schemas land. | Deployment Guild |
 | 2025-12-06 | Drafted policy incident runbook (`docs/runbooks/policy-incident.md`); set DEPLOY-POLICY-27-002 to DOING pending policy overlay/digests. | Deployment Guild |
 | 2025-12-06 | Header normalised to standard template; no content/status changes. | Project Mgmt |
 | 2025-12-06 | Seeded mock dev release manifest (`deploy/releases/2025.09-mock-dev.yaml`) covering VEX Lens and Findings/Vuln stacks; tasks moved to DOING (dev-mock) for development packaging. Production release still awaits real digests. | Deployment Guild |

docs/implplan/tasks-all.md

@@ -540,10 +540,10 @@
 | DEPLOY-PACKS-43-001 | DOING (dev-mock 2025-12-06) | 2025-12-06 | SPRINT_0501_0001_0001_ops_deployment_i | Deployment Guild · Task Runner Guild | ops/deployment | Ship remote Task Runner worker profiles, object storage bootstrap, approval workflow integration, and Offline Kit packaging instructions. Dependencies: DEPLOY-PACKS-42-001. | Needs #7 artifacts | AGDP0101 |
 | DEPLOY-POLICY-27-001 | DOING (dev-mock 2025-12-06) | 2025-12-05 | SPRINT_0501_0001_0001_ops_deployment_i | Deployment Guild · Policy Registry Guild | ops/deployment | Produce Helm/Compose overlays for Policy Registry + simulation workers (migrations, buckets, signing keys, tenancy defaults). | WEPO0101 | DVPL0105 |
 | DEPLOY-POLICY-27-002 | DOING (draft 2025-12-06) | 2025-12-06 | SPRINT_0502_0001_0001_ops_deployment_ii | Deployment Guild · Policy Guild | ops/deployment | Drafted `docs/runbooks/policy-incident.md` (publish/promote, freeze, evidence); awaiting policy overlay schema/digests from DEPLOY-POLICY-27-001. | DEPLOY-POLICY-27-001 | DVPL0105 |
-| DEPLOY-VEX-30-001 | DOING (dev-mock 2025-12-06) | 2025-12-06 | SPRINT_0502_0001_0001_ops_deployment_ii | Deployment + VEX Lens Guild | ops/deployment | Provide Helm/Compose overlays, scaling defaults, and offline kit instructions for VEX Lens service. | Wait for CCWO0101 schema | DVPL0101 |
+| DEPLOY-VEX-30-001 | DOING (dev-mock 2025-12-06) | 2025-12-06 | SPRINT_0502_0001_0001_ops_deployment_ii | Deployment + VEX Lens Guild | ops/deployment | Mock-ready runbook added (`docs/runbooks/vex-ops.md`); awaiting schema/digests for final Helm/Compose overlays. | Wait for CCWO0101 schema | DVPL0101 |
-| DEPLOY-VEX-30-002 | DOING (dev-mock 2025-12-06) | 2025-12-06 | SPRINT_0502_0001_0001_ops_deployment_ii | Deployment Guild | ops/deployment | Package Issuer Directory deployment manifests, backups, and security hardening guidance. Dependencies: DEPLOY-VEX-30-001. | Depends on #5 | DVPL0101 |
+| DEPLOY-VEX-30-002 | DOING (dev-mock 2025-12-06) | 2025-12-06 | SPRINT_0502_0001_0001_ops_deployment_ii | Deployment Guild | ops/deployment | Issuer Directory guidance covered in `docs/runbooks/vex-ops.md`; finalize once DEPLOY-VEX-30-001 pins production values. | Depends on #5 | DVPL0101 |
-| DEPLOY-VULN-29-001 | DOING (dev-mock 2025-12-06) | 2025-12-06 | SPRINT_0502_0001_0001_ops_deployment_ii | Deployment + Vuln Guild | ops/deployment | Produce Helm/Compose overlays for Findings Ledger + projector, including DB migrations, Merkle anchor jobs, and scaling guidance. | Needs CCWO0101 | DVPL0101 |
+| DEPLOY-VULN-29-001 | DOING (dev-mock 2025-12-06) | 2025-12-06 | SPRINT_0502_0001_0001_ops_deployment_ii | Deployment + Vuln Guild | ops/deployment | Mock-ready runbook added (`docs/runbooks/vuln-ops.md`); production overlays pending schema/digests. | Needs CCWO0101 | DVPL0101 |
-| DEPLOY-VULN-29-002 | DOING (dev-mock 2025-12-06) | 2025-12-06 | SPRINT_0502_0001_0001_ops_deployment_ii | Deployment Guild | ops/deployment | Package `stella-vuln-explorer-api` deployment manifests, health checks, autoscaling policies, and offline kit instructions with signed images. Dependencies: DEPLOY-VULN-29-001. | Depends on #7 | DVPL0101 |
+| DEPLOY-VULN-29-002 | DOING (dev-mock 2025-12-06) | 2025-12-06 | SPRINT_0502_0001_0001_ops_deployment_ii | Deployment Guild | ops/deployment | Vuln Explorer API steps captured in `docs/runbooks/vuln-ops.md`; finalize with real pins after DEPLOY-VULN-29-001. | Depends on #7 | DVPL0101 |
 | DETER-186-008 | TODO |  | SPRINT_0186_0001_0001_record_deterministic_execution | Scanner Guild | `src/Scanner/StellaOps.Scanner.WebService`, `src/Scanner/StellaOps.Scanner.Worker` | Wait for RLRC0101 fixture | Wait for RLRC0101 fixture | SCDT0101 |
 | DETER-186-009 | TODO |  | SPRINT_0186_0001_0001_record_deterministic_execution | Scanner Guild · QA Guild | `src/Scanner/StellaOps.Scanner.Replay`, `src/Scanner/__Tests` | Depends on #1 | Depends on #1 | SCDT0101 |
 | DETER-186-010 | TODO |  | SPRINT_0186_0001_0001_record_deterministic_execution | Scanner Guild · Export Center Guild | `src/Scanner/StellaOps.Scanner.WebService`, `docs/modules/scanner/operations/release.md` | Depends on #2 | Depends on #2 | SCDT0101 |
@@ -2754,10 +2754,10 @@
 | DEPLOY-PACKS-43-001 | DOING (dev-mock 2025-12-06) | 2025-12-06 | SPRINT_0501_0001_0001_ops_deployment_i | Deployment Guild · Task Runner Guild | ops/deployment | Ship remote Task Runner worker profiles, object storage bootstrap, approval workflow integration, and Offline Kit packaging instructions. Dependencies: DEPLOY-PACKS-42-001. | Needs #7 artifacts | AGDP0101 |
 | DEPLOY-POLICY-27-001 | DOING (dev-mock 2025-12-06) | 2025-12-05 | SPRINT_0501_0001_0001_ops_deployment_i | Deployment Guild · Policy Registry Guild | ops/deployment | Produce Helm/Compose overlays for Policy Registry + simulation workers, including Mongo migrations, object storage buckets, signing key secrets, and tenancy defaults. | Needs registry schema + secrets | AGDP0101 |
 | DEPLOY-POLICY-27-002 | DOING (draft 2025-12-06) | 2025-12-06 | SPRINT_0502_0001_0001_ops_deployment_ii | Deployment Guild · Policy Guild | ops/deployment | Drafted `docs/runbooks/policy-incident.md` (publish/promote, freeze, evidence); finalize once DEPLOY-POLICY-27-001 ships schema/digests. | Depends on 27-001 | AGDP0101 |
-| DEPLOY-VEX-30-001 | DOING (dev-mock 2025-12-06) | 2025-12-06 | SPRINT_0502_0001_0001_ops_deployment_ii | Deployment + VEX Lens Guild | ops/deployment | Provide Helm/Compose overlays, scaling defaults, and offline kit instructions for VEX Lens service. | Wait for CCWO0101 schema | DVPL0101 |
+| DEPLOY-VEX-30-001 | DOING (dev-mock 2025-12-06) | 2025-12-06 | SPRINT_0502_0001_0001_ops_deployment_ii | Deployment + VEX Lens Guild | ops/deployment | Mock-ready runbook added (`docs/runbooks/vex-ops.md`); awaiting schema/digests for final Helm/Compose overlays. | Wait for CCWO0101 schema | DVPL0101 |
-| DEPLOY-VEX-30-002 | DOING (dev-mock 2025-12-06) | 2025-12-06 | SPRINT_0502_0001_0001_ops_deployment_ii | Deployment Guild | ops/deployment | Package Issuer Directory deployment manifests, backups, and security hardening guidance. Dependencies: DEPLOY-VEX-30-001. | Depends on #5 | DVPL0101 |
+| DEPLOY-VEX-30-002 | DOING (dev-mock 2025-12-06) | 2025-12-06 | SPRINT_0502_0001_0001_ops_deployment_ii | Deployment Guild | ops/deployment | Issuer Directory guidance covered in `docs/runbooks/vex-ops.md`; finalize once DEPLOY-VEX-30-001 pins production values. | Depends on #5 | DVPL0101 |
-| DEPLOY-VULN-29-001 | DOING (dev-mock 2025-12-06) | 2025-12-06 | SPRINT_0502_0001_0001_ops_deployment_ii | Deployment + Vuln Guild | ops/deployment | Produce Helm/Compose overlays for Findings Ledger + projector, including DB migrations, Merkle anchor jobs, and scaling guidance. | Needs CCWO0101 | DVPL0101 |
+| DEPLOY-VULN-29-001 | DOING (dev-mock 2025-12-06) | 2025-12-06 | SPRINT_0502_0001_0001_ops_deployment_ii | Deployment + Vuln Guild | ops/deployment | Mock-ready runbook added (`docs/runbooks/vuln-ops.md`); production overlays pending schema/digests. | Needs CCWO0101 | DVPL0101 |
-| DEPLOY-VULN-29-002 | DOING (dev-mock 2025-12-06) | 2025-12-06 | SPRINT_0502_0001_0001_ops_deployment_ii | Deployment Guild | ops/deployment | Package `stella-vuln-explorer-api` deployment manifests, health checks, autoscaling policies, and offline kit instructions with signed images. Dependencies: DEPLOY-VULN-29-001. | Depends on #7 | DVPL0101 |
+| DEPLOY-VULN-29-002 | DOING (dev-mock 2025-12-06) | 2025-12-06 | SPRINT_0502_0001_0001_ops_deployment_ii | Deployment Guild | ops/deployment | Vuln Explorer API steps captured in `docs/runbooks/vuln-ops.md`; finalize with real pins after DEPLOY-VULN-29-001. | Depends on #7 | DVPL0101 |
 | DETER-186-008 | TODO |  | SPRINT_0186_0001_0001_record_deterministic_execution | Scanner Guild | `src/Scanner/StellaOps.Scanner.WebService`, `src/Scanner/StellaOps.Scanner.Worker` | Wait for RLRC0101 fixture | Wait for RLRC0101 fixture | SCDT0101 |
 | DETER-186-009 | TODO |  | SPRINT_0186_0001_0001_record_deterministic_execution | Scanner Guild · QA Guild | `src/Scanner/StellaOps.Scanner.Replay`, `src/Scanner/__Tests` | Depends on #1 | Depends on #1 | SCDT0101 |
 | DETER-186-010 | TODO |  | SPRINT_0186_0001_0001_record_deterministic_execution | Scanner Guild · Export Center Guild | `src/Scanner/StellaOps.Scanner.WebService`, `docs/modules/scanner/operations/release.md` | Depends on #2 | Depends on #2 | SCDT0101 |

docs/runbooks/vex-ops.md

@@ -1,15 +1,35 @@
-# VEX Ops Runbook — Draft Skeleton (2025-12-05 UTC)
+# VEX Ops Runbook (dev-mock ready)
 
-Status: draft placeholder. Inputs pending: DevOps rollout plan for signatures/ops.
+Status: DRAFT (2025-12-06 UTC). Safe for dev/mock exercises; production rollouts wait on policy/VEX final digests.
 
-## Recompute Storms
+## Pre-flight (dev vs. prod)
-- Steps to mitigate; throttling knobs (to fill).
+1) Release manifest guard
+   - Dev/mock: `python ops/devops/release/check_release_manifest.py deploy/releases/2025.09-mock-dev.yaml --downloads deploy/downloads/manifest.json`
+   - Prod: rerun against `deploy/releases/2025.09-stable.yaml` once VEX digests land.
+2) Render plan
+   - Helm (mock overlay): `helm template vex-mock ./deploy/helm/stellaops -f deploy/helm/stellaops/values-mock.yaml --debug --validate > /tmp/vex-mock.yaml`
+   - Compose (dev with overlay): `USE_MOCK=1 deploy/compose/scripts/quickstart.sh env/dev.env.example && docker compose --env-file env/dev.env.example -f deploy/compose/docker-compose.dev.yaml -f deploy/compose/docker-compose.mock.yaml config > /tmp/vex-compose.yaml`
+3) Backups (when touching prod data) — not required for mock, but in prod take Mongo snapshots for issuer-directory and VEX state before rollout.
 
-## Mapping Failures
+## Deploy (mock path)
-- Triage steps; retry/backfill guidance.
+- Helm dry-run already covers structural checks. To apply in a dev cluster: `helm upgrade --install stellaops ./deploy/helm/stellaops -f deploy/helm/stellaops/values-mock.yaml --atomic --timeout 10m`.
+- Observe VEX Lens pod logs: `kubectl logs deploy/vex-lens -n stellaops --tail=200 -f`.
+- Issuer Directory seed: ensure `issuer-directory-config` ConfigMap includes `csaf-publishers.json`; mock overlay already mounts default seed.
 
-## Signature Errors
+## Rollback
-- Diagnosis workflow; key rotation checks.
+- Helm: `helm rollback stellaops 1` (choose previous revision). Mock overlay uses `stellaops.dev/mock: "true"` annotations; safe to tear down after tests.
+- Compose: `docker compose --env-file env/dev.env.example -f docker-compose.dev.yaml -f docker-compose.mock.yaml down`.
+
+## Troubleshooting
+- Recompute storms: throttle via `VEX_LENS__MAX_PARALLELISM` env (set in values once schema lands); for now scale deployment down to 1 replica to reduce concurrency.
+- Mapping failures: capture request/response with `kubectl logs ... --since=10m`; rerun after clearing queue.
+- Signature errors: confirm Authority token audience/issuer; mock overlay uses the same auth settings as dev compose.
+
+## Evidence capture
+- Save `/tmp/vex-mock.yaml` and `/tmp/vex-compose.yaml` with the manifest used.
+- `kubectl get deploy/pod,svc -n stellaops -l app=vex-lens -o yaml > /tmp/vex-live.yaml`.
+- Tarball: `tar -czf vex-evidence-$(date -u +%Y%m%dT%H%M%SZ).tar.gz /tmp/vex-*`.
 
 ## Open TODOs
-- Add concrete commands and dashboards once rollout plan is delivered.
+- Replace mock digests with production pins and add env/schema knobs for VEX Lens once published.
+- Add Grafana panels for recompute throughput and mapping failure rate after metrics are exposed.

docs/runbooks/vuln-ops.md

@@ -1,22 +1,40 @@
-# Vuln Ops Runbook (Md.XI draft)
+# Vuln / Findings Ops Runbook (dev-mock ready)
 
-> Status: DRAFT — pending policy overlay outputs and Ops scenarios. Keep TODO.
+Status: DRAFT (2025-12-06 UTC). Safe for dev/mock exercises; production steps need final digests and schema from DEPLOY-VULN-29-001.
 
 ## Scope
-- Operational responses: projector lag, resolver storms, export failures, policy activation steps.
+- Findings Ledger + projector + Vuln Explorer API deployment/rollback, plus common incident drills (lag, storms, export failures).
 
-## Dependencies
+## Pre-flight (dev vs. prod)
-- Policy overlay outputs; GRAP0101 identifiers; export bundle spec.
+1) Release manifest guard
+   - Dev/mock: `python ops/devops/release/check_release_manifest.py deploy/releases/2025.09-mock-dev.yaml --downloads deploy/downloads/manifest.json`
+   - Prod: rerun against `deploy/releases/2025.09-stable.yaml` once ledger/api digests land.
+2) Render plan
+   - Helm (mock overlay): `helm template vuln-mock ./deploy/helm/stellaops -f deploy/helm/stellaops/values-mock.yaml --debug --validate > /tmp/vuln-mock.yaml`
+   - Compose (dev with overlay): `USE_MOCK=1 deploy/compose/scripts/quickstart.sh env/dev.env.example && docker compose --env-file env/dev.env.example -f docker-compose.dev.yaml -f docker-compose.mock.yaml config > /tmp/vuln-compose.yaml`
+3) Backups (prod only)
+   - Postgres dump for Findings Ledger DB; Mongo dump if projector uses Mongo cache; copy object-store buckets tied to projector anchors.
 
-## Outline
+## Deploy (mock path)
-- Projector lag: detection, remediation, replay steps.
+- Helm apply (dev): `helm upgrade --install stellaops ./deploy/helm/stellaops -f deploy/helm/stellaops/values-mock.yaml --atomic --timeout 10m`.
-- Resolver storms: rate limits, backpressure, queue drains.
+- Compose: quickstart already starts ledger + vuln API with mock pins; validate health at `https://localhost:8443/swagger` (dev certs).
-- Export failures: bundle retry, manifest verification, hash checks.
-- Policy activation: rollout checklist and rollback.
 
-### Hash Capture Checklist (when scenarios scripted)
+## Incident drills
-- `assets/vuln-explorer/runbook-projector-lag.md`
+- Projector lag: scale projector worker up (`kubectl scale deploy/findings-ledger -n stellaops --replicas=2`) then back down; monitor queue length (metric hook pending).
-- `assets/vuln-explorer/runbook-resolver-storm.json`
+- Resolver storms: temporarily set `ASPNETCORE_THREADPOOL_MINTHREADS` higher or scale API horizontally; in compose, use `docker compose restart vuln-explorer-api` after bumping `VULNEXPLORER__MAX_CONCURRENCY` env once schema lands.
-- `assets/vuln-explorer/runbook-export-failure.json`
+- Export failures: re-run export job after verifying hashes in `deploy/releases/*`; mock path skips signing but still exercises checksum validation via `ops/devops/release/check_release_manifest.py`.
-- `assets/vuln-explorer/runbook-policy-activation.md`
+
-_Last updated: 2025-12-05 (UTC)_
+## Rollback
+- Helm: `helm rollback stellaops 1` to previous revision.
+- Compose: `docker compose --env-file env/dev.env.example -f docker-compose.dev.yaml -f docker-compose.mock.yaml down`.
+
+## Evidence capture
+- Keep `/tmp/vuln-mock.yaml`, `/tmp/vuln-compose.yaml`, and the release manifest used.
+- `kubectl logs deployment/findings-ledger -n stellaops --since=30m > /tmp/ledger-logs.txt`
+- DB snapshot checksums if taken; bundle into `vuln-evidence-$(date -u +%Y%m%dT%H%M%SZ).tar.gz`.
+
+## Open TODOs
+- Replace mock digests with production pins; add concrete env knobs for projector and API when schemas publish.
+- Hook Prometheus counters for projector lag and resolver storm dashboards once metrics are exported.
+
+_Last updated: 2025-12-06 (UTC)_

src/Scanner/__Tests/StellaOps.Scanner.Analyzers.Lang.Bun.Tests/Fixtures/lang/bun/multi-workspace/package.json

@@ -0,0 +1,8 @@
+{
+  "name": "multi-workspace-fixture",
+  "version": "1.0.0",
+  "workspaces": ["packages/*"],
+  "dependencies": {
+    "lodash": "^4.17.21"
+  }
+}