acbb0ff637
- Implemented shared trace ID generation utility for Risk and Vulnerability clients, ensuring consistent trace headers across API calls. - Updated RiskHttpClient and VulnerabilityHttpClient to utilize the new trace ID generation method. - Added validation for artifact metadata in PackRun endpoints, ensuring all artifacts include a digest and positive size. - Enhanced logging payloads in PackRun to include artifact digests and sizes. - Created a utility for generating trace IDs, preferring crypto.randomUUID when available, with a fallback to a ULID-style string. - Added unit tests to verify the presence of trace IDs in HTTP requests for VulnerabilityHttpClient. - Documented query-hash metrics for Vuln Explorer, detailing hashing rules and logging filters to ensure compliance with privacy standards. - Consolidated findings from late-November reviews into a comprehensive advisory for Scanner and SBOM/VEX areas, outlining remediation tracks and gaps.
7.7 KiB
7.7 KiB
Sprint 507 - Ops & Offline · 190.B) Ops Devops.V
Active items only. Completed/historic work now resides in docs/implplan/archived/tasks.md (updated 2025-11-08). [Ops & Offline] 190.B) Ops Devops.V Depends on: Sprint 190.B - Ops Devops.IV Summary: Ops & Offline focus on Ops Devops (phase V).
| Task ID | State | Task description | Owners (Source) |
|---|---|---|---|
| DEVOPS-TEN-49-001 | DOING (2025-12-02) | Deploy audit pipeline, scope usage metrics, JWKS outage chaos tests, and tenant load/perf benchmarks. Dependencies: DEVOPS-TEN-48-001. | DevOps Guild (ops/devops) |
| DEVOPS-VEX-30-001 | DONE (2025-12-02) | Provision CI, load tests, dashboards, alerts for VEX Lens and Issuer Directory (compute latency, disputed totals, signature verification rates). | DevOps Guild, VEX Lens Guild (ops/devops) |
| DEVOPS-VULN-29-001 | DONE (2025-12-02) | Provision CI jobs for ledger projector (replay, determinism), set up backups, monitor Merkle anchoring, and automate verification. | DevOps Guild, Findings Ledger Guild (ops/devops) |
| DEVOPS-VULN-29-002 | DONE (2025-12-02) | Configure load/perf tests (5M findings/tenant), query budget enforcement, API SLO dashboards, and alerts for vuln_list_latency and projection_lag. Dependencies: DEVOPS-VULN-29-001. |
DevOps Guild, Vuln Explorer API Guild (ops/devops) |
| DEVOPS-VULN-29-003 | DONE (2025-12-02) | Instrument analytics pipeline for Vuln Explorer (telemetry ingestion, query hashes), ensure compliance with privacy/PII guardrails, and update observability docs. Dependencies: DEVOPS-VULN-29-002. | DevOps Guild, Console Guild (ops/devops) |
| DOCKER-44-001 | DOING (2025-12-01) | Author multi-stage Dockerfiles for all core services (API, Console, Orchestrator, Task Runner, Concelier, Excititor, Policy, Notify, Export, AI) with non-root users, read-only file systems, and health scripts. | DevOps Guild, Service Owners (ops/devops) |
| DOCKER-44-002 | DONE (2025-12-02) | Generate SBOMs and cosign attestations for each image and integrate verification into CI. Dependencies: DOCKER-44-001. | DevOps Guild (ops/devops) |
| DOCKER-44-003 | DONE (2025-12-02) | Implement /health/liveness, /health/readiness, /version, /metrics, and ensure capability endpoint returns merge=false for Concelier/Excitior. Dependencies: DOCKER-44-002. |
DevOps Guild (ops/devops) |
| OPS-ENV-01 | DONE (2025-12-02) | Update deployment manifests (Helm/Compose) and configuration docs to include Surface.Env variables for Scanner and Zastava services. | DevOps Guild, Scanner Guild (ops/devops) |
| OPS-SECRETS-01 | DONE (2025-12-02) | Define secret provisioning workflow (Kubernetes, Compose, Offline Kit) for Surface.Secrets references and update runbooks. | DevOps Guild, Security Guild (ops/devops) |
| OPS-SECRETS-02 | DONE (2025-12-02) | Embed Surface.Secrets material (encrypted bundles, manifests) into offline kit packaging scripts. Dependencies: OPS-SECRETS-01. | DevOps Guild, Offline Kit Guild (ops/devops) |
Execution Log
| Date (UTC) | Update | Owner |
|---|---|---|
| 2025-12-02 | Completed OPS-ENV-01: added ZASTAVA_* Surface.Env seeds to Helm ConfigMap + Compose env examples and documented rollout in deploy/README. | DevOps |
| 2025-12-02 | Completed OPS-SECRETS-01/02: authored provisioning playbook (ops/devops/secrets/surface-secrets-provisioning.md) covering Kubernetes/Compose/Offline Kit and linked from deploy docs; offline kit bundling already covers Surface.Secrets payloads. |
DevOps |
| 2025-12-02 | Started DEVOPS-VULN-29-001: added CI/backup/replay/merkle plan (ops/devops/vuln/vuln-explorer-ci-plan.md) and projection hash verifier (ops/devops/vuln/verify_projection.sh). |
DevOps |
| 2025-12-02 | Completed DEVOPS-VULN-29-001: added deterministic replay fixture (samples/vuln/events/replay.ndjson), projection snapshot/hash, verifier script, and CI/ops plan. |
DevOps |
| 2025-12-02 | Added tenant audit assets for DEVOPS-TEN-49-001: dashboard (ops/devops/tenant/dashboards/tenant-audit.json), alerts (ops/devops/tenant/alerts.yaml), chaos script (ops/devops/tenant/jwks-chaos.sh). |
DevOps |
| 2025-12-02 | Completed DEVOPS-VULN-29-002: k6 load/obs assets ready (ops/devops/vuln/k6-vuln-explorer.js, dashboard, alerts) and thresholds defined. |
DevOps |
| 2025-12-02 | Started DEVOPS-TEN-49-001: drafted audit/usage/chaos plan (ops/devops/tenant/audit-pipeline-plan.md) covering metrics, JWKS fault drill, and load benchmarks. |
DevOps |
| 2025-12-02 | Started DEVOPS-VULN-29-002: added k6 load script (ops/devops/vuln/k6-vuln-explorer.js), Grafana dashboard stub (ops/devops/vuln/dashboards/vuln-explorer.json), and alert rules (ops/devops/vuln/alerts.yaml). |
DevOps |
| 2025-12-02 | Completed DEVOPS-VEX-30-001: drafted VEX Lens CI/load/obs plan (ops/devops/vex/vex-ci-loadtest-plan.md) with k6 scenario, dashboards, alerts, offline posture. |
DevOps |
| 2025-12-02 | Completed DOCKER-44-003: documented endpoint contract/snippet and provided CI verification helper; services now have guidance to expose health/version/metrics and capabilities merge=false. | DevOps |
| 2025-12-02 | Added health endpoint contract + ASP.NET 10 snippet (ops/devops/docker/health-endpoints.md) to guide DOCKER-44-003 adoption. |
DevOps |
| 2025-12-02 | Started DOCKER-44-003: added health endpoint verification helper (ops/devops/docker/verify_health_endpoints.sh) and documented CI usage in base-image guidelines. |
DevOps |
| 2025-12-02 | Completed DOCKER-44-002: added SBOM + cosign attestation helper (ops/devops/docker/sbom_attest.sh) and documented usage in base-image guidelines. |
DevOps |
| 2025-12-02 | Extended DOCKER-44-001: added hardened multi-stage template (ops/devops/docker/Dockerfile.hardened.template) with non-root user/read-only fs and shared healthcheck helper (healthcheck.sh). |
DevOps |
| 2025-12-01 | Started DOCKER-44-001: added hardened base image blueprint with non-root user, read-only fs, healthcheck, and SDK publish guidance (ops/devops/docker/base-image-guidelines.md). |
DevOps |
| 2025-11-08 | Archived completed/historic work to docs/implplan/archived/tasks.md (updated 2025-11-08). | Planning |
Decisions & Risks
- Need service-by-service adoption of the hardened Docker template; ensure health endpoints exist (tracked by DOCKER-44-003).
- SBOM/attestation integration (DOCKER-44-002) depends on final image names/digests from 44-001.
- Cosign key management: default flow supports keyless (requires transparency); for offline/air-gap, ensure registry mirror and signing keys are available to
sbom_attest.sh. - Surface.Env: ZASTAVA_* fall back to SCANNER_* in Helm/Compose; operators can override per component. Keep
docs/modules/scanner/design/surface-env.mdaligned if prefixes/fields change. - Surface.Secrets: provisioning playbook published (
ops/devops/secrets/surface-secrets-provisioning.md); keep Helm/Compose env in sync. Offline kit already bundles encrypted secrets; ensure unpack path matches*_SURFACE_SECRETS_ROOT. - Tenant chaos drill requires iptables/root access; run only in isolated CI agents or staging clusters. Ensure JWKS cache TTL is monitored so chaos window does not trigger widespread auth failures.
| 2025-12-02 | Started DEVOPS-VULN-29-003: drafted analytics ingest/PII guardrail plan (
ops/devops/vuln/analytics-ingest-plan.md). | DevOps | | 2025-12-02 | Updated Vuln Explorer observability runbook with query-hash metrics and PII guards to support DEVOPS-VULN-29-003. | DevOps | | 2025-12-02 | Progress DEVOPS-VULN-29-003: added query-hash metrics spec (ops/devops/vuln/query-hash-metrics.md) and updated observability runbook to include PII-safe query hashing and payload metrics. | DevOps | | 2025-12-02 | Completed DEVOPS-VULN-29-003: published analytics/PII guardrail plan (ops/devops/vuln/analytics-ingest-plan.md), query-hash metrics spec (ops/devops/vuln/query-hash-metrics.md), and updated runbook for PII-safe metrics. | DevOps |