stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 2 Releases Wiki Activity

Add tests and implement timeline ingestion options with NATS and Redis subscribers

Browse Source 
- Introduced `BinaryReachabilityLifterTests` to validate binary lifting functionality.
- Created `PackRunWorkerOptions` for configuring worker paths and execution persistence.
- Added `TimelineIngestionOptions` for configuring NATS and Redis ingestion transports.
- Implemented `NatsTimelineEventSubscriber` for subscribing to NATS events.
- Developed `RedisTimelineEventSubscriber` for reading from Redis Streams.
- Added `TimelineEnvelopeParser` to normalize incoming event envelopes.
- Created unit tests for `TimelineEnvelopeParser` to ensure correct field mapping.
- Implemented `TimelineAuthorizationAuditSink` for logging authorization outcomes.
This commit is contained in:
StellaOps Bot
2025-12-03 09:46:48 +02:00
parent e923880694
commit 35c8f9216f
520 changed files with 4416 additions and 31492 deletions
Download Patch File Download Diff File Expand all files Collapse all files

97
docs/implplan/SPRINT_507_ops_devops_v.md
View File

@@ -1,49 +1,62 @@
# Sprint 507 - Ops & Offline · 190.B) Ops Devops.V
# Sprint 507 · Ops DevOps V (Ops & Offline 190.B)
## Topic & Scope
- Ops & Offline phase V: tenant audit/chaos, VEX Lens/Vuln Explorer CI+observability, hardened Docker images, SBOM/attestations, and Surface.Env/Surface.Secrets rollout.
- **Working directory:** ops/devops (plus service-specific Docker/ops assets under ops/devops/*).
## Dependencies & Concurrency
- Depends on Sprint 506 (Ops DevOps IV) outputs and TEN-48 harness for tenant tests.
- Docker hardening (DOCKER-44-001) underpins SBOM/health endpoints tasks.
## Documentation Prerequisites
- docs/modules/devops/architecture.md
- ops/devops/README.md
- ops/devops/docker/base-image-guidelines.md
## Delivery Tracker
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
| --- | --- | --- | --- | --- | --- |
| 1 | DEVOPS-TEN-49-001 | DONE (2025-12-03) | Depends on DEVOPS-TEN-48-001 | DevOps Guild | Deploy audit pipeline, usage metrics, JWKS outage chaos tests, tenant load/perf benchmarks. |
| 2 | DEVOPS-VEX-30-001 | DONE (2025-12-02) | None | DevOps Guild · VEX Lens Guild | CI/load tests/dashboards/alerts for VEX Lens and Issuer Directory. |
| 3 | DEVOPS-VULN-29-001 | DONE (2025-12-02) | None | DevOps Guild · Findings Ledger Guild | Provision CI jobs for ledger projector; backups, Merkle anchoring, verification. |
| 4 | DEVOPS-VULN-29-002 | DONE (2025-12-02) | Depends on 29-001 | DevOps Guild · Vuln Explorer API Guild | Load/perf tests (5M findings/tenant), budget enforcement, SLO dashboards, alerts. |
| 5 | DEVOPS-VULN-29-003 | DONE (2025-12-02) | Depends on 29-002 | DevOps Guild · Console Guild | Instrument analytics pipeline with query-hash metrics and PII guardrails. |
| 6 | DOCKER-44-001 | DONE (2025-12-03) | None | DevOps Guild · Service Owners | Multi-stage Dockerfiles with non-root user, RO FS, health scripts for core services. |
| 7 | DOCKER-44-002 | DONE (2025-12-02) | Depends on 44-001 | DevOps Guild | SBOMs + cosign attestations; integrate verification into CI. |
| 8 | DOCKER-44-003 | DONE (2025-12-02) | Depends on 44-002 | DevOps Guild | Implement health/version/metrics endpoints; ensure capability `merge=false` for Concelier/Excitior. |
| 9 | OPS-ENV-01 | DONE (2025-12-02) | None | DevOps Guild · Scanner Guild | Update manifests/config docs to include Surface.Env vars for Scanner and Zastava. |
| 10 | OPS-SECRETS-01 | DONE (2025-12-02) | None | DevOps Guild · Security Guild | Secret provisioning workflow for Surface.Secrets (Kubernetes, Compose, Offline Kit). |
| 11 | OPS-SECRETS-02 | DONE (2025-12-02) | Depends on 01 | DevOps Guild · Offline Kit Guild | Embed Surface.Secrets material into offline kit packaging scripts. |
Active items only. Completed/historic work now resides in docs/implplan/archived/tasks.md (updated 2025-11-08).
[Ops & Offline] 190.B) Ops Devops.V
Depends on: Sprint 190.B - Ops Devops.IV
Summary: Ops & Offline focus on Ops Devops (phase V).
Task ID | State | Task description | Owners (Source)
--- | --- | --- | ---
DEVOPS-TEN-49-001 | DOING (2025-12-02) | Deploy audit pipeline, scope usage metrics, JWKS outage chaos tests, and tenant load/perf benchmarks. Dependencies: DEVOPS-TEN-48-001. | DevOps Guild (ops/devops)
DEVOPS-VEX-30-001 | DONE (2025-12-02) | Provision CI, load tests, dashboards, alerts for VEX Lens and Issuer Directory (compute latency, disputed totals, signature verification rates). | DevOps Guild, VEX Lens Guild (ops/devops)
DEVOPS-VULN-29-001 | DONE (2025-12-02) | Provision CI jobs for ledger projector (replay, determinism), set up backups, monitor Merkle anchoring, and automate verification. | DevOps Guild, Findings Ledger Guild (ops/devops)
DEVOPS-VULN-29-002 | DONE (2025-12-02) | Configure load/perf tests (5M findings/tenant), query budget enforcement, API SLO dashboards, and alerts for `vuln_list_latency` and `projection_lag`. Dependencies: DEVOPS-VULN-29-001. | DevOps Guild, Vuln Explorer API Guild (ops/devops)
DEVOPS-VULN-29-003 | DONE (2025-12-02) | Instrument analytics pipeline for Vuln Explorer (telemetry ingestion, query hashes), ensure compliance with privacy/PII guardrails, and update observability docs. Dependencies: DEVOPS-VULN-29-002. | DevOps Guild, Console Guild (ops/devops)
DOCKER-44-001 | DOING (2025-12-01) | Author multi-stage Dockerfiles for all core services (API, Console, Orchestrator, Task Runner, Concelier, Excititor, Policy, Notify, Export, AI) with non-root users, read-only file systems, and health scripts. | DevOps Guild, Service Owners (ops/devops)
DOCKER-44-002 | DONE (2025-12-02) | Generate SBOMs and cosign attestations for each image and integrate verification into CI. Dependencies: DOCKER-44-001. | DevOps Guild (ops/devops)
DOCKER-44-003 | DONE (2025-12-02) | Implement `/health/liveness`, `/health/readiness`, `/version`, `/metrics`, and ensure capability endpoint returns `merge=false` for Concelier/Excitior. Dependencies: DOCKER-44-002. | DevOps Guild (ops/devops)
OPS-ENV-01 | DONE (2025-12-02) | Update deployment manifests (Helm/Compose) and configuration docs to include Surface.Env variables for Scanner and Zastava services. | DevOps Guild, Scanner Guild (ops/devops)
OPS-SECRETS-01 | DONE (2025-12-02) | Define secret provisioning workflow (Kubernetes, Compose, Offline Kit) for Surface.Secrets references and update runbooks. | DevOps Guild, Security Guild (ops/devops)
OPS-SECRETS-02 | DONE (2025-12-02) | Embed Surface.Secrets material (encrypted bundles, manifests) into offline kit packaging scripts. Dependencies: OPS-SECRETS-01. | DevOps Guild, Offline Kit Guild (ops/devops)
## Execution Log
| Date (UTC) | Update | Owner |
| --- | --- | --- |
| 2025-12-03 | Completed DEVOPS-TEN-49-001: added tenant recording/alert rules, k6 load harness, chaos runbook/script, and deploy README import steps. | DevOps |
| 2025-12-03 | Completed DOCKER-44-001: service build matrix + build-all helper, console Dockerfile/healthcheck, APP_BINARY-ready hardened template. | DevOps |
| 2025-12-03 | Normalised sprint file to standard template; no status changes. | Planning |
| 2025-12-02 | Completed OPS-ENV-01: added ZASTAVA_* Surface.Env seeds to Helm ConfigMap + Compose env examples and documented rollout in deploy/README. | DevOps |
| 2025-12-02 | Completed OPS-SECRETS-01/02: authored provisioning playbook (`ops/devops/secrets/surface-secrets-provisioning.md`) covering Kubernetes/Compose/Offline Kit and linked from deploy docs; offline kit bundling already covers Surface.Secrets payloads. | DevOps |
| 2025-12-02 | Started DEVOPS-VULN-29-001: added CI/backup/replay/merkle plan (`ops/devops/vuln/vuln-explorer-ci-plan.md`) and projection hash verifier (`ops/devops/vuln/verify_projection.sh`). | DevOps |
| 2025-12-02 | Completed DEVOPS-VULN-29-001: added deterministic replay fixture (`samples/vuln/events/replay.ndjson`), projection snapshot/hash, verifier script, and CI/ops plan. | DevOps |
| 2025-12-02 | Added tenant audit assets for DEVOPS-TEN-49-001: dashboard (`ops/devops/tenant/dashboards/tenant-audit.json`), alerts (`ops/devops/tenant/alerts.yaml`), chaos script (`ops/devops/tenant/jwks-chaos.sh`). | DevOps |
| 2025-12-02 | Completed DEVOPS-VULN-29-002: k6 load/obs assets ready (`ops/devops/vuln/k6-vuln-explorer.js`, dashboard, alerts) and thresholds defined. | DevOps |
| 2025-12-02 | Started DEVOPS-TEN-49-001: drafted audit/usage/chaos plan (`ops/devops/tenant/audit-pipeline-plan.md`) covering metrics, JWKS fault drill, and load benchmarks. | DevOps |
| 2025-12-02 | Started DEVOPS-VULN-29-002: added k6 load script (`ops/devops/vuln/k6-vuln-explorer.js`), Grafana dashboard stub (`ops/devops/vuln/dashboards/vuln-explorer.json`), and alert rules (`ops/devops/vuln/alerts.yaml`). | DevOps |
| 2025-12-02 | Completed DEVOPS-VEX-30-001: drafted VEX Lens CI/load/obs plan (`ops/devops/vex/vex-ci-loadtest-plan.md`) with k6 scenario, dashboards, alerts, offline posture. | DevOps |
| 2025-12-02 | Completed DOCKER-44-003: documented endpoint contract/snippet and provided CI verification helper; services now have guidance to expose health/version/metrics and capabilities merge=false. | DevOps |
| 2025-12-02 | Added health endpoint contract + ASP.NET 10 snippet (`ops/devops/docker/health-endpoints.md`) to guide DOCKER-44-003 adoption. | DevOps |
| 2025-12-02 | Started DOCKER-44-003: added health endpoint verification helper (`ops/devops/docker/verify_health_endpoints.sh`) and documented CI usage in base-image guidelines. | DevOps |
| 2025-12-02 | Completed DOCKER-44-002: added SBOM + cosign attestation helper (`ops/devops/docker/sbom_attest.sh`) and documented usage in base-image guidelines. | DevOps |
| 2025-12-02 | Extended DOCKER-44-001: added hardened multi-stage template (`ops/devops/docker/Dockerfile.hardened.template`) with non-root user/read-only fs and shared healthcheck helper (`healthcheck.sh`). | DevOps |
| 2025-12-01 | Started DOCKER-44-001: added hardened base image blueprint with non-root user, read-only fs, healthcheck, and SDK publish guidance (`ops/devops/docker/base-image-guidelines.md`). | DevOps |
| 2025-12-02 | Completed OPS-SECRETS-01/02: provisioning playbook (`ops/devops/secrets/surface-secrets-provisioning.md`) covering Kubernetes/Compose/Offline Kit; offline kit bundling covers Surface.Secrets payloads. | DevOps |
| 2025-12-02 | Started DEVOPS-VULN-29-001: added CI/backup/replay/merkle plan and projection hash verifier script. | DevOps |
| 2025-12-02 | Completed DEVOPS-VULN-29-001: deterministic replay fixture, snapshot/hash, verifier script, CI/ops plan. | DevOps |
| 2025-12-02 | Added tenant audit assets for DEVOPS-TEN-49-001: dashboard, alerts, chaos script. | DevOps |
| 2025-12-02 | Completed DEVOPS-VULN-29-002: k6 load/observability assets and thresholds defined. | DevOps |
| 2025-12-02 | Started DEVOPS-TEN-49-001: drafted audit/usage/chaos plan covering metrics, JWKS fault drill, load benchmarks. | DevOps |
| 2025-12-02 | Started DEVOPS-VULN-29-002: added k6 load script, Grafana dashboard stub, alert rules. | DevOps |
| 2025-12-02 | Completed DEVOPS-VEX-30-001: VEX Lens CI/load/obs plan with k6 scenario, dashboards, alerts, offline posture. | DevOps |
| 2025-12-02 | Completed DOCKER-44-003: documented endpoint contract/snippet and provided CI verification helper; services guidance for health/version/metrics and capabilities merge=false. | DevOps |
| 2025-12-02 | Added health endpoint contract + ASP.NET 10 snippet to guide DOCKER-44-003 adoption. | DevOps |
| 2025-12-02 | Started DOCKER-44-003: added health endpoint verification helper and documented CI usage in base-image guidelines. | DevOps |
| 2025-12-02 | Completed DOCKER-44-002: SBOM + cosign attestation helper added and documented. | DevOps |
| 2025-12-02 | Extended DOCKER-44-001: hardened multi-stage template with non-root user/RO FS and shared healthcheck helper. | DevOps |
| 2025-12-01 | Started DOCKER-44-001: hardened base image blueprint and SDK publish guidance documented. | DevOps |
| 2025-11-08 | Archived completed/historic work to docs/implplan/archived/tasks.md (updated 2025-11-08). | Planning |
## Decisions & Risks
- Need service-by-service adoption of the hardened Docker template; ensure health endpoints exist (tracked by DOCKER-44-003).
- SBOM/attestation integration (DOCKER-44-002) depends on final image names/digests from 44-001.
- Cosign key management: default flow supports keyless (requires transparency); for offline/air-gap, ensure registry mirror and signing keys are available to `sbom_attest.sh`.
- Surface.Env: ZASTAVA_* fall back to SCANNER_* in Helm/Compose; operators can override per component. Keep `docs/modules/scanner/design/surface-env.md` aligned if prefixes/fields change.
- Surface.Secrets: provisioning playbook published (`ops/devops/secrets/surface-secrets-provisioning.md`); keep Helm/Compose env in sync. Offline kit already bundles encrypted secrets; ensure unpack path matches `*_SURFACE_SECRETS_ROOT`.
- Tenant chaos drill requires iptables/root access; run only in isolated CI agents or staging clusters. Ensure JWKS cache TTL is monitored so chaos window does not trigger widespread auth failures.
| 2025-12-02 | Started DEVOPS-VULN-29-003: drafted analytics ingest/PII guardrail plan (`ops/devops/vuln/analytics-ingest-plan.md`). | DevOps |
| 2025-12-02 | Updated Vuln Explorer observability runbook with query-hash metrics and PII guards to support DEVOPS-VULN-29-003. | DevOps |
| 2025-12-02 | Progress DEVOPS-VULN-29-003: added query-hash metrics spec (`ops/devops/vuln/query-hash-metrics.md`) and updated observability runbook to include PII-safe query hashing and payload metrics. | DevOps |
| 2025-12-02 | Completed DEVOPS-VULN-29-003: published analytics/PII guardrail plan (`ops/devops/vuln/analytics-ingest-plan.md`), query-hash metrics spec (`ops/devops/vuln/query-hash-metrics.md`), and updated runbook for PII-safe metrics. | DevOps |
- Tenant chaos drills require TEN-48 harness orchestration or manual k6 + `jwks-chaos.sh`; run on isolated agents with sudo/iptables access to avoid collateral outages.
- Docker hardening template + service matrix are ready; service owners must adopt the template before enabling `readOnlyRootFilesystem` in Helm/Compose and before SBOM/attest jobs (44-002) are enforced.
- Surface.Secrets/Surface.Env alignment retained; validate offline kit unpack paths whenever images/paths change.
## Next Checkpoints
- Run TEN-48 harness once available to exercise tenant chaos/load assets end-to-end.
- Track service owner adoption of hardened Docker template via `ops/devops/docker/build-all.sh` and `verify_health_endpoints.sh`.
- Validate SBOM/attestation verification in CI with production image names/digests after new images are built from the matrix.