git.stella-ops.org/src/Zastava/StellaOps.Zastava.Webhook/IMPLEMENTATION_PLAN.md

Zastava Webhook · Wave 0 Implementation Notes

Authored 2025-10-19 by Zastava Webhook Guild.

ZASTAVA-WEBHOOK-12-101 — Admission Controller Host (TLS bootstrap + Authority auth)

Objectives

• Provide a deterministic, restart-safe .NET 10 host that exposes a Kubernetes ValidatingAdmissionWebhook endpoint.
• Load serving certificates at start-up only (per restart-time plug-in rule) and surface reload guidance via documentation rather than hot-reload.
• Authenticate outbound calls to Authority/Scanner using OpTok + DPoP as defined in docs/modules/zastava/ARCHITECTURE.md.

Plan

1. Project scaffolding
   • Create StellaOps.Zastava.Webhook project with minimal API pipeline (Program.cs, Startup equivalent via extension methods).
   • Reference shared helpers once ZASTAVA-CORE-12-201/202 land; temporarily stub interfaces behind IZastavaAdmissionRequest/IZastavaAdmissionResult.
2. TLS bootstrap
   • Support two certificate sources:
     1. Mounted secret path (/var/run/secrets/zastava-webhook/tls.{crt,key}) with optional CA bundle.
     2. CSR workflow: generate CSR + private key, submit to Kubernetes Certificates API when admission.tls.autoApprove enabled; persist signed cert/key to mounted emptyDir for reuse across replicas.
   • Validate cert/key pair on boot; abort start-up if invalid to preserve deterministic behavior.
   • Configure Kestrel for mutual TLS off (API Server already provides client auth) but enforce minimum TLS 1.3, strong cipher suite list, HTTP/2 disabled (K8s uses HTTP/1.1).
3. Authority auth
   • Bootstrap Authority client via shared runtime core (AddZastavaRuntimeCore + IZastavaAuthorityTokenProvider) so webhook reuses multitenant OpTok caching and guardrails.
   • Implement DPoP proof generator bound to webhook host keypair (prefer Ed25519) with configurable rotation period (default 24h, triggered at restart).
   • Add background health check verifying token freshness and surfacing metrics (zastava.authority_token_renew_failures_total).
4. Hosting concerns
   • Configure structured logging with correlation id from AdmissionReview UID.
   • Expose /healthz (reads cert expiry, Authority token status) and /metrics (Prometheus).
   • Add readiness gate that requires initial TLS and Authority bootstrap to succeed.

Deliverables

• Compilable host project with integration tests covering TLS load (mounted files + CSR mock) and Authority token acquisition.
• Documentation snippet for deploy charts describing secret/CSR wiring.

Open Questions

• Need confirmation from Core guild on DTO naming (AdmissionReviewEnvelope, AdmissionDecision) to avoid rework.
• Determine whether CSR auto-approval is acceptable for air-gapped clusters without Kubernetes cert-manager; may require fallback manual cert import path.

ZASTAVA-WEBHOOK-12-102 — Backend policy query & digest resolution

Objectives

• Resolve all images within AdmissionReview to immutable digests before policy evaluation.
• Call Scanner WebService /api/v1/scanner/policy/runtime with namespace/labels/images payload, enforce verdicts with deterministic error messaging.

Plan

1. Image resolution
   • Implement resolver service with pluggable strategies:
     • Use existing digest if present.
     • Resolve tags via registry HEAD (respecting admission.resolveTags flag); fallback to Observer-provided digest once core DTOs available.
   • Cache per-registry auth to minimise latency; adhere to allow/deny lists from configuration.
2. Scanner client
   • Define typed request/response models mirroring docs/modules/zastava/ARCHITECTURE.md structure (ttlSeconds, results[digest] -> { signed, hasSbom, policyVerdict, reasons, rekor }).
   • Implement retry policy (3 attempts, exponential backoff) and map HTTP errors to webhook fail-open/closed depending on namespace configuration.
   • Instrument latency (zastava.backend_latency_seconds) and failure counts.
3. Verdict enforcement
   • Evaluate per-image results: if any policyVerdict != pass (or warn when enforceWarnings=false), deny with aggregated reasons.
   • Attach ttlSeconds to admission response annotations for auditing.
   • Record structured logs with namespace, pod, image digest, decision, reasons, backend latency.
4. Contract coordination
   • Schedule joint review with Scanner WebService guild once SCANNER-RUNTIME-12-302 schema stabilises; track in TASKS sub-items.
   • Provide sample payload fixtures for CLI team (CLI-RUNTIME-13-005) to validate table output; ensure field names stay aligned.

Deliverables

• Registry resolver unit tests (tag->digest) with deterministic fixtures.
• HTTP client integration tests using Scanner stub returning varied verdict combinations.
• Documentation update summarising contract and failure handling.

Open Questions

• Confirm expected policy verdict enumeration (pass|warn|fail|error?) and textual reason codes.
• Need TTL behaviour: should webhook reduce TTL when backend returns > configured max?

ZASTAVA-WEBHOOK-12-103 — Caching, fail-open/closed toggles, metrics/logging

Objectives

• Provide deterministic caching layer respecting backend TTL while ensuring eviction on policy mutation.
• Allow namespace-scoped fail-open behaviour with explicit metrics and alerts.
• Surface actionable metrics/logging aligned with Architecture doc.

Plan

1. Cache design
   • In-memory LRU keyed by image digest; value carries verdict payload + expiry timestamp.
   • Support optional persistent seed (read-only) to prime hot digests for offline clusters (config: admission.cache.seedPath).
   • On startup, load seed file and emit metric zastava.cache_seed_entries_total.
   • Evict entries on TTL or when policyRevision annotation in AdmissionReview changes (requires hook from Core DTO).
2. Fail-open/closed toggles
   • Configuration: global default + namespace overrides through admission.failOpenNamespaces, admission.failClosedNamespaces.
   • Decision matrix:
     • Backend success + verdict PASS → allow.
     • Backend success + non-pass → deny unless namespace override says warn allowed.
     • Backend failure → allow if namespace fail-open, deny otherwise; annotate response with zastava.ops/fail-open=true.
   • Implement policy change event hook (future) to clear cache if observer signals revocation.
3. Metrics & logging
   • Counters: zastava.admission_requests_total{decision}, zastava.cache_hits_total{result=hit|miss}, zastava.fail_open_total, zastava.backend_failures_total{stage}.
   • Histograms: zastava.admission_latency_seconds (overall), zastava.resolve_latency_seconds.
   • Logs: structured JSON with decision, namespace, pod, imageDigest, reasons, cacheStatus, failMode.
   • Optionally emit OpenTelemetry span for admission path with attributes capturing backend latency + cache path.
4. Testing & ops hooks
   • Unit tests for cache TTL, namespace override logic, fail-open metric increments.
   • Integration test simulating backend outage ensuring fail-open/closed behaviour matches config.
   • Document runbook snippet describing interpreting metrics and toggling namespaces.

Open Questions

• Confirm whether cache entries should include policyRevision to detect backend policy updates; requires coordination with Policy guild.
• Need guidance on maximum cache size (default suggestions: 5k entries per replica?) to avoid memory blow-up.