feat(docs): Add comprehensive documentation for Vexer, Vulnerability Explorer, and Zastava modules

- Introduced AGENTS.md, README.md, TASKS.md, and implementation_plan.md for Vexer, detailing mission, responsibilities, key components, and operational notes. - Established similar documentation structure for Vulnerability Explorer and Zastava modules, including their respective workflows, integrations, and observability notes. - Created risk scoring profiles documentation outlining the core workflow, factor model, governance, and deliverables. - Ensured all modules adhere to the Aggregation-Only Contract and maintain determinism and provenance in outputs.
2025-10-30 00:09:39 +02:00
parent 3154c67978
commit 7b5bdcf4d3
503 changed files with 16136 additions and 54638 deletions
--- a/docs/modules/concelier/operations/mirror.md
+++ b/docs/modules/concelier/operations/mirror.md
@@ -0,0 +1,238 @@
+# Concelier & Excititor Mirror Operations
+
+This runbook describes how Stella Ops operates the managed mirrors under `*.stella-ops.org`.
+It covers Docker Compose and Helm deployment overlays, secret handling for multi-tenant
+authn, CDN fronting, and the recurring sync pipeline that keeps mirror bundles current.
+
+## 1. Prerequisites
+
+- **Authority access** – client credentials (`client_id` + secret) authorised for
+  `concelier.mirror.read` and `excititor.mirror.read` scopes. Secrets live outside git.
+- **Signed TLS certificates** – wildcard or per-domain (`mirror-primary`, `mirror-community`).
+  Store them under `deploy/compose/mirror-gateway/tls/` or in Kubernetes secrets.
+- **Mirror gateway credentials** – Basic Auth htpasswd files per domain. Generate with
+  `htpasswd -B`. Operators distribute credentials to downstream consumers.
+- **Export artifact source** – read access to the canonical S3 buckets (or rsync share)
+  that hold `concelier` JSON bundles and `excititor` VEX exports.
+- **Persistent volumes** – storage for Concelier job metadata and mirror export trees.
+  For Helm, provision PVCs (`concelier-mirror-jobs`, `concelier-mirror-exports`,
+  `excititor-mirror-exports`, `mirror-mongo-data`, `mirror-minio-data`) before rollout.
+
+### 1.1 Service configuration quick reference
+
+Concelier.WebService exposes the mirror HTTP endpoints once `CONCELIER__MIRROR__ENABLED=true`.
+Key knobs:
+
+- `CONCELIER__MIRROR__EXPORTROOT` – root folder containing export snapshots (`<exportId>/mirror/*`).
+- `CONCELIER__MIRROR__ACTIVEEXPORTID` – optional explicit export id; otherwise the service auto-falls back to the `latest/` symlink or newest directory.
+- `CONCELIER__MIRROR__REQUIREAUTHENTICATION` – default auth requirement; override per domain with `CONCELIER__MIRROR__DOMAINS__{n}__REQUIREAUTHENTICATION`.
+- `CONCELIER__MIRROR__MAXINDEXREQUESTSPERHOUR` – budget for `/concelier/exports/index.json`. Domains inherit this value unless they define `__MAXDOWNLOADREQUESTSPERHOUR`.
+- `CONCELIER__MIRROR__DOMAINS__{n}__ID` – domain identifier matching the exporter manifest; additional keys configure display name and rate budgets.
+
+> The service honours Stella Ops Authority when `CONCELIER__AUTHORITY__ENABLED=true` and `ALLOWANONYMOUSFALLBACK=false`. Use the bypass CIDR list (`CONCELIER__AUTHORITY__BYPASSNETWORKS__*`) for in-cluster ingress gateways that terminate Basic Auth. Unauthorized requests emit `WWW-Authenticate: Bearer` so downstream automation can detect token failures.
+
+Mirror responses carry deterministic cache headers: `/index.json` returns `Cache-Control: public, max-age=60`, while per-domain manifests/bundles include `Cache-Control: public, max-age=300, immutable`. Rate limiting surfaces `Retry-After` when quotas are exceeded.
+
+### 1.2 Mirror connector configuration
+
+Downstream Concelier instances ingest published bundles using the `StellaOpsMirrorConnector`. Operators running the connector in air‑gapped or limited connectivity environments can tune the following options (environment prefix `CONCELIER__SOURCES__STELLAOPSMIRROR__`):
+
+- `BASEADDRESS` – absolute mirror root (e.g., `https://mirror-primary.stella-ops.org`).
+- `INDEXPATH` – relative path to the mirror index (`/concelier/exports/index.json` by default).
+- `DOMAINID` – mirror domain identifier from the index (`primary`, `community`, etc.).
+- `HTTPTIMEOUT` – request timeout; raise when mirrors sit behind slow WAN links.
+- `SIGNATURE__ENABLED` – require detached JWS verification for `bundle.json`.
+- `SIGNATURE__KEYID` / `SIGNATURE__PROVIDER` – expected signing key metadata.
+- `SIGNATURE__PUBLICKEYPATH` – PEM fallback used when the mirror key registry is offline.
+
+The connector keeps a per-export fingerprint (bundle digest + generated-at timestamp) and tracks outstanding document IDs. If a scan is interrupted, the next run resumes parse/map work using the stored fingerprint and pending document lists—no network requests are reissued unless the upstream digest changes.
+
+## 2. Secret & certificate layout
+
+### Docker Compose (`deploy/compose/docker-compose.mirror.yaml`)
+
+- `deploy/compose/env/mirror.env.example` – copy to `.env` and adjust quotas or domain IDs.
+- `deploy/compose/mirror-secrets/` – mount read-only into `/run/secrets`. Place:
+  - `concelier-authority-client` – Authority client secret.
+  - `excititor-authority-client` (optional) – reserve for future authn.
+- `deploy/compose/mirror-gateway/tls/` – PEM-encoded cert/key pairs:
+  - `mirror-primary.crt`, `mirror-primary.key`
+  - `mirror-community.crt`, `mirror-community.key`
+- `deploy/compose/mirror-gateway/secrets/` – htpasswd files:
+  - `mirror-primary.htpasswd`
+  - `mirror-community.htpasswd`
+
+### Helm (`deploy/helm/stellaops/values-mirror.yaml`)
+
+Create secrets in the target namespace:
+
+```bash
+kubectl create secret generic concelier-mirror-auth \
+  --from-file=concelier-authority-client=concelier-authority-client
+
+kubectl create secret generic excititor-mirror-auth \
+  --from-file=excititor-authority-client=excititor-authority-client
+
+kubectl create secret tls mirror-gateway-tls \
+  --cert=mirror-primary.crt --key=mirror-primary.key
+
+kubectl create secret generic mirror-gateway-htpasswd \
+  --from-file=mirror-primary.htpasswd --from-file=mirror-community.htpasswd
+```
+
+> Keep Basic Auth lists short-lived (rotate quarterly) and document credential recipients.
+
+## 3. Deployment
+
+### 3.1 Docker Compose (edge mirrors, lab validation)
+
+1. `cp deploy/compose/env/mirror.env.example deploy/compose/env/mirror.env`
+2. Populate secrets/tls directories as described above.
+3. Sync mirror bundles (see §4) into `deploy/compose/mirror-data/…` and ensure they are mounted
+   on the host path backing the `concelier-exports` and `excititor-exports` volumes.
+4. Run the profile validator: `deploy/tools/validate-profiles.sh`.
+5. Launch: `docker compose --env-file env/mirror.env -f docker-compose.mirror.yaml up -d`.
+
+### 3.2 Helm (production mirrors)
+
+1. Provision PVCs sized for mirror bundles (baseline: 20 GiB per domain).
+2. Create secrets/tls config maps (§2).
+3. `helm upgrade --install mirror deploy/helm/stellaops -f deploy/helm/stellaops/values-mirror.yaml`.
+4. Annotate the `stellaops-mirror-gateway` service with ingress/LoadBalancer metadata required by
+   your CDN (e.g., AWS load balancer scheme internal + NLB idle timeout).
+
+## 4. Artifact sync workflow
+
+Mirrors never generate exports—they ingest signed bundles produced by the Concelier and Excititor
+export jobs. Recommended sync pattern:
+
+### 4.1 Compose host (systemd timer)
+
+`/usr/local/bin/mirror-sync.sh`:
+
+```bash
+#!/usr/bin/env bash
+set -euo pipefail
+export AWS_ACCESS_KEY_ID=…
+export AWS_SECRET_ACCESS_KEY=…
+
+aws s3 sync s3://mirror-stellaops/concelier/latest \
+  /opt/stellaops/mirror-data/concelier --delete --size-only
+
+aws s3 sync s3://mirror-stellaops/excititor/latest \
+  /opt/stellaops/mirror-data/excititor --delete --size-only
+```
+
+Schedule with a systemd timer every 5 minutes. The Compose volumes mount `/opt/stellaops/mirror-data/*`
+into the containers read-only, matching `CONCELIER__MIRROR__EXPORTROOT=/exports/json` and
+`EXCITITOR__ARTIFACTS__FILESYSTEM__ROOT=/exports`.
+
+### 4.2 Kubernetes (CronJob)
+
+Create a CronJob running the AWS CLI (or rclone) in the same namespace, writing into the PVCs:
+
+```yaml
+apiVersion: batch/v1
+kind: CronJob
+metadata:
+  name: mirror-sync
+spec:
+  schedule: "*/5 * * * *"
+  jobTemplate:
+    spec:
+      template:
+        spec:
+          containers:
+          - name: sync
+            image: public.ecr.aws/aws-cli/aws-cli@sha256:5df5f52c29f5e3ba46d0ad9e0e3afc98701c4a0f879400b4c5f80d943b5fadea
+            command:
+              - /bin/sh
+              - -c
+              - >
+                aws s3 sync s3://mirror-stellaops/concelier/latest /exports/concelier --delete --size-only &&
+                aws s3 sync s3://mirror-stellaops/excititor/latest /exports/excititor --delete --size-only
+            volumeMounts:
+              - name: concelier-exports
+                mountPath: /exports/concelier
+              - name: excititor-exports
+                mountPath: /exports/excititor
+            envFrom:
+              - secretRef:
+                  name: mirror-sync-aws
+          restartPolicy: OnFailure
+          volumes:
+            - name: concelier-exports
+              persistentVolumeClaim:
+                claimName: concelier-mirror-exports
+            - name: excititor-exports
+              persistentVolumeClaim:
+                claimName: excititor-mirror-exports
+```
+
+## 5. CDN integration
+
+1. Point the CDN origin at the mirror gateway (Compose host or Kubernetes LoadBalancer).
+2. Honour the response headers emitted by the gateway and Concelier/Excititor:
+   `Cache-Control: public, max-age=300, immutable` for mirror payloads.
+3. Configure origin shields in the CDN to prevent cache stampedes. Recommended TTLs:
+   - Index (`/concelier/exports/index.json`, `/excititor/mirror/*/index`) → 60 s.
+   - Bundle/manifest payloads → 300 s.
+4. Forward the `Authorization` header—Basic Auth terminates at the gateway.
+5. Enforce per-domain rate limits at the CDN (matching gateway budgets) and enable logging
+   to SIEM for anomaly detection.
+
+## 6. Smoke tests
+
+After each deployment or sync cycle (temporarily set low budgets if you need to observe 429 responses):
+
+```bash
+# Index with Basic Auth
+curl -u $PRIMARY_CREDS https://mirror-primary.stella-ops.org/concelier/exports/index.json | jq 'keys'
+
+# Mirror manifest signature and cache headers
+curl -u $PRIMARY_CREDS -I https://mirror-primary.stella-ops.org/concelier/exports/mirror/primary/manifest.json \
+  | tee /tmp/manifest-headers.txt
+grep -E '^Cache-Control: ' /tmp/manifest-headers.txt   # expect public, max-age=300, immutable
+
+# Excititor consensus bundle metadata
+curl -u $COMMUNITY_CREDS https://mirror-community.stella-ops.org/excititor/mirror/community/index \
+  | jq '.exports[].exportKey'
+
+# Signed bundle + detached JWS (spot check digests)
+curl -u $PRIMARY_CREDS https://mirror-primary.stella-ops.org/concelier/exports/mirror/primary/bundle.json.jws \
+  -o bundle.json.jws
+cosign verify-blob --signature bundle.json.jws --key mirror-key.pub bundle.json
+
+# Service-level auth check (inside cluster – no gateway credentials)
+kubectl exec deploy/stellaops-concelier -- curl -si http://localhost:8443/concelier/exports/mirror/primary/manifest.json \
+  | head -n 5   # expect HTTP/1.1 401 with WWW-Authenticate: Bearer
+
+# Rate limit smoke (repeat quickly; second call should return 429 + Retry-After)
+for i in 1 2; do
+  curl -s -o /dev/null -D - https://mirror-primary.stella-ops.org/concelier/exports/index.json \
+    -u $PRIMARY_CREDS | grep -E '^(HTTP/|Retry-After:)'
+  sleep 1
+done
+```
+
+Watch the gateway metrics (`nginx_vts` or access logs) for cache hits. In Kubernetes, `kubectl logs deploy/stellaops-mirror-gateway`
+should show `X-Cache-Status: HIT/MISS`.
+
+## 7. Maintenance & rotation
+
+- **Bundle freshness** – alert if sync job lag exceeds 15 minutes or if `concelier` logs
+  `Mirror export root is not configured`.
+- **Secret rotation** – change Authority client secrets and Basic Auth credentials quarterly.
+  Update the mounted secrets and restart deployments (`docker compose restart concelier` or
+  `kubectl rollout restart deploy/stellaops-concelier`).
+- **TLS renewal** – reissue certificates, place new files, and reload gateway (`docker compose exec mirror-gateway nginx -s reload`).
+- **Quota tuning** – adjust per-domain `MAXDOWNLOADREQUESTSPERHOUR` in `.env` or values file.
+  Align CDN rate limits and inform downstreams.
+
+## 8. References
+
+- Deployment profiles: `deploy/compose/docker-compose.mirror.yaml`,
+  `deploy/helm/stellaops/values-mirror.yaml`
+- Mirror architecture dossiers: `docs/modules/concelier/architecture.md`,
+  `docs/modules/excititor/mirrors.md`
+- Export bundling: `docs/modules/devops/architecture.md` §3, `docs/modules/excititor/architecture.md` §7