Add Policy DSL Validator, Schema Exporter, and Simulation Smoke tools

- Implemented PolicyDslValidator with command-line options for strict mode and JSON output. - Created PolicySchemaExporter to generate JSON schemas for policy-related models. - Developed PolicySimulationSmoke tool to validate policy simulations against expected outcomes. - Added project files and necessary dependencies for each tool. - Ensured proper error handling and usage instructions across tools.
2025-10-27 08:00:11 +02:00
parent 2b7b88ca77
commit 799f787de2
712 changed files with 49449 additions and 6124 deletions
--- a/deploy/compose/README.md
+++ b/deploy/compose/README.md
@@ -7,21 +7,52 @@ These Compose bundles ship the minimum services required to exercise the scanner
 | Path | Purpose |
 | ---- | ------- |
 | `docker-compose.dev.yaml` | Edge/nightly stack tuned for laptops and iterative work. |
-| `docker-compose.stage.yaml` | Stable channel stack mirroring pre-production clusters. |
-| `docker-compose.airgap.yaml` | Stable stack with air-gapped defaults (no outbound hostnames). |
-| `docker-compose.mirror.yaml` | Managed mirror topology for `*.stella-ops.org` distribution (Concelier + Excititor + CDN gateway). |
-| `env/*.env.example` | Seed `.env` files that document required secrets and ports per profile. |
+| `docker-compose.stage.yaml` | Stable channel stack mirroring pre-production clusters. |
+| `docker-compose.prod.yaml` | Production cutover stack with front-door network hand-off and Notify events enabled. |
+| `docker-compose.airgap.yaml` | Stable stack with air-gapped defaults (no outbound hostnames). |
+| `docker-compose.mirror.yaml` | Managed mirror topology for `*.stella-ops.org` distribution (Concelier + Excititor + CDN gateway). |
+| `docker-compose.telemetry.yaml` | Optional OpenTelemetry collector overlay (mutual TLS, OTLP ingest endpoints). |
+| `docker-compose.telemetry-storage.yaml` | Prometheus/Tempo/Loki storage overlay with multi-tenant defaults. |
+| `env/*.env.example` | Seed `.env` files that document required secrets and ports per profile. |

 ## Usage

-```bash
-cp env/dev.env.example dev.env
-docker compose --env-file dev.env -f docker-compose.dev.yaml config
-docker compose --env-file dev.env -f docker-compose.dev.yaml up -d
-```
-
+```bash
+cp env/dev.env.example dev.env
+docker compose --env-file dev.env -f docker-compose.dev.yaml config
+docker compose --env-file dev.env -f docker-compose.dev.yaml up -d
+```
+
 The stage and airgap variants behave the same way—swap the file names accordingly. All profiles expose 443/8443 for the UI and REST APIs, and they share a `stellaops` Docker network scoped to the compose project.

+> **Graph Explorer reminder:** If you enable Cartographer or Graph API containers alongside these profiles, update `etc/authority.yaml` so the `cartographer-service` client is marked with `properties.serviceIdentity: "cartographer"` and carries a tenant hint. The Authority host now refuses `graph:write` tokens without that marker, so apply the configuration change before rolling out the updated images.
+
+### Telemetry collector overlay
+
+The OpenTelemetry collector overlay is optional and can be layered on top of any profile:
+
+```bash
+./ops/devops/telemetry/generate_dev_tls.sh
+docker compose -f docker-compose.telemetry.yaml up -d
+python ../../ops/devops/telemetry/smoke_otel_collector.py --host localhost
+docker compose -f docker-compose.telemetry-storage.yaml up -d
+```
+
+The generator script creates a development CA plus server/client certificates under
+`deploy/telemetry/certs/`. The smoke test sends OTLP/HTTP payloads using the generated
+client certificate and asserts the collector reports accepted traces, metrics, and logs.
+The storage overlay starts Prometheus, Tempo, and Loki with multitenancy enabled so you
+can validate the end-to-end pipeline before promoting changes to staging. Adjust the
+configs in `deploy/telemetry/storage/` before running in production.
+Mount the same certificates when running workloads so the collector can enforce mutual TLS.
+
+For production cutovers copy `env/prod.env.example` to `prod.env`, update the secret placeholders, and create the external network expected by the profile:
+
+```bash
+docker network create stellaops_frontdoor
+docker compose --env-file prod.env -f docker-compose.prod.yaml config
+```
+
 ### Scanner event stream settings

 Scanner WebService can emit signed `scanner.report.*` events to Redis Streams when `SCANNER__EVENTS__ENABLED=true`. Each profile ships environment placeholders you can override in the `.env` file:
@@ -35,6 +66,10 @@ Scanner WebService can emit signed `scanner.report.*` events to Redis Streams wh

 Helm values mirror the same knobs under each service’s `env` map (see `deploy/helm/stellaops/values-*.yaml`).

+### Front-door network hand-off
+
+`docker-compose.prod.yaml` adds a `frontdoor` network so operators can attach Traefik, Envoy, or an on-prem load balancer that terminates TLS. Override `FRONTDOOR_NETWORK` in `prod.env` if your reverse proxy uses a different bridge name. Attach only the externally reachable services (Authority, Signer, Attestor, Concelier, Scanner Web, Notify Web, UI) to that network—internal infrastructure (Mongo, MinIO, RustFS, NATS) stays on the private `stellaops` network.
+
 ### Updating to a new release

 1. Import the new manifest into `deploy/releases/` (see `deploy/README.md`).
--- a/deploy/compose/docker-compose.prod.yaml
+++ b/deploy/compose/docker-compose.prod.yaml
@@ -0,0 +1,237 @@
+x-release-labels: &release-labels
+  com.stellaops.release.version: "2025.09.2"
+  com.stellaops.release.channel: "stable"
+  com.stellaops.profile: "prod"
+
+networks:
+  stellaops:
+    driver: bridge
+  frontdoor:
+    external: true
+    name: ${FRONTDOOR_NETWORK:-stellaops_frontdoor}
+
+volumes:
+  mongo-data:
+  minio-data:
+  rustfs-data:
+  concelier-jobs:
+  nats-data:
+
+services:
+  mongo:
+    image: docker.io/library/mongo@sha256:c258b26dbb7774f97f52aff52231ca5f228273a84329c5f5e451c3739457db49
+    command: ["mongod", "--bind_ip_all"]
+    restart: unless-stopped
+    environment:
+      MONGO_INITDB_ROOT_USERNAME: "${MONGO_INITDB_ROOT_USERNAME}"
+      MONGO_INITDB_ROOT_PASSWORD: "${MONGO_INITDB_ROOT_PASSWORD}"
+    volumes:
+      - mongo-data:/data/db
+    networks:
+      - stellaops
+    labels: *release-labels
+
+  minio:
+    image: docker.io/minio/minio@sha256:14cea493d9a34af32f524e538b8346cf79f3321eff8e708c1e2960462bd8936e
+    command: ["server", "/data", "--console-address", ":9001"]
+    restart: unless-stopped
+    environment:
+      MINIO_ROOT_USER: "${MINIO_ROOT_USER}"
+      MINIO_ROOT_PASSWORD: "${MINIO_ROOT_PASSWORD}"
+    volumes:
+      - minio-data:/data
+    ports:
+      - "${MINIO_CONSOLE_PORT:-9001}:9001"
+    networks:
+      - stellaops
+    labels: *release-labels
+
+  rustfs:
+    image: registry.stella-ops.org/stellaops/rustfs:2025.10.0-edge
+    command: ["serve", "--listen", "0.0.0.0:8080", "--root", "/data"]
+    restart: unless-stopped
+    environment:
+      RUSTFS__LOG__LEVEL: info
+      RUSTFS__STORAGE__PATH: /data
+    volumes:
+      - rustfs-data:/data
+    ports:
+      - "${RUSTFS_HTTP_PORT:-8080}:8080"
+    networks:
+      - stellaops
+    labels: *release-labels
+
+  nats:
+    image: docker.io/library/nats@sha256:c82559e4476289481a8a5196e675ebfe67eea81d95e5161e3e78eccfe766608e
+    command:
+      - "-js"
+      - "-sd"
+      - /data
+    restart: unless-stopped
+    ports:
+      - "${NATS_CLIENT_PORT:-4222}:4222"
+    volumes:
+      - nats-data:/data
+    networks:
+      - stellaops
+    labels: *release-labels
+
+  authority:
+    image: registry.stella-ops.org/stellaops/authority@sha256:b0348bad1d0b401cc3c71cb40ba034c8043b6c8874546f90d4783c9dbfcc0bf5
+    restart: unless-stopped
+    depends_on:
+      - mongo
+    environment:
+      STELLAOPS_AUTHORITY__ISSUER: "${AUTHORITY_ISSUER}"
+      STELLAOPS_AUTHORITY__MONGO__CONNECTIONSTRING: "mongodb://${MONGO_INITDB_ROOT_USERNAME}:${MONGO_INITDB_ROOT_PASSWORD}@mongo:27017"
+      STELLAOPS_AUTHORITY__PLUGINDIRECTORIES__0: "/app/plugins"
+      STELLAOPS_AUTHORITY__PLUGINS__CONFIGURATIONDIRECTORY: "/app/etc/authority.plugins"
+    volumes:
+      - ../../etc/authority.yaml:/etc/authority.yaml:ro
+      - ../../etc/authority.plugins:/app/etc/authority.plugins:ro
+    ports:
+      - "${AUTHORITY_PORT:-8440}:8440"
+    networks:
+      - stellaops
+      - frontdoor
+    labels: *release-labels
+
+  signer:
+    image: registry.stella-ops.org/stellaops/signer@sha256:8ad574e61f3a9e9bda8a58eb2700ae46813284e35a150b1137bc7c2b92ac0f2e
+    restart: unless-stopped
+    depends_on:
+      - authority
+    environment:
+      SIGNER__AUTHORITY__BASEURL: "https://authority:8440"
+      SIGNER__POE__INTROSPECTURL: "${SIGNER_POE_INTROSPECT_URL}"
+      SIGNER__STORAGE__MONGO__CONNECTIONSTRING: "mongodb://${MONGO_INITDB_ROOT_USERNAME}:${MONGO_INITDB_ROOT_PASSWORD}@mongo:27017"
+    ports:
+      - "${SIGNER_PORT:-8441}:8441"
+    networks:
+      - stellaops
+      - frontdoor
+    labels: *release-labels
+
+  attestor:
+    image: registry.stella-ops.org/stellaops/attestor@sha256:0534985f978b0b5d220d73c96fddd962cd9135f616811cbe3bff4666c5af568f
+    restart: unless-stopped
+    depends_on:
+      - signer
+    environment:
+      ATTESTOR__SIGNER__BASEURL: "https://signer:8441"
+      ATTESTOR__MONGO__CONNECTIONSTRING: "mongodb://${MONGO_INITDB_ROOT_USERNAME}:${MONGO_INITDB_ROOT_PASSWORD}@mongo:27017"
+    ports:
+      - "${ATTESTOR_PORT:-8442}:8442"
+    networks:
+      - stellaops
+      - frontdoor
+    labels: *release-labels
+
+  concelier:
+    image: registry.stella-ops.org/stellaops/concelier@sha256:c58cdcaee1d266d68d498e41110a589dd204b487d37381096bd61ab345a867c5
+    restart: unless-stopped
+    depends_on:
+      - mongo
+      - minio
+    environment:
+      CONCELIER__STORAGE__MONGO__CONNECTIONSTRING: "mongodb://${MONGO_INITDB_ROOT_USERNAME}:${MONGO_INITDB_ROOT_PASSWORD}@mongo:27017"
+      CONCELIER__STORAGE__S3__ENDPOINT: "http://minio:9000"
+      CONCELIER__STORAGE__S3__ACCESSKEYID: "${MINIO_ROOT_USER}"
+      CONCELIER__STORAGE__S3__SECRETACCESSKEY: "${MINIO_ROOT_PASSWORD}"
+      CONCELIER__AUTHORITY__BASEURL: "https://authority:8440"
+    volumes:
+      - concelier-jobs:/var/lib/concelier/jobs
+    ports:
+      - "${CONCELIER_PORT:-8445}:8445"
+    networks:
+      - stellaops
+      - frontdoor
+    labels: *release-labels
+
+  scanner-web:
+    image: registry.stella-ops.org/stellaops/scanner-web@sha256:14b23448c3f9586a9156370b3e8c1991b61907efa666ca37dd3aaed1e79fe3b7
+    restart: unless-stopped
+    depends_on:
+      - concelier
+      - rustfs
+      - nats
+    environment:
+      SCANNER__STORAGE__MONGO__CONNECTIONSTRING: "mongodb://${MONGO_INITDB_ROOT_USERNAME}:${MONGO_INITDB_ROOT_PASSWORD}@mongo:27017"
+      SCANNER__ARTIFACTSTORE__DRIVER: "rustfs"
+      SCANNER__ARTIFACTSTORE__ENDPOINT: "http://rustfs:8080/api/v1"
+      SCANNER__ARTIFACTSTORE__BUCKET: "scanner-artifacts"
+      SCANNER__ARTIFACTSTORE__TIMEOUTSECONDS: "30"
+      SCANNER__QUEUE__BROKER: "${SCANNER_QUEUE_BROKER}"
+      SCANNER__EVENTS__ENABLED: "${SCANNER_EVENTS_ENABLED:-true}"
+      SCANNER__EVENTS__DRIVER: "${SCANNER_EVENTS_DRIVER:-redis}"
+      SCANNER__EVENTS__DSN: "${SCANNER_EVENTS_DSN:-}"
+      SCANNER__EVENTS__STREAM: "${SCANNER_EVENTS_STREAM:-stella.events}"
+      SCANNER__EVENTS__PUBLISHTIMEOUTSECONDS: "${SCANNER_EVENTS_PUBLISH_TIMEOUT_SECONDS:-5}"
+      SCANNER__EVENTS__MAXSTREAMLENGTH: "${SCANNER_EVENTS_MAX_STREAM_LENGTH:-10000}"
+    ports:
+      - "${SCANNER_WEB_PORT:-8444}:8444"
+    networks:
+      - stellaops
+      - frontdoor
+    labels: *release-labels
+
+  scanner-worker:
+    image: registry.stella-ops.org/stellaops/scanner-worker@sha256:32e25e76386eb9ea8bee0a1ad546775db9a2df989fab61ac877e351881960dab
+    restart: unless-stopped
+    depends_on:
+      - scanner-web
+      - rustfs
+      - nats
+    environment:
+      SCANNER__STORAGE__MONGO__CONNECTIONSTRING: "mongodb://${MONGO_INITDB_ROOT_USERNAME}:${MONGO_INITDB_ROOT_PASSWORD}@mongo:27017"
+      SCANNER__ARTIFACTSTORE__DRIVER: "rustfs"
+      SCANNER__ARTIFACTSTORE__ENDPOINT: "http://rustfs:8080/api/v1"
+      SCANNER__ARTIFACTSTORE__BUCKET: "scanner-artifacts"
+      SCANNER__ARTIFACTSTORE__TIMEOUTSECONDS: "30"
+      SCANNER__QUEUE__BROKER: "${SCANNER_QUEUE_BROKER}"
+    networks:
+      - stellaops
+    labels: *release-labels
+
+  notify-web:
+    image: ${NOTIFY_WEB_IMAGE:-registry.stella-ops.org/stellaops/notify-web:2025.09.2}
+    restart: unless-stopped
+    depends_on:
+      - mongo
+      - authority
+    environment:
+      DOTNET_ENVIRONMENT: Production
+    volumes:
+      - ../../etc/notify.prod.yaml:/app/etc/notify.yaml:ro
+    ports:
+      - "${NOTIFY_WEB_PORT:-8446}:8446"
+    networks:
+      - stellaops
+      - frontdoor
+    labels: *release-labels
+
+  excititor:
+    image: registry.stella-ops.org/stellaops/excititor@sha256:59022e2016aebcef5c856d163ae705755d3f81949d41195256e935ef40a627fa
+    restart: unless-stopped
+    depends_on:
+      - concelier
+    environment:
+      EXCITITOR__CONCELIER__BASEURL: "https://concelier:8445"
+      EXCITITOR__STORAGE__MONGO__CONNECTIONSTRING: "mongodb://${MONGO_INITDB_ROOT_USERNAME}:${MONGO_INITDB_ROOT_PASSWORD}@mongo:27017"
+    networks:
+      - stellaops
+    labels: *release-labels
+
+  web-ui:
+    image: registry.stella-ops.org/stellaops/web-ui@sha256:10d924808c48e4353e3a241da62eb7aefe727a1d6dc830eb23a8e181013b3a23
+    restart: unless-stopped
+    depends_on:
+      - scanner-web
+    environment:
+      STELLAOPS_UI__BACKEND__BASEURL: "https://scanner-web:8444"
+    ports:
+      - "${UI_PORT:-8443}:8443"
+    networks:
+      - stellaops
+      - frontdoor
+    labels: *release-labels
--- a/deploy/compose/docker-compose.telemetry-storage.yaml
+++ b/deploy/compose/docker-compose.telemetry-storage.yaml
@@ -0,0 +1,57 @@
+version: "3.9"
+
+services:
+  prometheus:
+    image: prom/prometheus:v2.53.0
+    container_name: stellaops-prometheus
+    command:
+      - "--config.file=/etc/prometheus/prometheus.yaml"
+    volumes:
+      - ../telemetry/storage/prometheus.yaml:/etc/prometheus/prometheus.yaml:ro
+      - prometheus-data:/prometheus
+      - ../telemetry/certs:/etc/telemetry/tls:ro
+      - ../telemetry/storage/auth:/etc/telemetry/auth:ro
+    environment:
+      PROMETHEUS_COLLECTOR_TARGET: stellaops-otel-collector:9464
+    ports:
+      - "9090:9090"
+    depends_on:
+      - tempo
+      - loki
+
+  tempo:
+    image: grafana/tempo:2.5.0
+    container_name: stellaops-tempo
+    command:
+      - "-config.file=/etc/tempo/tempo.yaml"
+    volumes:
+      - ../telemetry/storage/tempo.yaml:/etc/tempo/tempo.yaml:ro
+      - ../telemetry/storage/tenants/tempo-overrides.yaml:/etc/telemetry/tenants/tempo-overrides.yaml:ro
+      - ../telemetry/certs:/etc/telemetry/tls:ro
+      - tempo-data:/var/tempo
+    ports:
+      - "3200:3200"
+    environment:
+      TEMPO_ZONE: docker
+
+  loki:
+    image: grafana/loki:3.1.0
+    container_name: stellaops-loki
+    command:
+      - "-config.file=/etc/loki/loki.yaml"
+    volumes:
+      - ../telemetry/storage/loki.yaml:/etc/loki/loki.yaml:ro
+      - ../telemetry/storage/tenants/loki-overrides.yaml:/etc/telemetry/tenants/loki-overrides.yaml:ro
+      - ../telemetry/certs:/etc/telemetry/tls:ro
+      - loki-data:/var/loki
+    ports:
+      - "3100:3100"
+
+volumes:
+  prometheus-data:
+  tempo-data:
+  loki-data:
+
+networks:
+  default:
+    name: stellaops-telemetry
--- a/deploy/compose/docker-compose.telemetry.yaml
+++ b/deploy/compose/docker-compose.telemetry.yaml
@@ -0,0 +1,34 @@
+version: "3.9"
+
+services:
+  otel-collector:
+    image: otel/opentelemetry-collector:0.105.0
+    container_name: stellaops-otel-collector
+    command:
+      - "--config=/etc/otel-collector/config.yaml"
+    environment:
+      STELLAOPS_OTEL_TLS_CERT: /etc/otel-collector/tls/collector.crt
+      STELLAOPS_OTEL_TLS_KEY: /etc/otel-collector/tls/collector.key
+      STELLAOPS_OTEL_TLS_CA: /etc/otel-collector/tls/ca.crt
+      STELLAOPS_OTEL_PROMETHEUS_ENDPOINT: 0.0.0.0:9464
+      STELLAOPS_OTEL_REQUIRE_CLIENT_CERT: "true"
+      STELLAOPS_TENANT_ID: dev
+    volumes:
+      - ../telemetry/otel-collector-config.yaml:/etc/otel-collector/config.yaml:ro
+      - ../telemetry/certs:/etc/otel-collector/tls:ro
+    ports:
+      - "4317:4317"    # OTLP gRPC (mTLS)
+      - "4318:4318"    # OTLP HTTP (mTLS)
+      - "9464:9464"    # Prometheus exporter (mTLS)
+      - "13133:13133"  # Health check
+      - "1777:1777"    # pprof
+    healthcheck:
+      test: ["CMD", "curl", "-fsk", "--cert", "/etc/otel-collector/tls/client.crt", "--key", "/etc/otel-collector/tls/client.key", "--cacert", "/etc/otel-collector/tls/ca.crt", "https://localhost:13133/healthz"]
+      interval: 30s
+      start_period: 15s
+      timeout: 5s
+      retries: 3
+
+networks:
+  default:
+    name: stellaops-telemetry
--- a/deploy/compose/env/prod.env.example
+++ b/deploy/compose/env/prod.env.example
@@ -0,0 +1,29 @@
+# Substitutions for docker-compose.prod.yaml
+# ⚠️ Replace all placeholder secrets with values sourced from your secret manager.
+MONGO_INITDB_ROOT_USERNAME=stellaops-prod
+MONGO_INITDB_ROOT_PASSWORD=REPLACE_WITH_STRONG_PASSWORD
+MINIO_ROOT_USER=stellaops-prod
+MINIO_ROOT_PASSWORD=REPLACE_WITH_STRONG_PASSWORD
+# Expose the MinIO console only to trusted operator networks.
+MINIO_CONSOLE_PORT=39001
+RUSTFS_HTTP_PORT=8080
+AUTHORITY_ISSUER=https://authority.prod.stella-ops.org
+AUTHORITY_PORT=8440
+SIGNER_POE_INTROSPECT_URL=https://licensing.prod.stella-ops.org/introspect
+SIGNER_PORT=8441
+ATTESTOR_PORT=8442
+CONCELIER_PORT=8445
+SCANNER_WEB_PORT=8444
+UI_PORT=8443
+NATS_CLIENT_PORT=4222
+SCANNER_QUEUE_BROKER=nats://nats:4222
+# `true` enables signed scanner events for Notify ingestion.
+SCANNER_EVENTS_ENABLED=true
+SCANNER_EVENTS_DRIVER=redis
+# Leave SCANNER_EVENTS_DSN empty to inherit the Redis queue DSN when SCANNER_QUEUE_BROKER uses redis://.
+SCANNER_EVENTS_DSN=
+SCANNER_EVENTS_STREAM=stella.events
+SCANNER_EVENTS_PUBLISH_TIMEOUT_SECONDS=5
+SCANNER_EVENTS_MAX_STREAM_LENGTH=10000
+# External reverse proxy (Traefik, Envoy, etc.) that terminates TLS.
+FRONTDOOR_NETWORK=stellaops_frontdoor