CD/CD consolidation

2025-12-26 17:32:23 +02:00
parent a866eb6277
commit c786faae84
638 changed files with 3821 additions and 181 deletions
--- a/devops/services/authority/AGENTS.md
+++ b/devops/services/authority/AGENTS.md
@@ -0,0 +1,20 @@
+# Authority DevOps Crew
+
+## Mission
+Operate and harden the StellaOps Authority platform in production and air-gapped environments: container images, deployment assets, observability defaults, backup/restore, and runtime key management.
+
+## Focus Areas
+- **Build & Packaging** – Dockerfiles, OCI bundles, offline artefact refresh.
+- **Deployment Tooling** – Compose/Kubernetes manifests, secrets bootstrap, upgrade paths.
+- **Observability** – Logging defaults, metrics/trace exporters, dashboards, alert policies.
+- **Continuity & Security** – Backup/restore guides, key rotation playbooks, revocation propagation.
+
+## Working Agreements
+- Track work directly in the relevant `docs/implplan/SPRINT_*.md` rows (TODO → DOING → DONE/BLOCKED); keep entries dated.
+- Validate container changes with the CI pipeline (`ops/authority` GitHub workflow) before marking DONE.
+- Update operator documentation in `docs/` together with any behavioural change.
+- Coordinate with Authority Core and Security Guild before altering sensitive defaults (rate limits, crypto providers, revocation jobs).
+
+## Required Reading
+- `docs/modules/platform/architecture-overview.md`
+- `docs/modules/airgap/airgap-mode.md`
--- a/devops/services/authority/Dockerfile
+++ b/devops/services/authority/Dockerfile
@@ -0,0 +1,38 @@
+# syntax=docker/dockerfile:1.7-labs
+
+#
+# StellaOps Authority – distroless container build
+# Produces a minimal image containing the Authority host and its plugins.
+#
+
+ARG SDK_IMAGE=mcr.microsoft.com/dotnet/nightly/sdk:10.0
+ARG RUNTIME_IMAGE=gcr.io/distroless/dotnet/aspnet:latest
+
+FROM ${SDK_IMAGE} AS build
+
+WORKDIR /src
+
+# Restore & publish
+COPY . .
+RUN dotnet restore src/StellaOps.sln
+RUN dotnet publish src/Authority/StellaOps.Authority/StellaOps.Authority/StellaOps.Authority.csproj \
+    -c Release \
+    -o /app/publish \
+    /p:UseAppHost=false
+
+FROM ${RUNTIME_IMAGE} AS runtime
+
+WORKDIR /app
+
+ENV ASPNETCORE_URLS=http://0.0.0.0:8080
+ENV STELLAOPS_AUTHORITY__PLUGINDIRECTORIES__0=/app/plugins
+ENV STELLAOPS_AUTHORITY__PLUGINS__CONFIGURATIONDIRECTORY=/app/etc/authority.plugins
+
+COPY --from=build /app/publish ./
+
+# Provide writable mount points for configs/keys/plugins
+VOLUME ["/app/etc", "/app/plugins", "/app/keys"]
+
+EXPOSE 8080
+
+ENTRYPOINT ["dotnet", "StellaOps.Authority.dll"]
--- a/devops/services/authority/README.md
+++ b/devops/services/authority/README.md
@@ -0,0 +1,62 @@
+# StellaOps Authority Container Scaffold
+
+This directory provides a distroless Dockerfile and `docker-compose` sample for bootstrapping the Authority service alongside MongoDB (required) and Redis (optional).
+
+## Prerequisites
+
+- Docker Engine 25+ and Compose V2
+- .NET 10 preview SDK (only required when building locally outside of Compose)
+- Populated Authority configuration at `etc/authority.yaml` and plugin manifests under `etc/authority.plugins/`
+
+## Usage
+
+```bash
+# 1. Ensure configuration files exist (copied from etc/authority.yaml.sample, etc/authority.plugins/*.yaml)
+# 2. Build and start the stack
+docker compose -f ops/authority/docker-compose.authority.yaml up --build
+```
+
+`authority.yaml` is mounted read-only at `/etc/authority.yaml` inside the container. Plugin manifests are mounted to `/app/etc/authority.plugins`. Update the issuer URL plus any Mongo credentials in the compose file or via an `.env`.
+
+To run with pre-built images, replace the `build:` block in the compose file with an `image:` reference.
+
+## Volumes
+
+- `mongo-data` – persists MongoDB state.
+- `redis-data` – optional Redis persistence (enable the service before use).
+- `authority-keys` – writable volume for Authority signing keys.
+
+## Environment overrides
+
+Key environment variables (mirroring `StellaOpsAuthorityOptions`):
+
+| Variable | Description |
+| --- | --- |
+| `STELLAOPS_AUTHORITY__ISSUER` | Public issuer URL advertised by Authority |
+| `STELLAOPS_AUTHORITY__PLUGINDIRECTORIES__0` | Primary plugin binaries directory inside the container |
+| `STELLAOPS_AUTHORITY__PLUGINS__CONFIGURATIONDIRECTORY` | Path to plugin manifest directory |
+
+For additional options, see `etc/authority.yaml.sample`.
+
+> **Graph Explorer reminder:** When enabling Cartographer or Graph API components, update `etc/authority.yaml` so the `cartographer-service` client includes `properties.serviceIdentity: "cartographer"` and a tenant hint. Authority now rejects `graph:write` tokens that lack this marker, so existing deployments must apply the update before rolling out the new build.
+
+> **Console endpoint reminder:** The Console UI now calls `/console/tenants`, `/console/profile`, and `/console/token/introspect`. Reverse proxies must forward the `X-Stella-Tenant` header (derived from the access token) so Authority can enforce tenancy; audit events are logged under `authority.console.*`. Admin actions obey a five-minute fresh-auth window reported by `/console/profile`, so keep session timeout prompts aligned with that value.
+
+## Key rotation automation (OPS3)
+
+The `key-rotation.sh` helper wraps the `/internal/signing/rotate` endpoint delivered with CORE10. It can run in CI/CD once the new PEM key is staged on the Authority host volume.
+
+```bash
+AUTHORITY_BOOTSTRAP_KEY=$(cat ~/.secrets/authority-bootstrap.key) \
+./key-rotation.sh \
+  --authority-url https://authority.stella-ops.local \
+  --key-id authority-signing-2025 \
+  --key-path ../certificates/authority-signing-2025.pem \
+  --meta rotatedBy=pipeline --meta changeTicket=OPS-1234
+```
+
+- `--key-path` should resolve from the Authority content root (same as `docs/11_AUTHORITY.md` SOP).
+- Provide `--source`/`--provider` if the key loader differs from the default file-based provider.
+- Pass `--dry-run` during rehearsals to inspect the JSON payload without invoking the API.
+
+After rotation, export a fresh revocation bundle (`stellaops-cli auth revoke export`) so downstream mirrors consume signatures from the new `kid`. The canonical operational steps live in `docs/11_AUTHORITY.md` – make sure any local automation keeps that guide as source of truth.
--- a/devops/services/authority/TASKS.completed.md
+++ b/devops/services/authority/TASKS.completed.md
@@ -0,0 +1,5 @@
+# Completed Tasks
+
+| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
+| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
+| OPS3.KEY-ROTATION | DONE (2025-10-12) | DevOps Crew, Authority Core | CORE10.JWKS | Implement key rotation tooling + pipeline hook once rotating JWKS lands. Document SOP and secret handling. | ✅ CLI/script rotates keys + updates JWKS; ✅ Pipeline job documented; ✅ docs/ops runbook updated. |
--- a/devops/services/authority/docker-compose.authority.yaml
+++ b/devops/services/authority/docker-compose.authority.yaml
@@ -0,0 +1,58 @@
+version: "3.9"
+
+services:
+  authority:
+    build:
+      context: ../..
+      dockerfile: ops/authority/Dockerfile
+    image: stellaops-authority:dev
+    container_name: stellaops-authority
+    depends_on:
+      mongo:
+        condition: service_started
+    environment:
+      # Override issuer to match your deployment URL.
+      STELLAOPS_AUTHORITY__ISSUER: "https://authority.localtest.me"
+      # Point the Authority host at the Mongo instance defined below.
+      STELLAOPS_AUTHORITY__PLUGINDIRECTORIES__0: "/app/plugins"
+      STELLAOPS_AUTHORITY__PLUGINS__CONFIGURATIONDIRECTORY: "/app/etc/authority.plugins"
+    volumes:
+      # Mount Authority configuration + plugins (edit etc/authority.yaml before running).
+      - ../../etc/authority.yaml:/etc/authority.yaml:ro
+      - ../../etc/authority.plugins:/app/etc/authority.plugins:ro
+      # Optional: persist plugin binaries or key material outside the container.
+      - authority-keys:/app/keys
+    ports:
+      - "8080:8080"
+    restart: unless-stopped
+
+  mongo:
+    image: mongo:7
+    container_name: stellaops-authority-mongo
+    command: ["mongod", "--bind_ip_all"]
+    environment:
+      MONGO_INITDB_ROOT_USERNAME: stellaops
+      MONGO_INITDB_ROOT_PASSWORD: stellaops
+    volumes:
+      - mongo-data:/data/db
+    ports:
+      - "27017:27017"
+    restart: unless-stopped
+
+  valkey:
+    image: valkey/valkey:8-alpine
+    container_name: stellaops-authority-valkey
+    command: ["valkey-server", "--save", "60", "1"]
+    volumes:
+      - valkey-data:/data
+    ports:
+      - "6379:6379"
+    restart: unless-stopped
+    # Uncomment to enable if/when Authority consumes Valkey.
+    # deploy:
+    #   replicas: 0
+
+volumes:
+  mongo-data:
+  valkey-data:
+  authority-keys:
--- a/devops/services/authority/key-rotation.sh
+++ b/devops/services/authority/key-rotation.sh
@@ -0,0 +1,189 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+usage() {
+  cat <<'USAGE'
+Usage: key-rotation.sh --authority-url URL --api-key TOKEN --key-id ID --key-path PATH [options]
+
+Required flags:
+  -u, --authority-url   Base Authority URL (e.g. https://authority.example.com)
+  -k, --api-key         Bootstrap API key (x-stellaops-bootstrap-key header)
+  -i, --key-id          Identifier (kid) for the new signing key
+  -p, --key-path        Path (relative to Authority content root or absolute) where the PEM key lives
+
+Optional flags:
+  -s, --source          Key source loader identifier (default: file)
+  -a, --algorithm       Signing algorithm (default: ES256)
+      --provider        Preferred crypto provider name
+  -m, --meta key=value  Additional metadata entries for the rotation record (repeatable)
+      --dry-run         Print the JSON payload instead of invoking the API
+  -h, --help            Show this help
+
+Environment fallbacks:
+  AUTHORITY_URL, AUTHORITY_BOOTSTRAP_KEY, AUTHORITY_KEY_SOURCE, AUTHORITY_KEY_PROVIDER
+
+Example:
+  AUTHORITY_BOOTSTRAP_KEY=$(cat key.txt) \\
+    ./key-rotation.sh -u https://authority.local \\
+      -i authority-signing-2025 \\
+      -p ../certificates/authority-signing-2025.pem \\
+      -m rotatedBy=pipeline -m ticket=OPS-1234
+USAGE
+}
+
+require_python() {
+  if command -v python3 >/dev/null 2>&1; then
+    PYTHON_BIN=python3
+  elif command -v python >/dev/null 2>&1; then
+    PYTHON_BIN=python
+  else
+    echo "error: python3 (or python) is required for JSON encoding" >&2
+    exit 1
+  fi
+}
+
+json_quote() {
+  "$PYTHON_BIN" - "$1" <<'PY'
+import json, sys
+print(json.dumps(sys.argv[1]))
+PY
+}
+
+AUTHORITY_URL="${AUTHORITY_URL:-}"
+API_KEY="${AUTHORITY_BOOTSTRAP_KEY:-}"
+KEY_ID=""
+KEY_PATH=""
+SOURCE="${AUTHORITY_KEY_SOURCE:-file}"
+ALGORITHM="ES256"
+PROVIDER="${AUTHORITY_KEY_PROVIDER:-}"
+DRY_RUN=false
+declare -a METADATA=()
+
+while [[ $# -gt 0 ]]; do
+  case "$1" in
+    -u|--authority-url)
+      AUTHORITY_URL="$2"
+      shift 2
+      ;;
+    -k|--api-key)
+      API_KEY="$2"
+      shift 2
+      ;;
+    -i|--key-id)
+      KEY_ID="$2"
+      shift 2
+      ;;
+    -p|--key-path)
+      KEY_PATH="$2"
+      shift 2
+      ;;
+    -s|--source)
+      SOURCE="$2"
+      shift 2
+      ;;
+    -a|--algorithm)
+      ALGORITHM="$2"
+      shift 2
+      ;;
+    --provider)
+      PROVIDER="$2"
+      shift 2
+      ;;
+    -m|--meta)
+      METADATA+=("$2")
+      shift 2
+      ;;
+    --dry-run)
+      DRY_RUN=true
+      shift
+      ;;
+    -h|--help)
+      usage
+      exit 0
+      ;;
+    *)
+      echo "Unknown option: $1" >&2
+      usage
+      exit 1
+      ;;
+  esac
+done
+
+if [[ -z "$AUTHORITY_URL" || -z "$API_KEY" || -z "$KEY_ID" || -z "$KEY_PATH" ]]; then
+  echo "error: missing required arguments" >&2
+  usage
+  exit 1
+fi
+
+case "$AUTHORITY_URL" in
+  http://*|https://*) ;;
+  *)
+    echo "error: --authority-url must include scheme (http/https)" >&2
+    exit 1
+    ;;
+esac
+
+require_python
+
+payload="{"
+payload+="\"keyId\":$(json_quote "$KEY_ID"),"
+payload+="\"location\":$(json_quote "$KEY_PATH"),"
+payload+="\"source\":$(json_quote "$SOURCE"),"
+payload+="\"algorithm\":$(json_quote "$ALGORITHM"),"
+if [[ -n "$PROVIDER" ]]; then
+  payload+="\"provider\":$(json_quote "$PROVIDER"),"
+fi
+
+if [[ ${#METADATA[@]} -gt 0 ]]; then
+  payload+="\"metadata\":{"
+  for entry in "${METADATA[@]}"; do
+    if [[ "$entry" != *=* ]]; then
+      echo "warning: ignoring metadata entry '$entry' (expected key=value)" >&2
+      continue
+    fi
+    key="${entry%%=*}"
+    value="${entry#*=}"
+    payload+="$(json_quote "$key"):$(json_quote "$value"),"
+  done
+  if [[ "${payload: -1}" == "," ]]; then
+    payload="${payload::-1}"
+  fi
+  payload+="},"
+fi
+
+if [[ "${payload: -1}" == "," ]]; then
+  payload="${payload::-1}"
+fi
+payload+="}"
+
+if [[ "$DRY_RUN" == true ]]; then
+  echo "# Dry run payload:"
+  echo "$payload"
+  exit 0
+fi
+
+tmp_response="$(mktemp)"
+cleanup() { rm -f "$tmp_response"; }
+trap cleanup EXIT
+
+http_code=$(curl -sS -o "$tmp_response" -w "%{http_code}" \
+  -X POST "${AUTHORITY_URL%/}/internal/signing/rotate" \
+  -H "Content-Type: application/json" \
+  -H "x-stellaops-bootstrap-key: $API_KEY" \
+  --data "$payload")
+
+if [[ "$http_code" != "200" && "$http_code" != "201" ]]; then
+  echo "error: rotation API returned HTTP $http_code" >&2
+  cat "$tmp_response" >&2 || true
+  exit 1
+fi
+
+echo "Rotation request accepted (HTTP $http_code). Response:"
+cat "$tmp_response"
+
+echo
+echo "Fetching JWKS to confirm active key..."
+curl -sS "${AUTHORITY_URL%/}/jwks" || true
+echo
+echo "Done. Remember to update authority.yaml with the new key metadata to keep restarts consistent."