CD/CD consolidation

2025-12-26 17:32:23 +02:00
parent a866eb6277
commit c786faae84
638 changed files with 3821 additions and 181 deletions
--- a/devops/services/console/Dockerfile.runner
+++ b/devops/services/console/Dockerfile.runner
@@ -0,0 +1,38 @@
+# syntax=docker/dockerfile:1.7
+# Offline-friendly console CI runner image with pre-baked npm and Playwright caches (DEVOPS-CONSOLE-23-001)
+ARG BASE_IMAGE=node:20-bookworm-slim
+ARG APP_DIR=src/Web/StellaOps.Web
+ARG SOURCE_DATE_EPOCH=1704067200
+
+FROM ${BASE_IMAGE}
+
+ENV DEBIAN_FRONTEND=noninteractive \
+    NPM_CONFIG_FUND=false \
+    NPM_CONFIG_AUDIT=false \
+    NPM_CONFIG_PROGRESS=false \
+    SOURCE_DATE_EPOCH=${SOURCE_DATE_EPOCH} \
+    PLAYWRIGHT_BROWSERS_PATH=/home/node/.cache/ms-playwright \
+    NPM_CONFIG_CACHE=/home/node/.npm \
+    CI=true
+
+RUN apt-get update && \
+    apt-get install -y --no-install-recommends git ca-certificates dumb-init wget curl && \
+    rm -rf /var/lib/apt/lists/*
+
+WORKDIR /tmp/console-seed
+COPY ${APP_DIR}/package.json ${APP_DIR}/package-lock.json ./
+
+ENV npm_config_cache=/tmp/npm-cache
+RUN npm ci --cache ${npm_config_cache} --prefer-offline --no-audit --progress=false --ignore-scripts && \
+    PLAYWRIGHT_BROWSERS_PATH=/tmp/ms-playwright npx playwright install chromium --with-deps && \
+    rm -rf node_modules
+
+RUN install -d -o node -g node /home/node/.npm /home/node/.cache && \
+    mv /tmp/npm-cache /home/node/.npm && \
+    mv /tmp/ms-playwright /home/node/.cache/ms-playwright && \
+    chown -R node:node /home/node/.npm /home/node/.cache
+
+WORKDIR /workspace
+USER node
+ENTRYPOINT ["/usr/bin/dumb-init","--"]
+CMD ["/bin/bash"]
--- a/devops/services/console/README.md
+++ b/devops/services/console/README.md
@@ -0,0 +1,44 @@
+# Console CI runner (offline-friendly)
+
+Status: runner spec + CI now wired to PRs; runner image scaffold + CI build workflow now available with baked npm + Playwright cache.
+
+## Runner profile
+- OS: Ubuntu 22.04 LTS (x86_64) with Docker available for Playwright deps if needed.
+- Node: 20.x (LTS). Enable corepack; prefer npm (default) to avoid extra downloads.
+- Caches:
+  - npm: `~/.npm` keyed by `src/Web/package-lock.json` hash.
+  - Playwright: `~/.cache/ms-playwright` pre-seeded with Chromium so `npm test -- --browsers=ChromeHeadless` can run offline. Seed once using `npx playwright install chromium` on a connected runner, then snapshot the directory into the runner image.
+  - Angular build cache: optional `~/.cache/angular` if using angular.json cache; safe to keep.
+- Artifacts retention: keep lint/test/build outputs 14 days; limit to 500 MB per run (coverage + dist + test reports). Artifacts path: `artifacts/` (dist, coverage, junit/trx if produced).
+
+## Pipeline steps (expected)
+1) Checkout
+2) Node 20 setup with npm cache restore (package-lock at `src/Web/package-lock.json`).
+3) Install: `npm ci --prefer-offline --no-audit --progress=false` in `src/Web`.
+4) Lint: `npm run lint -- --no-progress`.
+5) Unit: `npm test -- --watch=false --browsers=ChromeHeadless --no-progress` (headless Chromium from pre-seeded cache).
+6) Build: `npm run build -- --configuration=production --progress=false`.
+7) Artifact collect: `dist/`, `coverage/`, any `test-results/**`.
+
+## Offline/airgap notes
+- Do not hit external registries during CI; rely on pre-seeded npm mirror or cached tarballs. Runner image should contain npm cache prime. If mirror is used, set `NPM_CONFIG_REGISTRY=https://registry.npmjs.org` equivalent mirror URL inside the runner; default pipeline does not hard-code it.
+- Playwright browsers must be pre-baked; the workflow will not download them.
+
+### Runner image (with baked caches)
+- Dockerfile: `ops/devops/console/Dockerfile.runner` (Node 20, npm cache, Playwright Chromium cache). Builds with `npm ci` + `playwright install chromium --with-deps` during the image build.
+- Build locally: `IMAGE_TAG=stellaops/console-runner:offline OUTPUT_TAR=ops/devops/artifacts/console-runner/console-runner.tar ops/devops/console/build-runner-image.sh`
+  - `OUTPUT_TAR` optional; when set, the script saves the image for airgap transport.
+- Runner expectations: `NPM_CONFIG_CACHE=~/.npm`, `PLAYWRIGHT_BROWSERS_PATH=~/.cache/ms-playwright` (paths already baked). Register the runner with a label (e.g., `console-ci`) and point `.gitea/workflows/console-ci.yml` at that runner pool.
+- CI build helper: `ops/devops/console/build-runner-image-ci.sh` wraps the build, sets a run-scoped tag, emits metadata JSON, and saves a tarball under `ops/devops/artifacts/console-runner/`.
+- CI workflow: `.gitea/workflows/console-runner-image.yml` (manual + path-trigger) builds the runner image and uploads the tarball + metadata as an artifact named `console-runner-image-<run_id>`.
+
+### Seeding Playwright cache (one-time per runner image, host-based option)
+```bash
+ops/devops/console/seed_playwright.sh
+# then bake ~/.cache/ms-playwright into the runner image or mount it on the agent
+```
+
+## How to run
+- PR-triggered via `.gitea/workflows/console-ci.yml`; restrict runners to images with baked Playwright cache.
+- Manual `workflow_dispatch` remains available for dry runs or cache updates.
+- To refresh the runner image, run the `console-runner-image` workflow or execute `ops/devops/console/build-runner-image-ci.sh` locally to generate a tarball and metadata for distribution.
--- a/devops/services/console/build-console-image.sh
+++ b/devops/services/console/build-console-image.sh
@@ -0,0 +1,86 @@
+#!/usr/bin/env bash
+# Build console container image with SBOM and optional attestations
+# Usage: ./build-console-image.sh [tag] [registry]
+# Example: ./build-console-image.sh 2025.10.0-edge ghcr.io/stellaops
+
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_ROOT="$(cd "${SCRIPT_DIR}/../../.." && pwd)"
+
+TAG="${1:-$(date +%Y%m%dT%H%M%S)}"
+REGISTRY="${2:-registry.stella-ops.org/stellaops}"
+IMAGE_NAME="console"
+FULL_IMAGE="${REGISTRY}/${IMAGE_NAME}:${TAG}"
+
+# Freeze timestamps for reproducibility
+export SOURCE_DATE_EPOCH=${SOURCE_DATE_EPOCH:-1704067200}
+
+echo "==> Building console image: ${FULL_IMAGE}"
+
+# Build using the existing Dockerfile.console
+docker build \
+  --file "${REPO_ROOT}/ops/devops/docker/Dockerfile.console" \
+  --build-arg APP_DIR=src/Web/StellaOps.Web \
+  --build-arg APP_PORT=8080 \
+  --tag "${FULL_IMAGE}" \
+  --label "org.opencontainers.image.created=$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
+  --label "org.opencontainers.image.revision=$(git -C "${REPO_ROOT}" rev-parse HEAD 2>/dev/null || echo 'unknown')" \
+  --label "org.opencontainers.image.source=https://github.com/stellaops/stellaops" \
+  --label "org.opencontainers.image.title=StellaOps Console" \
+  --label "org.opencontainers.image.description=StellaOps Angular Console (non-root nginx)" \
+  "${REPO_ROOT}"
+
+# Get digest
+DIGEST=$(docker inspect --format='{{index .RepoDigests 0}}' "${FULL_IMAGE}" 2>/dev/null || echo "${FULL_IMAGE}")
+
+echo "==> Image built: ${FULL_IMAGE}"
+echo "==> Digest: ${DIGEST}"
+
+# Output metadata for CI
+mkdir -p "${SCRIPT_DIR}/../artifacts/console"
+cat > "${SCRIPT_DIR}/../artifacts/console/build-metadata.json" <<EOF
+{
+  "image": "${FULL_IMAGE}",
+  "digest": "${DIGEST}",
+  "tag": "${TAG}",
+  "registry": "${REGISTRY}",
+  "buildTime": "$(date -u +%Y-%m-%dT%H:%M:%SZ)",
+  "gitCommit": "$(git -C "${REPO_ROOT}" rev-parse HEAD 2>/dev/null || echo 'unknown')",
+  "sourceDateEpoch": "${SOURCE_DATE_EPOCH}"
+}
+EOF
+
+echo "==> Build metadata written to ops/devops/artifacts/console/build-metadata.json"
+
+# Generate SBOM if syft is available
+if command -v syft &>/dev/null; then
+  echo "==> Generating SBOM..."
+  syft "${FULL_IMAGE}" -o spdx-json > "${SCRIPT_DIR}/../artifacts/console/console.spdx.json"
+  syft "${FULL_IMAGE}" -o cyclonedx-json > "${SCRIPT_DIR}/../artifacts/console/console.cdx.json"
+  echo "==> SBOMs written to ops/devops/artifacts/console/"
+else
+  echo "==> Skipping SBOM generation (syft not found)"
+fi
+
+# Sign and attest if cosign is available and key is set
+if command -v cosign &>/dev/null; then
+  if [[ -n "${COSIGN_KEY:-}" ]]; then
+    echo "==> Signing image with cosign..."
+    cosign sign --key "${COSIGN_KEY}" "${FULL_IMAGE}"
+
+    if [[ -f "${SCRIPT_DIR}/../artifacts/console/console.spdx.json" ]]; then
+      echo "==> Attesting SBOM..."
+      cosign attest --predicate "${SCRIPT_DIR}/../artifacts/console/console.spdx.json" \
+        --type spdx --key "${COSIGN_KEY}" "${FULL_IMAGE}"
+    fi
+    echo "==> Image signed and attested"
+  else
+    echo "==> Skipping signing (COSIGN_KEY not set)"
+  fi
+else
+  echo "==> Skipping signing (cosign not found)"
+fi
+
+echo "==> Console image build complete"
+echo "    Image: ${FULL_IMAGE}"
--- a/devops/services/console/build-runner-image-ci.sh
+++ b/devops/services/console/build-runner-image-ci.sh
@@ -0,0 +1,44 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+# CI-friendly wrapper to build the console runner image with baked npm/Playwright caches
+# and emit a tarball + metadata for offline distribution.
+#
+# Inputs (env):
+#   RUN_ID           : unique run identifier (default: $GITHUB_RUN_ID or UTC timestamp)
+#   IMAGE_TAG        : optional override of image tag (default: stellaops/console-runner:offline-$RUN_ID)
+#   OUTPUT_TAR       : optional override of tarball path (default: ops/devops/artifacts/console-runner/console-runner-$RUN_ID.tar)
+#   APP_DIR          : optional override of app directory (default: src/Web/StellaOps.Web)
+
+ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)"
+RUN_ID="${RUN_ID:-${GITHUB_RUN_ID:-$(date -u +%Y%m%dT%H%M%SZ)}}"
+APP_DIR="${APP_DIR:-src/Web/StellaOps.Web}"
+IMAGE_TAG="${IMAGE_TAG:-stellaops/console-runner:offline-$RUN_ID}"
+OUTPUT_TAR="${OUTPUT_TAR:-$ROOT/ops/devops/artifacts/console-runner/console-runner-$RUN_ID.tar}"
+META_DIR="$(dirname "$OUTPUT_TAR")"
+META_JSON="$META_DIR/console-runner-$RUN_ID.json"
+
+mkdir -p "$META_DIR"
+
+IMAGE_TAG="$IMAGE_TAG" OUTPUT_TAR="$OUTPUT_TAR" APP_DIR="$APP_DIR" "$ROOT/ops/devops/console/build-runner-image.sh"
+
+digest="$(docker image inspect --format='{{index .RepoDigests 0}}' "$IMAGE_TAG" || true)"
+id="$(docker image inspect --format='{{.Id}}' "$IMAGE_TAG" || true)"
+
+cat > "$META_JSON" <<EOF
+{
+  "run_id": "$RUN_ID",
+  "image_tag": "$IMAGE_TAG",
+  "image_id": "$id",
+  "repo_digest": "$digest",
+  "output_tar": "$(python - <<PY
+import os, sys
+print(os.path.relpath("$OUTPUT_TAR","$ROOT"))
+PY
+)"
+}
+EOF
+
+echo "Built $IMAGE_TAG"
+echo "Saved tarball: $OUTPUT_TAR"
+echo "Metadata: $META_JSON"
--- a/devops/services/console/build-runner-image.sh
+++ b/devops/services/console/build-runner-image.sh
@@ -0,0 +1,29 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+# Builds the offline console CI runner image with baked npm/Playwright caches.
+# IMAGE_TAG: docker tag to produce (default: stellaops/console-runner:offline)
+# OUTPUT_TAR: optional path to save the image tarball for airgap use.
+
+ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)"
+IMAGE_TAG=${IMAGE_TAG:-stellaops/console-runner:offline}
+DOCKERFILE=${DOCKERFILE:-ops/devops/console/Dockerfile.runner}
+APP_DIR=${APP_DIR:-src/Web/StellaOps.Web}
+OUTPUT_TAR=${OUTPUT_TAR:-}
+
+if ! command -v docker >/dev/null 2>&1; then
+  echo "docker not found; install Docker/Podman before building the runner image." >&2
+  exit 1
+fi
+
+docker build -f "$ROOT/$DOCKERFILE" --build-arg APP_DIR="$APP_DIR" -t "$IMAGE_TAG" "$ROOT"
+
+if [[ -n "$OUTPUT_TAR" ]]; then
+  mkdir -p "$(dirname "$OUTPUT_TAR")"
+  docker save "$IMAGE_TAG" -o "$OUTPUT_TAR"
+fi
+
+echo "Runner image built: $IMAGE_TAG"
+if [[ -n "$OUTPUT_TAR" ]]; then
+  echo "Saved tarball: $OUTPUT_TAR"
+fi
--- a/devops/services/console/package-offline-bundle.sh
+++ b/devops/services/console/package-offline-bundle.sh
@@ -0,0 +1,131 @@
+#!/usr/bin/env bash
+# Package console for offline/airgap deployment
+# Usage: ./package-offline-bundle.sh [image] [output-dir]
+# Example: ./package-offline-bundle.sh registry.stella-ops.org/stellaops/console:2025.10.0 ./offline-bundle
+
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_ROOT="$(cd "${SCRIPT_DIR}/../../.." && pwd)"
+
+IMAGE="${1:-registry.stella-ops.org/stellaops/console:latest}"
+OUTPUT_DIR="${2:-${SCRIPT_DIR}/../artifacts/console/offline-bundle}"
+BUNDLE_NAME="console-offline-$(date +%Y%m%dT%H%M%S)"
+
+# Freeze timestamps
+export SOURCE_DATE_EPOCH=${SOURCE_DATE_EPOCH:-1704067200}
+
+echo "==> Creating offline bundle for: ${IMAGE}"
+mkdir -p "${OUTPUT_DIR}"
+
+# Save image as tarball
+IMAGE_TAR="${OUTPUT_DIR}/${BUNDLE_NAME}.tar"
+echo "==> Saving image to ${IMAGE_TAR}..."
+docker save "${IMAGE}" -o "${IMAGE_TAR}"
+
+# Calculate checksums
+echo "==> Generating checksums..."
+cd "${OUTPUT_DIR}"
+sha256sum "${BUNDLE_NAME}.tar" > "${BUNDLE_NAME}.tar.sha256"
+
+# Copy Helm values
+echo "==> Including Helm values overlay..."
+cp "${REPO_ROOT}/deploy/helm/stellaops/values-console.yaml" "${OUTPUT_DIR}/"
+
+# Copy Dockerfile for reference
+cp "${REPO_ROOT}/ops/devops/docker/Dockerfile.console" "${OUTPUT_DIR}/"
+
+# Generate SBOMs if syft available
+if command -v syft &>/dev/null; then
+  echo "==> Generating SBOMs..."
+  syft "${IMAGE}" -o spdx-json > "${OUTPUT_DIR}/${BUNDLE_NAME}.spdx.json"
+  syft "${IMAGE}" -o cyclonedx-json > "${OUTPUT_DIR}/${BUNDLE_NAME}.cdx.json"
+fi
+
+# Create manifest
+cat > "${OUTPUT_DIR}/manifest.json" <<EOF
+{
+  "bundle": "${BUNDLE_NAME}",
+  "image": "${IMAGE}",
+  "imageTarball": "${BUNDLE_NAME}.tar",
+  "checksumFile": "${BUNDLE_NAME}.tar.sha256",
+  "helmValues": "values-console.yaml",
+  "dockerfile": "Dockerfile.console",
+  "sbom": {
+    "spdx": "${BUNDLE_NAME}.spdx.json",
+    "cyclonedx": "${BUNDLE_NAME}.cdx.json"
+  },
+  "createdAt": "$(date -u +%Y-%m-%dT%H:%M:%SZ)",
+  "sourceDateEpoch": "${SOURCE_DATE_EPOCH}"
+}
+EOF
+
+# Create load script
+cat > "${OUTPUT_DIR}/load.sh" <<'LOAD'
+#!/usr/bin/env bash
+# Load console image into local Docker daemon
+set -euo pipefail
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+MANIFEST="${SCRIPT_DIR}/manifest.json"
+
+if [[ ! -f "${MANIFEST}" ]]; then
+  echo "ERROR: manifest.json not found" >&2
+  exit 1
+fi
+
+TARBALL=$(jq -r '.imageTarball' "${MANIFEST}")
+CHECKSUM_FILE=$(jq -r '.checksumFile' "${MANIFEST}")
+
+echo "==> Verifying checksum..."
+cd "${SCRIPT_DIR}"
+sha256sum -c "${CHECKSUM_FILE}"
+
+echo "==> Loading image..."
+docker load -i "${TARBALL}"
+
+IMAGE=$(jq -r '.image' "${MANIFEST}")
+echo "==> Image loaded: ${IMAGE}"
+LOAD
+chmod +x "${OUTPUT_DIR}/load.sh"
+
+# Create README
+cat > "${OUTPUT_DIR}/README.md" <<EOF
+# Console Offline Bundle
+
+This bundle contains the StellaOps Console container image and deployment assets
+for air-gapped environments.
+
+## Contents
+
+- \`${BUNDLE_NAME}.tar\` - Docker image tarball
+- \`${BUNDLE_NAME}.tar.sha256\` - SHA-256 checksum
+- \`values-console.yaml\` - Helm values overlay
+- \`Dockerfile.console\` - Reference Dockerfile
+- \`${BUNDLE_NAME}.spdx.json\` - SPDX SBOM (if generated)
+- \`${BUNDLE_NAME}.cdx.json\` - CycloneDX SBOM (if generated)
+- \`manifest.json\` - Bundle manifest
+- \`load.sh\` - Image load helper script
+
+## Usage
+
+1. Transfer this bundle to the air-gapped environment
+2. Verify checksums: \`sha256sum -c ${BUNDLE_NAME}.tar.sha256\`
+3. Load image: \`./load.sh\` or \`docker load -i ${BUNDLE_NAME}.tar\`
+4. Deploy with Helm: \`helm install stellaops ../stellaops -f values-console.yaml\`
+
+## Image Details
+
+- Image: \`${IMAGE}\`
+- Created: $(date -u +%Y-%m-%dT%H:%M:%SZ)
+- Non-root user: UID 101 (nginx-unprivileged)
+- Port: 8080
+
+## Verification
+
+The image runs as non-root and supports read-only root filesystem.
+Enable \`readOnlyRootFilesystem: true\` in your security context.
+EOF
+
+echo "==> Offline bundle created at: ${OUTPUT_DIR}"
+echo "==> Contents:"
+ls -la "${OUTPUT_DIR}"
--- a/devops/services/console/seed_playwright.sh
+++ b/devops/services/console/seed_playwright.sh
@@ -0,0 +1,22 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+# Seeds the Playwright browser cache for offline console CI runs.
+# Run on a connected runner once, then bake ~/.cache/ms-playwright into the runner image.
+
+ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
+pushd "$ROOT/src/Web" >/dev/null
+
+if ! command -v npx >/dev/null; then
+  echo "npx not found; install Node.js 20+ first" >&2
+  exit 1
+fi
+
+echo "Installing Playwright Chromium to ~/.cache/ms-playwright ..."
+PLAYWRIGHT_BROWSERS_PATH=${PLAYWRIGHT_BROWSERS_PATH:-~/.cache/ms-playwright}
+export PLAYWRIGHT_BROWSERS_PATH
+
+npx playwright install chromium --with-deps
+
+echo "Done. Cache directory: $PLAYWRIGHT_BROWSERS_PATH"
+popd >/dev/null