CD/CD consolidation

2025-12-26 17:32:23 +02:00
parent a866eb6277
commit c786faae84
638 changed files with 3821 additions and 181 deletions
--- a/devops/services/aoc/aoc-ci.md
+++ b/devops/services/aoc/aoc-ci.md
@@ -0,0 +1,25 @@
+# AOC Analyzer CI Contract (DEVOPS-AOC-19-001)
+
+## Scope
+Integrate AOC Roslyn analyzer and guard tests into CI to block banned writes in ingestion projects.
+
+## Steps
+1) Restore & build analyzers
+   - `dotnet restore src/Aoc/__Analyzers/StellaOps.Aoc.Analyzers/StellaOps.Aoc.Analyzers.csproj`
+   - `dotnet build src/Aoc/__Analyzers/StellaOps.Aoc.Analyzers/StellaOps.Aoc.Analyzers.csproj -c Release`
+2) Run analyzer on ingestion projects (Authority/Concelier/Excititor ingest paths)
+   - `dotnet build src/Concelier/StellaOps.Concelier.Ingestion/StellaOps.Concelier.Ingestion.csproj -c Release /p:RunAnalyzers=true /p:TreatWarningsAsErrors=true`
+   - `dotnet build src/Authority/StellaOps.Authority.Ingestion/StellaOps.Authority.Ingestion.csproj -c Release /p:RunAnalyzers=true /p:TreatWarningsAsErrors=true`
+   - `dotnet build src/Excititor/StellaOps.Excititor.Ingestion/StellaOps.Excititor.Ingestion.csproj -c Release /p:RunAnalyzers=true /p:TreatWarningsAsErrors=true`
+3) Guard tests
+   - `dotnet test src/Aoc/__Tests/StellaOps.Aoc.Analyzers.Tests/StellaOps.Aoc.Analyzers.Tests.csproj -c Release`
+4) Artefacts
+   - Upload `.artifacts/aoc-analyzer.log` and test TRX.
+
+## Determinism/Offline
+- Use local feeds (`local-nugets/`); no external fetches post-restore.
+- Build with `/p:ContinuousIntegrationBuild=true`.
+
+## Acceptance
+- CI fails on any analyzer warning in ingestion projects.
+- Tests pass; artefacts uploaded.
--- a/devops/services/aoc/aoc-verify-stage.md
+++ b/devops/services/aoc/aoc-verify-stage.md
@@ -0,0 +1,22 @@
+# AOC Verify Stage (DEVOPS-AOC-19-002)
+
+## Purpose
+Add CI stage to run `stella aoc verify --since <commit>` against seeded Mongo snapshots for Concelier + Excititor, publishing violation reports.
+
+## Inputs
+- `STAGING_MONGO_URI` (read-only snapshot).
+- Optional `AOC_VERIFY_SINCE` (defaults to `HEAD~1`).
+
+## Steps
+1) Seed snapshot (if needed)
+   - Restore snapshot into local Mongo or point to read-only staging snapshot.
+2) Run verify
+   - `dotnet run --project src/Aoc/StellaOps.Aoc.Cli -- verify --since ${AOC_VERIFY_SINCE:-HEAD~1} --mongo $STAGING_MONGO_URI --output .artifacts/aoc-verify.json`
+3) Fail on violations
+   - Parse `.artifacts/aoc-verify.json`; if `violations > 0`, fail with summary.
+4) Publish artifacts
+   - Upload `.artifacts/aoc-verify.json` and `.artifacts/aoc-verify.ndjson` (per-violation).
+
+## Acceptance
+- Stage fails when violations exist; passes clean otherwise.
+- Artifacts attached for auditing.
--- a/devops/services/aoc/backfill-release-plan.md
+++ b/devops/services/aoc/backfill-release-plan.md
@@ -0,0 +1,73 @@
+# AOC Backfill Release Plan (DEVOPS-STORE-AOC-19-005-REL)
+
+Scope: Release/offline-kit packaging for Concelier AOC backfill operations.
+
+## Prerequisites
+- Dataset hash from dev rehearsal (AOC-19-005 dev outputs)
+- AOC guard tests passing (DEVOPS-AOC-19-001/002/003 - DONE)
+- Supersedes rollout plan reviewed (ops/devops/aoc/supersedes-rollout.md)
+
+## Artefacts
+- Backfill runner bundle:
+  - `aoc-backfill-runner.tar.gz` - CLI tool + scripts
+  - `aoc-backfill-runner.sbom.json` - SPDX SBOM
+  - `aoc-backfill-runner.dsse.json` - Cosign attestation
+- Dataset bundle:
+  - `aoc-dataset-{hash}.tar.gz` - Seeded dataset
+  - `aoc-dataset-{hash}.manifest.json` - Manifest with checksums
+  - `aoc-dataset-{hash}.provenance.json` - SLSA provenance
+- Offline kit slice:
+  - All above + SHA256SUMS + verification scripts
+
+## Packaging Script
+
+```bash
+# Production (CI with secrets)
+./ops/devops/aoc/package-backfill-release.sh
+
+# Development (dev key)
+COSIGN_ALLOW_DEV_KEY=1 COSIGN_PASSWORD=stellaops-dev \
+  DATASET_HASH=dev-rehearsal-placeholder \
+  ./ops/devops/aoc/package-backfill-release.sh
+```
+
+## Pipeline Outline
+1) Build backfill runner from `src/Aoc/StellaOps.Aoc.Cli/`
+2) Generate SBOM with syft
+3) Sign with cosign (dev key fallback)
+4) Package dataset (when hash available)
+5) Create offline bundle with checksums
+6) Verification:
+   - `stella aoc verify --dry-run`
+   - `cosign verify-blob` for all bundles
+   - `sha256sum --check`
+7) Publish to release bucket + offline kit
+
+## Runbook
+1) Validate AOC guard tests pass in CI
+2) Run dev rehearsal with test dataset
+3) Capture dataset hash from rehearsal
+4) Execute packaging script with production key
+5) Verify all signatures and checksums
+6) Upload to release bucket
+7) Include in offline kit manifest
+
+## CI Workflow
+`.gitea/workflows/aoc-backfill-release.yml`
+
+## Verification
+```bash
+# Verify bundle signatures
+cosign verify-blob \
+  --key tools/cosign/cosign.dev.pub \
+  --bundle out/aoc/aoc-backfill-runner.dsse.json \
+  out/aoc/aoc-backfill-runner.tar.gz
+
+# Verify checksums
+cd out/aoc && sha256sum -c SHA256SUMS
+```
+
+## Owners
+- DevOps Guild (pipeline + packaging)
+- Concelier Storage Guild (dataset + backfill logic)
+- Platform Security (signing policy)
--- a/devops/services/aoc/package-backfill-release.sh
+++ b/devops/services/aoc/package-backfill-release.sh
@@ -0,0 +1,175 @@
+#!/usr/bin/env bash
+# Package AOC backfill release for offline kit
+# Usage: ./package-backfill-release.sh
+# Dev mode: COSIGN_ALLOW_DEV_KEY=1 COSIGN_PASSWORD=stellaops-dev DATASET_HASH=dev ./package-backfill-release.sh
+
+set -euo pipefail
+
+ROOT=$(cd "$(dirname "$0")/../../.." && pwd)
+OUT_DIR="${OUT_DIR:-$ROOT/out/aoc}"
+CREATED="${CREATED:-$(date -u +%Y-%m-%dT%H:%M:%SZ)}"
+DATASET_HASH="${DATASET_HASH:-}"
+
+mkdir -p "$OUT_DIR"
+
+echo "==> AOC Backfill Release Packaging"
+echo "    Output: $OUT_DIR"
+echo "    Dataset hash: ${DATASET_HASH:-<pending>}"
+
+# Key resolution (same pattern as advisory-ai packaging)
+resolve_key() {
+  if [[ -n "${COSIGN_KEY_FILE:-}" && -f "$COSIGN_KEY_FILE" ]]; then
+    echo "$COSIGN_KEY_FILE"
+  elif [[ -n "${COSIGN_PRIVATE_KEY_B64:-}" ]]; then
+    local tmp_key="$OUT_DIR/.cosign.key"
+    echo "$COSIGN_PRIVATE_KEY_B64" | base64 -d > "$tmp_key"
+    chmod 600 "$tmp_key"
+    echo "$tmp_key"
+  elif [[ -f "$ROOT/tools/cosign/cosign.key" ]]; then
+    echo "$ROOT/tools/cosign/cosign.key"
+  elif [[ "${COSIGN_ALLOW_DEV_KEY:-0}" == "1" && -f "$ROOT/tools/cosign/cosign.dev.key" ]]; then
+    echo "[info] Using development key (non-production)" >&2
+    echo "$ROOT/tools/cosign/cosign.dev.key"
+  else
+    echo "[error] No signing key available. Set COSIGN_PRIVATE_KEY_B64 or COSIGN_ALLOW_DEV_KEY=1" >&2
+    return 1
+  fi
+}
+
+# Build AOC CLI if not already built
+AOC_CLI_PROJECT="$ROOT/src/Aoc/StellaOps.Aoc.Cli/StellaOps.Aoc.Cli.csproj"
+AOC_CLI_OUT="$OUT_DIR/cli"
+
+if [[ -f "$AOC_CLI_PROJECT" ]]; then
+  echo "==> Building AOC CLI..."
+  dotnet publish "$AOC_CLI_PROJECT" \
+    -c Release \
+    -o "$AOC_CLI_OUT" \
+    --no-restore 2>/dev/null || echo "[info] Build skipped (may need restore)"
+else
+  echo "[info] AOC CLI project not found; using placeholder"
+  mkdir -p "$AOC_CLI_OUT"
+  echo "AOC CLI placeholder - build from src/Aoc/StellaOps.Aoc.Cli/" > "$AOC_CLI_OUT/README.txt"
+fi
+
+# Create backfill runner bundle
+echo "==> Creating backfill runner bundle..."
+RUNNER_TAR="$OUT_DIR/aoc-backfill-runner.tar.gz"
+tar -czf "$RUNNER_TAR" -C "$AOC_CLI_OUT" .
+
+# Compute hash
+sha256() {
+  sha256sum "$1" | awk '{print $1}'
+}
+RUNNER_HASH=$(sha256 "$RUNNER_TAR")
+
+# Generate manifest
+echo "==> Generating manifest..."
+MANIFEST="$OUT_DIR/aoc-backfill-runner.manifest.json"
+cat > "$MANIFEST" <<EOF
+{
+  "schemaVersion": "1.0.0",
+  "created": "$CREATED",
+  "runner": {
+    "path": "aoc-backfill-runner.tar.gz",
+    "sha256": "$RUNNER_HASH",
+    "size": $(stat -c%s "$RUNNER_TAR" 2>/dev/null || stat -f%z "$RUNNER_TAR")
+  },
+  "dataset": {
+    "hash": "${DATASET_HASH:-pending}",
+    "status": "$( [[ -n "$DATASET_HASH" ]] && echo "available" || echo "pending-dev-rehearsal" )"
+  },
+  "signing": {
+    "mode": "$( [[ "${COSIGN_ALLOW_DEV_KEY:-0}" == "1" ]] && echo "development" || echo "production" )"
+  }
+}
+EOF
+
+# Sign with cosign if available
+KEY_FILE=$(resolve_key) || true
+COSIGN="${COSIGN:-$ROOT/tools/cosign/cosign}"
+DSSE_OUT="$OUT_DIR/aoc-backfill-runner.dsse.json"
+
+if [[ -n "${KEY_FILE:-}" ]]; then
+  COSIGN_CMD="${COSIGN:-cosign}"
+  if command -v cosign &>/dev/null; then
+    COSIGN_CMD="cosign"
+  fi
+
+  echo "==> Signing bundle..."
+  COSIGN_PASSWORD="${COSIGN_PASSWORD:-}" "$COSIGN_CMD" sign-blob \
+    --key "$KEY_FILE" \
+    --bundle "$DSSE_OUT" \
+    --tlog-upload=false \
+    --yes \
+    "$RUNNER_TAR" 2>/dev/null || echo "[info] DSSE signing skipped"
+fi
+
+# Generate SBOM placeholder
+echo "==> Generating SBOM..."
+SBOM="$OUT_DIR/aoc-backfill-runner.sbom.json"
+cat > "$SBOM" <<EOF
+{
+  "spdxVersion": "SPDX-2.3",
+  "dataLicense": "CC0-1.0",
+  "SPDXID": "SPDXRef-DOCUMENT",
+  "name": "aoc-backfill-runner",
+  "documentNamespace": "https://stella-ops.org/sbom/aoc-backfill-runner/$CREATED",
+  "creationInfo": {
+    "created": "$CREATED",
+    "creators": ["Tool: stellaops-aoc-packager"]
+  },
+  "packages": [
+    {
+      "name": "StellaOps.Aoc.Cli",
+      "SPDXID": "SPDXRef-Package-aoc-cli",
+      "downloadLocation": "NOASSERTION",
+      "filesAnalyzed": false
+    }
+  ]
+}
+EOF
+
+# Generate provenance
+echo "==> Generating provenance..."
+PROVENANCE="$OUT_DIR/aoc-backfill-runner.provenance.json"
+cat > "$PROVENANCE" <<EOF
+{
+  "_type": "https://in-toto.io/Statement/v1",
+  "subject": [
+    {
+      "name": "aoc-backfill-runner.tar.gz",
+      "digest": {"sha256": "$RUNNER_HASH"}
+    }
+  ],
+  "predicateType": "https://slsa.dev/provenance/v1",
+  "predicate": {
+    "buildDefinition": {
+      "buildType": "https://stella-ops.org/aoc-backfill-release/v1",
+      "internalParameters": {
+        "created": "$CREATED",
+        "datasetHash": "${DATASET_HASH:-pending}"
+      }
+    },
+    "runDetails": {
+      "builder": {"id": "https://stella-ops.org/aoc-backfill-release"}
+    }
+  }
+}
+EOF
+
+# Generate checksums
+echo "==> Generating checksums..."
+cd "$OUT_DIR"
+sha256sum aoc-backfill-runner.tar.gz aoc-backfill-runner.manifest.json aoc-backfill-runner.sbom.json > SHA256SUMS
+
+# Cleanup temp key
+[[ -f "$OUT_DIR/.cosign.key" ]] && rm -f "$OUT_DIR/.cosign.key"
+
+echo "==> AOC backfill packaging complete"
+echo "    Runner: $RUNNER_TAR"
+echo "    Manifest: $MANIFEST"
+echo "    SBOM: $SBOM"
+echo "    Provenance: $PROVENANCE"
+echo "    Checksums: $OUT_DIR/SHA256SUMS"
+[[ -f "$DSSE_OUT" ]] && echo "    DSSE: $DSSE_OUT"
--- a/devops/services/aoc/supersedes-rollout.md
+++ b/devops/services/aoc/supersedes-rollout.md
@@ -0,0 +1,50 @@
+# Supersedes backfill rollout plan (DEVOPS-AOC-19-101)
+
+Scope: Concelier Link-Not-Merge backfill and supersedes processing once advisory_raw idempotency index is in staging.
+
+## Preconditions
+- Idempotency index verified in staging (`advisory_raw` duplicate inserts rejected; log hash recorded).
+- LNM migrations 21-101/102 applied (shards, TTL, tombstones).
+- Event transport to NATS/Redis disabled during backfill to avoid noisy downstream replays.
+- Offline kit mirror includes current hashes for `advisory_raw` and backfill bundle.
+
+## Rollout steps (staging → prod)
+1) **Freeze window** (announce 24h prior)
+   - Pause Concelier ingest workers (`CONCELIER_INGEST_ENABLED=false`).
+   - Stop outbox publisher or point to blackhole NATS subject.
+2) **Dry-run (staging)**
+   - Run backfill job with `--dry-run` to emit counts only.
+   - Verify: new supersedes records count == expected; no write errors; idempotency violations = 0.
+   - Capture logs + SHA256 of generated report.
+3) **Prod execution**
+   - Run backfill job with `--batch-size=500` and `--stop-on-error`.
+   - Monitor: insert rate, error rate, Mongo oplog lag; target <5% CPU on primary.
+4) **Validation**
+   - Run consistency check:
+     - `advisory_observations` count stable (no drop).
+     - Supersedes edges present for all prior conflicts.
+     - Idempotency index hit rate <0.1%.
+   - Run API spot check: `/advisories/summary` returns supersedes metadata; `advisory.linkset.updated` events absent during freeze.
+5) **Unfreeze**
+   - Re-enable ingest + outbox publisher.
+   - Trigger single `advisory.observation.updated@1` replay to confirm event path is healthy.
+
+## Rollback
+- If errors >0 or idempotency violations observed:
+  - Stop job, keep ingest paused.
+  - Run rollback script `ops/devops/scripts/rollback-lnm-backfill.js` to remove supersedes/tombstones inserted in current window.
+  - Restore Mongo from last checkpointed snapshot if rollback script fails.
+
+## Evidence to capture
+- Job command + arguments.
+- SHA256 of backfill bundle and report.
+- Idempotency violation count.
+- Post-run consistency report (JSON) stored under `ops/devops/artifacts/aoc-supersedes/<timestamp>/`.
+
+## Monitoring/Alerts
+- Add temporary Grafana panel for idempotency violations and Mongo ops/sec during job.
+- Alert if job runtime exceeds 2h or if oplog lag > 60s.
+
+## Owners
+- Run: DevOps Guild
+- Approvals: Concelier Storage Guild + Platform Security