git.stella-ops.org/docs/ops/concelier-mirror-operations.md

Concelier & Excititor Mirror Operations

This runbook describes how Stella Ops operates the managed mirrors under *.stella-ops.org. It covers Docker Compose and Helm deployment overlays, secret handling for multi-tenant authn, CDN fronting, and the recurring sync pipeline that keeps mirror bundles current.

1. Prerequisites

• Authority access – client credentials (client_id + secret) authorised for concelier.mirror.read and excititor.mirror.read scopes. Secrets live outside git.
• Signed TLS certificates – wildcard or per-domain (mirror-primary, mirror-community). Store them under deploy/compose/mirror-gateway/tls/ or in Kubernetes secrets.
• Mirror gateway credentials – Basic Auth htpasswd files per domain. Generate with htpasswd -B. Operators distribute credentials to downstream consumers.
• Export artifact source – read access to the canonical S3 buckets (or rsync share) that hold concelier JSON bundles and excititor VEX exports.
• Persistent volumes – storage for Concelier job metadata and mirror export trees. For Helm, provision PVCs (concelier-mirror-jobs, concelier-mirror-exports, excititor-mirror-exports, mirror-mongo-data, mirror-minio-data) before rollout.

1.1 Service configuration quick reference

Concelier.WebService exposes the mirror HTTP endpoints once CONCELIER__MIRROR__ENABLED=true. Key knobs:

• CONCELIER__MIRROR__EXPORTROOT – root folder containing export snapshots (<exportId>/mirror/*).
• CONCELIER__MIRROR__ACTIVEEXPORTID – optional explicit export id; otherwise the service auto-falls back to the latest/ symlink or newest directory.
• CONCELIER__MIRROR__REQUIREAUTHENTICATION – default auth requirement; override per domain with CONCELIER__MIRROR__DOMAINS__{n}__REQUIREAUTHENTICATION.
• CONCELIER__MIRROR__MAXINDEXREQUESTSPERHOUR – budget for /concelier/exports/index.json. Domains inherit this value unless they define __MAXDOWNLOADREQUESTSPERHOUR.
• CONCELIER__MIRROR__DOMAINS__{n}__ID – domain identifier matching the exporter manifest; additional keys configure display name and rate budgets.

The service honours Stella Ops Authority when CONCELIER__AUTHORITY__ENABLED=true and ALLOWANONYMOUSFALLBACK=false. Use the bypass CIDR list (CONCELIER__AUTHORITY__BYPASSNETWORKS__*) for in-cluster ingress gateways that terminate Basic Auth. Unauthorized requests emit WWW-Authenticate: Bearer so downstream automation can detect token failures.

Mirror responses carry deterministic cache headers: /index.json returns Cache-Control: public, max-age=60, while per-domain manifests/bundles include Cache-Control: public, max-age=300, immutable. Rate limiting surfaces Retry-After when quotas are exceeded.

1.2 Mirror connector configuration

Downstream Concelier instances ingest published bundles using the StellaOpsMirrorConnector. Operators running the connector in air‑gapped or limited connectivity environments can tune the following options (environment prefix CONCELIER__SOURCES__STELLAOPSMIRROR__):

• BASEADDRESS – absolute mirror root (e.g., https://mirror-primary.stella-ops.org).
• INDEXPATH – relative path to the mirror index (/concelier/exports/index.json by default).
• DOMAINID – mirror domain identifier from the index (primary, community, etc.).
• HTTPTIMEOUT – request timeout; raise when mirrors sit behind slow WAN links.
• SIGNATURE__ENABLED – require detached JWS verification for bundle.json.
• SIGNATURE__KEYID / SIGNATURE__PROVIDER – expected signing key metadata.
• SIGNATURE__PUBLICKEYPATH – PEM fallback used when the mirror key registry is offline.

The connector keeps a per-export fingerprint (bundle digest + generated-at timestamp) and tracks outstanding document IDs. If a scan is interrupted, the next run resumes parse/map work using the stored fingerprint and pending document lists—no network requests are reissued unless the upstream digest changes.

2. Secret & certificate layout

Docker Compose (deploy/compose/docker-compose.mirror.yaml)

• deploy/compose/env/mirror.env.example – copy to .env and adjust quotas or domain IDs.
• deploy/compose/mirror-secrets/ – mount read-only into /run/secrets. Place:
  • concelier-authority-client – Authority client secret.
  • excititor-authority-client (optional) – reserve for future authn.
• deploy/compose/mirror-gateway/tls/ – PEM-encoded cert/key pairs:
  • mirror-primary.crt, mirror-primary.key
  • mirror-community.crt, mirror-community.key
• deploy/compose/mirror-gateway/secrets/ – htpasswd files:
  • mirror-primary.htpasswd
  • mirror-community.htpasswd

Helm (deploy/helm/stellaops/values-mirror.yaml)

Create secrets in the target namespace:

kubectl create secret generic concelier-mirror-auth \
  --from-file=concelier-authority-client=concelier-authority-client

kubectl create secret generic excititor-mirror-auth \
  --from-file=excititor-authority-client=excititor-authority-client

kubectl create secret tls mirror-gateway-tls \
  --cert=mirror-primary.crt --key=mirror-primary.key

kubectl create secret generic mirror-gateway-htpasswd \
  --from-file=mirror-primary.htpasswd --from-file=mirror-community.htpasswd

Keep Basic Auth lists short-lived (rotate quarterly) and document credential recipients.

3. Deployment

3.1 Docker Compose (edge mirrors, lab validation)

1. cp deploy/compose/env/mirror.env.example deploy/compose/env/mirror.env
2. Populate secrets/tls directories as described above.
3. Sync mirror bundles (see §4) into deploy/compose/mirror-data/… and ensure they are mounted on the host path backing the concelier-exports and excititor-exports volumes.
4. Run the profile validator: deploy/tools/validate-profiles.sh.
5. Launch: docker compose --env-file env/mirror.env -f docker-compose.mirror.yaml up -d.

3.2 Helm (production mirrors)

1. Provision PVCs sized for mirror bundles (baseline: 20 GiB per domain).
2. Create secrets/tls config maps (§2).
3. helm upgrade --install mirror deploy/helm/stellaops -f deploy/helm/stellaops/values-mirror.yaml.
4. Annotate the stellaops-mirror-gateway service with ingress/LoadBalancer metadata required by your CDN (e.g., AWS load balancer scheme internal + NLB idle timeout).

4. Artifact sync workflow

Mirrors never generate exports—they ingest signed bundles produced by the Concelier and Excititor export jobs. Recommended sync pattern:

4.1 Compose host (systemd timer)

/usr/local/bin/mirror-sync.sh:

#!/usr/bin/env bash
set -euo pipefail
export AWS_ACCESS_KEY_ID=…
export AWS_SECRET_ACCESS_KEY=…

aws s3 sync s3://mirror-stellaops/concelier/latest \
  /opt/stellaops/mirror-data/concelier --delete --size-only

aws s3 sync s3://mirror-stellaops/excititor/latest \
  /opt/stellaops/mirror-data/excititor --delete --size-only

Schedule with a systemd timer every 5 minutes. The Compose volumes mount /opt/stellaops/mirror-data/* into the containers read-only, matching CONCELIER__MIRROR__EXPORTROOT=/exports/json and EXCITITOR__ARTIFACTS__FILESYSTEM__ROOT=/exports.

4.2 Kubernetes (CronJob)

Create a CronJob running the AWS CLI (or rclone) in the same namespace, writing into the PVCs:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: mirror-sync
spec:
  schedule: "*/5 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: sync
            image: public.ecr.aws/aws-cli/aws-cli@sha256:5df5f52c29f5e3ba46d0ad9e0e3afc98701c4a0f879400b4c5f80d943b5fadea
            command:
              - /bin/sh
              - -c
              - >
                aws s3 sync s3://mirror-stellaops/concelier/latest /exports/concelier --delete --size-only &&
                aws s3 sync s3://mirror-stellaops/excititor/latest /exports/excititor --delete --size-only
            volumeMounts:
              - name: concelier-exports
                mountPath: /exports/concelier
              - name: excititor-exports
                mountPath: /exports/excititor
            envFrom:
              - secretRef:
                  name: mirror-sync-aws
          restartPolicy: OnFailure
          volumes:
            - name: concelier-exports
              persistentVolumeClaim:
                claimName: concelier-mirror-exports
            - name: excititor-exports
              persistentVolumeClaim:
                claimName: excititor-mirror-exports

5. CDN integration

1. Point the CDN origin at the mirror gateway (Compose host or Kubernetes LoadBalancer).
2. Honour the response headers emitted by the gateway and Concelier/Excititor: Cache-Control: public, max-age=300, immutable for mirror payloads.
3. Configure origin shields in the CDN to prevent cache stampedes. Recommended TTLs:
   • Index (/concelier/exports/index.json, /excititor/mirror/*/index) → 60 s.
   • Bundle/manifest payloads → 300 s.
4. Forward the Authorization header—Basic Auth terminates at the gateway.
5. Enforce per-domain rate limits at the CDN (matching gateway budgets) and enable logging to SIEM for anomaly detection.

6. Smoke tests

After each deployment or sync cycle (temporarily set low budgets if you need to observe 429 responses):

# Index with Basic Auth
curl -u $PRIMARY_CREDS https://mirror-primary.stella-ops.org/concelier/exports/index.json | jq 'keys'

# Mirror manifest signature and cache headers
curl -u $PRIMARY_CREDS -I https://mirror-primary.stella-ops.org/concelier/exports/mirror/primary/manifest.json \
  | tee /tmp/manifest-headers.txt
grep -E '^Cache-Control: ' /tmp/manifest-headers.txt   # expect public, max-age=300, immutable

# Excititor consensus bundle metadata
curl -u $COMMUNITY_CREDS https://mirror-community.stella-ops.org/excititor/mirror/community/index \
  | jq '.exports[].exportKey'

# Signed bundle + detached JWS (spot check digests)
curl -u $PRIMARY_CREDS https://mirror-primary.stella-ops.org/concelier/exports/mirror/primary/bundle.json.jws \
  -o bundle.json.jws
cosign verify-blob --signature bundle.json.jws --key mirror-key.pub bundle.json

# Service-level auth check (inside cluster – no gateway credentials)
kubectl exec deploy/stellaops-concelier -- curl -si http://localhost:8443/concelier/exports/mirror/primary/manifest.json \
  | head -n 5   # expect HTTP/1.1 401 with WWW-Authenticate: Bearer

# Rate limit smoke (repeat quickly; second call should return 429 + Retry-After)
for i in 1 2; do
  curl -s -o /dev/null -D - https://mirror-primary.stella-ops.org/concelier/exports/index.json \
    -u $PRIMARY_CREDS | grep -E '^(HTTP/|Retry-After:)'
  sleep 1
done

Watch the gateway metrics (nginx_vts or access logs) for cache hits. In Kubernetes, kubectl logs deploy/stellaops-mirror-gateway should show X-Cache-Status: HIT/MISS.

7. Maintenance & rotation

• Bundle freshness – alert if sync job lag exceeds 15 minutes or if concelier logs Mirror export root is not configured.
• Secret rotation – change Authority client secrets and Basic Auth credentials quarterly. Update the mounted secrets and restart deployments (docker compose restart concelier or kubectl rollout restart deploy/stellaops-concelier).
• TLS renewal – reissue certificates, place new files, and reload gateway (docker compose exec mirror-gateway nginx -s reload).
• Quota tuning – adjust per-domain MAXDOWNLOADREQUESTSPERHOUR in .env or values file. Align CDN rate limits and inform downstreams.

8. References

• Deployment profiles: deploy/compose/docker-compose.mirror.yaml, deploy/helm/stellaops/values-mirror.yaml
• Mirror architecture dossiers: docs/ARCHITECTURE_CONCELIER.md, docs/ARCHITECTURE_EXCITITOR_MIRRORS.md
• Export bundling: docs/ARCHITECTURE_DEVOPS.md §3, docs/ARCHITECTURE_EXCITITOR.md §7