git.stella-ops.org/docs/ops/concelier-mirror-operations.md

Concelier & Excititor Mirror Operations

This runbook describes how Stella Ops operates the managed mirrors under *.stella-ops.org. It covers Docker Compose and Helm deployment overlays, secret handling for multi-tenant authn, CDN fronting, and the recurring sync pipeline that keeps mirror bundles current.

1. Prerequisites

• Authority access – client credentials (client_id + secret) authorised for concelier.mirror.read and excititor.mirror.read scopes. Secrets live outside git.
• Signed TLS certificates – wildcard or per-domain (mirror-primary, mirror-community). Store them under deploy/compose/mirror-gateway/tls/ or in Kubernetes secrets.
• Mirror gateway credentials – Basic Auth htpasswd files per domain. Generate with htpasswd -B. Operators distribute credentials to downstream consumers.
• Export artifact source – read access to the canonical S3 buckets (or rsync share) that hold concelier JSON bundles and excititor VEX exports.
• Persistent volumes – storage for Concelier job metadata and mirror export trees. For Helm, provision PVCs (concelier-mirror-jobs, concelier-mirror-exports, excititor-mirror-exports, mirror-mongo-data, mirror-minio-data) before rollout.

2. Secret & certificate layout

Docker Compose (deploy/compose/docker-compose.mirror.yaml)

• deploy/compose/env/mirror.env.example – copy to .env and adjust quotas or domain IDs.
• deploy/compose/mirror-secrets/ – mount read-only into /run/secrets. Place:
  • concelier-authority-client – Authority client secret.
  • excititor-authority-client (optional) – reserve for future authn.
• deploy/compose/mirror-gateway/tls/ – PEM-encoded cert/key pairs:
  • mirror-primary.crt, mirror-primary.key
  • mirror-community.crt, mirror-community.key
• deploy/compose/mirror-gateway/secrets/ – htpasswd files:
  • mirror-primary.htpasswd
  • mirror-community.htpasswd

Helm (deploy/helm/stellaops/values-mirror.yaml)

Create secrets in the target namespace:

kubectl create secret generic concelier-mirror-auth \
  --from-file=concelier-authority-client=concelier-authority-client

kubectl create secret generic excititor-mirror-auth \
  --from-file=excititor-authority-client=excititor-authority-client

kubectl create secret tls mirror-gateway-tls \
  --cert=mirror-primary.crt --key=mirror-primary.key

kubectl create secret generic mirror-gateway-htpasswd \
  --from-file=mirror-primary.htpasswd --from-file=mirror-community.htpasswd

Keep Basic Auth lists short-lived (rotate quarterly) and document credential recipients.

3. Deployment

3.1 Docker Compose (edge mirrors, lab validation)

1. cp deploy/compose/env/mirror.env.example deploy/compose/env/mirror.env
2. Populate secrets/tls directories as described above.
3. Sync mirror bundles (see §4) into deploy/compose/mirror-data/… and ensure they are mounted on the host path backing the concelier-exports and excititor-exports volumes.
4. Run the profile validator: deploy/tools/validate-profiles.sh.
5. Launch: docker compose --env-file env/mirror.env -f docker-compose.mirror.yaml up -d.

3.2 Helm (production mirrors)

1. Provision PVCs sized for mirror bundles (baseline: 20 GiB per domain).
2. Create secrets/tls config maps (§2).
3. helm upgrade --install mirror deploy/helm/stellaops -f deploy/helm/stellaops/values-mirror.yaml.
4. Annotate the stellaops-mirror-gateway service with ingress/LoadBalancer metadata required by your CDN (e.g., AWS load balancer scheme internal + NLB idle timeout).

4. Artifact sync workflow

Mirrors never generate exports—they ingest signed bundles produced by the Concelier and Excititor export jobs. Recommended sync pattern:

4.1 Compose host (systemd timer)

/usr/local/bin/mirror-sync.sh:

#!/usr/bin/env bash
set -euo pipefail
export AWS_ACCESS_KEY_ID=…
export AWS_SECRET_ACCESS_KEY=…

aws s3 sync s3://mirror-stellaops/concelier/latest \
  /opt/stellaops/mirror-data/concelier --delete --size-only

aws s3 sync s3://mirror-stellaops/excititor/latest \
  /opt/stellaops/mirror-data/excititor --delete --size-only

Schedule with a systemd timer every 5 minutes. The Compose volumes mount /opt/stellaops/mirror-data/* into the containers read-only, matching CONCELIER__MIRROR__EXPORTROOT=/exports/json and EXCITITOR__ARTIFACTS__FILESYSTEM__ROOT=/exports.

4.2 Kubernetes (CronJob)

Create a CronJob running the AWS CLI (or rclone) in the same namespace, writing into the PVCs:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: mirror-sync
spec:
  schedule: "*/5 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: sync
            image: public.ecr.aws/aws-cli/aws-cli@sha256:5df5f52c29f5e3ba46d0ad9e0e3afc98701c4a0f879400b4c5f80d943b5fadea
            command:
              - /bin/sh
              - -c
              - >
                aws s3 sync s3://mirror-stellaops/concelier/latest /exports/concelier --delete --size-only &&
                aws s3 sync s3://mirror-stellaops/excititor/latest /exports/excititor --delete --size-only
            volumeMounts:
              - name: concelier-exports
                mountPath: /exports/concelier
              - name: excititor-exports
                mountPath: /exports/excititor
            envFrom:
              - secretRef:
                  name: mirror-sync-aws
          restartPolicy: OnFailure
          volumes:
            - name: concelier-exports
              persistentVolumeClaim:
                claimName: concelier-mirror-exports
            - name: excititor-exports
              persistentVolumeClaim:
                claimName: excititor-mirror-exports

5. CDN integration

1. Point the CDN origin at the mirror gateway (Compose host or Kubernetes LoadBalancer).
2. Honour the response headers emitted by the gateway and Concelier/Excititor: Cache-Control: public, max-age=300, immutable for mirror payloads.
3. Configure origin shields in the CDN to prevent cache stampedes. Recommended TTLs:
   • Index (/concelier/exports/index.json, /excititor/mirror/*/index) → 60 s.
   • Bundle/manifest payloads → 300 s.
4. Forward the Authorization header—Basic Auth terminates at the gateway.
5. Enforce per-domain rate limits at the CDN (matching gateway budgets) and enable logging to SIEM for anomaly detection.

6. Smoke tests

After each deployment or sync cycle:

# Index with Basic Auth
curl -u $PRIMARY_CREDS https://mirror-primary.stella-ops.org/concelier/exports/index.json | jq 'keys'

# Mirror manifest signature
curl -u $PRIMARY_CREDS -I https://mirror-primary.stella-ops.org/concelier/exports/mirror/primary/manifest.json

# Excititor consensus bundle metadata
curl -u $COMMUNITY_CREDS https://mirror-community.stella-ops.org/excititor/mirror/community/index \
  | jq '.exports[].exportKey'

# Signed bundle + detached JWS (spot check digests)
curl -u $PRIMARY_CREDS https://mirror-primary.stella-ops.org/concelier/exports/mirror/primary/bundle.json.jws \
  -o bundle.json.jws
cosign verify-blob --signature bundle.json.jws --key mirror-key.pub bundle.json

Watch the gateway metrics (nginx_vts or access logs) for cache hits. In Kubernetes, kubectl logs deploy/stellaops-mirror-gateway should show X-Cache-Status: HIT/MISS.

7. Maintenance & rotation

• Bundle freshness – alert if sync job lag exceeds 15 minutes or if concelier logs Mirror export root is not configured.
• Secret rotation – change Authority client secrets and Basic Auth credentials quarterly. Update the mounted secrets and restart deployments (docker compose restart concelier or kubectl rollout restart deploy/stellaops-concelier).
• TLS renewal – reissue certificates, place new files, and reload gateway (docker compose exec mirror-gateway nginx -s reload).
• Quota tuning – adjust per-domain MAXDOWNLOADREQUESTSPERHOUR in .env or values file. Align CDN rate limits and inform downstreams.

8. References

• Deployment profiles: deploy/compose/docker-compose.mirror.yaml, deploy/helm/stellaops/values-mirror.yaml
• Mirror architecture dossiers: docs/ARCHITECTURE_CONCELIER.md, docs/ARCHITECTURE_EXCITITOR_MIRRORS.md
• Export bundling: docs/ARCHITECTURE_DEVOPS.md §3, docs/ARCHITECTURE_EXCITITOR.md §7