git.stella-ops.org/docs/modules/concelier/operations/mirror.md

Concelier & Excititor Mirror Operations

This runbook describes how Stella Ops operates the managed mirrors under *.stella-ops.org. It covers Docker Compose and Helm deployment overlays, secret handling for multi-tenant authn, CDN fronting, and the recurring sync pipeline that keeps mirror bundles current.

1. Prerequisites

• Authority access – client credentials (client_id + secret) authorised for concelier.mirror.read and excititor.mirror.read scopes. Secrets live outside git.
• Signed TLS certificates – wildcard or per-domain (mirror-primary, mirror-community). Store them under devops/compose/mirror-gateway/tls/ or in Kubernetes secrets.
• Mirror gateway credentials – Basic Auth htpasswd files per domain. Generate with htpasswd -B. Operators distribute credentials to downstream consumers.
• Export artifact source – read access to the canonical S3 buckets (or rsync share) that hold concelier JSON bundles and excititor VEX exports.
• Persistent volumes – storage for Concelier job metadata and mirror export trees. For Helm, provision PVCs (concelier-mirror-jobs, concelier-mirror-exports, excititor-mirror-exports) before rollout.

1.1 Service configuration quick reference

Concelier.WebService exposes the mirror HTTP endpoints once CONCELIER__MIRROR__ENABLED=true. Key knobs:

• CONCELIER__MIRROR__EXPORTROOT – root folder containing export snapshots (<exportId>/mirror/*).
• CONCELIER__MIRROR__ACTIVEEXPORTID – optional explicit export id; otherwise the service auto-falls back to the latest/ symlink or newest directory.
• CONCELIER__MIRROR__REQUIREAUTHENTICATION – default auth requirement; override per domain with CONCELIER__MIRROR__DOMAINS__{n}__REQUIREAUTHENTICATION.
• CONCELIER__MIRROR__MAXINDEXREQUESTSPERHOUR – budget for /concelier/exports/index.json. Domains inherit this value unless they define __MAXDOWNLOADREQUESTSPERHOUR.
• CONCELIER__MIRROR__DOMAINS__{n}__ID – domain identifier matching the exporter manifest; additional keys configure display name and rate budgets.

The service honours Stella Ops Authority when CONCELIER__AUTHORITY__ENABLED=true and ALLOWANONYMOUSFALLBACK=false. Use the bypass CIDR list (CONCELIER__AUTHORITY__BYPASSNETWORKS__*) for in-cluster ingress gateways that terminate Basic Auth. Unauthorized requests emit WWW-Authenticate: Bearer so downstream automation can detect token failures.

Mirror responses carry deterministic cache headers: /index.json returns Cache-Control: public, max-age=60, while per-domain manifests/bundles include Cache-Control: public, max-age=300, immutable. Rate limiting surfaces Retry-After when quotas are exceeded.

1.2 Mirror connector configuration

Downstream Concelier instances ingest published bundles using the StellaOpsMirrorConnector. Operators running the connector in air‑gapped or limited connectivity environments can tune the following options (environment prefix CONCELIER__SOURCES__STELLAOPSMIRROR__):

• BASEADDRESS – absolute mirror root (e.g., https://mirror-primary.stella-ops.org).
• INDEXPATH – relative path to the mirror index (/concelier/exports/index.json by default).
• DOMAINID – mirror domain identifier from the index (primary, community, etc.).
• HTTPTIMEOUT – request timeout; raise when mirrors sit behind slow WAN links.
• SIGNATURE__ENABLED – require detached JWS verification for bundle.json.
• SIGNATURE__KEYID / SIGNATURE__PROVIDER – expected signing key metadata.
• SIGNATURE__PUBLICKEYPATH – PEM fallback used when the mirror key registry is offline.

The connector keeps a per-export fingerprint (bundle digest + generated-at timestamp) and tracks outstanding document IDs. If a scan is interrupted, the next run resumes parse/map work using the stored fingerprint and pending document lists—no network requests are reissued unless the upstream digest changes.

2. Secret & certificate layout

Docker Compose (devops/compose/docker-compose.mirror.yaml)

• devops/compose/env/mirror.env.example – copy to .env and adjust quotas or domain IDs.
• devops/compose/mirror-secrets/ – mount read-only into /run/secrets. Place:
  • concelier-authority-client – Authority client secret.
  • excititor-authority-client (optional) – reserve for future authn.
• devops/compose/mirror-gateway/tls/ – PEM-encoded cert/key pairs:
  • mirror-primary.crt, mirror-primary.key
  • mirror-community.crt, mirror-community.key
• devops/compose/mirror-gateway/secrets/ – htpasswd files:
  • mirror-primary.htpasswd
  • mirror-community.htpasswd

Helm (devops/helm/stellaops/values-mirror.yaml)

Create secrets in the target namespace:

kubectl create secret generic concelier-mirror-auth \
  --from-file=concelier-authority-client=concelier-authority-client

kubectl create secret generic excititor-mirror-auth \
  --from-file=excititor-authority-client=excititor-authority-client

kubectl create secret tls mirror-gateway-tls \
  --cert=mirror-primary.crt --key=mirror-primary.key

kubectl create secret generic mirror-gateway-htpasswd \
  --from-file=mirror-primary.htpasswd --from-file=mirror-community.htpasswd

Keep Basic Auth lists short-lived (rotate quarterly) and document credential recipients.

3. Deployment

3.1 Docker Compose (edge mirrors, lab validation)

1. cp devops/compose/env/mirror.env.example devops/compose/env/mirror.env
2. Populate secrets/tls directories as described above.
3. Sync mirror bundles (see §4) into devops/compose/mirror-data/… and ensure they are mounted on the host path backing the concelier-exports and excititor-exports volumes.
4. Run the profile validator: deploy/tools/validate-profiles.sh.
5. Launch: docker compose --env-file env/mirror.env -f docker-compose.mirror.yaml up -d.

3.2 Helm (production mirrors)

1. Provision PVCs sized for mirror bundles (baseline: 20 GiB per domain).
2. Create secrets/tls config maps (§2).
3. helm upgrade --install mirror devops/helm/stellaops -f devops/helm/stellaops/values-mirror.yaml.
4. Annotate the stellaops-mirror-gateway service with ingress/LoadBalancer metadata required by your CDN (e.g., AWS load balancer scheme internal + NLB idle timeout).

4. Artifact sync workflow

Mirrors never generate exports—they ingest signed bundles produced by the Concelier and Excititor export jobs. Recommended sync pattern:

4.1 Compose host (systemd timer)

/usr/local/bin/mirror-sync.sh:

#!/usr/bin/env bash
set -euo pipefail
export AWS_ACCESS_KEY_ID=…
export AWS_SECRET_ACCESS_KEY=…

aws s3 sync s3://mirror-stellaops/concelier/latest \
  /opt/stellaops/mirror-data/concelier --delete --size-only

aws s3 sync s3://mirror-stellaops/excititor/latest \
  /opt/stellaops/mirror-data/excititor --delete --size-only

Schedule with a systemd timer every 5 minutes. The Compose volumes mount /opt/stellaops/mirror-data/* into the containers read-only, matching CONCELIER__MIRROR__EXPORTROOT=/exports/json and EXCITITOR__ARTIFACTS__FILESYSTEM__ROOT=/exports.

4.2 Kubernetes (CronJob)

Create a CronJob running the AWS CLI (or rclone) in the same namespace, writing into the PVCs:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: mirror-sync
spec:
  schedule: "*/5 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: sync
            image: public.ecr.aws/aws-cli/aws-cli@sha256:5df5f52c29f5e3ba46d0ad9e0e3afc98701c4a0f879400b4c5f80d943b5fadea
            command:
              - /bin/sh
              - -c
              - >
                aws s3 sync s3://mirror-stellaops/concelier/latest /exports/concelier --delete --size-only &&
                aws s3 sync s3://mirror-stellaops/excititor/latest /exports/excititor --delete --size-only
            volumeMounts:
              - name: concelier-exports
                mountPath: /exports/concelier
              - name: excititor-exports
                mountPath: /exports/excititor
            envFrom:
              - secretRef:
                  name: mirror-sync-aws
          restartPolicy: OnFailure
          volumes:
            - name: concelier-exports
              persistentVolumeClaim:
                claimName: concelier-mirror-exports
            - name: excititor-exports
              persistentVolumeClaim:
                claimName: excititor-mirror-exports

5. CDN integration

1. Point the CDN origin at the mirror gateway (Compose host or Kubernetes LoadBalancer).
2. Honour the response headers emitted by the gateway and Concelier/Excititor: Cache-Control: public, max-age=300, immutable for mirror payloads.
3. Configure origin shields in the CDN to prevent cache stampedes. Recommended TTLs:
   • Index (/concelier/exports/index.json, /excititor/mirror/*/index) → 60 s.
   • Bundle/manifest payloads → 300 s.
4. Forward the Authorization header—Basic Auth terminates at the gateway.
5. Enforce per-domain rate limits at the CDN (matching gateway budgets) and enable logging to SIEM for anomaly detection.

6. Smoke tests

After each deployment or sync cycle (temporarily set low budgets if you need to observe 429 responses):

# Index with Basic Auth
curl -u $PRIMARY_CREDS https://mirror-primary.stella-ops.org/concelier/exports/index.json | jq 'keys'

# Mirror manifest signature and cache headers
curl -u $PRIMARY_CREDS -I https://mirror-primary.stella-ops.org/concelier/exports/mirror/primary/manifest.json \
  | tee /tmp/manifest-headers.txt
grep -E '^Cache-Control: ' /tmp/manifest-headers.txt   # expect public, max-age=300, immutable

# Excititor consensus bundle metadata
curl -u $COMMUNITY_CREDS https://mirror-community.stella-ops.org/excititor/mirror/community/index \
  | jq '.exports[].exportKey'

# Signed bundle + detached JWS (spot check digests)
curl -u $PRIMARY_CREDS https://mirror-primary.stella-ops.org/concelier/exports/mirror/primary/bundle.json.jws \
  -o bundle.json.jws
cosign verify-blob --signature bundle.json.jws --key mirror-key.pub bundle.json

# Service-level auth check (inside cluster – no gateway credentials)
kubectl exec deploy/stellaops-concelier -- curl -si http://localhost:8443/concelier/exports/mirror/primary/manifest.json \
  | head -n 5   # expect HTTP/1.1 401 with WWW-Authenticate: Bearer

# Rate limit smoke (repeat quickly; second call should return 429 + Retry-After)
for i in 1 2; do
  curl -s -o /dev/null -D - https://mirror-primary.stella-ops.org/concelier/exports/index.json \
    -u $PRIMARY_CREDS | grep -E '^(HTTP/|Retry-After:)'
  sleep 1
done

Watch the gateway metrics (nginx_vts or access logs) for cache hits. In Kubernetes, kubectl logs deploy/stellaops-mirror-gateway should show X-Cache-Status: HIT/MISS.

7. Maintenance & rotation

• Bundle freshness – alert if sync job lag exceeds 15 minutes or if concelier logs Mirror export root is not configured.
• Secret rotation – change Authority client secrets and Basic Auth credentials quarterly. Update the mounted secrets and restart deployments (docker compose restart concelier or kubectl rollout restart deploy/stellaops-concelier).
• TLS renewal – reissue certificates, place new files, and reload gateway (docker compose exec mirror-gateway nginx -s reload).
• Quota tuning – adjust per-domain MAXDOWNLOADREQUESTSPERHOUR in .env or values file. Align CDN rate limits and inform downstreams.

8. Setting up as Mirror Consumer (UI)

Stella Ops provides a guided wizard for configuring an instance as a mirror consumer. The wizard replaces the manual env-var approach for most operators, while the environment variable path (section 1.2) remains available for headless, scripted, and air-gap-first deployments.

8.1 Accessing the wizard

The mirror client setup wizard is available at two entry points:

• Mirror Dashboard -- Navigate to Integrations > Advisory & VEX Sources > Mirror Dashboard. In the consumer panel header, click Configure. If the instance is in Direct mode with no consumer configured, the dashboard shows a "Switch to Mirror" call-to-action card that links directly to the wizard.
• Advisory Source Catalog -- Navigate to Integrations > Advisory & VEX Sources > Catalog. In the mirror context header, click Connect to Mirror (visible when no consumer connection is active).

The wizard route is advisory-vex-sources/mirror/client-setup (lazy-loaded under the Integration Hub).

8.2 Wizard flow (4 steps)

Step 1: Connect to Mirror

Enter the base address of the upstream mirror server (e.g., https://mirror-primary.stella-ops.org).

1. Click Test Connection -- the wizard calls POST /api/v1/mirror/test and reports success (with latency) or failure (with error message and remediation hint).
2. On successful connection, the wizard automatically calls POST /api/v1/mirror/consumer/discover to fetch the mirror index. A domain selector dropdown is populated with all available domains, showing each domain's display name, advisory count, bundle size, export formats, signature status, and last-generated timestamp.
3. Select the target domain from the dropdown.
4. Optionally expand Advanced Settings to override the index path (default: /concelier/exports/index.json) or adjust the HTTP timeout slider (5--300 seconds, default 30s).

Step 2: Signature Verification

On entering this step, the wizard automatically calls POST /api/v1/mirror/consumer/verify-signature to detect whether the selected domain's bundle is signed.

• If a signature is detected, the algorithm and key ID are pre-populated and a green "Signature detected" banner is shown.
• If no signature is detected, a warning banner is shown. You may proceed without verification.
• If the signature is invalid, a red banner is shown.

To configure signature verification manually:

1. Toggle Enable signature verification on.
2. Select the algorithm from the dropdown: ES256, ES384, ES512, RS256, or RS384.
3. Enter the Key ID (e.g., mirror-signing-key-01).
4. Paste the public key in PEM format or leave empty if the mirror key registry will resolve it.
5. Click Verify Sample to download a small bundle chunk and verify the signature against your configuration.

Signature verification is optional but strongly recommended. The wizard defaults to "enabled" when a signed bundle is detected.

Step 3: Sync Schedule & Caching

Operating mode -- choose one:

Mode	Behavior
Mirror	Consumer only. All advisory data comes from the upstream mirror. Direct source connectors are suspended.
Hybrid	Consumer + direct sources. Mirror data augments direct fetching. Both remain active.

A warning banner is shown when selecting Mirror mode, noting that direct source fetching will be disabled. Connectors remain configured and can be reactivated by switching back to Hybrid or Direct mode.

Sync schedule -- choose a preset:

• Manual, Hourly, Every 4 hours, Daily, Weekly

Bundle caching -- toggle on and set a TTL in hours (default: 168 hours / 7 days). Caching reduces bandwidth and improves resilience against transient mirror outages.

Air-gap import -- expand the collapsible "Air-Gap Import" section for offline bundle loading (see section 8.4).

Step 4: Review & Activate

The wizard shows a configuration summary card with all selected settings (mirror URL, domain, index path, timeout, mode, signature configuration, sync schedule, caching).

Pre-flight checks run automatically:

1. Mirror reachable -- re-tests connectivity.
2. Domain exists in index -- confirms the selected domain is still available.
3. Signature valid -- if verification is enabled, confirms the bundle signature verifies.
4. Sources superseded -- if in Mirror mode, lists the direct source connections that will be suspended.

All pre-flight checks must pass before the Activate Mirror Consumer button becomes enabled. On activation, the wizard calls PUT /api/v1/mirror/consumer (to persist the consumer connector config) and PUT /api/v1/mirror/config (to set the operating mode).

On success, a confirmation screen shows the configured mirror URL, domain, mode, and sync schedule with a link to the Mirror Dashboard.

8.3 UI wizard vs. environment variables

Both the UI wizard and environment variables configure the same StellaOpsMirrorConnector. The following table maps wizard fields to env vars:

Wizard field	Environment variable	Notes
Mirror Base Address	CONCELIER__SOURCES__STELLAOPSMIRROR__BASEADDRESS	Required
Domain	CONCELIER__SOURCES__STELLAOPSMIRROR__DOMAINID	Required
Index Path	CONCELIER__SOURCES__STELLAOPSMIRROR__INDEXPATH	Default: /concelier/exports/index.json
HTTP Timeout	CONCELIER__SOURCES__STELLAOPSMIRROR__HTTPTIMEOUT	Default: 30s
Signature Enabled	CONCELIER__SOURCES__STELLAOPSMIRROR__SIGNATURE__ENABLED	true or false
Algorithm	CONCELIER__SOURCES__STELLAOPSMIRROR__SIGNATURE__KEYID	Inferred from JWS header
Key ID	CONCELIER__SOURCES__STELLAOPSMIRROR__SIGNATURE__PROVIDER	e.g., mirror-signing-key-01
Public Key (PEM)	CONCELIER__SOURCES__STELLAOPSMIRROR__SIGNATURE__PUBLICKEYPATH	Path to PEM file on disk

When to use environment variables instead of the wizard:

• Automated provisioning (Terraform, Ansible, Helm values)
• Headless or CLI-only deployments
• Air-gap environments where the UI may not be accessible during initial setup
• GitOps workflows where configuration is declarative

When to use the wizard:

• First-time consumer setup with an unknown mirror
• Interactive domain discovery (the wizard fetches and displays the mirror index)
• Signature auto-detection (the wizard probes the bundle JWS header)
• Quick mode switching (Direct to Mirror/Hybrid) without container restarts

Configuration set via the wizard is persisted in-memory by the IMirrorConsumerConfigStore and takes effect immediately. Environment variables require a service restart to take effect. When both are set, the runtime (wizard) configuration takes precedence.

8.4 Air-gap bundle import

Air-gap environments can import mirror bundles via two paths:

CLI import (MirrorBundleImportService)

The stellaops-cli mirror import command is the primary offline method. It reads a bundle directory from the local filesystem, verifies checksums and DSSE envelopes, and imports the artifacts directly:

stellaops-cli mirror import \
  --bundle-path /data/mirror-bundles/export-2026-03-15 \
  --verify-checksums \
  --verify-dsse \
  --trust-roots /data/trust-roots/roots.pem

UI import (wizard Step 3)

The mirror client setup wizard provides an air-gap import section on Step 3 (Sync Schedule & Caching). Expand the Air-Gap Import panel:

1. Enter the Bundle Path -- a local filesystem path on the server where the bundle directory resides (e.g., /data/mirror-bundles/export-2026-03-15).
2. Optionally enter a Trust Roots Path for a PEM-encoded trust roots file.
3. Toggle Verify checksums (enabled by default) -- validates SHA-256 checksums for all artifacts in the bundle manifest.
4. Toggle Verify DSSE envelopes (enabled by default) -- validates Dead Simple Signing Envelope signatures.
5. Click Import Bundle -- calls POST /api/v1/mirror/import with the specified path and verification options.

The API endpoint operates on server-local filesystem paths, not file uploads. The bundle directory must be accessible to the Concelier service container. Mount the bundle directory into the container (e.g., via Docker volume) before triggering the import.

Import progress and results can be monitored via GET /api/v1/mirror/import/status, which returns the number of exports imported, total size, and any errors or warnings.

Feature	CLI	UI
Offline/air-gap primary path	Yes	Yes (requires server-local path)
Checksum verification	--verify-checksums	Toggle in wizard
DSSE verification	--verify-dsse	Toggle in wizard
Trust roots	--trust-roots <path>	Trust Roots Path field
Progress monitoring	Stdout	GET /api/v1/mirror/import/status
Automation-friendly	Yes	No (interactive)

9. References

• Deployment profiles: devops/compose/docker-compose.mirror.yaml, devops/helm/stellaops/values-mirror.yaml
• Mirror architecture dossiers: docs/modules/concelier/architecture.md, docs/modules/excititor/mirrors.md
• Export bundling: docs/modules/devops/architecture.md §3, docs/modules/excititor/architecture.md §7