# Concelier & Excititor Mirror Operations

This runbook describes how Stella Ops operates the managed mirrors under `*.stella-ops.org`.
It covers Docker Compose and Helm deployment overlays, secret handling for multi-tenant
authn, CDN fronting, and the recurring sync pipeline that keeps mirror bundles current.

## 1. Prerequisites

- **Authority access** – client credentials (`client_id` + secret) authorised for
  `concelier.mirror.read` and `excititor.mirror.read` scopes. Secrets live outside git.
- **Signed TLS certificates** – wildcard or per-domain (`mirror-primary`, `mirror-community`).
  Store them under `devops/compose/mirror-gateway/tls/` or in Kubernetes secrets.
- **Mirror gateway credentials** – Basic Auth htpasswd files per domain. Generate with
  `htpasswd -B`. Operators distribute credentials to downstream consumers.
- **Export artifact source** – read access to the canonical S3 buckets (or rsync share)
  that hold `concelier` JSON bundles and `excititor` VEX exports.
- **Persistent volumes** – storage for Concelier job metadata and mirror export trees.
  For Helm, provision PVCs (`concelier-mirror-jobs`, `concelier-mirror-exports`,
  `excititor-mirror-exports`) before rollout.

### 1.1 Service configuration quick reference

Concelier.WebService exposes the mirror HTTP endpoints once `CONCELIER__MIRROR__ENABLED=true`.
Key knobs:

- `CONCELIER__MIRROR__EXPORTROOT` – root folder containing export snapshots (`<exportId>/mirror/*`).
- `CONCELIER__MIRROR__ACTIVEEXPORTID` – optional explicit export id; otherwise the service auto-falls back to the `latest/` symlink or newest directory.
- `CONCELIER__MIRROR__REQUIREAUTHENTICATION` – default auth requirement; override per domain with `CONCELIER__MIRROR__DOMAINS__{n}__REQUIREAUTHENTICATION`.
- `CONCELIER__MIRROR__MAXINDEXREQUESTSPERHOUR` – budget for `/concelier/exports/index.json`. Domains inherit this value unless they define `__MAXDOWNLOADREQUESTSPERHOUR`.
- `CONCELIER__MIRROR__DOMAINS__{n}__ID` – domain identifier matching the exporter manifest; additional keys configure display name and rate budgets.

> The service honours Stella Ops Authority when `CONCELIER__AUTHORITY__ENABLED=true` and `ALLOWANONYMOUSFALLBACK=false`. Use the bypass CIDR list (`CONCELIER__AUTHORITY__BYPASSNETWORKS__*`) for in-cluster ingress gateways that terminate Basic Auth. Unauthorized requests emit `WWW-Authenticate: Bearer` so downstream automation can detect token failures.

Mirror responses carry deterministic cache headers: `/index.json` returns `Cache-Control: public, max-age=60`, while per-domain manifests/bundles include `Cache-Control: public, max-age=300, immutable`. Rate limiting surfaces `Retry-After` when quotas are exceeded.

### 1.2 Mirror connector configuration

Downstream Concelier instances ingest published bundles using the `StellaOpsMirrorConnector`. Operators running the connector in air‑gapped or limited connectivity environments can tune the following options (environment prefix `CONCELIER__SOURCES__STELLAOPSMIRROR__`):

- `BASEADDRESS` – absolute mirror root (e.g., `https://mirror-primary.stella-ops.org`).
- `INDEXPATH` – relative path to the mirror index (`/concelier/exports/index.json` by default).
- `DOMAINID` – mirror domain identifier from the index (`primary`, `community`, etc.).
- `HTTPTIMEOUT` – request timeout; raise when mirrors sit behind slow WAN links.
- `SIGNATURE__ENABLED` – require detached JWS verification for `bundle.json`.
- `SIGNATURE__KEYID` / `SIGNATURE__PROVIDER` – expected signing key metadata.
- `SIGNATURE__PUBLICKEYPATH` – PEM fallback used when the mirror key registry is offline.

The connector keeps a per-export fingerprint (bundle digest + generated-at timestamp) and tracks outstanding document IDs. If a scan is interrupted, the next run resumes parse/map work using the stored fingerprint and pending document lists—no network requests are reissued unless the upstream digest changes.

## 2. Secret & certificate layout

### Docker Compose (`devops/compose/docker-compose.mirror.yaml`)

- `devops/compose/env/mirror.env.example` – copy to `.env` and adjust quotas or domain IDs.
- `devops/compose/mirror-secrets/` – mount read-only into `/run/secrets`. Place:
  - `concelier-authority-client` – Authority client secret.
  - `excititor-authority-client` (optional) – reserve for future authn.
- `devops/compose/mirror-gateway/tls/` – PEM-encoded cert/key pairs:
  - `mirror-primary.crt`, `mirror-primary.key`
  - `mirror-community.crt`, `mirror-community.key`
- `devops/compose/mirror-gateway/secrets/` – htpasswd files:
  - `mirror-primary.htpasswd`
  - `mirror-community.htpasswd`

### Helm (`devops/helm/stellaops/values-mirror.yaml`)

Create secrets in the target namespace:

```bash
kubectl create secret generic concelier-mirror-auth \
  --from-file=concelier-authority-client=concelier-authority-client

kubectl create secret generic excititor-mirror-auth \
  --from-file=excititor-authority-client=excititor-authority-client

kubectl create secret tls mirror-gateway-tls \
  --cert=mirror-primary.crt --key=mirror-primary.key

kubectl create secret generic mirror-gateway-htpasswd \
  --from-file=mirror-primary.htpasswd --from-file=mirror-community.htpasswd
```

> Keep Basic Auth lists short-lived (rotate quarterly) and document credential recipients.

## 3. Deployment

### 3.1 Docker Compose (edge mirrors, lab validation)

1. `cp devops/compose/env/mirror.env.example devops/compose/env/mirror.env`
2. Populate secrets/tls directories as described above.
3. Sync mirror bundles (see §4) into `devops/compose/mirror-data/…` and ensure they are mounted
   on the host path backing the `concelier-exports` and `excititor-exports` volumes.
4. Run the profile validator: `deploy/tools/validate-profiles.sh`.
5. Launch: `docker compose --env-file env/mirror.env -f docker-compose.mirror.yaml up -d`.

### 3.2 Helm (production mirrors)

1. Provision PVCs sized for mirror bundles (baseline: 20 GiB per domain).
2. Create secrets/tls config maps (§2).
3. `helm upgrade --install mirror devops/helm/stellaops -f devops/helm/stellaops/values-mirror.yaml`.
4. Annotate the `stellaops-mirror-gateway` service with ingress/LoadBalancer metadata required by
   your CDN (e.g., AWS load balancer scheme internal + NLB idle timeout).

## 4. Artifact sync workflow

Mirrors never generate exports—they ingest signed bundles produced by the Concelier and Excititor
export jobs. Recommended sync pattern:

### 4.1 Compose host (systemd timer)

`/usr/local/bin/mirror-sync.sh`:

```bash
#!/usr/bin/env bash
set -euo pipefail
export AWS_ACCESS_KEY_ID=…
export AWS_SECRET_ACCESS_KEY=…

aws s3 sync s3://mirror-stellaops/concelier/latest \
  /opt/stellaops/mirror-data/concelier --delete --size-only

aws s3 sync s3://mirror-stellaops/excititor/latest \
  /opt/stellaops/mirror-data/excititor --delete --size-only
```

Schedule with a systemd timer every 5 minutes. The Compose volumes mount `/opt/stellaops/mirror-data/*`
into the containers read-only, matching `CONCELIER__MIRROR__EXPORTROOT=/exports/json` and
`EXCITITOR__ARTIFACTS__FILESYSTEM__ROOT=/exports`.

### 4.2 Kubernetes (CronJob)

Create a CronJob running the AWS CLI (or rclone) in the same namespace, writing into the PVCs:

```yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: mirror-sync
spec:
  schedule: "*/5 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: sync
            image: public.ecr.aws/aws-cli/aws-cli@sha256:5df5f52c29f5e3ba46d0ad9e0e3afc98701c4a0f879400b4c5f80d943b5fadea
            command:
              - /bin/sh
              - -c
              - >
                aws s3 sync s3://mirror-stellaops/concelier/latest /exports/concelier --delete --size-only &&
                aws s3 sync s3://mirror-stellaops/excititor/latest /exports/excititor --delete --size-only
            volumeMounts:
              - name: concelier-exports
                mountPath: /exports/concelier
              - name: excititor-exports
                mountPath: /exports/excititor
            envFrom:
              - secretRef:
                  name: mirror-sync-aws
          restartPolicy: OnFailure
          volumes:
            - name: concelier-exports
              persistentVolumeClaim:
                claimName: concelier-mirror-exports
            - name: excititor-exports
              persistentVolumeClaim:
                claimName: excititor-mirror-exports
```

## 5. CDN integration

1. Point the CDN origin at the mirror gateway (Compose host or Kubernetes LoadBalancer).
2. Honour the response headers emitted by the gateway and Concelier/Excititor:
   `Cache-Control: public, max-age=300, immutable` for mirror payloads.
3. Configure origin shields in the CDN to prevent cache stampedes. Recommended TTLs:
   - Index (`/concelier/exports/index.json`, `/excititor/mirror/*/index`) → 60 s.
   - Bundle/manifest payloads → 300 s.
4. Forward the `Authorization` header—Basic Auth terminates at the gateway.
5. Enforce per-domain rate limits at the CDN (matching gateway budgets) and enable logging
   to SIEM for anomaly detection.

## 6. Smoke tests

After each deployment or sync cycle (temporarily set low budgets if you need to observe 429 responses):

```bash
# Index with Basic Auth
curl -u $PRIMARY_CREDS https://mirror-primary.stella-ops.org/concelier/exports/index.json | jq 'keys'

# Mirror manifest signature and cache headers
curl -u $PRIMARY_CREDS -I https://mirror-primary.stella-ops.org/concelier/exports/mirror/primary/manifest.json \
  | tee /tmp/manifest-headers.txt
grep -E '^Cache-Control: ' /tmp/manifest-headers.txt   # expect public, max-age=300, immutable

# Excititor consensus bundle metadata
curl -u $COMMUNITY_CREDS https://mirror-community.stella-ops.org/excititor/mirror/community/index \
  | jq '.exports[].exportKey'

# Signed bundle + detached JWS (spot check digests)
curl -u $PRIMARY_CREDS https://mirror-primary.stella-ops.org/concelier/exports/mirror/primary/bundle.json.jws \
  -o bundle.json.jws
cosign verify-blob --signature bundle.json.jws --key mirror-key.pub bundle.json

# Service-level auth check (inside cluster – no gateway credentials)
kubectl exec deploy/stellaops-concelier -- curl -si http://localhost:8443/concelier/exports/mirror/primary/manifest.json \
  | head -n 5   # expect HTTP/1.1 401 with WWW-Authenticate: Bearer

# Rate limit smoke (repeat quickly; second call should return 429 + Retry-After)
for i in 1 2; do
  curl -s -o /dev/null -D - https://mirror-primary.stella-ops.org/concelier/exports/index.json \
    -u $PRIMARY_CREDS | grep -E '^(HTTP/|Retry-After:)'
  sleep 1
done
```

Watch the gateway metrics (`nginx_vts` or access logs) for cache hits. In Kubernetes, `kubectl logs deploy/stellaops-mirror-gateway`
should show `X-Cache-Status: HIT/MISS`.

## 7. Maintenance & rotation

- **Bundle freshness** – alert if sync job lag exceeds 15 minutes or if `concelier` logs
  `Mirror export root is not configured`.
- **Secret rotation** – change Authority client secrets and Basic Auth credentials quarterly.
  Update the mounted secrets and restart deployments (`docker compose restart concelier` or
  `kubectl rollout restart deploy/stellaops-concelier`).
- **TLS renewal** – reissue certificates, place new files, and reload gateway (`docker compose exec mirror-gateway nginx -s reload`).
- **Quota tuning** – adjust per-domain `MAXDOWNLOADREQUESTSPERHOUR` in `.env` or values file.
  Align CDN rate limits and inform downstreams.

## 8. Setting up as Mirror Consumer (UI)

Stella Ops provides a guided wizard for configuring an instance as a mirror consumer. The wizard replaces the manual env-var approach for most operators, while the environment variable path (section 1.2) remains available for headless, scripted, and air-gap-first deployments.

### 8.1 Accessing the wizard

The mirror client setup wizard is available at two entry points:

- **Mirror Dashboard** -- Navigate to **Integrations > Advisory & VEX Sources > Mirror Dashboard**. In the consumer panel header, click **Configure**. If the instance is in Direct mode with no consumer configured, the dashboard shows a "Switch to Mirror" call-to-action card that links directly to the wizard.
- **Advisory Source Catalog** -- Navigate to **Integrations > Advisory & VEX Sources > Catalog**. In the mirror context header, click **Connect to Mirror** (visible when no consumer connection is active).

The wizard route is `advisory-vex-sources/mirror/client-setup` (lazy-loaded under the Integration Hub).

### 8.2 Wizard flow (4 steps)

#### Step 1: Connect to Mirror

Enter the base address of the upstream mirror server (e.g., `https://mirror-primary.stella-ops.org`).

1. Click **Test Connection** -- the wizard calls `POST /api/v1/mirror/test` and reports success (with latency) or failure (with error message and remediation hint).
2. On successful connection, the wizard automatically calls `POST /api/v1/mirror/consumer/discover` to fetch the mirror index. A domain selector dropdown is populated with all available domains, showing each domain's display name, advisory count, bundle size, export formats, signature status, and last-generated timestamp.
3. Select the target domain from the dropdown.
4. Optionally expand **Advanced Settings** to override the index path (default: `/concelier/exports/index.json`) or adjust the HTTP timeout slider (5--300 seconds, default 30s).

#### Step 2: Signature Verification

On entering this step, the wizard automatically calls `POST /api/v1/mirror/consumer/verify-signature` to detect whether the selected domain's bundle is signed.

- If a signature is detected, the algorithm and key ID are pre-populated and a green "Signature detected" banner is shown.
- If no signature is detected, a warning banner is shown. You may proceed without verification.
- If the signature is invalid, a red banner is shown.

To configure signature verification manually:

1. Toggle **Enable signature verification** on.
2. Select the algorithm from the dropdown: ES256, ES384, ES512, RS256, or RS384.
3. Enter the Key ID (e.g., `mirror-signing-key-01`).
4. Paste the public key in PEM format or leave empty if the mirror key registry will resolve it.
5. Click **Verify Sample** to download a small bundle chunk and verify the signature against your configuration.

> Signature verification is optional but strongly recommended. The wizard defaults to "enabled" when a signed bundle is detected.

#### Step 3: Sync Schedule & Caching

**Operating mode** -- choose one:

| Mode | Behavior |
|------|----------|
| **Mirror** | Consumer only. All advisory data comes from the upstream mirror. Direct source connectors are suspended. |
| **Hybrid** | Consumer + direct sources. Mirror data augments direct fetching. Both remain active. |

A warning banner is shown when selecting Mirror mode, noting that direct source fetching will be disabled. Connectors remain configured and can be reactivated by switching back to Hybrid or Direct mode.

**Sync schedule** -- choose a preset:

- Manual, Hourly, Every 4 hours, Daily, Weekly

**Bundle caching** -- toggle on and set a TTL in hours (default: 168 hours / 7 days). Caching reduces bandwidth and improves resilience against transient mirror outages.

**Air-gap import** -- expand the collapsible "Air-Gap Import" section for offline bundle loading (see section 8.4).

#### Step 4: Review & Activate

The wizard shows a configuration summary card with all selected settings (mirror URL, domain, index path, timeout, mode, signature configuration, sync schedule, caching).

Pre-flight checks run automatically:

1. **Mirror reachable** -- re-tests connectivity.
2. **Domain exists in index** -- confirms the selected domain is still available.
3. **Signature valid** -- if verification is enabled, confirms the bundle signature verifies.
4. **Sources superseded** -- if in Mirror mode, lists the direct source connections that will be suspended.

All pre-flight checks must pass before the **Activate Mirror Consumer** button becomes enabled. On activation, the wizard calls `PUT /api/v1/mirror/consumer` (to persist the consumer connector config) and `PUT /api/v1/mirror/config` (to set the operating mode).

On success, a confirmation screen shows the configured mirror URL, domain, mode, and sync schedule with a link to the Mirror Dashboard.

### 8.3 UI wizard vs. environment variables

Both the UI wizard and environment variables configure the same `StellaOpsMirrorConnector`. The following table maps wizard fields to env vars:

| Wizard field | Environment variable | Notes |
|---|---|---|
| Mirror Base Address | `CONCELIER__SOURCES__STELLAOPSMIRROR__BASEADDRESS` | Required |
| Domain | `CONCELIER__SOURCES__STELLAOPSMIRROR__DOMAINID` | Required |
| Index Path | `CONCELIER__SOURCES__STELLAOPSMIRROR__INDEXPATH` | Default: `/concelier/exports/index.json` |
| HTTP Timeout | `CONCELIER__SOURCES__STELLAOPSMIRROR__HTTPTIMEOUT` | Default: 30s |
| Signature Enabled | `CONCELIER__SOURCES__STELLAOPSMIRROR__SIGNATURE__ENABLED` | `true` or `false` |
| Algorithm | `CONCELIER__SOURCES__STELLAOPSMIRROR__SIGNATURE__KEYID` | Inferred from JWS header |
| Key ID | `CONCELIER__SOURCES__STELLAOPSMIRROR__SIGNATURE__PROVIDER` | e.g., `mirror-signing-key-01` |
| Public Key (PEM) | `CONCELIER__SOURCES__STELLAOPSMIRROR__SIGNATURE__PUBLICKEYPATH` | Path to PEM file on disk |

**When to use environment variables instead of the wizard:**

- Automated provisioning (Terraform, Ansible, Helm values)
- Headless or CLI-only deployments
- Air-gap environments where the UI may not be accessible during initial setup
- GitOps workflows where configuration is declarative

**When to use the wizard:**

- First-time consumer setup with an unknown mirror
- Interactive domain discovery (the wizard fetches and displays the mirror index)
- Signature auto-detection (the wizard probes the bundle JWS header)
- Quick mode switching (Direct to Mirror/Hybrid) without container restarts

> Configuration set via the wizard is persisted in-memory by the `IMirrorConsumerConfigStore` and takes effect immediately. Environment variables require a service restart to take effect. When both are set, the runtime (wizard) configuration takes precedence.

### 8.4 Air-gap bundle import

Air-gap environments can import mirror bundles via two paths:

#### CLI import (`MirrorBundleImportService`)

The `stellaops-cli mirror import` command is the primary offline method. It reads a bundle directory from the local filesystem, verifies checksums and DSSE envelopes, and imports the artifacts directly:

```bash
stellaops-cli mirror import \
  --bundle-path /data/mirror-bundles/export-2026-03-15 \
  --verify-checksums \
  --verify-dsse \
  --trust-roots /data/trust-roots/roots.pem
```

#### UI import (wizard Step 3)

The mirror client setup wizard provides an air-gap import section on Step 3 (Sync Schedule & Caching). Expand the **Air-Gap Import** panel:

1. Enter the **Bundle Path** -- a local filesystem path on the server where the bundle directory resides (e.g., `/data/mirror-bundles/export-2026-03-15`).
2. Optionally enter a **Trust Roots Path** for a PEM-encoded trust roots file.
3. Toggle **Verify checksums** (enabled by default) -- validates SHA-256 checksums for all artifacts in the bundle manifest.
4. Toggle **Verify DSSE envelopes** (enabled by default) -- validates Dead Simple Signing Envelope signatures.
5. Click **Import Bundle** -- calls `POST /api/v1/mirror/import` with the specified path and verification options.

> The API endpoint operates on server-local filesystem paths, not file uploads. The bundle directory must be accessible to the Concelier service container. Mount the bundle directory into the container (e.g., via Docker volume) before triggering the import.

Import progress and results can be monitored via `GET /api/v1/mirror/import/status`, which returns the number of exports imported, total size, and any errors or warnings.

| Feature | CLI | UI |
|---|---|---|
| Offline/air-gap primary path | Yes | Yes (requires server-local path) |
| Checksum verification | `--verify-checksums` | Toggle in wizard |
| DSSE verification | `--verify-dsse` | Toggle in wizard |
| Trust roots | `--trust-roots <path>` | Trust Roots Path field |
| Progress monitoring | Stdout | `GET /api/v1/mirror/import/status` |
| Automation-friendly | Yes | No (interactive) |

## 9. References

- Deployment profiles: `devops/compose/docker-compose.mirror.yaml`,
  `devops/helm/stellaops/values-mirror.yaml`
- Mirror architecture dossiers: `docs/modules/concelier/architecture.md`,
  `docs/modules/excititor/mirrors.md`
- Export bundling: `docs/modules/devops/architecture.md` §3, `docs/modules/excititor/architecture.md` §7