audit work, fixed StellaOps.sln warnings/errors, fixed tests, sprints work, new advisories
This commit is contained in:
master
@@ -2,9 +2,44 @@
|
||||
|
||||
Per SPRINT_8200_0014_0003.
|
||||
|
||||
> **Related:** [Bundle Export Format](federation-bundle-export.md) for detailed bundle schema.
|
||||
|
||||
## Overview
|
||||
|
||||
Federation enables multi-site synchronization of canonical advisory data between Concelier instances. Sites can export bundles containing delta changes and import bundles from other sites to maintain synchronized vulnerability intelligence.
|
||||
Federation enables secure, cursor-based synchronization of canonical vulnerability advisories between StellaOps sites. It supports:
|
||||
|
||||
- **Delta exports**: Only changed records since the last cursor are included
|
||||
- **Air-gap transfers**: Bundles can be written to files for offline transfer
|
||||
- **Multi-site topology**: Multiple sites can synchronize independently
|
||||
- **Cryptographic verification**: DSSE signatures ensure bundle authenticity
|
||||
|
||||
## Bundle Format
|
||||
|
||||
Federation bundles are ZST-compressed TAR archives containing:
|
||||
|
||||
| File | Description |
|
||||
|------|-------------|
|
||||
| `MANIFEST.json` | Bundle metadata, cursor, counts, hash |
|
||||
| `canonicals.ndjson` | Canonical advisories (one per line) |
|
||||
| `edges.ndjson` | Source edges linking advisories to sources |
|
||||
| `deletions.ndjson` | Withdrawn/deleted advisory IDs |
|
||||
| `SIGNATURE.json` | Optional DSSE signature envelope |
|
||||
|
||||
## Cursor Format
|
||||
|
||||
Cursors use ISO-8601 timestamp with sequence number:
|
||||
|
||||
```
|
||||
{ISO-8601 timestamp}#{sequence number}
|
||||
|
||||
Examples:
|
||||
2025-01-15T10:00:00.000Z#0001
|
||||
2025-01-15T10:00:00.000Z#0002
|
||||
```
|
||||
|
||||
- Cursors are site-specific (each site maintains independent cursors)
|
||||
- Sequence numbers distinguish concurrent exports
|
||||
- Cursors are monotonically increasing within a site
|
||||
|
||||
## Architecture
|
||||
|
||||
@@ -384,3 +419,80 @@ stella feedser canonical get sha256:mergehash...
|
||||
6. **Maintain Key Trust:** Regularly rotate and verify federation signing keys
|
||||
|
||||
7. **Document Site Policies:** Keep a registry of trusted sites and their policies
|
||||
|
||||
## Multi-Site Topologies
|
||||
|
||||
### Hub-and-Spoke Topology
|
||||
|
||||
```
|
||||
┌─────────────┐
|
||||
│ Hub Site │
|
||||
│ (Primary) │
|
||||
└──────┬──────┘
|
||||
│
|
||||
┌──────────┼──────────┐
|
||||
▼ ▼ ▼
|
||||
┌──────────┐ ┌──────────┐ ┌──────────┐
|
||||
│ Site A │ │ Site B │ │ Site C │
|
||||
│ (Spoke) │ │ (Spoke) │ │ (Spoke) │
|
||||
└──────────┘ └──────────┘ └──────────┘
|
||||
```
|
||||
|
||||
### Mesh Topology
|
||||
|
||||
Each site can import from multiple sources for redundancy:
|
||||
|
||||
```yaml
|
||||
federation:
|
||||
import:
|
||||
allowed_sites:
|
||||
- "hub-primary"
|
||||
- "hub-secondary" # Redundancy
|
||||
```
|
||||
|
||||
## Verification Details
|
||||
|
||||
### Hash Verification
|
||||
|
||||
Bundle hash is computed over compressed content:
|
||||
|
||||
```
|
||||
SHA256(compressed bundle content)
|
||||
```
|
||||
|
||||
### DSSE Signature Format
|
||||
|
||||
DSSE envelope contains:
|
||||
|
||||
```json
|
||||
{
|
||||
"payloadType": "application/stellaops.federation.bundle+json",
|
||||
"payload": "base64(bundle_hash + site_id + cursor)",
|
||||
"signatures": [
|
||||
{
|
||||
"keyId": "signing-key-001",
|
||||
"algorithm": "ES256",
|
||||
"signature": "base64(signature)"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Monitoring Metrics
|
||||
|
||||
### Key Prometheus Metrics
|
||||
|
||||
- `federation_export_duration_seconds` - Export time
|
||||
- `federation_import_duration_seconds` - Import time
|
||||
- `federation_bundle_size_bytes` - Bundle sizes
|
||||
- `federation_items_processed_total` - Items processed by type
|
||||
- `federation_conflicts_total` - Merge conflicts encountered
|
||||
|
||||
## Security Considerations
|
||||
|
||||
1. **Never skip signature verification in production**
|
||||
2. **Validate allowed_sites whitelist**
|
||||
3. **Use TLS for API endpoints**
|
||||
4. **Rotate signing keys periodically**
|
||||
5. **Audit import events**
|
||||
6. **Monitor for duplicate bundle imports**
|
||||
|
||||
@@ -1,332 +0,0 @@
|
||||
# Federation Setup and Operations Guide
|
||||
|
||||
This guide covers the setup and operation of StellaOps federation for multi-site vulnerability data synchronization.
|
||||
|
||||
## Overview
|
||||
|
||||
Federation enables secure, cursor-based synchronization of canonical vulnerability advisories between StellaOps sites. It supports:
|
||||
|
||||
- **Delta exports**: Only changed records since the last cursor are included
|
||||
- **Air-gap transfers**: Bundles can be written to files for offline transfer
|
||||
- **Multi-site topology**: Multiple sites can synchronize independently
|
||||
- **Cryptographic verification**: DSSE signatures ensure bundle authenticity
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
|
||||
│ Site A │────▶│ Bundle │────▶│ Site B │
|
||||
│ (Export) │ │ (.zst) │ │ (Import) │
|
||||
└─────────────┘ └─────────────┘ └─────────────┘
|
||||
│
|
||||
▼
|
||||
┌───────────┐
|
||||
│ Site C │
|
||||
│ (Import) │
|
||||
└───────────┘
|
||||
```
|
||||
|
||||
## Bundle Format
|
||||
|
||||
Federation bundles are ZST-compressed TAR archives containing:
|
||||
|
||||
| File | Description |
|
||||
|------|-------------|
|
||||
| `MANIFEST.json` | Bundle metadata, cursor, counts, hash |
|
||||
| `canonicals.ndjson` | Canonical advisories (one per line) |
|
||||
| `edges.ndjson` | Source edges linking advisories to sources |
|
||||
| `deletions.ndjson` | Withdrawn/deleted advisory IDs |
|
||||
| `SIGNATURE.json` | Optional DSSE signature envelope |
|
||||
|
||||
## Configuration
|
||||
|
||||
### Export Site Configuration
|
||||
|
||||
```yaml
|
||||
# concelier.yaml
|
||||
federation:
|
||||
enabled: true
|
||||
site_id: "us-west-1" # Unique site identifier
|
||||
export:
|
||||
enabled: true
|
||||
default_compression_level: 3 # ZST level (1-19)
|
||||
sign_bundles: true # Sign exported bundles
|
||||
max_items_per_bundle: 10000 # Maximum items per export
|
||||
```
|
||||
|
||||
### Import Site Configuration
|
||||
|
||||
```yaml
|
||||
# concelier.yaml
|
||||
federation:
|
||||
enabled: true
|
||||
site_id: "eu-central-1"
|
||||
import:
|
||||
enabled: true
|
||||
skip_signature_verification: false # NEVER set true in production
|
||||
allowed_sites: # Trusted site IDs
|
||||
- "us-west-1"
|
||||
- "ap-south-1"
|
||||
conflict_resolution: "prefer_remote" # prefer_remote | prefer_local | fail
|
||||
force_cursor_validation: true # Reject out-of-order imports
|
||||
```
|
||||
|
||||
## API Endpoints
|
||||
|
||||
### Export Endpoints
|
||||
|
||||
```bash
|
||||
# Export delta bundle since cursor
|
||||
GET /api/v1/federation/export?since_cursor={cursor}
|
||||
|
||||
# Preview export (counts only)
|
||||
GET /api/v1/federation/export/preview?since_cursor={cursor}
|
||||
|
||||
# Get federation status
|
||||
GET /api/v1/federation/status
|
||||
```
|
||||
|
||||
### Import Endpoints
|
||||
|
||||
```bash
|
||||
# Import bundle
|
||||
POST /api/v1/federation/import
|
||||
Content-Type: application/zstd
|
||||
|
||||
# Validate bundle without importing
|
||||
POST /api/v1/federation/validate
|
||||
Content-Type: application/zstd
|
||||
|
||||
# List federated sites
|
||||
GET /api/v1/federation/sites
|
||||
|
||||
# Update site policy
|
||||
PUT /api/v1/federation/sites/{site_id}/policy
|
||||
```
|
||||
|
||||
## CLI Commands
|
||||
|
||||
### Export Operations
|
||||
|
||||
```bash
|
||||
# Export full bundle (no cursor = all data)
|
||||
feedser bundle export --output bundle.zst
|
||||
|
||||
# Export delta since last cursor
|
||||
feedser bundle export --since-cursor "2025-01-15T10:00:00Z#0001" --output delta.zst
|
||||
|
||||
# Preview export without creating bundle
|
||||
feedser bundle preview --since-cursor "2025-01-15T10:00:00Z#0001"
|
||||
|
||||
# Export without signing (testing only)
|
||||
feedser bundle export --no-sign --output unsigned.zst
|
||||
```
|
||||
|
||||
### Import Operations
|
||||
|
||||
```bash
|
||||
# Import bundle
|
||||
feedser bundle import bundle.zst
|
||||
|
||||
# Dry run (validate without importing)
|
||||
feedser bundle import bundle.zst --dry-run
|
||||
|
||||
# Import from stdin (pipe)
|
||||
cat bundle.zst | feedser bundle import -
|
||||
|
||||
# Force import (skip cursor validation)
|
||||
feedser bundle import bundle.zst --force
|
||||
```
|
||||
|
||||
### Site Management
|
||||
|
||||
```bash
|
||||
# List federated sites
|
||||
feedser sites list
|
||||
|
||||
# Show site details
|
||||
feedser sites show us-west-1
|
||||
|
||||
# Enable/disable site
|
||||
feedser sites enable ap-south-1
|
||||
feedser sites disable ap-south-1
|
||||
```
|
||||
|
||||
## Cursor Format
|
||||
|
||||
Cursors use ISO-8601 timestamp with sequence number:
|
||||
|
||||
```
|
||||
{ISO-8601 timestamp}#{sequence number}
|
||||
|
||||
Examples:
|
||||
2025-01-15T10:00:00.000Z#0001
|
||||
2025-01-15T10:00:00.000Z#0002
|
||||
```
|
||||
|
||||
- Cursors are site-specific (each site maintains independent cursors)
|
||||
- Sequence numbers distinguish concurrent exports
|
||||
- Cursors are monotonically increasing within a site
|
||||
|
||||
## Air-Gap Transfer Workflow
|
||||
|
||||
For environments without network connectivity:
|
||||
|
||||
```bash
|
||||
# On Source Site (connected to authority)
|
||||
feedser bundle export --since-cursor "$LAST_CURSOR" --output /media/usb/bundle.zst
|
||||
feedser bundle preview --since-cursor "$LAST_CURSOR" > /media/usb/manifest.txt
|
||||
|
||||
# Transfer media to target site...
|
||||
|
||||
# On Target Site (air-gapped)
|
||||
feedser bundle import /media/usb/bundle.zst --dry-run # Validate first
|
||||
feedser bundle import /media/usb/bundle.zst # Import
|
||||
```
|
||||
|
||||
## Multi-Site Synchronization
|
||||
|
||||
### Hub-and-Spoke Topology
|
||||
|
||||
```
|
||||
┌─────────────┐
|
||||
│ Hub Site │
|
||||
│ (Primary) │
|
||||
└──────┬──────┘
|
||||
│
|
||||
┌──────────┼──────────┐
|
||||
▼ ▼ ▼
|
||||
┌──────────┐ ┌──────────┐ ┌──────────┐
|
||||
│ Site A │ │ Site B │ │ Site C │
|
||||
│ (Spoke) │ │ (Spoke) │ │ (Spoke) │
|
||||
└──────────┘ └──────────┘ └──────────┘
|
||||
```
|
||||
|
||||
### Mesh Topology
|
||||
|
||||
Each site can import from multiple sources:
|
||||
|
||||
```yaml
|
||||
federation:
|
||||
import:
|
||||
allowed_sites:
|
||||
- "hub-primary"
|
||||
- "hub-secondary" # Redundancy
|
||||
```
|
||||
|
||||
## Merge Behavior
|
||||
|
||||
### Conflict Resolution
|
||||
|
||||
When importing, conflicts are resolved based on configuration:
|
||||
|
||||
| Strategy | Behavior |
|
||||
|----------|----------|
|
||||
| `prefer_remote` | Remote (bundle) value wins (default) |
|
||||
| `prefer_local` | Local value preserved |
|
||||
| `fail` | Import aborts on any conflict |
|
||||
|
||||
### Merge Actions
|
||||
|
||||
| Action | Description |
|
||||
|--------|-------------|
|
||||
| `Created` | New canonical added |
|
||||
| `Updated` | Existing canonical updated |
|
||||
| `Skipped` | No change needed (identical) |
|
||||
|
||||
## Verification
|
||||
|
||||
### Hash Verification
|
||||
|
||||
Bundle hash is computed over compressed content:
|
||||
|
||||
```
|
||||
SHA256(compressed bundle content)
|
||||
```
|
||||
|
||||
### Signature Verification
|
||||
|
||||
DSSE envelope contains:
|
||||
|
||||
```json
|
||||
{
|
||||
"payloadType": "application/stellaops.federation.bundle+json",
|
||||
"payload": "base64(bundle_hash + site_id + cursor)",
|
||||
"signatures": [
|
||||
{
|
||||
"keyId": "signing-key-001",
|
||||
"algorithm": "ES256",
|
||||
"signature": "base64(signature)"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
### Key Metrics
|
||||
|
||||
- `federation_export_duration_seconds` - Export time
|
||||
- `federation_import_duration_seconds` - Import time
|
||||
- `federation_bundle_size_bytes` - Bundle sizes
|
||||
- `federation_items_processed_total` - Items processed by type
|
||||
- `federation_conflicts_total` - Merge conflicts encountered
|
||||
|
||||
### Health Checks
|
||||
|
||||
```bash
|
||||
# Check federation status
|
||||
curl http://localhost:5000/api/v1/federation/status
|
||||
|
||||
# Response
|
||||
{
|
||||
"site_id": "us-west-1",
|
||||
"export_enabled": true,
|
||||
"import_enabled": true,
|
||||
"last_export": "2025-01-15T10:00:00Z",
|
||||
"last_import": "2025-01-15T09:30:00Z",
|
||||
"sites_synced": 2
|
||||
}
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
**Import fails with "cursor validation failed"**
|
||||
- Bundle cursor is not after current site cursor
|
||||
- Use `--force` to override (not recommended)
|
||||
- Check if bundle was already imported
|
||||
|
||||
**Signature verification failed**
|
||||
- Signing key not trusted on target site
|
||||
- Key expired or revoked
|
||||
- Use `--skip-signature` for testing only
|
||||
|
||||
**Large bundle timeout**
|
||||
- Increase `federation.export.timeout`
|
||||
- Use smaller `max_items_per_bundle`
|
||||
- Stream directly to file
|
||||
|
||||
### Debug Logging
|
||||
|
||||
```yaml
|
||||
logging:
|
||||
level:
|
||||
StellaOps.Concelier.Federation: Debug
|
||||
```
|
||||
|
||||
## Security Considerations
|
||||
|
||||
1. **Never skip signature verification in production**
|
||||
2. **Validate allowed_sites whitelist**
|
||||
3. **Use TLS for API endpoints**
|
||||
4. **Rotate signing keys periodically**
|
||||
5. **Audit import events**
|
||||
6. **Monitor for duplicate bundle imports**
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Bundle Export Format](federation-bundle-export.md)
|
||||
- [Sync Ledger Schema](../db/sync-ledger.md)
|
||||
- [Signing Configuration](../security/signing.md)
|
||||
@@ -1,238 +1,238 @@
|
||||
# Concelier & Excititor Mirror Operations
|
||||
|
||||
This runbook describes how Stella Ops operates the managed mirrors under `*.stella-ops.org`.
|
||||
It covers Docker Compose and Helm deployment overlays, secret handling for multi-tenant
|
||||
authn, CDN fronting, and the recurring sync pipeline that keeps mirror bundles current.
|
||||
|
||||
## 1. Prerequisites
|
||||
|
||||
- **Authority access** – client credentials (`client_id` + secret) authorised for
|
||||
`concelier.mirror.read` and `excititor.mirror.read` scopes. Secrets live outside git.
|
||||
- **Signed TLS certificates** – wildcard or per-domain (`mirror-primary`, `mirror-community`).
|
||||
Store them under `deploy/compose/mirror-gateway/tls/` or in Kubernetes secrets.
|
||||
- **Mirror gateway credentials** – Basic Auth htpasswd files per domain. Generate with
|
||||
`htpasswd -B`. Operators distribute credentials to downstream consumers.
|
||||
- **Export artifact source** – read access to the canonical S3 buckets (or rsync share)
|
||||
that hold `concelier` JSON bundles and `excititor` VEX exports.
|
||||
- **Persistent volumes** – storage for Concelier job metadata and mirror export trees.
|
||||
For Helm, provision PVCs (`concelier-mirror-jobs`, `concelier-mirror-exports`,
|
||||
`excititor-mirror-exports`, `mirror-mongo-data`, `mirror-minio-data`) before rollout.
|
||||
|
||||
### 1.1 Service configuration quick reference
|
||||
|
||||
Concelier.WebService exposes the mirror HTTP endpoints once `CONCELIER__MIRROR__ENABLED=true`.
|
||||
Key knobs:
|
||||
|
||||
- `CONCELIER__MIRROR__EXPORTROOT` – root folder containing export snapshots (`<exportId>/mirror/*`).
|
||||
- `CONCELIER__MIRROR__ACTIVEEXPORTID` – optional explicit export id; otherwise the service auto-falls back to the `latest/` symlink or newest directory.
|
||||
- `CONCELIER__MIRROR__REQUIREAUTHENTICATION` – default auth requirement; override per domain with `CONCELIER__MIRROR__DOMAINS__{n}__REQUIREAUTHENTICATION`.
|
||||
- `CONCELIER__MIRROR__MAXINDEXREQUESTSPERHOUR` – budget for `/concelier/exports/index.json`. Domains inherit this value unless they define `__MAXDOWNLOADREQUESTSPERHOUR`.
|
||||
- `CONCELIER__MIRROR__DOMAINS__{n}__ID` – domain identifier matching the exporter manifest; additional keys configure display name and rate budgets.
|
||||
|
||||
> The service honours Stella Ops Authority when `CONCELIER__AUTHORITY__ENABLED=true` and `ALLOWANONYMOUSFALLBACK=false`. Use the bypass CIDR list (`CONCELIER__AUTHORITY__BYPASSNETWORKS__*`) for in-cluster ingress gateways that terminate Basic Auth. Unauthorized requests emit `WWW-Authenticate: Bearer` so downstream automation can detect token failures.
|
||||
|
||||
Mirror responses carry deterministic cache headers: `/index.json` returns `Cache-Control: public, max-age=60`, while per-domain manifests/bundles include `Cache-Control: public, max-age=300, immutable`. Rate limiting surfaces `Retry-After` when quotas are exceeded.
|
||||
|
||||
### 1.2 Mirror connector configuration
|
||||
|
||||
Downstream Concelier instances ingest published bundles using the `StellaOpsMirrorConnector`. Operators running the connector in air‑gapped or limited connectivity environments can tune the following options (environment prefix `CONCELIER__SOURCES__STELLAOPSMIRROR__`):
|
||||
|
||||
- `BASEADDRESS` – absolute mirror root (e.g., `https://mirror-primary.stella-ops.org`).
|
||||
- `INDEXPATH` – relative path to the mirror index (`/concelier/exports/index.json` by default).
|
||||
- `DOMAINID` – mirror domain identifier from the index (`primary`, `community`, etc.).
|
||||
- `HTTPTIMEOUT` – request timeout; raise when mirrors sit behind slow WAN links.
|
||||
- `SIGNATURE__ENABLED` – require detached JWS verification for `bundle.json`.
|
||||
- `SIGNATURE__KEYID` / `SIGNATURE__PROVIDER` – expected signing key metadata.
|
||||
- `SIGNATURE__PUBLICKEYPATH` – PEM fallback used when the mirror key registry is offline.
|
||||
|
||||
The connector keeps a per-export fingerprint (bundle digest + generated-at timestamp) and tracks outstanding document IDs. If a scan is interrupted, the next run resumes parse/map work using the stored fingerprint and pending document lists—no network requests are reissued unless the upstream digest changes.
|
||||
|
||||
## 2. Secret & certificate layout
|
||||
|
||||
### Docker Compose (`deploy/compose/docker-compose.mirror.yaml`)
|
||||
|
||||
- `deploy/compose/env/mirror.env.example` – copy to `.env` and adjust quotas or domain IDs.
|
||||
- `deploy/compose/mirror-secrets/` – mount read-only into `/run/secrets`. Place:
|
||||
- `concelier-authority-client` – Authority client secret.
|
||||
- `excititor-authority-client` (optional) – reserve for future authn.
|
||||
- `deploy/compose/mirror-gateway/tls/` – PEM-encoded cert/key pairs:
|
||||
- `mirror-primary.crt`, `mirror-primary.key`
|
||||
- `mirror-community.crt`, `mirror-community.key`
|
||||
- `deploy/compose/mirror-gateway/secrets/` – htpasswd files:
|
||||
- `mirror-primary.htpasswd`
|
||||
- `mirror-community.htpasswd`
|
||||
|
||||
### Helm (`deploy/helm/stellaops/values-mirror.yaml`)
|
||||
|
||||
Create secrets in the target namespace:
|
||||
|
||||
```bash
|
||||
kubectl create secret generic concelier-mirror-auth \
|
||||
--from-file=concelier-authority-client=concelier-authority-client
|
||||
|
||||
kubectl create secret generic excititor-mirror-auth \
|
||||
--from-file=excititor-authority-client=excititor-authority-client
|
||||
|
||||
kubectl create secret tls mirror-gateway-tls \
|
||||
--cert=mirror-primary.crt --key=mirror-primary.key
|
||||
|
||||
kubectl create secret generic mirror-gateway-htpasswd \
|
||||
--from-file=mirror-primary.htpasswd --from-file=mirror-community.htpasswd
|
||||
```
|
||||
|
||||
> Keep Basic Auth lists short-lived (rotate quarterly) and document credential recipients.
|
||||
|
||||
## 3. Deployment
|
||||
|
||||
### 3.1 Docker Compose (edge mirrors, lab validation)
|
||||
|
||||
1. `cp deploy/compose/env/mirror.env.example deploy/compose/env/mirror.env`
|
||||
2. Populate secrets/tls directories as described above.
|
||||
3. Sync mirror bundles (see §4) into `deploy/compose/mirror-data/…` and ensure they are mounted
|
||||
on the host path backing the `concelier-exports` and `excititor-exports` volumes.
|
||||
4. Run the profile validator: `deploy/tools/validate-profiles.sh`.
|
||||
5. Launch: `docker compose --env-file env/mirror.env -f docker-compose.mirror.yaml up -d`.
|
||||
|
||||
### 3.2 Helm (production mirrors)
|
||||
|
||||
1. Provision PVCs sized for mirror bundles (baseline: 20 GiB per domain).
|
||||
2. Create secrets/tls config maps (§2).
|
||||
3. `helm upgrade --install mirror deploy/helm/stellaops -f deploy/helm/stellaops/values-mirror.yaml`.
|
||||
4. Annotate the `stellaops-mirror-gateway` service with ingress/LoadBalancer metadata required by
|
||||
your CDN (e.g., AWS load balancer scheme internal + NLB idle timeout).
|
||||
|
||||
## 4. Artifact sync workflow
|
||||
|
||||
Mirrors never generate exports—they ingest signed bundles produced by the Concelier and Excititor
|
||||
export jobs. Recommended sync pattern:
|
||||
|
||||
### 4.1 Compose host (systemd timer)
|
||||
|
||||
`/usr/local/bin/mirror-sync.sh`:
|
||||
|
||||
```bash
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
export AWS_ACCESS_KEY_ID=…
|
||||
export AWS_SECRET_ACCESS_KEY=…
|
||||
|
||||
aws s3 sync s3://mirror-stellaops/concelier/latest \
|
||||
/opt/stellaops/mirror-data/concelier --delete --size-only
|
||||
|
||||
aws s3 sync s3://mirror-stellaops/excititor/latest \
|
||||
/opt/stellaops/mirror-data/excititor --delete --size-only
|
||||
```
|
||||
|
||||
Schedule with a systemd timer every 5 minutes. The Compose volumes mount `/opt/stellaops/mirror-data/*`
|
||||
into the containers read-only, matching `CONCELIER__MIRROR__EXPORTROOT=/exports/json` and
|
||||
`EXCITITOR__ARTIFACTS__FILESYSTEM__ROOT=/exports`.
|
||||
|
||||
### 4.2 Kubernetes (CronJob)
|
||||
|
||||
Create a CronJob running the AWS CLI (or rclone) in the same namespace, writing into the PVCs:
|
||||
|
||||
```yaml
|
||||
apiVersion: batch/v1
|
||||
kind: CronJob
|
||||
metadata:
|
||||
name: mirror-sync
|
||||
spec:
|
||||
schedule: "*/5 * * * *"
|
||||
jobTemplate:
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: sync
|
||||
image: public.ecr.aws/aws-cli/aws-cli@sha256:5df5f52c29f5e3ba46d0ad9e0e3afc98701c4a0f879400b4c5f80d943b5fadea
|
||||
command:
|
||||
- /bin/sh
|
||||
- -c
|
||||
- >
|
||||
aws s3 sync s3://mirror-stellaops/concelier/latest /exports/concelier --delete --size-only &&
|
||||
aws s3 sync s3://mirror-stellaops/excititor/latest /exports/excititor --delete --size-only
|
||||
volumeMounts:
|
||||
- name: concelier-exports
|
||||
mountPath: /exports/concelier
|
||||
- name: excititor-exports
|
||||
mountPath: /exports/excititor
|
||||
envFrom:
|
||||
- secretRef:
|
||||
name: mirror-sync-aws
|
||||
restartPolicy: OnFailure
|
||||
volumes:
|
||||
- name: concelier-exports
|
||||
persistentVolumeClaim:
|
||||
claimName: concelier-mirror-exports
|
||||
- name: excititor-exports
|
||||
persistentVolumeClaim:
|
||||
claimName: excititor-mirror-exports
|
||||
```
|
||||
|
||||
## 5. CDN integration
|
||||
|
||||
1. Point the CDN origin at the mirror gateway (Compose host or Kubernetes LoadBalancer).
|
||||
2. Honour the response headers emitted by the gateway and Concelier/Excititor:
|
||||
`Cache-Control: public, max-age=300, immutable` for mirror payloads.
|
||||
3. Configure origin shields in the CDN to prevent cache stampedes. Recommended TTLs:
|
||||
- Index (`/concelier/exports/index.json`, `/excititor/mirror/*/index`) → 60 s.
|
||||
- Bundle/manifest payloads → 300 s.
|
||||
4. Forward the `Authorization` header—Basic Auth terminates at the gateway.
|
||||
5. Enforce per-domain rate limits at the CDN (matching gateway budgets) and enable logging
|
||||
to SIEM for anomaly detection.
|
||||
|
||||
## 6. Smoke tests
|
||||
|
||||
After each deployment or sync cycle (temporarily set low budgets if you need to observe 429 responses):
|
||||
|
||||
```bash
|
||||
# Index with Basic Auth
|
||||
curl -u $PRIMARY_CREDS https://mirror-primary.stella-ops.org/concelier/exports/index.json | jq 'keys'
|
||||
|
||||
# Mirror manifest signature and cache headers
|
||||
curl -u $PRIMARY_CREDS -I https://mirror-primary.stella-ops.org/concelier/exports/mirror/primary/manifest.json \
|
||||
| tee /tmp/manifest-headers.txt
|
||||
grep -E '^Cache-Control: ' /tmp/manifest-headers.txt # expect public, max-age=300, immutable
|
||||
|
||||
# Excititor consensus bundle metadata
|
||||
curl -u $COMMUNITY_CREDS https://mirror-community.stella-ops.org/excititor/mirror/community/index \
|
||||
| jq '.exports[].exportKey'
|
||||
|
||||
# Signed bundle + detached JWS (spot check digests)
|
||||
curl -u $PRIMARY_CREDS https://mirror-primary.stella-ops.org/concelier/exports/mirror/primary/bundle.json.jws \
|
||||
-o bundle.json.jws
|
||||
cosign verify-blob --signature bundle.json.jws --key mirror-key.pub bundle.json
|
||||
|
||||
# Service-level auth check (inside cluster – no gateway credentials)
|
||||
kubectl exec deploy/stellaops-concelier -- curl -si http://localhost:8443/concelier/exports/mirror/primary/manifest.json \
|
||||
| head -n 5 # expect HTTP/1.1 401 with WWW-Authenticate: Bearer
|
||||
|
||||
# Rate limit smoke (repeat quickly; second call should return 429 + Retry-After)
|
||||
for i in 1 2; do
|
||||
curl -s -o /dev/null -D - https://mirror-primary.stella-ops.org/concelier/exports/index.json \
|
||||
-u $PRIMARY_CREDS | grep -E '^(HTTP/|Retry-After:)'
|
||||
sleep 1
|
||||
done
|
||||
```
|
||||
|
||||
Watch the gateway metrics (`nginx_vts` or access logs) for cache hits. In Kubernetes, `kubectl logs deploy/stellaops-mirror-gateway`
|
||||
should show `X-Cache-Status: HIT/MISS`.
|
||||
|
||||
## 7. Maintenance & rotation
|
||||
|
||||
- **Bundle freshness** – alert if sync job lag exceeds 15 minutes or if `concelier` logs
|
||||
`Mirror export root is not configured`.
|
||||
- **Secret rotation** – change Authority client secrets and Basic Auth credentials quarterly.
|
||||
Update the mounted secrets and restart deployments (`docker compose restart concelier` or
|
||||
`kubectl rollout restart deploy/stellaops-concelier`).
|
||||
- **TLS renewal** – reissue certificates, place new files, and reload gateway (`docker compose exec mirror-gateway nginx -s reload`).
|
||||
- **Quota tuning** – adjust per-domain `MAXDOWNLOADREQUESTSPERHOUR` in `.env` or values file.
|
||||
Align CDN rate limits and inform downstreams.
|
||||
|
||||
## 8. References
|
||||
|
||||
- Deployment profiles: `deploy/compose/docker-compose.mirror.yaml`,
|
||||
`deploy/helm/stellaops/values-mirror.yaml`
|
||||
- Mirror architecture dossiers: `docs/modules/concelier/architecture.md`,
|
||||
`docs/modules/excititor/mirrors.md`
|
||||
- Export bundling: `docs/modules/devops/architecture.md` §3, `docs/modules/excititor/architecture.md` §7
|
||||
# Concelier & Excititor Mirror Operations
|
||||
|
||||
This runbook describes how Stella Ops operates the managed mirrors under `*.stella-ops.org`.
|
||||
It covers Docker Compose and Helm deployment overlays, secret handling for multi-tenant
|
||||
authn, CDN fronting, and the recurring sync pipeline that keeps mirror bundles current.
|
||||
|
||||
## 1. Prerequisites
|
||||
|
||||
- **Authority access** – client credentials (`client_id` + secret) authorised for
|
||||
`concelier.mirror.read` and `excititor.mirror.read` scopes. Secrets live outside git.
|
||||
- **Signed TLS certificates** – wildcard or per-domain (`mirror-primary`, `mirror-community`).
|
||||
Store them under `devops/compose/mirror-gateway/tls/` or in Kubernetes secrets.
|
||||
- **Mirror gateway credentials** – Basic Auth htpasswd files per domain. Generate with
|
||||
`htpasswd -B`. Operators distribute credentials to downstream consumers.
|
||||
- **Export artifact source** – read access to the canonical S3 buckets (or rsync share)
|
||||
that hold `concelier` JSON bundles and `excititor` VEX exports.
|
||||
- **Persistent volumes** – storage for Concelier job metadata and mirror export trees.
|
||||
For Helm, provision PVCs (`concelier-mirror-jobs`, `concelier-mirror-exports`,
|
||||
`excititor-mirror-exports`, `mirror-mongo-data`, `mirror-minio-data`) before rollout.
|
||||
|
||||
### 1.1 Service configuration quick reference
|
||||
|
||||
Concelier.WebService exposes the mirror HTTP endpoints once `CONCELIER__MIRROR__ENABLED=true`.
|
||||
Key knobs:
|
||||
|
||||
- `CONCELIER__MIRROR__EXPORTROOT` – root folder containing export snapshots (`<exportId>/mirror/*`).
|
||||
- `CONCELIER__MIRROR__ACTIVEEXPORTID` – optional explicit export id; otherwise the service auto-falls back to the `latest/` symlink or newest directory.
|
||||
- `CONCELIER__MIRROR__REQUIREAUTHENTICATION` – default auth requirement; override per domain with `CONCELIER__MIRROR__DOMAINS__{n}__REQUIREAUTHENTICATION`.
|
||||
- `CONCELIER__MIRROR__MAXINDEXREQUESTSPERHOUR` – budget for `/concelier/exports/index.json`. Domains inherit this value unless they define `__MAXDOWNLOADREQUESTSPERHOUR`.
|
||||
- `CONCELIER__MIRROR__DOMAINS__{n}__ID` – domain identifier matching the exporter manifest; additional keys configure display name and rate budgets.
|
||||
|
||||
> The service honours Stella Ops Authority when `CONCELIER__AUTHORITY__ENABLED=true` and `ALLOWANONYMOUSFALLBACK=false`. Use the bypass CIDR list (`CONCELIER__AUTHORITY__BYPASSNETWORKS__*`) for in-cluster ingress gateways that terminate Basic Auth. Unauthorized requests emit `WWW-Authenticate: Bearer` so downstream automation can detect token failures.
|
||||
|
||||
Mirror responses carry deterministic cache headers: `/index.json` returns `Cache-Control: public, max-age=60`, while per-domain manifests/bundles include `Cache-Control: public, max-age=300, immutable`. Rate limiting surfaces `Retry-After` when quotas are exceeded.
|
||||
|
||||
### 1.2 Mirror connector configuration
|
||||
|
||||
Downstream Concelier instances ingest published bundles using the `StellaOpsMirrorConnector`. Operators running the connector in air‑gapped or limited connectivity environments can tune the following options (environment prefix `CONCELIER__SOURCES__STELLAOPSMIRROR__`):
|
||||
|
||||
- `BASEADDRESS` – absolute mirror root (e.g., `https://mirror-primary.stella-ops.org`).
|
||||
- `INDEXPATH` – relative path to the mirror index (`/concelier/exports/index.json` by default).
|
||||
- `DOMAINID` – mirror domain identifier from the index (`primary`, `community`, etc.).
|
||||
- `HTTPTIMEOUT` – request timeout; raise when mirrors sit behind slow WAN links.
|
||||
- `SIGNATURE__ENABLED` – require detached JWS verification for `bundle.json`.
|
||||
- `SIGNATURE__KEYID` / `SIGNATURE__PROVIDER` – expected signing key metadata.
|
||||
- `SIGNATURE__PUBLICKEYPATH` – PEM fallback used when the mirror key registry is offline.
|
||||
|
||||
The connector keeps a per-export fingerprint (bundle digest + generated-at timestamp) and tracks outstanding document IDs. If a scan is interrupted, the next run resumes parse/map work using the stored fingerprint and pending document lists—no network requests are reissued unless the upstream digest changes.
|
||||
|
||||
## 2. Secret & certificate layout
|
||||
|
||||
### Docker Compose (`devops/compose/docker-compose.mirror.yaml`)
|
||||
|
||||
- `devops/compose/env/mirror.env.example` – copy to `.env` and adjust quotas or domain IDs.
|
||||
- `devops/compose/mirror-secrets/` – mount read-only into `/run/secrets`. Place:
|
||||
- `concelier-authority-client` – Authority client secret.
|
||||
- `excititor-authority-client` (optional) – reserve for future authn.
|
||||
- `devops/compose/mirror-gateway/tls/` – PEM-encoded cert/key pairs:
|
||||
- `mirror-primary.crt`, `mirror-primary.key`
|
||||
- `mirror-community.crt`, `mirror-community.key`
|
||||
- `devops/compose/mirror-gateway/secrets/` – htpasswd files:
|
||||
- `mirror-primary.htpasswd`
|
||||
- `mirror-community.htpasswd`
|
||||
|
||||
### Helm (`devops/helm/stellaops/values-mirror.yaml`)
|
||||
|
||||
Create secrets in the target namespace:
|
||||
|
||||
```bash
|
||||
kubectl create secret generic concelier-mirror-auth \
|
||||
--from-file=concelier-authority-client=concelier-authority-client
|
||||
|
||||
kubectl create secret generic excititor-mirror-auth \
|
||||
--from-file=excititor-authority-client=excititor-authority-client
|
||||
|
||||
kubectl create secret tls mirror-gateway-tls \
|
||||
--cert=mirror-primary.crt --key=mirror-primary.key
|
||||
|
||||
kubectl create secret generic mirror-gateway-htpasswd \
|
||||
--from-file=mirror-primary.htpasswd --from-file=mirror-community.htpasswd
|
||||
```
|
||||
|
||||
> Keep Basic Auth lists short-lived (rotate quarterly) and document credential recipients.
|
||||
|
||||
## 3. Deployment
|
||||
|
||||
### 3.1 Docker Compose (edge mirrors, lab validation)
|
||||
|
||||
1. `cp devops/compose/env/mirror.env.example devops/compose/env/mirror.env`
|
||||
2. Populate secrets/tls directories as described above.
|
||||
3. Sync mirror bundles (see §4) into `devops/compose/mirror-data/…` and ensure they are mounted
|
||||
on the host path backing the `concelier-exports` and `excititor-exports` volumes.
|
||||
4. Run the profile validator: `deploy/tools/validate-profiles.sh`.
|
||||
5. Launch: `docker compose --env-file env/mirror.env -f docker-compose.mirror.yaml up -d`.
|
||||
|
||||
### 3.2 Helm (production mirrors)
|
||||
|
||||
1. Provision PVCs sized for mirror bundles (baseline: 20 GiB per domain).
|
||||
2. Create secrets/tls config maps (§2).
|
||||
3. `helm upgrade --install mirror devops/helm/stellaops -f devops/helm/stellaops/values-mirror.yaml`.
|
||||
4. Annotate the `stellaops-mirror-gateway` service with ingress/LoadBalancer metadata required by
|
||||
your CDN (e.g., AWS load balancer scheme internal + NLB idle timeout).
|
||||
|
||||
## 4. Artifact sync workflow
|
||||
|
||||
Mirrors never generate exports—they ingest signed bundles produced by the Concelier and Excititor
|
||||
export jobs. Recommended sync pattern:
|
||||
|
||||
### 4.1 Compose host (systemd timer)
|
||||
|
||||
`/usr/local/bin/mirror-sync.sh`:
|
||||
|
||||
```bash
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
export AWS_ACCESS_KEY_ID=…
|
||||
export AWS_SECRET_ACCESS_KEY=…
|
||||
|
||||
aws s3 sync s3://mirror-stellaops/concelier/latest \
|
||||
/opt/stellaops/mirror-data/concelier --delete --size-only
|
||||
|
||||
aws s3 sync s3://mirror-stellaops/excititor/latest \
|
||||
/opt/stellaops/mirror-data/excititor --delete --size-only
|
||||
```
|
||||
|
||||
Schedule with a systemd timer every 5 minutes. The Compose volumes mount `/opt/stellaops/mirror-data/*`
|
||||
into the containers read-only, matching `CONCELIER__MIRROR__EXPORTROOT=/exports/json` and
|
||||
`EXCITITOR__ARTIFACTS__FILESYSTEM__ROOT=/exports`.
|
||||
|
||||
### 4.2 Kubernetes (CronJob)
|
||||
|
||||
Create a CronJob running the AWS CLI (or rclone) in the same namespace, writing into the PVCs:
|
||||
|
||||
```yaml
|
||||
apiVersion: batch/v1
|
||||
kind: CronJob
|
||||
metadata:
|
||||
name: mirror-sync
|
||||
spec:
|
||||
schedule: "*/5 * * * *"
|
||||
jobTemplate:
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: sync
|
||||
image: public.ecr.aws/aws-cli/aws-cli@sha256:5df5f52c29f5e3ba46d0ad9e0e3afc98701c4a0f879400b4c5f80d943b5fadea
|
||||
command:
|
||||
- /bin/sh
|
||||
- -c
|
||||
- >
|
||||
aws s3 sync s3://mirror-stellaops/concelier/latest /exports/concelier --delete --size-only &&
|
||||
aws s3 sync s3://mirror-stellaops/excititor/latest /exports/excititor --delete --size-only
|
||||
volumeMounts:
|
||||
- name: concelier-exports
|
||||
mountPath: /exports/concelier
|
||||
- name: excititor-exports
|
||||
mountPath: /exports/excititor
|
||||
envFrom:
|
||||
- secretRef:
|
||||
name: mirror-sync-aws
|
||||
restartPolicy: OnFailure
|
||||
volumes:
|
||||
- name: concelier-exports
|
||||
persistentVolumeClaim:
|
||||
claimName: concelier-mirror-exports
|
||||
- name: excititor-exports
|
||||
persistentVolumeClaim:
|
||||
claimName: excititor-mirror-exports
|
||||
```
|
||||
|
||||
## 5. CDN integration
|
||||
|
||||
1. Point the CDN origin at the mirror gateway (Compose host or Kubernetes LoadBalancer).
|
||||
2. Honour the response headers emitted by the gateway and Concelier/Excititor:
|
||||
`Cache-Control: public, max-age=300, immutable` for mirror payloads.
|
||||
3. Configure origin shields in the CDN to prevent cache stampedes. Recommended TTLs:
|
||||
- Index (`/concelier/exports/index.json`, `/excititor/mirror/*/index`) → 60 s.
|
||||
- Bundle/manifest payloads → 300 s.
|
||||
4. Forward the `Authorization` header—Basic Auth terminates at the gateway.
|
||||
5. Enforce per-domain rate limits at the CDN (matching gateway budgets) and enable logging
|
||||
to SIEM for anomaly detection.
|
||||
|
||||
## 6. Smoke tests
|
||||
|
||||
After each deployment or sync cycle (temporarily set low budgets if you need to observe 429 responses):
|
||||
|
||||
```bash
|
||||
# Index with Basic Auth
|
||||
curl -u $PRIMARY_CREDS https://mirror-primary.stella-ops.org/concelier/exports/index.json | jq 'keys'
|
||||
|
||||
# Mirror manifest signature and cache headers
|
||||
curl -u $PRIMARY_CREDS -I https://mirror-primary.stella-ops.org/concelier/exports/mirror/primary/manifest.json \
|
||||
| tee /tmp/manifest-headers.txt
|
||||
grep -E '^Cache-Control: ' /tmp/manifest-headers.txt # expect public, max-age=300, immutable
|
||||
|
||||
# Excititor consensus bundle metadata
|
||||
curl -u $COMMUNITY_CREDS https://mirror-community.stella-ops.org/excititor/mirror/community/index \
|
||||
| jq '.exports[].exportKey'
|
||||
|
||||
# Signed bundle + detached JWS (spot check digests)
|
||||
curl -u $PRIMARY_CREDS https://mirror-primary.stella-ops.org/concelier/exports/mirror/primary/bundle.json.jws \
|
||||
-o bundle.json.jws
|
||||
cosign verify-blob --signature bundle.json.jws --key mirror-key.pub bundle.json
|
||||
|
||||
# Service-level auth check (inside cluster – no gateway credentials)
|
||||
kubectl exec deploy/stellaops-concelier -- curl -si http://localhost:8443/concelier/exports/mirror/primary/manifest.json \
|
||||
| head -n 5 # expect HTTP/1.1 401 with WWW-Authenticate: Bearer
|
||||
|
||||
# Rate limit smoke (repeat quickly; second call should return 429 + Retry-After)
|
||||
for i in 1 2; do
|
||||
curl -s -o /dev/null -D - https://mirror-primary.stella-ops.org/concelier/exports/index.json \
|
||||
-u $PRIMARY_CREDS | grep -E '^(HTTP/|Retry-After:)'
|
||||
sleep 1
|
||||
done
|
||||
```
|
||||
|
||||
Watch the gateway metrics (`nginx_vts` or access logs) for cache hits. In Kubernetes, `kubectl logs deploy/stellaops-mirror-gateway`
|
||||
should show `X-Cache-Status: HIT/MISS`.
|
||||
|
||||
## 7. Maintenance & rotation
|
||||
|
||||
- **Bundle freshness** – alert if sync job lag exceeds 15 minutes or if `concelier` logs
|
||||
`Mirror export root is not configured`.
|
||||
- **Secret rotation** – change Authority client secrets and Basic Auth credentials quarterly.
|
||||
Update the mounted secrets and restart deployments (`docker compose restart concelier` or
|
||||
`kubectl rollout restart deploy/stellaops-concelier`).
|
||||
- **TLS renewal** – reissue certificates, place new files, and reload gateway (`docker compose exec mirror-gateway nginx -s reload`).
|
||||
- **Quota tuning** – adjust per-domain `MAXDOWNLOADREQUESTSPERHOUR` in `.env` or values file.
|
||||
Align CDN rate limits and inform downstreams.
|
||||
|
||||
## 8. References
|
||||
|
||||
- Deployment profiles: `devops/compose/docker-compose.mirror.yaml`,
|
||||
`devops/helm/stellaops/values-mirror.yaml`
|
||||
- Mirror architecture dossiers: `docs/modules/concelier/architecture.md`,
|
||||
`docs/modules/excititor/mirrors.md`
|
||||
- Export bundling: `docs/modules/devops/architecture.md` §3, `docs/modules/excititor/architecture.md` §7
|
||||
|
||||
Reference in New Issue
Block a user