git.stella-ops.org/docs/deploy/containers.md

# Container Deployment Guide — AOC Update

> **Audience:** DevOps Guild, platform operators deploying StellaOps services.  
> **Scope:** Deployment configuration changes required by the Aggregation-Only Contract (AOC), including schema validators, guard environment flags, and verifier identities.

This guide supplements existing deployment manuals with AOC-specific configuration. It assumes familiarity with the base Compose/Helm manifests described in `ops/deployment/` and `docs/ARCHITECTURE_DEVOPS.md`.

---

## 1 · Schema validator enablement

### 1.1 MongoDB validators

- Apply JSON schema validators to `advisory_raw` and `vex_raw` collections before enabling AOC guards.
- Before enabling validators or the idempotency index, run the duplicate audit helper to confirm no conflicting raw advisories remain:
  ```bash
  mongo concelier ops/devops/scripts/check-advisory-raw-duplicates.js --eval 'var LIMIT=200;'
  ```
  Resolve any reported rows prior to rollout.
- Use the migration script provided in `ops/devops/scripts/apply-aoc-validators.js`:

```bash
kubectl exec -n concelier deploy/concelier-mongo -- \
  mongo concelier ops/devops/scripts/apply-aoc-validators.js

kubectl exec -n excititor deploy/excititor-mongo -- \
  mongo excititor ops/devops/scripts/apply-aoc-validators.js
```

- Validators enforce required fields (`tenant`, `source`, `upstream`, `linkset`) and reject forbidden keys at DB level.
- Rollback plan: validators are applied with `validationLevel: moderate`—downgrade via the same script with `--remove` if required.

### 1.2 Migration order

1. Deploy validators in maintenance window.
2. Roll out Concelier/Excititor images with guard middleware enabled (`AOC_GUARD_ENABLED=true`).
3. Run smoke tests (`stella sources ingest --dry-run` fixtures) before resuming production ingestion.

### 1.3 Supersedes backfill verification

1. **Duplicate audit:** Confirm `mongo concelier ops/devops/scripts/check-advisory-raw-duplicates.js --eval 'var LIMIT=200;'` reports no conflicts before restarting Concelier with the new migrations.
2. **Post-migration check:** After the service restarts, validate that `db.advisory` is a view pointing to `advisory_backup_20251028`:
   ```bash
   mongo concelier --quiet --eval 'db.getCollectionInfos({ name: "advisory" })[0]'
   ```
   The `type` should be `"view"` and `options.viewOn` should equal `"advisory_backup_20251028"`.
3. **Supersedes chain spot-check:** Inspect a sample set to ensure deterministic chaining:
   ```bash
   mongo concelier --quiet --eval '
     db.advisory_raw.aggregate([
       { $match: { "upstream.upstream_id": { $exists: true } } },
       { $sort: { "tenant": 1, "source.vendor": 1, "upstream.upstream_id": 1, "upstream.retrieved_at": 1 } },
       { $limit: 5 },
       { $project: { _id: 1, supersedes: 1 } }
     ]).forEach(printjson)'
   ```
   Each revision should reference the previous `_id` (or `null` for the first revision). Record findings in the change ticket before proceeding to production.

---

## 2 · Container environment flags

Add the following environment variables to Concelier/Excititor deployments:

| Variable | Default | Description |
|----------|---------|-------------|
| `AOC_GUARD_ENABLED` | `true` | Enables `AOCWriteGuard` interception. Set `false` only for controlled rollback. |
| `AOC_ALLOW_SUPERSEDES_RETROFIT` | `false` | Allows temporary supersedes backfill during migration. Remove after cutover. |
| `AOC_METRICS_ENABLED` | `true` | Emits `ingestion_write_total`, `aoc_violation_total`, etc. |
| `AOC_TENANT_HEADER` | `X-Stella-Tenant` | Header name expected from Gateway. |
| `AOC_VERIFIER_USER` | `stella-aoc-verify` | Read-only service user used by UI/CLI verification. |

Compose snippet:

```yaml
environment:
  - AOC_GUARD_ENABLED=true
  - AOC_ALLOW_SUPERSEDES_RETROFIT=false
  - AOC_METRICS_ENABLED=true
  - AOC_TENANT_HEADER=X-Stella-Tenant
  - AOC_VERIFIER_USER=stella-aoc-verify
```

Ensure `AOC_VERIFIER_USER` exists in Authority with `aoc:verify` scope and no write permissions.

---

## 3 · Verifier identity

- Create a dedicated client (`stella-aoc-verify`) via Authority bootstrap:

```yaml
clients:
  - clientId: stella-aoc-verify
    grantTypes: [client_credentials]
    scopes: [aoc:verify, advisory:read, vex:read]
    tenants: [default]
```

- Store credentials in secret store (`Kubernetes Secret`, `Docker swarm secret`).
- Bind credentials to `stella aoc verify` CI jobs and Console verification service.
- Rotate quarterly; document in `ops/authority-key-rotation.md`.

---

## 4 · Deployment steps

1. **Pre-checks:** Confirm database backups, alerting in maintenance mode, and staging environment validated.
2. **Apply validators:** Run scripts per § 1.1.
3. **Update manifests:** Inject environment variables (§ 2) and mount guard configuration configmaps.
4. **Redeploy services:** Rolling restart Concelier/Excititor pods. Monitor `ingestion_write_total` for steady throughput.
5. **Seed verifier:** Deploy read-only verifier user and store credentials.
6. **Run verification:** Execute `stella aoc verify --since 24h` and ensure exit code `0`.
7. **Update dashboards:** Point Grafana panels to new metrics (`aoc_violation_total`).
8. **Record handoff:** Capture console screenshots and verification logs for release notes.

---

## 5 · Offline Kit updates

- Ship validator scripts with Offline Kit (`offline-kit/scripts/apply-aoc-validators.js`).
- Include pre-generated verification reports for air-gapped deployments.
- Document offline CLI workflow in bundle README referencing `docs/cli/cli-reference.md`.
- Ensure `stella-aoc-verify` credentials are scoped to offline tenant and rotated during bundle refresh.

---

## 6 · Rollback plan

1. Disable guard via `AOC_GUARD_ENABLED=false` on Concelier/Excititor and rollout.
2. Remove validators with the migration script (`--remove`).
3. Pause verification jobs to prevent noise.
4. Investigate and remediate upstream issues before re-enabling guards.

---

## 7 · References

- [Aggregation-Only Contract reference](../ingestion/aggregation-only-contract.md)
- [Authority scopes & tenancy](../security/authority-scopes.md)
- [Observability guide](../observability/observability.md)
- [CLI AOC commands](../cli/cli-reference.md)
- [Concelier architecture](../ARCHITECTURE_CONCELIER.md)
- [Excititor architecture](../ARCHITECTURE_EXCITITOR.md)

---

## 8 · Compliance checklist

- [ ] Validators documented and scripts referenced for online/offline deployments.
- [ ] Environment variables cover guard enablement, metrics, and tenant header.
- [ ] Read-only verifier user installation steps included.
- [ ] Offline kit instructions align with validator/verification workflow.
- [ ] Rollback procedure captured.
- [ ] Cross-links to AOC docs, Authority scopes, and observability guides present.
- [ ] DevOps Guild sign-off tracked (owner: @devops-guild, due 2025-10-29).

---

*Last updated: 2025-10-26 (Sprint 19).*