master
7b5bdcf4d3
- Introduced AGENTS.md, README.md, TASKS.md, and implementation_plan.md for Vexer, detailing mission, responsibilities, key components, and operational notes. - Established similar documentation structure for Vulnerability Explorer and Zastava modules, including their respective workflows, integrations, and observability notes. - Created risk scoring profiles documentation outlining the core workflow, factor model, governance, and deliverables. - Ensured all modules adhere to the Aggregation-Only Contract and maintain determinism and provenance in outputs.
161 lines
6.9 KiB
Markdown
161 lines
6.9 KiB
Markdown
# Container Deployment Guide — AOC Update
|
||
|
||
> **Audience:** DevOps Guild, platform operators deploying StellaOps services.
|
||
> **Scope:** Deployment configuration changes required by the Aggregation-Only Contract (AOC), including schema validators, guard environment flags, and verifier identities.
|
||
|
||
This guide supplements existing deployment manuals with AOC-specific configuration. It assumes familiarity with the base Compose/Helm manifests described in `ops/deployment/` and `docs/modules/devops/architecture.md`.
|
||
|
||
---
|
||
|
||
## 1 · Schema validator enablement
|
||
|
||
### 1.1 MongoDB validators
|
||
|
||
- Apply JSON schema validators to `advisory_raw` and `vex_raw` collections before enabling AOC guards.
|
||
- Before enabling validators or the idempotency index, run the duplicate audit helper to confirm no conflicting raw advisories remain:
|
||
```bash
|
||
mongo concelier ops/devops/scripts/check-advisory-raw-duplicates.js --eval 'var LIMIT=200;'
|
||
```
|
||
Resolve any reported rows prior to rollout.
|
||
- Use the migration script provided in `ops/devops/scripts/apply-aoc-validators.js`:
|
||
|
||
```bash
|
||
kubectl exec -n concelier deploy/concelier-mongo -- \
|
||
mongo concelier ops/devops/scripts/apply-aoc-validators.js
|
||
|
||
kubectl exec -n excititor deploy/excititor-mongo -- \
|
||
mongo excititor ops/devops/scripts/apply-aoc-validators.js
|
||
```
|
||
|
||
- Validators enforce required fields (`tenant`, `source`, `upstream`, `linkset`) and reject forbidden keys at DB level.
|
||
- Rollback plan: validators are applied with `validationLevel: moderate`—downgrade via the same script with `--remove` if required.
|
||
|
||
### 1.2 Migration order
|
||
|
||
1. Deploy validators in maintenance window.
|
||
2. Roll out Concelier/Excititor images with guard middleware enabled (`AOC_GUARD_ENABLED=true`).
|
||
3. Run smoke tests (`stella sources ingest --dry-run` fixtures) before resuming production ingestion.
|
||
|
||
### 1.3 Supersedes backfill verification
|
||
|
||
1. **Duplicate audit:** Confirm `mongo concelier ops/devops/scripts/check-advisory-raw-duplicates.js --eval 'var LIMIT=200;'` reports no conflicts before restarting Concelier with the new migrations.
|
||
2. **Post-migration check:** After the service restarts, validate that `db.advisory` is a view pointing to `advisory_backup_20251028`:
|
||
```bash
|
||
mongo concelier --quiet --eval 'db.getCollectionInfos({ name: "advisory" })[0]'
|
||
```
|
||
The `type` should be `"view"` and `options.viewOn` should equal `"advisory_backup_20251028"`.
|
||
3. **Supersedes chain spot-check:** Inspect a sample set to ensure deterministic chaining:
|
||
```bash
|
||
mongo concelier --quiet --eval '
|
||
db.advisory_raw.aggregate([
|
||
{ $match: { "upstream.upstream_id": { $exists: true } } },
|
||
{ $sort: { "tenant": 1, "source.vendor": 1, "upstream.upstream_id": 1, "upstream.retrieved_at": 1 } },
|
||
{ $limit: 5 },
|
||
{ $project: { _id: 1, supersedes: 1 } }
|
||
]).forEach(printjson)'
|
||
```
|
||
Each revision should reference the previous `_id` (or `null` for the first revision). Record findings in the change ticket before proceeding to production.
|
||
|
||
---
|
||
|
||
## 2 · Container environment flags
|
||
|
||
Add the following environment variables to Concelier/Excititor deployments:
|
||
|
||
| Variable | Default | Description |
|
||
|----------|---------|-------------|
|
||
| `AOC_GUARD_ENABLED` | `true` | Enables `AOCWriteGuard` interception. Set `false` only for controlled rollback. |
|
||
| `AOC_ALLOW_SUPERSEDES_RETROFIT` | `false` | Allows temporary supersedes backfill during migration. Remove after cutover. |
|
||
| `AOC_METRICS_ENABLED` | `true` | Emits `ingestion_write_total`, `aoc_violation_total`, etc. |
|
||
| `AOC_TENANT_HEADER` | `X-Stella-Tenant` | Header name expected from Gateway. |
|
||
| `AOC_VERIFIER_USER` | `stella-aoc-verify` | Read-only service user used by UI/CLI verification. |
|
||
|
||
Compose snippet:
|
||
|
||
```yaml
|
||
environment:
|
||
- AOC_GUARD_ENABLED=true
|
||
- AOC_ALLOW_SUPERSEDES_RETROFIT=false
|
||
- AOC_METRICS_ENABLED=true
|
||
- AOC_TENANT_HEADER=X-Stella-Tenant
|
||
- AOC_VERIFIER_USER=stella-aoc-verify
|
||
```
|
||
|
||
Ensure `AOC_VERIFIER_USER` exists in Authority with `aoc:verify` scope and no write permissions.
|
||
|
||
---
|
||
|
||
## 3 · Verifier identity
|
||
|
||
- Create a dedicated client (`stella-aoc-verify`) via Authority bootstrap:
|
||
|
||
```yaml
|
||
clients:
|
||
- clientId: stella-aoc-verify
|
||
grantTypes: [client_credentials]
|
||
scopes: [aoc:verify, advisory:read, vex:read]
|
||
tenants: [default]
|
||
```
|
||
|
||
- Store credentials in secret store (`Kubernetes Secret`, `Docker swarm secret`).
|
||
- Bind credentials to `stella aoc verify` CI jobs and Console verification service.
|
||
- Rotate quarterly; document in `ops/authority-key-rotation.md`.
|
||
|
||
---
|
||
|
||
## 4 · Deployment steps
|
||
|
||
1. **Pre-checks:** Confirm database backups, alerting in maintenance mode, and staging environment validated.
|
||
2. **Apply validators:** Run scripts per § 1.1.
|
||
3. **Update manifests:** Inject environment variables (§ 2) and mount guard configuration configmaps.
|
||
4. **Redeploy services:** Rolling restart Concelier/Excititor pods. Monitor `ingestion_write_total` for steady throughput.
|
||
5. **Seed verifier:** Deploy read-only verifier user and store credentials.
|
||
6. **Run verification:** Execute `stella aoc verify --since 24h` and ensure exit code `0`.
|
||
7. **Update dashboards:** Point Grafana panels to new metrics (`aoc_violation_total`).
|
||
8. **Record handoff:** Capture console screenshots and verification logs for release notes.
|
||
|
||
---
|
||
|
||
## 5 · Offline Kit updates
|
||
|
||
- Ship validator scripts with Offline Kit (`offline-kit/scripts/apply-aoc-validators.js`).
|
||
- Include pre-generated verification reports for air-gapped deployments.
|
||
- Document offline CLI workflow in bundle README referencing `docs/modules/cli/guides/cli-reference.md`.
|
||
- Ensure `stella-aoc-verify` credentials are scoped to offline tenant and rotated during bundle refresh.
|
||
|
||
---
|
||
|
||
## 6 · Rollback plan
|
||
|
||
1. Disable guard via `AOC_GUARD_ENABLED=false` on Concelier/Excititor and rollout.
|
||
2. Remove validators with the migration script (`--remove`).
|
||
3. Pause verification jobs to prevent noise.
|
||
4. Investigate and remediate upstream issues before re-enabling guards.
|
||
|
||
---
|
||
|
||
## 7 · References
|
||
|
||
- [Aggregation-Only Contract reference](../ingestion/aggregation-only-contract.md)
|
||
- [Authority scopes & tenancy](../security/authority-scopes.md)
|
||
- [Observability guide](../observability/observability.md)
|
||
- [CLI AOC commands](../modules/cli/guides/cli-reference.md)
|
||
- [Concelier architecture](../modules/concelier/architecture.md)
|
||
- [Excititor architecture](../modules/excititor/architecture.md)
|
||
|
||
---
|
||
|
||
## 8 · Compliance checklist
|
||
|
||
- [ ] Validators documented and scripts referenced for online/offline deployments.
|
||
- [ ] Environment variables cover guard enablement, metrics, and tenant header.
|
||
- [ ] Read-only verifier user installation steps included.
|
||
- [ ] Offline kit instructions align with validator/verification workflow.
|
||
- [ ] Rollback procedure captured.
|
||
- [ ] Cross-links to AOC docs, Authority scopes, and observability guides present.
|
||
- [ ] DevOps Guild sign-off tracked (owner: @devops-guild, due 2025-10-29).
|
||
|
||
---
|
||
|
||
*Last updated: 2025-10-26 (Sprint 19).*
|