57 lines
3.3 KiB
Markdown
57 lines
3.3 KiB
Markdown
# Raw Linkset Backfill & Adoption Plan
|
|
|
|
_Last updated: 2025-10-31_
|
|
Owners: Concelier Storage Guild, DevOps Guild, Policy Guild
|
|
|
|
## Context
|
|
|
|
- Concelier observations now emit both a **canonical linkset** (deduped, normalised identifiers) and a **raw linkset** (`rawLinkset`) that preserves upstream ordering, duplicates, and original pointer metadata.
|
|
- Existing `concelier.advisory_observations` documents created before 2025-10-31 do **not** contain the `rawLinkset` field.
|
|
- Policy Engine selection joiners (`POLICY-ENGINE-20-003`) will switch to the raw projection once backfill completes and consumers validate fixtures.
|
|
|
|
## Objectives
|
|
|
|
1. Populate `rawLinkset` for historical observations across online clusters and Offline Kit bundles without breaking append-only guarantees.
|
|
2. Provide migration scripts + runbook so operators can rehearse in staging (and air-gapped deployments) before production rollout.
|
|
3. Unblock Policy Engine adoption by guaranteeing dual projections exist for all tenants.
|
|
|
|
## Deliverables
|
|
|
|
- [ ] **Migration script** (`20251104_advisory_observations_raw_linkset_backfill.csx`)
|
|
- Iterates observations lacking `rawLinkset`
|
|
- Rehydrates raw document via existing snapshot (or cached DTO)
|
|
- Reuses `AdvisoryObservationFactory.CreateRawLinkset`
|
|
- Writes using `$set` with optimistic retry; preserves `updatedAt` via `setOnInsert`
|
|
- [ ] **Offline Kit updater** (extend `ops/offline-kit/scripts/export_offline_bundle.py`) to patch bundles in-place
|
|
- [ ] **Runbook** covering:
|
|
- Pre-check query: `db.concelier.advisory_observations.countDocuments({ rawLinkset: { $exists: false } })`
|
|
- Backup procedure (`mongodump` or snapshot requirement)
|
|
- Dry-run mode limiting batches by tenant
|
|
- Metrics/telemetry expectations (`concelier.migrations.documents_processed_total`)
|
|
- Rollback (no-op because field addition; note to retain snapshot for verification)
|
|
- [ ] **Fixture updates** ensuring storage/CLI/Policy tests include `rawLinkset`
|
|
- [ ] **Policy Engine follow-up** to flip joiners once `rawLinkset` population reaches 100% (tracked via metrics).
|
|
|
|
## Timeline
|
|
|
|
| Date (UTC) | Milestone | Notes |
|
|
|------------|-----------|-------|
|
|
| 2025-10-31 | Handshake w/ Policy | Agreement to consume `rawLinkset`; this document created. |
|
|
| 2025-11-01 | Draft migration script | Validate against staging dataset snapshots. |
|
|
| 2025-11-04 | Storage task CONCELIER-STORE-AOC-19-005 due | Deliver script + runbook for review. |
|
|
| 2025-11-06 | Staging backfill rehearsal | Target < 30 min runtime on 5M observations. |
|
|
| 2025-11-08 | Policy fixtures updated | POL engine branch consumes `rawLinkset`. |
|
|
| 2025-11-11 | Production rollout window | Pending DevOps sign-off after rehearsals. |
|
|
|
|
## Open Questions
|
|
|
|
- Do we need archival of the canonical-only projection for backwards compatibility exports? (Policy to confirm.)
|
|
- Offline Kit delta: should we regenerate entire bundle or ship incremental patch? (DevOps reviewing.)
|
|
- Metrics: add `raw_linkset_missing_total` counter to detect regressions post-backfill?
|
|
|
|
## Next Actions
|
|
|
|
- [ ] Concelier Storage Guild: prototype migration script, share for review (`2025-11-01`).
|
|
- [ ] DevOps Guild: schedule staging rehearsal + update `docs/deploy/containers.md` with new runbook section.
|
|
- [ ] Policy Guild: prepare feature flag/branch to switch joiners once metrics show zero missing `rawLinkset`.
|