stela ops usage fixes roles propagation and timoeut, one account to support multi tenants, migrations consolidation, search to support documentation, doctor and open api vector db search
This commit is contained in:
master
126
docs/db/MIGRATION_CONSOLIDATION_PLAN.md
Normal file
126
docs/db/MIGRATION_CONSOLIDATION_PLAN.md
Normal file
@@ -0,0 +1,126 @@
|
||||
# Migration Consolidation Plan (On-Prem Upgradeable)
|
||||
|
||||
Date: 2026-02-22 (UTC)
|
||||
Status: Active sprint plan for `MGC-02`, `MGC-03`, and `MGC-12`
|
||||
|
||||
## Objective
|
||||
|
||||
Consolidate StellaOps database migrations to one canonical mechanism and one operational runner entrypoint so upgrades are deterministic for self-hosted and long-lived on-prem installations.
|
||||
|
||||
## Canonical Migration Mechanism
|
||||
|
||||
The canonical mechanism is the shared PostgreSQL migration stack in `src/__Libraries/StellaOps.Infrastructure.Postgres/Migrations/`:
|
||||
|
||||
- `MigrationRunner` for execution, advisory lock coordination, checksum validation, and migration history.
|
||||
- `StartupMigrationHost` for automatic startup-time migration execution.
|
||||
- `MigrationStatusService` for status inspection.
|
||||
|
||||
Module discovery for this runner is plug-in based:
|
||||
|
||||
- One migration module plug-in per web service implementing `src/Platform/__Libraries/StellaOps.Platform.Database/IMigrationModulePlugin.cs`.
|
||||
- Consolidated module registry auto-discovers plug-ins through `src/Platform/__Libraries/StellaOps.Platform.Database/MigrationModulePluginDiscovery.cs`.
|
||||
- Current built-in plug-ins are in `src/Platform/__Libraries/StellaOps.Platform.Database/MigrationModulePlugins.cs`.
|
||||
- Optional external plug-in directories can be injected with `STELLAOPS_MIGRATION_PLUGIN_DIR` (path-list separated by OS path separator).
|
||||
|
||||
Canonical history table format is:
|
||||
|
||||
- `<schema>.schema_migrations`
|
||||
- Required columns: `migration_name`, `category`, `checksum`, `applied_at`, `applied_by`, `duration_ms`
|
||||
|
||||
## Canonical Migration Types and Numbering
|
||||
|
||||
Only these categories are valid:
|
||||
|
||||
- `startup`: automatic; prefix `001-099`
|
||||
- `release`: manual gate; prefix `100+`
|
||||
- `seed`: automatic; prefix `S001-S999`
|
||||
- `data`: manual/background; prefix `DM001-DM999`
|
||||
|
||||
Naming standard:
|
||||
|
||||
- `NNN_description.sql`
|
||||
- `SNNN_description.sql`
|
||||
- `DMNNN_description.sql`
|
||||
|
||||
Legacy naming is allowed only for already shipped files. New migrations must follow the canonical format.
|
||||
|
||||
## Canonical Runner Entrypoint Policy
|
||||
|
||||
Upgrade and deployment gate entrypoint:
|
||||
|
||||
- `stella system migrations-status --module all`
|
||||
- `stella system migrations-verify --module all`
|
||||
- `stella system migrations-run --module all --category release --dry-run`
|
||||
- `stella system migrations-run --module all --category release --force`
|
||||
|
||||
Startup behavior:
|
||||
|
||||
- Services may run only `startup` and `seed` categories automatically through `StartupMigrationHost` / `AddStartupMigrations(...)`.
|
||||
- Pending `release` or `data` migrations must block startup when startup migration hosts are enabled.
|
||||
|
||||
Compose/bootstrap behavior:
|
||||
|
||||
- `devops/compose/postgres-init` remains bootstrap-only (first initialization).
|
||||
- Release upgrades must never rely on `docker-entrypoint-initdb.d`; they must use the canonical CLI entrypoint.
|
||||
|
||||
UI/API execution path (implemented):
|
||||
|
||||
- Module registry ownership is platform-level: `src/Platform/__Libraries/StellaOps.Platform.Database/MigrationModuleRegistry.cs`.
|
||||
- UI-triggered migration execution must call Platform WebService administrative APIs (no browser-direct database execution).
|
||||
- Platform endpoint contract:
|
||||
- `GET /api/v1/admin/migrations/modules`
|
||||
- `GET /api/v1/admin/migrations/status?module=<name|all>`
|
||||
- `GET /api/v1/admin/migrations/verify?module=<name|all>`
|
||||
- `POST /api/v1/admin/migrations/run`
|
||||
- Endpoint authorization policy: `platform.setup.admin` (`PlatformPolicies.SetupAdmin`).
|
||||
- Endpoint implementation and server-side runner path:
|
||||
- `src/Platform/StellaOps.Platform.WebService/Endpoints/MigrationAdminEndpoints.cs`
|
||||
- `src/Platform/StellaOps.Platform.WebService/Services/PlatformMigrationAdminService.cs`
|
||||
|
||||
## Legacy Compatibility Mapping
|
||||
|
||||
The following non-canonical history tables must be migrated to `<schema>.schema_migrations` via one-time compatibility scripts/adapters:
|
||||
|
||||
| Module | Legacy history table | Mapping target |
|
||||
| --- | --- | --- |
|
||||
| EvidenceLocker | `evidence_locker.evidence_schema_version` | `evidence_locker.schema_migrations` |
|
||||
| ExportCenter | `export_center.export_schema_version` | `export_center.schema_migrations` |
|
||||
| BinaryIndex | `binaries.schema_migrations` (`name`, `applied_at`) | `binaries.schema_migrations` canonical columns |
|
||||
| Plugin Registry | `<schema>.plugin_migrations` | `<schema>.schema_migrations` |
|
||||
|
||||
Compatibility rules:
|
||||
|
||||
- Preserve applied order and timestamps.
|
||||
- Preserve or derive deterministic checksums for already-applied scripts.
|
||||
- Never rewrite already-applied migration file contents.
|
||||
|
||||
## Cutover Waves (Runner Entrypoint Consolidation)
|
||||
|
||||
| Wave | Goal | Module Set | Current Mechanism | Target Mechanism | Rollback Marker |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| W0 | Governance baseline | Docs + CLI contract | Mixed | Canonical policy published | Docs and runbooks merged |
|
||||
| W1 | Expand CLI/shared-runner coverage | Authority, Scheduler, Concelier, Policy, Notify, Excititor, Scanner, AirGap, TimelineIndexer, ReleaseOrchestrator | Shared runner partial coverage | Canonical CLI entrypoint for all shared-runner modules | `migrations-status` parity before/after |
|
||||
| W2 | Convert custom history runners | EvidenceLocker, ExportCenter, BinaryIndex, Plugin Registry | Custom runners/history tables | Shared `MigrationRunner` with compatibility mapping | Legacy table snapshot and compare report |
|
||||
| W3 | Wire currently unwired migration folders | Modules marked unwired in `docs/db/MIGRATION_INVENTORY.md` | Embedded SQL without runtime invocation | Shared runner via startup/CLI registration | Per-module dry-run evidence |
|
||||
| W4 | Count consolidation/baselining | High-count modules (for example Platform release scripts) | Long historical chains | Approved baseline/squash strategy | Rehearsed upgrade from previous GA |
|
||||
| W5 | On-prem rehearsal gate | Full product stack | Mixed in-flight state | Canonical entrypoint only | Deterministic replay report |
|
||||
|
||||
## Migration Count Consolidation Rules
|
||||
|
||||
- Baseline/squash is allowed only after:
|
||||
- at least one release where compatibility mappings are shipped,
|
||||
- deterministic replay tests pass for clean install and upgrade paths,
|
||||
- installed-version compatibility window is documented.
|
||||
- A baseline migration must not remove upgradeability from supported in-field versions.
|
||||
- Squash output must retain checksum determinism and clear mapping to superseded migration ranges.
|
||||
|
||||
## EF Core v10 Phase Gate
|
||||
|
||||
The Dapper-to-EF Core v10 transition starts only after waves W1-W5 complete and gate evidence is accepted.
|
||||
|
||||
Exit criteria before EF phase opens:
|
||||
|
||||
- One canonical migration mechanism in production paths.
|
||||
- One canonical operational entrypoint in runbooks and CI/CD automation.
|
||||
- Legacy history tables mapped and validated.
|
||||
- Migration replay determinism proven for clean install and upgrade scenarios.
|
||||
Reference in New Issue
Block a user