stela ops usage fixes roles propagation and timoeut, one account to support multi tenants, migrations consolidation, search to support documentation, doctor and open api vector db search

This commit is contained in:
master
2026-02-22 19:27:54 +02:00
parent a29f438f53
commit bd8fee6ed8
373 changed files with 832097 additions and 3369 deletions

View File

@@ -1,13 +1,50 @@
# PostgreSQL Migration Strategy
**Version:** 1.0
**Last Updated:** 2025-12-03
**Version:** 1.1
**Last Updated:** 2026-02-22
**Status:** Active
## Overview
This document defines the migration strategy for StellaOps PostgreSQL databases. It covers initial setup, per-release migrations, multi-instance coordination, and air-gapped operation.
## Consolidation Baseline (2026-02-22)
This strategy is under active consolidation. The current repository baseline is tracked in:
- `docs/db/MIGRATION_INVENTORY.md`
- `docs/db/MIGRATION_CONSOLIDATION_PLAN.md`
Current-state realities that must be accounted for in operations:
- Multiple migration mechanisms are active (shared `MigrationRunner`, `StartupMigrationHost` wrappers, custom runners with custom history tables, compose bootstrap init SQL, and unwired migration folders).
- CLI migration coverage is currently limited to the modules registered in `src/Platform/__Libraries/StellaOps.Platform.Database/MigrationModuleRegistry.cs`.
- Registry module population is plug-in based (`IMigrationModulePlugin`) with one migration plug-in per web service.
- Platform migration admin endpoints (`/api/v1/admin/migrations/*`) use the same platform-owned registry for UI/backend orchestration.
- Several services contain migration SQL but have no verified runtime invocation path in non-test code.
Until consolidation is complete, treat this document as the target policy and `docs/db/MIGRATION_INVENTORY.md` as the source of truth for current implementation state.
## Canonical Mechanism and Entrypoint (Target)
For on-prem upgradeable deployments, the target state is:
- One migration mechanism:
- `src/__Libraries/StellaOps.Infrastructure.Postgres/Migrations/MigrationRunner.cs`
- `src/__Libraries/StellaOps.Infrastructure.Postgres/Migrations/StartupMigrationHost.cs`
- One migration history table format: `<schema>.schema_migrations`
- One operational release gate with one shared module catalog:
- `stella system migrations-status --module all`
- `stella system migrations-verify --module all`
- `stella system migrations-run --module all --category release --dry-run`
- `stella system migrations-run --module all --category release --force`
- `GET /api/v1/admin/migrations/status?module=all`
- `GET /api/v1/admin/migrations/verify?module=all`
- `POST /api/v1/admin/migrations/run`
Service startup paths may run only automatic categories (`startup`, `seed`) through `StartupMigrationHost`.
Pending `release` or `data` migrations are handled through the canonical CLI/API execution path before service rollout.
## Principles
1. **Forward-Only**: No down migrations. Fixes are applied as new forward migrations.
@@ -40,9 +77,9 @@ Run automatically when application starts. Must complete within 60 seconds.
- Large data migrations (> 10,000 rows affected)
- Any operation requiring `ACCESS EXCLUSIVE` lock for extended periods
### Category B: Release Migrations (Manual/CLI)
### Category B: Release Migrations (Manual/CLI/API)
Require explicit execution via CLI before deployment. Used for breaking changes.
Require explicit execution through CLI or Platform migration admin API before deployment. Used for breaking changes.
**Typical Operations:**
- Dropping deprecated columns/tables
@@ -128,7 +165,7 @@ src/<Module>/__Libraries/StellaOps.<Module>.Storage.Postgres/
┌─────────────────────────────────────────────────────────────┐
│ 5. Check for pending Category B migrations │
│ - If any 100+ migrations are pending: FAIL STARTUP │
│ - Log: "Run 'stellaops migrate' before deployment"
│ - Log: "Run 'stella system migrations-run ...' before deployment" │
└─────────────────────────────────────────────────────────────┘
@@ -155,17 +192,28 @@ src/<Module>/__Libraries/StellaOps.<Module>.Storage.Postgres/
└─────────────────────────────────────────────────────────────┘
```
### Release Migration (CLI)
### Release Migration (CLI/API)
```bash
# Before deployment - run breaking migrations
stellaops system migrations-run --module Authority --category release
stella system migrations-run --module Authority --category release --force
# Verify migration state
stellaops system migrations-status --module Authority
stella system migrations-status --module Authority
# Dry run (show what would be executed)
stellaops system migrations-run --module Authority --dry-run
stella system migrations-run --module Authority --category release --dry-run
```
```bash
# API equivalent (used by Platform UI/backend orchestration)
curl -sk -H "Authorization: Bearer ${TOKEN}" \
"https://platform.stella-ops.local/api/v1/admin/migrations/status?module=Authority"
curl -sk -H "Authorization: Bearer ${TOKEN}" \
-H "Content-Type: application/json" \
-d '{"module":"Authority","category":"release","dryRun":true}' \
"https://platform.stella-ops.local/api/v1/admin/migrations/run"
```
## Multi-Instance Coordination
@@ -285,16 +333,16 @@ At startup, migrations are validated for:
```bash
# 1. Review pending migrations
stellaops system migrations-status --module all
stella system migrations-status --module all
# 2. Backup database (if required)
pg_dump -Fc stellaops > backup_$(date +%Y%m%d).dump
# 3. Run release migrations in maintenance window
stellaops system migrations-run --category release --module all
stella system migrations-run --category release --module all --force
# 4. Verify schema state
stellaops system migrations-verify --module all
stella system migrations-verify --module all
```
### Deployment
@@ -307,10 +355,10 @@ stellaops system migrations-verify --module all
```bash
# Check migration status
stellaops system migrations-status --module all
stella system migrations-status --module all
# Run any data migrations (background)
stellaops system migrations-run --category data --module all
stella system migrations-run --category data --module all
```
## Rollback Strategy
@@ -401,7 +449,7 @@ EOF
dotnet run --project src/Authority/StellaOps.Authority.WebService
# 4. Verify migration applied
stellaops system migrations-status --module Authority
stella system migrations-status --module Authority
```
### Testing Migrations
@@ -411,8 +459,8 @@ stellaops system migrations-status --module Authority
dotnet test --filter "Category=Migration"
# Test idempotency (run twice)
stellaops system migrations-run --module Authority
stellaops system migrations-run --module Authority # Should be no-op
stella system migrations-run --module Authority
stella system migrations-run --module Authority # Should be no-op
```
## Troubleshooting
@@ -454,7 +502,7 @@ ERROR: Migration checksum mismatch for '003_add_audit_columns.sql'
```
ERROR: Cannot start application - pending release migrations require manual execution
Pending: 100_drop_legacy_columns.sql
Run: stellaops system migrations-run --module Authority --category release
Run: stella system migrations-run --module Authority --category release --force
```
**Resolution**: Run CLI migration command before deployment.