wip: doctor/cli/docs/api to vector db consolidation; api hardening for descriptions, tenant, and scopes; migrations and conversions of all DALs to EF v10
This commit is contained in:
master
434
docs/db/EF_CORE_MODEL_GENERATION_STANDARDS.md
Normal file
434
docs/db/EF_CORE_MODEL_GENERATION_STANDARDS.md
Normal file
@@ -0,0 +1,434 @@
|
||||
# EF Core v10 Model Generation Standards
|
||||
|
||||
> Authoritative reference for EF Core model generation conventions in Stella Ops.
|
||||
> Derived from completed reference implementations: TimelineIndexer (Sprint 063) and AirGap (Sprint 064).
|
||||
|
||||
## 1. DbContext Structure
|
||||
|
||||
### 1.1 File Layout
|
||||
|
||||
Every module's EF Core implementation must follow this directory structure:
|
||||
|
||||
```
|
||||
src/<Module>/__Libraries/StellaOps.<Module>.Persistence/
|
||||
EfCore/
|
||||
Context/
|
||||
<Module>DbContext.cs # Main DbContext (scaffolded)
|
||||
<Module>DbContext.Partial.cs # Relationship overlays (manual)
|
||||
<Module>DesignTimeDbContextFactory.cs # For dotnet ef CLI
|
||||
Models/
|
||||
<EntityName>.cs # Scaffolded entity POCOs
|
||||
<EntityName>.Partials.cs # Navigation properties / enum overlays (manual)
|
||||
CompiledModels/ # Auto-generated by dotnet ef dbcontext optimize
|
||||
<Module>DbContextModel.cs
|
||||
<Module>DbContextModelBuilder.cs
|
||||
<EntityName>EntityType.cs # Per-entity compiled metadata
|
||||
<Module>DbContextAssemblyAttributes.cs # May be excluded from compilation
|
||||
Postgres/
|
||||
<Module>DbContextFactory.cs # Runtime factory with compiled model hookup
|
||||
<Module>DataSource.cs # NpgsqlDataSource + enum mapping
|
||||
Repositories/
|
||||
Postgres<Store>.cs # EF-backed repository implementations
|
||||
```
|
||||
|
||||
### 1.2 DbContext Class Rules
|
||||
|
||||
1. **Use `partial class`** to separate scaffolded configuration from manual overlays.
|
||||
2. **Main file** contains `OnModelCreating` with table/index/column mappings.
|
||||
3. **Partial file** contains `OnModelCreatingPartial` with relationship configuration, enum mappings, and navigation property wiring.
|
||||
4. **Schema injection**: accept schema name via constructor parameter with a default fallback.
|
||||
|
||||
```csharp
|
||||
public partial class <Module>DbContext : DbContext
|
||||
{
|
||||
private readonly string _schemaName;
|
||||
|
||||
public <Module>DbContext(DbContextOptions<<Module>DbContext> options, string? schemaName = null)
|
||||
: base(options)
|
||||
{
|
||||
_schemaName = string.IsNullOrWhiteSpace(schemaName)
|
||||
? "<default_schema>"
|
||||
: schemaName.Trim();
|
||||
}
|
||||
|
||||
// DbSet properties for each entity
|
||||
public virtual DbSet<EntityName> EntityNames { get; set; }
|
||||
|
||||
protected override void OnModelCreating(ModelBuilder modelBuilder)
|
||||
{
|
||||
var schemaName = _schemaName;
|
||||
// Table, index, column, default value configurations
|
||||
// ...
|
||||
OnModelCreatingPartial(modelBuilder);
|
||||
}
|
||||
|
||||
partial void OnModelCreatingPartial(ModelBuilder modelBuilder);
|
||||
}
|
||||
```
|
||||
|
||||
### 1.3 Naming Conventions
|
||||
|
||||
| Aspect | Convention | Example |
|
||||
| --- | --- | --- |
|
||||
| DbContext class | `<Module>DbContext` | `AirGapDbContext` |
|
||||
| Entity class (DB-aligned) | snake_case matching table | `timeline_event` |
|
||||
| Entity class (domain-aligned) | PascalCase | `BundleVersion` |
|
||||
| DbSet property | PascalCase plural or snake_case plural | `BundleVersions` or `timeline_events` |
|
||||
| Column mapping | Always explicit `HasColumnName("snake_case")` | `.HasColumnName("tenant_id")` |
|
||||
| Table mapping | Always explicit `ToTable("name", schema)` | `.ToTable("states", "airgap")` |
|
||||
| Index naming | `idx_<module>_<table>_<columns>` or `ix_<table>_<columns>` | `idx_airgap_bundle_versions_tenant` |
|
||||
| Key naming | `<table>_pkey` | `bundle_versions_pkey` |
|
||||
|
||||
**Decision**: Both DB-aligned snake_case and domain-aligned PascalCase entity naming are valid. Choose one per module and be consistent. New modules should prefer PascalCase entities with explicit column mappings.
|
||||
|
||||
## 2. Entity Model Rules
|
||||
|
||||
### 2.1 Base Entity Structure
|
||||
|
||||
- Use `partial class` to separate scaffolded properties from manual additions.
|
||||
- Scaffolded file: scalar properties only (no navigation properties).
|
||||
- Partial file: navigation properties, custom enum properties, collection initializers.
|
||||
|
||||
```csharp
|
||||
// Scaffolded file: <EntityName>.cs
|
||||
public partial class BundleVersion
|
||||
{
|
||||
public string TenantId { get; set; } = null!;
|
||||
public string BundleType { get; set; } = null!;
|
||||
public string VersionString { get; set; } = null!;
|
||||
public DateTime CreatedAt { get; set; }
|
||||
public DateTime UpdatedAt { get; set; }
|
||||
}
|
||||
|
||||
// Manual overlay: <EntityName>.Partials.cs
|
||||
public partial class BundleVersion
|
||||
{
|
||||
// Navigation properties added manually
|
||||
public virtual ICollection<BundleVersionHistory> Histories { get; set; } = new List<BundleVersionHistory>();
|
||||
}
|
||||
```
|
||||
|
||||
### 2.2 Type Mapping Rules
|
||||
|
||||
| CLR Type | PostgreSQL Type | Convention |
|
||||
| --- | --- | --- |
|
||||
| `string` | `text` / `varchar` | Use `= null!` for required, `string?` for nullable |
|
||||
| `DateTime` | `timestamp without time zone` | Store UTC, use `HasDefaultValueSql("now()")` for DB defaults |
|
||||
| `Guid` | `uuid` | Use `HasDefaultValueSql("gen_random_uuid()")` if DB-generated |
|
||||
| `string` (JSON) | `jsonb` | Store as `string`, deserialize in domain layer; annotate with `HasColumnType("jsonb")` |
|
||||
| Custom enum | Custom PostgreSQL enum type | Map via `[PgName]` attribute + `DataSourceBuilder.MapEnum<T>()` |
|
||||
| `long` (identity) | `bigint GENERATED` | Use `ValueGenerated.OnAdd` + `NpgsqlValueGenerationStrategy.IdentityByDefaultColumn` |
|
||||
|
||||
### 2.3 PostgreSQL Enum Mapping
|
||||
|
||||
1. Define CLR enum with `[PgName]` attributes:
|
||||
|
||||
```csharp
|
||||
public enum EventSeverity
|
||||
{
|
||||
[PgName("info")] Info,
|
||||
[PgName("notice")] Notice,
|
||||
[PgName("warn")] Warn,
|
||||
[PgName("error")] Error,
|
||||
[PgName("critical")] Critical
|
||||
}
|
||||
```
|
||||
|
||||
2. Register in DataSource builder:
|
||||
|
||||
```csharp
|
||||
protected override void ConfigureDataSourceBuilder(NpgsqlDataSourceBuilder builder)
|
||||
{
|
||||
base.ConfigureDataSourceBuilder(builder);
|
||||
builder.MapEnum<EventSeverity>("<schema>.event_severity");
|
||||
}
|
||||
```
|
||||
|
||||
3. Register in DbContext `OnModelCreating`:
|
||||
|
||||
```csharp
|
||||
modelBuilder.HasPostgresEnum("<schema>", "event_severity",
|
||||
new[] { "info", "notice", "warn", "error", "critical" });
|
||||
```
|
||||
|
||||
## 3. Design-Time Factory
|
||||
|
||||
Every module must provide an `IDesignTimeDbContextFactory<T>` for `dotnet ef` CLI tooling.
|
||||
|
||||
```csharp
|
||||
public sealed class <Module>DesignTimeDbContextFactory
|
||||
: IDesignTimeDbContextFactory<<Module>DbContext>
|
||||
{
|
||||
private const string DefaultConnectionString =
|
||||
"Host=localhost;Port=55433;Database=postgres;Username=postgres;Password=postgres";
|
||||
private const string ConnectionStringEnvironmentVariable =
|
||||
"STELLAOPS_<MODULE_UPPER>_EF_CONNECTION";
|
||||
|
||||
public <Module>DbContext CreateDbContext(string[] args)
|
||||
{
|
||||
var connectionString = ResolveConnectionString();
|
||||
var options = new DbContextOptionsBuilder<<Module>DbContext>()
|
||||
.UseNpgsql(connectionString)
|
||||
.Options;
|
||||
return new <Module>DbContext(options);
|
||||
}
|
||||
|
||||
private static string ResolveConnectionString()
|
||||
{
|
||||
var fromEnvironment =
|
||||
Environment.GetEnvironmentVariable(ConnectionStringEnvironmentVariable);
|
||||
return string.IsNullOrWhiteSpace(fromEnvironment)
|
||||
? DefaultConnectionString
|
||||
: fromEnvironment;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Rules**:
|
||||
- Environment variable naming: `STELLAOPS_<MODULE_UPPER>_EF_CONNECTION`
|
||||
- Default connection: localhost dev database
|
||||
- Design-time factory must NOT use compiled models (uses reflection-based discovery)
|
||||
- Port should match module's dev compose port or default `55433`
|
||||
|
||||
## 4. Compiled Model Generation
|
||||
|
||||
### 4.1 Generation Command
|
||||
|
||||
```bash
|
||||
dotnet ef dbcontext optimize \
|
||||
--project src/<Module>/__Libraries/StellaOps.<Module>.Persistence/ \
|
||||
--output-dir EfCore/CompiledModels \
|
||||
--namespace StellaOps.<Module>.Persistence.EfCore.CompiledModels
|
||||
```
|
||||
|
||||
### 4.2 Generated Artifacts
|
||||
|
||||
The `dotnet ef dbcontext optimize` command produces:
|
||||
- `<Module>DbContextModel.cs` - Singleton `RuntimeModel` with thread-safe initialization
|
||||
- `<Module>DbContextModelBuilder.cs` - Entity type registration and annotations
|
||||
- Per-entity `<EntityName>EntityType.cs` files with property/key/index metadata
|
||||
- `<Module>DbContextAssemblyAttributes.cs` - Assembly-level `[DbContextModel]` attribute
|
||||
|
||||
### 4.3 Assembly Attribute Exclusion
|
||||
|
||||
**Critical**: When a module supports non-default schemas for integration testing, the assembly attribute file must be excluded from compilation to prevent automatic compiled model binding:
|
||||
|
||||
```xml
|
||||
<!-- In .csproj -->
|
||||
<ItemGroup>
|
||||
<Compile Remove="EfCore\CompiledModels\<Module>DbContextAssemblyAttributes.cs" />
|
||||
</ItemGroup>
|
||||
```
|
||||
|
||||
This ensures non-default schemas build runtime models dynamically while the default schema path uses the static compiled model.
|
||||
|
||||
### 4.4 Regeneration Workflow
|
||||
|
||||
When the DbContext or model configuration changes:
|
||||
1. Update `OnModelCreating` / partial files as needed
|
||||
2. Run `dotnet ef dbcontext optimize` to regenerate compiled models
|
||||
3. Verify assembly attribute exclusion is still in `.csproj`
|
||||
4. Run sequential build and tests to validate
|
||||
|
||||
## 5. Runtime DbContext Factory
|
||||
|
||||
Every module must provide a static runtime factory that applies the compiled model for the default schema:
|
||||
|
||||
```csharp
|
||||
internal static class <Module>DbContextFactory
|
||||
{
|
||||
public static <Module>DbContext Create(
|
||||
NpgsqlConnection connection,
|
||||
int commandTimeoutSeconds,
|
||||
string schemaName)
|
||||
{
|
||||
var normalizedSchema = string.IsNullOrWhiteSpace(schemaName)
|
||||
? <Module>DataSource.DefaultSchemaName
|
||||
: schemaName.Trim();
|
||||
|
||||
var optionsBuilder = new DbContextOptionsBuilder<<Module>DbContext>()
|
||||
.UseNpgsql(connection, npgsql => npgsql.CommandTimeout(commandTimeoutSeconds));
|
||||
|
||||
// Use static compiled model ONLY for default schema path
|
||||
if (string.Equals(normalizedSchema, <Module>DataSource.DefaultSchemaName,
|
||||
StringComparison.Ordinal))
|
||||
{
|
||||
optionsBuilder.UseModel(<Module>DbContextModel.Instance);
|
||||
}
|
||||
|
||||
return new <Module>DbContext(optionsBuilder.Options, normalizedSchema);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Rules**:
|
||||
- Compiled model applied only when schema matches the default (deterministic path).
|
||||
- Non-default schemas (integration tests) use reflection-based model building.
|
||||
- Accept `NpgsqlConnection` from the module's `DataSource` (connection pooling).
|
||||
- Accept command timeout as parameter (configurable per operation).
|
||||
|
||||
## 6. DataSource Registration
|
||||
|
||||
Every module extends `DataSourceBase` for connection management and enum mapping:
|
||||
|
||||
```csharp
|
||||
public sealed class <Module>DataSource : DataSourceBase
|
||||
{
|
||||
public const string DefaultSchemaName = "<schema>";
|
||||
|
||||
public <Module>DataSource(
|
||||
IOptions<PostgresOptions> options,
|
||||
ILogger<<Module>DataSource> logger)
|
||||
: base(EnsureSchema(options.Value), logger) { }
|
||||
|
||||
protected override string ModuleName => "<Module>";
|
||||
|
||||
protected override void ConfigureDataSourceBuilder(NpgsqlDataSourceBuilder builder)
|
||||
{
|
||||
base.ConfigureDataSourceBuilder(builder);
|
||||
// Map custom PostgreSQL enum types
|
||||
builder.MapEnum<CustomEnum>("<schema>.enum_type_name");
|
||||
}
|
||||
|
||||
private static PostgresOptions EnsureSchema(PostgresOptions baseOptions)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(baseOptions.SchemaName))
|
||||
baseOptions.SchemaName = DefaultSchemaName;
|
||||
return baseOptions;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## 7. Project File (.csproj) Configuration
|
||||
|
||||
Required elements for every EF Core-enabled persistence project:
|
||||
|
||||
```xml
|
||||
<ItemGroup>
|
||||
<!-- Embed SQL migrations as resources -->
|
||||
<EmbeddedResource Include="Migrations\**\*.sql"
|
||||
LogicalName="%(RecursiveDir)%(Filename)%(Extension)" />
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<!-- Exclude assembly attribute for non-default schema support -->
|
||||
<Compile Remove="EfCore\CompiledModels\<Module>DbContextAssemblyAttributes.cs" />
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<PackageReference Include="Microsoft.EntityFrameworkCore" />
|
||||
<PackageReference Include="Microsoft.EntityFrameworkCore.Design" PrivateAssets="all" />
|
||||
<PackageReference Include="Npgsql" />
|
||||
<PackageReference Include="Npgsql.EntityFrameworkCore.PostgreSQL" />
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="..\..\..\..\__Libraries\StellaOps.Infrastructure.Postgres\StellaOps.Infrastructure.Postgres.csproj" />
|
||||
<ProjectReference Include="..\..\..\..\__Libraries\StellaOps.Infrastructure.EfCore\StellaOps.Infrastructure.EfCore.csproj" />
|
||||
</ItemGroup>
|
||||
```
|
||||
|
||||
## 8. Dependency Injection Pattern
|
||||
|
||||
```csharp
|
||||
public static IServiceCollection Add<Module>Persistence(
|
||||
this IServiceCollection services,
|
||||
IConfiguration configuration,
|
||||
string sectionName = "Postgres:<Module>")
|
||||
{
|
||||
services.Configure<PostgresOptions>(configuration.GetSection(sectionName));
|
||||
services.AddSingleton<<Module>DataSource>();
|
||||
services.AddHostedService<<Module>MigrationHostedService>();
|
||||
services.AddScoped<IRepository, ConcreteRepository>();
|
||||
return services;
|
||||
}
|
||||
```
|
||||
|
||||
**Lifecycle rules**:
|
||||
- `DataSource`: Singleton (connection pool reuse)
|
||||
- `MigrationRunner`: Singleton
|
||||
- `MigrationHostedService`: Hosted service (runs at startup)
|
||||
- Repositories: Scoped (per-request)
|
||||
|
||||
## 9. Repository Pattern with EF Core
|
||||
|
||||
### 9.1 Read Operations
|
||||
|
||||
```csharp
|
||||
await using var connection = await DataSource.OpenConnectionAsync(tenantId, "reader", ct);
|
||||
await using var dbContext = <Module>DbContextFactory.Create(connection, CommandTimeoutSeconds, GetSchemaName());
|
||||
|
||||
var result = await dbContext.Entities
|
||||
.AsNoTracking()
|
||||
.Where(e => e.TenantId == tenantId)
|
||||
.ToListAsync(ct);
|
||||
```
|
||||
|
||||
**Rules**:
|
||||
- Always use `AsNoTracking()` for read-only queries.
|
||||
- Create DbContext per operation (not cached).
|
||||
- Use tenant-scoped connection from DataSource.
|
||||
|
||||
### 9.2 Write Operations
|
||||
|
||||
```csharp
|
||||
await using var connection = await DataSource.OpenConnectionAsync(tenantId, "writer", ct);
|
||||
await using var dbContext = <Module>DbContextFactory.Create(connection, CommandTimeoutSeconds, GetSchemaName());
|
||||
|
||||
dbContext.Entities.Add(newEntity);
|
||||
try
|
||||
{
|
||||
await dbContext.SaveChangesAsync(ct);
|
||||
}
|
||||
catch (DbUpdateException ex) when (IsUniqueViolation(ex))
|
||||
{
|
||||
// Handle idempotency
|
||||
}
|
||||
```
|
||||
|
||||
### 9.3 Unique Violation Detection
|
||||
|
||||
```csharp
|
||||
private static bool IsUniqueViolation(DbUpdateException exception)
|
||||
{
|
||||
Exception? current = exception;
|
||||
while (current is not null)
|
||||
{
|
||||
if (current is PostgresException { SqlState: PostgresErrorCodes.UniqueViolation })
|
||||
return true;
|
||||
current = current.InnerException;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
```
|
||||
|
||||
## 10. Schema Compatibility Rules
|
||||
|
||||
### 10.1 SQL Migration Governance Preserved
|
||||
|
||||
- SQL migrations remain the authoritative schema definition.
|
||||
- EF Core models are scaffolded FROM the existing schema, not the reverse.
|
||||
- No EF Core auto-migrations (`EnsureCreated`, `Migrate`) are permitted at runtime.
|
||||
- Schema changes require new SQL migration files following existing naming/category conventions.
|
||||
|
||||
### 10.2 Schema Validation Checks (Per Module)
|
||||
|
||||
Before marking a module's EF conversion as complete:
|
||||
1. Verify all tables referenced by repositories are represented as DbSets.
|
||||
2. Verify column names, types, and nullability match the SQL migration schema.
|
||||
3. Verify indices defined in SQL are reflected in `OnModelCreating` (for query plan awareness).
|
||||
4. Verify foreign key relationships match SQL constraints.
|
||||
5. Verify PostgreSQL-specific types (jsonb, custom enums, extensions) are properly mapped.
|
||||
|
||||
### 10.3 Tenant Isolation Preserved
|
||||
|
||||
- Tenant isolation via RLS policies remains in SQL migrations.
|
||||
- Connection-level tenant context set via `DataSource.OpenConnectionAsync(tenantId, role)`.
|
||||
- No application-level tenant filtering replaces RLS (RLS is the authoritative enforcement).
|
||||
- Non-tenant-scoped modules (e.g., VexHub) document their global scope explicitly.
|
||||
|
||||
## Revision History
|
||||
|
||||
| Date | Change | Author |
|
||||
| --- | --- | --- |
|
||||
| 2026-02-22 | Initial standards derived from TimelineIndexer and AirGap reference implementations. | Documentation Author |
|
||||
149
docs/db/EF_CORE_RUNTIME_CUTOVER_STRATEGY.md
Normal file
149
docs/db/EF_CORE_RUNTIME_CUTOVER_STRATEGY.md
Normal file
@@ -0,0 +1,149 @@
|
||||
# EF Core Runtime Cutover Strategy (Dapper/Npgsql to EF Core)
|
||||
|
||||
> Authoritative reference for how modules transition read/write paths from Dapper/raw Npgsql repositories to EF Core-backed repositories without breaking deterministic behavior.
|
||||
> Supports the EF Core v10 Dapper Transition Phase Gate (Sprint 062).
|
||||
|
||||
## 1. Cutover Pattern Overview
|
||||
|
||||
The transition follows a **repository-level in-place replacement** pattern:
|
||||
- Repository interfaces remain unchanged (no consumer-facing API changes).
|
||||
- Repository implementations are rewritten internally from Dapper/Npgsql SQL to EF Core operations.
|
||||
- Both old and new implementations satisfy the same interface contract and behavioral invariants.
|
||||
- The cutover is atomic per repository class (no partial Dapper+EF mixing within a single repository).
|
||||
|
||||
This pattern was validated in TimelineIndexer (Sprint 063) and AirGap (Sprint 064).
|
||||
|
||||
## 2. Per-Module Cutover Sequence
|
||||
|
||||
Each module follows this ordered sequence:
|
||||
|
||||
### Step 1: Pre-Cutover Baseline
|
||||
- Ensure all existing tests pass (sequential, `/m:1`, no parallelism).
|
||||
- Document current DAL technology and repository class inventory.
|
||||
- Verify migration is registered in Platform migration module registry.
|
||||
|
||||
### Step 2: EF Core Scaffold
|
||||
- Provision schema from migration SQL.
|
||||
- Run `dotnet ef dbcontext scaffold` for module schema.
|
||||
- Place scaffolded output in `EfCore/Context/` and `EfCore/Models/`.
|
||||
- Add partial overlays for relationships, enums, navigation properties.
|
||||
|
||||
### Step 3: Repository Rewrite
|
||||
- For each repository class:
|
||||
1. Replace Dapper `connection.QueryAsync<T>()` / `connection.ExecuteAsync()` calls with EF Core `dbContext.Entities.Where().ToListAsync()` / `dbContext.SaveChangesAsync()`.
|
||||
2. Replace raw `NpgsqlCommand` + `NpgsqlDataReader` patterns with EF Core LINQ queries.
|
||||
3. Preserve transaction boundaries (use `dbContext.Database.BeginTransactionAsync()` where the original used explicit transactions).
|
||||
4. Preserve idempotency handling (catch `DbUpdateException` with `UniqueViolation` instead of raw `PostgresException`).
|
||||
5. Preserve ordering semantics (`.OrderByDescending()` matching original `ORDER BY` clauses).
|
||||
6. Preserve tenant scoping (connection obtained from `DataSource.OpenConnectionAsync(tenantId, role)`).
|
||||
|
||||
### Step 4: Compiled Model Generation
|
||||
- Run `dotnet ef dbcontext optimize` to generate compiled model artifacts.
|
||||
- Wire `UseModel(<Module>DbContextModel.Instance)` in runtime factory for default schema.
|
||||
- Exclude assembly attributes if module tests use non-default schemas.
|
||||
|
||||
### Step 5: Post-Cutover Validation
|
||||
- Run targeted module tests sequentially.
|
||||
- Verify no behavioral regressions in ordering, idempotency, tenant isolation.
|
||||
- Update module docs to reflect EF-backed DAL.
|
||||
|
||||
## 3. Dapper Retirement Criteria
|
||||
|
||||
A module's Dapper dependency can be removed when ALL of these are true:
|
||||
1. Every repository interface implementation uses EF Core exclusively (no remaining Dapper calls).
|
||||
2. No utility code depends on Dapper extension methods (`SqlMapper`, `DynamicParameters`, etc.).
|
||||
3. Sequential build/test passes without the Dapper package reference.
|
||||
4. The `<PackageReference Include="Dapper" />` is removed from the persistence `.csproj`.
|
||||
|
||||
**Important**: Do not remove Dapper from `.csproj` until ALL repositories in the module are converted. Mixed Dapper+EF within a persistence project is acceptable during the transition window.
|
||||
|
||||
## 4. Adapter Pattern (When Required)
|
||||
|
||||
For modules with complex DAL logic (e.g., Scanner with 36 migrations, Policy with mixed DAL), a temporary adapter pattern may be used:
|
||||
|
||||
```csharp
|
||||
// Temporary adapter that delegates to either Dapper or EF implementation
|
||||
internal sealed class HybridRepository : IRepository
|
||||
{
|
||||
private readonly DapperRepository _legacy;
|
||||
private readonly EfCoreRepository _modern;
|
||||
private readonly bool _useEfCore;
|
||||
|
||||
public HybridRepository(DapperRepository legacy, EfCoreRepository modern, IOptions<DalOptions> options)
|
||||
{
|
||||
_legacy = legacy;
|
||||
_modern = modern;
|
||||
_useEfCore = options.Value.UseEfCore;
|
||||
}
|
||||
|
||||
public Task<T> GetAsync(string id, CancellationToken ct)
|
||||
=> _useEfCore ? _modern.GetAsync(id, ct) : _legacy.GetAsync(id, ct);
|
||||
}
|
||||
```
|
||||
|
||||
**Adapter rules**:
|
||||
- Only use for Wave B/C modules (orders 17+) where complexity justifies gradual rollout.
|
||||
- Wave A modules (orders 2-16, single migration) must use direct replacement (no adapter).
|
||||
- Configuration flag: `DalOptions.UseEfCore` (default: `true` for new deployments, `false` for upgrades until validated).
|
||||
- Retirement: adapter removed once EF path is validated in production and upgrade rehearsal passes.
|
||||
|
||||
## 5. Rollback Strategy Per Wave
|
||||
|
||||
### Wave A (Orders 2-16: Single-Migration Modules)
|
||||
- **Rollback**: revert the repository `.cs` files to pre-conversion state via git.
|
||||
- **Risk**: minimal; single migration, small repositories, no schema changes.
|
||||
- **Decision authority**: Developer can self-approve rollback.
|
||||
|
||||
### Wave B (Orders 17-23: Medium-Complexity Modules)
|
||||
- **Rollback**: revert repository files + remove compiled model artifacts.
|
||||
- **Risk**: moderate; some modules have multiple migration sources or shared-runner dependencies.
|
||||
- **Decision authority**: Developer + Project Manager approval.
|
||||
- **Validation**: must re-run sequential build/test after rollback to confirm clean state.
|
||||
|
||||
### Wave C (Orders 24-32: High-Complexity Modules)
|
||||
- **Rollback**: revert all EF artifacts, restore Dapper package reference if removed, re-run migrations.
|
||||
- **Risk**: high; custom histories, large migration chains, mixed DAL internals.
|
||||
- **Decision authority**: Project Manager + Platform owner approval.
|
||||
- **Validation**: must re-run full module test suite + Platform registry validation + migration status check.
|
||||
- **Mitigation**: adapter pattern (Section 4) available for controlled rollout.
|
||||
|
||||
## 6. Behavioral Invariants (Must Preserve)
|
||||
|
||||
Every cutover must preserve these behaviors identically:
|
||||
|
||||
| Invariant | Validation Method |
|
||||
| --- | --- |
|
||||
| Ordering semantics | Compare query results ordering (pre/post); verify `ORDER BY` equivalence in LINQ `.OrderBy`/`.OrderByDescending` |
|
||||
| Idempotency | Duplicate insert/upsert tests must produce same outcome (no error, no duplicate data) |
|
||||
| Tenant isolation | Multi-tenant integration tests verify data never leaks across tenant boundaries |
|
||||
| Transaction atomicity | Multi-step write operations remain atomic (all-or-nothing); test rollback scenarios |
|
||||
| NULL handling | Nullable columns preserve NULL vs empty-string distinction; Dapper and EF handle this differently |
|
||||
| JSON column fidelity | JSONB columns round-trip without key reordering, whitespace changes, or precision loss |
|
||||
| Enum mapping | PostgreSQL custom enum values map to same CLR enum members before and after |
|
||||
| Default value generation | DB-level defaults (`now()`, `gen_random_uuid()`) remain authoritative (not application-generated) |
|
||||
| Connection timeout behavior | Command timeouts are still respected per-operation |
|
||||
|
||||
## 7. Known Dapper-to-EF Behavioral Differences
|
||||
|
||||
| Aspect | Dapper Behavior | EF Core Behavior | Mitigation |
|
||||
| --- | --- | --- | --- |
|
||||
| NULL string columns | Returns `null` | Returns `null` (same) | No action needed |
|
||||
| Empty result sets | Returns empty collection | Returns empty collection | No action needed |
|
||||
| DateTime precision | Depends on reader conversion | Npgsql provider handles | Verify in tests |
|
||||
| JSONB deserialization | Manual `JsonSerializer.Deserialize` | Manual (stored as string) | Keep domain-layer deserialization |
|
||||
| Bulk insert | `connection.ExecuteAsync` with multi-row SQL | `dbContext.AddRange()` + `SaveChangesAsync()` | Verify batch performance; use `ExecuteUpdateAsync` for large batches if needed |
|
||||
| UPSERT (ON CONFLICT) | Raw SQL `INSERT ... ON CONFLICT DO UPDATE` | `dbContext.Add` + catch `UniqueViolation` + update, or use raw SQL via `dbContext.Database.ExecuteSqlRawAsync` | Prefer catch-and-update for simple cases; use raw SQL for complex multi-column conflict clauses |
|
||||
|
||||
## 8. Module Classification for Cutover Approach
|
||||
|
||||
| Approach | Modules | Criteria |
|
||||
| --- | --- | --- |
|
||||
| **Direct replacement** | All Wave A (orders 2-16), Authority, Notify | Single migration, small repository surface, no custom history |
|
||||
| **Direct replacement with careful testing** | Graph, Signals, Unknowns, Excititor | 2-3 migrations, moderate repository complexity |
|
||||
| **Adapter-eligible** | Scheduler, EvidenceLocker, Policy, BinaryIndex, Concelier, Attestor, Orchestrator, Findings, Scanner, Platform | 4+ migrations, custom histories, mixed DAL, large repository surface |
|
||||
|
||||
## Revision History
|
||||
|
||||
| Date | Change | Author |
|
||||
| --- | --- | --- |
|
||||
| 2026-02-22 | Initial cutover strategy derived from TimelineIndexer and AirGap reference implementations. | Documentation Author |
|
||||
@@ -19,8 +19,14 @@ Module discovery for this runner is plug-in based:
|
||||
|
||||
- One migration module plug-in per web service implementing `src/Platform/__Libraries/StellaOps.Platform.Database/IMigrationModulePlugin.cs`.
|
||||
- Consolidated module registry auto-discovers plug-ins through `src/Platform/__Libraries/StellaOps.Platform.Database/MigrationModulePluginDiscovery.cs`.
|
||||
- Each service plug-in may flatten multiple migration sources (assembly + resource prefix) into one service-level runner module.
|
||||
- Current built-in plug-ins are in `src/Platform/__Libraries/StellaOps.Platform.Database/MigrationModulePlugins.cs`.
|
||||
- Optional external plug-in directories can be injected with `STELLAOPS_MIGRATION_PLUGIN_DIR` (path-list separated by OS path separator).
|
||||
- Consolidated execution behavior:
|
||||
- When `<schema>.schema_migrations` is empty, CLI/API runner paths execute one synthesized per-service migration (`100_consolidated_<service>.sql`) built from the plug-in source set.
|
||||
- After a successful non-dry-run consolidated execution, legacy per-file history rows are backfilled so future incremental upgrades remain compatible.
|
||||
- If consolidated history exists but legacy backfill is partially missing, CLI/API runner paths auto-backfill missing legacy rows before source-set execution.
|
||||
- Therefore, bootstrap execution is one migration per service module, but the resulting history table intentionally retains per-file entries for compatibility.
|
||||
|
||||
Canonical history table format is:
|
||||
|
||||
@@ -67,6 +73,7 @@ UI/API execution path (implemented):
|
||||
|
||||
- Module registry ownership is platform-level: `src/Platform/__Libraries/StellaOps.Platform.Database/MigrationModuleRegistry.cs`.
|
||||
- UI-triggered migration execution must call Platform WebService administrative APIs (no browser-direct database execution).
|
||||
- Platform service applies the same consolidated-empty-db behavior as CLI using `MigrationModuleConsolidation`.
|
||||
- Platform endpoint contract:
|
||||
- `GET /api/v1/admin/migrations/modules`
|
||||
- `GET /api/v1/admin/migrations/status?module=<name|all>`
|
||||
@@ -124,3 +131,17 @@ Exit criteria before EF phase opens:
|
||||
- One canonical operational entrypoint in runbooks and CI/CD automation.
|
||||
- Legacy history tables mapped and validated.
|
||||
- Migration replay determinism proven for clean install and upgrade scenarios.
|
||||
|
||||
Gate decision (2026-02-22 UTC): `GO`
|
||||
|
||||
- Gate evidence accepted:
|
||||
- `docs/db/rehearsals/20260222_mgc06_retry_seq_after_fix6/` (clean install + idempotent rerun evidence)
|
||||
- `docs/db/rehearsals/20260222_mgc06_rollback_retry_seq3/` (sequential rollback/retry rehearsal from partial state)
|
||||
- EF transition implementation sprint opened:
|
||||
- `docs/implplan/SPRINT_20260222_062_DOCS_efcore_v10_dapper_transition_phase_gate.md`
|
||||
|
||||
Governance boundary for EF phase:
|
||||
|
||||
- Migration registry ownership remains platform/infrastructure-owned in `src/Platform/__Libraries/StellaOps.Platform.Database/`.
|
||||
- Migration execution from UI must continue through Platform migration admin APIs; UI must not execute database operations directly.
|
||||
- Consolidated migration policy (categories, numbering, checksum/history compatibility) remains authoritative and cannot be relaxed by ORM refactors.
|
||||
|
||||
@@ -13,14 +13,14 @@ Scope: `src/**/Migrations/**/*.sql` and `src/**/migrations/**/*.sql`, excluding
|
||||
| Policy | Mixed Npgsql + Dapper (module-level) | `src/Policy/__Libraries/StellaOps.Policy.Persistence/Migrations` | 6 | Shared `MigrationRunner` resources | `CLI+PlatformAdminApi+SeedOnly`; `PolicyMigrator` is data conversion, not schema runner |
|
||||
| Notify | Npgsql repositories (no Dapper usage observed in module) | `src/Notify/__Libraries/StellaOps.Notify.Persistence/Migrations` | 2 | Shared `MigrationRunner` resources | `CLI+PlatformAdminApi+SeedOnly`; startup migration host not wired |
|
||||
| Excititor | Npgsql repositories (no Dapper usage observed in module) | `src/Excititor/__Libraries/StellaOps.Excititor.Persistence/Migrations` | 3 | Shared `MigrationRunner` resources | `CLI+PlatformAdminApi+SeedOnly`; startup migration host not wired |
|
||||
| Scanner | Dapper/Npgsql | `src/Scanner/__Libraries/StellaOps.Scanner.Storage/Postgres/Migrations`, `src/Scanner/__Libraries/StellaOps.Scanner.Triage/Migrations` | 35 | Shared `StartupMigrationHost` + `MigrationRunner` | `ScannerStartupHost + CLI + PlatformAdminApi` |
|
||||
| Scanner | Dapper/Npgsql | `src/Scanner/__Libraries/StellaOps.Scanner.Storage/Postgres/Migrations`, `src/Scanner/__Libraries/StellaOps.Scanner.Triage/Migrations` | 36 | Shared `StartupMigrationHost` + `MigrationRunner` (service plug-in source-set aggregation) | `ScannerStartupHost + CLI + PlatformAdminApi` |
|
||||
| AirGap | Npgsql repositories (no Dapper usage observed in module) | `src/AirGap/__Libraries/StellaOps.AirGap.Persistence/Migrations` | 1 | Shared `StartupMigrationHost` + `MigrationRunner` | `AirGapStartupHost + CLI + PlatformAdminApi` |
|
||||
| TimelineIndexer | Npgsql repositories (no Dapper usage observed in module) | `src/TimelineIndexer/StellaOps.TimelineIndexer/StellaOps.TimelineIndexer.Infrastructure/Db/Migrations` | 1 | Shared `MigrationRunner` via module wrapper | `TimelineIndexerMigrationHostedService + CLI + PlatformAdminApi` |
|
||||
| EvidenceLocker | Dapper/Npgsql | `src/EvidenceLocker/StellaOps.EvidenceLocker/StellaOps.EvidenceLocker.Infrastructure/Db/Migrations`, `src/EvidenceLocker/StellaOps.EvidenceLocker/Migrations` | 5 | Custom SQL runner with custom history table | `EvidenceLockerMigrationHostedService` (`evidence_schema_version`) |
|
||||
| ExportCenter | Npgsql repositories (no Dapper usage observed in module) | `src/ExportCenter/StellaOps.ExportCenter/StellaOps.ExportCenter.Infrastructure/Db/Migrations` | 1 | Custom SQL runner with custom history table | `ExportCenterMigrationHostedService` (`export_schema_version`) |
|
||||
| BinaryIndex | Dapper/Npgsql | `src/BinaryIndex/__Libraries/StellaOps.BinaryIndex.Persistence/Migrations`, `src/BinaryIndex/__Libraries/StellaOps.BinaryIndex.GoldenSet/Migrations` | 6 | Custom SQL runner with custom history table | Runner class exists; no runtime invocation found in non-test code |
|
||||
| BinaryIndex | EF Core v10 + compiled models (mixed: FunctionCorpusRepository and PostgresGoldenSetStore remain Dapper/Npgsql) | `src/BinaryIndex/__Libraries/StellaOps.BinaryIndex.Persistence/Migrations`, `src/BinaryIndex/__Libraries/StellaOps.BinaryIndex.GoldenSet/Migrations` | 6 | Custom SQL runner with custom history table; Platform migration registry plugin wired (BinaryIndexMigrationModulePlugin) | Runner class exists + CLI + PlatformAdminApi |
|
||||
| Plugin Registry | Npgsql repositories (no Dapper usage observed in module) | `src/Plugin/StellaOps.Plugin.Registry/Migrations` | 1 | Custom SQL runner with custom history table | Runner registered in DI; no runtime invocation found in non-test code |
|
||||
| Platform | Npgsql repositories (no Dapper usage observed in module) | `src/Platform/__Libraries/StellaOps.Platform.Database/Migrations/Release` | 56 | Shared `MigrationRunner` via module wrapper | `CLI+PlatformAdminApi`; no automatic runtime invocation found in non-test code |
|
||||
| Platform | Npgsql repositories (no Dapper usage observed in module) | `src/Platform/__Libraries/StellaOps.Platform.Database/Migrations/Release` | 57 | Shared `MigrationRunner` via module wrapper | `CLI+PlatformAdminApi`; no automatic runtime invocation found in non-test code |
|
||||
| Graph | Npgsql repositories (no Dapper usage observed in module) | `src/Graph/__Libraries/StellaOps.Graph.Indexer.Persistence/Migrations`, `src/Graph/__Libraries/StellaOps.Graph.Core/migrations` | 2 | Embedded SQL files only | No runtime invocation found in non-test code |
|
||||
| IssuerDirectory | Npgsql repositories (no Dapper usage observed in module) | `src/IssuerDirectory/__Libraries/StellaOps.IssuerDirectory.Persistence/Migrations` | 1 | Embedded SQL files only | No runtime invocation found in non-test code |
|
||||
| Findings Ledger | Npgsql repositories (no Dapper usage observed in module) | `src/Findings/StellaOps.Findings.Ledger/migrations` | 12 | Embedded SQL files only | No runtime invocation found in non-test code |
|
||||
@@ -65,6 +65,7 @@ Scope: `src/**/Migrations/**/*.sql` and `src/**/migrations/**/*.sql`, excluding
|
||||
- Platform migration registry: `src/Platform/__Libraries/StellaOps.Platform.Database/MigrationModuleRegistry.cs`
|
||||
- `ScannerStartupHost + CLI + PlatformAdminApi`:
|
||||
- Startup host: `src/Scanner/__Libraries/StellaOps.Scanner.Storage/Extensions/ServiceCollectionExtensions.cs`
|
||||
- Service plug-in source-set declaration: `src/Platform/__Libraries/StellaOps.Platform.Database/MigrationModulePlugins.cs` (`ScannerMigrationModulePlugin`)
|
||||
- Plug-in discovery: `src/Platform/__Libraries/StellaOps.Platform.Database/MigrationModulePluginDiscovery.cs`
|
||||
- Platform API: `src/Platform/StellaOps.Platform.WebService/Endpoints/MigrationAdminEndpoints.cs`
|
||||
- Platform migration registry: `src/Platform/__Libraries/StellaOps.Platform.Database/MigrationModuleRegistry.cs`
|
||||
@@ -95,6 +96,7 @@ Scope: `src/**/Migrations/**/*.sql` and `src/**/migrations/**/*.sql`, excluding
|
||||
- Primary consolidation objective for this sprint:
|
||||
- Reduce to one canonical runner contract and one canonical runtime entrypoint policy across startup, CLI, and compose/upgrade workflows.
|
||||
- Execute UI-triggered migration flows through Platform WebService administrative APIs that consume the platform-owned migration registry.
|
||||
- Execute one synthesized per-plugin consolidated migration for empty-history installs, with legacy history backfill preserving incremental upgrade compatibility.
|
||||
|
||||
## Target Wave Assignment (Consolidation)
|
||||
|
||||
|
||||
@@ -20,6 +20,10 @@ Current-state realities that must be accounted for in operations:
|
||||
- Multiple migration mechanisms are active (shared `MigrationRunner`, `StartupMigrationHost` wrappers, custom runners with custom history tables, compose bootstrap init SQL, and unwired migration folders).
|
||||
- CLI migration coverage is currently limited to the modules registered in `src/Platform/__Libraries/StellaOps.Platform.Database/MigrationModuleRegistry.cs`.
|
||||
- Registry module population is plug-in based (`IMigrationModulePlugin`) with one migration plug-in per web service.
|
||||
- Service plug-ins can flatten multiple migration sources into one service-level module (for example Scanner storage + triage) while preserving one runner entrypoint per service.
|
||||
- Consolidated runner behavior for CLI/API: when a module has no applied history rows, one synthesized `100_consolidated_<service>.sql` migration is executed from the service source set, then legacy per-file history rows are backfilled for upgrade compatibility.
|
||||
- Consolidated runner behavior is self-healing for partial backfill states: if consolidated history exists and only some legacy rows are present, missing legacy rows are backfilled before per-source execution.
|
||||
- This means one-per-service execution for first bootstrap, not a permanent single-row history model.
|
||||
- Platform migration admin endpoints (`/api/v1/admin/migrations/*`) use the same platform-owned registry for UI/backend orchestration.
|
||||
- Several services contain migration SQL but have no verified runtime invocation path in non-test code.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user