up
This commit is contained in:
StellaOps Bot
404
docs/db/tasks/PHASE_0_FOUNDATIONS.md
Normal file
404
docs/db/tasks/PHASE_0_FOUNDATIONS.md
Normal file
@@ -0,0 +1,404 @@
|
||||
# Phase 0: Foundations
|
||||
|
||||
**Sprint:** 1
|
||||
**Duration:** 1 sprint
|
||||
**Status:** TODO
|
||||
**Dependencies:** None
|
||||
|
||||
---
|
||||
|
||||
## Objectives
|
||||
|
||||
1. Provision PostgreSQL cluster for staging and production
|
||||
2. Create shared infrastructure library (`StellaOps.Infrastructure.Postgres`)
|
||||
3. Set up CI/CD pipeline for PostgreSQL migrations
|
||||
4. Establish Testcontainers-based integration testing
|
||||
|
||||
---
|
||||
|
||||
## Deliverables
|
||||
|
||||
| Deliverable | Acceptance Criteria |
|
||||
|-------------|---------------------|
|
||||
| PostgreSQL cluster | Running in staging with proper configuration |
|
||||
| Shared library | DataSource, migrations, extensions implemented |
|
||||
| CI pipeline | PostgreSQL tests running on every PR |
|
||||
| Documentation | SPECIFICATION.md, RULES.md reviewed and approved |
|
||||
|
||||
---
|
||||
|
||||
## Task Breakdown
|
||||
|
||||
### T0.1: PostgreSQL Cluster Provisioning
|
||||
|
||||
**Status:** TODO
|
||||
**Assignee:** TBD
|
||||
**Estimate:** 2 days
|
||||
|
||||
**Description:**
|
||||
Provision PostgreSQL 16+ cluster with appropriate configuration for StellaOps workload.
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T0.1.1: Select PostgreSQL hosting (managed vs self-hosted)
|
||||
- [ ] T0.1.2: Create staging cluster with single primary
|
||||
- [ ] T0.1.3: Configure connection pooling (PgBouncer or built-in)
|
||||
- [ ] T0.1.4: Set up backup and restore procedures
|
||||
- [ ] T0.1.5: Configure monitoring (pg_stat_statements, Prometheus exporter)
|
||||
- [ ] T0.1.6: Document connection strings and access credentials
|
||||
- [ ] T0.1.7: Configure SSL/TLS for connections
|
||||
|
||||
**Configuration Requirements:**
|
||||
```
|
||||
PostgreSQL Version: 16+
|
||||
Max Connections: 100 (via pooler: 500)
|
||||
Shared Buffers: 25% of RAM
|
||||
Work Mem: 64MB
|
||||
Maintenance Work Mem: 512MB
|
||||
WAL Level: replica
|
||||
Max WAL Size: 2GB
|
||||
```
|
||||
|
||||
**Verification:**
|
||||
- [ ] Can connect from development machines
|
||||
- [ ] Can connect from CI/CD runners
|
||||
- [ ] Monitoring dashboard shows metrics
|
||||
- [ ] Backup tested and verified
|
||||
|
||||
---
|
||||
|
||||
### T0.2: Create StellaOps.Infrastructure.Postgres Library
|
||||
|
||||
**Status:** TODO
|
||||
**Assignee:** TBD
|
||||
**Estimate:** 3 days
|
||||
|
||||
**Description:**
|
||||
Create shared library with reusable PostgreSQL infrastructure components.
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T0.2.1: Create project `src/Shared/StellaOps.Infrastructure.Postgres/`
|
||||
- [ ] T0.2.2: Add Npgsql NuGet package reference
|
||||
- [ ] T0.2.3: Implement `DataSourceBase` abstract class
|
||||
- [ ] T0.2.4: Implement `IPostgresMigration` interface
|
||||
- [ ] T0.2.5: Implement `PostgresMigrationRunner` class
|
||||
- [ ] T0.2.6: Implement `NpgsqlExtensions` helper methods
|
||||
- [ ] T0.2.7: Implement `ServiceCollectionExtensions` for DI
|
||||
- [ ] T0.2.8: Add XML documentation to all public APIs
|
||||
- [ ] T0.2.9: Add unit tests for migration runner
|
||||
|
||||
**Files to Create:**
|
||||
```
|
||||
src/Shared/StellaOps.Infrastructure.Postgres/
|
||||
├── StellaOps.Infrastructure.Postgres.csproj
|
||||
├── DataSourceBase.cs
|
||||
├── PostgresOptions.cs
|
||||
├── Migrations/
|
||||
│ ├── IPostgresMigration.cs
|
||||
│ └── PostgresMigrationRunner.cs
|
||||
├── Extensions/
|
||||
│ ├── NpgsqlExtensions.cs
|
||||
│ └── NpgsqlCommandExtensions.cs
|
||||
└── ServiceCollectionExtensions.cs
|
||||
```
|
||||
|
||||
**DataSourceBase Implementation:**
|
||||
```csharp
|
||||
public abstract class DataSourceBase : IAsyncDisposable
|
||||
{
|
||||
protected readonly NpgsqlDataSource DataSource;
|
||||
protected readonly PostgresOptions Options;
|
||||
|
||||
protected DataSourceBase(IOptions<PostgresOptions> options)
|
||||
{
|
||||
Options = options.Value;
|
||||
var builder = new NpgsqlDataSourceBuilder(Options.ConnectionString);
|
||||
ConfigureDataSource(builder);
|
||||
DataSource = builder.Build();
|
||||
}
|
||||
|
||||
protected virtual void ConfigureDataSource(NpgsqlDataSourceBuilder builder)
|
||||
{
|
||||
// Override in derived classes for module-specific config
|
||||
}
|
||||
|
||||
public async Task<NpgsqlConnection> OpenConnectionAsync(
|
||||
string tenantId,
|
||||
CancellationToken cancellationToken = default)
|
||||
{
|
||||
var connection = await DataSource.OpenConnectionAsync(cancellationToken);
|
||||
await ConfigureSessionAsync(connection, tenantId, cancellationToken);
|
||||
return connection;
|
||||
}
|
||||
|
||||
protected virtual async Task ConfigureSessionAsync(
|
||||
NpgsqlConnection connection,
|
||||
string tenantId,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
await using var cmd = connection.CreateCommand();
|
||||
cmd.CommandText = $"""
|
||||
SET app.tenant_id = '{tenantId}';
|
||||
SET timezone = 'UTC';
|
||||
SET statement_timeout = '{Options.CommandTimeoutSeconds}s';
|
||||
""";
|
||||
await cmd.ExecuteNonQueryAsync(cancellationToken);
|
||||
}
|
||||
|
||||
public async ValueTask DisposeAsync()
|
||||
{
|
||||
await DataSource.DisposeAsync();
|
||||
GC.SuppressFinalize(this);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Verification:**
|
||||
- [ ] Project builds without errors
|
||||
- [ ] Unit tests pass
|
||||
- [ ] Can be referenced from module projects
|
||||
|
||||
---
|
||||
|
||||
### T0.3: Migration Framework Implementation
|
||||
|
||||
**Status:** TODO
|
||||
**Assignee:** TBD
|
||||
**Estimate:** 2 days
|
||||
|
||||
**Description:**
|
||||
Implement idempotent migration framework for schema management.
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T0.3.1: Define `IPostgresMigration` interface
|
||||
- [ ] T0.3.2: Implement `PostgresMigrationRunner` with transaction support
|
||||
- [ ] T0.3.3: Implement migration tracking table (`_migrations`)
|
||||
- [ ] T0.3.4: Add `IHostedService` for automatic migration on startup
|
||||
- [ ] T0.3.5: Add CLI command for manual migration execution
|
||||
- [ ] T0.3.6: Add migration rollback support (optional)
|
||||
|
||||
**Migration Interface:**
|
||||
```csharp
|
||||
public interface IPostgresMigration
|
||||
{
|
||||
/// <summary>
|
||||
/// Unique migration identifier (e.g., "V001_CreateAuthoritySchema")
|
||||
/// </summary>
|
||||
string Id { get; }
|
||||
|
||||
/// <summary>
|
||||
/// Human-readable description
|
||||
/// </summary>
|
||||
string Description { get; }
|
||||
|
||||
/// <summary>
|
||||
/// Apply the migration
|
||||
/// </summary>
|
||||
Task UpAsync(NpgsqlConnection connection, CancellationToken cancellationToken);
|
||||
|
||||
/// <summary>
|
||||
/// Rollback the migration (optional)
|
||||
/// </summary>
|
||||
Task DownAsync(NpgsqlConnection connection, CancellationToken cancellationToken);
|
||||
}
|
||||
```
|
||||
|
||||
**Verification:**
|
||||
- [ ] Migrations run idempotently (can run multiple times)
|
||||
- [ ] Migration state tracked correctly
|
||||
- [ ] Failed migrations roll back cleanly
|
||||
|
||||
---
|
||||
|
||||
### T0.4: CI/CD Pipeline Configuration
|
||||
|
||||
**Status:** TODO
|
||||
**Assignee:** TBD
|
||||
**Estimate:** 2 days
|
||||
|
||||
**Description:**
|
||||
Add PostgreSQL integration testing to CI/CD pipeline.
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T0.4.1: Add Testcontainers.PostgreSql NuGet package to test projects
|
||||
- [ ] T0.4.2: Create `PostgresTestFixture` base class
|
||||
- [ ] T0.4.3: Update CI workflow to support PostgreSQL containers
|
||||
- [ ] T0.4.4: Add parallel test execution configuration
|
||||
- [ ] T0.4.5: Add test coverage reporting for PostgreSQL code
|
||||
|
||||
**PostgresTestFixture:**
|
||||
```csharp
|
||||
public sealed class PostgresTestFixture : IAsyncLifetime
|
||||
{
|
||||
private readonly PostgreSqlContainer _container;
|
||||
private NpgsqlDataSource? _dataSource;
|
||||
|
||||
public PostgresTestFixture()
|
||||
{
|
||||
_container = new PostgreSqlBuilder()
|
||||
.WithImage("postgres:16-alpine")
|
||||
.WithDatabase("stellaops_test")
|
||||
.WithUsername("test")
|
||||
.WithPassword("test")
|
||||
.WithWaitStrategy(Wait.ForUnixContainer()
|
||||
.UntilPortIsAvailable(5432))
|
||||
.Build();
|
||||
}
|
||||
|
||||
public string ConnectionString => _container.GetConnectionString();
|
||||
public NpgsqlDataSource DataSource => _dataSource
|
||||
?? throw new InvalidOperationException("Fixture not initialized");
|
||||
|
||||
public async Task InitializeAsync()
|
||||
{
|
||||
await _container.StartAsync();
|
||||
_dataSource = NpgsqlDataSource.Create(ConnectionString);
|
||||
}
|
||||
|
||||
public async Task DisposeAsync()
|
||||
{
|
||||
if (_dataSource is not null)
|
||||
await _dataSource.DisposeAsync();
|
||||
await _container.DisposeAsync();
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**CI Workflow Update:**
|
||||
```yaml
|
||||
# .gitea/workflows/build-test-deploy.yml
|
||||
- name: Run PostgreSQL Integration Tests
|
||||
run: |
|
||||
dotnet test src/StellaOps.sln \
|
||||
--filter "Category=PostgresIntegration" \
|
||||
--logger "trx;LogFileName=postgres-test-results.trx"
|
||||
env:
|
||||
TESTCONTAINERS_RYUK_DISABLED: true
|
||||
```
|
||||
|
||||
**Verification:**
|
||||
- [ ] CI pipeline runs PostgreSQL tests
|
||||
- [ ] Tests can run in parallel without conflicts
|
||||
- [ ] Test results reported correctly
|
||||
|
||||
---
|
||||
|
||||
### T0.5: Persistence Configuration
|
||||
|
||||
**Status:** TODO
|
||||
**Assignee:** TBD
|
||||
**Estimate:** 1 day
|
||||
|
||||
**Description:**
|
||||
Add persistence backend configuration to all services.
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T0.5.1: Define `PersistenceOptions` class
|
||||
- [ ] T0.5.2: Add configuration section to `appsettings.json`
|
||||
- [ ] T0.5.3: Update service registration to read persistence config
|
||||
- [ ] T0.5.4: Add configuration validation on startup
|
||||
|
||||
**PersistenceOptions:**
|
||||
```csharp
|
||||
public sealed class PersistenceOptions
|
||||
{
|
||||
public const string SectionName = "Persistence";
|
||||
|
||||
public string Authority { get; set; } = "Mongo";
|
||||
public string Scheduler { get; set; } = "Mongo";
|
||||
public string Concelier { get; set; } = "Mongo";
|
||||
public string Excititor { get; set; } = "Mongo";
|
||||
public string Notify { get; set; } = "Mongo";
|
||||
public string Policy { get; set; } = "Mongo";
|
||||
}
|
||||
```
|
||||
|
||||
**Configuration Template:**
|
||||
```json
|
||||
{
|
||||
"Persistence": {
|
||||
"Authority": "Mongo",
|
||||
"Scheduler": "Mongo",
|
||||
"Concelier": "Mongo",
|
||||
"Excititor": "Mongo",
|
||||
"Notify": "Mongo",
|
||||
"Policy": "Mongo"
|
||||
},
|
||||
"Postgres": {
|
||||
"ConnectionString": "Host=localhost;Database=stellaops;Username=stellaops;Password=secret",
|
||||
"CommandTimeoutSeconds": 30,
|
||||
"ConnectionTimeoutSeconds": 15
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Verification:**
|
||||
- [ ] Configuration loads correctly
|
||||
- [ ] Invalid configuration throws on startup
|
||||
- [ ] Environment variables can override settings
|
||||
|
||||
---
|
||||
|
||||
### T0.6: Documentation Review
|
||||
|
||||
**Status:** TODO
|
||||
**Assignee:** TBD
|
||||
**Estimate:** 1 day
|
||||
|
||||
**Description:**
|
||||
Review and finalize database documentation.
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T0.6.1: Review SPECIFICATION.md for completeness
|
||||
- [ ] T0.6.2: Review RULES.md for clarity
|
||||
- [ ] T0.6.3: Review VERIFICATION.md for test coverage
|
||||
- [ ] T0.6.4: Get Architecture Team sign-off
|
||||
- [ ] T0.6.5: Publish to team wiki/docs site
|
||||
|
||||
**Verification:**
|
||||
- [ ] All documents reviewed by 2+ team members
|
||||
- [ ] No outstanding questions or TODOs
|
||||
- [ ] Architecture Team approval received
|
||||
|
||||
---
|
||||
|
||||
## Exit Criteria
|
||||
|
||||
- [ ] PostgreSQL cluster running and accessible
|
||||
- [ ] `StellaOps.Infrastructure.Postgres` library implemented and tested
|
||||
- [ ] CI pipeline running PostgreSQL integration tests
|
||||
- [ ] Persistence configuration framework in place
|
||||
- [ ] Documentation reviewed and approved
|
||||
|
||||
---
|
||||
|
||||
## Risks & Mitigations
|
||||
|
||||
| Risk | Likelihood | Impact | Mitigation |
|
||||
|------|------------|--------|------------|
|
||||
| PostgreSQL provisioning delays | Medium | High | Start early, have backup plan |
|
||||
| Testcontainers compatibility issues | Low | Medium | Test on CI runners early |
|
||||
| Configuration complexity | Low | Low | Use existing patterns from Orchestrator |
|
||||
|
||||
---
|
||||
|
||||
## Dependencies on Later Phases
|
||||
|
||||
Phase 0 must complete before any module conversion (Phases 1-6) can begin. The following are required:
|
||||
|
||||
1. PostgreSQL cluster operational
|
||||
2. Shared library published
|
||||
3. CI pipeline validated
|
||||
4. Configuration framework deployed
|
||||
|
||||
---
|
||||
|
||||
## Notes
|
||||
|
||||
- Use Orchestrator module as reference for all patterns
|
||||
- Prioritize getting CI pipeline working early
|
||||
- Document all configuration decisions
|
||||
|
||||
---
|
||||
|
||||
*Phase Version: 1.0.0*
|
||||
*Last Updated: 2025-11-28*
|
||||
495
docs/db/tasks/PHASE_1_AUTHORITY.md
Normal file
495
docs/db/tasks/PHASE_1_AUTHORITY.md
Normal file
@@ -0,0 +1,495 @@
|
||||
# Phase 1: Authority Module Conversion
|
||||
|
||||
**Sprint:** 2
|
||||
**Duration:** 1 sprint
|
||||
**Status:** TODO
|
||||
**Dependencies:** Phase 0 (Foundations)
|
||||
|
||||
---
|
||||
|
||||
## Objectives
|
||||
|
||||
1. Create `StellaOps.Authority.Storage.Postgres` project
|
||||
2. Implement full Authority schema in PostgreSQL
|
||||
3. Implement all repository interfaces
|
||||
4. Enable dual-write mode for validation
|
||||
5. Switch Authority to PostgreSQL-only after verification
|
||||
|
||||
---
|
||||
|
||||
## Deliverables
|
||||
|
||||
| Deliverable | Acceptance Criteria |
|
||||
|-------------|---------------------|
|
||||
| Authority schema | All tables created with indexes |
|
||||
| Repository implementations | All 9 interfaces implemented |
|
||||
| Dual-write wrapper | Optional, for safe rollout |
|
||||
| Integration tests | 100% coverage of CRUD operations |
|
||||
| Verification report | MongoDB vs PostgreSQL comparison passed |
|
||||
|
||||
---
|
||||
|
||||
## Schema Reference
|
||||
|
||||
See [SPECIFICATION.md](../SPECIFICATION.md) Section 5.1 for complete Authority schema.
|
||||
|
||||
**Tables:**
|
||||
- `authority.tenants`
|
||||
- `authority.users`
|
||||
- `authority.roles`
|
||||
- `authority.user_roles`
|
||||
- `authority.service_accounts`
|
||||
- `authority.clients`
|
||||
- `authority.scopes`
|
||||
- `authority.tokens`
|
||||
- `authority.revocations`
|
||||
- `authority.login_attempts`
|
||||
- `authority.licenses`
|
||||
- `authority.license_usage`
|
||||
|
||||
---
|
||||
|
||||
## Task Breakdown
|
||||
|
||||
### T1.1: Create Authority.Storage.Postgres Project
|
||||
|
||||
**Status:** TODO
|
||||
**Assignee:** TBD
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
**Description:**
|
||||
Create the PostgreSQL storage project for Authority module.
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T1.1.1: Create project `src/Authority/__Libraries/StellaOps.Authority.Storage.Postgres/`
|
||||
- [ ] T1.1.2: Add reference to `StellaOps.Infrastructure.Postgres`
|
||||
- [ ] T1.1.3: Add reference to `StellaOps.Authority.Core`
|
||||
- [ ] T1.1.4: Create `AuthorityDataSource` class
|
||||
- [ ] T1.1.5: Create `AuthorityPostgresOptions` class
|
||||
- [ ] T1.1.6: Create `ServiceCollectionExtensions.cs`
|
||||
|
||||
**Project Structure:**
|
||||
```
|
||||
src/Authority/__Libraries/StellaOps.Authority.Storage.Postgres/
|
||||
├── StellaOps.Authority.Storage.Postgres.csproj
|
||||
├── AuthorityDataSource.cs
|
||||
├── AuthorityPostgresOptions.cs
|
||||
├── Repositories/
|
||||
│ ├── PostgresUserRepository.cs
|
||||
│ ├── PostgresRoleRepository.cs
|
||||
│ ├── PostgresServiceAccountRepository.cs
|
||||
│ ├── PostgresClientRepository.cs
|
||||
│ ├── PostgresScopeRepository.cs
|
||||
│ ├── PostgresTokenRepository.cs
|
||||
│ ├── PostgresRevocationRepository.cs
|
||||
│ ├── PostgresLoginAttemptRepository.cs
|
||||
│ └── PostgresLicenseRepository.cs
|
||||
├── Migrations/
|
||||
│ └── V001_CreateAuthoritySchema.cs
|
||||
└── ServiceCollectionExtensions.cs
|
||||
```
|
||||
|
||||
**Verification:**
|
||||
- [ ] Project builds without errors
|
||||
- [ ] Can be referenced from Authority.WebService
|
||||
|
||||
---
|
||||
|
||||
### T1.2: Implement Schema Migrations
|
||||
|
||||
**Status:** TODO
|
||||
**Assignee:** TBD
|
||||
**Estimate:** 1 day
|
||||
|
||||
**Description:**
|
||||
Create PostgreSQL schema migration for Authority tables.
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T1.2.1: Create `V001_CreateAuthoritySchema` migration
|
||||
- [ ] T1.2.2: Include all tables from SPECIFICATION.md
|
||||
- [ ] T1.2.3: Include all indexes
|
||||
- [ ] T1.2.4: Add seed data for system roles/permissions
|
||||
- [ ] T1.2.5: Test migration idempotency
|
||||
|
||||
**Migration Implementation:**
|
||||
```csharp
|
||||
public sealed class V001_CreateAuthoritySchema : IPostgresMigration
|
||||
{
|
||||
public string Id => "V001_CreateAuthoritySchema";
|
||||
public string Description => "Create Authority schema with all tables and indexes";
|
||||
|
||||
public async Task UpAsync(NpgsqlConnection connection, CancellationToken ct)
|
||||
{
|
||||
await using var cmd = connection.CreateCommand();
|
||||
cmd.CommandText = AuthoritySchemaSql;
|
||||
await cmd.ExecuteNonQueryAsync(ct);
|
||||
}
|
||||
|
||||
public Task DownAsync(NpgsqlConnection connection, CancellationToken ct)
|
||||
=> throw new NotSupportedException("Rollback not supported for schema creation");
|
||||
|
||||
private const string AuthoritySchemaSql = """
|
||||
CREATE SCHEMA IF NOT EXISTS authority;
|
||||
|
||||
CREATE TABLE IF NOT EXISTS authority.tenants (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
code TEXT NOT NULL UNIQUE,
|
||||
display_name TEXT NOT NULL,
|
||||
status TEXT NOT NULL DEFAULT 'active'
|
||||
CHECK (status IN ('active', 'suspended', 'trial', 'terminated')),
|
||||
settings JSONB DEFAULT '{}',
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||
);
|
||||
|
||||
-- ... rest of schema from SPECIFICATION.md
|
||||
""";
|
||||
}
|
||||
```
|
||||
|
||||
**Verification:**
|
||||
- [ ] Migration creates all tables
|
||||
- [ ] Migration is idempotent
|
||||
- [ ] Indexes created correctly
|
||||
|
||||
---
|
||||
|
||||
### T1.3: Implement User Repository
|
||||
|
||||
**Status:** TODO
|
||||
**Assignee:** TBD
|
||||
**Estimate:** 1 day
|
||||
|
||||
**Description:**
|
||||
Implement `IUserRepository` for PostgreSQL.
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T1.3.1: Implement `GetByIdAsync`
|
||||
- [ ] T1.3.2: Implement `GetByUsernameAsync`
|
||||
- [ ] T1.3.3: Implement `GetBySubjectIdAsync`
|
||||
- [ ] T1.3.4: Implement `ListAsync` with pagination
|
||||
- [ ] T1.3.5: Implement `CreateAsync`
|
||||
- [ ] T1.3.6: Implement `UpdateAsync`
|
||||
- [ ] T1.3.7: Implement `DeleteAsync`
|
||||
- [ ] T1.3.8: Implement `GetRolesAsync`
|
||||
- [ ] T1.3.9: Implement `AssignRoleAsync`
|
||||
- [ ] T1.3.10: Implement `RevokeRoleAsync`
|
||||
- [ ] T1.3.11: Write integration tests
|
||||
|
||||
**Interface Reference:**
|
||||
```csharp
|
||||
public interface IUserRepository
|
||||
{
|
||||
Task<User?> GetByIdAsync(string tenantId, Guid userId, CancellationToken ct);
|
||||
Task<User?> GetByUsernameAsync(string tenantId, string username, CancellationToken ct);
|
||||
Task<User?> GetBySubjectIdAsync(Guid subjectId, CancellationToken ct);
|
||||
Task<PagedResult<User>> ListAsync(string tenantId, UserQuery query, CancellationToken ct);
|
||||
Task<User> CreateAsync(User user, CancellationToken ct);
|
||||
Task<User> UpdateAsync(User user, CancellationToken ct);
|
||||
Task<bool> DeleteAsync(string tenantId, Guid userId, CancellationToken ct);
|
||||
Task<IReadOnlyList<Role>> GetRolesAsync(string tenantId, Guid userId, CancellationToken ct);
|
||||
Task AssignRoleAsync(string tenantId, Guid userId, Guid roleId, CancellationToken ct);
|
||||
Task RevokeRoleAsync(string tenantId, Guid userId, Guid roleId, CancellationToken ct);
|
||||
}
|
||||
```
|
||||
|
||||
**Verification:**
|
||||
- [ ] All methods implemented
|
||||
- [ ] Integration tests pass
|
||||
- [ ] Tenant isolation verified
|
||||
|
||||
---
|
||||
|
||||
### T1.4: Implement Service Account Repository
|
||||
|
||||
**Status:** TODO
|
||||
**Assignee:** TBD
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
**Description:**
|
||||
Implement `IServiceAccountRepository` for PostgreSQL.
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T1.4.1: Implement `GetByIdAsync`
|
||||
- [ ] T1.4.2: Implement `GetByAccountIdAsync`
|
||||
- [ ] T1.4.3: Implement `ListAsync`
|
||||
- [ ] T1.4.4: Implement `CreateAsync`
|
||||
- [ ] T1.4.5: Implement `UpdateAsync`
|
||||
- [ ] T1.4.6: Implement `DeleteAsync`
|
||||
- [ ] T1.4.7: Write integration tests
|
||||
|
||||
**Verification:**
|
||||
- [ ] All methods implemented
|
||||
- [ ] Integration tests pass
|
||||
|
||||
---
|
||||
|
||||
### T1.5: Implement Client Repository
|
||||
|
||||
**Status:** TODO
|
||||
**Assignee:** TBD
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
**Description:**
|
||||
Implement `IClientRepository` for PostgreSQL (OpenIddict compatible).
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T1.5.1: Implement `GetByIdAsync`
|
||||
- [ ] T1.5.2: Implement `GetByClientIdAsync`
|
||||
- [ ] T1.5.3: Implement `ListAsync`
|
||||
- [ ] T1.5.4: Implement `CreateAsync`
|
||||
- [ ] T1.5.5: Implement `UpdateAsync`
|
||||
- [ ] T1.5.6: Implement `DeleteAsync`
|
||||
- [ ] T1.5.7: Write integration tests
|
||||
|
||||
**Verification:**
|
||||
- [ ] All methods implemented
|
||||
- [ ] Integration tests pass
|
||||
|
||||
---
|
||||
|
||||
### T1.6: Implement Token Repository
|
||||
|
||||
**Status:** TODO
|
||||
**Assignee:** TBD
|
||||
**Estimate:** 1 day
|
||||
|
||||
**Description:**
|
||||
Implement `ITokenRepository` for PostgreSQL.
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T1.6.1: Implement `GetByIdAsync`
|
||||
- [ ] T1.6.2: Implement `GetByHashAsync`
|
||||
- [ ] T1.6.3: Implement `CreateAsync`
|
||||
- [ ] T1.6.4: Implement `RevokeAsync`
|
||||
- [ ] T1.6.5: Implement `PruneExpiredAsync`
|
||||
- [ ] T1.6.6: Implement `GetActiveTokensAsync`
|
||||
- [ ] T1.6.7: Write integration tests
|
||||
|
||||
**Verification:**
|
||||
- [ ] All methods implemented
|
||||
- [ ] Token lookup by hash is fast
|
||||
- [ ] Expired token pruning works
|
||||
|
||||
---
|
||||
|
||||
### T1.7: Implement Remaining Repositories
|
||||
|
||||
**Status:** TODO
|
||||
**Assignee:** TBD
|
||||
**Estimate:** 1.5 days
|
||||
|
||||
**Description:**
|
||||
Implement remaining repository interfaces.
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T1.7.1: Implement `IRoleRepository`
|
||||
- [ ] T1.7.2: Implement `IScopeRepository`
|
||||
- [ ] T1.7.3: Implement `IRevocationRepository`
|
||||
- [ ] T1.7.4: Implement `ILoginAttemptRepository`
|
||||
- [ ] T1.7.5: Implement `ILicenseRepository`
|
||||
- [ ] T1.7.6: Write integration tests for all
|
||||
|
||||
**Verification:**
|
||||
- [ ] All repositories implemented
|
||||
- [ ] All integration tests pass
|
||||
|
||||
---
|
||||
|
||||
### T1.8: Add Configuration Switch
|
||||
|
||||
**Status:** TODO
|
||||
**Assignee:** TBD
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
**Description:**
|
||||
Add configuration-based backend selection for Authority.
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T1.8.1: Update `ServiceCollectionExtensions` in Authority.WebService
|
||||
- [ ] T1.8.2: Add conditional registration based on `Persistence:Authority`
|
||||
- [ ] T1.8.3: Test switching between Mongo and Postgres
|
||||
- [ ] T1.8.4: Document configuration options
|
||||
|
||||
**Implementation:**
|
||||
```csharp
|
||||
public static IServiceCollection AddAuthorityStorage(
|
||||
this IServiceCollection services,
|
||||
IConfiguration configuration)
|
||||
{
|
||||
var backend = configuration.GetValue<string>("Persistence:Authority") ?? "Mongo";
|
||||
|
||||
return backend.ToLowerInvariant() switch
|
||||
{
|
||||
"postgres" => services.AddAuthorityPostgresStorage(configuration),
|
||||
"mongo" => services.AddAuthorityMongoStorage(configuration),
|
||||
_ => throw new ArgumentException($"Unknown Authority backend: {backend}")
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
**Verification:**
|
||||
- [ ] Can switch between backends via configuration
|
||||
- [ ] Invalid configuration throws clear error
|
||||
|
||||
---
|
||||
|
||||
### T1.9: Implement Dual-Write Wrapper (Optional)
|
||||
|
||||
**Status:** TODO
|
||||
**Assignee:** TBD
|
||||
**Estimate:** 1 day
|
||||
|
||||
**Description:**
|
||||
Implement dual-write repository wrapper for safe migration.
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T1.9.1: Create `DualWriteUserRepository`
|
||||
- [ ] T1.9.2: Implement write-to-both logic
|
||||
- [ ] T1.9.3: Implement read-from-primary-with-fallback logic
|
||||
- [ ] T1.9.4: Add metrics for dual-write operations
|
||||
- [ ] T1.9.5: Add logging for inconsistencies
|
||||
- [ ] T1.9.6: Create similar wrappers for other critical repositories
|
||||
|
||||
**Configuration Options:**
|
||||
```csharp
|
||||
public sealed class DualWriteOptions
|
||||
{
|
||||
public string PrimaryBackend { get; set; } = "Postgres";
|
||||
public bool WriteToBoth { get; set; } = true;
|
||||
public bool FallbackToSecondary { get; set; } = true;
|
||||
public bool ConvertOnRead { get; set; } = true;
|
||||
}
|
||||
```
|
||||
|
||||
**Verification:**
|
||||
- [ ] Writes go to both backends
|
||||
- [ ] Reads work with fallback
|
||||
- [ ] Inconsistencies are logged
|
||||
|
||||
---
|
||||
|
||||
### T1.10: Run Verification Tests
|
||||
|
||||
**Status:** TODO
|
||||
**Assignee:** TBD
|
||||
**Estimate:** 1 day
|
||||
|
||||
**Description:**
|
||||
Verify PostgreSQL implementation matches MongoDB behavior.
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T1.10.1: Run comparison tests for User repository
|
||||
- [ ] T1.10.2: Run comparison tests for Token repository
|
||||
- [ ] T1.10.3: Verify token issuance/verification flow
|
||||
- [ ] T1.10.4: Verify login flow
|
||||
- [ ] T1.10.5: Document any differences found
|
||||
- [ ] T1.10.6: Generate verification report
|
||||
|
||||
**Verification Tests:**
|
||||
```csharp
|
||||
[Fact]
|
||||
public async Task Users_Should_Match_Between_Mongo_And_Postgres()
|
||||
{
|
||||
var tenantIds = await GetSampleTenantIds(10);
|
||||
|
||||
foreach (var tenantId in tenantIds)
|
||||
{
|
||||
var mongoUsers = await _mongoRepo.ListAsync(tenantId, new UserQuery());
|
||||
var postgresUsers = await _postgresRepo.ListAsync(tenantId, new UserQuery());
|
||||
|
||||
postgresUsers.Items.Should().BeEquivalentTo(mongoUsers.Items,
|
||||
options => options.Excluding(u => u.Id));
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Verification:**
|
||||
- [ ] All comparison tests pass
|
||||
- [ ] No data discrepancies found
|
||||
- [ ] Verification report approved
|
||||
|
||||
---
|
||||
|
||||
### T1.11: Backfill Data (If Required)
|
||||
|
||||
**Status:** TODO
|
||||
**Assignee:** TBD
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
**Description:**
|
||||
Backfill existing MongoDB data to PostgreSQL.
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T1.11.1: Create backfill script for tenants
|
||||
- [ ] T1.11.2: Create backfill script for users
|
||||
- [ ] T1.11.3: Create backfill script for service accounts
|
||||
- [ ] T1.11.4: Create backfill script for clients/scopes
|
||||
- [ ] T1.11.5: Create backfill script for active tokens
|
||||
- [ ] T1.11.6: Verify record counts match
|
||||
- [ ] T1.11.7: Verify sample records match
|
||||
|
||||
**Verification:**
|
||||
- [ ] All Tier A data backfilled
|
||||
- [ ] Record counts match
|
||||
- [ ] Sample verification passed
|
||||
|
||||
---
|
||||
|
||||
### T1.12: Switch to PostgreSQL-Only
|
||||
|
||||
**Status:** TODO
|
||||
**Assignee:** TBD
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
**Description:**
|
||||
Switch Authority to PostgreSQL-only mode.
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T1.12.1: Update configuration to `"Authority": "Postgres"`
|
||||
- [ ] T1.12.2: Deploy to staging
|
||||
- [ ] T1.12.3: Run full integration test suite
|
||||
- [ ] T1.12.4: Monitor for errors/issues
|
||||
- [ ] T1.12.5: Deploy to production
|
||||
- [ ] T1.12.6: Monitor production metrics
|
||||
|
||||
**Verification:**
|
||||
- [ ] All tests pass in staging
|
||||
- [ ] No errors in production
|
||||
- [ ] Performance metrics acceptable
|
||||
|
||||
---
|
||||
|
||||
## Exit Criteria
|
||||
|
||||
- [ ] All repository interfaces implemented for PostgreSQL
|
||||
- [ ] All integration tests pass
|
||||
- [ ] Verification tests pass (MongoDB vs PostgreSQL comparison)
|
||||
- [ ] Configuration switch working
|
||||
- [ ] Authority running on PostgreSQL in production
|
||||
- [ ] MongoDB Authority collections archived
|
||||
|
||||
---
|
||||
|
||||
## Risks & Mitigations
|
||||
|
||||
| Risk | Likelihood | Impact | Mitigation |
|
||||
|------|------------|--------|------------|
|
||||
| Token verification regression | Low | High | Extensive testing, dual-write |
|
||||
| OAuth flow breakage | Low | High | Test all OAuth flows |
|
||||
| Performance regression | Medium | Medium | Load testing before switch |
|
||||
|
||||
---
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
1. Change configuration: `"Authority": "Mongo"`
|
||||
2. Deploy configuration change
|
||||
3. MongoDB still has all data (dual-write period)
|
||||
4. Investigate and fix PostgreSQL issues
|
||||
5. Re-attempt conversion
|
||||
|
||||
---
|
||||
|
||||
*Phase Version: 1.0.0*
|
||||
*Last Updated: 2025-11-28*
|
||||
305
docs/db/tasks/PHASE_2_SCHEDULER.md
Normal file
305
docs/db/tasks/PHASE_2_SCHEDULER.md
Normal file
@@ -0,0 +1,305 @@
|
||||
# Phase 2: Scheduler Module Conversion
|
||||
|
||||
**Sprint:** 3
|
||||
**Duration:** 1 sprint
|
||||
**Status:** TODO
|
||||
**Dependencies:** Phase 0 (Foundations)
|
||||
|
||||
---
|
||||
|
||||
## Objectives
|
||||
|
||||
1. Create `StellaOps.Scheduler.Storage.Postgres` project
|
||||
2. Implement Scheduler schema in PostgreSQL
|
||||
3. Implement 7+ repository interfaces
|
||||
4. Replace MongoDB job tracking with PostgreSQL
|
||||
5. Implement PostgreSQL advisory locks for distributed locking
|
||||
|
||||
---
|
||||
|
||||
## Deliverables
|
||||
|
||||
| Deliverable | Acceptance Criteria |
|
||||
|-------------|---------------------|
|
||||
| Scheduler schema | All tables created with indexes |
|
||||
| Repository implementations | All 7+ interfaces implemented |
|
||||
| Advisory locks | Distributed locking working |
|
||||
| Integration tests | 100% coverage of CRUD operations |
|
||||
| Verification report | Schedule execution verified |
|
||||
|
||||
---
|
||||
|
||||
## Schema Reference
|
||||
|
||||
See [SPECIFICATION.md](../SPECIFICATION.md) Section 5.4 for complete Scheduler schema.
|
||||
|
||||
**Tables:**
|
||||
- `scheduler.schedules`
|
||||
- `scheduler.triggers`
|
||||
- `scheduler.runs`
|
||||
- `scheduler.graph_jobs`
|
||||
- `scheduler.policy_jobs`
|
||||
- `scheduler.impact_snapshots`
|
||||
- `scheduler.workers`
|
||||
- `scheduler.execution_logs`
|
||||
- `scheduler.locks`
|
||||
- `scheduler.run_summaries`
|
||||
- `scheduler.audit`
|
||||
|
||||
---
|
||||
|
||||
## Task Breakdown
|
||||
|
||||
### T2.1: Create Scheduler.Storage.Postgres Project
|
||||
|
||||
**Status:** TODO
|
||||
**Assignee:** TBD
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T2.1.1: Create project structure
|
||||
- [ ] T2.1.2: Add NuGet references
|
||||
- [ ] T2.1.3: Create `SchedulerDataSource` class
|
||||
- [ ] T2.1.4: Create `ServiceCollectionExtensions.cs`
|
||||
|
||||
---
|
||||
|
||||
### T2.2: Implement Schema Migrations
|
||||
|
||||
**Status:** TODO
|
||||
**Assignee:** TBD
|
||||
**Estimate:** 1 day
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T2.2.1: Create `V001_CreateSchedulerSchema` migration
|
||||
- [ ] T2.2.2: Include all tables and indexes
|
||||
- [ ] T2.2.3: Add partial index for active schedules
|
||||
- [ ] T2.2.4: Test migration idempotency
|
||||
|
||||
---
|
||||
|
||||
### T2.3: Implement Schedule Repository
|
||||
|
||||
**Status:** TODO
|
||||
**Assignee:** TBD
|
||||
**Estimate:** 1 day
|
||||
|
||||
**Interface:**
|
||||
```csharp
|
||||
public interface IScheduleRepository
|
||||
{
|
||||
Task<Schedule?> GetAsync(string tenantId, string scheduleId, CancellationToken ct);
|
||||
Task<IReadOnlyList<Schedule>> ListAsync(string tenantId, ScheduleQueryOptions? options, CancellationToken ct);
|
||||
Task UpsertAsync(Schedule schedule, CancellationToken ct);
|
||||
Task<bool> SoftDeleteAsync(string tenantId, string scheduleId, string deletedBy, DateTimeOffset deletedAt, CancellationToken ct);
|
||||
Task<IReadOnlyList<Schedule>> GetDueSchedulesAsync(DateTimeOffset now, CancellationToken ct);
|
||||
}
|
||||
```
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T2.3.1: Implement all interface methods
|
||||
- [ ] T2.3.2: Handle soft delete correctly
|
||||
- [ ] T2.3.3: Implement GetDueSchedules for trigger calculation
|
||||
- [ ] T2.3.4: Write integration tests
|
||||
|
||||
---
|
||||
|
||||
### T2.4: Implement Run Repository
|
||||
|
||||
**Status:** TODO
|
||||
**Assignee:** TBD
|
||||
**Estimate:** 1 day
|
||||
|
||||
**Interface:**
|
||||
```csharp
|
||||
public interface IRunRepository
|
||||
{
|
||||
Task<Run?> GetAsync(string tenantId, Guid runId, CancellationToken ct);
|
||||
Task<IReadOnlyList<Run>> ListAsync(string tenantId, RunQueryOptions? options, CancellationToken ct);
|
||||
Task<Run> CreateAsync(Run run, CancellationToken ct);
|
||||
Task<Run> UpdateAsync(Run run, CancellationToken ct);
|
||||
Task<IReadOnlyList<Run>> GetPendingRunsAsync(string tenantId, CancellationToken ct);
|
||||
Task<IReadOnlyList<Run>> GetRunsByScheduleAsync(string tenantId, Guid scheduleId, int limit, CancellationToken ct);
|
||||
}
|
||||
```
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T2.4.1: Implement all interface methods
|
||||
- [ ] T2.4.2: Handle state transitions
|
||||
- [ ] T2.4.3: Implement efficient pagination
|
||||
- [ ] T2.4.4: Write integration tests
|
||||
|
||||
---
|
||||
|
||||
### T2.5: Implement Graph Job Repository
|
||||
|
||||
**Status:** TODO
|
||||
**Assignee:** TBD
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T2.5.1: Implement CRUD operations
|
||||
- [ ] T2.5.2: Implement status queries
|
||||
- [ ] T2.5.3: Write integration tests
|
||||
|
||||
---
|
||||
|
||||
### T2.6: Implement Policy Job Repository
|
||||
|
||||
**Status:** TODO
|
||||
**Assignee:** TBD
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T2.6.1: Implement CRUD operations
|
||||
- [ ] T2.6.2: Implement status queries
|
||||
- [ ] T2.6.3: Write integration tests
|
||||
|
||||
---
|
||||
|
||||
### T2.7: Implement Impact Snapshot Repository
|
||||
|
||||
**Status:** TODO
|
||||
**Assignee:** TBD
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T2.7.1: Implement CRUD operations
|
||||
- [ ] T2.7.2: Implement queries by run
|
||||
- [ ] T2.7.3: Write integration tests
|
||||
|
||||
---
|
||||
|
||||
### T2.8: Implement Distributed Locking
|
||||
|
||||
**Status:** TODO
|
||||
**Assignee:** TBD
|
||||
**Estimate:** 1 day
|
||||
|
||||
**Description:**
|
||||
Implement distributed locking using PostgreSQL advisory locks.
|
||||
|
||||
**Options:**
|
||||
1. PostgreSQL advisory locks (`pg_advisory_lock`)
|
||||
2. Table-based locks with SELECT FOR UPDATE SKIP LOCKED
|
||||
3. Combination approach
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T2.8.1: Choose locking strategy
|
||||
- [ ] T2.8.2: Implement `IDistributedLock` interface
|
||||
- [ ] T2.8.3: Implement lock acquisition with timeout
|
||||
- [ ] T2.8.4: Implement lock renewal
|
||||
- [ ] T2.8.5: Implement lock release
|
||||
- [ ] T2.8.6: Write concurrency tests
|
||||
|
||||
**Implementation Example:**
|
||||
```csharp
|
||||
public sealed class PostgresDistributedLock : IDistributedLock
|
||||
{
|
||||
private readonly SchedulerDataSource _dataSource;
|
||||
|
||||
public async Task<IAsyncDisposable?> TryAcquireAsync(
|
||||
string lockKey,
|
||||
TimeSpan timeout,
|
||||
CancellationToken ct)
|
||||
{
|
||||
var lockId = ComputeLockId(lockKey);
|
||||
await using var connection = await _dataSource.OpenConnectionAsync("system", ct);
|
||||
|
||||
await using var cmd = connection.CreateCommand();
|
||||
cmd.CommandText = "SELECT pg_try_advisory_lock(@lock_id)";
|
||||
cmd.Parameters.AddWithValue("lock_id", lockId);
|
||||
|
||||
var acquired = await cmd.ExecuteScalarAsync(ct) is true;
|
||||
if (!acquired) return null;
|
||||
|
||||
return new LockHandle(connection, lockId);
|
||||
}
|
||||
|
||||
private static long ComputeLockId(string key)
|
||||
=> unchecked((long)key.GetHashCode());
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### T2.9: Implement Worker Registration
|
||||
|
||||
**Status:** TODO
|
||||
**Assignee:** TBD
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T2.9.1: Implement worker registration
|
||||
- [ ] T2.9.2: Implement heartbeat updates
|
||||
- [ ] T2.9.3: Implement dead worker detection
|
||||
- [ ] T2.9.4: Write integration tests
|
||||
|
||||
---
|
||||
|
||||
### T2.10: Add Configuration Switch
|
||||
|
||||
**Status:** TODO
|
||||
**Assignee:** TBD
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T2.10.1: Update service registration
|
||||
- [ ] T2.10.2: Test backend switching
|
||||
- [ ] T2.10.3: Document configuration
|
||||
|
||||
---
|
||||
|
||||
### T2.11: Run Verification Tests
|
||||
|
||||
**Status:** TODO
|
||||
**Assignee:** TBD
|
||||
**Estimate:** 1 day
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T2.11.1: Test schedule CRUD
|
||||
- [ ] T2.11.2: Test run creation and state transitions
|
||||
- [ ] T2.11.3: Test trigger calculation
|
||||
- [ ] T2.11.4: Test distributed locking under concurrency
|
||||
- [ ] T2.11.5: Test job execution end-to-end
|
||||
- [ ] T2.11.6: Generate verification report
|
||||
|
||||
---
|
||||
|
||||
### T2.12: Switch to PostgreSQL-Only
|
||||
|
||||
**Status:** TODO
|
||||
**Assignee:** TBD
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T2.12.1: Update configuration
|
||||
- [ ] T2.12.2: Deploy to staging
|
||||
- [ ] T2.12.3: Run integration tests
|
||||
- [ ] T2.12.4: Deploy to production
|
||||
- [ ] T2.12.5: Monitor metrics
|
||||
|
||||
---
|
||||
|
||||
## Exit Criteria
|
||||
|
||||
- [ ] All repository interfaces implemented
|
||||
- [ ] Distributed locking working correctly
|
||||
- [ ] All integration tests pass
|
||||
- [ ] Schedule execution working end-to-end
|
||||
- [ ] Scheduler running on PostgreSQL in production
|
||||
|
||||
---
|
||||
|
||||
## Risks & Mitigations
|
||||
|
||||
| Risk | Likelihood | Impact | Mitigation |
|
||||
|------|------------|--------|------------|
|
||||
| Lock contention | Medium | Medium | Test under load, tune timeouts |
|
||||
| Trigger calculation errors | Low | High | Extensive testing with edge cases |
|
||||
| State transition bugs | Medium | Medium | State machine tests |
|
||||
|
||||
---
|
||||
|
||||
*Phase Version: 1.0.0*
|
||||
*Last Updated: 2025-11-28*
|
||||
183
docs/db/tasks/PHASE_3_NOTIFY.md
Normal file
183
docs/db/tasks/PHASE_3_NOTIFY.md
Normal file
@@ -0,0 +1,183 @@
|
||||
# Phase 3: Notify Module Conversion
|
||||
|
||||
**Sprint:** 4
|
||||
**Duration:** 1 sprint
|
||||
**Status:** TODO
|
||||
**Dependencies:** Phase 0 (Foundations)
|
||||
|
||||
---
|
||||
|
||||
## Objectives
|
||||
|
||||
1. Create `StellaOps.Notify.Storage.Postgres` project
|
||||
2. Implement Notify schema in PostgreSQL
|
||||
3. Implement 15 repository interfaces
|
||||
4. Handle delivery tracking and escalation state
|
||||
|
||||
---
|
||||
|
||||
## Deliverables
|
||||
|
||||
| Deliverable | Acceptance Criteria |
|
||||
|-------------|---------------------|
|
||||
| Notify schema | All tables created with indexes |
|
||||
| Repository implementations | All 15 interfaces implemented |
|
||||
| Integration tests | 100% coverage of CRUD operations |
|
||||
| Verification report | Notification delivery verified |
|
||||
|
||||
---
|
||||
|
||||
## Schema Reference
|
||||
|
||||
See [SPECIFICATION.md](../SPECIFICATION.md) Section 5.5 for complete Notify schema.
|
||||
|
||||
**Tables:**
|
||||
- `notify.channels`
|
||||
- `notify.rules`
|
||||
- `notify.templates`
|
||||
- `notify.deliveries`
|
||||
- `notify.digests`
|
||||
- `notify.quiet_hours`
|
||||
- `notify.maintenance_windows`
|
||||
- `notify.escalation_policies`
|
||||
- `notify.escalation_states`
|
||||
- `notify.on_call_schedules`
|
||||
- `notify.inbox`
|
||||
- `notify.incidents`
|
||||
- `notify.audit`
|
||||
|
||||
---
|
||||
|
||||
## Task Breakdown
|
||||
|
||||
### T3.1: Create Notify.Storage.Postgres Project
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Create project structure
|
||||
- [ ] Add NuGet references
|
||||
- [ ] Create `NotifyDataSource` class
|
||||
- [ ] Create `ServiceCollectionExtensions.cs`
|
||||
|
||||
---
|
||||
|
||||
### T3.2: Implement Schema Migrations
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 1 day
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Create schema migration
|
||||
- [ ] Include all tables and indexes
|
||||
- [ ] Test migration idempotency
|
||||
|
||||
---
|
||||
|
||||
### T3.3: Implement Channel Repository
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Implement CRUD operations
|
||||
- [ ] Handle channel types (email, slack, teams, etc.)
|
||||
- [ ] Write integration tests
|
||||
|
||||
---
|
||||
|
||||
### T3.4: Implement Rule Repository
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Implement CRUD operations
|
||||
- [ ] Handle filter JSONB
|
||||
- [ ] Write integration tests
|
||||
|
||||
---
|
||||
|
||||
### T3.5: Implement Template Repository
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Implement CRUD operations
|
||||
- [ ] Handle localization
|
||||
- [ ] Write integration tests
|
||||
|
||||
---
|
||||
|
||||
### T3.6: Implement Delivery Repository
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 1 day
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Implement CRUD operations
|
||||
- [ ] Handle status transitions
|
||||
- [ ] Implement retry logic
|
||||
- [ ] Write integration tests
|
||||
|
||||
---
|
||||
|
||||
### T3.7: Implement Remaining Repositories
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 2 days
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Implement Digest repository
|
||||
- [ ] Implement QuietHours repository
|
||||
- [ ] Implement MaintenanceWindow repository
|
||||
- [ ] Implement EscalationPolicy repository
|
||||
- [ ] Implement EscalationState repository
|
||||
- [ ] Implement OnCallSchedule repository
|
||||
- [ ] Implement Inbox repository
|
||||
- [ ] Implement Incident repository
|
||||
- [ ] Implement Audit repository
|
||||
- [ ] Write integration tests for all
|
||||
|
||||
---
|
||||
|
||||
### T3.8: Add Configuration Switch
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
---
|
||||
|
||||
### T3.9: Run Verification Tests
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 1 day
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Test notification delivery flow
|
||||
- [ ] Test escalation handling
|
||||
- [ ] Test digest aggregation
|
||||
- [ ] Generate verification report
|
||||
|
||||
---
|
||||
|
||||
### T3.10: Switch to PostgreSQL-Only
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
---
|
||||
|
||||
## Exit Criteria
|
||||
|
||||
- [ ] All 15 repository interfaces implemented
|
||||
- [ ] All integration tests pass
|
||||
- [ ] Notification delivery working end-to-end
|
||||
- [ ] Notify running on PostgreSQL in production
|
||||
|
||||
---
|
||||
|
||||
*Phase Version: 1.0.0*
|
||||
*Last Updated: 2025-11-28*
|
||||
147
docs/db/tasks/PHASE_4_POLICY.md
Normal file
147
docs/db/tasks/PHASE_4_POLICY.md
Normal file
@@ -0,0 +1,147 @@
|
||||
# Phase 4: Policy Module Conversion
|
||||
|
||||
**Sprint:** 5
|
||||
**Duration:** 1 sprint
|
||||
**Status:** TODO
|
||||
**Dependencies:** Phase 0 (Foundations)
|
||||
|
||||
---
|
||||
|
||||
## Objectives
|
||||
|
||||
1. Create `StellaOps.Policy.Storage.Postgres` project
|
||||
2. Implement Policy schema in PostgreSQL
|
||||
3. Handle policy pack versioning correctly
|
||||
4. Implement risk profiles with version history
|
||||
|
||||
---
|
||||
|
||||
## Deliverables
|
||||
|
||||
| Deliverable | Acceptance Criteria |
|
||||
|-------------|---------------------|
|
||||
| Policy schema | All tables created with indexes |
|
||||
| Repository implementations | All 4+ interfaces implemented |
|
||||
| Version management | Pack versioning working correctly |
|
||||
| Integration tests | 100% coverage of CRUD operations |
|
||||
|
||||
---
|
||||
|
||||
## Schema Reference
|
||||
|
||||
See [SPECIFICATION.md](../SPECIFICATION.md) Section 5.6 for complete Policy schema.
|
||||
|
||||
**Tables:**
|
||||
- `policy.packs`
|
||||
- `policy.pack_versions`
|
||||
- `policy.rules`
|
||||
- `policy.risk_profiles`
|
||||
- `policy.evaluation_runs`
|
||||
- `policy.explanations`
|
||||
- `policy.exceptions`
|
||||
- `policy.audit`
|
||||
|
||||
---
|
||||
|
||||
## Task Breakdown
|
||||
|
||||
### T4.1: Create Policy.Storage.Postgres Project
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
---
|
||||
|
||||
### T4.2: Implement Schema Migrations
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 1 day
|
||||
|
||||
---
|
||||
|
||||
### T4.3: Implement Pack Repository
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 1 day
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Implement CRUD for packs
|
||||
- [ ] Implement version management
|
||||
- [ ] Handle active version promotion
|
||||
- [ ] Write integration tests
|
||||
|
||||
---
|
||||
|
||||
### T4.4: Implement Risk Profile Repository
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 1 day
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Implement CRUD operations
|
||||
- [ ] Handle version history
|
||||
- [ ] Implement GetVersionAsync
|
||||
- [ ] Implement ListVersionsAsync
|
||||
- [ ] Write integration tests
|
||||
|
||||
---
|
||||
|
||||
### T4.5: Implement Remaining Repositories
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 1.5 days
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Implement Evaluation Run repository
|
||||
- [ ] Implement Explanation repository
|
||||
- [ ] Implement Exception repository
|
||||
- [ ] Implement Audit repository
|
||||
- [ ] Write integration tests
|
||||
|
||||
---
|
||||
|
||||
### T4.6: Add Configuration Switch
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
---
|
||||
|
||||
### T4.7: Run Verification Tests
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 1 day
|
||||
|
||||
---
|
||||
|
||||
### T4.8: Migrate Active Policy Packs
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Export active packs from MongoDB
|
||||
- [ ] Import to PostgreSQL
|
||||
- [ ] Verify version numbers
|
||||
- [ ] Verify active version settings
|
||||
|
||||
---
|
||||
|
||||
### T4.9: Switch to PostgreSQL-Only
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
---
|
||||
|
||||
## Exit Criteria
|
||||
|
||||
- [ ] All repository interfaces implemented
|
||||
- [ ] Pack versioning working correctly
|
||||
- [ ] All integration tests pass
|
||||
- [ ] Policy running on PostgreSQL in production
|
||||
|
||||
---
|
||||
|
||||
*Phase Version: 1.0.0*
|
||||
*Last Updated: 2025-11-28*
|
||||
334
docs/db/tasks/PHASE_5_VULNERABILITIES.md
Normal file
334
docs/db/tasks/PHASE_5_VULNERABILITIES.md
Normal file
@@ -0,0 +1,334 @@
|
||||
# Phase 5: Vulnerability Index Conversion (Concelier)
|
||||
|
||||
**Sprint:** 6-7
|
||||
**Duration:** 2 sprints
|
||||
**Status:** TODO
|
||||
**Dependencies:** Phase 0 (Foundations)
|
||||
|
||||
---
|
||||
|
||||
## Objectives
|
||||
|
||||
1. Create `StellaOps.Concelier.Storage.Postgres` project
|
||||
2. Implement full vulnerability schema in PostgreSQL
|
||||
3. Build advisory conversion pipeline
|
||||
4. Maintain deterministic vulnerability matching
|
||||
|
||||
---
|
||||
|
||||
## Deliverables
|
||||
|
||||
| Deliverable | Acceptance Criteria |
|
||||
|-------------|---------------------|
|
||||
| Vuln schema | All tables created with indexes |
|
||||
| Conversion pipeline | MongoDB advisories converted to PostgreSQL |
|
||||
| Matching verification | Same CVEs found for identical SBOMs |
|
||||
| Integration tests | 100% coverage of query operations |
|
||||
|
||||
---
|
||||
|
||||
## Schema Reference
|
||||
|
||||
See [SPECIFICATION.md](../SPECIFICATION.md) Section 5.2 for complete vulnerability schema.
|
||||
|
||||
**Tables:**
|
||||
- `vuln.sources`
|
||||
- `vuln.feed_snapshots`
|
||||
- `vuln.advisory_snapshots`
|
||||
- `vuln.advisories`
|
||||
- `vuln.advisory_aliases`
|
||||
- `vuln.advisory_cvss`
|
||||
- `vuln.advisory_affected`
|
||||
- `vuln.advisory_references`
|
||||
- `vuln.advisory_credits`
|
||||
- `vuln.advisory_weaknesses`
|
||||
- `vuln.kev_flags`
|
||||
- `vuln.source_states`
|
||||
- `vuln.merge_events`
|
||||
|
||||
---
|
||||
|
||||
## Sprint 5a: Schema & Repositories
|
||||
|
||||
### T5a.1: Create Concelier.Storage.Postgres Project
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Create project structure
|
||||
- [ ] Add NuGet references
|
||||
- [ ] Create `ConcelierDataSource` class
|
||||
- [ ] Create `ServiceCollectionExtensions.cs`
|
||||
|
||||
---
|
||||
|
||||
### T5a.2: Implement Schema Migrations
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 1.5 days
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Create schema migration
|
||||
- [ ] Include all tables
|
||||
- [ ] Add full-text search index
|
||||
- [ ] Add PURL lookup index
|
||||
- [ ] Test migration idempotency
|
||||
|
||||
---
|
||||
|
||||
### T5a.3: Implement Source Repository
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Implement CRUD operations
|
||||
- [ ] Implement GetByKeyAsync
|
||||
- [ ] Write integration tests
|
||||
|
||||
---
|
||||
|
||||
### T5a.4: Implement Advisory Repository
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 2 days
|
||||
|
||||
**Interface:**
|
||||
```csharp
|
||||
public interface IAdvisoryRepository
|
||||
{
|
||||
Task<Advisory?> GetByKeyAsync(string advisoryKey, CancellationToken ct);
|
||||
Task<Advisory?> GetByAliasAsync(string aliasType, string aliasValue, CancellationToken ct);
|
||||
Task<IReadOnlyList<Advisory>> SearchAsync(AdvisorySearchQuery query, CancellationToken ct);
|
||||
Task<Advisory> UpsertAsync(Advisory advisory, CancellationToken ct);
|
||||
Task<IReadOnlyList<Advisory>> GetAffectingPackageAsync(string purl, CancellationToken ct);
|
||||
Task<IReadOnlyList<Advisory>> GetAffectingPackageNameAsync(string ecosystem, string name, CancellationToken ct);
|
||||
}
|
||||
```
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Implement GetByKeyAsync
|
||||
- [ ] Implement GetByAliasAsync (CVE lookup)
|
||||
- [ ] Implement SearchAsync with full-text search
|
||||
- [ ] Implement UpsertAsync with all child tables
|
||||
- [ ] Implement GetAffectingPackageAsync (PURL match)
|
||||
- [ ] Implement GetAffectingPackageNameAsync
|
||||
- [ ] Write integration tests
|
||||
|
||||
---
|
||||
|
||||
### T5a.5: Implement Child Table Repositories
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 2 days
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Implement Alias repository
|
||||
- [ ] Implement CVSS repository
|
||||
- [ ] Implement Affected repository
|
||||
- [ ] Implement Reference repository
|
||||
- [ ] Implement Credit repository
|
||||
- [ ] Implement Weakness repository
|
||||
- [ ] Implement KEV repository
|
||||
- [ ] Write integration tests
|
||||
|
||||
---
|
||||
|
||||
### T5a.6: Implement Source State Repository
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Implement CRUD operations
|
||||
- [ ] Implement cursor management
|
||||
- [ ] Write integration tests
|
||||
|
||||
---
|
||||
|
||||
## Sprint 5b: Conversion & Verification
|
||||
|
||||
### T5b.1: Build Advisory Conversion Service
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 2 days
|
||||
|
||||
**Description:**
|
||||
Create service to convert MongoDB advisory documents to PostgreSQL relational structure.
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Parse MongoDB `AdvisoryDocument` structure
|
||||
- [ ] Map to `vuln.advisories` table
|
||||
- [ ] Extract and normalize aliases
|
||||
- [ ] Extract and normalize CVSS metrics
|
||||
- [ ] Extract and normalize affected packages
|
||||
- [ ] Preserve provenance JSONB
|
||||
- [ ] Handle version ranges (keep as JSONB)
|
||||
- [ ] Handle normalized versions (keep as JSONB)
|
||||
|
||||
**Conversion Logic:**
|
||||
```csharp
|
||||
public sealed class AdvisoryConverter
|
||||
{
|
||||
public async Task ConvertAsync(
|
||||
IMongoCollection<AdvisoryDocument> source,
|
||||
IAdvisoryRepository target,
|
||||
CancellationToken ct)
|
||||
{
|
||||
await foreach (var doc in source.AsAsyncEnumerable(ct))
|
||||
{
|
||||
var advisory = MapToAdvisory(doc);
|
||||
await target.UpsertAsync(advisory, ct);
|
||||
}
|
||||
}
|
||||
|
||||
private Advisory MapToAdvisory(AdvisoryDocument doc)
|
||||
{
|
||||
// Extract from BsonDocument payload
|
||||
var payload = doc.Payload;
|
||||
return new Advisory
|
||||
{
|
||||
AdvisoryKey = doc.Id,
|
||||
PrimaryVulnId = payload["primaryVulnId"].AsString,
|
||||
Title = payload["title"]?.AsString,
|
||||
Summary = payload["summary"]?.AsString,
|
||||
// ... etc
|
||||
Provenance = BsonSerializer.Deserialize<JsonElement>(payload["provenance"]),
|
||||
};
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### T5b.2: Build Feed Import Pipeline
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 1 day
|
||||
|
||||
**Description:**
|
||||
Modify feed import to write directly to PostgreSQL.
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Update NVD importer to use PostgreSQL
|
||||
- [ ] Update OSV importer to use PostgreSQL
|
||||
- [ ] Update GHSA importer to use PostgreSQL
|
||||
- [ ] Update vendor feed importers
|
||||
- [ ] Test incremental imports
|
||||
|
||||
---
|
||||
|
||||
### T5b.3: Run Parallel Import
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 1 day
|
||||
|
||||
**Description:**
|
||||
Run imports to both MongoDB and PostgreSQL simultaneously.
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Configure dual-import mode
|
||||
- [ ] Run import cycle
|
||||
- [ ] Compare record counts
|
||||
- [ ] Sample comparison checks
|
||||
|
||||
---
|
||||
|
||||
### T5b.4: Verify Vulnerability Matching
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 2 days
|
||||
|
||||
**Description:**
|
||||
Verify that vulnerability matching produces identical results.
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Select sample SBOMs (various ecosystems)
|
||||
- [ ] Run matching with MongoDB backend
|
||||
- [ ] Run matching with PostgreSQL backend
|
||||
- [ ] Compare findings (must be identical)
|
||||
- [ ] Document any differences
|
||||
- [ ] Fix any issues found
|
||||
|
||||
**Verification Tests:**
|
||||
```csharp
|
||||
[Theory]
|
||||
[MemberData(nameof(GetSampleSboms))]
|
||||
public async Task Scanner_Should_Find_Same_Vulns(string sbomPath)
|
||||
{
|
||||
var sbom = await LoadSbom(sbomPath);
|
||||
|
||||
_config["Persistence:Concelier"] = "Mongo";
|
||||
var mongoFindings = await _scanner.ScanAsync(sbom);
|
||||
|
||||
_config["Persistence:Concelier"] = "Postgres";
|
||||
var postgresFindings = await _scanner.ScanAsync(sbom);
|
||||
|
||||
// Strict ordering for determinism
|
||||
postgresFindings.Should().BeEquivalentTo(mongoFindings,
|
||||
options => options.WithStrictOrdering());
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### T5b.5: Performance Optimization
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 1 day
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Analyze slow queries with EXPLAIN ANALYZE
|
||||
- [ ] Optimize indexes for common queries
|
||||
- [ ] Consider partial indexes for active advisories
|
||||
- [ ] Benchmark PostgreSQL vs MongoDB performance
|
||||
|
||||
---
|
||||
|
||||
### T5b.6: Switch Scanner to PostgreSQL
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Update configuration
|
||||
- [ ] Deploy to staging
|
||||
- [ ] Run full scan suite
|
||||
- [ ] Deploy to production
|
||||
|
||||
---
|
||||
|
||||
## Exit Criteria
|
||||
|
||||
- [ ] All repository interfaces implemented
|
||||
- [ ] Advisory conversion pipeline working
|
||||
- [ ] Vulnerability matching produces identical results
|
||||
- [ ] Feed imports working on PostgreSQL
|
||||
- [ ] Concelier running on PostgreSQL in production
|
||||
|
||||
---
|
||||
|
||||
## Risks & Mitigations
|
||||
|
||||
| Risk | Likelihood | Impact | Mitigation |
|
||||
|------|------------|--------|------------|
|
||||
| Matching discrepancies | Medium | High | Extensive comparison testing |
|
||||
| Performance regression on queries | Medium | Medium | Index optimization, query tuning |
|
||||
| Data loss during conversion | Low | High | Verify counts, sample checks |
|
||||
|
||||
---
|
||||
|
||||
## Data Volume Estimates
|
||||
|
||||
| Table | Estimated Rows | Growth Rate |
|
||||
|-------|----------------|-------------|
|
||||
| advisories | 300,000+ | ~100/day |
|
||||
| advisory_aliases | 600,000+ | ~200/day |
|
||||
| advisory_affected | 2,000,000+ | ~1000/day |
|
||||
| advisory_cvss | 400,000+ | ~150/day |
|
||||
|
||||
---
|
||||
|
||||
*Phase Version: 1.0.0*
|
||||
*Last Updated: 2025-11-28*
|
||||
434
docs/db/tasks/PHASE_6_VEX_GRAPH.md
Normal file
434
docs/db/tasks/PHASE_6_VEX_GRAPH.md
Normal file
@@ -0,0 +1,434 @@
|
||||
# Phase 6: VEX & Graph Conversion (Excititor)
|
||||
|
||||
**Sprint:** 8-10
|
||||
**Duration:** 2-3 sprints
|
||||
**Status:** TODO
|
||||
**Dependencies:** Phase 5 (Vulnerabilities)
|
||||
|
||||
---
|
||||
|
||||
## Objectives
|
||||
|
||||
1. Create `StellaOps.Excititor.Storage.Postgres` project
|
||||
2. Implement VEX schema in PostgreSQL
|
||||
3. Handle graph nodes/edges efficiently
|
||||
4. Preserve graph_revision_id stability (determinism critical)
|
||||
5. Maintain VEX statement lattice logic
|
||||
|
||||
---
|
||||
|
||||
## Deliverables
|
||||
|
||||
| Deliverable | Acceptance Criteria |
|
||||
|-------------|---------------------|
|
||||
| VEX schema | All tables created with indexes |
|
||||
| Graph storage | Nodes/edges efficiently stored |
|
||||
| Statement storage | VEX statements with full provenance |
|
||||
| Revision stability | Same inputs produce same revision_id |
|
||||
| Integration tests | 100% coverage |
|
||||
|
||||
---
|
||||
|
||||
## Schema Reference
|
||||
|
||||
See [SPECIFICATION.md](../SPECIFICATION.md) Section 5.3 for complete VEX schema.
|
||||
|
||||
**Tables:**
|
||||
- `vex.projects`
|
||||
- `vex.graph_revisions`
|
||||
- `vex.graph_nodes`
|
||||
- `vex.graph_edges`
|
||||
- `vex.statements`
|
||||
- `vex.observations`
|
||||
- `vex.linksets`
|
||||
- `vex.linkset_events`
|
||||
- `vex.consensus`
|
||||
- `vex.consensus_holds`
|
||||
- `vex.unknowns_snapshots`
|
||||
- `vex.unknown_items`
|
||||
- `vex.evidence_manifests`
|
||||
- `vex.cvss_receipts`
|
||||
- `vex.attestations`
|
||||
- `vex.timeline_events`
|
||||
|
||||
---
|
||||
|
||||
## Sprint 6a: Core Schema & Repositories
|
||||
|
||||
### T6a.1: Create Excititor.Storage.Postgres Project
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Create project structure
|
||||
- [ ] Add NuGet references
|
||||
- [ ] Create `ExcititorDataSource` class
|
||||
- [ ] Create `ServiceCollectionExtensions.cs`
|
||||
|
||||
---
|
||||
|
||||
### T6a.2: Implement Schema Migrations
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 1.5 days
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Create schema migration
|
||||
- [ ] Include all tables
|
||||
- [ ] Add indexes for graph traversal
|
||||
- [ ] Add indexes for VEX lookups
|
||||
- [ ] Test migration idempotency
|
||||
|
||||
---
|
||||
|
||||
### T6a.3: Implement Project Repository
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Implement CRUD operations
|
||||
- [ ] Handle tenant scoping
|
||||
- [ ] Write integration tests
|
||||
|
||||
---
|
||||
|
||||
### T6a.4: Implement VEX Statement Repository
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 1.5 days
|
||||
|
||||
**Interface:**
|
||||
```csharp
|
||||
public interface IVexStatementRepository
|
||||
{
|
||||
Task<VexStatement?> GetAsync(string tenantId, Guid statementId, CancellationToken ct);
|
||||
Task<IReadOnlyList<VexStatement>> GetByVulnerabilityAsync(
|
||||
string tenantId, string vulnerabilityId, CancellationToken ct);
|
||||
Task<IReadOnlyList<VexStatement>> GetByProjectAsync(
|
||||
string tenantId, Guid projectId, CancellationToken ct);
|
||||
Task<VexStatement> UpsertAsync(VexStatement statement, CancellationToken ct);
|
||||
Task<IReadOnlyList<VexStatement>> GetByGraphRevisionAsync(
|
||||
Guid graphRevisionId, CancellationToken ct);
|
||||
}
|
||||
```
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Implement all interface methods
|
||||
- [ ] Handle status and justification enums
|
||||
- [ ] Preserve evidence JSONB
|
||||
- [ ] Preserve provenance JSONB
|
||||
- [ ] Write integration tests
|
||||
|
||||
---
|
||||
|
||||
### T6a.5: Implement VEX Observation Repository
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 1 day
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Implement CRUD operations
|
||||
- [ ] Handle unique constraint on composite key
|
||||
- [ ] Implement FindByVulnerabilityAndProductAsync
|
||||
- [ ] Write integration tests
|
||||
|
||||
---
|
||||
|
||||
### T6a.6: Implement Linkset Repository
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Implement CRUD operations
|
||||
- [ ] Implement event logging
|
||||
- [ ] Write integration tests
|
||||
|
||||
---
|
||||
|
||||
### T6a.7: Implement Consensus Repository
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Implement CRUD operations
|
||||
- [ ] Implement hold management
|
||||
- [ ] Write integration tests
|
||||
|
||||
---
|
||||
|
||||
## Sprint 6b: Graph Storage
|
||||
|
||||
### T6b.1: Implement Graph Revision Repository
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 1 day
|
||||
|
||||
**Interface:**
|
||||
```csharp
|
||||
public interface IGraphRevisionRepository
|
||||
{
|
||||
Task<GraphRevision?> GetByIdAsync(Guid id, CancellationToken ct);
|
||||
Task<GraphRevision?> GetByRevisionIdAsync(string revisionId, CancellationToken ct);
|
||||
Task<GraphRevision?> GetLatestByProjectAsync(Guid projectId, CancellationToken ct);
|
||||
Task<GraphRevision> CreateAsync(GraphRevision revision, CancellationToken ct);
|
||||
Task<IReadOnlyList<GraphRevision>> GetHistoryAsync(
|
||||
Guid projectId, int limit, CancellationToken ct);
|
||||
}
|
||||
```
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Implement all interface methods
|
||||
- [ ] Handle revision_id uniqueness
|
||||
- [ ] Handle parent_revision_id linking
|
||||
- [ ] Write integration tests
|
||||
|
||||
---
|
||||
|
||||
### T6b.2: Implement Graph Node Repository
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 1.5 days
|
||||
|
||||
**Interface:**
|
||||
```csharp
|
||||
public interface IGraphNodeRepository
|
||||
{
|
||||
Task<GraphNode?> GetByIdAsync(long nodeId, CancellationToken ct);
|
||||
Task<GraphNode?> GetByKeyAsync(Guid graphRevisionId, string nodeKey, CancellationToken ct);
|
||||
Task<IReadOnlyList<GraphNode>> GetByRevisionAsync(
|
||||
Guid graphRevisionId, CancellationToken ct);
|
||||
Task BulkInsertAsync(
|
||||
Guid graphRevisionId, IEnumerable<GraphNode> nodes, CancellationToken ct);
|
||||
Task<int> GetCountAsync(Guid graphRevisionId, CancellationToken ct);
|
||||
}
|
||||
```
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Implement all interface methods
|
||||
- [ ] Implement bulk insert for efficiency
|
||||
- [ ] Handle node_key uniqueness per revision
|
||||
- [ ] Write integration tests
|
||||
|
||||
**Bulk Insert Optimization:**
|
||||
```csharp
|
||||
public async Task BulkInsertAsync(
|
||||
Guid graphRevisionId,
|
||||
IEnumerable<GraphNode> nodes,
|
||||
CancellationToken ct)
|
||||
{
|
||||
await using var connection = await _dataSource.OpenConnectionAsync("system", ct);
|
||||
await using var writer = await connection.BeginBinaryImportAsync(
|
||||
"COPY vex.graph_nodes (graph_revision_id, node_key, node_type, purl, name, version, attributes) " +
|
||||
"FROM STDIN (FORMAT BINARY)", ct);
|
||||
|
||||
foreach (var node in nodes)
|
||||
{
|
||||
await writer.StartRowAsync(ct);
|
||||
await writer.WriteAsync(graphRevisionId, ct);
|
||||
await writer.WriteAsync(node.NodeKey, ct);
|
||||
await writer.WriteAsync(node.NodeType, ct);
|
||||
await writer.WriteAsync(node.Purl, NpgsqlDbType.Text, ct);
|
||||
await writer.WriteAsync(node.Name, NpgsqlDbType.Text, ct);
|
||||
await writer.WriteAsync(node.Version, NpgsqlDbType.Text, ct);
|
||||
await writer.WriteAsync(JsonSerializer.Serialize(node.Attributes), NpgsqlDbType.Jsonb, ct);
|
||||
}
|
||||
|
||||
await writer.CompleteAsync(ct);
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### T6b.3: Implement Graph Edge Repository
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 1.5 days
|
||||
|
||||
**Interface:**
|
||||
```csharp
|
||||
public interface IGraphEdgeRepository
|
||||
{
|
||||
Task<IReadOnlyList<GraphEdge>> GetByRevisionAsync(
|
||||
Guid graphRevisionId, CancellationToken ct);
|
||||
Task<IReadOnlyList<GraphEdge>> GetOutgoingAsync(
|
||||
long fromNodeId, CancellationToken ct);
|
||||
Task<IReadOnlyList<GraphEdge>> GetIncomingAsync(
|
||||
long toNodeId, CancellationToken ct);
|
||||
Task BulkInsertAsync(
|
||||
Guid graphRevisionId, IEnumerable<GraphEdge> edges, CancellationToken ct);
|
||||
Task<int> GetCountAsync(Guid graphRevisionId, CancellationToken ct);
|
||||
}
|
||||
```
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Implement all interface methods
|
||||
- [ ] Implement bulk insert for efficiency
|
||||
- [ ] Optimize for traversal queries
|
||||
- [ ] Write integration tests
|
||||
|
||||
---
|
||||
|
||||
### T6b.4: Verify Graph Revision ID Stability
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 1 day
|
||||
|
||||
**Description:**
|
||||
Critical: Same SBOM + feeds + policy must produce identical revision_id.
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Document revision_id computation algorithm
|
||||
- [ ] Verify nodes are inserted in deterministic order
|
||||
- [ ] Verify edges are inserted in deterministic order
|
||||
- [ ] Write stability tests
|
||||
|
||||
**Stability Test:**
|
||||
```csharp
|
||||
[Fact]
|
||||
public async Task Same_Inputs_Should_Produce_Same_RevisionId()
|
||||
{
|
||||
var sbom = await LoadSbom("testdata/stable-sbom.json");
|
||||
var feedSnapshot = "feed-v1.2.3";
|
||||
var policyVersion = "policy-v1.0";
|
||||
|
||||
// Compute multiple times
|
||||
var revisions = new List<string>();
|
||||
for (int i = 0; i < 5; i++)
|
||||
{
|
||||
var graph = await _graphService.ComputeGraphAsync(
|
||||
sbom, feedSnapshot, policyVersion);
|
||||
revisions.Add(graph.RevisionId);
|
||||
}
|
||||
|
||||
// All must be identical
|
||||
revisions.Distinct().Should().HaveCount(1);
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Sprint 6c: Migration & Verification
|
||||
|
||||
### T6c.1: Build Graph Conversion Service
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 1.5 days
|
||||
|
||||
**Description:**
|
||||
Convert existing MongoDB graphs to PostgreSQL.
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Parse MongoDB graph documents
|
||||
- [ ] Map to graph_revisions table
|
||||
- [ ] Extract and insert nodes
|
||||
- [ ] Extract and insert edges
|
||||
- [ ] Verify node/edge counts match
|
||||
|
||||
---
|
||||
|
||||
### T6c.2: Build VEX Conversion Service
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 1 day
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Parse MongoDB VEX statements
|
||||
- [ ] Map to vex.statements table
|
||||
- [ ] Preserve provenance
|
||||
- [ ] Preserve evidence
|
||||
|
||||
---
|
||||
|
||||
### T6c.3: Run Dual Pipeline Comparison
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 2 days
|
||||
|
||||
**Description:**
|
||||
Run graph computation on both backends and compare.
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Select sample projects
|
||||
- [ ] Compute graphs with MongoDB
|
||||
- [ ] Compute graphs with PostgreSQL
|
||||
- [ ] Compare revision_ids (must match)
|
||||
- [ ] Compare node counts
|
||||
- [ ] Compare edge counts
|
||||
- [ ] Compare VEX statements
|
||||
- [ ] Document any differences
|
||||
|
||||
---
|
||||
|
||||
### T6c.4: Migrate Projects
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 1 day
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Identify projects to migrate (active VEX)
|
||||
- [ ] Run conversion for each project
|
||||
- [ ] Verify latest graph revision
|
||||
- [ ] Verify VEX statements
|
||||
|
||||
---
|
||||
|
||||
### T6c.5: Switch to PostgreSQL-Only
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Update configuration
|
||||
- [ ] Deploy to staging
|
||||
- [ ] Run full test suite
|
||||
- [ ] Deploy to production
|
||||
- [ ] Monitor metrics
|
||||
|
||||
---
|
||||
|
||||
## Exit Criteria
|
||||
|
||||
- [ ] All repository interfaces implemented
|
||||
- [ ] Graph storage working efficiently
|
||||
- [ ] Graph revision IDs stable (deterministic)
|
||||
- [ ] VEX statements preserved correctly
|
||||
- [ ] All comparison tests pass
|
||||
- [ ] Excititor running on PostgreSQL in production
|
||||
|
||||
---
|
||||
|
||||
## Risks & Mitigations
|
||||
|
||||
| Risk | Likelihood | Impact | Mitigation |
|
||||
|------|------------|--------|------------|
|
||||
| Revision ID instability | Medium | Critical | Deterministic ordering tests |
|
||||
| Graph storage performance | Medium | High | Bulk insert, index optimization |
|
||||
| VEX lattice logic errors | Low | High | Extensive comparison testing |
|
||||
|
||||
---
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
### Graph Storage
|
||||
|
||||
- Use `BIGSERIAL` for node/edge IDs (high volume)
|
||||
- Use `COPY` for bulk inserts (10-100x faster)
|
||||
- Index `(graph_revision_id, node_key)` for lookups
|
||||
- Index `(from_node_id)` and `(to_node_id)` for traversal
|
||||
|
||||
### Estimated Volumes
|
||||
|
||||
| Table | Estimated Rows per Project | Total Estimated |
|
||||
|-------|---------------------------|-----------------|
|
||||
| graph_nodes | 1,000 - 50,000 | 10M+ |
|
||||
| graph_edges | 2,000 - 100,000 | 20M+ |
|
||||
| vex_statements | 100 - 5,000 | 1M+ |
|
||||
|
||||
---
|
||||
|
||||
*Phase Version: 1.0.0*
|
||||
*Last Updated: 2025-11-28*
|
||||
305
docs/db/tasks/PHASE_7_CLEANUP.md
Normal file
305
docs/db/tasks/PHASE_7_CLEANUP.md
Normal file
@@ -0,0 +1,305 @@
|
||||
# Phase 7: Cleanup & Optimization
|
||||
|
||||
**Sprint:** 11
|
||||
**Duration:** 1 sprint
|
||||
**Status:** TODO
|
||||
**Dependencies:** All previous phases completed
|
||||
|
||||
---
|
||||
|
||||
## Objectives
|
||||
|
||||
1. Remove MongoDB dependencies from converted modules
|
||||
2. Archive MongoDB data
|
||||
3. Optimize PostgreSQL performance
|
||||
4. Update documentation
|
||||
5. Update air-gap kit
|
||||
|
||||
---
|
||||
|
||||
## Deliverables
|
||||
|
||||
| Deliverable | Acceptance Criteria |
|
||||
|-------------|---------------------|
|
||||
| Code cleanup | MongoDB code removed from converted modules |
|
||||
| Data archive | MongoDB data archived and documented |
|
||||
| Performance tuning | Query times within acceptable range |
|
||||
| Documentation | All docs updated for PostgreSQL |
|
||||
| Air-gap kit | PostgreSQL support added |
|
||||
|
||||
---
|
||||
|
||||
## Task Breakdown
|
||||
|
||||
### T7.1: Remove MongoDB Dependencies
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 2 days
|
||||
|
||||
**Description:**
|
||||
Remove MongoDB storage projects and references from converted modules.
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T7.1.1: Remove `StellaOps.Authority.Storage.Mongo` project
|
||||
- [ ] T7.1.2: Remove `StellaOps.Scheduler.Storage.Mongo` project
|
||||
- [ ] T7.1.3: Remove `StellaOps.Notify.Storage.Mongo` project
|
||||
- [ ] T7.1.4: Remove `StellaOps.Policy.Storage.Mongo` project
|
||||
- [ ] T7.1.5: Remove `StellaOps.Concelier.Storage.Mongo` project
|
||||
- [ ] T7.1.6: Remove `StellaOps.Excititor.Storage.Mongo` project
|
||||
- [ ] T7.1.7: Update solution files
|
||||
- [ ] T7.1.8: Remove dual-write wrappers
|
||||
- [ ] T7.1.9: Remove MongoDB configuration options
|
||||
- [ ] T7.1.10: Run full build to verify no broken references
|
||||
|
||||
**Verification:**
|
||||
- [ ] Solution builds without MongoDB packages
|
||||
- [ ] No MongoDB references in converted modules
|
||||
- [ ] All tests pass
|
||||
|
||||
---
|
||||
|
||||
### T7.2: Archive MongoDB Data
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 1 day
|
||||
|
||||
**Description:**
|
||||
Archive MongoDB databases for historical reference.
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T7.2.1: Take final MongoDB backup
|
||||
- [ ] T7.2.2: Export to BSON/JSON archives
|
||||
- [ ] T7.2.3: Store archives in secure location
|
||||
- [ ] T7.2.4: Document archive contents and structure
|
||||
- [ ] T7.2.5: Set retention policy for archives
|
||||
- [ ] T7.2.6: Schedule MongoDB cluster decommission
|
||||
|
||||
**Archive Structure:**
|
||||
```
|
||||
archives/
|
||||
├── mongodb-authority-2025-XX-XX.bson.gz
|
||||
├── mongodb-scheduler-2025-XX-XX.bson.gz
|
||||
├── mongodb-notify-2025-XX-XX.bson.gz
|
||||
├── mongodb-policy-2025-XX-XX.bson.gz
|
||||
├── mongodb-concelier-2025-XX-XX.bson.gz
|
||||
├── mongodb-excititor-2025-XX-XX.bson.gz
|
||||
└── ARCHIVE_MANIFEST.md
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### T7.3: PostgreSQL Performance Optimization
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 2 days
|
||||
|
||||
**Description:**
|
||||
Analyze and optimize PostgreSQL performance.
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T7.3.1: Enable `pg_stat_statements` extension
|
||||
- [ ] T7.3.2: Identify slow queries
|
||||
- [ ] T7.3.3: Analyze query plans with EXPLAIN ANALYZE
|
||||
- [ ] T7.3.4: Add missing indexes
|
||||
- [ ] T7.3.5: Remove unused indexes
|
||||
- [ ] T7.3.6: Tune PostgreSQL configuration
|
||||
- [ ] T7.3.7: Set up query monitoring dashboard
|
||||
- [ ] T7.3.8: Document performance baselines
|
||||
|
||||
**Configuration Tuning:**
|
||||
```ini
|
||||
# postgresql.conf optimizations
|
||||
shared_buffers = 25% of RAM
|
||||
effective_cache_size = 75% of RAM
|
||||
work_mem = 64MB
|
||||
maintenance_work_mem = 512MB
|
||||
random_page_cost = 1.1 # for SSD
|
||||
effective_io_concurrency = 200 # for SSD
|
||||
max_parallel_workers_per_gather = 4
|
||||
```
|
||||
|
||||
**Monitoring Queries:**
|
||||
```sql
|
||||
-- Top slow queries
|
||||
SELECT query, calls, mean_time, total_time
|
||||
FROM pg_stat_statements
|
||||
ORDER BY mean_time DESC
|
||||
LIMIT 20;
|
||||
|
||||
-- Unused indexes
|
||||
SELECT schemaname, tablename, indexname
|
||||
FROM pg_stat_user_indexes
|
||||
WHERE idx_scan = 0;
|
||||
|
||||
-- Table bloat
|
||||
SELECT schemaname, tablename,
|
||||
pg_size_pretty(pg_total_relation_size(schemaname || '.' || tablename)) as size
|
||||
FROM pg_stat_user_tables
|
||||
ORDER BY pg_total_relation_size(schemaname || '.' || tablename) DESC;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### T7.4: Update Documentation
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 1.5 days
|
||||
|
||||
**Description:**
|
||||
Update all documentation to reflect PostgreSQL as the primary database.
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T7.4.1: Update `docs/07_HIGH_LEVEL_ARCHITECTURE.md`
|
||||
- [ ] T7.4.2: Update module architecture docs
|
||||
- [ ] T7.4.3: Update deployment guides
|
||||
- [ ] T7.4.4: Update operations runbooks
|
||||
- [ ] T7.4.5: Update troubleshooting guides
|
||||
- [ ] T7.4.6: Update `CLAUDE.md` technology stack
|
||||
- [ ] T7.4.7: Create PostgreSQL operations guide
|
||||
- [ ] T7.4.8: Document backup/restore procedures
|
||||
- [ ] T7.4.9: Document scaling recommendations
|
||||
|
||||
**New Documents:**
|
||||
- `docs/operations/postgresql-guide.md`
|
||||
- `docs/operations/postgresql-backup-restore.md`
|
||||
- `docs/operations/postgresql-troubleshooting.md`
|
||||
|
||||
---
|
||||
|
||||
### T7.5: Update Air-Gap Kit
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 1 day
|
||||
|
||||
**Description:**
|
||||
Update offline/air-gap kit to include PostgreSQL.
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T7.5.1: Add PostgreSQL container image to kit
|
||||
- [ ] T7.5.2: Update kit scripts for PostgreSQL setup
|
||||
- [ ] T7.5.3: Include schema migrations in kit
|
||||
- [ ] T7.5.4: Update kit documentation
|
||||
- [ ] T7.5.5: Test kit installation in air-gapped environment
|
||||
- [ ] T7.5.6: Update `docs/24_OFFLINE_KIT.md`
|
||||
|
||||
**Air-Gap Kit Structure:**
|
||||
```
|
||||
offline-kit/
|
||||
├── images/
|
||||
│ ├── postgres-16-alpine.tar
|
||||
│ └── stellaops-*.tar
|
||||
├── schemas/
|
||||
│ ├── authority.sql
|
||||
│ ├── vuln.sql
|
||||
│ ├── vex.sql
|
||||
│ ├── scheduler.sql
|
||||
│ ├── notify.sql
|
||||
│ └── policy.sql
|
||||
├── scripts/
|
||||
│ ├── setup-postgres.sh
|
||||
│ ├── run-migrations.sh
|
||||
│ └── import-data.sh
|
||||
└── docs/
|
||||
└── OFFLINE_SETUP.md
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### T7.6: Final Verification
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 1 day
|
||||
|
||||
**Description:**
|
||||
Run final verification of all systems.
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T7.6.1: Run full integration test suite
|
||||
- [ ] T7.6.2: Run performance benchmark suite
|
||||
- [ ] T7.6.3: Verify all modules on PostgreSQL
|
||||
- [ ] T7.6.4: Verify determinism tests pass
|
||||
- [ ] T7.6.5: Verify air-gap kit works
|
||||
- [ ] T7.6.6: Generate final verification report
|
||||
- [ ] T7.6.7: Get sign-off from stakeholders
|
||||
|
||||
---
|
||||
|
||||
### T7.7: Decommission MongoDB
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
**Description:**
|
||||
Final decommission of MongoDB infrastructure.
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T7.7.1: Verify no services using MongoDB
|
||||
- [ ] T7.7.2: Stop MongoDB instances
|
||||
- [ ] T7.7.3: Archive final state
|
||||
- [ ] T7.7.4: Remove MongoDB from infrastructure
|
||||
- [ ] T7.7.5: Update monitoring/alerting
|
||||
- [ ] T7.7.6: Update cost projections
|
||||
|
||||
---
|
||||
|
||||
## Exit Criteria
|
||||
|
||||
- [ ] All MongoDB code removed from converted modules
|
||||
- [ ] MongoDB data archived
|
||||
- [ ] PostgreSQL performance optimized
|
||||
- [ ] All documentation updated
|
||||
- [ ] Air-gap kit updated and tested
|
||||
- [ ] Final verification report approved
|
||||
- [ ] MongoDB infrastructure decommissioned
|
||||
|
||||
---
|
||||
|
||||
## Post-Conversion Monitoring
|
||||
|
||||
### First Week
|
||||
|
||||
- Monitor error rates closely
|
||||
- Track query performance
|
||||
- Watch for any data inconsistencies
|
||||
- Have rollback plan ready (restore MongoDB)
|
||||
|
||||
### First Month
|
||||
|
||||
- Review query statistics weekly
|
||||
- Optimize any slow queries found
|
||||
- Monitor storage growth
|
||||
- Adjust vacuum settings if needed
|
||||
|
||||
### Ongoing
|
||||
|
||||
- Regular performance reviews
|
||||
- Index maintenance
|
||||
- Backup verification
|
||||
- Capacity planning
|
||||
|
||||
---
|
||||
|
||||
## Rollback Considerations
|
||||
|
||||
**Note:** After Phase 7 completion, rollback to MongoDB becomes significantly more complex. Ensure all stakeholders understand:
|
||||
|
||||
1. MongoDB archives are read-only backup
|
||||
2. Any new data created after cutover is PostgreSQL-only
|
||||
3. Full rollback would require data export/import
|
||||
|
||||
---
|
||||
|
||||
## Success Metrics
|
||||
|
||||
| Metric | Target | Measurement |
|
||||
|--------|--------|-------------|
|
||||
| Query latency (p95) | < 100ms | pg_stat_statements |
|
||||
| Error rate | < 0.01% | Application logs |
|
||||
| Storage efficiency | < 120% of MongoDB | Disk usage |
|
||||
| Test coverage | 100% | CI reports |
|
||||
| Documentation coverage | 100% | Manual review |
|
||||
|
||||
---
|
||||
|
||||
*Phase Version: 1.0.0*
|
||||
*Last Updated: 2025-11-28*
|
||||
Reference in New Issue
Block a user