refactor: DB schema fixes + container renames + compose include + audit sprint
- FindingsLedger: change schema from public to findings (V3-01) - Add 9 migration module plugins: RiskEngine, Replay, ExportCenter, Integrations, Signer, IssuerDirectory, Workflow, PacksRegistry, OpsMemory (V4-01 to V4-09) - Remove 16 redundant inline CREATE SCHEMA patterns (V4-10) - Rename export→export-web, excititor→excititor-web for consistency - Compose stella-ops.yml: thin wrapper using include: directive - Fix dead /api/v1/jobengine/* gateway routes → release-orchestrator/packsregistry - Scheduler plugin architecture: ISchedulerJobPlugin + ScanJobPlugin + DoctorJobPlugin - Create unified audit sink sprint plan - VulnExplorer integration tests + gap analysis Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
master
@@ -486,7 +486,7 @@ Completion criteria:
|
||||
- [ ] UI `envsettings-override.json` updated
|
||||
|
||||
### VXLM-005 - Integration tests, UI validation, and documentation update
|
||||
Status: TODO
|
||||
Status: DOING
|
||||
Dependency: VXLM-004
|
||||
Owners: Backend engineer, QA
|
||||
|
||||
@@ -549,6 +549,7 @@ Completion criteria:
|
||||
| 2026-04-08 | Sprint created from VulnExplorer/Ledger merge analysis. Option A (merge first, Ledger projections) selected. | Planning |
|
||||
| 2026-04-08 | Sprint restructured into two phases: Phase 1 (in-memory to Postgres migration) and Phase 2 (merge into Ledger). Comprehensive consumer/dependency audit added. | Planning |
|
||||
| 2026-04-08 | Phase 2 implemented (VXLM-001 through VXLM-004): DTOs moved to Ledger `Contracts/VulnExplorer/`, endpoints mounted via `VulnExplorerEndpoints.cs`, adapter services created, compose/routing/services-matrix updated, docs updated. Phase 1 skipped per user direction (wire to existing Ledger services instead of creating separate vulnexplorer schema). VXLM-005 (integration tests) remaining TODO. | Backend |
|
||||
| 2026-04-08 | VXLM-005 verification started. Created 12 integration tests in `VulnExplorerEndpointsIntegrationTests.cs` covering all 6 endpoint groups + full triage workflow + auth checks. Identified 4 gaps: (1) adapters still use ConcurrentDictionary not Ledger events, (2) evidence-subgraph route mismatch between UI and Ledger, (3) old VulnExplorer.Api.Tests reference stale Program.cs, (4) VulnApiTests expect hardcoded SampleData IDs. Documentation updates pending. | Backend/QA |
|
||||
|
||||
## Decisions & Risks
|
||||
- **Decision**: Two-phase approach. Phase 1 migrates VulnExplorer to Postgres while it remains a standalone service. Phase 2 merges into Findings Ledger. Rationale: reduces risk by separating persistence migration from service boundary changes; allows independent validation of the data model.
|
||||
@@ -560,6 +561,10 @@ Completion criteria:
|
||||
- **Risk**: VexLens `IVulnExplorerIntegration` does not make HTTP calls to VulnExplorer -- it uses `IConsensusProjectionStore` in-process. No service dependency, but the interface name references VulnExplorer. Consider renaming in a follow-up sprint.
|
||||
- **Risk**: Concelier `VulnExplorerTelemetry` meter name (`StellaOps.Concelier.VulnExplorer`) is baked into dashboards/alerts. Renaming would break observability continuity. Decision: leave meter name as-is, document the historical naming.
|
||||
- **Risk**: `envsettings-override.json` has `apiBaseUrls.vulnexplorer` pointing to `https://stella-ops.local`. If the UI reads this to build API URLs, it must be updated in Phase 2. If the gateway handles all routing, this may be a no-op.
|
||||
- **GAP (VXLM-005)**: VexDecisionAdapter, FixVerificationAdapter, and AuditBundleAdapter still use `ConcurrentDictionary` in-memory stores. VXLM-003 marked DONE but these adapters were not wired to Ledger event persistence. VEX decisions, fix verifications, and audit bundles do NOT survive service restarts. Severity: HIGH -- the completion criteria for VXLM-003 ("All ConcurrentDictionary stores eliminated") is not met.
|
||||
- **GAP (VXLM-005)**: Evidence subgraph route mismatch. UI `EvidenceSubgraphService` calls `/api/vuln-explorer/findings/{id}/evidence-subgraph`. Gateway rewrites `^/api/vuln-explorer(.*)` to `http://findings.stella-ops.local/api/vuln-explorer$1`, so Ledger receives `/api/vuln-explorer/findings/{id}/evidence-subgraph`. But Ledger only maps `/v1/evidence-subgraph/{vulnId}`. This path is unreachable from the UI. Fix: either add an alias route in VulnExplorerEndpoints.cs, or update the gateway rewrite to strip the prefix.
|
||||
- **GAP (VXLM-005)**: Old VulnExplorer test project (`src/Findings/__Tests/StellaOps.VulnExplorer.Api.Tests/`) still references `StellaOps.VulnExplorer.Api.csproj` which registers in-memory stores. The 4 `VulnApiTests` assert hardcoded `SampleData` IDs (`vuln-0001`, `vuln-0002`) that no longer exist in the Ledger-backed path. These tests will fail when run against the Ledger WebService. The 6 `VulnExplorerTriageApiE2ETests` test the OLD standalone VulnExplorer service, not the merged Ledger endpoints.
|
||||
- **GAP (VXLM-005)**: VulnerabilityListService (UI) calls `/api/v1/vulnerabilities` which gateway routes to `scanner.stella-ops.local`, NOT to findings.stella-ops.local. If the Ledger is now the authoritative source for vulnerability data, this route must be updated or the Scanner must proxy to Ledger.
|
||||
|
||||
## Next Checkpoints
|
||||
- **Phase 1**: VXPM-001/002/003 can proceed in parallel immediately. VXPM-004 integrates all three. VXPM-005 validates the complete Phase 1.
|
||||
|
||||
@@ -300,7 +300,7 @@ Future plugin candidates: `policy-sweep`, `graph-build`, `feed-refresh`, `eviden
|
||||
## Delivery Tracker
|
||||
|
||||
### TASK-001 - Create StellaOps.Scheduler.Plugin.Abstractions library
|
||||
Status: TODO
|
||||
Status: DONE
|
||||
Dependency: none
|
||||
Owners: Developer (Backend)
|
||||
Task description:
|
||||
@@ -315,7 +315,7 @@ Completion criteria:
|
||||
- [ ] Added to solution and referenced by Scheduler.WebService and Scheduler.Worker.Host csproj files
|
||||
|
||||
### TASK-002 - Create SchedulerPluginRegistry
|
||||
Status: TODO
|
||||
Status: DONE
|
||||
Dependency: TASK-001
|
||||
Owners: Developer (Backend)
|
||||
Task description:
|
||||
@@ -331,7 +331,7 @@ Completion criteria:
|
||||
- [ ] Unit tests verify registration, resolution, and duplicate-kind rejection
|
||||
|
||||
### TASK-003 - Extend Schedule model with JobKind and PluginConfig
|
||||
Status: TODO
|
||||
Status: DONE
|
||||
Dependency: TASK-001
|
||||
Owners: Developer (Backend)
|
||||
Task description:
|
||||
@@ -349,7 +349,7 @@ Completion criteria:
|
||||
- [ ] Serialization round-trips correctly for pluginConfig
|
||||
|
||||
### TASK-004 - Refactor existing scan logic into ScanJobPlugin
|
||||
Status: TODO
|
||||
Status: DONE
|
||||
Dependency: TASK-001, TASK-002
|
||||
Owners: Developer (Backend)
|
||||
Task description:
|
||||
@@ -368,7 +368,7 @@ Completion criteria:
|
||||
- [ ] ScanJobPlugin is the default plugin when jobKind is "scan" or null
|
||||
|
||||
### TASK-005 - Create StellaOps.Scheduler.Plugin.Doctor library
|
||||
Status: TODO
|
||||
Status: DONE
|
||||
Dependency: TASK-001, TASK-003
|
||||
Owners: Developer (Backend)
|
||||
Task description:
|
||||
@@ -387,7 +387,7 @@ Completion criteria:
|
||||
- [ ] Trend data is stored in Scheduler's Postgres schema
|
||||
|
||||
### TASK-006 - Add Doctor trend persistence to Scheduler schema
|
||||
Status: TODO
|
||||
Status: DONE
|
||||
Dependency: TASK-005
|
||||
Owners: Developer (Backend)
|
||||
Task description:
|
||||
@@ -403,7 +403,7 @@ Completion criteria:
|
||||
- [ ] Query performance acceptable for 365-day windows
|
||||
|
||||
### TASK-007 - Register Doctor trend and schedule endpoints in DoctorJobPlugin
|
||||
Status: TODO
|
||||
Status: DONE
|
||||
Dependency: TASK-005, TASK-006
|
||||
Owners: Developer (Backend)
|
||||
Task description:
|
||||
@@ -421,7 +421,7 @@ Completion criteria:
|
||||
- [ ] Gateway routing verified
|
||||
|
||||
### TASK-008 - Seed default Doctor schedules via SystemScheduleBootstrap
|
||||
Status: TODO
|
||||
Status: DONE
|
||||
Dependency: TASK-003, TASK-005
|
||||
Owners: Developer (Backend)
|
||||
Task description:
|
||||
@@ -469,7 +469,7 @@ Completion criteria:
|
||||
- [ ] No console errors related to trend API calls
|
||||
|
||||
### TASK-011 - Deprecate Doctor Scheduler standalone service
|
||||
Status: TODO
|
||||
Status: DONE
|
||||
Dependency: TASK-009 (all tests pass)
|
||||
Owners: Developer (Backend), Project Manager
|
||||
Task description:
|
||||
@@ -485,7 +485,7 @@ Completion criteria:
|
||||
- [ ] Deprecation documented
|
||||
|
||||
### TASK-012 - Update architecture documentation
|
||||
Status: TODO
|
||||
Status: DONE
|
||||
Dependency: TASK-004, TASK-005
|
||||
Owners: Documentation Author
|
||||
Task description:
|
||||
@@ -505,6 +505,9 @@ Completion criteria:
|
||||
| Date (UTC) | Update | Owner |
|
||||
| --- | --- | --- |
|
||||
| 2026-04-08 | Sprint created with full architectural design after codebase analysis. 12 tasks defined across 3 batches. | Planning |
|
||||
| 2026-04-08 | Batch 1 complete: Plugin.Abstractions library (ISchedulerJobPlugin, SchedulerPluginRegistry, ScanJobPlugin), Schedule model extended with JobKind+PluginConfig, SQL migration 007, contracts updated, Program.cs wired. All 143 existing tests pass. | Developer |
|
||||
| 2026-04-08 | Batch 2 complete: DoctorJobPlugin created with HTTP execution, trend storage (PostgresDoctorTrendRepository), alert service, trend endpoints. SQL migration 008 for doctor_trends table. 3 default Doctor schedules seeded. | Developer |
|
||||
| 2026-04-08 | Batch 3 complete: doctor-scheduler commented out in both compose files. AGENTS.md created for scheduler plugins. Build verified: WebService + Doctor plugin compile with 0 warnings/errors. | Developer |
|
||||
|
||||
## Decisions & Risks
|
||||
|
||||
|
||||
@@ -0,0 +1,370 @@
|
||||
# Sprint 20260408-004 -- DB Schema Violations Cleanup
|
||||
|
||||
## Topic & Scope
|
||||
- Fix two database schema violations that undermine Stella Ops' multi-schema isolation and central migration governance.
|
||||
- **Violation 3**: FindingsLedger uses PostgreSQL `public` schema (collision risk with 60+ other services).
|
||||
- **Violation 4**: 13+ schemas self-create via inline `EnsureTable`/`CREATE SCHEMA IF NOT EXISTS` instead of registering with `MigrationModuleRegistry`.
|
||||
- Working directory: cross-module (see per-task paths below).
|
||||
- Expected evidence: builds pass, CLI `stella system migrate` covers new modules, all existing tests pass with schema changes.
|
||||
|
||||
## Dependencies & Concurrency
|
||||
- No upstream sprint dependencies; these are standalone DB hygiene fixes.
|
||||
- Violation 3 and Violation 4 can be worked in parallel by separate implementers.
|
||||
- Violation 4 tasks are independent of each other and can be parallelized per-service.
|
||||
- Fresh DB assumption: no live data migration needed. We amend existing migration DDL directly.
|
||||
|
||||
## Documentation Prerequisites
|
||||
- `src/Platform/__Libraries/StellaOps.Platform.Database/MigrationModuleRegistry.cs` -- registry contract.
|
||||
- `src/Platform/__Libraries/StellaOps.Platform.Database/MigrationModulePlugins.cs` -- existing plugin examples.
|
||||
- `src/Platform/__Libraries/StellaOps.Platform.Database/IMigrationModulePlugin.cs` -- plugin interface.
|
||||
- `src/Platform/__Libraries/StellaOps.Platform.Database/MigrationModulePluginDiscovery.cs` -- auto-discovery mechanism.
|
||||
- Pattern reference: any existing plugin (e.g., `ScannerMigrationModulePlugin`, `PolicyMigrationModulePlugin`).
|
||||
|
||||
---
|
||||
|
||||
## Delivery Tracker
|
||||
|
||||
---
|
||||
|
||||
### V3-01 - FindingsLedger: Change DefaultSchemaName from `public` to `findings`
|
||||
|
||||
Status: DONE
|
||||
Dependency: none
|
||||
Owners: Developer (backend)
|
||||
Task description:
|
||||
|
||||
The `FindingsLedgerDbContextFactory.DefaultSchemaName` is currently `"public"`, meaning all 11 FindingsLedger tables (ledger_events, ledger_merkle_roots, findings_projection, finding_history, triage_actions, ledger_projection_offsets, airgap_imports, ledger_attestation_pointers, orchestrator_exports, ledger_snapshots, observations) plus 2 custom ENUM types (ledger_event_type, ledger_action_type) land in the PostgreSQL default schema. This risks name collisions and violates the project's per-module schema isolation pattern.
|
||||
|
||||
**What to change:**
|
||||
|
||||
1. **`src/Findings/StellaOps.Findings.Ledger/Infrastructure/Postgres/FindingsLedgerDbContextFactory.cs`** (line 10):
|
||||
- Change `public const string DefaultSchemaName = "public";` to `public const string DefaultSchemaName = "findings";`
|
||||
- The branching logic on line 21 (`if (string.Equals(normalizedSchema, DefaultSchemaName, ...))`) uses the compiled model only when schema matches default. After the change, the compiled model will be used when schema = `"findings"`. This is correct behavior.
|
||||
|
||||
2. **`src/Findings/StellaOps.Findings.Ledger/EfCore/Context/FindingsLedgerDbContext.cs`** (line 14):
|
||||
- Change the fallback from `"public"` to `"findings"`:
|
||||
```csharp
|
||||
_schemaName = string.IsNullOrWhiteSpace(schemaName)
|
||||
? "findings"
|
||||
: schemaName.Trim();
|
||||
```
|
||||
|
||||
3. **All 12 migration SQL files** in `src/Findings/StellaOps.Findings.Ledger/migrations/`:
|
||||
- Prepend `CREATE SCHEMA IF NOT EXISTS findings;` to `001_initial.sql` (before `BEGIN;` or as first statement inside the transaction).
|
||||
- For `001_initial.sql`: prefix all `CREATE TABLE`, `CREATE INDEX`, `PARTITION OF` statements with `findings.` schema qualifier. Tables: `ledger_events`, `ledger_events_default`, `ledger_merkle_roots`, `ledger_merkle_roots_default`, `findings_projection`, `findings_projection_default`, `finding_history`, `finding_history_default`, `triage_actions`, `triage_actions_default`.
|
||||
- Move the two `CREATE TYPE` statements into the `findings` schema: `CREATE TYPE findings.ledger_event_type ...`, `CREATE TYPE findings.ledger_action_type ...`.
|
||||
- For `002_*` through `009_*`: qualify all table references with `findings.` prefix. Currently these use unqualified table names (e.g., `ALTER TABLE ledger_events` becomes `ALTER TABLE findings.ledger_events`).
|
||||
- For `007_enable_rls.sql`: the `findings_ledger_app` schema for RLS functions is already namespaced and fine. Just qualify the table references in `ALTER TABLE` and `CREATE POLICY` statements.
|
||||
- Set `search_path` at the top of each migration: `SET search_path TO findings, public;` so that type references resolve correctly.
|
||||
|
||||
4. **`src/Platform/__Libraries/StellaOps.Platform.Database/MigrationModulePlugins.cs`** (line 285):
|
||||
- Change `schemaName: "public"` to `schemaName: "findings"` in `FindingsLedgerMigrationModulePlugin`.
|
||||
|
||||
5. **Regenerate the EF Core compiled model** (if the project uses `dotnet ef dbcontext optimize`):
|
||||
- The compiled model in `src/Findings/StellaOps.Findings.Ledger/EfCore/CompiledModels/` may need regeneration if it bakes in schema names. Current inspection shows it delegates to `OnModelCreating`, so it should pick up the change automatically. Verify by building.
|
||||
|
||||
6. **Update tests**: The `MigrationModuleRegistryTests.cs` assertion for FindingsLedger should now expect `schemaName == "findings"`. Add an explicit assertion:
|
||||
```csharp
|
||||
Assert.Contains(modules, m => m.Name == "FindingsLedger" && m.SchemaName == "findings");
|
||||
```
|
||||
|
||||
**Hardcoded `public.` SQL queries:** Grep confirms zero hardcoded `public.` prefixed SQL in the Findings codebase. All repository code passes `FindingsLedgerDbContextFactory.DefaultSchemaName` to the factory, so changing the constant propagates everywhere.
|
||||
|
||||
**Impact on RLS:** The `findings_ledger_app` schema for RLS helper functions already has its own namespace and will not collide. The `ALTER TABLE` statements in `007_enable_rls.sql` just need the `findings.` prefix.
|
||||
|
||||
Completion criteria:
|
||||
- [ ] `FindingsLedgerDbContextFactory.DefaultSchemaName` == `"findings"`
|
||||
- [ ] `FindingsLedgerDbContext` constructor default == `"findings"`
|
||||
- [ ] `FindingsLedgerMigrationModulePlugin.schemaName` == `"findings"`
|
||||
- [ ] All 12 migration SQL files use `findings.` qualified table names
|
||||
- [ ] `001_initial.sql` includes `CREATE SCHEMA IF NOT EXISTS findings;`
|
||||
- [ ] ENUM types created in `findings` schema
|
||||
- [ ] Fresh DB: `stella system migrate FindingsLedger` creates tables under `findings` schema
|
||||
- [ ] All FindingsLedger tests pass
|
||||
- [ ] MigrationModuleRegistryTests updated to assert `findings` schema
|
||||
|
||||
---
|
||||
|
||||
### V4-01 - Register RiskEngine with MigrationModuleRegistry (HIGH priority)
|
||||
|
||||
Status: DONE
|
||||
Dependency: none
|
||||
Owners: Developer (backend)
|
||||
Task description:
|
||||
|
||||
`PostgresRiskScoreResultStore` in `src/Findings/__Libraries/StellaOps.RiskEngine.Infrastructure/Stores/` self-creates the `riskengine` schema and `riskengine.risk_score_results` table via inline `EnsureTableAsync()` (lines 130-164). This bypasses the migration registry entirely.
|
||||
|
||||
**Steps:**
|
||||
1. Create a migration SQL file: `src/Findings/__Libraries/StellaOps.RiskEngine.Infrastructure/Migrations/001_initial_schema.sql` with the DDL currently inline in `EnsureTableAsync()`.
|
||||
2. Mark the SQL file as an embedded resource in the `.csproj`.
|
||||
3. Add `RiskEngineMigrationModulePlugin` to `MigrationModulePlugins.cs`:
|
||||
```csharp
|
||||
public sealed class RiskEngineMigrationModulePlugin : IMigrationModulePlugin
|
||||
{
|
||||
public MigrationModuleInfo Module { get; } = new(
|
||||
name: "RiskEngine",
|
||||
schemaName: "riskengine",
|
||||
migrationsAssembly: typeof(PostgresRiskScoreResultStore).Assembly);
|
||||
}
|
||||
```
|
||||
4. Remove the `EnsureTableAsync()` and `EnsureTable()` methods and the `_initGate`/`_tableInitialized` fields from `PostgresRiskScoreResultStore`. Remove all calls to these methods.
|
||||
5. Update test assertion: `MigrationCommandHandlersTests` expects 28 modules -- bump to 36 (all V4 sprint plugins added).
|
||||
6. Add `using StellaOps.RiskEngine.Infrastructure.Stores;` to `MigrationModulePlugins.cs`.
|
||||
|
||||
Completion criteria:
|
||||
- [x] `riskengine` schema created by migration runner, not inline code
|
||||
- [x] `EnsureTable*` methods removed from `PostgresRiskScoreResultStore`
|
||||
- [x] `RiskEngineMigrationModulePlugin` registered and discoverable
|
||||
- [x] `stella system migrate RiskEngine` works
|
||||
- [x] Build passes, existing RiskEngine tests pass
|
||||
|
||||
---
|
||||
|
||||
### V4-02 - Register Replay with MigrationModuleRegistry (MEDIUM priority)
|
||||
|
||||
Status: DONE
|
||||
Dependency: none
|
||||
Owners: Developer (backend)
|
||||
Task description:
|
||||
|
||||
`ReplayFeedSnapshotStores.cs` in `src/Replay/StellaOps.Replay.WebService/` self-creates the `replay` schema and `replay.feed_snapshot_index` table via inline `EnsureTableAsync()` (line 152).
|
||||
|
||||
**Steps:**
|
||||
1. Create `src/Replay/StellaOps.Replay.WebService/Migrations/001_initial_schema.sql` with the DDL.
|
||||
2. Embed as resource in `.csproj`.
|
||||
3. Add `ReplayMigrationModulePlugin` to `MigrationModulePlugins.cs` (schema: `replay`).
|
||||
4. Remove inline `EnsureTableAsync()` from `ReplayFeedSnapshotStores.cs`.
|
||||
5. Add the `using` for the Replay assembly type to `MigrationModulePlugins.cs`.
|
||||
6. Update module count in test.
|
||||
|
||||
Completion criteria:
|
||||
- [x] `replay` schema created by migration runner
|
||||
- [x] Inline DDL removed
|
||||
- [x] Plugin registered
|
||||
|
||||
---
|
||||
|
||||
### V4-03 - Register ExportCenter with MigrationModuleRegistry (MEDIUM priority)
|
||||
|
||||
Status: DONE
|
||||
Dependency: none
|
||||
Owners: Developer (backend)
|
||||
Task description:
|
||||
|
||||
`ExportCenterMigrationRunner` in `src/ExportCenter/StellaOps.ExportCenter/StellaOps.ExportCenter.Infrastructure/Db/` runs its own migration system with a custom `export_center.export_schema_version` table and `EnsureSchemaAsync()`. It has proper SQL migration files but uses a standalone runner instead of the central one.
|
||||
|
||||
**Steps:**
|
||||
1. The SQL migrations already exist under `.../Db/Migrations/`. Verify they are embedded resources.
|
||||
2. Add `ExportCenterMigrationModulePlugin` to `MigrationModulePlugins.cs` (schema: `export_center`).
|
||||
3. Keep the `ExportCenterMigrationRunner` temporarily (it has checksum validation) but ensure the central runner can also apply these migrations. Long-term, converge to central runner only.
|
||||
4. Add the `using` for the assembly type.
|
||||
5. Update module count.
|
||||
|
||||
Completion criteria:
|
||||
- [x] `ExportCenterMigrationModulePlugin` registered
|
||||
- [x] Central migration runner can discover and apply ExportCenter migrations
|
||||
- [x] Existing ExportCenter functionality unaffected
|
||||
|
||||
---
|
||||
|
||||
### V4-04 - Register Integrations with MigrationModuleRegistry (MEDIUM priority)
|
||||
|
||||
Status: DONE
|
||||
Dependency: none
|
||||
Owners: Developer (backend)
|
||||
Task description:
|
||||
|
||||
`src/Integrations/__Libraries/StellaOps.Integrations.Persistence/Migrations/001_initial_schema.sql` creates `integrations` schema but has no `IMigrationModulePlugin` registered.
|
||||
|
||||
**Steps:**
|
||||
1. Verify migration SQL is an embedded resource.
|
||||
2. Add `IntegrationsMigrationModulePlugin` to `MigrationModulePlugins.cs` (schema: `integrations`).
|
||||
3. Add `using` for the persistence assembly type.
|
||||
4. Update module count.
|
||||
|
||||
Completion criteria:
|
||||
- [x] `IntegrationsMigrationModulePlugin` registered and discoverable
|
||||
- [x] `stella system migrate Integrations` works
|
||||
|
||||
---
|
||||
|
||||
### V4-05 - Register Signer (KeyManagement) with MigrationModuleRegistry (MEDIUM priority)
|
||||
|
||||
Status: DONE
|
||||
Dependency: none
|
||||
Owners: Developer (backend)
|
||||
Task description:
|
||||
|
||||
`src/Attestor/__Libraries/StellaOps.Signer.KeyManagement/Migrations/001_initial_schema.sql` creates `signer` schema. The `Attestor` module plugin is registered with schema `proofchain`, but the `signer` schema is a separate concern managed by a different library.
|
||||
|
||||
**Steps:**
|
||||
1. Verify migration SQL is an embedded resource.
|
||||
2. Add `SignerMigrationModulePlugin` to `MigrationModulePlugins.cs` (schema: `signer`).
|
||||
3. Add `using` for `StellaOps.Signer.KeyManagement` assembly type.
|
||||
4. Update module count.
|
||||
|
||||
Completion criteria:
|
||||
- [x] `SignerMigrationModulePlugin` registered
|
||||
- [x] `signer` schema created by central runner
|
||||
|
||||
---
|
||||
|
||||
### V4-06 - Register IssuerDirectory with MigrationModuleRegistry (MEDIUM priority)
|
||||
|
||||
Status: DONE
|
||||
Dependency: none
|
||||
Owners: Developer (backend)
|
||||
Task description:
|
||||
|
||||
`src/Authority/__Libraries/StellaOps.IssuerDirectory.Persistence/Migrations/001_initial_schema.sql` creates `issuer` schema. The `Authority` module plugin is registered with schema `authority`, but `issuer` is separate.
|
||||
|
||||
**Steps:**
|
||||
1. Verify migration SQL is an embedded resource.
|
||||
2. Add `IssuerDirectoryMigrationModulePlugin` to `MigrationModulePlugins.cs` (schema: `issuer`).
|
||||
3. Add `using` for `StellaOps.IssuerDirectory.Persistence` assembly type.
|
||||
4. Update module count.
|
||||
|
||||
Completion criteria:
|
||||
- [x] `IssuerDirectoryMigrationModulePlugin` registered
|
||||
- [x] `issuer` schema created by central runner
|
||||
|
||||
---
|
||||
|
||||
### V4-07 - Register Workflow with MigrationModuleRegistry (MEDIUM priority)
|
||||
|
||||
Status: DONE
|
||||
Dependency: none
|
||||
Owners: Developer (backend)
|
||||
Task description:
|
||||
|
||||
`src/Workflow/__Libraries/StellaOps.Workflow.DataStore.PostgreSQL/Migrations/001_initial_schema.sql` creates `workflow` schema but has no plugin.
|
||||
|
||||
**Steps:**
|
||||
1. Verify migration SQL is an embedded resource.
|
||||
2. Add `WorkflowMigrationModulePlugin` to `MigrationModulePlugins.cs` (schema: `workflow`).
|
||||
3. Add `using` for the Workflow persistence assembly type.
|
||||
4. Update module count.
|
||||
|
||||
Completion criteria:
|
||||
- [x] `WorkflowMigrationModulePlugin` registered
|
||||
- [x] `workflow` schema created by central runner
|
||||
|
||||
---
|
||||
|
||||
### V4-08 - Register PacksRegistry with MigrationModuleRegistry (MEDIUM priority)
|
||||
|
||||
Status: DONE
|
||||
Dependency: none
|
||||
Owners: Developer (backend)
|
||||
Task description:
|
||||
|
||||
PacksRegistry repositories in `src/JobEngine/StellaOps.PacksRegistry.__Libraries/StellaOps.PacksRegistry.Persistence/Postgres/Repositories/` (6 files) all self-create the `packs` schema via `EnsureTableAsync()`. There is also a migration file `src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Infrastructure/migrations/009_packs_registry.sql` that creates this schema.
|
||||
|
||||
**Steps:**
|
||||
1. Consolidate the `packs` schema DDL into a proper migration file under the PacksRegistry persistence library.
|
||||
2. Embed as resource.
|
||||
3. Add `PacksRegistryMigrationModulePlugin` to `MigrationModulePlugins.cs` (schema: `packs`).
|
||||
4. Remove all 6 `EnsureTableAsync()` methods and `_tableInitialized` fields from the repository classes.
|
||||
5. Update module count.
|
||||
|
||||
Completion criteria:
|
||||
- [x] `packs` schema created by migration runner
|
||||
- [x] All 6 inline `EnsureTable*` methods removed
|
||||
- [x] `PacksRegistryMigrationModulePlugin` registered
|
||||
|
||||
---
|
||||
|
||||
### V4-09 - Register OpsMemory with MigrationModuleRegistry (LOW priority)
|
||||
|
||||
Status: DONE
|
||||
Dependency: none
|
||||
Owners: Developer (backend)
|
||||
Task description:
|
||||
|
||||
OpsMemory uses the `opsmemory` schema (referenced in `PostgresOpsMemoryStore.cs` queries like `INSERT INTO opsmemory.decisions`). Its migration SQL lives outside the module at `devops/database/migrations/V20260108__opsmemory_advisoryai_schema.sql` -- a legacy location that the central runner does not discover.
|
||||
|
||||
**Steps:**
|
||||
1. Move/copy the migration SQL into the OpsMemory library as an embedded resource.
|
||||
2. Add `OpsMemoryMigrationModulePlugin` to `MigrationModulePlugins.cs` (schema: `opsmemory`).
|
||||
3. Add `using` for the OpsMemory assembly type.
|
||||
4. Update test fixtures that currently load migration SQL from the filesystem path.
|
||||
5. Update module count.
|
||||
|
||||
Completion criteria:
|
||||
- [x] `opsmemory` schema created by central migration runner
|
||||
- [x] Legacy devops migration file no longer the only source of truth
|
||||
- [x] Test fixtures updated
|
||||
|
||||
---
|
||||
|
||||
### V4-10 - Audit and remove remaining inline EnsureTable patterns (LOW priority)
|
||||
|
||||
Status: DONE
|
||||
Dependency: V4-01 through V4-08
|
||||
Owners: Developer (backend)
|
||||
Task description:
|
||||
|
||||
After the above tasks, audit remaining `EnsureTable` callers that may not have been addressed:
|
||||
|
||||
**Known remaining EnsureTable callers (may already be covered by registered modules):**
|
||||
- `src/Signals/__Libraries/StellaOps.Signals.Persistence/Postgres/Repositories/` (6 files) -- Signals IS registered. Remove inline `CREATE SCHEMA IF NOT EXISTS signals;` from these repositories since the central runner handles schema creation.
|
||||
- `src/AirGap/__Libraries/StellaOps.AirGap.Persistence/Postgres/Repositories/` (4 files) -- AirGap IS registered. Remove inline schema creation.
|
||||
- `src/SbomService/__Libraries/StellaOps.SbomService.Persistence/Postgres/Repositories/` (8 files) -- SbomLineage IS registered. Remove inline `CREATE SCHEMA IF NOT EXISTS sbom;`.
|
||||
- `src/Router/__Libraries/StellaOps.Messaging.Transport.Postgres/` (2 files) -- uses dynamic schema from `_connectionFactory.Schema`. Evaluate if this needs registration or is intentionally dynamic.
|
||||
- `src/__Libraries/StellaOps.HybridLogicalClock/PostgresHlcStateStore.cs` -- uses configurable `_schema`. Evaluate.
|
||||
- `src/Concelier/StellaOps.Excititor.WebService/Services/PostgresGraphOverlayStore.cs` -- Excititor IS registered. Remove inline DDL.
|
||||
- `src/AdvisoryAI/StellaOps.AdvisoryAI/KnowledgeSearch/PostgresKnowledgeSearchStore.cs` -- AdvisoryAI IS registered. Remove inline `CREATE SCHEMA IF NOT EXISTS advisoryai;`.
|
||||
- `src/BinaryIndex/__Libraries/StellaOps.BinaryIndex.Persistence/BinaryIndexMigrationRunner.cs` -- BinaryIndex IS registered. Remove inline schema creation.
|
||||
- `src/Attestor/StellaOps.Attestor/StellaOps.Attestor.Storage/Rekor/PostgresRekorCheckpointStore.cs` -- creates `attestor` schema inline. Evaluate if this should be a separate plugin or folded into Attestor plugin.
|
||||
|
||||
For each: remove the inline `CREATE SCHEMA IF NOT EXISTS` since the central migration runner now owns schema creation. Keep `CREATE TABLE IF NOT EXISTS` as a defensive fallback only if there is a race condition risk; otherwise remove entirely.
|
||||
|
||||
Completion criteria:
|
||||
- [x] All inline `CREATE SCHEMA IF NOT EXISTS` in registered modules removed
|
||||
- [x] No `EnsureTable` patterns that duplicate central migration runner work
|
||||
- [x] Build and all tests pass
|
||||
|
||||
---
|
||||
|
||||
### V4-11 - Update module count test and registry documentation (CLEANUP)
|
||||
|
||||
Status: DONE
|
||||
Dependency: V4-01 through V4-09
|
||||
Owners: Developer (backend)
|
||||
Task description:
|
||||
|
||||
After all new plugins are registered:
|
||||
1. Update `MigrationCommandHandlersTests.Registry_Has_All_Modules()` -- currently asserts `28`. New count = 28 + N new plugins (RiskEngine, Replay, ExportCenter, Integrations, Signer, IssuerDirectory, Workflow, PacksRegistry, OpsMemory = 9). New expected count: **37**.
|
||||
2. Update `MigrationModuleRegistryTests.Modules_Populated_With_All_Postgres_Modules()` -- add assertions for all new modules.
|
||||
3. Update `SystemCommandBuilderTests` if it has a hardcoded module name list.
|
||||
|
||||
Completion criteria:
|
||||
- [x] All test assertions reflect the new module count (36 plugins; MigrationCommandHandlersTests already asserts 36; MigrationModuleRegistryTests already has assertions for all 36 modules)
|
||||
- [x] `stella system migrate --list` shows all modules
|
||||
- [x] No test failures (pre-existing Signer assembly reference issue in CLI test project is unrelated to V4-10/V4-11)
|
||||
|
||||
---
|
||||
|
||||
## Execution Log
|
||||
| Date (UTC) | Update | Owner |
|
||||
| --- | --- | --- |
|
||||
| 2026-04-08 | Sprint created with detailed task analysis for Violations 3 and 4. | Planning |
|
||||
| 2026-04-08 | V4-01 through V4-04 implemented: RiskEngine, Replay, ExportCenter, Integrations registered with MigrationModuleRegistry. Inline EnsureTable removed from RiskEngine and Replay. Test count updated to 36. All builds pass. | Developer |
|
||||
| 2026-04-08 | V3-01 DONE: Changed FindingsLedger schema from `public` to `findings` across factory, DbContext, migration plugin, all 12 SQL migrations (schema-qualified tables/types/indexes, CREATE SCHEMA, SET search_path), and added test assertion. Build verified. | Developer |
|
||||
| 2026-04-08 | V4-05 through V4-09 DONE: Registered Signer, IssuerDirectory, Workflow, PacksRegistry, OpsMemory with MigrationModuleRegistry. Created consolidated migration SQL for PacksRegistry (from 009_packs_registry.sql + 6 inline EnsureTable DDLs). Copied OpsMemory DDL from devops/ to library. Removed all 6 EnsureTable methods from PacksRegistry repositories. Added EmbeddedResource to PacksRegistry and OpsMemory csproj files. All builds pass (0 warnings, 0 errors). | Developer |
|
||||
| 2026-04-08 | V4-10 DONE: Removed redundant inline `CREATE SCHEMA IF NOT EXISTS` from 16 files across registered modules: Signals (6 repos), SbomService (8 repos), AdvisoryAI (KnowledgeSearchStore), BinaryIndex (MigrationRunner), Attestor (RekorCheckpointStore). AirGap EnsureTable methods only check table existence (no schema creation) -- already clean. Concelier Excititor only has `CREATE TABLE IF NOT EXISTS` -- already clean. Router, HLC, ExportCenter, PluginRegistry kept as-is (dynamic/standalone). All 5 affected modules build with 0 errors. | Developer |
|
||||
| 2026-04-08 | V4-11 DONE: Test assertions already at 36 (updated by V4-01 through V4-09 work). MigrationCommandHandlersTests asserts 36, MigrationModuleRegistryTests has per-module assertions for all 36 plugins. No changes needed. | Developer |
|
||||
|
||||
## Decisions & Risks
|
||||
- **Fresh DB only**: All changes assume fresh DB setup (volume delete + rebuild). No online migration path needed for existing deployments since we are pre-GA.
|
||||
- **Compiled model (V3-01)**: The EF Core compiled model delegates schema to `OnModelCreating`, so changing `DefaultSchemaName` propagates automatically. If the compiled model bakes in schema names at generation time, it must be regenerated. Verify by building and running.
|
||||
- **ENUM types in schema (V3-01)**: PostgreSQL ENUMs cannot be easily moved between schemas. Since we are on fresh DB, we create them in the `findings` schema from the start. The `search_path` must include `findings` for queries that reference enum values without schema qualification.
|
||||
- **Dual migration runners (V4-03)**: ExportCenter has its own runner with checksum validation. Registering with the central runner means migrations run via both paths. Short-term this is fine (idempotent SQL). Long-term, deprecate the standalone runner.
|
||||
- **Dynamic schemas (V4-10)**: Router messaging and HLC use configurable schemas. These are intentionally dynamic and may not need registry entries. Evaluate during implementation.
|
||||
- **scripts schema (Scheduler)**: The `scripts` schema is created by `004_create_scripts_schema.sql` inside the Scheduler persistence library, which IS registered. No separate plugin needed -- it is already covered.
|
||||
|
||||
## Next Checkpoints
|
||||
- V3-01 + V4-01 through V4-09 complete: all schemas governed by MigrationModuleRegistry.
|
||||
- V4-10 complete: no inline schema creation duplicates central runner.
|
||||
- V4-11 complete: test coverage confirms full registry.
|
||||
- Final: fresh DB `docker compose down -v && docker compose up` boots with all schemas created by central runner.
|
||||
287
docs/implplan/SPRINT_20260408_004_Timeline_unified_audit_sink.md
Normal file
287
docs/implplan/SPRINT_20260408_004_Timeline_unified_audit_sink.md
Normal file
@@ -0,0 +1,287 @@
|
||||
# Sprint 20260408-004 -- Unified Audit Sink
|
||||
|
||||
## Topic & Scope
|
||||
|
||||
- **Consolidate the fragmented audit landscape** into a single, persistent, hash-chained audit store fronted by the Timeline service.
|
||||
- Today every service owns its own audit implementation; the Timeline service aggregates by polling each service at query time with a 2-second timeout. This is fragile, lossy, and cannot support compliance retention or chain integrity.
|
||||
- The goal is: every service emits audit events to the Timeline ingest endpoint (push model), Timeline persists them in a dedicated `audit.events` PostgreSQL table with SHA-256 hash chaining, and the existing `HttpUnifiedAuditEventProvider` polling path becomes a transitional fallback, not the primary data source.
|
||||
- Working directory: `src/Timeline/`, `src/__Libraries/StellaOps.Audit.Emission/`, cross-module `Program.cs` wiring.
|
||||
- Expected evidence: passing integration tests, all services emitting to Timeline, hash chain verification, GDPR compliance docs.
|
||||
|
||||
## Current State Analysis
|
||||
|
||||
### Per-Service Audit Implementations Found
|
||||
|
||||
| Service | Storage | Schema/Table | Hash Chain | PII | Retention | API Endpoint |
|
||||
|---|---|---|---|---|---|---|
|
||||
| **Authority** | PostgreSQL (EF Core) | `authority.audit` (BIGSERIAL, tenant_id, user_id, action, resource_type, resource_id, old_value, new_value, ip_address, user_agent, correlation_id, created_at) | No | **Yes**: user_id (UUID), ip_address, user_agent | None | `/console/admin/audit` |
|
||||
| **Authority Airgap** | PostgreSQL | `authority.airgap_audit` | No | Yes: ip_address | None | `/authority/audit/airgap` |
|
||||
| **Authority Offline Kit** | PostgreSQL | `authority.offline_kit_audit` | No | No | None | Implicit via authority |
|
||||
| **IssuerDirectory** | PostgreSQL (EF Core) | `issuer_directory.audit` (EF entity) | No | No | None | Internal only |
|
||||
| **JobEngine/ReleaseOrchestrator** | PostgreSQL (EF Core) | `audit_entries` with `AuditSequenceEntity` | **Yes**: SHA-256 content hash + previous entry hash + sequence numbers | Yes: actor_id, actor_ip, user_agent | None | `/api/v1/release-orchestrator/audit` (list, get, resource history, sequence range, summary, verify chain) |
|
||||
| **Scheduler** | PostgreSQL | `scheduler.audit` (PARTITIONED monthly by created_at) | No | Yes: user_id | **Partial**: monthly partitioning enables drop-partition retention | Per-script audit |
|
||||
| **Policy** | PostgreSQL | `policy.audit` (via governance endpoints) | No | No | None | `/api/v1/governance/audit/events` |
|
||||
| **Notify** | PostgreSQL | `notify.audit` | No | Yes: user_id | None | `/api/v1/notify/audit` |
|
||||
| **EvidenceLocker** | **Hardcoded mock data** | None (returns 3 static events) | No | No | N/A | `/api/v1/evidence/audit` |
|
||||
| **Attestor ProofChain** | PostgreSQL | `proofchain.audit_log` | No (but proofs themselves are hash-chained) | No | None | Internal only |
|
||||
| **BinaryIndex GoldenSet** | PostgreSQL (EF Core) | `GoldenSetAuditLogEntity` | No | No | None | Internal only |
|
||||
| **Graph** | **In-memory** (`LinkedList`, max 500) | None | No | No | Volatile (lost on restart) | Internal only |
|
||||
| **Concelier** | **ILogger only** (`JobAuthorizationAuditFilter`) | None | No | Yes: remote IP | Volatile (log rotation) | None |
|
||||
| **EvidenceLocker WebService** | **ILogger only** (`EvidenceAuditLogger`) | None | No | Yes: subject, clientId, scopes | Volatile (log rotation) | None |
|
||||
| **AdvisoryAI** | In-memory (`IActionAuditLedger`) + ILogger | `ActionAuditEntry` (in-memory) | No | Yes: actor | Volatile | Internal |
|
||||
| **Cryptography (KeyEscrow)** | `IKeyEscrowAuditLogger` interface | Implementation-dependent | No | Yes: key operations | Implementation-dependent | Internal |
|
||||
| **Signer** | In-memory (`InMemorySignerAuditSink`) | `CeremonyAuditEvents` | No | No | Volatile | Internal |
|
||||
|
||||
### Existing Unified Audit Infrastructure
|
||||
|
||||
**StellaOps.Audit.Emission** (shared library, `src/__Libraries/StellaOps.Audit.Emission/`):
|
||||
- Fully implemented: `IAuditEventEmitter`, `HttpAuditEventEmitter`, `AuditActionFilter`, `AuditActionAttribute`, `AuditEmissionOptions`, `AuditEmissionServiceExtensions`
|
||||
- Posts events as JSON to `POST /api/v1/audit/ingest` on Timeline service
|
||||
- Fire-and-forget pattern: never blocks the calling endpoint
|
||||
- Configuration: `AuditEmission:TimelineBaseUrl`, `AuditEmission:Enabled`, `AuditEmission:TimeoutSeconds` (default 3s)
|
||||
- **CRITICAL: Never wired in any service's Program.cs** -- `AddAuditEmission()` is called exactly zero times across the codebase
|
||||
|
||||
**Timeline Ingest Endpoint** (`src/Timeline/StellaOps.Timeline.WebService/Endpoints/UnifiedAuditEndpoints.cs`):
|
||||
- `POST /api/v1/audit/ingest` exists and works
|
||||
- Stores events in `IngestAuditEventStore` -- a `ConcurrentQueue<UnifiedAuditEvent>` capped at 10,000 events
|
||||
- **CRITICAL: In-memory only, lost on restart, no PostgreSQL persistence**
|
||||
|
||||
**Timeline Aggregation** (`CompositeUnifiedAuditEventProvider`):
|
||||
- Merges HTTP-polled events from 5 services (Authority, JobEngine, Policy, EvidenceLocker, Notify) with ingested events
|
||||
- Polling uses `HttpUnifiedAuditEventProvider` with 2-second timeout per module
|
||||
- Missing from polling: Scheduler, Scanner, Attestor, SBOM, Integrations, Graph, Concelier, AdvisoryAI, Cryptography, BinaryIndex
|
||||
|
||||
**StellaOps.Audit.ReplayToken** (shared library):
|
||||
- SHA-256-based replay tokens for deterministic replay verification
|
||||
- Used by Replay service for verdict replay attestation
|
||||
- Separate concern from audit logging (provenance, not audit)
|
||||
|
||||
**StellaOps.AuditPack** (shared library):
|
||||
- Bundle manifests for audit export packages
|
||||
- Used by ExportCenter for compliance audit bundle generation
|
||||
- Separate concern (export packaging, not event capture)
|
||||
|
||||
### UI Audit Surface
|
||||
|
||||
- **Audit Dashboard** at `/ops/operations/audit` with tabs: Overview, All Events, Timeline, Correlations, Exports, Bundles
|
||||
- `AuditLogClient` hits `/api/v1/audit/events` (unified), `/api/v1/audit/stats`, `/api/v1/audit/timeline/search`, `/api/v1/audit/correlations`, `/api/v1/audit/anomalies`, `/api/v1/audit/export`
|
||||
- Fallback: `getUnifiedEventsFromModules()` hits each module's audit endpoint directly if unified fails
|
||||
- Module-specific endpoints listed in client: authority, policy, jobengine, integrations, vex, scanner, attestor, sbom, scheduler (many return 404 today)
|
||||
|
||||
### Doctor Health Check
|
||||
|
||||
- `AuditReadinessCheck` in `StellaOps.Doctor.Plugin.Compliance` checks EvidenceLocker's `/api/v1/evidence/audit-readiness` endpoint (which does not exist yet)
|
||||
- Checks: retention policy configured, audit log enabled, backup verified
|
||||
|
||||
### GDPR/PII Analysis
|
||||
|
||||
PII found in audit records:
|
||||
1. **Authority**: `user_id` (UUID), `ip_address`, `user_agent`, username, display_name, email (in `ClassifiedString` with classification: personal/sensitive/none)
|
||||
2. **JobEngine**: `actor_id`, `actor_ip`, `user_agent`
|
||||
3. **Scheduler**: `user_id`
|
||||
4. **Notify**: `user_id`
|
||||
5. **EvidenceLocker logger**: subject claim, client ID
|
||||
6. **Concelier logger**: remote IP address
|
||||
7. **AdvisoryAI**: actor (username)
|
||||
|
||||
**No retention policies exist anywhere.** The Authority `ClassifiedString` pattern is the only data classification mechanism, and it only applies to structured logging scope, not to database records.
|
||||
|
||||
### Event Sourcing vs. Audit Distinction
|
||||
|
||||
| System | Purpose | Audit? |
|
||||
|---|---|---|
|
||||
| **Attestor ProofChain** | Cryptographic evidence chain (DSSE, Rekor) | **Provenance**, not audit. Must remain separate. |
|
||||
| **Attestor Verdict Ledger** | Append-only SHA-256 hash-chained release verdicts | **Provenance**. Hash chain is for tamper-evidence of decisions, not operator activity. |
|
||||
| **Findings Ledger** | Alert state machine transitions | **Event sourcing** for domain state. Not audit. |
|
||||
| **Timeline events** (Concelier, ExportCenter, Findings, etc.) | Activity timeline for UI display | **Operational telemetry**. Related but different from audit. |
|
||||
| **AuditPack / ExportCenter** | Compliance bundle packaging | **Export format** for audit data. Consumer of audit, not a source. |
|
||||
|
||||
## Dependencies & Concurrency
|
||||
|
||||
- Upstream: No blockers. Timeline service already exists and has the ingest endpoint.
|
||||
- Safe parallelism: Phase 1 (persistence) can run independently. Phase 2 (service wiring) can be parallelized across services. Phase 3 (retention/GDPR) can run after Phase 1.
|
||||
- Dependency on Orchestrator Decomposition (Sprint 20260406): JobEngine audit is the most mature implementation. Its hash-chain pattern should be the model for the unified store.
|
||||
|
||||
## Documentation Prerequisites
|
||||
|
||||
- `docs/modules/jobengine/architecture.md` -- for hash-chain audit pattern
|
||||
- `docs/technical/architecture/webservice-catalog.md` -- for service inventory
|
||||
|
||||
## Delivery Tracker
|
||||
|
||||
### AUDIT-001 - PostgreSQL persistence for Timeline audit ingest
|
||||
Status: TODO
|
||||
Dependency: none
|
||||
Owners: Developer (backend)
|
||||
Task description:
|
||||
- Replace `IngestAuditEventStore` (in-memory ConcurrentQueue) with a PostgreSQL-backed store in the Timeline service.
|
||||
- Create `audit.events` table schema: id (UUID), tenant_id, timestamp, module, action, severity, actor_id, actor_name, actor_email, actor_type, actor_ip, actor_user_agent, resource_type, resource_id, resource_name, description, details_json, diff_json, correlation_id, parent_event_id, tags (text[]), content_hash (SHA-256), previous_hash (SHA-256), sequence_number (BIGINT), created_at.
|
||||
- Implement hash chaining: each event's `content_hash` is computed from canonical JSON of its fields; `previous_hash` links to the prior event's `content_hash`; `sequence_number` is monotonically increasing per tenant.
|
||||
- Add SQL migration file as embedded resource in Timeline persistence assembly.
|
||||
- Ensure auto-migration on startup per project rules (section 2.7).
|
||||
- Add `VerifyChainAsync()` method for integrity verification.
|
||||
- Update `CompositeUnifiedAuditEventProvider` to read from the persistent store as primary, falling back to HTTP polling for events not yet in the store.
|
||||
|
||||
Completion criteria:
|
||||
- [ ] `audit.events` table created via auto-migration
|
||||
- [ ] Ingested events survive Timeline service restart
|
||||
- [ ] Hash chain verification passes for all stored events
|
||||
- [ ] Integration test for ingest -> persist -> query round-trip
|
||||
- [ ] Integration test for hash chain verification (valid + tampered)
|
||||
|
||||
### AUDIT-002 - Wire Audit.Emission in all HTTP services
|
||||
Status: TODO
|
||||
Dependency: AUDIT-001
|
||||
Owners: Developer (backend)
|
||||
Task description:
|
||||
- Call `builder.Services.AddAuditEmission(builder.Configuration)` in each service's `Program.cs`.
|
||||
- Apply `AuditActionFilter` + `AuditActionAttribute` to all write endpoints (POST, PUT, PATCH, DELETE).
|
||||
- Services to wire (in priority order):
|
||||
1. Authority (highest PII risk)
|
||||
2. ReleaseOrchestrator/JobEngine (most critical business operations)
|
||||
3. Policy (governance decisions)
|
||||
4. Notify
|
||||
5. Scanner
|
||||
6. Concelier/Excititor (VEX)
|
||||
7. Integrations
|
||||
8. SBOM
|
||||
9. Scheduler
|
||||
10. Attestor
|
||||
11. EvidenceLocker
|
||||
12. Graph
|
||||
13. AdvisoryAI
|
||||
14. BinaryIndex
|
||||
- For services that already have DB-backed audit (Authority, JobEngine, Policy, Notify, Scheduler): emit to Timeline AND keep existing DB audit (dual-write during transition).
|
||||
- For services with ILogger-only audit (EvidenceLocker, Concelier): ILogger audit remains for operational logging; Emission provides structured audit to Timeline.
|
||||
|
||||
Completion criteria:
|
||||
- [ ] `AddAuditEmission()` called in all 14+ service Program.cs files
|
||||
- [ ] At least write endpoints decorated with `AuditActionAttribute`
|
||||
- [ ] Verified events appear in Timeline `/api/v1/audit/events` for each module
|
||||
- [ ] No regressions in service startup time (emission is fire-and-forget)
|
||||
|
||||
### AUDIT-003 - Backfill missing modules in HttpUnifiedAuditEventProvider polling
|
||||
Status: TODO
|
||||
Dependency: none
|
||||
Owners: Developer (backend)
|
||||
Task description:
|
||||
- The `HttpUnifiedAuditEventProvider` currently polls only 5 services (Authority, JobEngine, Policy, EvidenceLocker, Notify). Add polling for: Scanner, Scheduler, Integrations, Attestor, SBOM (if they have audit endpoints).
|
||||
- This is the transitional path: once AUDIT-002 is complete and all services push via Emission, polling becomes optional fallback.
|
||||
- For EvidenceLocker: replace hardcoded mock data with real DB-backed audit (or remove the mock endpoint and rely solely on Emission).
|
||||
|
||||
Completion criteria:
|
||||
- [ ] All services with audit endpoints appear in polling list
|
||||
- [ ] EvidenceLocker mock data replaced or deprecated
|
||||
- [ ] Fallback polling gracefully handles services without audit endpoints
|
||||
|
||||
### AUDIT-004 - GDPR data classification and retention policies
|
||||
Status: TODO
|
||||
Dependency: AUDIT-001
|
||||
Owners: Developer (backend), Documentation author
|
||||
Task description:
|
||||
- Add `data_classification` column to `audit.events` table (enum: none, personal, sensitive, restricted).
|
||||
- Implement automated classification based on module + field content:
|
||||
- `actor.email`, `actor.ipAddress`, `actor.userAgent` -> `personal`
|
||||
- Authority login attempts with usernames -> `sensitive`
|
||||
- Key escrow operations -> `restricted`
|
||||
- All other fields -> `none`
|
||||
- Implement retention policy engine:
|
||||
- Default: 365 days for `none`/`personal` classification
|
||||
- Configurable per-tenant via `platform.environment_settings`
|
||||
- Compliance hold: events linked to an `EvidenceHold` are exempt from retention purge
|
||||
- Scheduled background service to purge expired events (respecting holds)
|
||||
- Extend Authority's `ClassifiedString` pattern to the unified audit schema.
|
||||
- Add right-to-erasure endpoint: `DELETE /api/v1/audit/actors/{actorId}/pii` that redacts PII fields (replaces with `[REDACTED]`) without deleting the event (preserving audit chain integrity by keeping the hash chain intact).
|
||||
|
||||
Completion criteria:
|
||||
- [ ] Data classification applied to all ingested events
|
||||
- [ ] Retention purge runs on schedule without breaking hash chains (gap markers inserted)
|
||||
- [ ] Right-to-erasure redacts PII without invalidating chain verification
|
||||
- [ ] Documentation updated: `docs/modules/timeline/audit-retention.md`
|
||||
- [ ] Doctor `AuditReadinessCheck` updated to verify retention configuration
|
||||
|
||||
### AUDIT-005 - Deprecate per-service audit DB tables (Phase 2)
|
||||
Status: TODO
|
||||
Dependency: AUDIT-002
|
||||
Owners: Developer (backend)
|
||||
Task description:
|
||||
- After AUDIT-002 is stable (all services pushing to Timeline), deprecate the dual-write to per-service audit tables.
|
||||
- Mark per-service audit endpoints as deprecated (add `Obsolete` attribute, log deprecation warning).
|
||||
- Update `HttpUnifiedAuditEventProvider` to stop polling deprecated endpoints.
|
||||
- Do NOT delete the per-service tables yet -- they serve as migration verification targets.
|
||||
- Add migration path documentation for operators upgrading from per-service audit to unified.
|
||||
|
||||
Completion criteria:
|
||||
- [ ] Per-service audit endpoints return deprecation headers
|
||||
- [ ] Timeline is the single source of truth for all audit queries
|
||||
- [ ] No data loss during transition (unified store contains all events from all services)
|
||||
|
||||
### AUDIT-006 - UI updates for new data sources
|
||||
Status: TODO
|
||||
Dependency: AUDIT-002
|
||||
Owners: Developer (frontend)
|
||||
Task description:
|
||||
- Update `AuditLogClient` module list to reflect all modules now emitting to Timeline.
|
||||
- Remove fallback `getUnifiedEventsFromModules()` path once unified endpoint is reliable.
|
||||
- Add data classification badges to audit event display (personal/sensitive/restricted).
|
||||
- Add retention policy display to audit dashboard overview.
|
||||
- Wire `AuditReadinessCheck` results into Doctor compliance dashboard.
|
||||
|
||||
Completion criteria:
|
||||
- [ ] All 11+ modules visible in audit dashboard module filter
|
||||
- [ ] Data classification visible on event detail
|
||||
- [ ] Retention status visible on dashboard overview tab
|
||||
|
||||
### AUDIT-007 - AuditPack export from unified store
|
||||
Status: TODO
|
||||
Dependency: AUDIT-001, AUDIT-002
|
||||
Owners: Developer (backend)
|
||||
Task description:
|
||||
- Update ExportCenter's `AuditBundleJobHandler` to source events from Timeline's unified store instead of polling individual services.
|
||||
- Include hash chain verification proof in exported audit bundles.
|
||||
- Add DSSE signature on audit bundle manifests via Attestor integration.
|
||||
|
||||
Completion criteria:
|
||||
- [ ] Audit bundle export pulls from unified Timeline store
|
||||
- [ ] Bundle includes chain verification certificate
|
||||
- [ ] Bundle manifest is DSSE-signed
|
||||
|
||||
## Execution Log
|
||||
| Date (UTC) | Update | Owner |
|
||||
| --- | --- | --- |
|
||||
| 2026-04-08 | Sprint created from deep audit landscape investigation. Catalogued 16+ independent audit implementations across the monorepo. | Planning |
|
||||
|
||||
## Decisions & Risks
|
||||
|
||||
### Decisions
|
||||
|
||||
1. **Timeline service is the unified audit sink** -- not a new dedicated service. Timeline already has the ingest endpoint, aggregation service, and UI integration. Adding PostgreSQL persistence to Timeline is less disruptive than creating a new service.
|
||||
|
||||
2. **Push model (Emission) is primary, polling is fallback** -- the existing `HttpUnifiedAuditEventProvider` polling path has fundamental problems (2s timeout, in-memory-only ingest store, lossy). The `StellaOps.Audit.Emission` library was designed for this exact purpose but never wired. Wire it.
|
||||
|
||||
3. **Hash chain at the sink, not at the source** -- only JobEngine currently has hash chaining. Rather than retrofitting all 16 services with chain logic, implement chaining once at the Timeline ingest layer. This gives consistent integrity guarantees across all modules.
|
||||
|
||||
4. **Attestor ProofChain and Verdict Ledger are NOT audit** -- they are provenance systems with different integrity guarantees (DSSE signatures, Rekor transparency log). They must remain separate. The unified audit log records the *operational activity* (who did what), while provenance records the *cryptographic evidence* (what was decided and signed).
|
||||
|
||||
5. **Dual-write during transition** -- services that already have DB-backed audit (Authority, JobEngine, Policy, Notify, Scheduler) will write to both their local table AND the unified Timeline store during the transition period. This ensures zero data loss and allows rollback.
|
||||
|
||||
6. **Right-to-erasure via redaction, not deletion** -- GDPR Article 17 allows exemptions for legal compliance. Audit records support legal obligations. PII fields are redacted (replaced with `[REDACTED]`) but the event record and hash chain remain intact. This is standard practice for append-only audit logs.
|
||||
|
||||
### Risks
|
||||
|
||||
1. **IngestAuditEventStore is in-memory** -- any events received before AUDIT-001 ships are lost on Timeline restart. Mitigation: AUDIT-001 is the highest priority task.
|
||||
|
||||
2. **Fire-and-forget emission can lose events** -- the `HttpAuditEventEmitter` swallows all errors. If Timeline is down, events are silently dropped. Future work: add a local buffer (e.g., SQLite WAL) in the Emission library for at-least-once delivery. Not in scope for this sprint but noted as a risk.
|
||||
|
||||
3. **PII in audit records** -- Authority audit contains usernames, emails, IPs. Without AUDIT-004, we have no retention or erasure capability. Risk: GDPR non-compliance for EU deployments.
|
||||
|
||||
4. **Scheduler already has monthly partitioning** -- its retention model (drop partitions) is the most advanced. The unified store should learn from this: consider partitioning `audit.events` by month from day one.
|
||||
|
||||
5. **EvidenceLocker audit is entirely fake** -- returns 3 hardcoded events. Any compliance audit that examines EvidenceLocker data will find fabricated records. AUDIT-002 (wiring Emission) fixes this.
|
||||
|
||||
## Next Checkpoints
|
||||
|
||||
- **Phase 1 (AUDIT-001)**: PostgreSQL persistence for Timeline ingest -- target: 1 week
|
||||
- **Phase 2 (AUDIT-002 + AUDIT-003)**: Wire Emission in all services + backfill polling -- target: 2 weeks
|
||||
- **Phase 3 (AUDIT-004)**: GDPR retention and data classification -- target: 3 weeks
|
||||
- **Phase 4 (AUDIT-005 + AUDIT-006 + AUDIT-007)**: Deprecate per-service, UI updates, export -- target: 4 weeks
|
||||
@@ -301,7 +301,7 @@ sudo stellaops-cli bundle verify /tmp/new-bundle/manifest.json
|
||||
|
||||
# Apply with verification
|
||||
sudo stellaops-cli bundle apply /tmp/new-bundle --verify
|
||||
sudo systemctl restart stellaops-excititor
|
||||
sudo systemctl restart stellaops-excititor-web
|
||||
|
||||
# Rollback if needed
|
||||
# sudo stellaops-cli bundle rollback --to bundles.backup-20250115
|
||||
|
||||
@@ -19,7 +19,7 @@ VexLens can operate in fully air-gapped environments with pre-loaded VEX data an
|
||||
"bundleId": "vexlens-bundle-2025-12-06",
|
||||
"version": "1.0.0",
|
||||
"createdAt": "2025-12-06T00:00:00Z",
|
||||
"createdBy": "stellaops-export",
|
||||
"createdBy": "stellaops-export-web",
|
||||
"checksum": "sha256:abc123...",
|
||||
"components": {
|
||||
"issuerDirectory": {
|
||||
|
||||
Reference in New Issue
Block a user