up
This commit is contained in:
StellaOps Bot
491
docs/db/CONVERSION_PLAN.md
Normal file
491
docs/db/CONVERSION_PLAN.md
Normal file
@@ -0,0 +1,491 @@
|
||||
# MongoDB to PostgreSQL Conversion Plan
|
||||
|
||||
**Version:** 2.0.0
|
||||
**Status:** APPROVED
|
||||
**Created:** 2025-11-28
|
||||
**Last Updated:** 2025-11-28
|
||||
|
||||
---
|
||||
|
||||
## Executive Summary
|
||||
|
||||
This document outlines the strategic plan to **convert** (not migrate) StellaOps from MongoDB to PostgreSQL for control-plane domains. The conversion follows a "strangler fig" pattern, introducing PostgreSQL repositories alongside existing MongoDB implementations and gradually switching each bounded context.
|
||||
|
||||
**Key Finding:** StellaOps already has production-ready PostgreSQL patterns in the Orchestrator and Findings modules that serve as templates for all other modules.
|
||||
|
||||
### Related Documents
|
||||
|
||||
| Document | Purpose |
|
||||
|----------|---------|
|
||||
| [SPECIFICATION.md](./SPECIFICATION.md) | Schema designs, naming conventions, data types |
|
||||
| [RULES.md](./RULES.md) | Database coding rules and patterns |
|
||||
| [VERIFICATION.md](./VERIFICATION.md) | Testing and verification requirements |
|
||||
| [tasks/](./tasks/) | Detailed task definitions per phase |
|
||||
|
||||
---
|
||||
|
||||
## 1. Principles & Scope
|
||||
|
||||
### 1.1 Goals
|
||||
|
||||
Convert **control-plane** domains from MongoDB to PostgreSQL:
|
||||
|
||||
| Domain | Current DB | Target | Priority |
|
||||
|--------|-----------|--------|----------|
|
||||
| Authority | `stellaops_authority` | PostgreSQL | P0 |
|
||||
| Scheduler | `stellaops_scheduler` | PostgreSQL | P0 |
|
||||
| Notify | `stellaops_notify` | PostgreSQL | P1 |
|
||||
| Policy | `stellaops_policy` | PostgreSQL | P1 |
|
||||
| Vulnerabilities (Concelier) | `concelier` | PostgreSQL | P2 |
|
||||
| VEX & Graph (Excititor) | `excititor` | PostgreSQL | P2 |
|
||||
| PacksRegistry | `stellaops_packs` | PostgreSQL | P3 |
|
||||
| IssuerDirectory | `stellaops_issuer` | PostgreSQL | P3 |
|
||||
|
||||
### 1.2 Non-Goals
|
||||
|
||||
- Scanner result storage (remains object storage + Mongo for now)
|
||||
- Real-time event streams (separate infrastructure)
|
||||
- Legacy data archive (can remain in MongoDB read-only)
|
||||
|
||||
### 1.3 Constraints
|
||||
|
||||
**MUST Preserve:**
|
||||
- Deterministic, replayable scans
|
||||
- "Preserve/prune source" rule for Concelier/Excititor
|
||||
- Lattice logic in `Scanner.WebService` (not in DB)
|
||||
- Air-gap friendliness and offline-kit packaging
|
||||
- Multi-tenant isolation patterns
|
||||
- Zero downtime during conversion
|
||||
|
||||
### 1.4 Conversion vs Migration
|
||||
|
||||
This is a **conversion**, not a 1:1 document→row mapping:
|
||||
|
||||
| Approach | When to Use |
|
||||
|----------|-------------|
|
||||
| **Normalize** | Identities, jobs, schedules, relationships |
|
||||
| **Keep JSONB** | Advisory payloads, provenance trails, evidence manifests |
|
||||
| **Drop/Archive** | Ephemeral data (caches, locks), historical logs |
|
||||
|
||||
---
|
||||
|
||||
## 2. Architecture
|
||||
|
||||
### 2.1 Strangler Fig Pattern
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Service Layer │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ Repository Interface │
|
||||
│ (e.g., IScheduleRepository) │
|
||||
├──────────────────────┬──────────────────────────────────────┤
|
||||
│ MongoRepository │ PostgresRepository │
|
||||
│ (existing) │ (new) │
|
||||
├──────────────────────┴──────────────────────────────────────┤
|
||||
│ DI Container (configured switch) │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### 2.2 Configuration-Driven Backend Selection
|
||||
|
||||
```json
|
||||
{
|
||||
"Persistence": {
|
||||
"Authority": "Postgres",
|
||||
"Scheduler": "Postgres",
|
||||
"Concelier": "Mongo",
|
||||
"Excititor": "Mongo",
|
||||
"Notify": "Postgres",
|
||||
"Policy": "Mongo"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 2.3 Existing PostgreSQL Patterns
|
||||
|
||||
The codebase already contains production-ready patterns:
|
||||
|
||||
| Module | Location | Reusable Components |
|
||||
|--------|----------|---------------------|
|
||||
| Orchestrator | `src/Orchestrator/.../Infrastructure/Postgres/` | DataSource, tenant context, repository pattern |
|
||||
| Findings | `src/Findings/StellaOps.Findings.Ledger/Infrastructure/Postgres/` | Ledger events, Merkle anchors, projections |
|
||||
|
||||
**Reference Implementation:** `OrchestratorDataSource.cs`
|
||||
|
||||
---
|
||||
|
||||
## 3. Data Tiering
|
||||
|
||||
### 3.1 Tier Definitions
|
||||
|
||||
| Tier | Description | Strategy |
|
||||
|------|-------------|----------|
|
||||
| **A** | Critical business data | Full conversion with verification |
|
||||
| **B** | Important but recoverable | Convert active records only |
|
||||
| **C** | Ephemeral/cache data | Fresh start, no migration |
|
||||
|
||||
### 3.2 Module Tiering
|
||||
|
||||
#### Authority
|
||||
| Collection | Tier | Strategy |
|
||||
|------------|------|----------|
|
||||
| `authority_users` | A | Full conversion |
|
||||
| `authority_clients` | A | Full conversion |
|
||||
| `authority_scopes` | A | Full conversion |
|
||||
| `authority_tokens` | B | Active tokens only |
|
||||
| `authority_service_accounts` | A | Full conversion |
|
||||
| `authority_login_attempts` | B | Recent 90 days |
|
||||
| `authority_revocations` | A | Full conversion |
|
||||
|
||||
#### Scheduler
|
||||
| Collection | Tier | Strategy |
|
||||
|------------|------|----------|
|
||||
| `schedules` | A | Full conversion |
|
||||
| `runs` | B | Recent 180 days |
|
||||
| `graph_jobs` | B | Active/recent only |
|
||||
| `policy_jobs` | B | Active/recent only |
|
||||
| `impact_snapshots` | B | Recent 90 days |
|
||||
| `locks` | C | Fresh start |
|
||||
|
||||
#### Concelier (Vulnerabilities)
|
||||
| Collection | Tier | Strategy |
|
||||
|------------|------|----------|
|
||||
| `advisory` | A | Full conversion |
|
||||
| `advisory_raw` | B | GridFS refs only |
|
||||
| `alias` | A | Full conversion |
|
||||
| `affected` | A | Full conversion |
|
||||
| `source` | A | Full conversion |
|
||||
| `source_state` | A | Full conversion |
|
||||
| `jobs`, `locks` | C | Fresh start |
|
||||
|
||||
#### Excititor (VEX)
|
||||
| Collection | Tier | Strategy |
|
||||
|------------|------|----------|
|
||||
| `vex.statements` | A | Full conversion |
|
||||
| `vex.observations` | A | Full conversion |
|
||||
| `vex.linksets` | A | Full conversion |
|
||||
| `vex.consensus` | A | Full conversion |
|
||||
| `vex.raw` | B | Active/recent only |
|
||||
| `vex.cache` | C | Fresh start |
|
||||
|
||||
---
|
||||
|
||||
## 4. Execution Phases
|
||||
|
||||
### Phase Overview
|
||||
|
||||
```
|
||||
Phase 0: Foundations [1 sprint]
|
||||
│
|
||||
├─→ Phase 1: Authority [1 sprint]
|
||||
│
|
||||
├─→ Phase 2: Scheduler [1 sprint]
|
||||
│
|
||||
├─→ Phase 3: Notify [1 sprint]
|
||||
│
|
||||
├─→ Phase 4: Policy [1 sprint]
|
||||
│
|
||||
└─→ Phase 5: Concelier [2 sprints]
|
||||
│
|
||||
└─→ Phase 6: Excititor [2-3 sprints]
|
||||
│
|
||||
└─→ Phase 7: Cleanup [1 sprint]
|
||||
```
|
||||
|
||||
### Phase Summary
|
||||
|
||||
| Phase | Scope | Duration | Dependencies | Deliverable |
|
||||
|-------|-------|----------|--------------|-------------|
|
||||
| 0 | Foundations | 1 sprint | None | PostgreSQL infrastructure, shared library |
|
||||
| 1 | Authority | 1 sprint | Phase 0 | Identity management on PostgreSQL |
|
||||
| 2 | Scheduler | 1 sprint | Phase 0 | Job scheduling on PostgreSQL |
|
||||
| 3 | Notify | 1 sprint | Phase 0 | Notifications on PostgreSQL |
|
||||
| 4 | Policy | 1 sprint | Phase 0 | Policy engine on PostgreSQL |
|
||||
| 5 | Concelier | 2 sprints | Phase 0 | Vulnerability index on PostgreSQL |
|
||||
| 6 | Excititor | 2-3 sprints | Phase 5 | VEX & graphs on PostgreSQL |
|
||||
| 7 | Cleanup | 1 sprint | All | MongoDB retired, docs updated |
|
||||
|
||||
**Total: 10-12 sprints**
|
||||
|
||||
### Detailed Task Definitions
|
||||
|
||||
See:
|
||||
- [tasks/PHASE_0_FOUNDATIONS.md](./tasks/PHASE_0_FOUNDATIONS.md)
|
||||
- [tasks/PHASE_1_AUTHORITY.md](./tasks/PHASE_1_AUTHORITY.md)
|
||||
- [tasks/PHASE_2_SCHEDULER.md](./tasks/PHASE_2_SCHEDULER.md)
|
||||
- [tasks/PHASE_3_NOTIFY.md](./tasks/PHASE_3_NOTIFY.md)
|
||||
- [tasks/PHASE_4_POLICY.md](./tasks/PHASE_4_POLICY.md)
|
||||
- [tasks/PHASE_5_VULNERABILITIES.md](./tasks/PHASE_5_VULNERABILITIES.md)
|
||||
- [tasks/PHASE_6_VEX_GRAPH.md](./tasks/PHASE_6_VEX_GRAPH.md)
|
||||
- [tasks/PHASE_7_CLEANUP.md](./tasks/PHASE_7_CLEANUP.md)
|
||||
|
||||
---
|
||||
|
||||
## 5. Conversion Strategy
|
||||
|
||||
### 5.1 Per-Module Approach
|
||||
|
||||
```
|
||||
1. Create PostgreSQL storage project
|
||||
2. Implement schema migrations
|
||||
3. Implement repository interfaces
|
||||
4. Add configuration switch
|
||||
5. Enable dual-write (if Tier A)
|
||||
6. Run verification tests
|
||||
7. Switch to PostgreSQL-only
|
||||
8. Archive MongoDB data
|
||||
```
|
||||
|
||||
### 5.2 Dual-Write Pattern
|
||||
|
||||
For Tier A data requiring historical continuity:
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────────────────┐
|
||||
│ DualWriteRepository │
|
||||
├──────────────────────────────────────────────────────────────┤
|
||||
│ Write: PostgreSQL (primary) + MongoDB (secondary) │
|
||||
│ Read: PostgreSQL (primary) → MongoDB (fallback) │
|
||||
│ Config: WriteToBoth, FallbackToMongo, ConvertOnRead │
|
||||
└──────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### 5.3 Fresh Start Pattern
|
||||
|
||||
For Tier C ephemeral data:
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────────────────┐
|
||||
│ 1. Deploy PostgreSQL schema │
|
||||
│ 2. Switch configuration to PostgreSQL │
|
||||
│ 3. New data goes to PostgreSQL only │
|
||||
│ 4. Old MongoDB data ages out naturally │
|
||||
└──────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. Risk Assessment
|
||||
|
||||
### 6.1 Technical Risks
|
||||
|
||||
| Risk | Impact | Likelihood | Mitigation |
|
||||
|------|--------|------------|------------|
|
||||
| Data loss during conversion | High | Low | Dual-write mode, extensive verification |
|
||||
| Performance regression | Medium | Medium | Load testing before switch, index optimization |
|
||||
| Determinism violation | High | Medium | Automated verification tests, parallel pipeline |
|
||||
| Schema evolution conflicts | Medium | Low | Migration framework, schema versioning |
|
||||
| Transaction semantics differences | Medium | Low | Code review, integration tests |
|
||||
|
||||
### 6.2 Operational Risks
|
||||
|
||||
| Risk | Impact | Likelihood | Mitigation |
|
||||
|------|--------|------------|------------|
|
||||
| Extended conversion timeline | Medium | Medium | Phase-based approach, clear milestones |
|
||||
| Team learning curve | Low | Medium | Reference implementations, documentation |
|
||||
| Rollback complexity | Medium | Low | Keep Mongo data until verified, feature flags |
|
||||
|
||||
### 6.3 Rollback Strategy
|
||||
|
||||
Each phase has independent rollback capability:
|
||||
|
||||
| Level | Action | Recovery Time |
|
||||
|-------|--------|---------------|
|
||||
| Configuration | Change `Persistence:<Module>` to `Mongo` | Minutes |
|
||||
| Data | MongoDB data retained during dual-write | None needed |
|
||||
| Code | Git revert (PostgreSQL code isolated) | Hours |
|
||||
|
||||
---
|
||||
|
||||
## 7. Success Criteria
|
||||
|
||||
### 7.1 Per-Module Criteria
|
||||
|
||||
- [ ] All existing integration tests pass with PostgreSQL backend
|
||||
- [ ] No performance regression >10% on critical paths
|
||||
- [ ] Deterministic outputs verified against MongoDB baseline
|
||||
- [ ] Zero data loss during conversion
|
||||
- [ ] Tenant isolation verified
|
||||
|
||||
### 7.2 Overall Criteria
|
||||
|
||||
- [ ] All control-plane modules running on PostgreSQL
|
||||
- [ ] MongoDB retired from production for converted modules
|
||||
- [ ] Air-gap kit updated with PostgreSQL support
|
||||
- [ ] Documentation updated for PostgreSQL operations
|
||||
- [ ] Runbooks updated for PostgreSQL troubleshooting
|
||||
|
||||
---
|
||||
|
||||
## 8. Project Structure
|
||||
|
||||
### 8.1 New Projects
|
||||
|
||||
```
|
||||
src/
|
||||
├── Shared/
|
||||
│ └── StellaOps.Infrastructure.Postgres/
|
||||
│ ├── DataSourceBase.cs
|
||||
│ ├── Migrations/
|
||||
│ │ ├── IPostgresMigration.cs
|
||||
│ │ └── PostgresMigrationRunner.cs
|
||||
│ ├── Extensions/
|
||||
│ │ └── NpgsqlExtensions.cs
|
||||
│ └── ServiceCollectionExtensions.cs
|
||||
│
|
||||
├── Authority/
|
||||
│ └── __Libraries/
|
||||
│ └── StellaOps.Authority.Storage.Postgres/
|
||||
│ ├── AuthorityDataSource.cs
|
||||
│ ├── Repositories/
|
||||
│ ├── Migrations/
|
||||
│ └── ServiceCollectionExtensions.cs
|
||||
│
|
||||
├── Scheduler/
|
||||
│ └── __Libraries/
|
||||
│ └── StellaOps.Scheduler.Storage.Postgres/
|
||||
│
|
||||
├── Notify/
|
||||
│ └── __Libraries/
|
||||
│ └── StellaOps.Notify.Storage.Postgres/
|
||||
│
|
||||
├── Policy/
|
||||
│ └── __Libraries/
|
||||
│ └── StellaOps.Policy.Storage.Postgres/
|
||||
│
|
||||
├── Concelier/
|
||||
│ └── __Libraries/
|
||||
│ └── StellaOps.Concelier.Storage.Postgres/
|
||||
│
|
||||
└── Excititor/
|
||||
└── __Libraries/
|
||||
└── StellaOps.Excititor.Storage.Postgres/
|
||||
```
|
||||
|
||||
### 8.2 Schema Files
|
||||
|
||||
```
|
||||
docs/db/
|
||||
├── schemas/
|
||||
│ ├── authority.sql
|
||||
│ ├── vuln.sql
|
||||
│ ├── vex.sql
|
||||
│ ├── scheduler.sql
|
||||
│ ├── notify.sql
|
||||
│ └── policy.sql
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 9. Timeline
|
||||
|
||||
### 9.1 Sprint Schedule
|
||||
|
||||
| Sprint | Phase | Focus |
|
||||
|--------|-------|-------|
|
||||
| 1 | 0 | PostgreSQL infrastructure, shared library |
|
||||
| 2 | 1 | Authority module conversion |
|
||||
| 3 | 2 | Scheduler module conversion |
|
||||
| 4 | 3 | Notify module conversion |
|
||||
| 5 | 4 | Policy module conversion |
|
||||
| 6-7 | 5 | Concelier/Vulnerability conversion |
|
||||
| 8-10 | 6 | Excititor/VEX conversion |
|
||||
| 11 | 7 | Cleanup, optimization, documentation |
|
||||
|
||||
### 9.2 Milestones
|
||||
|
||||
| Milestone | Sprint | Criteria |
|
||||
|-----------|--------|----------|
|
||||
| M1: Infrastructure Ready | 1 | PostgreSQL cluster operational, CI tests passing |
|
||||
| M2: Identity Converted | 2 | Authority on PostgreSQL, auth flows working |
|
||||
| M3: Scheduling Converted | 3 | Scheduler on PostgreSQL, jobs executing |
|
||||
| M4: Core Services Converted | 5 | Notify + Policy on PostgreSQL |
|
||||
| M5: Vulnerability Index Converted | 7 | Concelier on PostgreSQL, scans deterministic |
|
||||
| M6: VEX Converted | 10 | Excititor on PostgreSQL, graphs stable |
|
||||
| M7: MongoDB Retired | 11 | All modules converted, Mongo archived |
|
||||
|
||||
---
|
||||
|
||||
## 10. Governance
|
||||
|
||||
### 10.1 Decision Log
|
||||
|
||||
| Date | Decision | Rationale | Approver |
|
||||
|------|----------|-----------|----------|
|
||||
| 2025-11-28 | Strangler fig pattern | Allows gradual rollout with rollback | Architecture Team |
|
||||
| 2025-11-28 | JSONB for semi-structured data | Preserves flexibility, simplifies conversion | Architecture Team |
|
||||
| 2025-11-28 | Phase 0 first | Infrastructure must be stable before modules | Architecture Team |
|
||||
|
||||
### 10.2 Change Control
|
||||
|
||||
Changes to this plan require:
|
||||
1. Impact assessment documented
|
||||
2. Risk analysis updated
|
||||
3. Approval from Architecture Team
|
||||
4. Updated task definitions in `docs/db/tasks/`
|
||||
|
||||
### 10.3 Status Reporting
|
||||
|
||||
Weekly status updates in sprint files tracking:
|
||||
- Tasks completed
|
||||
- Blockers encountered
|
||||
- Verification results
|
||||
- Next sprint objectives
|
||||
|
||||
---
|
||||
|
||||
## Appendix A: Reference Implementation
|
||||
|
||||
### DataSource Pattern
|
||||
|
||||
```csharp
|
||||
public sealed class ModuleDataSource : IAsyncDisposable
|
||||
{
|
||||
private readonly NpgsqlDataSource _dataSource;
|
||||
|
||||
public async Task<NpgsqlConnection> OpenConnectionAsync(
|
||||
string tenantId,
|
||||
CancellationToken cancellationToken = default)
|
||||
{
|
||||
var connection = await _dataSource.OpenConnectionAsync(cancellationToken);
|
||||
await ConfigureSessionAsync(connection, tenantId, cancellationToken);
|
||||
return connection;
|
||||
}
|
||||
|
||||
private static async Task ConfigureSessionAsync(
|
||||
NpgsqlConnection connection,
|
||||
string tenantId,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
await using var cmd = connection.CreateCommand();
|
||||
cmd.CommandText = $"""
|
||||
SET app.tenant_id = '{tenantId}';
|
||||
SET timezone = 'UTC';
|
||||
SET statement_timeout = '30s';
|
||||
""";
|
||||
await cmd.ExecuteNonQueryAsync(cancellationToken);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Repository Pattern
|
||||
|
||||
See [RULES.md](./RULES.md) Section 1 for complete repository implementation guidelines.
|
||||
|
||||
---
|
||||
|
||||
## Appendix B: Glossary
|
||||
|
||||
| Term | Definition |
|
||||
|------|------------|
|
||||
| **Strangler Fig** | Pattern where new system grows alongside old, gradually replacing it |
|
||||
| **Dual-Write** | Writing to both MongoDB and PostgreSQL during transition |
|
||||
| **Tier A/B/C** | Data classification by criticality for migration strategy |
|
||||
| **DataSource** | Npgsql connection factory with tenant context configuration |
|
||||
| **Determinism** | Property that same inputs always produce same outputs |
|
||||
|
||||
---
|
||||
|
||||
*Document Version: 2.0.0*
|
||||
*Last Updated: 2025-11-28*
|
||||
60
docs/db/README.md
Normal file
60
docs/db/README.md
Normal file
@@ -0,0 +1,60 @@
|
||||
# StellaOps Database Documentation
|
||||
|
||||
This directory contains all documentation related to the StellaOps database architecture, including the MongoDB to PostgreSQL conversion project.
|
||||
|
||||
## Document Index
|
||||
|
||||
| Document | Purpose |
|
||||
|----------|---------|
|
||||
| [SPECIFICATION.md](./SPECIFICATION.md) | PostgreSQL schema design specification, data types, naming conventions |
|
||||
| [RULES.md](./RULES.md) | Database coding rules, patterns, and constraints for all developers |
|
||||
| [CONVERSION_PLAN.md](./CONVERSION_PLAN.md) | Strategic plan for MongoDB to PostgreSQL conversion |
|
||||
| [VERIFICATION.md](./VERIFICATION.md) | Testing and verification requirements for database changes |
|
||||
|
||||
## Task Definitions
|
||||
|
||||
Sprint-level task definitions for the conversion project:
|
||||
|
||||
| Phase | Document | Status |
|
||||
|-------|----------|--------|
|
||||
| Phase 0 | [tasks/PHASE_0_FOUNDATIONS.md](./tasks/PHASE_0_FOUNDATIONS.md) | TODO |
|
||||
| Phase 1 | [tasks/PHASE_1_AUTHORITY.md](./tasks/PHASE_1_AUTHORITY.md) | TODO |
|
||||
| Phase 2 | [tasks/PHASE_2_SCHEDULER.md](./tasks/PHASE_2_SCHEDULER.md) | TODO |
|
||||
| Phase 3 | [tasks/PHASE_3_NOTIFY.md](./tasks/PHASE_3_NOTIFY.md) | TODO |
|
||||
| Phase 4 | [tasks/PHASE_4_POLICY.md](./tasks/PHASE_4_POLICY.md) | TODO |
|
||||
| Phase 5 | [tasks/PHASE_5_VULNERABILITIES.md](./tasks/PHASE_5_VULNERABILITIES.md) | TODO |
|
||||
| Phase 6 | [tasks/PHASE_6_VEX_GRAPH.md](./tasks/PHASE_6_VEX_GRAPH.md) | TODO |
|
||||
| Phase 7 | [tasks/PHASE_7_CLEANUP.md](./tasks/PHASE_7_CLEANUP.md) | TODO |
|
||||
|
||||
## Schema Reference
|
||||
|
||||
Schema DDL files (generated from specifications):
|
||||
|
||||
| Schema | File | Tables |
|
||||
|--------|------|--------|
|
||||
| authority | [schemas/authority.sql](./schemas/authority.sql) | 12 |
|
||||
| vuln | [schemas/vuln.sql](./schemas/vuln.sql) | 12 |
|
||||
| vex | [schemas/vex.sql](./schemas/vex.sql) | 13 |
|
||||
| scheduler | [schemas/scheduler.sql](./schemas/scheduler.sql) | 10 |
|
||||
| notify | [schemas/notify.sql](./schemas/notify.sql) | 14 |
|
||||
| policy | [schemas/policy.sql](./schemas/policy.sql) | 8 |
|
||||
|
||||
## Quick Links
|
||||
|
||||
- **For developers**: Start with [RULES.md](./RULES.md) for coding conventions
|
||||
- **For architects**: Review [SPECIFICATION.md](./SPECIFICATION.md) for design rationale
|
||||
- **For project managers**: See [CONVERSION_PLAN.md](./CONVERSION_PLAN.md) for timeline and phases
|
||||
- **For QA**: Check [VERIFICATION.md](./VERIFICATION.md) for testing requirements
|
||||
|
||||
## Key Principles
|
||||
|
||||
1. **Determinism First**: All database operations must produce reproducible, stable outputs
|
||||
2. **Tenant Isolation**: Multi-tenancy via `tenant_id` column with row-level security
|
||||
3. **Strangler Fig Pattern**: Gradual conversion with rollback capability per module
|
||||
4. **JSONB for Flexibility**: Semi-structured data stays as JSONB, relational data normalizes
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Architecture Overview](../07_HIGH_LEVEL_ARCHITECTURE.md)
|
||||
- [Module Dossiers](../modules/)
|
||||
- [Air-Gap Operations](../24_OFFLINE_KIT.md)
|
||||
839
docs/db/RULES.md
Normal file
839
docs/db/RULES.md
Normal file
@@ -0,0 +1,839 @@
|
||||
# Database Coding Rules
|
||||
|
||||
**Version:** 1.0.0
|
||||
**Status:** APPROVED
|
||||
**Last Updated:** 2025-11-28
|
||||
|
||||
---
|
||||
|
||||
## Purpose
|
||||
|
||||
This document defines mandatory rules and guidelines for all database-related code in StellaOps. These rules ensure consistency, maintainability, determinism, and security across all modules.
|
||||
|
||||
**Compliance is mandatory.** Deviations require explicit approval documented in the relevant sprint file.
|
||||
|
||||
---
|
||||
|
||||
## 1. Repository Pattern Rules
|
||||
|
||||
### 1.1 Interface Location
|
||||
|
||||
**RULE:** Repository interfaces MUST be defined in the Core/Domain layer, NOT in the storage layer.
|
||||
|
||||
```
|
||||
✓ CORRECT:
|
||||
src/Scheduler/__Libraries/StellaOps.Scheduler.Core/Repositories/IScheduleRepository.cs
|
||||
|
||||
✗ INCORRECT:
|
||||
src/Scheduler/__Libraries/StellaOps.Scheduler.Storage.Postgres/IScheduleRepository.cs
|
||||
```
|
||||
|
||||
### 1.2 Implementation Naming
|
||||
|
||||
**RULE:** Repository implementations MUST be prefixed with the storage technology.
|
||||
|
||||
```csharp
|
||||
// ✓ CORRECT
|
||||
public sealed class PostgresScheduleRepository : IScheduleRepository
|
||||
public sealed class MongoScheduleRepository : IScheduleRepository
|
||||
|
||||
// ✗ INCORRECT
|
||||
public sealed class ScheduleRepository : IScheduleRepository
|
||||
```
|
||||
|
||||
### 1.3 Dependency Injection
|
||||
|
||||
**RULE:** PostgreSQL repositories MUST be registered as `Scoped`. MongoDB repositories MAY be `Singleton`.
|
||||
|
||||
```csharp
|
||||
// PostgreSQL - always scoped (connection per request)
|
||||
services.AddScoped<IScheduleRepository, PostgresScheduleRepository>();
|
||||
|
||||
// MongoDB - singleton is acceptable (stateless)
|
||||
services.AddSingleton<IScheduleRepository, MongoScheduleRepository>();
|
||||
```
|
||||
|
||||
### 1.4 No Direct SQL in Services
|
||||
|
||||
**RULE:** Business logic services MUST NOT contain raw SQL. All database access MUST go through repository interfaces.
|
||||
|
||||
```csharp
|
||||
// ✓ CORRECT
|
||||
public class ScheduleService
|
||||
{
|
||||
private readonly IScheduleRepository _repository;
|
||||
|
||||
public Task<Schedule?> GetAsync(string id)
|
||||
=> _repository.GetAsync(id);
|
||||
}
|
||||
|
||||
// ✗ INCORRECT
|
||||
public class ScheduleService
|
||||
{
|
||||
private readonly NpgsqlDataSource _dataSource;
|
||||
|
||||
public async Task<Schedule?> GetAsync(string id)
|
||||
{
|
||||
await using var conn = await _dataSource.OpenConnectionAsync();
|
||||
// Direct SQL here - FORBIDDEN
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Connection Management Rules
|
||||
|
||||
### 2.1 DataSource Pattern
|
||||
|
||||
**RULE:** Every module MUST have its own DataSource class that configures tenant context.
|
||||
|
||||
```csharp
|
||||
public sealed class SchedulerDataSource : IAsyncDisposable
|
||||
{
|
||||
private readonly NpgsqlDataSource _dataSource;
|
||||
|
||||
public async Task<NpgsqlConnection> OpenConnectionAsync(
|
||||
string tenantId,
|
||||
CancellationToken cancellationToken = default)
|
||||
{
|
||||
var connection = await _dataSource.OpenConnectionAsync(cancellationToken);
|
||||
await ConfigureSessionAsync(connection, tenantId, cancellationToken);
|
||||
return connection;
|
||||
}
|
||||
|
||||
private static async Task ConfigureSessionAsync(
|
||||
NpgsqlConnection connection,
|
||||
string tenantId,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
// MANDATORY: Set tenant context and UTC timezone
|
||||
await using var cmd = connection.CreateCommand();
|
||||
cmd.CommandText = $"""
|
||||
SET app.tenant_id = '{tenantId}';
|
||||
SET timezone = 'UTC';
|
||||
SET statement_timeout = '30s';
|
||||
""";
|
||||
await cmd.ExecuteNonQueryAsync(cancellationToken);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 2.2 Connection Disposal
|
||||
|
||||
**RULE:** All NpgsqlConnection instances MUST be disposed via `await using`.
|
||||
|
||||
```csharp
|
||||
// ✓ CORRECT
|
||||
await using var connection = await _dataSource.OpenConnectionAsync(tenantId, ct);
|
||||
|
||||
// ✗ INCORRECT
|
||||
var connection = await _dataSource.OpenConnectionAsync(tenantId, ct);
|
||||
// Missing disposal
|
||||
```
|
||||
|
||||
### 2.3 Command Disposal
|
||||
|
||||
**RULE:** All NpgsqlCommand instances MUST be disposed via `await using`.
|
||||
|
||||
```csharp
|
||||
// ✓ CORRECT
|
||||
await using var cmd = connection.CreateCommand();
|
||||
|
||||
// ✗ INCORRECT
|
||||
var cmd = connection.CreateCommand();
|
||||
```
|
||||
|
||||
### 2.4 Reader Disposal
|
||||
|
||||
**RULE:** All NpgsqlDataReader instances MUST be disposed via `await using`.
|
||||
|
||||
```csharp
|
||||
// ✓ CORRECT
|
||||
await using var reader = await cmd.ExecuteReaderAsync(ct);
|
||||
|
||||
// ✗ INCORRECT
|
||||
var reader = await cmd.ExecuteReaderAsync(ct);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Tenant Isolation Rules
|
||||
|
||||
### 3.1 Tenant ID Required
|
||||
|
||||
**RULE:** Every tenant-scoped repository method MUST require `tenantId` as the first parameter.
|
||||
|
||||
```csharp
|
||||
// ✓ CORRECT
|
||||
Task<Schedule?> GetAsync(string tenantId, string scheduleId, CancellationToken ct);
|
||||
Task<IReadOnlyList<Schedule>> ListAsync(string tenantId, QueryOptions? options, CancellationToken ct);
|
||||
|
||||
// ✗ INCORRECT
|
||||
Task<Schedule?> GetAsync(string scheduleId, CancellationToken ct);
|
||||
```
|
||||
|
||||
### 3.2 Tenant Filtering
|
||||
|
||||
**RULE:** All queries MUST include `tenant_id` in the WHERE clause for tenant-scoped tables.
|
||||
|
||||
```csharp
|
||||
// ✓ CORRECT
|
||||
cmd.CommandText = """
|
||||
SELECT * FROM scheduler.schedules
|
||||
WHERE tenant_id = @tenant_id AND id = @id
|
||||
""";
|
||||
|
||||
// ✗ INCORRECT - Missing tenant filter
|
||||
cmd.CommandText = """
|
||||
SELECT * FROM scheduler.schedules
|
||||
WHERE id = @id
|
||||
""";
|
||||
```
|
||||
|
||||
### 3.3 Session Context Verification
|
||||
|
||||
**RULE:** DataSource MUST set `app.tenant_id` on every connection before executing any queries.
|
||||
|
||||
```csharp
|
||||
// ✓ CORRECT - Connection opened via DataSource sets tenant context
|
||||
await using var connection = await _dataSource.OpenConnectionAsync(tenantId, ct);
|
||||
|
||||
// ✗ INCORRECT - Direct connection without tenant context
|
||||
await using var connection = await _rawDataSource.OpenConnectionAsync(ct);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. SQL Writing Rules
|
||||
|
||||
### 4.1 Parameterized Queries Only
|
||||
|
||||
**RULE:** All user-provided values MUST be passed as parameters. String interpolation is FORBIDDEN for values.
|
||||
|
||||
```csharp
|
||||
// ✓ CORRECT
|
||||
cmd.CommandText = "SELECT * FROM users WHERE id = @id";
|
||||
cmd.Parameters.AddWithValue("id", userId);
|
||||
|
||||
// ✗ INCORRECT - SQL INJECTION VULNERABILITY
|
||||
cmd.CommandText = $"SELECT * FROM users WHERE id = '{userId}'";
|
||||
```
|
||||
|
||||
### 4.2 SQL String Constants
|
||||
|
||||
**RULE:** SQL strings MUST be defined as `const` or `static readonly` fields, or as raw string literals in methods.
|
||||
|
||||
```csharp
|
||||
// ✓ CORRECT - Raw string literal
|
||||
cmd.CommandText = """
|
||||
SELECT id, name, created_at
|
||||
FROM scheduler.schedules
|
||||
WHERE tenant_id = @tenant_id
|
||||
ORDER BY created_at DESC
|
||||
""";
|
||||
|
||||
// ✓ CORRECT - Constant
|
||||
private const string SelectScheduleSql = """
|
||||
SELECT id, name, created_at
|
||||
FROM scheduler.schedules
|
||||
WHERE tenant_id = @tenant_id
|
||||
""";
|
||||
|
||||
// ✗ INCORRECT - Dynamic string building without reason
|
||||
cmd.CommandText = "SELECT " + columns + " FROM " + table;
|
||||
```
|
||||
|
||||
### 4.3 Schema Qualification
|
||||
|
||||
**RULE:** All table references MUST include the schema name.
|
||||
|
||||
```csharp
|
||||
// ✓ CORRECT
|
||||
cmd.CommandText = "SELECT * FROM scheduler.schedules";
|
||||
|
||||
// ✗ INCORRECT - Missing schema
|
||||
cmd.CommandText = "SELECT * FROM schedules";
|
||||
```
|
||||
|
||||
### 4.4 Column Listing
|
||||
|
||||
**RULE:** SELECT statements MUST list columns explicitly. `SELECT *` is FORBIDDEN in production code.
|
||||
|
||||
```csharp
|
||||
// ✓ CORRECT
|
||||
cmd.CommandText = """
|
||||
SELECT id, tenant_id, name, enabled, created_at
|
||||
FROM scheduler.schedules
|
||||
""";
|
||||
|
||||
// ✗ INCORRECT
|
||||
cmd.CommandText = "SELECT * FROM scheduler.schedules";
|
||||
```
|
||||
|
||||
### 4.5 Consistent Casing
|
||||
|
||||
**RULE:** SQL keywords MUST be lowercase for consistency with PostgreSQL conventions.
|
||||
|
||||
```csharp
|
||||
// ✓ CORRECT
|
||||
cmd.CommandText = """
|
||||
select id, name
|
||||
from scheduler.schedules
|
||||
where tenant_id = @tenant_id
|
||||
order by created_at desc
|
||||
""";
|
||||
|
||||
// ✗ INCORRECT - Mixed casing
|
||||
cmd.CommandText = """
|
||||
SELECT id, name
|
||||
FROM scheduler.schedules
|
||||
WHERE tenant_id = @tenant_id
|
||||
""";
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Data Type Rules
|
||||
|
||||
### 5.1 UUID Handling
|
||||
|
||||
**RULE:** UUIDs MUST be passed as `Guid` type to Npgsql, NOT as strings.
|
||||
|
||||
```csharp
|
||||
// ✓ CORRECT
|
||||
cmd.Parameters.AddWithValue("id", Guid.Parse(scheduleId));
|
||||
|
||||
// ✗ INCORRECT
|
||||
cmd.Parameters.AddWithValue("id", scheduleId); // String
|
||||
```
|
||||
|
||||
### 5.2 Timestamp Handling
|
||||
|
||||
**RULE:** All timestamps MUST be `DateTimeOffset` or `DateTime` with `Kind = Utc`.
|
||||
|
||||
```csharp
|
||||
// ✓ CORRECT
|
||||
cmd.Parameters.AddWithValue("created_at", DateTimeOffset.UtcNow);
|
||||
cmd.Parameters.AddWithValue("created_at", DateTime.UtcNow);
|
||||
|
||||
// ✗ INCORRECT - Local time
|
||||
cmd.Parameters.AddWithValue("created_at", DateTime.Now);
|
||||
```
|
||||
|
||||
### 5.3 JSONB Serialization
|
||||
|
||||
**RULE:** JSONB columns MUST be serialized using `System.Text.Json.JsonSerializer` with consistent options.
|
||||
|
||||
```csharp
|
||||
// ✓ CORRECT
|
||||
var json = JsonSerializer.Serialize(obj, JsonSerializerOptions.Default);
|
||||
cmd.Parameters.AddWithValue("config", json);
|
||||
|
||||
// ✗ INCORRECT - Newtonsoft or inconsistent serialization
|
||||
var json = Newtonsoft.Json.JsonConvert.SerializeObject(obj);
|
||||
```
|
||||
|
||||
### 5.4 Null Handling
|
||||
|
||||
**RULE:** Nullable values MUST use `DBNull.Value` when null.
|
||||
|
||||
```csharp
|
||||
// ✓ CORRECT
|
||||
cmd.Parameters.AddWithValue("description", (object?)schedule.Description ?? DBNull.Value);
|
||||
|
||||
// ✗ INCORRECT - Will fail or behave unexpectedly
|
||||
cmd.Parameters.AddWithValue("description", schedule.Description); // If null
|
||||
```
|
||||
|
||||
### 5.5 Array Handling
|
||||
|
||||
**RULE:** PostgreSQL arrays MUST be passed as .NET arrays with explicit type.
|
||||
|
||||
```csharp
|
||||
// ✓ CORRECT
|
||||
cmd.Parameters.AddWithValue("tags", schedule.Tags.ToArray());
|
||||
|
||||
// ✗ INCORRECT - List won't map correctly
|
||||
cmd.Parameters.AddWithValue("tags", schedule.Tags);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. Transaction Rules
|
||||
|
||||
### 6.1 Explicit Transactions
|
||||
|
||||
**RULE:** Operations affecting multiple tables MUST use explicit transactions.
|
||||
|
||||
```csharp
|
||||
// ✓ CORRECT
|
||||
await using var transaction = await connection.BeginTransactionAsync(ct);
|
||||
try
|
||||
{
|
||||
// Multiple operations
|
||||
await cmd1.ExecuteNonQueryAsync(ct);
|
||||
await cmd2.ExecuteNonQueryAsync(ct);
|
||||
await transaction.CommitAsync(ct);
|
||||
}
|
||||
catch
|
||||
{
|
||||
await transaction.RollbackAsync(ct);
|
||||
throw;
|
||||
}
|
||||
```
|
||||
|
||||
### 6.2 Transaction Isolation
|
||||
|
||||
**RULE:** Default isolation level is `ReadCommitted`. Stricter levels MUST be documented.
|
||||
|
||||
```csharp
|
||||
// ✓ CORRECT - Default
|
||||
await using var transaction = await connection.BeginTransactionAsync(ct);
|
||||
|
||||
// ✓ CORRECT - Explicit stricter level with documentation
|
||||
// Using Serializable for financial consistency requirement
|
||||
await using var transaction = await connection.BeginTransactionAsync(
|
||||
IsolationLevel.Serializable, ct);
|
||||
```
|
||||
|
||||
### 6.3 No Nested Transactions
|
||||
|
||||
**RULE:** Nested transactions are NOT supported. Use savepoints if needed.
|
||||
|
||||
```csharp
|
||||
// ✗ INCORRECT - Nested transaction
|
||||
await using var tx1 = await connection.BeginTransactionAsync(ct);
|
||||
await using var tx2 = await connection.BeginTransactionAsync(ct); // FAILS
|
||||
|
||||
// ✓ CORRECT - Savepoint for partial rollback
|
||||
await using var transaction = await connection.BeginTransactionAsync(ct);
|
||||
await transaction.SaveAsync("savepoint1", ct);
|
||||
// ... operations ...
|
||||
await transaction.RollbackAsync("savepoint1", ct); // Partial rollback
|
||||
await transaction.CommitAsync(ct);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7. Error Handling Rules
|
||||
|
||||
### 7.1 PostgreSQL Exception Handling
|
||||
|
||||
**RULE:** Catch `PostgresException` for database-specific errors, not generic exceptions.
|
||||
|
||||
```csharp
|
||||
// ✓ CORRECT
|
||||
try
|
||||
{
|
||||
await cmd.ExecuteNonQueryAsync(ct);
|
||||
}
|
||||
catch (PostgresException ex) when (ex.SqlState == "23505") // Unique violation
|
||||
{
|
||||
throw new DuplicateEntityException($"Entity already exists: {ex.ConstraintName}");
|
||||
}
|
||||
|
||||
// ✗ INCORRECT - Too broad
|
||||
catch (Exception ex)
|
||||
{
|
||||
// Can't distinguish database errors from other errors
|
||||
}
|
||||
```
|
||||
|
||||
### 7.2 Constraint Violation Handling
|
||||
|
||||
**RULE:** Unique constraint violations MUST be translated to domain exceptions.
|
||||
|
||||
| SQL State | Meaning | Domain Exception |
|
||||
|-----------|---------|------------------|
|
||||
| `23505` | Unique violation | `DuplicateEntityException` |
|
||||
| `23503` | Foreign key violation | `ReferenceNotFoundException` |
|
||||
| `23502` | Not null violation | `ValidationException` |
|
||||
| `23514` | Check constraint | `ValidationException` |
|
||||
|
||||
### 7.3 Timeout Handling
|
||||
|
||||
**RULE:** Query timeouts MUST be caught and logged with context.
|
||||
|
||||
```csharp
|
||||
try
|
||||
{
|
||||
await cmd.ExecuteNonQueryAsync(ct);
|
||||
}
|
||||
catch (NpgsqlException ex) when (ex.InnerException is TimeoutException)
|
||||
{
|
||||
_logger.LogWarning(ex, "Query timeout for schedule {ScheduleId}", scheduleId);
|
||||
throw new QueryTimeoutException("Database query timed out", ex);
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8. Pagination Rules
|
||||
|
||||
### 8.1 Keyset Pagination
|
||||
|
||||
**RULE:** Use keyset pagination, NOT offset pagination for large result sets.
|
||||
|
||||
```csharp
|
||||
// ✓ CORRECT - Keyset pagination
|
||||
cmd.CommandText = """
|
||||
select id, name, created_at
|
||||
from scheduler.schedules
|
||||
where tenant_id = @tenant_id
|
||||
and (created_at, id) < (@cursor_created_at, @cursor_id)
|
||||
order by created_at desc, id desc
|
||||
limit @page_size
|
||||
""";
|
||||
|
||||
// ✗ INCORRECT - Offset pagination (slow for large offsets)
|
||||
cmd.CommandText = """
|
||||
select id, name, created_at
|
||||
from scheduler.schedules
|
||||
where tenant_id = @tenant_id
|
||||
order by created_at desc
|
||||
limit @page_size offset @offset
|
||||
""";
|
||||
```
|
||||
|
||||
### 8.2 Default Page Size
|
||||
|
||||
**RULE:** Default page size MUST be 50. Maximum page size MUST be 1000.
|
||||
|
||||
```csharp
|
||||
public class QueryOptions
|
||||
{
|
||||
public int PageSize { get; init; } = 50;
|
||||
|
||||
public int GetValidatedPageSize()
|
||||
=> Math.Clamp(PageSize, 1, 1000);
|
||||
}
|
||||
```
|
||||
|
||||
### 8.3 Continuation Tokens
|
||||
|
||||
**RULE:** Pagination cursors MUST be opaque, encoded tokens containing sort key values.
|
||||
|
||||
```csharp
|
||||
public record PaginationCursor(DateTimeOffset CreatedAt, Guid Id)
|
||||
{
|
||||
public string Encode()
|
||||
=> Convert.ToBase64String(
|
||||
JsonSerializer.SerializeToUtf8Bytes(this));
|
||||
|
||||
public static PaginationCursor? Decode(string? token)
|
||||
=> string.IsNullOrEmpty(token)
|
||||
? null
|
||||
: JsonSerializer.Deserialize<PaginationCursor>(
|
||||
Convert.FromBase64String(token));
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 9. Ordering Rules
|
||||
|
||||
### 9.1 Deterministic Ordering
|
||||
|
||||
**RULE:** All queries returning multiple rows MUST have an ORDER BY clause that produces deterministic results.
|
||||
|
||||
```csharp
|
||||
// ✓ CORRECT - Deterministic (includes unique column)
|
||||
cmd.CommandText = """
|
||||
select * from scheduler.runs
|
||||
order by created_at desc, id asc
|
||||
""";
|
||||
|
||||
// ✗ INCORRECT - Non-deterministic (created_at may have ties)
|
||||
cmd.CommandText = """
|
||||
select * from scheduler.runs
|
||||
order by created_at desc
|
||||
""";
|
||||
```
|
||||
|
||||
### 9.2 Stable Ordering for JSONB Arrays
|
||||
|
||||
**RULE:** When serializing arrays to JSONB, ensure consistent ordering.
|
||||
|
||||
```csharp
|
||||
// ✓ CORRECT - Sorted before serialization
|
||||
var sortedTags = schedule.Tags.OrderBy(t => t).ToList();
|
||||
cmd.Parameters.AddWithValue("tags", sortedTags.ToArray());
|
||||
|
||||
// ✗ INCORRECT - Order may vary
|
||||
cmd.Parameters.AddWithValue("tags", schedule.Tags.ToArray());
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 10. Audit Rules
|
||||
|
||||
### 10.1 Timestamp Columns
|
||||
|
||||
**RULE:** All mutable tables MUST have `created_at` and `updated_at` columns.
|
||||
|
||||
```sql
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||
```
|
||||
|
||||
### 10.2 Update Timestamp
|
||||
|
||||
**RULE:** `updated_at` MUST be set on every UPDATE operation.
|
||||
|
||||
```csharp
|
||||
// ✓ CORRECT
|
||||
cmd.CommandText = """
|
||||
update scheduler.schedules
|
||||
set name = @name, updated_at = @updated_at
|
||||
where id = @id
|
||||
""";
|
||||
cmd.Parameters.AddWithValue("updated_at", DateTimeOffset.UtcNow);
|
||||
|
||||
// ✗ INCORRECT - Missing updated_at
|
||||
cmd.CommandText = """
|
||||
update scheduler.schedules
|
||||
set name = @name
|
||||
where id = @id
|
||||
""";
|
||||
```
|
||||
|
||||
### 10.3 Soft Delete Pattern
|
||||
|
||||
**RULE:** For audit-required entities, use soft delete with `deleted_at` and `deleted_by`.
|
||||
|
||||
```csharp
|
||||
cmd.CommandText = """
|
||||
update scheduler.schedules
|
||||
set deleted_at = @deleted_at, deleted_by = @deleted_by
|
||||
where tenant_id = @tenant_id and id = @id and deleted_at is null
|
||||
""";
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 11. Testing Rules
|
||||
|
||||
### 11.1 Integration Test Database
|
||||
|
||||
**RULE:** Integration tests MUST use Testcontainers with PostgreSQL.
|
||||
|
||||
```csharp
|
||||
public class PostgresFixture : IAsyncLifetime
|
||||
{
|
||||
private readonly PostgreSqlContainer _container = new PostgreSqlBuilder()
|
||||
.WithImage("postgres:16")
|
||||
.Build();
|
||||
|
||||
public string ConnectionString => _container.GetConnectionString();
|
||||
|
||||
public Task InitializeAsync() => _container.StartAsync();
|
||||
public Task DisposeAsync() => _container.DisposeAsync().AsTask();
|
||||
}
|
||||
```
|
||||
|
||||
### 11.2 Test Isolation
|
||||
|
||||
**RULE:** Each test MUST run in a transaction that is rolled back after the test.
|
||||
|
||||
```csharp
|
||||
public class ScheduleRepositoryTests : IClassFixture<PostgresFixture>
|
||||
{
|
||||
[Fact]
|
||||
public async Task GetAsync_ReturnsSchedule_WhenExists()
|
||||
{
|
||||
await using var connection = await _fixture.OpenConnectionAsync();
|
||||
await using var transaction = await connection.BeginTransactionAsync();
|
||||
|
||||
try
|
||||
{
|
||||
// Arrange, Act, Assert
|
||||
}
|
||||
finally
|
||||
{
|
||||
await transaction.RollbackAsync();
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 11.3 Determinism Tests
|
||||
|
||||
**RULE:** Every repository MUST have tests verifying deterministic output ordering.
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public async Task ListAsync_ReturnsDeterministicOrder()
|
||||
{
|
||||
// Insert records with same created_at
|
||||
// Verify order is consistent across multiple calls
|
||||
var result1 = await _repository.ListAsync(tenantId);
|
||||
var result2 = await _repository.ListAsync(tenantId);
|
||||
|
||||
result1.Should().BeEquivalentTo(result2, options =>
|
||||
options.WithStrictOrdering());
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 12. Migration Rules
|
||||
|
||||
### 12.1 Idempotent Migrations
|
||||
|
||||
**RULE:** All migrations MUST be idempotent using `IF NOT EXISTS` / `IF EXISTS`.
|
||||
|
||||
```sql
|
||||
-- ✓ CORRECT
|
||||
CREATE TABLE IF NOT EXISTS scheduler.schedules (...);
|
||||
CREATE INDEX IF NOT EXISTS idx_schedules_tenant ON scheduler.schedules(tenant_id);
|
||||
|
||||
-- ✗ INCORRECT
|
||||
CREATE TABLE scheduler.schedules (...); -- Fails if exists
|
||||
```
|
||||
|
||||
### 12.2 No Breaking Changes
|
||||
|
||||
**RULE:** Migrations MUST NOT break existing code. Use expand-contract pattern.
|
||||
|
||||
```
|
||||
Expand Phase:
|
||||
1. Add new column as nullable
|
||||
2. Deploy code that writes to both old and new columns
|
||||
3. Backfill new column
|
||||
|
||||
Contract Phase:
|
||||
4. Deploy code that reads from new column only
|
||||
5. Add NOT NULL constraint
|
||||
6. Drop old column
|
||||
```
|
||||
|
||||
### 12.3 Index Creation
|
||||
|
||||
**RULE:** Large table indexes MUST be created with `CONCURRENTLY`.
|
||||
|
||||
```sql
|
||||
-- ✓ CORRECT - Won't lock table
|
||||
CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_large_table_col
|
||||
ON schema.large_table(column);
|
||||
|
||||
-- ✗ INCORRECT - Locks table during creation
|
||||
CREATE INDEX idx_large_table_col ON schema.large_table(column);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 13. Configuration Rules
|
||||
|
||||
### 13.1 Backend Selection
|
||||
|
||||
**RULE:** Storage backend MUST be configurable per module.
|
||||
|
||||
```json
|
||||
{
|
||||
"Persistence": {
|
||||
"Authority": "Postgres",
|
||||
"Scheduler": "Postgres",
|
||||
"Concelier": "Mongo"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 13.2 Connection String Security
|
||||
|
||||
**RULE:** Connection strings MUST NOT be logged or included in exception messages.
|
||||
|
||||
```csharp
|
||||
// ✓ CORRECT
|
||||
catch (NpgsqlException ex)
|
||||
{
|
||||
_logger.LogError(ex, "Database connection failed for module {Module}", moduleName);
|
||||
throw;
|
||||
}
|
||||
|
||||
// ✗ INCORRECT
|
||||
catch (NpgsqlException ex)
|
||||
{
|
||||
_logger.LogError("Failed to connect: {ConnectionString}", connectionString);
|
||||
}
|
||||
```
|
||||
|
||||
### 13.3 Timeout Configuration
|
||||
|
||||
**RULE:** Command timeout MUST be configurable with sensible defaults.
|
||||
|
||||
```csharp
|
||||
public class PostgresOptions
|
||||
{
|
||||
public int CommandTimeoutSeconds { get; set; } = 30;
|
||||
public int ConnectionTimeoutSeconds { get; set; } = 15;
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 14. Documentation Rules
|
||||
|
||||
### 14.1 Repository Method Documentation
|
||||
|
||||
**RULE:** All public repository methods MUST have XML documentation.
|
||||
|
||||
```csharp
|
||||
/// <summary>
|
||||
/// Retrieves a schedule by its unique identifier.
|
||||
/// </summary>
|
||||
/// <param name="tenantId">The tenant identifier for isolation.</param>
|
||||
/// <param name="scheduleId">The schedule's unique identifier.</param>
|
||||
/// <param name="cancellationToken">Cancellation token.</param>
|
||||
/// <returns>The schedule if found; otherwise, null.</returns>
|
||||
Task<Schedule?> GetAsync(string tenantId, string scheduleId, CancellationToken cancellationToken);
|
||||
```
|
||||
|
||||
### 14.2 SQL Comment Headers
|
||||
|
||||
**RULE:** Complex SQL queries SHOULD have a comment explaining the purpose.
|
||||
|
||||
```csharp
|
||||
cmd.CommandText = """
|
||||
-- Find schedules due to fire within the next minute
|
||||
-- Uses compound index (tenant_id, next_fire_time) for efficiency
|
||||
select s.id, s.name, t.next_fire_time
|
||||
from scheduler.schedules s
|
||||
join scheduler.triggers t on t.schedule_id = s.id
|
||||
where s.tenant_id = @tenant_id
|
||||
and s.enabled = true
|
||||
and t.next_fire_time <= @window_end
|
||||
order by t.next_fire_time asc
|
||||
""";
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Enforcement
|
||||
|
||||
### Code Review Checklist
|
||||
|
||||
- [ ] Repository interfaces in Core layer
|
||||
- [ ] PostgreSQL repositories prefixed with `Postgres`
|
||||
- [ ] All connections disposed with `await using`
|
||||
- [ ] Tenant ID required and used in all queries
|
||||
- [ ] Parameterized queries (no string interpolation for values)
|
||||
- [ ] Schema-qualified table names
|
||||
- [ ] Explicit column lists (no `SELECT *`)
|
||||
- [ ] Deterministic ORDER BY clauses
|
||||
- [ ] Timestamps are UTC
|
||||
- [ ] JSONB serialized with System.Text.Json
|
||||
- [ ] PostgresException caught for constraint violations
|
||||
- [ ] Integration tests use Testcontainers
|
||||
|
||||
### Automated Checks
|
||||
|
||||
These rules are enforced by:
|
||||
- Roslyn analyzers in `StellaOps.Analyzers`
|
||||
- SQL linting in CI pipeline
|
||||
- Integration test requirements
|
||||
|
||||
---
|
||||
|
||||
*Document Version: 1.0.0*
|
||||
*Last Updated: 2025-11-28*
|
||||
1326
docs/db/SPECIFICATION.md
Normal file
1326
docs/db/SPECIFICATION.md
Normal file
File diff suppressed because it is too large
Load Diff
961
docs/db/VERIFICATION.md
Normal file
961
docs/db/VERIFICATION.md
Normal file
@@ -0,0 +1,961 @@
|
||||
# Database Verification Requirements
|
||||
|
||||
**Version:** 1.0.0
|
||||
**Status:** DRAFT
|
||||
**Last Updated:** 2025-11-28
|
||||
|
||||
---
|
||||
|
||||
## Purpose
|
||||
|
||||
This document defines the verification and testing requirements for the MongoDB to PostgreSQL conversion. It ensures that the conversion maintains data integrity, determinism, and functional correctness.
|
||||
|
||||
---
|
||||
|
||||
## 1. Verification Principles
|
||||
|
||||
### 1.1 Core Guarantees
|
||||
|
||||
The conversion MUST maintain these guarantees:
|
||||
|
||||
| Guarantee | Description | Verification Method |
|
||||
|-----------|-------------|---------------------|
|
||||
| **Data Integrity** | No data loss during conversion | Record count comparison, checksum validation |
|
||||
| **Determinism** | Same inputs produce identical outputs | Parallel pipeline comparison |
|
||||
| **Functional Equivalence** | APIs behave identically | Integration test suite |
|
||||
| **Performance Parity** | No significant degradation | Benchmark comparison |
|
||||
| **Tenant Isolation** | Data remains properly isolated | Cross-tenant query tests |
|
||||
|
||||
### 1.2 Verification Levels
|
||||
|
||||
```
|
||||
Level 1: Unit Tests
|
||||
└── Individual repository method correctness
|
||||
|
||||
Level 2: Integration Tests
|
||||
└── End-to-end repository operations with real PostgreSQL
|
||||
|
||||
Level 3: Comparison Tests
|
||||
└── MongoDB vs PostgreSQL output comparison
|
||||
|
||||
Level 4: Load Tests
|
||||
└── Performance and scalability verification
|
||||
|
||||
Level 5: Production Verification
|
||||
└── Dual-write monitoring and validation
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Test Infrastructure
|
||||
|
||||
### 2.1 Testcontainers Setup
|
||||
|
||||
All PostgreSQL integration tests MUST use Testcontainers:
|
||||
|
||||
```csharp
|
||||
public sealed class PostgresTestFixture : IAsyncLifetime
|
||||
{
|
||||
private readonly PostgreSqlContainer _container;
|
||||
private NpgsqlDataSource? _dataSource;
|
||||
|
||||
public PostgresTestFixture()
|
||||
{
|
||||
_container = new PostgreSqlBuilder()
|
||||
.WithImage("postgres:16-alpine")
|
||||
.WithDatabase("stellaops_test")
|
||||
.WithUsername("test")
|
||||
.WithPassword("test")
|
||||
.WithWaitStrategy(Wait.ForUnixContainer()
|
||||
.UntilPortIsAvailable(5432))
|
||||
.Build();
|
||||
}
|
||||
|
||||
public string ConnectionString => _container.GetConnectionString();
|
||||
public NpgsqlDataSource DataSource => _dataSource
|
||||
?? throw new InvalidOperationException("Not initialized");
|
||||
|
||||
public async Task InitializeAsync()
|
||||
{
|
||||
await _container.StartAsync();
|
||||
_dataSource = NpgsqlDataSource.Create(ConnectionString);
|
||||
await RunMigrationsAsync();
|
||||
}
|
||||
|
||||
public async Task DisposeAsync()
|
||||
{
|
||||
if (_dataSource is not null)
|
||||
await _dataSource.DisposeAsync();
|
||||
await _container.DisposeAsync();
|
||||
}
|
||||
|
||||
private async Task RunMigrationsAsync()
|
||||
{
|
||||
await using var connection = await _dataSource!.OpenConnectionAsync();
|
||||
var migrationRunner = new PostgresMigrationRunner(_dataSource, GetMigrations());
|
||||
await migrationRunner.RunAsync();
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 2.2 Test Database State Management
|
||||
|
||||
```csharp
|
||||
public abstract class PostgresRepositoryTestBase : IAsyncLifetime
|
||||
{
|
||||
protected readonly PostgresTestFixture Fixture;
|
||||
protected NpgsqlConnection Connection = null!;
|
||||
protected NpgsqlTransaction Transaction = null!;
|
||||
|
||||
protected PostgresRepositoryTestBase(PostgresTestFixture fixture)
|
||||
{
|
||||
Fixture = fixture;
|
||||
}
|
||||
|
||||
public async Task InitializeAsync()
|
||||
{
|
||||
Connection = await Fixture.DataSource.OpenConnectionAsync();
|
||||
Transaction = await Connection.BeginTransactionAsync();
|
||||
|
||||
// Set test tenant context
|
||||
await using var cmd = Connection.CreateCommand();
|
||||
cmd.CommandText = "SET app.tenant_id = 'test-tenant-id'";
|
||||
await cmd.ExecuteNonQueryAsync();
|
||||
}
|
||||
|
||||
public async Task DisposeAsync()
|
||||
{
|
||||
await Transaction.RollbackAsync();
|
||||
await Transaction.DisposeAsync();
|
||||
await Connection.DisposeAsync();
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 2.3 Test Data Builders
|
||||
|
||||
```csharp
|
||||
public sealed class ScheduleBuilder
|
||||
{
|
||||
private Guid _id = Guid.NewGuid();
|
||||
private string _tenantId = "test-tenant";
|
||||
private string _name = "test-schedule";
|
||||
private bool _enabled = true;
|
||||
private string? _cronExpression = "0 * * * *";
|
||||
|
||||
public ScheduleBuilder WithId(Guid id) { _id = id; return this; }
|
||||
public ScheduleBuilder WithTenant(string tenantId) { _tenantId = tenantId; return this; }
|
||||
public ScheduleBuilder WithName(string name) { _name = name; return this; }
|
||||
public ScheduleBuilder Enabled(bool enabled = true) { _enabled = enabled; return this; }
|
||||
public ScheduleBuilder WithCron(string? cron) { _cronExpression = cron; return this; }
|
||||
|
||||
public Schedule Build() => new()
|
||||
{
|
||||
Id = _id,
|
||||
TenantId = _tenantId,
|
||||
Name = _name,
|
||||
Enabled = _enabled,
|
||||
CronExpression = _cronExpression,
|
||||
Timezone = "UTC",
|
||||
Mode = ScheduleMode.Scheduled,
|
||||
CreatedAt = DateTimeOffset.UtcNow,
|
||||
UpdatedAt = DateTimeOffset.UtcNow
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Unit Test Requirements
|
||||
|
||||
### 3.1 Repository CRUD Tests
|
||||
|
||||
Every repository implementation MUST have tests for:
|
||||
|
||||
```csharp
|
||||
public class PostgresScheduleRepositoryTests : PostgresRepositoryTestBase
|
||||
{
|
||||
private readonly PostgresScheduleRepository _repository;
|
||||
|
||||
public PostgresScheduleRepositoryTests(PostgresTestFixture fixture)
|
||||
: base(fixture)
|
||||
{
|
||||
_repository = new PostgresScheduleRepository(/* ... */);
|
||||
}
|
||||
|
||||
// CREATE
|
||||
[Fact]
|
||||
public async Task UpsertAsync_CreatesNewSchedule_WhenNotExists()
|
||||
{
|
||||
var schedule = new ScheduleBuilder().Build();
|
||||
|
||||
await _repository.UpsertAsync(schedule, CancellationToken.None);
|
||||
|
||||
var retrieved = await _repository.GetAsync(
|
||||
schedule.TenantId, schedule.Id.ToString(), CancellationToken.None);
|
||||
retrieved.Should().BeEquivalentTo(schedule);
|
||||
}
|
||||
|
||||
// READ
|
||||
[Fact]
|
||||
public async Task GetAsync_ReturnsNull_WhenNotExists()
|
||||
{
|
||||
var result = await _repository.GetAsync(
|
||||
"tenant", Guid.NewGuid().ToString(), CancellationToken.None);
|
||||
|
||||
result.Should().BeNull();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task GetAsync_ReturnsSchedule_WhenExists()
|
||||
{
|
||||
var schedule = new ScheduleBuilder().Build();
|
||||
await _repository.UpsertAsync(schedule, CancellationToken.None);
|
||||
|
||||
var result = await _repository.GetAsync(
|
||||
schedule.TenantId, schedule.Id.ToString(), CancellationToken.None);
|
||||
|
||||
result.Should().NotBeNull();
|
||||
result!.Id.Should().Be(schedule.Id);
|
||||
}
|
||||
|
||||
// UPDATE
|
||||
[Fact]
|
||||
public async Task UpsertAsync_UpdatesExisting_WhenExists()
|
||||
{
|
||||
var schedule = new ScheduleBuilder().Build();
|
||||
await _repository.UpsertAsync(schedule, CancellationToken.None);
|
||||
|
||||
schedule = schedule with { Name = "updated-name" };
|
||||
await _repository.UpsertAsync(schedule, CancellationToken.None);
|
||||
|
||||
var retrieved = await _repository.GetAsync(
|
||||
schedule.TenantId, schedule.Id.ToString(), CancellationToken.None);
|
||||
retrieved!.Name.Should().Be("updated-name");
|
||||
}
|
||||
|
||||
// DELETE
|
||||
[Fact]
|
||||
public async Task SoftDeleteAsync_SetsDeletedAt_WhenExists()
|
||||
{
|
||||
var schedule = new ScheduleBuilder().Build();
|
||||
await _repository.UpsertAsync(schedule, CancellationToken.None);
|
||||
|
||||
var result = await _repository.SoftDeleteAsync(
|
||||
schedule.TenantId, schedule.Id.ToString(),
|
||||
"test-user", DateTimeOffset.UtcNow, CancellationToken.None);
|
||||
|
||||
result.Should().BeTrue();
|
||||
var retrieved = await _repository.GetAsync(
|
||||
schedule.TenantId, schedule.Id.ToString(), CancellationToken.None);
|
||||
retrieved.Should().BeNull(); // Soft-deleted not returned
|
||||
}
|
||||
|
||||
// LIST
|
||||
[Fact]
|
||||
public async Task ListAsync_ReturnsAllForTenant()
|
||||
{
|
||||
var schedule1 = new ScheduleBuilder().WithName("schedule-1").Build();
|
||||
var schedule2 = new ScheduleBuilder().WithName("schedule-2").Build();
|
||||
await _repository.UpsertAsync(schedule1, CancellationToken.None);
|
||||
await _repository.UpsertAsync(schedule2, CancellationToken.None);
|
||||
|
||||
var results = await _repository.ListAsync(
|
||||
schedule1.TenantId, null, CancellationToken.None);
|
||||
|
||||
results.Should().HaveCount(2);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 3.2 Tenant Isolation Tests
|
||||
|
||||
```csharp
|
||||
public class TenantIsolationTests : PostgresRepositoryTestBase
|
||||
{
|
||||
[Fact]
|
||||
public async Task GetAsync_DoesNotReturnOtherTenantData()
|
||||
{
|
||||
var tenant1Schedule = new ScheduleBuilder()
|
||||
.WithTenant("tenant-1")
|
||||
.WithName("tenant1-schedule")
|
||||
.Build();
|
||||
var tenant2Schedule = new ScheduleBuilder()
|
||||
.WithTenant("tenant-2")
|
||||
.WithName("tenant2-schedule")
|
||||
.Build();
|
||||
|
||||
await _repository.UpsertAsync(tenant1Schedule, CancellationToken.None);
|
||||
await _repository.UpsertAsync(tenant2Schedule, CancellationToken.None);
|
||||
|
||||
// Tenant 1 should not see Tenant 2's data
|
||||
var result = await _repository.GetAsync(
|
||||
"tenant-1", tenant2Schedule.Id.ToString(), CancellationToken.None);
|
||||
|
||||
result.Should().BeNull();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ListAsync_OnlyReturnsTenantData()
|
||||
{
|
||||
// Create schedules for two tenants
|
||||
for (int i = 0; i < 5; i++)
|
||||
{
|
||||
await _repository.UpsertAsync(
|
||||
new ScheduleBuilder().WithTenant("tenant-1").Build(),
|
||||
CancellationToken.None);
|
||||
await _repository.UpsertAsync(
|
||||
new ScheduleBuilder().WithTenant("tenant-2").Build(),
|
||||
CancellationToken.None);
|
||||
}
|
||||
|
||||
var tenant1Results = await _repository.ListAsync(
|
||||
"tenant-1", null, CancellationToken.None);
|
||||
var tenant2Results = await _repository.ListAsync(
|
||||
"tenant-2", null, CancellationToken.None);
|
||||
|
||||
tenant1Results.Should().HaveCount(5);
|
||||
tenant2Results.Should().HaveCount(5);
|
||||
tenant1Results.Should().OnlyContain(s => s.TenantId == "tenant-1");
|
||||
tenant2Results.Should().OnlyContain(s => s.TenantId == "tenant-2");
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 3.3 Determinism Tests
|
||||
|
||||
```csharp
|
||||
public class DeterminismTests : PostgresRepositoryTestBase
|
||||
{
|
||||
[Fact]
|
||||
public async Task ListAsync_ReturnsDeterministicOrder()
|
||||
{
|
||||
// Insert multiple schedules with same created_at
|
||||
var baseTime = DateTimeOffset.UtcNow;
|
||||
var schedules = Enumerable.Range(0, 10)
|
||||
.Select(i => new ScheduleBuilder()
|
||||
.WithName($"schedule-{i}")
|
||||
.Build() with { CreatedAt = baseTime })
|
||||
.ToList();
|
||||
|
||||
foreach (var schedule in schedules)
|
||||
await _repository.UpsertAsync(schedule, CancellationToken.None);
|
||||
|
||||
// Multiple calls should return same order
|
||||
var results1 = await _repository.ListAsync("test-tenant", null, CancellationToken.None);
|
||||
var results2 = await _repository.ListAsync("test-tenant", null, CancellationToken.None);
|
||||
var results3 = await _repository.ListAsync("test-tenant", null, CancellationToken.None);
|
||||
|
||||
results1.Select(s => s.Id).Should().Equal(results2.Select(s => s.Id));
|
||||
results2.Select(s => s.Id).Should().Equal(results3.Select(s => s.Id));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task JsonbSerialization_IsDeterministic()
|
||||
{
|
||||
var schedule = new ScheduleBuilder()
|
||||
.Build() with
|
||||
{
|
||||
Selection = new ScheduleSelector
|
||||
{
|
||||
Tags = new[] { "z", "a", "m" },
|
||||
Repositories = new[] { "repo-2", "repo-1" }
|
||||
}
|
||||
};
|
||||
|
||||
await _repository.UpsertAsync(schedule, CancellationToken.None);
|
||||
|
||||
// Retrieve and re-save multiple times
|
||||
for (int i = 0; i < 3; i++)
|
||||
{
|
||||
var retrieved = await _repository.GetAsync(
|
||||
schedule.TenantId, schedule.Id.ToString(), CancellationToken.None);
|
||||
await _repository.UpsertAsync(retrieved!, CancellationToken.None);
|
||||
}
|
||||
|
||||
// Final retrieval should have identical JSONB
|
||||
var final = await _repository.GetAsync(
|
||||
schedule.TenantId, schedule.Id.ToString(), CancellationToken.None);
|
||||
|
||||
// Arrays should be consistently ordered
|
||||
final!.Selection.Tags.Should().BeInAscendingOrder();
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Comparison Test Requirements
|
||||
|
||||
### 4.1 MongoDB vs PostgreSQL Comparison Framework
|
||||
|
||||
```csharp
|
||||
public abstract class ComparisonTestBase<TEntity, TRepository>
|
||||
where TRepository : class
|
||||
{
|
||||
protected readonly TRepository MongoRepository;
|
||||
protected readonly TRepository PostgresRepository;
|
||||
|
||||
protected abstract Task<TEntity?> GetFromMongo(string tenantId, string id);
|
||||
protected abstract Task<TEntity?> GetFromPostgres(string tenantId, string id);
|
||||
protected abstract Task<IReadOnlyList<TEntity>> ListFromMongo(string tenantId);
|
||||
protected abstract Task<IReadOnlyList<TEntity>> ListFromPostgres(string tenantId);
|
||||
|
||||
[Fact]
|
||||
public async Task Get_ReturnsSameEntity_FromBothBackends()
|
||||
{
|
||||
var entityId = GetTestEntityId();
|
||||
var tenantId = GetTestTenantId();
|
||||
|
||||
var mongoResult = await GetFromMongo(tenantId, entityId);
|
||||
var postgresResult = await GetFromPostgres(tenantId, entityId);
|
||||
|
||||
postgresResult.Should().BeEquivalentTo(mongoResult, options =>
|
||||
options.Excluding(e => e.Path.Contains("Id"))); // IDs may differ
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task List_ReturnsSameEntities_FromBothBackends()
|
||||
{
|
||||
var tenantId = GetTestTenantId();
|
||||
|
||||
var mongoResults = await ListFromMongo(tenantId);
|
||||
var postgresResults = await ListFromPostgres(tenantId);
|
||||
|
||||
postgresResults.Should().BeEquivalentTo(mongoResults, options =>
|
||||
options
|
||||
.Excluding(e => e.Path.Contains("Id"))
|
||||
.WithStrictOrdering()); // Order must match
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 4.2 Advisory Matching Comparison
|
||||
|
||||
```csharp
|
||||
public class AdvisoryMatchingComparisonTests
|
||||
{
|
||||
[Theory]
|
||||
[MemberData(nameof(GetSampleSboms))]
|
||||
public async Task VulnerabilityMatching_ProducesSameResults(string sbomPath)
|
||||
{
|
||||
var sbom = await LoadSbomAsync(sbomPath);
|
||||
|
||||
// Configure Mongo backend
|
||||
var mongoConfig = CreateConfig("Mongo");
|
||||
var mongoScanner = CreateScanner(mongoConfig);
|
||||
var mongoFindings = await mongoScanner.ScanAsync(sbom);
|
||||
|
||||
// Configure Postgres backend
|
||||
var postgresConfig = CreateConfig("Postgres");
|
||||
var postgresScanner = CreateScanner(postgresConfig);
|
||||
var postgresFindings = await postgresScanner.ScanAsync(sbom);
|
||||
|
||||
// Compare findings
|
||||
postgresFindings.Should().BeEquivalentTo(mongoFindings, options =>
|
||||
options
|
||||
.WithStrictOrdering()
|
||||
.Using<DateTimeOffset>(ctx =>
|
||||
ctx.Subject.Should().BeCloseTo(ctx.Expectation, TimeSpan.FromSeconds(1)))
|
||||
.WhenTypeIs<DateTimeOffset>());
|
||||
}
|
||||
|
||||
public static IEnumerable<object[]> GetSampleSboms()
|
||||
{
|
||||
yield return new object[] { "testdata/sbom-alpine-3.18.json" };
|
||||
yield return new object[] { "testdata/sbom-debian-12.json" };
|
||||
yield return new object[] { "testdata/sbom-nodejs-app.json" };
|
||||
yield return new object[] { "testdata/sbom-python-app.json" };
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 4.3 VEX Graph Comparison
|
||||
|
||||
```csharp
|
||||
public class GraphRevisionComparisonTests
|
||||
{
|
||||
[Theory]
|
||||
[MemberData(nameof(GetTestProjects))]
|
||||
public async Task GraphComputation_ProducesIdenticalRevisionId(string projectId)
|
||||
{
|
||||
// Compute graph with Mongo backend
|
||||
var mongoGraph = await ComputeGraphAsync(projectId, "Mongo");
|
||||
|
||||
// Compute graph with Postgres backend
|
||||
var postgresGraph = await ComputeGraphAsync(projectId, "Postgres");
|
||||
|
||||
// Revision ID MUST be identical (hash-stable)
|
||||
postgresGraph.RevisionId.Should().Be(mongoGraph.RevisionId);
|
||||
|
||||
// Node and edge counts should match
|
||||
postgresGraph.NodeCount.Should().Be(mongoGraph.NodeCount);
|
||||
postgresGraph.EdgeCount.Should().Be(mongoGraph.EdgeCount);
|
||||
|
||||
// VEX statements should match
|
||||
var mongoStatements = await GetStatementsAsync(projectId, "Mongo");
|
||||
var postgresStatements = await GetStatementsAsync(projectId, "Postgres");
|
||||
|
||||
postgresStatements.Should().BeEquivalentTo(mongoStatements, options =>
|
||||
options
|
||||
.Excluding(s => s.Id)
|
||||
.WithStrictOrdering());
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Performance Test Requirements
|
||||
|
||||
### 5.1 Benchmark Framework
|
||||
|
||||
```csharp
|
||||
[MemoryDiagnoser]
|
||||
[SimpleJob(RuntimeMoniker.Net80)]
|
||||
public class RepositoryBenchmarks
|
||||
{
|
||||
private IScheduleRepository _mongoRepository = null!;
|
||||
private IScheduleRepository _postgresRepository = null!;
|
||||
private string _tenantId = null!;
|
||||
|
||||
[GlobalSetup]
|
||||
public async Task Setup()
|
||||
{
|
||||
// Initialize both repositories
|
||||
_mongoRepository = await CreateMongoRepositoryAsync();
|
||||
_postgresRepository = await CreatePostgresRepositoryAsync();
|
||||
_tenantId = await SeedTestDataAsync();
|
||||
}
|
||||
|
||||
[Benchmark(Baseline = true)]
|
||||
public async Task<Schedule?> Mongo_GetById()
|
||||
{
|
||||
return await _mongoRepository.GetAsync(_tenantId, _testScheduleId, CancellationToken.None);
|
||||
}
|
||||
|
||||
[Benchmark]
|
||||
public async Task<Schedule?> Postgres_GetById()
|
||||
{
|
||||
return await _postgresRepository.GetAsync(_tenantId, _testScheduleId, CancellationToken.None);
|
||||
}
|
||||
|
||||
[Benchmark(Baseline = true)]
|
||||
public async Task<IReadOnlyList<Schedule>> Mongo_List100()
|
||||
{
|
||||
return await _mongoRepository.ListAsync(_tenantId,
|
||||
new QueryOptions { PageSize = 100 }, CancellationToken.None);
|
||||
}
|
||||
|
||||
[Benchmark]
|
||||
public async Task<IReadOnlyList<Schedule>> Postgres_List100()
|
||||
{
|
||||
return await _postgresRepository.ListAsync(_tenantId,
|
||||
new QueryOptions { PageSize = 100 }, CancellationToken.None);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 5.2 Performance Acceptance Criteria
|
||||
|
||||
| Operation | Mongo Baseline | Postgres Target | Maximum Acceptable |
|
||||
|-----------|----------------|-----------------|-------------------|
|
||||
| Get by ID | X ms | ≤ X ms | ≤ 1.5X ms |
|
||||
| List (100 items) | Y ms | ≤ Y ms | ≤ 1.5Y ms |
|
||||
| Insert | Z ms | ≤ Z ms | ≤ 2Z ms |
|
||||
| Update | W ms | ≤ W ms | ≤ 2W ms |
|
||||
| Complex query | V ms | ≤ V ms | ≤ 2V ms |
|
||||
|
||||
### 5.3 Load Test Scenarios
|
||||
|
||||
```yaml
|
||||
# k6 load test configuration
|
||||
scenarios:
|
||||
constant_load:
|
||||
executor: constant-arrival-rate
|
||||
rate: 100
|
||||
timeUnit: 1s
|
||||
duration: 5m
|
||||
preAllocatedVUs: 50
|
||||
maxVUs: 100
|
||||
|
||||
spike_test:
|
||||
executor: ramping-arrival-rate
|
||||
startRate: 10
|
||||
timeUnit: 1s
|
||||
stages:
|
||||
- duration: 1m
|
||||
target: 10
|
||||
- duration: 1m
|
||||
target: 100
|
||||
- duration: 2m
|
||||
target: 100
|
||||
- duration: 1m
|
||||
target: 10
|
||||
|
||||
thresholds:
|
||||
http_req_duration:
|
||||
- p(95) < 200 # 95th percentile under 200ms
|
||||
- p(99) < 500 # 99th percentile under 500ms
|
||||
http_req_failed:
|
||||
- rate < 0.01 # Error rate under 1%
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. Data Integrity Verification
|
||||
|
||||
### 6.1 Record Count Verification
|
||||
|
||||
```csharp
|
||||
public class DataIntegrityVerifier
|
||||
{
|
||||
public async Task<VerificationResult> VerifyCountsAsync(string module)
|
||||
{
|
||||
var results = new Dictionary<string, (long mongo, long postgres)>();
|
||||
|
||||
foreach (var collection in GetCollections(module))
|
||||
{
|
||||
var mongoCount = await _mongoDb.GetCollection<BsonDocument>(collection)
|
||||
.CountDocumentsAsync(FilterDefinition<BsonDocument>.Empty);
|
||||
|
||||
var postgresCount = await GetPostgresCountAsync(collection);
|
||||
|
||||
results[collection] = (mongoCount, postgresCount);
|
||||
}
|
||||
|
||||
return new VerificationResult
|
||||
{
|
||||
Module = module,
|
||||
Counts = results,
|
||||
AllMatch = results.All(r => r.Value.mongo == r.Value.postgres)
|
||||
};
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 6.2 Checksum Verification
|
||||
|
||||
```csharp
|
||||
public class ChecksumVerifier
|
||||
{
|
||||
public async Task<bool> VerifyAdvisoryChecksumAsync(string advisoryKey)
|
||||
{
|
||||
var mongoAdvisory = await _mongoAdvisoryRepo.GetAsync(advisoryKey);
|
||||
var postgresAdvisory = await _postgresAdvisoryRepo.GetAsync(advisoryKey);
|
||||
|
||||
if (mongoAdvisory is null || postgresAdvisory is null)
|
||||
return mongoAdvisory is null && postgresAdvisory is null;
|
||||
|
||||
var mongoChecksum = ComputeChecksum(mongoAdvisory);
|
||||
var postgresChecksum = ComputeChecksum(postgresAdvisory);
|
||||
|
||||
return mongoChecksum == postgresChecksum;
|
||||
}
|
||||
|
||||
private string ComputeChecksum(Advisory advisory)
|
||||
{
|
||||
// Serialize to canonical JSON and hash
|
||||
var json = JsonSerializer.Serialize(advisory, new JsonSerializerOptions
|
||||
{
|
||||
PropertyNamingPolicy = JsonNamingPolicy.CamelCase,
|
||||
WriteIndented = false,
|
||||
DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull
|
||||
});
|
||||
|
||||
using var sha256 = SHA256.Create();
|
||||
var hash = sha256.ComputeHash(Encoding.UTF8.GetBytes(json));
|
||||
return Convert.ToHexString(hash);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 6.3 Referential Integrity Verification
|
||||
|
||||
```csharp
|
||||
public class ReferentialIntegrityTests
|
||||
{
|
||||
[Fact]
|
||||
public async Task AllForeignKeys_ReferenceExistingRecords()
|
||||
{
|
||||
await using var connection = await _dataSource.OpenConnectionAsync();
|
||||
await using var cmd = connection.CreateCommand();
|
||||
|
||||
// Check for orphaned references
|
||||
cmd.CommandText = """
|
||||
SELECT 'advisory_aliases' as table_name, COUNT(*) as orphan_count
|
||||
FROM vuln.advisory_aliases a
|
||||
LEFT JOIN vuln.advisories adv ON a.advisory_id = adv.id
|
||||
WHERE adv.id IS NULL
|
||||
|
||||
UNION ALL
|
||||
|
||||
SELECT 'advisory_cvss', COUNT(*)
|
||||
FROM vuln.advisory_cvss c
|
||||
LEFT JOIN vuln.advisories adv ON c.advisory_id = adv.id
|
||||
WHERE adv.id IS NULL
|
||||
|
||||
-- Add more tables...
|
||||
""";
|
||||
|
||||
await using var reader = await cmd.ExecuteReaderAsync();
|
||||
while (await reader.ReadAsync())
|
||||
{
|
||||
var tableName = reader.GetString(0);
|
||||
var orphanCount = reader.GetInt64(1);
|
||||
orphanCount.Should().Be(0, $"Table {tableName} has orphaned references");
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7. Production Verification
|
||||
|
||||
### 7.1 Dual-Write Monitoring
|
||||
|
||||
```csharp
|
||||
public class DualWriteMonitor
|
||||
{
|
||||
private readonly IMetrics _metrics;
|
||||
|
||||
public async Task RecordWriteAsync(
|
||||
string module,
|
||||
string operation,
|
||||
bool mongoSuccess,
|
||||
bool postgresSuccess,
|
||||
TimeSpan mongoDuration,
|
||||
TimeSpan postgresDuration)
|
||||
{
|
||||
_metrics.Counter("dual_write_total", new[]
|
||||
{
|
||||
("module", module),
|
||||
("operation", operation),
|
||||
("mongo_success", mongoSuccess.ToString()),
|
||||
("postgres_success", postgresSuccess.ToString())
|
||||
}).Inc();
|
||||
|
||||
_metrics.Histogram("dual_write_duration_ms", new[]
|
||||
{
|
||||
("module", module),
|
||||
("operation", operation),
|
||||
("backend", "mongo")
|
||||
}).Observe(mongoDuration.TotalMilliseconds);
|
||||
|
||||
_metrics.Histogram("dual_write_duration_ms", new[]
|
||||
{
|
||||
("module", module),
|
||||
("operation", operation),
|
||||
("backend", "postgres")
|
||||
}).Observe(postgresDuration.TotalMilliseconds);
|
||||
|
||||
if (mongoSuccess != postgresSuccess)
|
||||
{
|
||||
_metrics.Counter("dual_write_inconsistency", new[]
|
||||
{
|
||||
("module", module),
|
||||
("operation", operation)
|
||||
}).Inc();
|
||||
|
||||
_logger.LogWarning(
|
||||
"Dual-write inconsistency: {Module}/{Operation} - Mongo: {Mongo}, Postgres: {Postgres}",
|
||||
module, operation, mongoSuccess, postgresSuccess);
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 7.2 Read Comparison Sampling
|
||||
|
||||
```csharp
|
||||
public class ReadComparisonSampler : BackgroundService
|
||||
{
|
||||
private readonly IOptions<SamplingOptions> _options;
|
||||
private readonly Random _random = new();
|
||||
|
||||
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
|
||||
{
|
||||
while (!stoppingToken.IsCancellationRequested)
|
||||
{
|
||||
if (_random.NextDouble() < _options.Value.SampleRate) // e.g., 1%
|
||||
{
|
||||
await CompareRandomRecordAsync(stoppingToken);
|
||||
}
|
||||
|
||||
await Task.Delay(_options.Value.Interval, stoppingToken);
|
||||
}
|
||||
}
|
||||
|
||||
private async Task CompareRandomRecordAsync(CancellationToken ct)
|
||||
{
|
||||
var entityId = await GetRandomEntityIdAsync(ct);
|
||||
|
||||
var mongoEntity = await _mongoRepo.GetAsync(entityId, ct);
|
||||
var postgresEntity = await _postgresRepo.GetAsync(entityId, ct);
|
||||
|
||||
if (!AreEquivalent(mongoEntity, postgresEntity))
|
||||
{
|
||||
_logger.LogError(
|
||||
"Read comparison mismatch for entity {EntityId}",
|
||||
entityId);
|
||||
|
||||
_metrics.Counter("read_comparison_mismatch").Inc();
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 7.3 Rollback Verification
|
||||
|
||||
```csharp
|
||||
public class RollbackVerificationTests
|
||||
{
|
||||
[Fact]
|
||||
public async Task Rollback_RestoresMongoAsSource_WhenPostgresFails()
|
||||
{
|
||||
// Simulate Postgres failure
|
||||
await _postgresDataSource.DisposeAsync();
|
||||
|
||||
// Verify system falls back to Mongo
|
||||
var config = _configuration.GetSection("Persistence");
|
||||
config["Scheduler"] = "Mongo"; // Simulate config change
|
||||
|
||||
// Operations should continue working
|
||||
var schedule = await _scheduleRepository.GetAsync(
|
||||
"tenant", "schedule-id", CancellationToken.None);
|
||||
|
||||
schedule.Should().NotBeNull();
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8. Module-Specific Verification
|
||||
|
||||
### 8.1 Authority Verification
|
||||
|
||||
| Test | Description | Pass Criteria |
|
||||
|------|-------------|---------------|
|
||||
| User CRUD | Create, read, update, delete users | All operations succeed |
|
||||
| Role assignment | Assign/revoke roles | Roles correctly applied |
|
||||
| Token issuance | Issue OAuth tokens | Tokens valid and verifiable |
|
||||
| Token verification | Verify issued tokens | Verification succeeds |
|
||||
| Login tracking | Record login attempts | Attempts logged correctly |
|
||||
| License validation | Check license validity | Same result both backends |
|
||||
|
||||
### 8.2 Scheduler Verification
|
||||
|
||||
| Test | Description | Pass Criteria |
|
||||
|------|-------------|---------------|
|
||||
| Schedule CRUD | All CRUD operations | Data integrity preserved |
|
||||
| Trigger calculation | Next fire time calculation | Identical results |
|
||||
| Run history | Run creation and completion | Correct state transitions |
|
||||
| Impact snapshots | Finding aggregation | Same counts and severity |
|
||||
| Worker registration | Worker heartbeats | Consistent status |
|
||||
|
||||
### 8.3 Vulnerability Verification
|
||||
|
||||
| Test | Description | Pass Criteria |
|
||||
|------|-------------|---------------|
|
||||
| Advisory ingest | Import from feed | All advisories imported |
|
||||
| Alias resolution | CVE → Advisory lookup | Same advisory returned |
|
||||
| CVSS lookup | Get CVSS scores | Identical scores |
|
||||
| Affected package match | PURL matching | Same vulnerabilities found |
|
||||
| KEV flag lookup | Check KEV status | Correct flag status |
|
||||
|
||||
### 8.4 VEX Verification
|
||||
|
||||
| Test | Description | Pass Criteria |
|
||||
|------|-------------|---------------|
|
||||
| Graph revision | Compute revision ID | Identical revision IDs |
|
||||
| Node/edge counts | Graph structure | Same counts |
|
||||
| VEX statements | Status determination | Same statuses |
|
||||
| Consensus computation | Aggregate signals | Same consensus |
|
||||
| Evidence manifest | Merkle root | Identical roots |
|
||||
|
||||
---
|
||||
|
||||
## 9. Verification Checklist
|
||||
|
||||
### Per-Module Checklist
|
||||
|
||||
- [ ] All unit tests pass with PostgreSQL
|
||||
- [ ] Tenant isolation tests pass
|
||||
- [ ] Determinism tests pass
|
||||
- [ ] Performance benchmarks within tolerance
|
||||
- [ ] Record counts match between MongoDB and PostgreSQL
|
||||
- [ ] Checksum verification passes for sample data
|
||||
- [ ] Referential integrity verified
|
||||
- [ ] Comparison tests pass for all scenarios
|
||||
- [ ] Load tests pass with acceptable metrics
|
||||
|
||||
### Pre-Production Checklist
|
||||
|
||||
- [ ] Dual-write monitoring in place
|
||||
- [ ] Read comparison sampling enabled
|
||||
- [ ] Rollback procedure tested
|
||||
- [ ] Performance baselines established
|
||||
- [ ] Alert thresholds configured
|
||||
- [ ] Runbook documented
|
||||
|
||||
### Post-Switch Checklist
|
||||
|
||||
- [ ] No dual-write inconsistencies for 7 days
|
||||
- [ ] Read comparison sampling shows 100% match
|
||||
- [ ] Performance within acceptable range
|
||||
- [ ] No data integrity alerts
|
||||
- [ ] MongoDB reads disabled
|
||||
- [ ] MongoDB backups archived
|
||||
|
||||
---
|
||||
|
||||
## 10. Reporting
|
||||
|
||||
### 10.1 Verification Report Template
|
||||
|
||||
```markdown
|
||||
# Database Conversion Verification Report
|
||||
|
||||
## Module: [Module Name]
|
||||
## Date: [YYYY-MM-DD]
|
||||
## Status: [PASS/FAIL]
|
||||
|
||||
### Summary
|
||||
- Total Tests: X
|
||||
- Passed: Y
|
||||
- Failed: Z
|
||||
|
||||
### Unit Tests
|
||||
| Category | Passed | Failed | Notes |
|
||||
|----------|--------|--------|-------|
|
||||
| CRUD | | | |
|
||||
| Isolation| | | |
|
||||
| Determinism | | | |
|
||||
|
||||
### Comparison Tests
|
||||
| Test | Status | Notes |
|
||||
|------|--------|-------|
|
||||
| | | |
|
||||
|
||||
### Performance
|
||||
| Operation | Mongo | Postgres | Diff |
|
||||
|-----------|-------|----------|------|
|
||||
| | | | |
|
||||
|
||||
### Data Integrity
|
||||
- Record count match: [YES/NO]
|
||||
- Checksum verification: [PASS/FAIL]
|
||||
- Referential integrity: [PASS/FAIL]
|
||||
|
||||
### Sign-off
|
||||
- [ ] QA Engineer
|
||||
- [ ] Tech Lead
|
||||
- [ ] Product Owner
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
*Document Version: 1.0.0*
|
||||
*Last Updated: 2025-11-28*
|
||||
404
docs/db/tasks/PHASE_0_FOUNDATIONS.md
Normal file
404
docs/db/tasks/PHASE_0_FOUNDATIONS.md
Normal file
@@ -0,0 +1,404 @@
|
||||
# Phase 0: Foundations
|
||||
|
||||
**Sprint:** 1
|
||||
**Duration:** 1 sprint
|
||||
**Status:** TODO
|
||||
**Dependencies:** None
|
||||
|
||||
---
|
||||
|
||||
## Objectives
|
||||
|
||||
1. Provision PostgreSQL cluster for staging and production
|
||||
2. Create shared infrastructure library (`StellaOps.Infrastructure.Postgres`)
|
||||
3. Set up CI/CD pipeline for PostgreSQL migrations
|
||||
4. Establish Testcontainers-based integration testing
|
||||
|
||||
---
|
||||
|
||||
## Deliverables
|
||||
|
||||
| Deliverable | Acceptance Criteria |
|
||||
|-------------|---------------------|
|
||||
| PostgreSQL cluster | Running in staging with proper configuration |
|
||||
| Shared library | DataSource, migrations, extensions implemented |
|
||||
| CI pipeline | PostgreSQL tests running on every PR |
|
||||
| Documentation | SPECIFICATION.md, RULES.md reviewed and approved |
|
||||
|
||||
---
|
||||
|
||||
## Task Breakdown
|
||||
|
||||
### T0.1: PostgreSQL Cluster Provisioning
|
||||
|
||||
**Status:** TODO
|
||||
**Assignee:** TBD
|
||||
**Estimate:** 2 days
|
||||
|
||||
**Description:**
|
||||
Provision PostgreSQL 16+ cluster with appropriate configuration for StellaOps workload.
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T0.1.1: Select PostgreSQL hosting (managed vs self-hosted)
|
||||
- [ ] T0.1.2: Create staging cluster with single primary
|
||||
- [ ] T0.1.3: Configure connection pooling (PgBouncer or built-in)
|
||||
- [ ] T0.1.4: Set up backup and restore procedures
|
||||
- [ ] T0.1.5: Configure monitoring (pg_stat_statements, Prometheus exporter)
|
||||
- [ ] T0.1.6: Document connection strings and access credentials
|
||||
- [ ] T0.1.7: Configure SSL/TLS for connections
|
||||
|
||||
**Configuration Requirements:**
|
||||
```
|
||||
PostgreSQL Version: 16+
|
||||
Max Connections: 100 (via pooler: 500)
|
||||
Shared Buffers: 25% of RAM
|
||||
Work Mem: 64MB
|
||||
Maintenance Work Mem: 512MB
|
||||
WAL Level: replica
|
||||
Max WAL Size: 2GB
|
||||
```
|
||||
|
||||
**Verification:**
|
||||
- [ ] Can connect from development machines
|
||||
- [ ] Can connect from CI/CD runners
|
||||
- [ ] Monitoring dashboard shows metrics
|
||||
- [ ] Backup tested and verified
|
||||
|
||||
---
|
||||
|
||||
### T0.2: Create StellaOps.Infrastructure.Postgres Library
|
||||
|
||||
**Status:** TODO
|
||||
**Assignee:** TBD
|
||||
**Estimate:** 3 days
|
||||
|
||||
**Description:**
|
||||
Create shared library with reusable PostgreSQL infrastructure components.
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T0.2.1: Create project `src/Shared/StellaOps.Infrastructure.Postgres/`
|
||||
- [ ] T0.2.2: Add Npgsql NuGet package reference
|
||||
- [ ] T0.2.3: Implement `DataSourceBase` abstract class
|
||||
- [ ] T0.2.4: Implement `IPostgresMigration` interface
|
||||
- [ ] T0.2.5: Implement `PostgresMigrationRunner` class
|
||||
- [ ] T0.2.6: Implement `NpgsqlExtensions` helper methods
|
||||
- [ ] T0.2.7: Implement `ServiceCollectionExtensions` for DI
|
||||
- [ ] T0.2.8: Add XML documentation to all public APIs
|
||||
- [ ] T0.2.9: Add unit tests for migration runner
|
||||
|
||||
**Files to Create:**
|
||||
```
|
||||
src/Shared/StellaOps.Infrastructure.Postgres/
|
||||
├── StellaOps.Infrastructure.Postgres.csproj
|
||||
├── DataSourceBase.cs
|
||||
├── PostgresOptions.cs
|
||||
├── Migrations/
|
||||
│ ├── IPostgresMigration.cs
|
||||
│ └── PostgresMigrationRunner.cs
|
||||
├── Extensions/
|
||||
│ ├── NpgsqlExtensions.cs
|
||||
│ └── NpgsqlCommandExtensions.cs
|
||||
└── ServiceCollectionExtensions.cs
|
||||
```
|
||||
|
||||
**DataSourceBase Implementation:**
|
||||
```csharp
|
||||
public abstract class DataSourceBase : IAsyncDisposable
|
||||
{
|
||||
protected readonly NpgsqlDataSource DataSource;
|
||||
protected readonly PostgresOptions Options;
|
||||
|
||||
protected DataSourceBase(IOptions<PostgresOptions> options)
|
||||
{
|
||||
Options = options.Value;
|
||||
var builder = new NpgsqlDataSourceBuilder(Options.ConnectionString);
|
||||
ConfigureDataSource(builder);
|
||||
DataSource = builder.Build();
|
||||
}
|
||||
|
||||
protected virtual void ConfigureDataSource(NpgsqlDataSourceBuilder builder)
|
||||
{
|
||||
// Override in derived classes for module-specific config
|
||||
}
|
||||
|
||||
public async Task<NpgsqlConnection> OpenConnectionAsync(
|
||||
string tenantId,
|
||||
CancellationToken cancellationToken = default)
|
||||
{
|
||||
var connection = await DataSource.OpenConnectionAsync(cancellationToken);
|
||||
await ConfigureSessionAsync(connection, tenantId, cancellationToken);
|
||||
return connection;
|
||||
}
|
||||
|
||||
protected virtual async Task ConfigureSessionAsync(
|
||||
NpgsqlConnection connection,
|
||||
string tenantId,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
await using var cmd = connection.CreateCommand();
|
||||
cmd.CommandText = $"""
|
||||
SET app.tenant_id = '{tenantId}';
|
||||
SET timezone = 'UTC';
|
||||
SET statement_timeout = '{Options.CommandTimeoutSeconds}s';
|
||||
""";
|
||||
await cmd.ExecuteNonQueryAsync(cancellationToken);
|
||||
}
|
||||
|
||||
public async ValueTask DisposeAsync()
|
||||
{
|
||||
await DataSource.DisposeAsync();
|
||||
GC.SuppressFinalize(this);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Verification:**
|
||||
- [ ] Project builds without errors
|
||||
- [ ] Unit tests pass
|
||||
- [ ] Can be referenced from module projects
|
||||
|
||||
---
|
||||
|
||||
### T0.3: Migration Framework Implementation
|
||||
|
||||
**Status:** TODO
|
||||
**Assignee:** TBD
|
||||
**Estimate:** 2 days
|
||||
|
||||
**Description:**
|
||||
Implement idempotent migration framework for schema management.
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T0.3.1: Define `IPostgresMigration` interface
|
||||
- [ ] T0.3.2: Implement `PostgresMigrationRunner` with transaction support
|
||||
- [ ] T0.3.3: Implement migration tracking table (`_migrations`)
|
||||
- [ ] T0.3.4: Add `IHostedService` for automatic migration on startup
|
||||
- [ ] T0.3.5: Add CLI command for manual migration execution
|
||||
- [ ] T0.3.6: Add migration rollback support (optional)
|
||||
|
||||
**Migration Interface:**
|
||||
```csharp
|
||||
public interface IPostgresMigration
|
||||
{
|
||||
/// <summary>
|
||||
/// Unique migration identifier (e.g., "V001_CreateAuthoritySchema")
|
||||
/// </summary>
|
||||
string Id { get; }
|
||||
|
||||
/// <summary>
|
||||
/// Human-readable description
|
||||
/// </summary>
|
||||
string Description { get; }
|
||||
|
||||
/// <summary>
|
||||
/// Apply the migration
|
||||
/// </summary>
|
||||
Task UpAsync(NpgsqlConnection connection, CancellationToken cancellationToken);
|
||||
|
||||
/// <summary>
|
||||
/// Rollback the migration (optional)
|
||||
/// </summary>
|
||||
Task DownAsync(NpgsqlConnection connection, CancellationToken cancellationToken);
|
||||
}
|
||||
```
|
||||
|
||||
**Verification:**
|
||||
- [ ] Migrations run idempotently (can run multiple times)
|
||||
- [ ] Migration state tracked correctly
|
||||
- [ ] Failed migrations roll back cleanly
|
||||
|
||||
---
|
||||
|
||||
### T0.4: CI/CD Pipeline Configuration
|
||||
|
||||
**Status:** TODO
|
||||
**Assignee:** TBD
|
||||
**Estimate:** 2 days
|
||||
|
||||
**Description:**
|
||||
Add PostgreSQL integration testing to CI/CD pipeline.
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T0.4.1: Add Testcontainers.PostgreSql NuGet package to test projects
|
||||
- [ ] T0.4.2: Create `PostgresTestFixture` base class
|
||||
- [ ] T0.4.3: Update CI workflow to support PostgreSQL containers
|
||||
- [ ] T0.4.4: Add parallel test execution configuration
|
||||
- [ ] T0.4.5: Add test coverage reporting for PostgreSQL code
|
||||
|
||||
**PostgresTestFixture:**
|
||||
```csharp
|
||||
public sealed class PostgresTestFixture : IAsyncLifetime
|
||||
{
|
||||
private readonly PostgreSqlContainer _container;
|
||||
private NpgsqlDataSource? _dataSource;
|
||||
|
||||
public PostgresTestFixture()
|
||||
{
|
||||
_container = new PostgreSqlBuilder()
|
||||
.WithImage("postgres:16-alpine")
|
||||
.WithDatabase("stellaops_test")
|
||||
.WithUsername("test")
|
||||
.WithPassword("test")
|
||||
.WithWaitStrategy(Wait.ForUnixContainer()
|
||||
.UntilPortIsAvailable(5432))
|
||||
.Build();
|
||||
}
|
||||
|
||||
public string ConnectionString => _container.GetConnectionString();
|
||||
public NpgsqlDataSource DataSource => _dataSource
|
||||
?? throw new InvalidOperationException("Fixture not initialized");
|
||||
|
||||
public async Task InitializeAsync()
|
||||
{
|
||||
await _container.StartAsync();
|
||||
_dataSource = NpgsqlDataSource.Create(ConnectionString);
|
||||
}
|
||||
|
||||
public async Task DisposeAsync()
|
||||
{
|
||||
if (_dataSource is not null)
|
||||
await _dataSource.DisposeAsync();
|
||||
await _container.DisposeAsync();
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**CI Workflow Update:**
|
||||
```yaml
|
||||
# .gitea/workflows/build-test-deploy.yml
|
||||
- name: Run PostgreSQL Integration Tests
|
||||
run: |
|
||||
dotnet test src/StellaOps.sln \
|
||||
--filter "Category=PostgresIntegration" \
|
||||
--logger "trx;LogFileName=postgres-test-results.trx"
|
||||
env:
|
||||
TESTCONTAINERS_RYUK_DISABLED: true
|
||||
```
|
||||
|
||||
**Verification:**
|
||||
- [ ] CI pipeline runs PostgreSQL tests
|
||||
- [ ] Tests can run in parallel without conflicts
|
||||
- [ ] Test results reported correctly
|
||||
|
||||
---
|
||||
|
||||
### T0.5: Persistence Configuration
|
||||
|
||||
**Status:** TODO
|
||||
**Assignee:** TBD
|
||||
**Estimate:** 1 day
|
||||
|
||||
**Description:**
|
||||
Add persistence backend configuration to all services.
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T0.5.1: Define `PersistenceOptions` class
|
||||
- [ ] T0.5.2: Add configuration section to `appsettings.json`
|
||||
- [ ] T0.5.3: Update service registration to read persistence config
|
||||
- [ ] T0.5.4: Add configuration validation on startup
|
||||
|
||||
**PersistenceOptions:**
|
||||
```csharp
|
||||
public sealed class PersistenceOptions
|
||||
{
|
||||
public const string SectionName = "Persistence";
|
||||
|
||||
public string Authority { get; set; } = "Mongo";
|
||||
public string Scheduler { get; set; } = "Mongo";
|
||||
public string Concelier { get; set; } = "Mongo";
|
||||
public string Excititor { get; set; } = "Mongo";
|
||||
public string Notify { get; set; } = "Mongo";
|
||||
public string Policy { get; set; } = "Mongo";
|
||||
}
|
||||
```
|
||||
|
||||
**Configuration Template:**
|
||||
```json
|
||||
{
|
||||
"Persistence": {
|
||||
"Authority": "Mongo",
|
||||
"Scheduler": "Mongo",
|
||||
"Concelier": "Mongo",
|
||||
"Excititor": "Mongo",
|
||||
"Notify": "Mongo",
|
||||
"Policy": "Mongo"
|
||||
},
|
||||
"Postgres": {
|
||||
"ConnectionString": "Host=localhost;Database=stellaops;Username=stellaops;Password=secret",
|
||||
"CommandTimeoutSeconds": 30,
|
||||
"ConnectionTimeoutSeconds": 15
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Verification:**
|
||||
- [ ] Configuration loads correctly
|
||||
- [ ] Invalid configuration throws on startup
|
||||
- [ ] Environment variables can override settings
|
||||
|
||||
---
|
||||
|
||||
### T0.6: Documentation Review
|
||||
|
||||
**Status:** TODO
|
||||
**Assignee:** TBD
|
||||
**Estimate:** 1 day
|
||||
|
||||
**Description:**
|
||||
Review and finalize database documentation.
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T0.6.1: Review SPECIFICATION.md for completeness
|
||||
- [ ] T0.6.2: Review RULES.md for clarity
|
||||
- [ ] T0.6.3: Review VERIFICATION.md for test coverage
|
||||
- [ ] T0.6.4: Get Architecture Team sign-off
|
||||
- [ ] T0.6.5: Publish to team wiki/docs site
|
||||
|
||||
**Verification:**
|
||||
- [ ] All documents reviewed by 2+ team members
|
||||
- [ ] No outstanding questions or TODOs
|
||||
- [ ] Architecture Team approval received
|
||||
|
||||
---
|
||||
|
||||
## Exit Criteria
|
||||
|
||||
- [ ] PostgreSQL cluster running and accessible
|
||||
- [ ] `StellaOps.Infrastructure.Postgres` library implemented and tested
|
||||
- [ ] CI pipeline running PostgreSQL integration tests
|
||||
- [ ] Persistence configuration framework in place
|
||||
- [ ] Documentation reviewed and approved
|
||||
|
||||
---
|
||||
|
||||
## Risks & Mitigations
|
||||
|
||||
| Risk | Likelihood | Impact | Mitigation |
|
||||
|------|------------|--------|------------|
|
||||
| PostgreSQL provisioning delays | Medium | High | Start early, have backup plan |
|
||||
| Testcontainers compatibility issues | Low | Medium | Test on CI runners early |
|
||||
| Configuration complexity | Low | Low | Use existing patterns from Orchestrator |
|
||||
|
||||
---
|
||||
|
||||
## Dependencies on Later Phases
|
||||
|
||||
Phase 0 must complete before any module conversion (Phases 1-6) can begin. The following are required:
|
||||
|
||||
1. PostgreSQL cluster operational
|
||||
2. Shared library published
|
||||
3. CI pipeline validated
|
||||
4. Configuration framework deployed
|
||||
|
||||
---
|
||||
|
||||
## Notes
|
||||
|
||||
- Use Orchestrator module as reference for all patterns
|
||||
- Prioritize getting CI pipeline working early
|
||||
- Document all configuration decisions
|
||||
|
||||
---
|
||||
|
||||
*Phase Version: 1.0.0*
|
||||
*Last Updated: 2025-11-28*
|
||||
495
docs/db/tasks/PHASE_1_AUTHORITY.md
Normal file
495
docs/db/tasks/PHASE_1_AUTHORITY.md
Normal file
@@ -0,0 +1,495 @@
|
||||
# Phase 1: Authority Module Conversion
|
||||
|
||||
**Sprint:** 2
|
||||
**Duration:** 1 sprint
|
||||
**Status:** TODO
|
||||
**Dependencies:** Phase 0 (Foundations)
|
||||
|
||||
---
|
||||
|
||||
## Objectives
|
||||
|
||||
1. Create `StellaOps.Authority.Storage.Postgres` project
|
||||
2. Implement full Authority schema in PostgreSQL
|
||||
3. Implement all repository interfaces
|
||||
4. Enable dual-write mode for validation
|
||||
5. Switch Authority to PostgreSQL-only after verification
|
||||
|
||||
---
|
||||
|
||||
## Deliverables
|
||||
|
||||
| Deliverable | Acceptance Criteria |
|
||||
|-------------|---------------------|
|
||||
| Authority schema | All tables created with indexes |
|
||||
| Repository implementations | All 9 interfaces implemented |
|
||||
| Dual-write wrapper | Optional, for safe rollout |
|
||||
| Integration tests | 100% coverage of CRUD operations |
|
||||
| Verification report | MongoDB vs PostgreSQL comparison passed |
|
||||
|
||||
---
|
||||
|
||||
## Schema Reference
|
||||
|
||||
See [SPECIFICATION.md](../SPECIFICATION.md) Section 5.1 for complete Authority schema.
|
||||
|
||||
**Tables:**
|
||||
- `authority.tenants`
|
||||
- `authority.users`
|
||||
- `authority.roles`
|
||||
- `authority.user_roles`
|
||||
- `authority.service_accounts`
|
||||
- `authority.clients`
|
||||
- `authority.scopes`
|
||||
- `authority.tokens`
|
||||
- `authority.revocations`
|
||||
- `authority.login_attempts`
|
||||
- `authority.licenses`
|
||||
- `authority.license_usage`
|
||||
|
||||
---
|
||||
|
||||
## Task Breakdown
|
||||
|
||||
### T1.1: Create Authority.Storage.Postgres Project
|
||||
|
||||
**Status:** TODO
|
||||
**Assignee:** TBD
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
**Description:**
|
||||
Create the PostgreSQL storage project for Authority module.
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T1.1.1: Create project `src/Authority/__Libraries/StellaOps.Authority.Storage.Postgres/`
|
||||
- [ ] T1.1.2: Add reference to `StellaOps.Infrastructure.Postgres`
|
||||
- [ ] T1.1.3: Add reference to `StellaOps.Authority.Core`
|
||||
- [ ] T1.1.4: Create `AuthorityDataSource` class
|
||||
- [ ] T1.1.5: Create `AuthorityPostgresOptions` class
|
||||
- [ ] T1.1.6: Create `ServiceCollectionExtensions.cs`
|
||||
|
||||
**Project Structure:**
|
||||
```
|
||||
src/Authority/__Libraries/StellaOps.Authority.Storage.Postgres/
|
||||
├── StellaOps.Authority.Storage.Postgres.csproj
|
||||
├── AuthorityDataSource.cs
|
||||
├── AuthorityPostgresOptions.cs
|
||||
├── Repositories/
|
||||
│ ├── PostgresUserRepository.cs
|
||||
│ ├── PostgresRoleRepository.cs
|
||||
│ ├── PostgresServiceAccountRepository.cs
|
||||
│ ├── PostgresClientRepository.cs
|
||||
│ ├── PostgresScopeRepository.cs
|
||||
│ ├── PostgresTokenRepository.cs
|
||||
│ ├── PostgresRevocationRepository.cs
|
||||
│ ├── PostgresLoginAttemptRepository.cs
|
||||
│ └── PostgresLicenseRepository.cs
|
||||
├── Migrations/
|
||||
│ └── V001_CreateAuthoritySchema.cs
|
||||
└── ServiceCollectionExtensions.cs
|
||||
```
|
||||
|
||||
**Verification:**
|
||||
- [ ] Project builds without errors
|
||||
- [ ] Can be referenced from Authority.WebService
|
||||
|
||||
---
|
||||
|
||||
### T1.2: Implement Schema Migrations
|
||||
|
||||
**Status:** TODO
|
||||
**Assignee:** TBD
|
||||
**Estimate:** 1 day
|
||||
|
||||
**Description:**
|
||||
Create PostgreSQL schema migration for Authority tables.
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T1.2.1: Create `V001_CreateAuthoritySchema` migration
|
||||
- [ ] T1.2.2: Include all tables from SPECIFICATION.md
|
||||
- [ ] T1.2.3: Include all indexes
|
||||
- [ ] T1.2.4: Add seed data for system roles/permissions
|
||||
- [ ] T1.2.5: Test migration idempotency
|
||||
|
||||
**Migration Implementation:**
|
||||
```csharp
|
||||
public sealed class V001_CreateAuthoritySchema : IPostgresMigration
|
||||
{
|
||||
public string Id => "V001_CreateAuthoritySchema";
|
||||
public string Description => "Create Authority schema with all tables and indexes";
|
||||
|
||||
public async Task UpAsync(NpgsqlConnection connection, CancellationToken ct)
|
||||
{
|
||||
await using var cmd = connection.CreateCommand();
|
||||
cmd.CommandText = AuthoritySchemaSql;
|
||||
await cmd.ExecuteNonQueryAsync(ct);
|
||||
}
|
||||
|
||||
public Task DownAsync(NpgsqlConnection connection, CancellationToken ct)
|
||||
=> throw new NotSupportedException("Rollback not supported for schema creation");
|
||||
|
||||
private const string AuthoritySchemaSql = """
|
||||
CREATE SCHEMA IF NOT EXISTS authority;
|
||||
|
||||
CREATE TABLE IF NOT EXISTS authority.tenants (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
code TEXT NOT NULL UNIQUE,
|
||||
display_name TEXT NOT NULL,
|
||||
status TEXT NOT NULL DEFAULT 'active'
|
||||
CHECK (status IN ('active', 'suspended', 'trial', 'terminated')),
|
||||
settings JSONB DEFAULT '{}',
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||
);
|
||||
|
||||
-- ... rest of schema from SPECIFICATION.md
|
||||
""";
|
||||
}
|
||||
```
|
||||
|
||||
**Verification:**
|
||||
- [ ] Migration creates all tables
|
||||
- [ ] Migration is idempotent
|
||||
- [ ] Indexes created correctly
|
||||
|
||||
---
|
||||
|
||||
### T1.3: Implement User Repository
|
||||
|
||||
**Status:** TODO
|
||||
**Assignee:** TBD
|
||||
**Estimate:** 1 day
|
||||
|
||||
**Description:**
|
||||
Implement `IUserRepository` for PostgreSQL.
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T1.3.1: Implement `GetByIdAsync`
|
||||
- [ ] T1.3.2: Implement `GetByUsernameAsync`
|
||||
- [ ] T1.3.3: Implement `GetBySubjectIdAsync`
|
||||
- [ ] T1.3.4: Implement `ListAsync` with pagination
|
||||
- [ ] T1.3.5: Implement `CreateAsync`
|
||||
- [ ] T1.3.6: Implement `UpdateAsync`
|
||||
- [ ] T1.3.7: Implement `DeleteAsync`
|
||||
- [ ] T1.3.8: Implement `GetRolesAsync`
|
||||
- [ ] T1.3.9: Implement `AssignRoleAsync`
|
||||
- [ ] T1.3.10: Implement `RevokeRoleAsync`
|
||||
- [ ] T1.3.11: Write integration tests
|
||||
|
||||
**Interface Reference:**
|
||||
```csharp
|
||||
public interface IUserRepository
|
||||
{
|
||||
Task<User?> GetByIdAsync(string tenantId, Guid userId, CancellationToken ct);
|
||||
Task<User?> GetByUsernameAsync(string tenantId, string username, CancellationToken ct);
|
||||
Task<User?> GetBySubjectIdAsync(Guid subjectId, CancellationToken ct);
|
||||
Task<PagedResult<User>> ListAsync(string tenantId, UserQuery query, CancellationToken ct);
|
||||
Task<User> CreateAsync(User user, CancellationToken ct);
|
||||
Task<User> UpdateAsync(User user, CancellationToken ct);
|
||||
Task<bool> DeleteAsync(string tenantId, Guid userId, CancellationToken ct);
|
||||
Task<IReadOnlyList<Role>> GetRolesAsync(string tenantId, Guid userId, CancellationToken ct);
|
||||
Task AssignRoleAsync(string tenantId, Guid userId, Guid roleId, CancellationToken ct);
|
||||
Task RevokeRoleAsync(string tenantId, Guid userId, Guid roleId, CancellationToken ct);
|
||||
}
|
||||
```
|
||||
|
||||
**Verification:**
|
||||
- [ ] All methods implemented
|
||||
- [ ] Integration tests pass
|
||||
- [ ] Tenant isolation verified
|
||||
|
||||
---
|
||||
|
||||
### T1.4: Implement Service Account Repository
|
||||
|
||||
**Status:** TODO
|
||||
**Assignee:** TBD
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
**Description:**
|
||||
Implement `IServiceAccountRepository` for PostgreSQL.
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T1.4.1: Implement `GetByIdAsync`
|
||||
- [ ] T1.4.2: Implement `GetByAccountIdAsync`
|
||||
- [ ] T1.4.3: Implement `ListAsync`
|
||||
- [ ] T1.4.4: Implement `CreateAsync`
|
||||
- [ ] T1.4.5: Implement `UpdateAsync`
|
||||
- [ ] T1.4.6: Implement `DeleteAsync`
|
||||
- [ ] T1.4.7: Write integration tests
|
||||
|
||||
**Verification:**
|
||||
- [ ] All methods implemented
|
||||
- [ ] Integration tests pass
|
||||
|
||||
---
|
||||
|
||||
### T1.5: Implement Client Repository
|
||||
|
||||
**Status:** TODO
|
||||
**Assignee:** TBD
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
**Description:**
|
||||
Implement `IClientRepository` for PostgreSQL (OpenIddict compatible).
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T1.5.1: Implement `GetByIdAsync`
|
||||
- [ ] T1.5.2: Implement `GetByClientIdAsync`
|
||||
- [ ] T1.5.3: Implement `ListAsync`
|
||||
- [ ] T1.5.4: Implement `CreateAsync`
|
||||
- [ ] T1.5.5: Implement `UpdateAsync`
|
||||
- [ ] T1.5.6: Implement `DeleteAsync`
|
||||
- [ ] T1.5.7: Write integration tests
|
||||
|
||||
**Verification:**
|
||||
- [ ] All methods implemented
|
||||
- [ ] Integration tests pass
|
||||
|
||||
---
|
||||
|
||||
### T1.6: Implement Token Repository
|
||||
|
||||
**Status:** TODO
|
||||
**Assignee:** TBD
|
||||
**Estimate:** 1 day
|
||||
|
||||
**Description:**
|
||||
Implement `ITokenRepository` for PostgreSQL.
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T1.6.1: Implement `GetByIdAsync`
|
||||
- [ ] T1.6.2: Implement `GetByHashAsync`
|
||||
- [ ] T1.6.3: Implement `CreateAsync`
|
||||
- [ ] T1.6.4: Implement `RevokeAsync`
|
||||
- [ ] T1.6.5: Implement `PruneExpiredAsync`
|
||||
- [ ] T1.6.6: Implement `GetActiveTokensAsync`
|
||||
- [ ] T1.6.7: Write integration tests
|
||||
|
||||
**Verification:**
|
||||
- [ ] All methods implemented
|
||||
- [ ] Token lookup by hash is fast
|
||||
- [ ] Expired token pruning works
|
||||
|
||||
---
|
||||
|
||||
### T1.7: Implement Remaining Repositories
|
||||
|
||||
**Status:** TODO
|
||||
**Assignee:** TBD
|
||||
**Estimate:** 1.5 days
|
||||
|
||||
**Description:**
|
||||
Implement remaining repository interfaces.
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T1.7.1: Implement `IRoleRepository`
|
||||
- [ ] T1.7.2: Implement `IScopeRepository`
|
||||
- [ ] T1.7.3: Implement `IRevocationRepository`
|
||||
- [ ] T1.7.4: Implement `ILoginAttemptRepository`
|
||||
- [ ] T1.7.5: Implement `ILicenseRepository`
|
||||
- [ ] T1.7.6: Write integration tests for all
|
||||
|
||||
**Verification:**
|
||||
- [ ] All repositories implemented
|
||||
- [ ] All integration tests pass
|
||||
|
||||
---
|
||||
|
||||
### T1.8: Add Configuration Switch
|
||||
|
||||
**Status:** TODO
|
||||
**Assignee:** TBD
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
**Description:**
|
||||
Add configuration-based backend selection for Authority.
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T1.8.1: Update `ServiceCollectionExtensions` in Authority.WebService
|
||||
- [ ] T1.8.2: Add conditional registration based on `Persistence:Authority`
|
||||
- [ ] T1.8.3: Test switching between Mongo and Postgres
|
||||
- [ ] T1.8.4: Document configuration options
|
||||
|
||||
**Implementation:**
|
||||
```csharp
|
||||
public static IServiceCollection AddAuthorityStorage(
|
||||
this IServiceCollection services,
|
||||
IConfiguration configuration)
|
||||
{
|
||||
var backend = configuration.GetValue<string>("Persistence:Authority") ?? "Mongo";
|
||||
|
||||
return backend.ToLowerInvariant() switch
|
||||
{
|
||||
"postgres" => services.AddAuthorityPostgresStorage(configuration),
|
||||
"mongo" => services.AddAuthorityMongoStorage(configuration),
|
||||
_ => throw new ArgumentException($"Unknown Authority backend: {backend}")
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
**Verification:**
|
||||
- [ ] Can switch between backends via configuration
|
||||
- [ ] Invalid configuration throws clear error
|
||||
|
||||
---
|
||||
|
||||
### T1.9: Implement Dual-Write Wrapper (Optional)
|
||||
|
||||
**Status:** TODO
|
||||
**Assignee:** TBD
|
||||
**Estimate:** 1 day
|
||||
|
||||
**Description:**
|
||||
Implement dual-write repository wrapper for safe migration.
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T1.9.1: Create `DualWriteUserRepository`
|
||||
- [ ] T1.9.2: Implement write-to-both logic
|
||||
- [ ] T1.9.3: Implement read-from-primary-with-fallback logic
|
||||
- [ ] T1.9.4: Add metrics for dual-write operations
|
||||
- [ ] T1.9.5: Add logging for inconsistencies
|
||||
- [ ] T1.9.6: Create similar wrappers for other critical repositories
|
||||
|
||||
**Configuration Options:**
|
||||
```csharp
|
||||
public sealed class DualWriteOptions
|
||||
{
|
||||
public string PrimaryBackend { get; set; } = "Postgres";
|
||||
public bool WriteToBoth { get; set; } = true;
|
||||
public bool FallbackToSecondary { get; set; } = true;
|
||||
public bool ConvertOnRead { get; set; } = true;
|
||||
}
|
||||
```
|
||||
|
||||
**Verification:**
|
||||
- [ ] Writes go to both backends
|
||||
- [ ] Reads work with fallback
|
||||
- [ ] Inconsistencies are logged
|
||||
|
||||
---
|
||||
|
||||
### T1.10: Run Verification Tests
|
||||
|
||||
**Status:** TODO
|
||||
**Assignee:** TBD
|
||||
**Estimate:** 1 day
|
||||
|
||||
**Description:**
|
||||
Verify PostgreSQL implementation matches MongoDB behavior.
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T1.10.1: Run comparison tests for User repository
|
||||
- [ ] T1.10.2: Run comparison tests for Token repository
|
||||
- [ ] T1.10.3: Verify token issuance/verification flow
|
||||
- [ ] T1.10.4: Verify login flow
|
||||
- [ ] T1.10.5: Document any differences found
|
||||
- [ ] T1.10.6: Generate verification report
|
||||
|
||||
**Verification Tests:**
|
||||
```csharp
|
||||
[Fact]
|
||||
public async Task Users_Should_Match_Between_Mongo_And_Postgres()
|
||||
{
|
||||
var tenantIds = await GetSampleTenantIds(10);
|
||||
|
||||
foreach (var tenantId in tenantIds)
|
||||
{
|
||||
var mongoUsers = await _mongoRepo.ListAsync(tenantId, new UserQuery());
|
||||
var postgresUsers = await _postgresRepo.ListAsync(tenantId, new UserQuery());
|
||||
|
||||
postgresUsers.Items.Should().BeEquivalentTo(mongoUsers.Items,
|
||||
options => options.Excluding(u => u.Id));
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Verification:**
|
||||
- [ ] All comparison tests pass
|
||||
- [ ] No data discrepancies found
|
||||
- [ ] Verification report approved
|
||||
|
||||
---
|
||||
|
||||
### T1.11: Backfill Data (If Required)
|
||||
|
||||
**Status:** TODO
|
||||
**Assignee:** TBD
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
**Description:**
|
||||
Backfill existing MongoDB data to PostgreSQL.
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T1.11.1: Create backfill script for tenants
|
||||
- [ ] T1.11.2: Create backfill script for users
|
||||
- [ ] T1.11.3: Create backfill script for service accounts
|
||||
- [ ] T1.11.4: Create backfill script for clients/scopes
|
||||
- [ ] T1.11.5: Create backfill script for active tokens
|
||||
- [ ] T1.11.6: Verify record counts match
|
||||
- [ ] T1.11.7: Verify sample records match
|
||||
|
||||
**Verification:**
|
||||
- [ ] All Tier A data backfilled
|
||||
- [ ] Record counts match
|
||||
- [ ] Sample verification passed
|
||||
|
||||
---
|
||||
|
||||
### T1.12: Switch to PostgreSQL-Only
|
||||
|
||||
**Status:** TODO
|
||||
**Assignee:** TBD
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
**Description:**
|
||||
Switch Authority to PostgreSQL-only mode.
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T1.12.1: Update configuration to `"Authority": "Postgres"`
|
||||
- [ ] T1.12.2: Deploy to staging
|
||||
- [ ] T1.12.3: Run full integration test suite
|
||||
- [ ] T1.12.4: Monitor for errors/issues
|
||||
- [ ] T1.12.5: Deploy to production
|
||||
- [ ] T1.12.6: Monitor production metrics
|
||||
|
||||
**Verification:**
|
||||
- [ ] All tests pass in staging
|
||||
- [ ] No errors in production
|
||||
- [ ] Performance metrics acceptable
|
||||
|
||||
---
|
||||
|
||||
## Exit Criteria
|
||||
|
||||
- [ ] All repository interfaces implemented for PostgreSQL
|
||||
- [ ] All integration tests pass
|
||||
- [ ] Verification tests pass (MongoDB vs PostgreSQL comparison)
|
||||
- [ ] Configuration switch working
|
||||
- [ ] Authority running on PostgreSQL in production
|
||||
- [ ] MongoDB Authority collections archived
|
||||
|
||||
---
|
||||
|
||||
## Risks & Mitigations
|
||||
|
||||
| Risk | Likelihood | Impact | Mitigation |
|
||||
|------|------------|--------|------------|
|
||||
| Token verification regression | Low | High | Extensive testing, dual-write |
|
||||
| OAuth flow breakage | Low | High | Test all OAuth flows |
|
||||
| Performance regression | Medium | Medium | Load testing before switch |
|
||||
|
||||
---
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
1. Change configuration: `"Authority": "Mongo"`
|
||||
2. Deploy configuration change
|
||||
3. MongoDB still has all data (dual-write period)
|
||||
4. Investigate and fix PostgreSQL issues
|
||||
5. Re-attempt conversion
|
||||
|
||||
---
|
||||
|
||||
*Phase Version: 1.0.0*
|
||||
*Last Updated: 2025-11-28*
|
||||
305
docs/db/tasks/PHASE_2_SCHEDULER.md
Normal file
305
docs/db/tasks/PHASE_2_SCHEDULER.md
Normal file
@@ -0,0 +1,305 @@
|
||||
# Phase 2: Scheduler Module Conversion
|
||||
|
||||
**Sprint:** 3
|
||||
**Duration:** 1 sprint
|
||||
**Status:** TODO
|
||||
**Dependencies:** Phase 0 (Foundations)
|
||||
|
||||
---
|
||||
|
||||
## Objectives
|
||||
|
||||
1. Create `StellaOps.Scheduler.Storage.Postgres` project
|
||||
2. Implement Scheduler schema in PostgreSQL
|
||||
3. Implement 7+ repository interfaces
|
||||
4. Replace MongoDB job tracking with PostgreSQL
|
||||
5. Implement PostgreSQL advisory locks for distributed locking
|
||||
|
||||
---
|
||||
|
||||
## Deliverables
|
||||
|
||||
| Deliverable | Acceptance Criteria |
|
||||
|-------------|---------------------|
|
||||
| Scheduler schema | All tables created with indexes |
|
||||
| Repository implementations | All 7+ interfaces implemented |
|
||||
| Advisory locks | Distributed locking working |
|
||||
| Integration tests | 100% coverage of CRUD operations |
|
||||
| Verification report | Schedule execution verified |
|
||||
|
||||
---
|
||||
|
||||
## Schema Reference
|
||||
|
||||
See [SPECIFICATION.md](../SPECIFICATION.md) Section 5.4 for complete Scheduler schema.
|
||||
|
||||
**Tables:**
|
||||
- `scheduler.schedules`
|
||||
- `scheduler.triggers`
|
||||
- `scheduler.runs`
|
||||
- `scheduler.graph_jobs`
|
||||
- `scheduler.policy_jobs`
|
||||
- `scheduler.impact_snapshots`
|
||||
- `scheduler.workers`
|
||||
- `scheduler.execution_logs`
|
||||
- `scheduler.locks`
|
||||
- `scheduler.run_summaries`
|
||||
- `scheduler.audit`
|
||||
|
||||
---
|
||||
|
||||
## Task Breakdown
|
||||
|
||||
### T2.1: Create Scheduler.Storage.Postgres Project
|
||||
|
||||
**Status:** TODO
|
||||
**Assignee:** TBD
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T2.1.1: Create project structure
|
||||
- [ ] T2.1.2: Add NuGet references
|
||||
- [ ] T2.1.3: Create `SchedulerDataSource` class
|
||||
- [ ] T2.1.4: Create `ServiceCollectionExtensions.cs`
|
||||
|
||||
---
|
||||
|
||||
### T2.2: Implement Schema Migrations
|
||||
|
||||
**Status:** TODO
|
||||
**Assignee:** TBD
|
||||
**Estimate:** 1 day
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T2.2.1: Create `V001_CreateSchedulerSchema` migration
|
||||
- [ ] T2.2.2: Include all tables and indexes
|
||||
- [ ] T2.2.3: Add partial index for active schedules
|
||||
- [ ] T2.2.4: Test migration idempotency
|
||||
|
||||
---
|
||||
|
||||
### T2.3: Implement Schedule Repository
|
||||
|
||||
**Status:** TODO
|
||||
**Assignee:** TBD
|
||||
**Estimate:** 1 day
|
||||
|
||||
**Interface:**
|
||||
```csharp
|
||||
public interface IScheduleRepository
|
||||
{
|
||||
Task<Schedule?> GetAsync(string tenantId, string scheduleId, CancellationToken ct);
|
||||
Task<IReadOnlyList<Schedule>> ListAsync(string tenantId, ScheduleQueryOptions? options, CancellationToken ct);
|
||||
Task UpsertAsync(Schedule schedule, CancellationToken ct);
|
||||
Task<bool> SoftDeleteAsync(string tenantId, string scheduleId, string deletedBy, DateTimeOffset deletedAt, CancellationToken ct);
|
||||
Task<IReadOnlyList<Schedule>> GetDueSchedulesAsync(DateTimeOffset now, CancellationToken ct);
|
||||
}
|
||||
```
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T2.3.1: Implement all interface methods
|
||||
- [ ] T2.3.2: Handle soft delete correctly
|
||||
- [ ] T2.3.3: Implement GetDueSchedules for trigger calculation
|
||||
- [ ] T2.3.4: Write integration tests
|
||||
|
||||
---
|
||||
|
||||
### T2.4: Implement Run Repository
|
||||
|
||||
**Status:** TODO
|
||||
**Assignee:** TBD
|
||||
**Estimate:** 1 day
|
||||
|
||||
**Interface:**
|
||||
```csharp
|
||||
public interface IRunRepository
|
||||
{
|
||||
Task<Run?> GetAsync(string tenantId, Guid runId, CancellationToken ct);
|
||||
Task<IReadOnlyList<Run>> ListAsync(string tenantId, RunQueryOptions? options, CancellationToken ct);
|
||||
Task<Run> CreateAsync(Run run, CancellationToken ct);
|
||||
Task<Run> UpdateAsync(Run run, CancellationToken ct);
|
||||
Task<IReadOnlyList<Run>> GetPendingRunsAsync(string tenantId, CancellationToken ct);
|
||||
Task<IReadOnlyList<Run>> GetRunsByScheduleAsync(string tenantId, Guid scheduleId, int limit, CancellationToken ct);
|
||||
}
|
||||
```
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T2.4.1: Implement all interface methods
|
||||
- [ ] T2.4.2: Handle state transitions
|
||||
- [ ] T2.4.3: Implement efficient pagination
|
||||
- [ ] T2.4.4: Write integration tests
|
||||
|
||||
---
|
||||
|
||||
### T2.5: Implement Graph Job Repository
|
||||
|
||||
**Status:** TODO
|
||||
**Assignee:** TBD
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T2.5.1: Implement CRUD operations
|
||||
- [ ] T2.5.2: Implement status queries
|
||||
- [ ] T2.5.3: Write integration tests
|
||||
|
||||
---
|
||||
|
||||
### T2.6: Implement Policy Job Repository
|
||||
|
||||
**Status:** TODO
|
||||
**Assignee:** TBD
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T2.6.1: Implement CRUD operations
|
||||
- [ ] T2.6.2: Implement status queries
|
||||
- [ ] T2.6.3: Write integration tests
|
||||
|
||||
---
|
||||
|
||||
### T2.7: Implement Impact Snapshot Repository
|
||||
|
||||
**Status:** TODO
|
||||
**Assignee:** TBD
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T2.7.1: Implement CRUD operations
|
||||
- [ ] T2.7.2: Implement queries by run
|
||||
- [ ] T2.7.3: Write integration tests
|
||||
|
||||
---
|
||||
|
||||
### T2.8: Implement Distributed Locking
|
||||
|
||||
**Status:** TODO
|
||||
**Assignee:** TBD
|
||||
**Estimate:** 1 day
|
||||
|
||||
**Description:**
|
||||
Implement distributed locking using PostgreSQL advisory locks.
|
||||
|
||||
**Options:**
|
||||
1. PostgreSQL advisory locks (`pg_advisory_lock`)
|
||||
2. Table-based locks with SELECT FOR UPDATE SKIP LOCKED
|
||||
3. Combination approach
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T2.8.1: Choose locking strategy
|
||||
- [ ] T2.8.2: Implement `IDistributedLock` interface
|
||||
- [ ] T2.8.3: Implement lock acquisition with timeout
|
||||
- [ ] T2.8.4: Implement lock renewal
|
||||
- [ ] T2.8.5: Implement lock release
|
||||
- [ ] T2.8.6: Write concurrency tests
|
||||
|
||||
**Implementation Example:**
|
||||
```csharp
|
||||
public sealed class PostgresDistributedLock : IDistributedLock
|
||||
{
|
||||
private readonly SchedulerDataSource _dataSource;
|
||||
|
||||
public async Task<IAsyncDisposable?> TryAcquireAsync(
|
||||
string lockKey,
|
||||
TimeSpan timeout,
|
||||
CancellationToken ct)
|
||||
{
|
||||
var lockId = ComputeLockId(lockKey);
|
||||
await using var connection = await _dataSource.OpenConnectionAsync("system", ct);
|
||||
|
||||
await using var cmd = connection.CreateCommand();
|
||||
cmd.CommandText = "SELECT pg_try_advisory_lock(@lock_id)";
|
||||
cmd.Parameters.AddWithValue("lock_id", lockId);
|
||||
|
||||
var acquired = await cmd.ExecuteScalarAsync(ct) is true;
|
||||
if (!acquired) return null;
|
||||
|
||||
return new LockHandle(connection, lockId);
|
||||
}
|
||||
|
||||
private static long ComputeLockId(string key)
|
||||
=> unchecked((long)key.GetHashCode());
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### T2.9: Implement Worker Registration
|
||||
|
||||
**Status:** TODO
|
||||
**Assignee:** TBD
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T2.9.1: Implement worker registration
|
||||
- [ ] T2.9.2: Implement heartbeat updates
|
||||
- [ ] T2.9.3: Implement dead worker detection
|
||||
- [ ] T2.9.4: Write integration tests
|
||||
|
||||
---
|
||||
|
||||
### T2.10: Add Configuration Switch
|
||||
|
||||
**Status:** TODO
|
||||
**Assignee:** TBD
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T2.10.1: Update service registration
|
||||
- [ ] T2.10.2: Test backend switching
|
||||
- [ ] T2.10.3: Document configuration
|
||||
|
||||
---
|
||||
|
||||
### T2.11: Run Verification Tests
|
||||
|
||||
**Status:** TODO
|
||||
**Assignee:** TBD
|
||||
**Estimate:** 1 day
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T2.11.1: Test schedule CRUD
|
||||
- [ ] T2.11.2: Test run creation and state transitions
|
||||
- [ ] T2.11.3: Test trigger calculation
|
||||
- [ ] T2.11.4: Test distributed locking under concurrency
|
||||
- [ ] T2.11.5: Test job execution end-to-end
|
||||
- [ ] T2.11.6: Generate verification report
|
||||
|
||||
---
|
||||
|
||||
### T2.12: Switch to PostgreSQL-Only
|
||||
|
||||
**Status:** TODO
|
||||
**Assignee:** TBD
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T2.12.1: Update configuration
|
||||
- [ ] T2.12.2: Deploy to staging
|
||||
- [ ] T2.12.3: Run integration tests
|
||||
- [ ] T2.12.4: Deploy to production
|
||||
- [ ] T2.12.5: Monitor metrics
|
||||
|
||||
---
|
||||
|
||||
## Exit Criteria
|
||||
|
||||
- [ ] All repository interfaces implemented
|
||||
- [ ] Distributed locking working correctly
|
||||
- [ ] All integration tests pass
|
||||
- [ ] Schedule execution working end-to-end
|
||||
- [ ] Scheduler running on PostgreSQL in production
|
||||
|
||||
---
|
||||
|
||||
## Risks & Mitigations
|
||||
|
||||
| Risk | Likelihood | Impact | Mitigation |
|
||||
|------|------------|--------|------------|
|
||||
| Lock contention | Medium | Medium | Test under load, tune timeouts |
|
||||
| Trigger calculation errors | Low | High | Extensive testing with edge cases |
|
||||
| State transition bugs | Medium | Medium | State machine tests |
|
||||
|
||||
---
|
||||
|
||||
*Phase Version: 1.0.0*
|
||||
*Last Updated: 2025-11-28*
|
||||
183
docs/db/tasks/PHASE_3_NOTIFY.md
Normal file
183
docs/db/tasks/PHASE_3_NOTIFY.md
Normal file
@@ -0,0 +1,183 @@
|
||||
# Phase 3: Notify Module Conversion
|
||||
|
||||
**Sprint:** 4
|
||||
**Duration:** 1 sprint
|
||||
**Status:** TODO
|
||||
**Dependencies:** Phase 0 (Foundations)
|
||||
|
||||
---
|
||||
|
||||
## Objectives
|
||||
|
||||
1. Create `StellaOps.Notify.Storage.Postgres` project
|
||||
2. Implement Notify schema in PostgreSQL
|
||||
3. Implement 15 repository interfaces
|
||||
4. Handle delivery tracking and escalation state
|
||||
|
||||
---
|
||||
|
||||
## Deliverables
|
||||
|
||||
| Deliverable | Acceptance Criteria |
|
||||
|-------------|---------------------|
|
||||
| Notify schema | All tables created with indexes |
|
||||
| Repository implementations | All 15 interfaces implemented |
|
||||
| Integration tests | 100% coverage of CRUD operations |
|
||||
| Verification report | Notification delivery verified |
|
||||
|
||||
---
|
||||
|
||||
## Schema Reference
|
||||
|
||||
See [SPECIFICATION.md](../SPECIFICATION.md) Section 5.5 for complete Notify schema.
|
||||
|
||||
**Tables:**
|
||||
- `notify.channels`
|
||||
- `notify.rules`
|
||||
- `notify.templates`
|
||||
- `notify.deliveries`
|
||||
- `notify.digests`
|
||||
- `notify.quiet_hours`
|
||||
- `notify.maintenance_windows`
|
||||
- `notify.escalation_policies`
|
||||
- `notify.escalation_states`
|
||||
- `notify.on_call_schedules`
|
||||
- `notify.inbox`
|
||||
- `notify.incidents`
|
||||
- `notify.audit`
|
||||
|
||||
---
|
||||
|
||||
## Task Breakdown
|
||||
|
||||
### T3.1: Create Notify.Storage.Postgres Project
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Create project structure
|
||||
- [ ] Add NuGet references
|
||||
- [ ] Create `NotifyDataSource` class
|
||||
- [ ] Create `ServiceCollectionExtensions.cs`
|
||||
|
||||
---
|
||||
|
||||
### T3.2: Implement Schema Migrations
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 1 day
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Create schema migration
|
||||
- [ ] Include all tables and indexes
|
||||
- [ ] Test migration idempotency
|
||||
|
||||
---
|
||||
|
||||
### T3.3: Implement Channel Repository
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Implement CRUD operations
|
||||
- [ ] Handle channel types (email, slack, teams, etc.)
|
||||
- [ ] Write integration tests
|
||||
|
||||
---
|
||||
|
||||
### T3.4: Implement Rule Repository
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Implement CRUD operations
|
||||
- [ ] Handle filter JSONB
|
||||
- [ ] Write integration tests
|
||||
|
||||
---
|
||||
|
||||
### T3.5: Implement Template Repository
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Implement CRUD operations
|
||||
- [ ] Handle localization
|
||||
- [ ] Write integration tests
|
||||
|
||||
---
|
||||
|
||||
### T3.6: Implement Delivery Repository
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 1 day
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Implement CRUD operations
|
||||
- [ ] Handle status transitions
|
||||
- [ ] Implement retry logic
|
||||
- [ ] Write integration tests
|
||||
|
||||
---
|
||||
|
||||
### T3.7: Implement Remaining Repositories
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 2 days
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Implement Digest repository
|
||||
- [ ] Implement QuietHours repository
|
||||
- [ ] Implement MaintenanceWindow repository
|
||||
- [ ] Implement EscalationPolicy repository
|
||||
- [ ] Implement EscalationState repository
|
||||
- [ ] Implement OnCallSchedule repository
|
||||
- [ ] Implement Inbox repository
|
||||
- [ ] Implement Incident repository
|
||||
- [ ] Implement Audit repository
|
||||
- [ ] Write integration tests for all
|
||||
|
||||
---
|
||||
|
||||
### T3.8: Add Configuration Switch
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
---
|
||||
|
||||
### T3.9: Run Verification Tests
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 1 day
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Test notification delivery flow
|
||||
- [ ] Test escalation handling
|
||||
- [ ] Test digest aggregation
|
||||
- [ ] Generate verification report
|
||||
|
||||
---
|
||||
|
||||
### T3.10: Switch to PostgreSQL-Only
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
---
|
||||
|
||||
## Exit Criteria
|
||||
|
||||
- [ ] All 15 repository interfaces implemented
|
||||
- [ ] All integration tests pass
|
||||
- [ ] Notification delivery working end-to-end
|
||||
- [ ] Notify running on PostgreSQL in production
|
||||
|
||||
---
|
||||
|
||||
*Phase Version: 1.0.0*
|
||||
*Last Updated: 2025-11-28*
|
||||
147
docs/db/tasks/PHASE_4_POLICY.md
Normal file
147
docs/db/tasks/PHASE_4_POLICY.md
Normal file
@@ -0,0 +1,147 @@
|
||||
# Phase 4: Policy Module Conversion
|
||||
|
||||
**Sprint:** 5
|
||||
**Duration:** 1 sprint
|
||||
**Status:** TODO
|
||||
**Dependencies:** Phase 0 (Foundations)
|
||||
|
||||
---
|
||||
|
||||
## Objectives
|
||||
|
||||
1. Create `StellaOps.Policy.Storage.Postgres` project
|
||||
2. Implement Policy schema in PostgreSQL
|
||||
3. Handle policy pack versioning correctly
|
||||
4. Implement risk profiles with version history
|
||||
|
||||
---
|
||||
|
||||
## Deliverables
|
||||
|
||||
| Deliverable | Acceptance Criteria |
|
||||
|-------------|---------------------|
|
||||
| Policy schema | All tables created with indexes |
|
||||
| Repository implementations | All 4+ interfaces implemented |
|
||||
| Version management | Pack versioning working correctly |
|
||||
| Integration tests | 100% coverage of CRUD operations |
|
||||
|
||||
---
|
||||
|
||||
## Schema Reference
|
||||
|
||||
See [SPECIFICATION.md](../SPECIFICATION.md) Section 5.6 for complete Policy schema.
|
||||
|
||||
**Tables:**
|
||||
- `policy.packs`
|
||||
- `policy.pack_versions`
|
||||
- `policy.rules`
|
||||
- `policy.risk_profiles`
|
||||
- `policy.evaluation_runs`
|
||||
- `policy.explanations`
|
||||
- `policy.exceptions`
|
||||
- `policy.audit`
|
||||
|
||||
---
|
||||
|
||||
## Task Breakdown
|
||||
|
||||
### T4.1: Create Policy.Storage.Postgres Project
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
---
|
||||
|
||||
### T4.2: Implement Schema Migrations
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 1 day
|
||||
|
||||
---
|
||||
|
||||
### T4.3: Implement Pack Repository
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 1 day
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Implement CRUD for packs
|
||||
- [ ] Implement version management
|
||||
- [ ] Handle active version promotion
|
||||
- [ ] Write integration tests
|
||||
|
||||
---
|
||||
|
||||
### T4.4: Implement Risk Profile Repository
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 1 day
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Implement CRUD operations
|
||||
- [ ] Handle version history
|
||||
- [ ] Implement GetVersionAsync
|
||||
- [ ] Implement ListVersionsAsync
|
||||
- [ ] Write integration tests
|
||||
|
||||
---
|
||||
|
||||
### T4.5: Implement Remaining Repositories
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 1.5 days
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Implement Evaluation Run repository
|
||||
- [ ] Implement Explanation repository
|
||||
- [ ] Implement Exception repository
|
||||
- [ ] Implement Audit repository
|
||||
- [ ] Write integration tests
|
||||
|
||||
---
|
||||
|
||||
### T4.6: Add Configuration Switch
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
---
|
||||
|
||||
### T4.7: Run Verification Tests
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 1 day
|
||||
|
||||
---
|
||||
|
||||
### T4.8: Migrate Active Policy Packs
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Export active packs from MongoDB
|
||||
- [ ] Import to PostgreSQL
|
||||
- [ ] Verify version numbers
|
||||
- [ ] Verify active version settings
|
||||
|
||||
---
|
||||
|
||||
### T4.9: Switch to PostgreSQL-Only
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
---
|
||||
|
||||
## Exit Criteria
|
||||
|
||||
- [ ] All repository interfaces implemented
|
||||
- [ ] Pack versioning working correctly
|
||||
- [ ] All integration tests pass
|
||||
- [ ] Policy running on PostgreSQL in production
|
||||
|
||||
---
|
||||
|
||||
*Phase Version: 1.0.0*
|
||||
*Last Updated: 2025-11-28*
|
||||
334
docs/db/tasks/PHASE_5_VULNERABILITIES.md
Normal file
334
docs/db/tasks/PHASE_5_VULNERABILITIES.md
Normal file
@@ -0,0 +1,334 @@
|
||||
# Phase 5: Vulnerability Index Conversion (Concelier)
|
||||
|
||||
**Sprint:** 6-7
|
||||
**Duration:** 2 sprints
|
||||
**Status:** TODO
|
||||
**Dependencies:** Phase 0 (Foundations)
|
||||
|
||||
---
|
||||
|
||||
## Objectives
|
||||
|
||||
1. Create `StellaOps.Concelier.Storage.Postgres` project
|
||||
2. Implement full vulnerability schema in PostgreSQL
|
||||
3. Build advisory conversion pipeline
|
||||
4. Maintain deterministic vulnerability matching
|
||||
|
||||
---
|
||||
|
||||
## Deliverables
|
||||
|
||||
| Deliverable | Acceptance Criteria |
|
||||
|-------------|---------------------|
|
||||
| Vuln schema | All tables created with indexes |
|
||||
| Conversion pipeline | MongoDB advisories converted to PostgreSQL |
|
||||
| Matching verification | Same CVEs found for identical SBOMs |
|
||||
| Integration tests | 100% coverage of query operations |
|
||||
|
||||
---
|
||||
|
||||
## Schema Reference
|
||||
|
||||
See [SPECIFICATION.md](../SPECIFICATION.md) Section 5.2 for complete vulnerability schema.
|
||||
|
||||
**Tables:**
|
||||
- `vuln.sources`
|
||||
- `vuln.feed_snapshots`
|
||||
- `vuln.advisory_snapshots`
|
||||
- `vuln.advisories`
|
||||
- `vuln.advisory_aliases`
|
||||
- `vuln.advisory_cvss`
|
||||
- `vuln.advisory_affected`
|
||||
- `vuln.advisory_references`
|
||||
- `vuln.advisory_credits`
|
||||
- `vuln.advisory_weaknesses`
|
||||
- `vuln.kev_flags`
|
||||
- `vuln.source_states`
|
||||
- `vuln.merge_events`
|
||||
|
||||
---
|
||||
|
||||
## Sprint 5a: Schema & Repositories
|
||||
|
||||
### T5a.1: Create Concelier.Storage.Postgres Project
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Create project structure
|
||||
- [ ] Add NuGet references
|
||||
- [ ] Create `ConcelierDataSource` class
|
||||
- [ ] Create `ServiceCollectionExtensions.cs`
|
||||
|
||||
---
|
||||
|
||||
### T5a.2: Implement Schema Migrations
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 1.5 days
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Create schema migration
|
||||
- [ ] Include all tables
|
||||
- [ ] Add full-text search index
|
||||
- [ ] Add PURL lookup index
|
||||
- [ ] Test migration idempotency
|
||||
|
||||
---
|
||||
|
||||
### T5a.3: Implement Source Repository
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Implement CRUD operations
|
||||
- [ ] Implement GetByKeyAsync
|
||||
- [ ] Write integration tests
|
||||
|
||||
---
|
||||
|
||||
### T5a.4: Implement Advisory Repository
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 2 days
|
||||
|
||||
**Interface:**
|
||||
```csharp
|
||||
public interface IAdvisoryRepository
|
||||
{
|
||||
Task<Advisory?> GetByKeyAsync(string advisoryKey, CancellationToken ct);
|
||||
Task<Advisory?> GetByAliasAsync(string aliasType, string aliasValue, CancellationToken ct);
|
||||
Task<IReadOnlyList<Advisory>> SearchAsync(AdvisorySearchQuery query, CancellationToken ct);
|
||||
Task<Advisory> UpsertAsync(Advisory advisory, CancellationToken ct);
|
||||
Task<IReadOnlyList<Advisory>> GetAffectingPackageAsync(string purl, CancellationToken ct);
|
||||
Task<IReadOnlyList<Advisory>> GetAffectingPackageNameAsync(string ecosystem, string name, CancellationToken ct);
|
||||
}
|
||||
```
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Implement GetByKeyAsync
|
||||
- [ ] Implement GetByAliasAsync (CVE lookup)
|
||||
- [ ] Implement SearchAsync with full-text search
|
||||
- [ ] Implement UpsertAsync with all child tables
|
||||
- [ ] Implement GetAffectingPackageAsync (PURL match)
|
||||
- [ ] Implement GetAffectingPackageNameAsync
|
||||
- [ ] Write integration tests
|
||||
|
||||
---
|
||||
|
||||
### T5a.5: Implement Child Table Repositories
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 2 days
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Implement Alias repository
|
||||
- [ ] Implement CVSS repository
|
||||
- [ ] Implement Affected repository
|
||||
- [ ] Implement Reference repository
|
||||
- [ ] Implement Credit repository
|
||||
- [ ] Implement Weakness repository
|
||||
- [ ] Implement KEV repository
|
||||
- [ ] Write integration tests
|
||||
|
||||
---
|
||||
|
||||
### T5a.6: Implement Source State Repository
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Implement CRUD operations
|
||||
- [ ] Implement cursor management
|
||||
- [ ] Write integration tests
|
||||
|
||||
---
|
||||
|
||||
## Sprint 5b: Conversion & Verification
|
||||
|
||||
### T5b.1: Build Advisory Conversion Service
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 2 days
|
||||
|
||||
**Description:**
|
||||
Create service to convert MongoDB advisory documents to PostgreSQL relational structure.
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Parse MongoDB `AdvisoryDocument` structure
|
||||
- [ ] Map to `vuln.advisories` table
|
||||
- [ ] Extract and normalize aliases
|
||||
- [ ] Extract and normalize CVSS metrics
|
||||
- [ ] Extract and normalize affected packages
|
||||
- [ ] Preserve provenance JSONB
|
||||
- [ ] Handle version ranges (keep as JSONB)
|
||||
- [ ] Handle normalized versions (keep as JSONB)
|
||||
|
||||
**Conversion Logic:**
|
||||
```csharp
|
||||
public sealed class AdvisoryConverter
|
||||
{
|
||||
public async Task ConvertAsync(
|
||||
IMongoCollection<AdvisoryDocument> source,
|
||||
IAdvisoryRepository target,
|
||||
CancellationToken ct)
|
||||
{
|
||||
await foreach (var doc in source.AsAsyncEnumerable(ct))
|
||||
{
|
||||
var advisory = MapToAdvisory(doc);
|
||||
await target.UpsertAsync(advisory, ct);
|
||||
}
|
||||
}
|
||||
|
||||
private Advisory MapToAdvisory(AdvisoryDocument doc)
|
||||
{
|
||||
// Extract from BsonDocument payload
|
||||
var payload = doc.Payload;
|
||||
return new Advisory
|
||||
{
|
||||
AdvisoryKey = doc.Id,
|
||||
PrimaryVulnId = payload["primaryVulnId"].AsString,
|
||||
Title = payload["title"]?.AsString,
|
||||
Summary = payload["summary"]?.AsString,
|
||||
// ... etc
|
||||
Provenance = BsonSerializer.Deserialize<JsonElement>(payload["provenance"]),
|
||||
};
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### T5b.2: Build Feed Import Pipeline
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 1 day
|
||||
|
||||
**Description:**
|
||||
Modify feed import to write directly to PostgreSQL.
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Update NVD importer to use PostgreSQL
|
||||
- [ ] Update OSV importer to use PostgreSQL
|
||||
- [ ] Update GHSA importer to use PostgreSQL
|
||||
- [ ] Update vendor feed importers
|
||||
- [ ] Test incremental imports
|
||||
|
||||
---
|
||||
|
||||
### T5b.3: Run Parallel Import
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 1 day
|
||||
|
||||
**Description:**
|
||||
Run imports to both MongoDB and PostgreSQL simultaneously.
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Configure dual-import mode
|
||||
- [ ] Run import cycle
|
||||
- [ ] Compare record counts
|
||||
- [ ] Sample comparison checks
|
||||
|
||||
---
|
||||
|
||||
### T5b.4: Verify Vulnerability Matching
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 2 days
|
||||
|
||||
**Description:**
|
||||
Verify that vulnerability matching produces identical results.
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Select sample SBOMs (various ecosystems)
|
||||
- [ ] Run matching with MongoDB backend
|
||||
- [ ] Run matching with PostgreSQL backend
|
||||
- [ ] Compare findings (must be identical)
|
||||
- [ ] Document any differences
|
||||
- [ ] Fix any issues found
|
||||
|
||||
**Verification Tests:**
|
||||
```csharp
|
||||
[Theory]
|
||||
[MemberData(nameof(GetSampleSboms))]
|
||||
public async Task Scanner_Should_Find_Same_Vulns(string sbomPath)
|
||||
{
|
||||
var sbom = await LoadSbom(sbomPath);
|
||||
|
||||
_config["Persistence:Concelier"] = "Mongo";
|
||||
var mongoFindings = await _scanner.ScanAsync(sbom);
|
||||
|
||||
_config["Persistence:Concelier"] = "Postgres";
|
||||
var postgresFindings = await _scanner.ScanAsync(sbom);
|
||||
|
||||
// Strict ordering for determinism
|
||||
postgresFindings.Should().BeEquivalentTo(mongoFindings,
|
||||
options => options.WithStrictOrdering());
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### T5b.5: Performance Optimization
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 1 day
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Analyze slow queries with EXPLAIN ANALYZE
|
||||
- [ ] Optimize indexes for common queries
|
||||
- [ ] Consider partial indexes for active advisories
|
||||
- [ ] Benchmark PostgreSQL vs MongoDB performance
|
||||
|
||||
---
|
||||
|
||||
### T5b.6: Switch Scanner to PostgreSQL
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Update configuration
|
||||
- [ ] Deploy to staging
|
||||
- [ ] Run full scan suite
|
||||
- [ ] Deploy to production
|
||||
|
||||
---
|
||||
|
||||
## Exit Criteria
|
||||
|
||||
- [ ] All repository interfaces implemented
|
||||
- [ ] Advisory conversion pipeline working
|
||||
- [ ] Vulnerability matching produces identical results
|
||||
- [ ] Feed imports working on PostgreSQL
|
||||
- [ ] Concelier running on PostgreSQL in production
|
||||
|
||||
---
|
||||
|
||||
## Risks & Mitigations
|
||||
|
||||
| Risk | Likelihood | Impact | Mitigation |
|
||||
|------|------------|--------|------------|
|
||||
| Matching discrepancies | Medium | High | Extensive comparison testing |
|
||||
| Performance regression on queries | Medium | Medium | Index optimization, query tuning |
|
||||
| Data loss during conversion | Low | High | Verify counts, sample checks |
|
||||
|
||||
---
|
||||
|
||||
## Data Volume Estimates
|
||||
|
||||
| Table | Estimated Rows | Growth Rate |
|
||||
|-------|----------------|-------------|
|
||||
| advisories | 300,000+ | ~100/day |
|
||||
| advisory_aliases | 600,000+ | ~200/day |
|
||||
| advisory_affected | 2,000,000+ | ~1000/day |
|
||||
| advisory_cvss | 400,000+ | ~150/day |
|
||||
|
||||
---
|
||||
|
||||
*Phase Version: 1.0.0*
|
||||
*Last Updated: 2025-11-28*
|
||||
434
docs/db/tasks/PHASE_6_VEX_GRAPH.md
Normal file
434
docs/db/tasks/PHASE_6_VEX_GRAPH.md
Normal file
@@ -0,0 +1,434 @@
|
||||
# Phase 6: VEX & Graph Conversion (Excititor)
|
||||
|
||||
**Sprint:** 8-10
|
||||
**Duration:** 2-3 sprints
|
||||
**Status:** TODO
|
||||
**Dependencies:** Phase 5 (Vulnerabilities)
|
||||
|
||||
---
|
||||
|
||||
## Objectives
|
||||
|
||||
1. Create `StellaOps.Excititor.Storage.Postgres` project
|
||||
2. Implement VEX schema in PostgreSQL
|
||||
3. Handle graph nodes/edges efficiently
|
||||
4. Preserve graph_revision_id stability (determinism critical)
|
||||
5. Maintain VEX statement lattice logic
|
||||
|
||||
---
|
||||
|
||||
## Deliverables
|
||||
|
||||
| Deliverable | Acceptance Criteria |
|
||||
|-------------|---------------------|
|
||||
| VEX schema | All tables created with indexes |
|
||||
| Graph storage | Nodes/edges efficiently stored |
|
||||
| Statement storage | VEX statements with full provenance |
|
||||
| Revision stability | Same inputs produce same revision_id |
|
||||
| Integration tests | 100% coverage |
|
||||
|
||||
---
|
||||
|
||||
## Schema Reference
|
||||
|
||||
See [SPECIFICATION.md](../SPECIFICATION.md) Section 5.3 for complete VEX schema.
|
||||
|
||||
**Tables:**
|
||||
- `vex.projects`
|
||||
- `vex.graph_revisions`
|
||||
- `vex.graph_nodes`
|
||||
- `vex.graph_edges`
|
||||
- `vex.statements`
|
||||
- `vex.observations`
|
||||
- `vex.linksets`
|
||||
- `vex.linkset_events`
|
||||
- `vex.consensus`
|
||||
- `vex.consensus_holds`
|
||||
- `vex.unknowns_snapshots`
|
||||
- `vex.unknown_items`
|
||||
- `vex.evidence_manifests`
|
||||
- `vex.cvss_receipts`
|
||||
- `vex.attestations`
|
||||
- `vex.timeline_events`
|
||||
|
||||
---
|
||||
|
||||
## Sprint 6a: Core Schema & Repositories
|
||||
|
||||
### T6a.1: Create Excititor.Storage.Postgres Project
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Create project structure
|
||||
- [ ] Add NuGet references
|
||||
- [ ] Create `ExcititorDataSource` class
|
||||
- [ ] Create `ServiceCollectionExtensions.cs`
|
||||
|
||||
---
|
||||
|
||||
### T6a.2: Implement Schema Migrations
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 1.5 days
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Create schema migration
|
||||
- [ ] Include all tables
|
||||
- [ ] Add indexes for graph traversal
|
||||
- [ ] Add indexes for VEX lookups
|
||||
- [ ] Test migration idempotency
|
||||
|
||||
---
|
||||
|
||||
### T6a.3: Implement Project Repository
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Implement CRUD operations
|
||||
- [ ] Handle tenant scoping
|
||||
- [ ] Write integration tests
|
||||
|
||||
---
|
||||
|
||||
### T6a.4: Implement VEX Statement Repository
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 1.5 days
|
||||
|
||||
**Interface:**
|
||||
```csharp
|
||||
public interface IVexStatementRepository
|
||||
{
|
||||
Task<VexStatement?> GetAsync(string tenantId, Guid statementId, CancellationToken ct);
|
||||
Task<IReadOnlyList<VexStatement>> GetByVulnerabilityAsync(
|
||||
string tenantId, string vulnerabilityId, CancellationToken ct);
|
||||
Task<IReadOnlyList<VexStatement>> GetByProjectAsync(
|
||||
string tenantId, Guid projectId, CancellationToken ct);
|
||||
Task<VexStatement> UpsertAsync(VexStatement statement, CancellationToken ct);
|
||||
Task<IReadOnlyList<VexStatement>> GetByGraphRevisionAsync(
|
||||
Guid graphRevisionId, CancellationToken ct);
|
||||
}
|
||||
```
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Implement all interface methods
|
||||
- [ ] Handle status and justification enums
|
||||
- [ ] Preserve evidence JSONB
|
||||
- [ ] Preserve provenance JSONB
|
||||
- [ ] Write integration tests
|
||||
|
||||
---
|
||||
|
||||
### T6a.5: Implement VEX Observation Repository
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 1 day
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Implement CRUD operations
|
||||
- [ ] Handle unique constraint on composite key
|
||||
- [ ] Implement FindByVulnerabilityAndProductAsync
|
||||
- [ ] Write integration tests
|
||||
|
||||
---
|
||||
|
||||
### T6a.6: Implement Linkset Repository
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Implement CRUD operations
|
||||
- [ ] Implement event logging
|
||||
- [ ] Write integration tests
|
||||
|
||||
---
|
||||
|
||||
### T6a.7: Implement Consensus Repository
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Implement CRUD operations
|
||||
- [ ] Implement hold management
|
||||
- [ ] Write integration tests
|
||||
|
||||
---
|
||||
|
||||
## Sprint 6b: Graph Storage
|
||||
|
||||
### T6b.1: Implement Graph Revision Repository
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 1 day
|
||||
|
||||
**Interface:**
|
||||
```csharp
|
||||
public interface IGraphRevisionRepository
|
||||
{
|
||||
Task<GraphRevision?> GetByIdAsync(Guid id, CancellationToken ct);
|
||||
Task<GraphRevision?> GetByRevisionIdAsync(string revisionId, CancellationToken ct);
|
||||
Task<GraphRevision?> GetLatestByProjectAsync(Guid projectId, CancellationToken ct);
|
||||
Task<GraphRevision> CreateAsync(GraphRevision revision, CancellationToken ct);
|
||||
Task<IReadOnlyList<GraphRevision>> GetHistoryAsync(
|
||||
Guid projectId, int limit, CancellationToken ct);
|
||||
}
|
||||
```
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Implement all interface methods
|
||||
- [ ] Handle revision_id uniqueness
|
||||
- [ ] Handle parent_revision_id linking
|
||||
- [ ] Write integration tests
|
||||
|
||||
---
|
||||
|
||||
### T6b.2: Implement Graph Node Repository
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 1.5 days
|
||||
|
||||
**Interface:**
|
||||
```csharp
|
||||
public interface IGraphNodeRepository
|
||||
{
|
||||
Task<GraphNode?> GetByIdAsync(long nodeId, CancellationToken ct);
|
||||
Task<GraphNode?> GetByKeyAsync(Guid graphRevisionId, string nodeKey, CancellationToken ct);
|
||||
Task<IReadOnlyList<GraphNode>> GetByRevisionAsync(
|
||||
Guid graphRevisionId, CancellationToken ct);
|
||||
Task BulkInsertAsync(
|
||||
Guid graphRevisionId, IEnumerable<GraphNode> nodes, CancellationToken ct);
|
||||
Task<int> GetCountAsync(Guid graphRevisionId, CancellationToken ct);
|
||||
}
|
||||
```
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Implement all interface methods
|
||||
- [ ] Implement bulk insert for efficiency
|
||||
- [ ] Handle node_key uniqueness per revision
|
||||
- [ ] Write integration tests
|
||||
|
||||
**Bulk Insert Optimization:**
|
||||
```csharp
|
||||
public async Task BulkInsertAsync(
|
||||
Guid graphRevisionId,
|
||||
IEnumerable<GraphNode> nodes,
|
||||
CancellationToken ct)
|
||||
{
|
||||
await using var connection = await _dataSource.OpenConnectionAsync("system", ct);
|
||||
await using var writer = await connection.BeginBinaryImportAsync(
|
||||
"COPY vex.graph_nodes (graph_revision_id, node_key, node_type, purl, name, version, attributes) " +
|
||||
"FROM STDIN (FORMAT BINARY)", ct);
|
||||
|
||||
foreach (var node in nodes)
|
||||
{
|
||||
await writer.StartRowAsync(ct);
|
||||
await writer.WriteAsync(graphRevisionId, ct);
|
||||
await writer.WriteAsync(node.NodeKey, ct);
|
||||
await writer.WriteAsync(node.NodeType, ct);
|
||||
await writer.WriteAsync(node.Purl, NpgsqlDbType.Text, ct);
|
||||
await writer.WriteAsync(node.Name, NpgsqlDbType.Text, ct);
|
||||
await writer.WriteAsync(node.Version, NpgsqlDbType.Text, ct);
|
||||
await writer.WriteAsync(JsonSerializer.Serialize(node.Attributes), NpgsqlDbType.Jsonb, ct);
|
||||
}
|
||||
|
||||
await writer.CompleteAsync(ct);
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### T6b.3: Implement Graph Edge Repository
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 1.5 days
|
||||
|
||||
**Interface:**
|
||||
```csharp
|
||||
public interface IGraphEdgeRepository
|
||||
{
|
||||
Task<IReadOnlyList<GraphEdge>> GetByRevisionAsync(
|
||||
Guid graphRevisionId, CancellationToken ct);
|
||||
Task<IReadOnlyList<GraphEdge>> GetOutgoingAsync(
|
||||
long fromNodeId, CancellationToken ct);
|
||||
Task<IReadOnlyList<GraphEdge>> GetIncomingAsync(
|
||||
long toNodeId, CancellationToken ct);
|
||||
Task BulkInsertAsync(
|
||||
Guid graphRevisionId, IEnumerable<GraphEdge> edges, CancellationToken ct);
|
||||
Task<int> GetCountAsync(Guid graphRevisionId, CancellationToken ct);
|
||||
}
|
||||
```
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Implement all interface methods
|
||||
- [ ] Implement bulk insert for efficiency
|
||||
- [ ] Optimize for traversal queries
|
||||
- [ ] Write integration tests
|
||||
|
||||
---
|
||||
|
||||
### T6b.4: Verify Graph Revision ID Stability
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 1 day
|
||||
|
||||
**Description:**
|
||||
Critical: Same SBOM + feeds + policy must produce identical revision_id.
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Document revision_id computation algorithm
|
||||
- [ ] Verify nodes are inserted in deterministic order
|
||||
- [ ] Verify edges are inserted in deterministic order
|
||||
- [ ] Write stability tests
|
||||
|
||||
**Stability Test:**
|
||||
```csharp
|
||||
[Fact]
|
||||
public async Task Same_Inputs_Should_Produce_Same_RevisionId()
|
||||
{
|
||||
var sbom = await LoadSbom("testdata/stable-sbom.json");
|
||||
var feedSnapshot = "feed-v1.2.3";
|
||||
var policyVersion = "policy-v1.0";
|
||||
|
||||
// Compute multiple times
|
||||
var revisions = new List<string>();
|
||||
for (int i = 0; i < 5; i++)
|
||||
{
|
||||
var graph = await _graphService.ComputeGraphAsync(
|
||||
sbom, feedSnapshot, policyVersion);
|
||||
revisions.Add(graph.RevisionId);
|
||||
}
|
||||
|
||||
// All must be identical
|
||||
revisions.Distinct().Should().HaveCount(1);
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Sprint 6c: Migration & Verification
|
||||
|
||||
### T6c.1: Build Graph Conversion Service
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 1.5 days
|
||||
|
||||
**Description:**
|
||||
Convert existing MongoDB graphs to PostgreSQL.
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Parse MongoDB graph documents
|
||||
- [ ] Map to graph_revisions table
|
||||
- [ ] Extract and insert nodes
|
||||
- [ ] Extract and insert edges
|
||||
- [ ] Verify node/edge counts match
|
||||
|
||||
---
|
||||
|
||||
### T6c.2: Build VEX Conversion Service
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 1 day
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Parse MongoDB VEX statements
|
||||
- [ ] Map to vex.statements table
|
||||
- [ ] Preserve provenance
|
||||
- [ ] Preserve evidence
|
||||
|
||||
---
|
||||
|
||||
### T6c.3: Run Dual Pipeline Comparison
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 2 days
|
||||
|
||||
**Description:**
|
||||
Run graph computation on both backends and compare.
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Select sample projects
|
||||
- [ ] Compute graphs with MongoDB
|
||||
- [ ] Compute graphs with PostgreSQL
|
||||
- [ ] Compare revision_ids (must match)
|
||||
- [ ] Compare node counts
|
||||
- [ ] Compare edge counts
|
||||
- [ ] Compare VEX statements
|
||||
- [ ] Document any differences
|
||||
|
||||
---
|
||||
|
||||
### T6c.4: Migrate Projects
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 1 day
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Identify projects to migrate (active VEX)
|
||||
- [ ] Run conversion for each project
|
||||
- [ ] Verify latest graph revision
|
||||
- [ ] Verify VEX statements
|
||||
|
||||
---
|
||||
|
||||
### T6c.5: Switch to PostgreSQL-Only
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] Update configuration
|
||||
- [ ] Deploy to staging
|
||||
- [ ] Run full test suite
|
||||
- [ ] Deploy to production
|
||||
- [ ] Monitor metrics
|
||||
|
||||
---
|
||||
|
||||
## Exit Criteria
|
||||
|
||||
- [ ] All repository interfaces implemented
|
||||
- [ ] Graph storage working efficiently
|
||||
- [ ] Graph revision IDs stable (deterministic)
|
||||
- [ ] VEX statements preserved correctly
|
||||
- [ ] All comparison tests pass
|
||||
- [ ] Excititor running on PostgreSQL in production
|
||||
|
||||
---
|
||||
|
||||
## Risks & Mitigations
|
||||
|
||||
| Risk | Likelihood | Impact | Mitigation |
|
||||
|------|------------|--------|------------|
|
||||
| Revision ID instability | Medium | Critical | Deterministic ordering tests |
|
||||
| Graph storage performance | Medium | High | Bulk insert, index optimization |
|
||||
| VEX lattice logic errors | Low | High | Extensive comparison testing |
|
||||
|
||||
---
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
### Graph Storage
|
||||
|
||||
- Use `BIGSERIAL` for node/edge IDs (high volume)
|
||||
- Use `COPY` for bulk inserts (10-100x faster)
|
||||
- Index `(graph_revision_id, node_key)` for lookups
|
||||
- Index `(from_node_id)` and `(to_node_id)` for traversal
|
||||
|
||||
### Estimated Volumes
|
||||
|
||||
| Table | Estimated Rows per Project | Total Estimated |
|
||||
|-------|---------------------------|-----------------|
|
||||
| graph_nodes | 1,000 - 50,000 | 10M+ |
|
||||
| graph_edges | 2,000 - 100,000 | 20M+ |
|
||||
| vex_statements | 100 - 5,000 | 1M+ |
|
||||
|
||||
---
|
||||
|
||||
*Phase Version: 1.0.0*
|
||||
*Last Updated: 2025-11-28*
|
||||
305
docs/db/tasks/PHASE_7_CLEANUP.md
Normal file
305
docs/db/tasks/PHASE_7_CLEANUP.md
Normal file
@@ -0,0 +1,305 @@
|
||||
# Phase 7: Cleanup & Optimization
|
||||
|
||||
**Sprint:** 11
|
||||
**Duration:** 1 sprint
|
||||
**Status:** TODO
|
||||
**Dependencies:** All previous phases completed
|
||||
|
||||
---
|
||||
|
||||
## Objectives
|
||||
|
||||
1. Remove MongoDB dependencies from converted modules
|
||||
2. Archive MongoDB data
|
||||
3. Optimize PostgreSQL performance
|
||||
4. Update documentation
|
||||
5. Update air-gap kit
|
||||
|
||||
---
|
||||
|
||||
## Deliverables
|
||||
|
||||
| Deliverable | Acceptance Criteria |
|
||||
|-------------|---------------------|
|
||||
| Code cleanup | MongoDB code removed from converted modules |
|
||||
| Data archive | MongoDB data archived and documented |
|
||||
| Performance tuning | Query times within acceptable range |
|
||||
| Documentation | All docs updated for PostgreSQL |
|
||||
| Air-gap kit | PostgreSQL support added |
|
||||
|
||||
---
|
||||
|
||||
## Task Breakdown
|
||||
|
||||
### T7.1: Remove MongoDB Dependencies
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 2 days
|
||||
|
||||
**Description:**
|
||||
Remove MongoDB storage projects and references from converted modules.
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T7.1.1: Remove `StellaOps.Authority.Storage.Mongo` project
|
||||
- [ ] T7.1.2: Remove `StellaOps.Scheduler.Storage.Mongo` project
|
||||
- [ ] T7.1.3: Remove `StellaOps.Notify.Storage.Mongo` project
|
||||
- [ ] T7.1.4: Remove `StellaOps.Policy.Storage.Mongo` project
|
||||
- [ ] T7.1.5: Remove `StellaOps.Concelier.Storage.Mongo` project
|
||||
- [ ] T7.1.6: Remove `StellaOps.Excititor.Storage.Mongo` project
|
||||
- [ ] T7.1.7: Update solution files
|
||||
- [ ] T7.1.8: Remove dual-write wrappers
|
||||
- [ ] T7.1.9: Remove MongoDB configuration options
|
||||
- [ ] T7.1.10: Run full build to verify no broken references
|
||||
|
||||
**Verification:**
|
||||
- [ ] Solution builds without MongoDB packages
|
||||
- [ ] No MongoDB references in converted modules
|
||||
- [ ] All tests pass
|
||||
|
||||
---
|
||||
|
||||
### T7.2: Archive MongoDB Data
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 1 day
|
||||
|
||||
**Description:**
|
||||
Archive MongoDB databases for historical reference.
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T7.2.1: Take final MongoDB backup
|
||||
- [ ] T7.2.2: Export to BSON/JSON archives
|
||||
- [ ] T7.2.3: Store archives in secure location
|
||||
- [ ] T7.2.4: Document archive contents and structure
|
||||
- [ ] T7.2.5: Set retention policy for archives
|
||||
- [ ] T7.2.6: Schedule MongoDB cluster decommission
|
||||
|
||||
**Archive Structure:**
|
||||
```
|
||||
archives/
|
||||
├── mongodb-authority-2025-XX-XX.bson.gz
|
||||
├── mongodb-scheduler-2025-XX-XX.bson.gz
|
||||
├── mongodb-notify-2025-XX-XX.bson.gz
|
||||
├── mongodb-policy-2025-XX-XX.bson.gz
|
||||
├── mongodb-concelier-2025-XX-XX.bson.gz
|
||||
├── mongodb-excititor-2025-XX-XX.bson.gz
|
||||
└── ARCHIVE_MANIFEST.md
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### T7.3: PostgreSQL Performance Optimization
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 2 days
|
||||
|
||||
**Description:**
|
||||
Analyze and optimize PostgreSQL performance.
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T7.3.1: Enable `pg_stat_statements` extension
|
||||
- [ ] T7.3.2: Identify slow queries
|
||||
- [ ] T7.3.3: Analyze query plans with EXPLAIN ANALYZE
|
||||
- [ ] T7.3.4: Add missing indexes
|
||||
- [ ] T7.3.5: Remove unused indexes
|
||||
- [ ] T7.3.6: Tune PostgreSQL configuration
|
||||
- [ ] T7.3.7: Set up query monitoring dashboard
|
||||
- [ ] T7.3.8: Document performance baselines
|
||||
|
||||
**Configuration Tuning:**
|
||||
```ini
|
||||
# postgresql.conf optimizations
|
||||
shared_buffers = 25% of RAM
|
||||
effective_cache_size = 75% of RAM
|
||||
work_mem = 64MB
|
||||
maintenance_work_mem = 512MB
|
||||
random_page_cost = 1.1 # for SSD
|
||||
effective_io_concurrency = 200 # for SSD
|
||||
max_parallel_workers_per_gather = 4
|
||||
```
|
||||
|
||||
**Monitoring Queries:**
|
||||
```sql
|
||||
-- Top slow queries
|
||||
SELECT query, calls, mean_time, total_time
|
||||
FROM pg_stat_statements
|
||||
ORDER BY mean_time DESC
|
||||
LIMIT 20;
|
||||
|
||||
-- Unused indexes
|
||||
SELECT schemaname, tablename, indexname
|
||||
FROM pg_stat_user_indexes
|
||||
WHERE idx_scan = 0;
|
||||
|
||||
-- Table bloat
|
||||
SELECT schemaname, tablename,
|
||||
pg_size_pretty(pg_total_relation_size(schemaname || '.' || tablename)) as size
|
||||
FROM pg_stat_user_tables
|
||||
ORDER BY pg_total_relation_size(schemaname || '.' || tablename) DESC;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### T7.4: Update Documentation
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 1.5 days
|
||||
|
||||
**Description:**
|
||||
Update all documentation to reflect PostgreSQL as the primary database.
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T7.4.1: Update `docs/07_HIGH_LEVEL_ARCHITECTURE.md`
|
||||
- [ ] T7.4.2: Update module architecture docs
|
||||
- [ ] T7.4.3: Update deployment guides
|
||||
- [ ] T7.4.4: Update operations runbooks
|
||||
- [ ] T7.4.5: Update troubleshooting guides
|
||||
- [ ] T7.4.6: Update `CLAUDE.md` technology stack
|
||||
- [ ] T7.4.7: Create PostgreSQL operations guide
|
||||
- [ ] T7.4.8: Document backup/restore procedures
|
||||
- [ ] T7.4.9: Document scaling recommendations
|
||||
|
||||
**New Documents:**
|
||||
- `docs/operations/postgresql-guide.md`
|
||||
- `docs/operations/postgresql-backup-restore.md`
|
||||
- `docs/operations/postgresql-troubleshooting.md`
|
||||
|
||||
---
|
||||
|
||||
### T7.5: Update Air-Gap Kit
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 1 day
|
||||
|
||||
**Description:**
|
||||
Update offline/air-gap kit to include PostgreSQL.
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T7.5.1: Add PostgreSQL container image to kit
|
||||
- [ ] T7.5.2: Update kit scripts for PostgreSQL setup
|
||||
- [ ] T7.5.3: Include schema migrations in kit
|
||||
- [ ] T7.5.4: Update kit documentation
|
||||
- [ ] T7.5.5: Test kit installation in air-gapped environment
|
||||
- [ ] T7.5.6: Update `docs/24_OFFLINE_KIT.md`
|
||||
|
||||
**Air-Gap Kit Structure:**
|
||||
```
|
||||
offline-kit/
|
||||
├── images/
|
||||
│ ├── postgres-16-alpine.tar
|
||||
│ └── stellaops-*.tar
|
||||
├── schemas/
|
||||
│ ├── authority.sql
|
||||
│ ├── vuln.sql
|
||||
│ ├── vex.sql
|
||||
│ ├── scheduler.sql
|
||||
│ ├── notify.sql
|
||||
│ └── policy.sql
|
||||
├── scripts/
|
||||
│ ├── setup-postgres.sh
|
||||
│ ├── run-migrations.sh
|
||||
│ └── import-data.sh
|
||||
└── docs/
|
||||
└── OFFLINE_SETUP.md
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### T7.6: Final Verification
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 1 day
|
||||
|
||||
**Description:**
|
||||
Run final verification of all systems.
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T7.6.1: Run full integration test suite
|
||||
- [ ] T7.6.2: Run performance benchmark suite
|
||||
- [ ] T7.6.3: Verify all modules on PostgreSQL
|
||||
- [ ] T7.6.4: Verify determinism tests pass
|
||||
- [ ] T7.6.5: Verify air-gap kit works
|
||||
- [ ] T7.6.6: Generate final verification report
|
||||
- [ ] T7.6.7: Get sign-off from stakeholders
|
||||
|
||||
---
|
||||
|
||||
### T7.7: Decommission MongoDB
|
||||
|
||||
**Status:** TODO
|
||||
**Estimate:** 0.5 days
|
||||
|
||||
**Description:**
|
||||
Final decommission of MongoDB infrastructure.
|
||||
|
||||
**Subtasks:**
|
||||
- [ ] T7.7.1: Verify no services using MongoDB
|
||||
- [ ] T7.7.2: Stop MongoDB instances
|
||||
- [ ] T7.7.3: Archive final state
|
||||
- [ ] T7.7.4: Remove MongoDB from infrastructure
|
||||
- [ ] T7.7.5: Update monitoring/alerting
|
||||
- [ ] T7.7.6: Update cost projections
|
||||
|
||||
---
|
||||
|
||||
## Exit Criteria
|
||||
|
||||
- [ ] All MongoDB code removed from converted modules
|
||||
- [ ] MongoDB data archived
|
||||
- [ ] PostgreSQL performance optimized
|
||||
- [ ] All documentation updated
|
||||
- [ ] Air-gap kit updated and tested
|
||||
- [ ] Final verification report approved
|
||||
- [ ] MongoDB infrastructure decommissioned
|
||||
|
||||
---
|
||||
|
||||
## Post-Conversion Monitoring
|
||||
|
||||
### First Week
|
||||
|
||||
- Monitor error rates closely
|
||||
- Track query performance
|
||||
- Watch for any data inconsistencies
|
||||
- Have rollback plan ready (restore MongoDB)
|
||||
|
||||
### First Month
|
||||
|
||||
- Review query statistics weekly
|
||||
- Optimize any slow queries found
|
||||
- Monitor storage growth
|
||||
- Adjust vacuum settings if needed
|
||||
|
||||
### Ongoing
|
||||
|
||||
- Regular performance reviews
|
||||
- Index maintenance
|
||||
- Backup verification
|
||||
- Capacity planning
|
||||
|
||||
---
|
||||
|
||||
## Rollback Considerations
|
||||
|
||||
**Note:** After Phase 7 completion, rollback to MongoDB becomes significantly more complex. Ensure all stakeholders understand:
|
||||
|
||||
1. MongoDB archives are read-only backup
|
||||
2. Any new data created after cutover is PostgreSQL-only
|
||||
3. Full rollback would require data export/import
|
||||
|
||||
---
|
||||
|
||||
## Success Metrics
|
||||
|
||||
| Metric | Target | Measurement |
|
||||
|--------|--------|-------------|
|
||||
| Query latency (p95) | < 100ms | pg_stat_statements |
|
||||
| Error rate | < 0.01% | Application logs |
|
||||
| Storage efficiency | < 120% of MongoDB | Disk usage |
|
||||
| Test coverage | 100% | CI reports |
|
||||
| Documentation coverage | 100% | Manual review |
|
||||
|
||||
---
|
||||
|
||||
*Phase Version: 1.0.0*
|
||||
*Last Updated: 2025-11-28*
|
||||
@@ -55,11 +55,12 @@
|
||||
| 12 | ORCH-OBS-53-001 | BLOCKED (2025-11-19) | PREP-ORCH-OBS-53-001-DEPENDS-ON-52-001-EVIDEN | Orchestrator Service Guild · Evidence Locker Guild | Generate job capsule inputs for Evidence Locker; invoke snapshot hooks; enforce redaction guard. |
|
||||
| 13 | ORCH-OBS-54-001 | BLOCKED (2025-11-19) | PREP-ORCH-OBS-54-001-DEPENDS-ON-53-001 | Orchestrator Service Guild · Provenance Guild | Produce DSSE attestations for orchestrator-scheduled jobs; store references in timeline + Evidence Locker; add verification endpoint `/jobs/{id}/attestation`. |
|
||||
| 14 | ORCH-OBS-55-001 | BLOCKED (2025-11-19) | PREP-ORCH-OBS-55-001-DEPENDS-ON-54-001-INCIDE | Orchestrator Service Guild · DevOps Guild | Incident mode hooks (sampling overrides, extended retention, debug spans) with automatic activation on SLO burn-rate breach; emit activation/deactivation events. |
|
||||
| 15 | ORCH-SVC-32-001 | BLOCKED (2025-11-19) | PREP-ORCH-SVC-32-001-UPSTREAM-READINESS-AIRGA | Orchestrator Service Guild | Bootstrap service project/config and Postgres schema/migrations for sources, runs, jobs, dag_edges, artifacts, quotas, schedules. |
|
||||
| 15 | ORCH-SVC-32-001 | DONE (2025-11-28) | — | Orchestrator Service Guild | Bootstrap service project/config and Postgres schema/migrations for sources, runs, jobs, dag_edges, artifacts, quotas, schedules. |
|
||||
|
||||
## Execution Log
|
||||
| Date (UTC) | Update | Owner |
|
||||
| --- | --- | --- |
|
||||
| 2025-11-28 | ORCH-SVC-32-001 DONE: Implemented Postgres schema/migrations (001_initial.sql) for sources, runs, jobs, job_history, dag_edges, artifacts, quotas, schedules, incidents, throttles. Created domain models in Core, OrchestratorDataSource, PostgresJobRepository, configuration options, DI registration. Build verified. | Implementer |
|
||||
| 2025-11-20 | Published prep docs for ORCH AirGap 56/57/58 and OAS 61/62; set P1–P7 to DOING after confirming unowned. | Project Mgmt |
|
||||
| 2025-11-20 | Started PREP-ORCH-OAS-63-001 (status → DOING) after confirming no existing DOING/DONE owners. | Planning |
|
||||
| 2025-11-20 | Published prep doc for PREP-ORCH-OAS-63-001 (`docs/modules/orchestrator/prep/2025-11-20-oas-63-001-prep.md`) and marked P8 DONE; awaits OAS 61/62 freeze before implementation. | Implementer |
|
||||
|
||||
@@ -20,15 +20,15 @@
|
||||
## Delivery Tracker
|
||||
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| 1 | ORCH-SVC-32-002 | TODO | Depends on ORCH-SVC-32-001 (Sprint 0151). | Orchestrator Service Guild (`src/Orchestrator/StellaOps.Orchestrator`) | Implement scheduler DAG planner + dependency resolver, job state machine, critical-path metadata (no control actions yet). |
|
||||
| 2 | ORCH-SVC-32-003 | TODO | Depends on 32-002. | Orchestrator Service Guild | Expose read-only REST APIs (sources, runs, jobs, DAG) with OpenAPI, validation, pagination, tenant scoping. |
|
||||
| 3 | ORCH-SVC-32-004 | TODO | Depends on 32-003. | Orchestrator Service Guild | Implement WebSocket/SSE stream for job/run updates; emit structured metrics counters/histograms; add health probes. |
|
||||
| 4 | ORCH-SVC-32-005 | TODO | Depends on 32-004. | Orchestrator Service Guild | Deliver worker claim/heartbeat/progress endpoints capturing artifact metadata/checksums and enforcing idempotency keys. |
|
||||
| 5 | ORCH-SVC-33-001 | TODO | Depends on 32-005. | Orchestrator Service Guild | Enable `sources` tests (control-plane validation). |
|
||||
| 6 | ORCH-SVC-33-002 | TODO | Depends on 33-001. | Orchestrator Service Guild | Per-source/tenant adaptive token-bucket limiter, concurrency caps, backpressure reacting to upstream 429/503. |
|
||||
| 7 | ORCH-SVC-33-003 | TODO | Depends on 33-002. | Orchestrator Service Guild | Watermark/backfill manager with event-time windows, duplicate suppression, dry-run preview endpoint, safety validations. |
|
||||
| 8 | ORCH-SVC-33-004 | TODO | Depends on 33-003. | Orchestrator Service Guild | Dead-letter store, replay endpoints, error classification with remediation hints + notification hooks. |
|
||||
| 9 | ORCH-SVC-34-001 | TODO | Depends on 33-004. | Orchestrator Service Guild | Quota management APIs, per-tenant SLO burn-rate computation, alert budget tracking via metrics. |
|
||||
| 1 | ORCH-SVC-32-002 | DONE | Depends on ORCH-SVC-32-001 (Sprint 0151). | Orchestrator Service Guild (`src/Orchestrator/StellaOps.Orchestrator`) | Implement scheduler DAG planner + dependency resolver, job state machine, critical-path metadata (no control actions yet). |
|
||||
| 2 | ORCH-SVC-32-003 | DONE | Depends on 32-002. | Orchestrator Service Guild | Expose read-only REST APIs (sources, runs, jobs, DAG) with OpenAPI, validation, pagination, tenant scoping. |
|
||||
| 3 | ORCH-SVC-32-004 | DONE | Depends on 32-003. | Orchestrator Service Guild | Implement WebSocket/SSE stream for job/run updates; emit structured metrics counters/histograms; add health probes. |
|
||||
| 4 | ORCH-SVC-32-005 | DONE | Depends on 32-004. | Orchestrator Service Guild | Deliver worker claim/heartbeat/progress endpoints capturing artifact metadata/checksums and enforcing idempotency keys. |
|
||||
| 5 | ORCH-SVC-33-001 | DONE | Depends on 32-005. | Orchestrator Service Guild | Enable `sources` tests (control-plane validation). |
|
||||
| 6 | ORCH-SVC-33-002 | DONE | Depends on 33-001. | Orchestrator Service Guild | Per-source/tenant adaptive token-bucket limiter, concurrency caps, backpressure reacting to upstream 429/503. |
|
||||
| 7 | ORCH-SVC-33-003 | DONE | Depends on 33-002. | Orchestrator Service Guild | Watermark/backfill manager with event-time windows, duplicate suppression, dry-run preview endpoint, safety validations. |
|
||||
| 8 | ORCH-SVC-33-004 | DONE | Depends on 33-003. | Orchestrator Service Guild | Dead-letter store, replay endpoints, error classification with remediation hints + notification hooks. |
|
||||
| 9 | ORCH-SVC-34-001 | DONE | Depends on 33-004. | Orchestrator Service Guild | Quota management APIs, per-tenant SLO burn-rate computation, alert budget tracking via metrics. |
|
||||
| 10 | ORCH-SVC-34-002 | TODO | Depends on 34-001. | Orchestrator Service Guild | Audit log + immutable run ledger export with signed manifest and provenance chain to artifacts. |
|
||||
| 11 | ORCH-SVC-34-003 | TODO | Depends on 34-002. | Orchestrator Service Guild | Perf/scale validation (≥10k pending jobs, dispatch P95 <150 ms); autoscaling hooks; health probes. |
|
||||
| 12 | ORCH-SVC-34-004 | TODO | Depends on 34-003. | Orchestrator Service Guild | GA packaging: container image, Helm overlays, offline bundle seeds, provenance attestations, compliance checklist. |
|
||||
@@ -42,6 +42,15 @@
|
||||
| 2025-11-08 | Sprint stub (legacy format) created; awaiting orchestrator phase I completion. | Planning |
|
||||
| 2025-11-19 | Normalized sprint to standard template and renamed from `SPRINT_152_orchestrator_ii.md` to `SPRINT_0152_0001_0002_orchestrator_ii.md`; content preserved. | Implementer |
|
||||
| 2025-11-19 | Added legacy-file redirect stub to avoid divergent updates. | Implementer |
|
||||
| 2025-11-28 | ORCH-SVC-32-002 DONE: Implemented JobStateMachine (status transitions/validation), DagPlanner (cycle detection, topological sort, critical path, dependency resolution), RetryPolicy (exponential backoff with jitter), JobScheduler (scheduling coordination). Added unit tests (67 tests passing). | Implementer |
|
||||
| 2025-11-28 | ORCH-SVC-32-003 DONE: Implemented REST APIs for sources, runs, jobs, and DAG. Added TenantResolver, EndpointHelpers, pagination support with cursors. Endpoints: SourceEndpoints (list, get), RunEndpoints (list, get, jobs, summary), JobEndpoints (list, get, detail, summary, by-idempotency-key), DagEndpoints (run DAG, edges, ready-jobs, blocked-jobs, parents, children). Build succeeds, 67 tests pass. | Implementer |
|
||||
| 2025-11-28 | ORCH-SVC-32-004 DONE: Implemented SSE streaming for jobs and runs. Created SseWriter utility, StreamOptions configuration, JobStreamCoordinator (job state changes), RunStreamCoordinator (run progress). Added StreamEndpoints (/api/v1/orchestrator/stream/jobs/{jobId}, /api/v1/orchestrator/stream/runs/{runId}). Enhanced HealthEndpoints with /healthz, /readyz, /livez, /health/details including database, memory, and thread pool checks. Metrics already implemented in Infrastructure. 67 tests pass. | Implementer |
|
||||
| 2025-11-28 | ORCH-SVC-32-005 DONE: Implemented worker endpoints for claim/heartbeat/progress/complete. Created WorkerContracts (ClaimRequest/Response, HeartbeatRequest/Response, ProgressRequest/Response, CompleteRequest/Response, ArtifactInput). Added IArtifactRepository interface and PostgresArtifactRepository. Created WorkerEndpoints with POST /api/v1/orchestrator/worker/claim, POST /worker/jobs/{jobId}/heartbeat, POST /worker/jobs/{jobId}/progress, POST /worker/jobs/{jobId}/complete. Added idempotency key enforcement and artifact metadata/checksum capture. Enhanced OrchestratorMetrics with ArtifactCreated, HeartbeatReceived, ProgressReported counters. Build succeeds, 67 tests pass. | Implementer |
|
||||
| 2025-11-28 | ORCH-SVC-33-001 DONE: Enabled sources control-plane validation. Created PostgresSourceRepository (CRUD, pause/resume, list with filters) and PostgresRunRepository (CRUD, status updates, job count incrementing). Added OrchestratorMetrics for sources (SourceCreated, SourcePaused, SourceResumed) and runs (RunCreated, RunCompleted). Registered all repositories in DI container. Created comprehensive control-plane tests: SourceTests (17 tests for Source domain validation, pause/resume semantics, configuration handling) and RunTests (27 tests for Run lifecycle, status transitions, job counting invariants). Build succeeds, 111 tests pass (+44 new tests). | Implementer |
|
||||
| 2025-11-28 | ORCH-SVC-33-002 DONE: Implemented per-source/tenant adaptive rate limiting. Created Throttle domain model (ThrottleReasons constants). Built RateLimiting components: TokenBucket (token bucket algorithm with refill/consume/snapshot), ConcurrencyLimiter (max active jobs tracking with acquire/release), BackpressureHandler (429/503 handling with exponential backoff and jitter), HourlyCounter (hourly rate tracking with automatic reset), AdaptiveRateLimiter (combines all strategies with rollback on partial failures). Created IQuotaRepository/IThrottleRepository interfaces and PostgresQuotaRepository/PostgresThrottleRepository implementations with full CRUD and state management. Added OrchestratorMetrics for quotas (QuotaCreated/Paused/Resumed), throttles (ThrottleCreated/Deactivated), rate limiting (RateLimitDenied, BackpressureEvent, TokenBucketUtilization, ConcurrencyUtilization). Registered repositories in DI container. Comprehensive test coverage: TokenBucketTests, ConcurrencyLimiterTests, BackpressureHandlerTests, AdaptiveRateLimiterTests, HourlyCounterTests. Build succeeds, 232 tests pass (+121 new tests). | Implementer |
|
||||
| 2025-11-28 | ORCH-SVC-33-003 DONE: Implemented watermark/backfill manager with event-time windows, duplicate suppression, dry-run preview, and safety validations. Created database migration (002_backfill.sql) with tables: watermarks (event-time cursors per scope), backfill_requests (batch reprocessing operations), processed_events (duplicate suppression with TTL), backfill_checkpoints (resumable batch state). Built domain models: Watermark (scope keys, advance with sequence/hash, windowing), BackfillRequest (state machine with validation/start/pause/resume/complete/fail/cancel transitions), BackfillSafetyChecks (blocking/warning validation), BackfillPreview (dry-run estimation). Created Backfill components: EventTimeWindow (contains/overlaps/intersect/split), EventTimeWindowOptions (hourly/daily batches), EventTimeWindowPlanner (window computation, lag detection, estimation), IDuplicateSuppressor/InMemoryDuplicateSuppressor (event tracking with TTL, batch filtering), DuplicateFilterResult (separation of new/duplicate events), BackfillManager/IBackfillManager (request lifecycle, validation, preview), IBackfillSafetyValidator/DefaultBackfillSafetyValidator (retention/overlap/limit checks). Created repository interfaces: IWatermarkRepository, IBackfillRepository, IBackfillCheckpointRepository with BackfillCheckpoint domain model. Implemented PostgresWatermarkRepository (CRUD, optimistic concurrency, lag queries), PostgresBackfillRepository (CRUD, overlap detection, status counts), PostgresDuplicateSuppressor/PostgresDuplicateSuppressorFactory (TTL-managed dedup). Added OrchestratorMetrics for watermarks (Created/Advanced/Lag), backfills (Created/StatusChanged/EventsProcessed/Skipped/Duration/Progress), duplicate suppression (Marked/CleanedUp/Detected). Registered services in DI container. Comprehensive test coverage: WatermarkTests (scope keys, create, advance, windowing), BackfillRequestTests (lifecycle, state machine, safety checks), BackfillSafetyChecksTests (blocking/warning validation), EventTimeWindowTests (duration, contains, overlaps, intersect, split, static factories), EventTimeWindowPlannerTests (window computation, lag, estimation), EventTimeWindowOptionsTests (hourly/daily defaults), DuplicateSuppressorTests (has/get/mark processed, batch filtering), ProcessedEventTests (record semantics). Build succeeds, 288 tests pass (+56 new tests). | Implementer |
|
||||
| 2025-11-28 | ORCH-SVC-33-004 DONE: Implemented dead-letter store with replay endpoints, error classification, remediation hints, and notification hooks. Created database migration (003_dead_letter.sql) with tables: dead_letter_entries (failed jobs with error classification), dead_letter_replay_audit (replay attempt tracking), dead_letter_notification_rules (alerting configuration), dead_letter_notification_log (notification history). Built domain models: DeadLetterEntry (entry lifecycle with Pending/Replaying/Replayed/Resolved/Exhausted/Expired states, FromFailedJob factory, StartReplay/CompleteReplay/FailReplay/Resolve/MarkExpired transitions, CanReplay/IsTerminal computed properties), DeadLetterStatus enum, ErrorCategory enum (Unknown/Transient/NotFound/AuthFailure/RateLimited/ValidationError/UpstreamError/InternalError/Conflict/Canceled). Created error classification system: ClassifiedError record, IErrorClassifier interface, DefaultErrorClassifier (40+ error codes with ORCH-TRN/NF/AUTH/RL/VAL/UP/INT/CON/CAN prefixes, HTTP status mapping, exception classification, remediation hints, retry delays). Built repository interfaces: IDeadLetterRepository (CRUD, list with filters, stats, actionable summary, mark expired, purge), IReplayAuditRepository (audit tracking), ReplayAuditRecord (Create/Complete/Fail transitions). Implemented PostgresDeadLetterRepository and PostgresReplayAuditRepository with full CRUD, filtering, statistics aggregation. Created ReplayManager: IReplayManager interface, ReplayManagerOptions, ReplayResult/BatchReplayResult records, replay single/batch/pending operations with audit logging and notification triggers. Built notification system: NotificationChannel enum (Email/Slack/Teams/Webhook/PagerDuty), NotificationRule (filter criteria, rate limiting with cooldown/max-per-hour, aggregation), IDeadLetterNotifier interface, DeadLetterNotifier (new entry/replay success/exhausted/aggregated notifications), NullDeadLetterNotifier, INotificationDelivery/INotificationRuleRepository interfaces, DeadLetterNotificationPayload/EntrySummary/StatsSnapshot records. Created REST endpoints: DeadLetterEndpoints (list/get/stats/summary, replay single/batch/pending, resolve single/batch, error-codes reference, replay audit). Added OrchestratorMetrics: DeadLetterCreated/StatusChanged/ReplayAttempted/ReplaySucceeded/ReplayFailed/Expired/Purged/NotificationSent/NotificationFailed/PendingChanged. Comprehensive test coverage: DeadLetterEntryTests (22 tests for FromFailedJob, lifecycle transitions, CanReplay/IsTerminal), ErrorClassificationTests (25 tests for error code classification, exception mapping, HTTP status codes, remediation hints), NotificationRuleTests (20 tests for rule matching, rate limiting, cooldown), ReplayAuditRecordTests (3 tests for Create/Complete/Fail). Build succeeds, 402 tests pass (+114 new tests). | Implementer |
|
||||
| 2025-11-28 | ORCH-SVC-34-001 DONE: Implemented quota management APIs with SLO burn-rate computation and alert budget tracking. Created Slo domain model (Domain/Slo.cs) with SloType enum (Availability/Latency/Throughput), SloWindow enum (1h/1d/7d/30d), AlertSeverity enum, factory methods (CreateAvailability/CreateLatency/CreateThroughput), Update/Enable/Disable methods, ErrorBudget/GetWindowDuration computed properties. Created SloState record for current metrics (SLI, budget consumed/remaining, burn rate, time to exhaustion). Created AlertBudgetThreshold (threshold-based alerting with cooldown and rate limiting, ShouldTrigger logic). Created SloAlert (alert lifecycle with Acknowledge/Resolve). Built BurnRateEngine (SloManagement/BurnRateEngine.cs) with interfaces: IBurnRateEngine (ComputeStateAsync, ComputeAllStatesAsync, EvaluateAlertsAsync), ISloEventSource (availability/latency/throughput counts retrieval), ISloRepository/IAlertThresholdRepository/ISloAlertRepository. Created database migration (004_slo_quotas.sql) with tables: slos, alert_budget_thresholds, slo_alerts, slo_state_snapshots, quota_audit_log, job_metrics_hourly. Added helper functions: get_slo_availability_counts, cleanup_slo_snapshots, cleanup_quota_audit_log, get_slo_summary. Created REST API contracts (QuotaContracts.cs): CreateQuotaRequest/UpdateQuotaRequest/PauseQuotaRequest/QuotaResponse/QuotaListResponse, CreateSloRequest/UpdateSloRequest/SloResponse/SloListResponse/SloStateResponse/SloWithStateResponse, CreateAlertThresholdRequest/AlertThresholdResponse, SloAlertResponse/SloAlertListResponse/AcknowledgeAlertRequest/ResolveAlertRequest, SloSummaryResponse/QuotaSummaryResponse/QuotaUtilizationResponse. Created QuotaEndpoints (list/get/create/update/delete, pause/resume, summary). Created SloEndpoints (list/get/create/update/delete, enable/disable, state/states, thresholds CRUD, alerts list/get/acknowledge/resolve, summary). Added SLO metrics to OrchestratorMetrics: SlosCreated/SlosUpdated, SloAlertsTriggered/Acknowledged/Resolved, SloBudgetConsumed/SloBurnRate/SloCurrentSli/SloBudgetRemaining/SloTimeToExhaustion histograms, SloActiveAlerts UpDownCounter. Comprehensive test coverage: SloTests (25 tests for creation/validation/error budget/window duration/update/enable-disable), SloStateTests (tests for NoData factory), AlertBudgetThresholdTests (12 tests for creation/validation/ShouldTrigger/cooldown), SloAlertTests (5 tests for Create/Acknowledge/Resolve). Build succeeds, 450 tests pass (+48 new tests). | Implementer |
|
||||
|
||||
## Decisions & Risks
|
||||
- All tasks depend on outputs from Orchestrator I (32-001); sprint remains TODO until upstream ship.
|
||||
|
||||
@@ -27,8 +27,8 @@
|
||||
| 1 | CVSS-MODEL-190-001 | DONE (2025-11-28) | None; foundational. | Policy Guild · Signals Guild (`src/Policy/StellaOps.Policy.Scoring`) | Design and implement CVSS v4.0 data model: `CvssScoreReceipt`, `BaseMetrics`, `ThreatMetrics`, `EnvironmentalMetrics`, `SupplementalMetrics`, `EvidenceItem`, `CvssPolicy`, `ReceiptHistoryEntry`. Include EF Core mappings and MongoDB schema. Evidence: Created `StellaOps.Policy.Scoring` project with `CvssMetrics.cs` (all CVSS v4.0 metric enums/records), `CvssScoreReceipt.cs` (receipt model with scores, evidence, history), `CvssPolicy.cs` (policy configuration), JSON schemas `cvss-policy-schema@1.json` and `cvss-receipt-schema@1.json`, and `AGENTS.md`. |
|
||||
| 2 | CVSS-ENGINE-190-002 | DONE (2025-11-28) | Depends on 190-001 for types. | Policy Guild (`src/Policy/StellaOps.Policy.Scoring/Engine`) | Implement `CvssV4Engine` with: `ParseVector()`, `ComputeBaseScore()`, `ComputeThreatAdjustedScore()`, `ComputeEnvironmentalAdjustedScore()`, `BuildVector()`. Follow FIRST spec v4.0 exactly for math/rounding. Evidence: `ICvssV4Engine.cs` interface, `CvssV4Engine.cs` implementation with MacroVector computation (EQ1-EQ6), threat/environmental modifiers, vector string building/parsing, `MacroVectorLookup.cs` with score tables. |
|
||||
| 3 | CVSS-TESTS-190-003 | DONE (2025-11-28) | Depends on 190-002. | Policy Guild · QA Guild (`src/Policy/__Tests/StellaOps.Policy.Scoring.Tests`) | Unit tests for CVSS v4.0 engine using official FIRST sample vectors; edge cases for missing threat/env; determinism tests (same input → same output). Evidence: Created `StellaOps.Policy.Scoring.Tests` project with `CvssV4EngineTests.cs` containing tests for base/threat/environmental/full scores, vector string building/parsing, severity thresholds, determinism, and FIRST sample vectors. |
|
||||
| 4 | CVSS-POLICY-190-004 | TODO | Depends on 190-002. | Policy Guild (`src/Policy/StellaOps.Policy.Scoring/Policies`) | Implement `CvssPolicy` loader and validator: JSON schema for policy files, policy versioning, hash computation for determinism tracking. |
|
||||
| 5 | CVSS-RECEIPT-190-005 | TODO | Depends on 190-002, 190-004. | Policy Guild (`src/Policy/StellaOps.Policy.Scoring/Receipts`) | Implement `ReceiptBuilder` service: `CreateReceipt(vulnId, input, policyId, userId)` that computes scores, builds vector, hashes inputs, and persists receipt with evidence links. |
|
||||
| 4 | CVSS-POLICY-190-004 | DONE (2025-11-28) | Depends on 190-002. | Policy Guild (`src/Policy/StellaOps.Policy.Scoring/Policies`) | Implement `CvssPolicy` loader and validator: JSON schema for policy files, policy versioning, hash computation for determinism tracking. |
|
||||
| 5 | CVSS-RECEIPT-190-005 | DONE (2025-11-28) | Depends on 190-002, 190-004. | Policy Guild (`src/Policy/StellaOps.Policy.Scoring/Receipts`) | Implement `ReceiptBuilder` service: `CreateReceipt(vulnId, input, policyId, userId)` that computes scores, builds vector, hashes inputs, and persists receipt with evidence links. |
|
||||
| 6 | CVSS-DSSE-190-006 | TODO | Depends on 190-005; uses Attestor primitives. | Policy Guild · Attestor Guild (`src/Policy/StellaOps.Policy.Scoring`, `src/Attestor/StellaOps.Attestor.Envelope`) | Attach DSSE attestations to score receipts: create `stella.ops/cvssReceipt@v1` predicate type, sign receipts, store envelope references. |
|
||||
| 7 | CVSS-HISTORY-190-007 | TODO | Depends on 190-005. | Policy Guild (`src/Policy/StellaOps.Policy.Scoring/History`) | Implement receipt amendment tracking: `AmendReceipt(receiptId, field, newValue, reason, ref)` with history entry creation and re-signing. |
|
||||
| 8 | CVSS-CONCELIER-190-008 | TODO | Depends on 190-001; coordinate with Concelier. | Concelier Guild · Policy Guild (`src/Concelier/__Libraries/StellaOps.Concelier.Core`) | Ingest vendor-provided CVSS v4.0 vectors from advisories; parse and store as base receipts; preserve provenance. |
|
||||
@@ -40,7 +40,7 @@
|
||||
## Wave Coordination
|
||||
| Wave | Guild owners | Shared prerequisites | Status | Notes |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| W1 Foundation | Policy Guild | None | TODO | Tasks 1-4: Data model, engine, tests, policy loader. |
|
||||
| W1 Foundation | Policy Guild | None | DONE (2025-11-28) | Tasks 1-4: Data model, engine, tests, policy loader. |
|
||||
| W2 Receipt Pipeline | Policy Guild · Attestor Guild | W1 complete | TODO | Tasks 5-7: Receipt builder, DSSE, history. |
|
||||
| W3 Integration | Concelier · Policy · CLI · UI Guilds | W2 complete | TODO | Tasks 8-11: Vendor ingest, APIs, CLI, UI. |
|
||||
| W4 Documentation | Docs Guild | W3 complete | TODO | Task 12: Full documentation. |
|
||||
@@ -59,7 +59,7 @@
|
||||
| # | Action | Owner | Due (UTC) | Status | Notes |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| 1 | Review FIRST CVSS v4.0 spec and identify implementation gaps. | Policy Guild | TBD | Open | Reference: https://www.first.org/cvss/v4-0/ |
|
||||
| 2 | Draft CvssPolicy JSON schema for team review. | Policy Guild | TBD | Open | |
|
||||
| 2 | Draft CvssPolicy JSON schema for team review. | Policy Guild | 2025-11-28 | DONE | Schema implemented and embedded at `src/Policy/StellaOps.Policy.Scoring/Schemas/cvss-policy-schema@1.json`; loader validates against it. |
|
||||
|
||||
## Decisions & Risks
|
||||
| ID | Risk | Impact | Mitigation / Owner |
|
||||
@@ -76,3 +76,6 @@
|
||||
| 2025-11-28 | Started CVSS-ENGINE-190-002: Implementing scoring engine with MacroVector lookup tables per FIRST CVSS v4.0 specification. | Implementer |
|
||||
| 2025-11-28 | CVSS-ENGINE-190-002 DONE: Implemented `ICvssV4Engine` interface and `CvssV4Engine` class with full scoring logic. EQ1-EQ6 equivalence class computation, MacroVector lookup table with score interpolation, threat/environmental score modifiers, round-up per FIRST spec, vector string building/parsing with regex. Started CVSS-TESTS-190-003. | Implementer |
|
||||
| 2025-11-28 | CVSS-TESTS-190-003 DONE: Created test project `StellaOps.Policy.Scoring.Tests` with `CvssV4EngineTests.cs`. Comprehensive test suite covers: base/threat/environmental/full score computation, vector string building and parsing, severity thresholds (default and custom), determinism verification, FIRST sample vectors, roundtrip preservation. Wave 1 (Foundation) complete - all 4 tasks DONE. | Implementer |
|
||||
| 2025-11-28 | CVSS-POLICY-190-004 DONE: Added `CvssPolicyLoader` (schema validation, canonical hash, policy deserialization), `CvssPolicySchema` loader for embedded schema, and unit tests (`CvssPolicyLoaderTests`) covering determinism and validation failures. | Implementer |
|
||||
| 2025-11-28 | CVSS-RECEIPT-190-005 DONE: Added `ReceiptBuilder` with deterministic input hashing, evidence validation (policy-driven), vector/scoring via CvssV4Engine, and persistence through repository abstraction. Added `CreateReceiptRequest`, `IReceiptRepository`, unit tests (`ReceiptBuilderTests`) with in-memory repo; all 37 tests passing. | Implementer |
|
||||
| 2025-11-28 | Ran `dotnet test src/Policy/__Tests/StellaOps.Policy.Scoring.Tests` (Release); 35 tests passed. Adjusted MacroVector lookup for FIRST sample vectors; duplicate PackageReference warnings remain to be cleaned separately. | Implementer |
|
||||
|
||||
123
docs/implplan/SPRINT_0215_0001_0001_vuln_triage_ux.md
Normal file
123
docs/implplan/SPRINT_0215_0001_0001_vuln_triage_ux.md
Normal file
@@ -0,0 +1,123 @@
|
||||
# Sprint 0215.0001.0001 - Experience & SDKs - Vulnerability Triage UX
|
||||
|
||||
## Topic & Scope
|
||||
- Implement vulnerability triage workspace with VEX-first decisioning UX aligned with industry patterns (Snyk, GitLab, Harbor/Trivy, Anchore).
|
||||
- Build evidence-first finding cards, VEX modal, attestation views, and audit bundle export.
|
||||
- **Working directory:** `src/UI/StellaOps.UI`
|
||||
|
||||
## Dependencies & Concurrency
|
||||
- Upstream sprints: SPRINT_0209_0001_0001_ui_i (UI I), SPRINT_210_ui_ii (UI II - VEX tab).
|
||||
- Backend dependencies: Vuln Explorer APIs (`/v1/findings`, `/v1/vex-decisions`), Attestor service, Export Center.
|
||||
- Parallel tracks: Can run alongside UI II/III for shared component work.
|
||||
- Blockers to flag: VEX decision API schema finalization, Attestation viewer predicates.
|
||||
|
||||
## Documentation Prerequisites
|
||||
- `docs/README.md`
|
||||
- `docs/07_HIGH_LEVEL_ARCHITECTURE.md`
|
||||
- `docs/modules/platform/architecture-overview.md`
|
||||
- `docs/modules/ui/architecture.md`
|
||||
- `docs/modules/vuln-explorer/architecture.md`
|
||||
- `docs/modules/vex-lens/architecture.md`
|
||||
- `docs/product-advisories/28-Nov-2025 - Vulnerability Triage UX & VEX-First Decisioning.md` (canonical)
|
||||
- `docs/product-advisories/27-Nov-2025 - Explainability Layer for Vulnerability Verdicts.md`
|
||||
- `docs/schemas/vex-decision.schema.json`
|
||||
- `docs/schemas/audit-bundle-index.schema.json`
|
||||
|
||||
## Delivery Tracker
|
||||
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| 1 | UI-TRIAGE-01-001 | TODO | - | UI Guild (src/UI/StellaOps.UI) | Create Artifacts List view with columns: Artifact, Type, Environment(s), Open/Total vulns, Max severity, Attestations badge, Last scan. Include sorting, filtering, and "View vulnerabilities" primary action. |
|
||||
| 2 | UI-TRIAGE-01-002 | TODO | UI-TRIAGE-01-001 | UI Guild (src/UI/StellaOps.UI) | Build Vulnerability Workspace split layout: left panel with finding cards (CVE, package, severity, path), right panel with Explainability tabs (Overview, Reachability, Policy, Attestations). |
|
||||
| 3 | UI-TRIAGE-01-003 | TODO | UI-TRIAGE-01-002 | UI Guild (src/UI/StellaOps.UI) | Implement evidence-first Finding Card component with severity badge, package info, location path, and primary actions (Fix PR, VEX, Attach Evidence). Include `New`, `VEX: Not affected`, `Policy: blocked` badges. |
|
||||
| 4 | UI-TRIAGE-01-004 | TODO | UI-TRIAGE-01-003 | UI Guild (src/UI/StellaOps.UI) | Build Explainability Panel Overview tab: title, severity, package/version, scanner+DB date, finding history timeline, current VEX decision summary. |
|
||||
| 5 | UI-TRIAGE-01-005 | TODO | UI-TRIAGE-01-004 | UI Guild (src/UI/StellaOps.UI) | Build Explainability Panel Reachability tab: call path visualization, module list, runtime usage indicators (when available from scanner). |
|
||||
| 6 | UI-TRIAGE-01-006 | TODO | UI-TRIAGE-01-004 | UI Guild (src/UI/StellaOps.UI) | Build Explainability Panel Policy tab: policy evaluation result, gate details with "this gate failed because..." explanation, links to gate definitions. |
|
||||
| 7 | UI-TRIAGE-01-007 | TODO | UI-TRIAGE-01-004 | UI Guild (src/UI/StellaOps.UI) | Build Explainability Panel Attestations tab: list attestations mentioning artifact/vulnerabilityId/scan with type, subject, predicate, signer, verified badge. |
|
||||
| 8 | UI-VEX-02-001 | TODO | UI-TRIAGE-01-003 | UI Guild; Excititor Guild (src/UI/StellaOps.UI) | Create VEX Modal component with status radio buttons (Not Affected, Affected-mitigated, Affected-unmitigated, Fixed), justification type select, justification text area. |
|
||||
| 9 | UI-VEX-02-002 | TODO | UI-VEX-02-001 | UI Guild (src/UI/StellaOps.UI) | Add VEX Modal scope section: environments multi-select, projects multi-select with clear scope preview. |
|
||||
| 10 | UI-VEX-02-003 | TODO | UI-VEX-02-002 | UI Guild (src/UI/StellaOps.UI) | Add VEX Modal validity section: notBefore date (default now), notAfter date with expiry recommendations and warnings for long durations. |
|
||||
| 11 | UI-VEX-02-004 | TODO | UI-VEX-02-003 | UI Guild (src/UI/StellaOps.UI) | Add VEX Modal evidence section: add links (PR, ticket, doc, commit), attach attestation picker, evidence preview list with remove action. |
|
||||
| 12 | UI-VEX-02-005 | TODO | UI-VEX-02-004 | UI Guild (src/UI/StellaOps.UI) | Add VEX Modal review section: summary preview of VEX statement to be created, "Will generate signed attestation" indicator, View raw JSON toggle for power users. |
|
||||
| 13 | UI-VEX-02-006 | TODO | UI-VEX-02-005 | UI Guild (src/UI/StellaOps.UI) | Wire VEX Modal to backend: POST /vex-decisions on save, handle success/error states, update finding card VEX badge on completion. |
|
||||
| 14 | UI-VEX-02-007 | TODO | UI-VEX-02-006 | UI Guild (src/UI/StellaOps.UI) | Add bulk VEX action: multi-select findings from list, open VEX modal with bulk context, apply decision to all selected findings. |
|
||||
| 15 | UI-ATT-03-001 | TODO | UI-TRIAGE-01-007 | UI Guild; Attestor Guild (src/UI/StellaOps.UI) | Create Attestations View per artifact: table with Type, Subject, Predicate type, Scanner/policy engine, Signer (keyId + trusted badge), Created at, Verified status. |
|
||||
| 16 | UI-ATT-03-002 | TODO | UI-ATT-03-001 | UI Guild (src/UI/StellaOps.UI) | Build Attestation Detail modal: header (statement id, subject, signer), predicate preview (vuln scan counts, SBOM bomRef, VEX decision status), verify command snippet. |
|
||||
| 17 | UI-ATT-03-003 | TODO | UI-ATT-03-002 | UI Guild (src/UI/StellaOps.UI) | Add "Signed evidence" pill to finding cards: clicking opens attestation detail modal, shows human-readable JSON view. |
|
||||
| 18 | UI-GATE-04-001 | TODO | UI-TRIAGE-01-006 | UI Guild; Policy Guild (src/UI/StellaOps.UI) | Create Policy & Gating View: matrix of gates vs subject types (CI Build, Registry Admission, Runtime Admission), rule descriptions, last evaluation stats. |
|
||||
| 19 | UI-GATE-04-002 | TODO | UI-GATE-04-001 | UI Guild (src/UI/StellaOps.UI) | Add gate drill-down: recent evaluations list, artifact links, policy attestation links, condition failure explanations. |
|
||||
| 20 | UI-GATE-04-003 | TODO | UI-GATE-04-002 | UI Guild (src/UI/StellaOps.UI) | Add "Ready to deploy" badge on artifact cards when all gates pass and required attestations verified. |
|
||||
| 21 | UI-AUDIT-05-001 | TODO | UI-TRIAGE-01-001 | UI Guild; Export Center Guild (src/UI/StellaOps.UI) | Create "Create immutable audit bundle" button on Artifact page, Pipeline run detail, and Policy evaluation detail views. |
|
||||
| 22 | UI-AUDIT-05-002 | TODO | UI-AUDIT-05-001 | UI Guild (src/UI/StellaOps.UI) | Build Audit Bundle creation wizard: subject artifact+digest selection, time window picker, content checklist (Vuln reports, SBOM, VEX, Policy evals, Attestations). |
|
||||
| 23 | UI-AUDIT-05-003 | TODO | UI-AUDIT-05-002 | UI Guild (src/UI/StellaOps.UI) | Wire audit bundle creation to POST /audit-bundles, show progress, display bundle ID, hash, download button, and OCI reference on completion. |
|
||||
| 24 | UI-AUDIT-05-004 | TODO | UI-AUDIT-05-003 | UI Guild (src/UI/StellaOps.UI) | Add audit bundle history view: list previously created bundles with bundleId, createdAt, subject, download/view actions. |
|
||||
| 25 | API-VEX-06-001 | TODO | - | API Guild (src/VulnExplorer) | Implement POST /v1/vex-decisions endpoint with VexDecisionDto request/response per schema, validation, attestation generation trigger. |
|
||||
| 26 | API-VEX-06-002 | TODO | API-VEX-06-001 | API Guild (src/VulnExplorer) | Implement PATCH /v1/vex-decisions/{id} for updating existing decisions with supersedes tracking. |
|
||||
| 27 | API-VEX-06-003 | TODO | API-VEX-06-002 | API Guild (src/VulnExplorer) | Implement GET /v1/vex-decisions with filters for vulnerabilityId, subject, status, scope, validFor. |
|
||||
| 28 | API-AUDIT-07-001 | TODO | - | API Guild (src/ExportCenter) | Implement POST /v1/audit-bundles endpoint with bundle creation, index generation, ZIP/OCI artifact production. |
|
||||
| 29 | API-AUDIT-07-002 | TODO | API-AUDIT-07-001 | API Guild (src/ExportCenter) | Implement GET /v1/audit-bundles/{bundleId} for bundle download with integrity verification. |
|
||||
| 30 | SCHEMA-08-001 | TODO | - | Platform Guild | Create docs/schemas/vex-decision.schema.json with JSON Schema 2020-12 definition per advisory. |
|
||||
| 31 | SCHEMA-08-002 | TODO | SCHEMA-08-001 | Platform Guild | Create docs/schemas/attestation-vuln-scan.schema.json for vulnerability scan attestation predicate. |
|
||||
| 32 | SCHEMA-08-003 | TODO | SCHEMA-08-002 | Platform Guild | Create docs/schemas/audit-bundle-index.schema.json for audit bundle manifest structure. |
|
||||
| 33 | DTO-09-001 | TODO | SCHEMA-08-001 | API Guild | Create VexDecisionDto, SubjectRefDto, EvidenceRefDto, VexScopeDto, ValidForDto C# DTOs per advisory. |
|
||||
| 34 | DTO-09-002 | TODO | SCHEMA-08-002 | API Guild | Create VulnScanAttestationDto, AttestationSubjectDto, VulnScanPredicateDto C# DTOs per advisory. |
|
||||
| 35 | DTO-09-003 | TODO | SCHEMA-08-003 | API Guild | Create AuditBundleIndexDto, BundleArtifactDto, BundleVexDecisionEntryDto C# DTOs per advisory. |
|
||||
| 36 | TS-10-001 | TODO | SCHEMA-08-001 | UI Guild | Create TypeScript interfaces for VexDecision, SubjectRef, EvidenceRef, VexScope, ValidFor per advisory. |
|
||||
| 37 | TS-10-002 | TODO | SCHEMA-08-002 | UI Guild | Create TypeScript interfaces for VulnScanAttestation, AttestationSubject, VulnScanPredicate per advisory. |
|
||||
| 38 | TS-10-003 | TODO | SCHEMA-08-003 | UI Guild | Create TypeScript interfaces for AuditBundleIndex, BundleArtifact, BundleVexDecisionEntry per advisory. |
|
||||
|
||||
## Wave Coordination
|
||||
- **Wave A (Schemas & DTOs):** SCHEMA-08-*, DTO-09-*, TS-10-* - Foundation work
|
||||
- **Wave B (Backend APIs):** API-VEX-06-*, API-AUDIT-07-* - Depends on Wave A
|
||||
- **Wave C (UI Components):** UI-TRIAGE-01-*, UI-VEX-02-*, UI-ATT-03-*, UI-GATE-04-*, UI-AUDIT-05-* - Depends on Wave A, can start mockable components in parallel
|
||||
|
||||
## Wave Detail Snapshots
|
||||
### Wave A - Schemas & Types
|
||||
- Duration: 2-3 days
|
||||
- Deliverables: JSON schemas in docs/schemas/, C# DTOs in src/VulnExplorer, TypeScript interfaces in src/UI
|
||||
- Exit criteria: Schemas validate, DTOs compile, TS interfaces pass type checks
|
||||
|
||||
### Wave B - Backend APIs
|
||||
- Duration: 3-5 days
|
||||
- Deliverables: VEX decision CRUD endpoints, audit bundle generation endpoint
|
||||
- Exit criteria: API tests pass, OpenAPI spec updated, deterministic outputs verified
|
||||
|
||||
### Wave C - UI Components
|
||||
- Duration: 5-7 days
|
||||
- Deliverables: Triage workspace, VEX modal, attestation views, audit bundle wizard
|
||||
- Exit criteria: Accessibility audit passes, responsive design verified, e2e tests green
|
||||
|
||||
## Interlocks
|
||||
- VEX-Lens module (Excititor) for VEX document normalization and consensus
|
||||
- Attestor service for VEX attestation signing
|
||||
- Export Center for audit bundle ZIP/OCI generation
|
||||
- Policy Engine for gate evaluation data
|
||||
|
||||
## Upcoming Checkpoints
|
||||
- 2025-12-02 15:00 UTC - Schema review (owners: Platform Guild, API Guild)
|
||||
- 2025-12-05 15:00 UTC - API contract freeze (owners: API Guild, UI Guild)
|
||||
- 2025-12-10 15:00 UTC - UI component review (owners: UI Guild, UX)
|
||||
- 2025-12-13 15:00 UTC - Integration testing go/no-go (owners: All guilds)
|
||||
|
||||
## Action Tracker
|
||||
| # | Action | Owner | Due | Status |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| 1 | Finalize VEX decision schema with Excititor team | Platform Guild | 2025-12-02 | TODO |
|
||||
| 2 | Confirm attestation predicate types with Attestor team | API Guild | 2025-12-03 | TODO |
|
||||
| 3 | Review audit bundle format with Export Center team | API Guild | 2025-12-04 | TODO |
|
||||
| 4 | Accessibility review of VEX modal with Accessibility Guild | UI Guild | 2025-12-09 | TODO |
|
||||
|
||||
## Decisions & Risks
|
||||
| Risk | Impact | Mitigation / Next Step |
|
||||
| --- | --- | --- |
|
||||
| VEX schema changes after Wave A | Rework DTOs and TS interfaces | Lock schema by checkpoint 1; version DTOs if needed |
|
||||
| Attestation service not ready | UI-ATT-* tasks blocked | Mock attestation data; feature flag attestation views |
|
||||
| Export Center capacity | Audit bundle generation slow | Async generation with progress; queue management |
|
||||
| Bulk VEX operations performance | UI-VEX-02-007 slow for large selections | Batch API endpoint; pagination; background processing |
|
||||
|
||||
## Execution Log
|
||||
| Date (UTC) | Update | Owner |
|
||||
| --- | --- | --- |
|
||||
| 2025-11-28 | Sprint created from product advisory `28-Nov-2025 - Vulnerability Triage UX & VEX-First Decisioning.md`. 38 tasks defined across 5 UI task groups, 2 API task groups, 3 schema tasks, 3 DTO tasks, 3 TS interface tasks. | Project mgmt |
|
||||
|
||||
---
|
||||
*Sprint created: 2025-11-28*
|
||||
@@ -33,10 +33,10 @@ Dependency: Sprint 135 - 6. Scanner.VI — Scanner & Surface focus on Scanner (p
|
||||
| `SCANNER-ENG-0021` | DONE (2025-11-28) | Implement pkgutil receipt collector per `design/macos-analyzer.md` §3.2. | Scanner Guild (docs/modules/scanner) | — |
|
||||
| `SCANNER-ENG-0022` | DONE (2025-11-28) | Implement macOS bundle inspector & capability overlays per `design/macos-analyzer.md` §3.3. | Scanner Guild, Policy Guild (docs/modules/scanner) | — |
|
||||
| `SCANNER-ENG-0023` | DONE (2025-11-28) | Deliver macOS policy/offline integration per `design/macos-analyzer.md` §5–6. | Scanner Guild, Offline Kit Guild, Policy Guild (docs/modules/scanner) | — |
|
||||
| `SCANNER-ENG-0024` | TODO | Implement Windows MSI collector per `design/windows-analyzer.md` §3.1. | Scanner Guild (docs/modules/scanner) | — |
|
||||
| `SCANNER-ENG-0025` | TODO | Implement WinSxS manifest collector per `design/windows-analyzer.md` §3.2. | Scanner Guild (docs/modules/scanner) | — |
|
||||
| `SCANNER-ENG-0026` | TODO | Implement Windows Chocolatey & registry collectors per `design/windows-analyzer.md` §3.3–3.4. | Scanner Guild (docs/modules/scanner) | — |
|
||||
| `SCANNER-ENG-0027` | TODO | Deliver Windows policy/offline integration per `design/windows-analyzer.md` §5–6. | Scanner Guild, Policy Guild, Offline Kit Guild (docs/modules/scanner) | — |
|
||||
| `SCANNER-ENG-0024` | DONE (2025-11-28) | Implement Windows MSI collector per `design/windows-analyzer.md` §3.1. | Scanner Guild (docs/modules/scanner) | — |
|
||||
| `SCANNER-ENG-0025` | DONE (2025-11-28) | Implement WinSxS manifest collector per `design/windows-analyzer.md` §3.2. | Scanner Guild (docs/modules/scanner) | — |
|
||||
| `SCANNER-ENG-0026` | DONE (2025-11-28) | Implement Windows Chocolatey & registry collectors per `design/windows-analyzer.md` §3.3–3.4. | Scanner Guild (docs/modules/scanner) | — |
|
||||
| `SCANNER-ENG-0027` | DONE (2025-11-28) | Deliver Windows policy/offline integration per `design/windows-analyzer.md` §5–6. | Scanner Guild, Policy Guild, Offline Kit Guild (docs/modules/scanner) | — |
|
||||
| `SCHED-SURFACE-02` | TODO | Integrate Scheduler worker prefetch using Surface manifest reader and persist manifest pointers with rerun plans. | Scheduler Worker Guild (src/Scheduler/__Libraries/StellaOps.Scheduler.Worker) | SURFACE-FS-02, SCHED-SURFACE-01. Reference `docs/modules/scanner/design/surface-fs-consumers.md` §3 for implementation checklist |
|
||||
| `ZASTAVA-SURFACE-02` | TODO | Use Surface manifest reader helpers to resolve `cas://` pointers and enrich drift diagnostics with manifest provenance. | Zastava Observer Guild (src/Zastava/StellaOps.Zastava.Observer) | SURFACE-FS-02, ZASTAVA-SURFACE-01. Reference `docs/modules/scanner/design/surface-fs-consumers.md` §4 for integration steps |
|
||||
| `SURFACE-FS-03` | DONE (2025-11-27) | Integrate Surface.FS writer into Scanner Worker analyzer pipeline to persist layer + entry-trace fragments. | Scanner Guild (src/Scanner/__Libraries/StellaOps.Scanner.Surface.FS) | SURFACE-FS-02 |
|
||||
@@ -90,3 +90,7 @@ Dependency: Sprint 135 - 6. Scanner.VI — Scanner & Surface focus on Scanner (p
|
||||
| 2025-11-28 | Created `docs/modules/scanner/guides/surface-fs-workflow.md` with end-to-end workflow including artefact generation, storage layout, consumption, and offline kit handling; SURFACE-FS-06 DONE. | Implementer |
|
||||
| 2025-11-28 | Created `StellaOps.Scanner.Analyzers.OS.Homebrew` library with `HomebrewReceiptParser` (INSTALL_RECEIPT.json parsing), `HomebrewPackageAnalyzer` (Cellar discovery for Intel/Apple Silicon), and `HomebrewAnalyzerPlugin`; added `BuildHomebrew` PURL builder, `HomebrewCellar` evidence source; 23 tests passing. SCANNER-ENG-0020 DONE. | Implementer |
|
||||
| 2025-11-28 | Created `StellaOps.Scanner.Analyzers.OS.Pkgutil` library with `PkgutilReceiptParser` (plist parsing), `BomParser` (BOM file enumeration), `PkgutilPackageAnalyzer` (receipt discovery from /var/db/receipts), and `PkgutilAnalyzerPlugin`; added `BuildPkgutil` PURL builder, `PkgutilReceipt` evidence source; 9 tests passing. SCANNER-ENG-0021 DONE. | Implementer |
|
||||
| 2025-11-28 | Created `StellaOps.Scanner.Analyzers.OS.Windows.Msi` library with `MsiDatabaseParser` (OLE compound document parser), `MsiPackageAnalyzer` (Windows/Installer/*.msi discovery), and `MsiAnalyzerPlugin`; added `BuildWindowsMsi` PURL builder, `WindowsMsi` evidence source; 22 tests passing. SCANNER-ENG-0024 DONE. | Implementer |
|
||||
| 2025-11-28 | Created `StellaOps.Scanner.Analyzers.OS.Windows.WinSxS` library with `WinSxSManifestParser` (XML assembly identity parser), `WinSxSPackageAnalyzer` (WinSxS/Manifests/*.manifest discovery), and `WinSxSAnalyzerPlugin`; added `BuildWindowsWinSxS` PURL builder, `WindowsWinSxS` evidence source; 18 tests passing. SCANNER-ENG-0025 DONE. | Implementer |
|
||||
| 2025-11-28 | Created `StellaOps.Scanner.Analyzers.OS.Windows.Chocolatey` library with `NuspecParser` (nuspec + directory name fallback), `ChocolateyPackageAnalyzer` (ProgramData/Chocolatey/lib discovery), and `ChocolateyAnalyzerPlugin`; added `BuildChocolatey` PURL builder, `WindowsChocolatey` evidence source; 44 tests passing. SCANNER-ENG-0026 DONE. | Implementer |
|
||||
| 2025-11-28 | Updated `docs/modules/scanner/design/windows-analyzer.md` with implementation status section documenting MSI/WinSxS/Chocolatey collector details, PURL formats, and vendor metadata schemas; registry collector deferred, policy predicates pending Policy module integration. SCANNER-ENG-0027 DONE. | Implementer |
|
||||
|
||||
@@ -15,8 +15,8 @@ ORCH-SVC-33-001 | TODO | Enable `sources test. Dependencies: ORCH-SVC-32-005. |
|
||||
ORCH-SVC-33-002 | TODO | Implement per-source/tenant adaptive token-bucket rate limiter, concurrency caps, and backpressure signals reacting to upstream 429/503. Dependencies: ORCH-SVC-33-001. | Orchestrator Service Guild (src/Orchestrator/StellaOps.Orchestrator)
|
||||
ORCH-SVC-33-003 | TODO | Add watermark/backfill manager with event-time windows, duplicate suppression, dry-run preview endpoint, and safety validations. Dependencies: ORCH-SVC-33-002. | Orchestrator Service Guild (src/Orchestrator/StellaOps.Orchestrator)
|
||||
ORCH-SVC-33-004 | TODO | Deliver dead-letter store, replay endpoints, and error classification surfaces with remediation hints + notification hooks. Dependencies: ORCH-SVC-33-003. | Orchestrator Service Guild (src/Orchestrator/StellaOps.Orchestrator)
|
||||
ORCH-SVC-34-001 | TODO | Implement quota management APIs, per-tenant SLO burn-rate computation, and alert budget tracking surfaced via metrics. Dependencies: ORCH-SVC-33-004. | Orchestrator Service Guild (src/Orchestrator/StellaOps.Orchestrator)
|
||||
ORCH-SVC-34-002 | TODO | Build audit log + immutable run ledger export with signed manifest support, including provenance chain to artifacts. Dependencies: ORCH-SVC-34-001. | Orchestrator Service Guild (src/Orchestrator/StellaOps.Orchestrator)
|
||||
ORCH-SVC-34-001 | DONE | Implement quota management APIs, per-tenant SLO burn-rate computation, and alert budget tracking surfaced via metrics. Dependencies: ORCH-SVC-33-004. | Orchestrator Service Guild (src/Orchestrator/StellaOps.Orchestrator)
|
||||
ORCH-SVC-34-002 | DONE | Build audit log + immutable run ledger export with signed manifest support, including provenance chain to artifacts. Dependencies: ORCH-SVC-34-001. | Orchestrator Service Guild (src/Orchestrator/StellaOps.Orchestrator)
|
||||
ORCH-SVC-34-003 | TODO | Execute perf/scale validation (≥10k pending jobs, dispatch P95 <150 ms) and add autoscaling hooks with health probes. Dependencies: ORCH-SVC-34-002. | Orchestrator Service Guild (src/Orchestrator/StellaOps.Orchestrator)
|
||||
ORCH-SVC-34-004 | TODO | Package orchestrator container, Helm overlays, offline bundle seeds, provenance attestations, and compliance checklist for GA. Dependencies: ORCH-SVC-34-003. | Orchestrator Service Guild (src/Orchestrator/StellaOps.Orchestrator)
|
||||
ORCH-SVC-35-101 | TODO | Register `export` job type with quotas/rate policies, expose telemetry, and ensure exporter workers heartbeat via orchestrator contracts. Dependencies: ORCH-SVC-34-004. | Orchestrator Service Guild (src/Orchestrator/StellaOps.Orchestrator)
|
||||
|
||||
@@ -1,15 +1,25 @@
|
||||
# Sprint 185 - Replay Core · 185.A) Shared Replay Primitives
|
||||
|
||||
[Replay Core] 185.A) Shared Replay Primitives
|
||||
Depends on: Sprint 160 Export & Evidence
|
||||
[Replay Core] 185.A) Shared Replay Primitives
|
||||
Depends on: Sprint 160 Export & Evidence
|
||||
Summary: Stand up a shared replay library, hashing/cononicalisation helpers, and baseline documentation for deterministic bundles.
|
||||
|
||||
Task ID | State | Task description | Owners (Source)
|
||||
--- | --- | --- | ---
|
||||
REPLAY-CORE-185-001 | TODO | Scaffold `StellaOps.Replay.Core` with manifest schema types, canonical JSON rules, Merkle utilities, and DSSE payload builders; add `AGENTS.md`/`TASKS.md` for the new library; cross-reference `docs/replay/DETERMINISTIC_REPLAY.md` section 3 when updating the library charter. | BE-Base Platform Guild (`src/__Libraries/StellaOps.Replay.Core`)
|
||||
REPLAY-CORE-185-002 | TODO | Implement deterministic bundle writer (tar.zst, CAS naming) and hashing abstractions, updating `docs/modules/platform/architecture-overview.md` with a “Replay CAS” subsection that documents layout/retention expectations. | Platform Guild (src/__Libraries/StellaOps.Replay.Core)
|
||||
REPLAY-CORE-185-003 | TODO | Define Mongo collections (`replay_runs`, `replay_bundles`, `replay_subjects`) and indices, then author `docs/data/replay_schema.md` detailing schema fields, constraints, and offline sync strategy. | Platform Data Guild (src/__Libraries/StellaOps.Replay.Core)
|
||||
DOCS-REPLAY-185-003 | TODO | Author `docs/data/replay_schema.md` detailing `replay_runs`, `replay_bundles`, `replay_subjects` collections, index guidance, and offline sync strategy aligned with Replay CAS. | Docs Guild, Platform Data Guild (docs)
|
||||
DOCS-REPLAY-185-004 | TODO | Expand `docs/replay/DEVS_GUIDE_REPLAY.md` with integration guidance for consuming services (Scanner, Evidence Locker, CLI) and add checklist derived from `docs/replay/DETERMINISTIC_REPLAY.md` Section 11. | Docs Guild (docs)
|
||||
REPLAY-CORE-185-001 | DONE (2025-11-28) | Scaffold `StellaOps.Replay.Core` with manifest schema types, canonical JSON rules, Merkle utilities, and DSSE payload builders; add `AGENTS.md`/`TASKS.md` for the new library; cross-reference `docs/replay/DETERMINISTIC_REPLAY.md` section 3 when updating the library charter. | BE-Base Platform Guild (`src/__Libraries/StellaOps.Replay.Core`)
|
||||
REPLAY-CORE-185-002 | DONE (2025-11-28) | Implement deterministic bundle writer (tar.zst, CAS naming) and hashing abstractions, updating `docs/modules/platform/architecture-overview.md` with a "Replay CAS" subsection that documents layout/retention expectations. | Platform Guild (src/__Libraries/StellaOps.Replay.Core)
|
||||
REPLAY-CORE-185-003 | DONE (2025-11-28) | Define Mongo collections (`replay_runs`, `replay_bundles`, `replay_subjects`) and indices, then author `docs/data/replay_schema.md` detailing schema fields, constraints, and offline sync strategy. | Platform Data Guild (src/__Libraries/StellaOps.Replay.Core)
|
||||
DOCS-REPLAY-185-003 | DONE (2025-11-28) | Author `docs/data/replay_schema.md` detailing `replay_runs`, `replay_bundles`, `replay_subjects` collections, index guidance, and offline sync strategy aligned with Replay CAS. | Docs Guild, Platform Data Guild (docs)
|
||||
DOCS-REPLAY-185-004 | DONE (2025-11-28) | Expand `docs/replay/DEVS_GUIDE_REPLAY.md` with integration guidance for consuming services (Scanner, Evidence Locker, CLI) and add checklist derived from `docs/replay/DETERMINISTIC_REPLAY.md` Section 11. | Docs Guild (docs)
|
||||
|
||||
> 2025-11-03: Replay CAS section published in `docs/modules/platform/architecture-overview.md` §5 — owners can move REPLAY-CORE-185-001/002 to **DOING** once library scaffolding begins.
|
||||
|
||||
## Implementation Status (2025-11-28)
|
||||
|
||||
All tasks verified complete:
|
||||
|
||||
- **REPLAY-CORE-185-001**: Library scaffolded with `CanonicalJson.cs`, `DeterministicHash.cs`, `DsseEnvelope.cs`, `ReplayManifest.cs`, `ReplayManifestExtensions.cs`; `AGENTS.md` published.
|
||||
- **REPLAY-CORE-185-002**: `ReplayBundleWriter.cs` and `ReplayBundleEntry.cs` implement tar.zst CAS bundle operations; Replay CAS documented in architecture-overview.md §5.
|
||||
- **REPLAY-CORE-185-003**: `ReplayMongoModels.cs` defines `ReplayRunDocument`, `ReplayBundleDocument`, `ReplaySubjectDocument` with `ReplayIndexes` constants.
|
||||
- **DOCS-REPLAY-185-003**: `docs/data/replay_schema.md` published with collection schemas, indexes, and determinism constraints.
|
||||
- **DOCS-REPLAY-185-004**: `docs/replay/DEVS_GUIDE_REPLAY.md` expanded with developer checklist, storage schema references, and workflow guidance.
|
||||
|
||||
@@ -5,6 +5,14 @@ Active items only. Completed/historic work now resides in docs/implplan/archived
|
||||
[Experience & SDKs] 180.E) UI.II
|
||||
Depends on: Sprint 180.E - UI.I
|
||||
Summary: Experience & SDKs focus on UI (phase II).
|
||||
|
||||
## Related Sprints & Advisories
|
||||
|
||||
- **SPRINT_0215_0001_0001_vuln_triage_ux.md** - Comprehensive vulnerability triage UX with VEX-first decisioning
|
||||
- **Advisory:** `28-Nov-2025 - Vulnerability Triage UX & VEX-First Decisioning.md`
|
||||
- **Schemas:** `docs/schemas/vex-decision.schema.json`, `docs/schemas/audit-bundle-index.schema.json`
|
||||
|
||||
Note: UI-LNM-22-003 (VEX tab) should align with VEX decision model defined in SPRINT_0215. The VEX modal and decision workflows are detailed in the new sprint.
|
||||
Task ID | State | Task description | Owners (Source)
|
||||
--- | --- | --- | ---
|
||||
UI-LNM-22-002 | TODO | Implement filters (source, severity bucket, conflict-only, CVSS vector presence) and pagination/lazy loading for large linksets. Docs depend on finalized filtering UX. Dependencies: UI-LNM-22-001. | UI Guild (src/UI/StellaOps.UI)
|
||||
|
||||
@@ -0,0 +1,89 @@
|
||||
# PostgreSQL Conversion Project Overview
|
||||
|
||||
## Project Summary
|
||||
|
||||
**Objective:** Convert StellaOps control-plane domains from MongoDB to PostgreSQL using a strangler fig pattern for gradual rollout.
|
||||
|
||||
**Timeline:** 10-12 sprints (Phases 0-7)
|
||||
|
||||
**Reference Documentation:** `docs/db/` directory
|
||||
|
||||
## Sprint Index
|
||||
|
||||
| Sprint | Phase | Module | Status | Dependencies |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| [3400](SPRINT_3400_0001_0001_postgres_foundations.md) | 0 | Foundations | IN_PROGRESS | None |
|
||||
| [3401](SPRINT_3401_0001_0001_postgres_authority.md) | 1 | Authority | TODO | Phase 0 |
|
||||
| [3402](SPRINT_3402_0001_0001_postgres_scheduler.md) | 2 | Scheduler | TODO | Phase 0 |
|
||||
| [3403](SPRINT_3403_0001_0001_postgres_notify.md) | 3 | Notify | TODO | Phase 0 |
|
||||
| [3404](SPRINT_3404_0001_0001_postgres_policy.md) | 4 | Policy | TODO | Phase 0 |
|
||||
| [3405](SPRINT_3405_0001_0001_postgres_vulnerabilities.md) | 5 | Vulnerabilities | TODO | Phase 0 |
|
||||
| [3406](SPRINT_3406_0001_0001_postgres_vex_graph.md) | 6 | VEX & Graph | TODO | Phase 5 |
|
||||
| [3407](SPRINT_3407_0001_0001_postgres_cleanup.md) | 7 | Cleanup | TODO | All |
|
||||
|
||||
## Dependency Graph
|
||||
|
||||
```
|
||||
Phase 0 (Foundations)
|
||||
├─→ Phase 1 (Authority) ──┐
|
||||
├─→ Phase 2 (Scheduler) ──┤
|
||||
├─→ Phase 3 (Notify) ──┼─→ Phase 7 (Cleanup)
|
||||
├─→ Phase 4 (Policy) ──┤
|
||||
└─→ Phase 5 (Vulnerabilities) ─→ Phase 6 (VEX/Graph) ─┘
|
||||
```
|
||||
|
||||
## Key Principles
|
||||
|
||||
1. **Strangler Fig Pattern:** Introduce PostgreSQL repositories alongside MongoDB, gradually switch per module.
|
||||
2. **Dual-Write for Tier A:** Critical data (auth, tokens) uses dual-write during transition.
|
||||
3. **Determinism Preserved:** Same inputs must produce identical outputs (especially graph_revision_id).
|
||||
4. **Multi-Tenancy:** Row-level isolation via `tenant_id` column.
|
||||
5. **Offline-First:** All operations must work in air-gapped environments.
|
||||
|
||||
## Data Tiering
|
||||
|
||||
| Tier | Examples | Migration Strategy |
|
||||
| --- | --- | --- |
|
||||
| **Tier A (Critical)** | Tenants, users, tokens, API keys | Dual-write, extensive verification |
|
||||
| **Tier B (Important)** | Jobs, advisories, VEX statements | Conversion with comparison tests |
|
||||
| **Tier C (Ephemeral)** | Metrics, audit logs | Recreate from scratch |
|
||||
|
||||
## Critical Success Factors
|
||||
|
||||
1. **Graph Revision ID Stability** - Phase 6 determinism is CRITICAL
|
||||
2. **Vulnerability Matching Parity** - Phase 5 must produce identical results
|
||||
3. **Zero Data Loss** - Tier A data must be 100% preserved
|
||||
4. **Performance Parity** - PostgreSQL must match or exceed MongoDB performance
|
||||
|
||||
## Documentation
|
||||
|
||||
| Document | Location | Purpose |
|
||||
| --- | --- | --- |
|
||||
| Specification | `docs/db/SPECIFICATION.md` | Complete PostgreSQL schema design |
|
||||
| Rules | `docs/db/RULES.md` | Coding conventions and patterns |
|
||||
| Verification | `docs/db/VERIFICATION.md` | Testing requirements |
|
||||
| Conversion Plan | `docs/db/CONVERSION_PLAN.md` | Strategic plan |
|
||||
| Task Definitions | `docs/db/tasks/PHASE_*.md` | Detailed task breakdowns |
|
||||
|
||||
## Current Status
|
||||
|
||||
### Phase 0: Foundations - IN PROGRESS
|
||||
- [x] `StellaOps.Infrastructure.Postgres` library created
|
||||
- [x] `DataSourceBase` implemented
|
||||
- [x] `RepositoryBase` implemented
|
||||
- [x] `MigrationRunner` implemented
|
||||
- [x] `PostgresOptions` and `PersistenceOptions` created
|
||||
- [x] `PostgresFixture` for testing created
|
||||
- [ ] Projects added to solution file
|
||||
- [ ] PostgreSQL cluster provisioned
|
||||
- [ ] CI pipeline integrated
|
||||
|
||||
### Upcoming
|
||||
- Phase 1-4 can run in parallel after Phase 0 completes
|
||||
- Phase 5 must complete before Phase 6
|
||||
- Phase 7 runs after all other phases complete
|
||||
|
||||
---
|
||||
|
||||
*Created: 2025-11-28*
|
||||
*Last Updated: 2025-11-28*
|
||||
74
docs/implplan/SPRINT_3400_0001_0001_postgres_foundations.md
Normal file
74
docs/implplan/SPRINT_3400_0001_0001_postgres_foundations.md
Normal file
@@ -0,0 +1,74 @@
|
||||
# Sprint 3400 · PostgreSQL Conversion: Phase 0 - Foundations
|
||||
|
||||
## Topic & Scope
|
||||
- Phase 0 of MongoDB to PostgreSQL conversion: Infrastructure & shared library setup.
|
||||
- Create shared PostgreSQL infrastructure library (`StellaOps.Infrastructure.Postgres`).
|
||||
- Establish patterns for DataSource, Repository, and Migration framework.
|
||||
- Set up CI/CD pipeline for PostgreSQL testing.
|
||||
- **Working directory:** src/__Libraries/StellaOps.Infrastructure.Postgres
|
||||
|
||||
## Dependencies & Concurrency
|
||||
- Upstream: None (foundational work).
|
||||
- Concurrency: Independent; must complete before Phase 1-7 sprints begin.
|
||||
- Reference: `docs/db/tasks/PHASE_0_FOUNDATIONS.md`
|
||||
|
||||
## Documentation Prerequisites
|
||||
- docs/db/README.md
|
||||
- docs/db/SPECIFICATION.md
|
||||
- docs/db/RULES.md
|
||||
- docs/db/VERIFICATION.md
|
||||
- docs/db/CONVERSION_PLAN.md
|
||||
|
||||
## Delivery Tracker
|
||||
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| 1 | PG-T0.1.1 | DONE | Infrastructure library created | Infrastructure Guild | Create `StellaOps.Infrastructure.Postgres` project structure |
|
||||
| 2 | PG-T0.1.2 | DONE | NuGet references added | Infrastructure Guild | Add Npgsql 9.x and Microsoft.Extensions packages |
|
||||
| 3 | PG-T0.2.1 | DONE | DataSourceBase implemented | Infrastructure Guild | Create abstract `DataSourceBase` class with connection pooling |
|
||||
| 4 | PG-T0.2.2 | DONE | Tenant context implemented | Infrastructure Guild | Implement `OpenConnectionAsync` with `SET app.current_tenant` |
|
||||
| 5 | PG-T0.2.3 | DONE | Session configuration implemented | Infrastructure Guild | Add UTC timezone, statement timeout, search path |
|
||||
| 6 | PG-T0.3.1 | DONE | RepositoryBase implemented | Infrastructure Guild | Create `RepositoryBase<TDataSource>` with query helpers |
|
||||
| 7 | PG-T0.3.2 | DONE | Parameter helpers implemented | Infrastructure Guild | Add JSONB, array, and nullable parameter helpers |
|
||||
| 8 | PG-T0.3.3 | DONE | Pagination helpers implemented | Infrastructure Guild | Add `BuildOrderByClause` and `BuildPaginationClause` |
|
||||
| 9 | PG-T0.4.1 | DONE | MigrationRunner implemented | Infrastructure Guild | Create SQL migration runner with checksum tracking |
|
||||
| 10 | PG-T0.4.2 | DONE | Schema management implemented | Infrastructure Guild | Add schema creation and migration table setup |
|
||||
| 11 | PG-T0.5.1 | DONE | PostgresOptions created | Infrastructure Guild | Create options class for connection settings |
|
||||
| 12 | PG-T0.5.2 | DONE | PersistenceOptions created | Infrastructure Guild | Create backend switching options (Mongo/Postgres/DualWrite) |
|
||||
| 13 | PG-T0.5.3 | DONE | DI extensions created | Infrastructure Guild | Create `ServiceCollectionExtensions` for registration |
|
||||
| 14 | PG-T0.6.1 | DONE | PostgresFixture created | Infrastructure Guild | Create test fixture with Testcontainers support |
|
||||
| 15 | PG-T0.6.2 | DONE | Test project created | Infrastructure Guild | Create `StellaOps.Infrastructure.Postgres.Tests` project |
|
||||
| 16 | PG-T0.6.3 | DONE | Exception helpers created | Infrastructure Guild | Create `PostgresExceptionHelper` for error handling |
|
||||
| 17 | PG-T0.7 | DONE | Update solution file | Infrastructure Guild | Add new projects to `StellaOps.sln` |
|
||||
| 18 | PG-T0.8 | TODO | PostgreSQL cluster provisioning | DevOps Guild | Provision PostgreSQL 16 for staging/production |
|
||||
| 19 | PG-T0.9 | TODO | CI pipeline integration | DevOps Guild | Add PostgreSQL Testcontainers to CI workflow |
|
||||
|
||||
## Execution Log
|
||||
| Date (UTC) | Update | Owner |
|
||||
| --- | --- | --- |
|
||||
| 2025-11-28 | Created `StellaOps.Infrastructure.Postgres` library with DataSourceBase, RepositoryBase, MigrationRunner | Infrastructure Guild |
|
||||
| 2025-11-28 | Added PostgresOptions, PersistenceOptions, and ServiceCollectionExtensions | Infrastructure Guild |
|
||||
| 2025-11-28 | Created PostgresFixture for Testcontainers integration | Infrastructure Guild |
|
||||
| 2025-11-28 | Created test project; verified build succeeds | Infrastructure Guild |
|
||||
| 2025-11-28 | Sprint file created | Planning |
|
||||
| 2025-11-28 | Added all 7 PostgreSQL storage projects to StellaOps.sln | Infrastructure Guild |
|
||||
| 2025-11-28 | Created DataSource classes for all 6 modules | Infrastructure Guild |
|
||||
| 2025-11-28 | Created repository implementations for Authority, Scheduler, Concelier, Excititor | Infrastructure Guild |
|
||||
| 2025-11-28 | All PostgreSQL storage projects build successfully | Infrastructure Guild |
|
||||
|
||||
## Decisions & Risks
|
||||
- Using Npgsql 9.x for latest features and performance improvements.
|
||||
- Tenant context set via `set_config('app.current_tenant', ...)` for RLS compatibility.
|
||||
- Migration runner uses SHA256 checksums for change detection.
|
||||
- Test isolation via unique schema names per test class.
|
||||
|
||||
## Exit Criteria
|
||||
- [ ] All infrastructure library components implemented and tested
|
||||
- [ ] Projects added to solution file
|
||||
- [ ] CI/CD pipeline running PostgreSQL tests
|
||||
- [ ] PostgreSQL cluster provisioned for staging
|
||||
|
||||
## Next Checkpoints
|
||||
- Phase 1 (Authority) can begin once CI pipeline is integrated.
|
||||
|
||||
---
|
||||
*Reference: docs/db/tasks/PHASE_0_FOUNDATIONS.md*
|
||||
70
docs/implplan/SPRINT_3401_0001_0001_postgres_authority.md
Normal file
70
docs/implplan/SPRINT_3401_0001_0001_postgres_authority.md
Normal file
@@ -0,0 +1,70 @@
|
||||
# Sprint 3401 · PostgreSQL Conversion: Phase 1 - Authority Module
|
||||
|
||||
## Topic & Scope
|
||||
- Phase 1 of MongoDB to PostgreSQL conversion: Authority module (IAM, tenants, tokens).
|
||||
- Create `StellaOps.Authority.Storage.Postgres` project.
|
||||
- Implement all 12+ repository interfaces for Authority schema.
|
||||
- Tier A data: requires dual-write verification before cutover.
|
||||
- **Working directory:** src/Authority/__Libraries/StellaOps.Authority.Storage.Postgres
|
||||
|
||||
## Dependencies & Concurrency
|
||||
- Upstream: Sprint 3400 (Phase 0 - Foundations) must be DONE.
|
||||
- Concurrency: Can run in parallel with Phase 2-4 after foundations complete.
|
||||
- Reference: `docs/db/tasks/PHASE_1_AUTHORITY.md`
|
||||
|
||||
## Documentation Prerequisites
|
||||
- docs/db/README.md
|
||||
- docs/db/SPECIFICATION.md (Section 5.1 - Authority Schema)
|
||||
- docs/db/RULES.md
|
||||
- src/Authority/AGENTS.md
|
||||
|
||||
## Delivery Tracker
|
||||
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| 1 | PG-T1.1 | TODO | Depends on PG-T0.7 | Authority Guild | Create `StellaOps.Authority.Storage.Postgres` project structure |
|
||||
| 2 | PG-T1.2.1 | TODO | Depends on PG-T1.1 | Authority Guild | Create schema migration for `authority` schema |
|
||||
| 3 | PG-T1.2.2 | TODO | Depends on PG-T1.2.1 | Authority Guild | Create `tenants` table with indexes |
|
||||
| 4 | PG-T1.2.3 | TODO | Depends on PG-T1.2.1 | Authority Guild | Create `users`, `roles`, `permissions` tables |
|
||||
| 5 | PG-T1.2.4 | TODO | Depends on PG-T1.2.1 | Authority Guild | Create `tokens`, `refresh_tokens`, `api_keys` tables |
|
||||
| 6 | PG-T1.2.5 | TODO | Depends on PG-T1.2.1 | Authority Guild | Create `sessions`, `audit` tables |
|
||||
| 7 | PG-T1.3 | TODO | Depends on PG-T1.2 | Authority Guild | Implement `AuthorityDataSource` class |
|
||||
| 8 | PG-T1.4.1 | TODO | Depends on PG-T1.3 | Authority Guild | Implement `ITenantRepository` |
|
||||
| 9 | PG-T1.4.2 | TODO | Depends on PG-T1.3 | Authority Guild | Implement `IUserRepository` with password hash handling |
|
||||
| 10 | PG-T1.4.3 | TODO | Depends on PG-T1.3 | Authority Guild | Implement `IRoleRepository` |
|
||||
| 11 | PG-T1.4.4 | TODO | Depends on PG-T1.3 | Authority Guild | Implement `IPermissionRepository` |
|
||||
| 12 | PG-T1.5.1 | TODO | Depends on PG-T1.3 | Authority Guild | Implement `ITokenRepository` |
|
||||
| 13 | PG-T1.5.2 | TODO | Depends on PG-T1.3 | Authority Guild | Implement `IRefreshTokenRepository` |
|
||||
| 14 | PG-T1.5.3 | TODO | Depends on PG-T1.3 | Authority Guild | Implement `IApiKeyRepository` |
|
||||
| 15 | PG-T1.6.1 | TODO | Depends on PG-T1.3 | Authority Guild | Implement `ISessionRepository` |
|
||||
| 16 | PG-T1.6.2 | TODO | Depends on PG-T1.3 | Authority Guild | Implement `IAuditRepository` |
|
||||
| 17 | PG-T1.7 | TODO | Depends on PG-T1.4-6 | Authority Guild | Add configuration switch in `ServiceCollectionExtensions` |
|
||||
| 18 | PG-T1.8.1 | TODO | Depends on PG-T1.7 | Authority Guild | Write integration tests for all repositories |
|
||||
| 19 | PG-T1.8.2 | TODO | Depends on PG-T1.8.1 | Authority Guild | Write determinism tests for token generation |
|
||||
| 20 | PG-T1.9 | TODO | Depends on PG-T1.8 | Authority Guild | Optional: Implement dual-write wrapper for Tier A verification |
|
||||
| 21 | PG-T1.10 | TODO | Depends on PG-T1.8 | Authority Guild | Run backfill from MongoDB to PostgreSQL |
|
||||
| 22 | PG-T1.11 | TODO | Depends on PG-T1.10 | Authority Guild | Verify data integrity: row counts, checksums |
|
||||
| 23 | PG-T1.12 | TODO | Depends on PG-T1.11 | Authority Guild | Switch Authority to PostgreSQL-only |
|
||||
|
||||
## Execution Log
|
||||
| Date (UTC) | Update | Owner |
|
||||
| --- | --- | --- |
|
||||
| 2025-11-28 | Sprint file created | Planning |
|
||||
|
||||
## Decisions & Risks
|
||||
- Password hashes stored as TEXT; Argon2id parameters in separate columns.
|
||||
- Token expiry uses `TIMESTAMPTZ` for timezone-aware comparisons.
|
||||
- Audit log may grow large; consider partitioning by `created_at` in production.
|
||||
- Dual-write mode optional but recommended for Tier A data verification.
|
||||
|
||||
## Exit Criteria
|
||||
- [ ] All 12+ repository interfaces implemented
|
||||
- [ ] Schema migrations idempotent and tested
|
||||
- [ ] All integration tests pass with Testcontainers
|
||||
- [ ] Data backfill completed and verified
|
||||
- [ ] Authority running on PostgreSQL in staging
|
||||
|
||||
## Next Checkpoints
|
||||
- Coordinate with Phase 2 (Scheduler) for any shared user/tenant references.
|
||||
|
||||
---
|
||||
*Reference: docs/db/tasks/PHASE_1_AUTHORITY.md*
|
||||
70
docs/implplan/SPRINT_3402_0001_0001_postgres_scheduler.md
Normal file
70
docs/implplan/SPRINT_3402_0001_0001_postgres_scheduler.md
Normal file
@@ -0,0 +1,70 @@
|
||||
# Sprint 3402 · PostgreSQL Conversion: Phase 2 - Scheduler Module
|
||||
|
||||
## Topic & Scope
|
||||
- Phase 2 of MongoDB to PostgreSQL conversion: Scheduler module.
|
||||
- Create `StellaOps.Scheduler.Storage.Postgres` project.
|
||||
- Implement job queue, triggers, and distributed locking with PostgreSQL advisory locks.
|
||||
- Critical: preserve deterministic trigger calculation.
|
||||
- **Working directory:** src/Scheduler/__Libraries/StellaOps.Scheduler.Storage.Postgres
|
||||
|
||||
## Dependencies & Concurrency
|
||||
- Upstream: Sprint 3400 (Phase 0 - Foundations) must be DONE.
|
||||
- Concurrency: Can run in parallel with Phase 1, 3, 4 after foundations complete.
|
||||
- Reference: `docs/db/tasks/PHASE_2_SCHEDULER.md`
|
||||
|
||||
## Documentation Prerequisites
|
||||
- docs/db/README.md
|
||||
- docs/db/SPECIFICATION.md (Section 5.4 - Scheduler Schema)
|
||||
- docs/db/RULES.md
|
||||
- src/Scheduler/AGENTS.md
|
||||
|
||||
## Delivery Tracker
|
||||
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| 1 | PG-T2.1 | TODO | Depends on PG-T0.7 | Scheduler Guild | Create `StellaOps.Scheduler.Storage.Postgres` project structure |
|
||||
| 2 | PG-T2.2.1 | TODO | Depends on PG-T2.1 | Scheduler Guild | Create schema migration for `scheduler` schema |
|
||||
| 3 | PG-T2.2.2 | TODO | Depends on PG-T2.2.1 | Scheduler Guild | Create `jobs` table with status enum and indexes |
|
||||
| 4 | PG-T2.2.3 | TODO | Depends on PG-T2.2.1 | Scheduler Guild | Create `triggers` table with cron expression support |
|
||||
| 5 | PG-T2.2.4 | TODO | Depends on PG-T2.2.1 | Scheduler Guild | Create `workers`, `leases` tables |
|
||||
| 6 | PG-T2.2.5 | TODO | Depends on PG-T2.2.1 | Scheduler Guild | Create `job_history`, `metrics` tables |
|
||||
| 7 | PG-T2.3 | TODO | Depends on PG-T2.2 | Scheduler Guild | Implement `SchedulerDataSource` class |
|
||||
| 8 | PG-T2.4.1 | TODO | Depends on PG-T2.3 | Scheduler Guild | Implement `IJobRepository` with `FOR UPDATE SKIP LOCKED` |
|
||||
| 9 | PG-T2.4.2 | TODO | Depends on PG-T2.3 | Scheduler Guild | Implement `ITriggerRepository` with next-fire calculation |
|
||||
| 10 | PG-T2.4.3 | TODO | Depends on PG-T2.3 | Scheduler Guild | Implement `IWorkerRepository` for heartbeat tracking |
|
||||
| 11 | PG-T2.5.1 | TODO | Depends on PG-T2.3 | Scheduler Guild | Implement distributed lock using `pg_advisory_lock` |
|
||||
| 12 | PG-T2.5.2 | TODO | Depends on PG-T2.5.1 | Scheduler Guild | Implement `IDistributedLockRepository` interface |
|
||||
| 13 | PG-T2.6.1 | TODO | Depends on PG-T2.3 | Scheduler Guild | Implement `IJobHistoryRepository` |
|
||||
| 14 | PG-T2.6.2 | TODO | Depends on PG-T2.3 | Scheduler Guild | Implement `IMetricsRepository` |
|
||||
| 15 | PG-T2.7 | TODO | Depends on PG-T2.4-6 | Scheduler Guild | Add configuration switch in `ServiceCollectionExtensions` |
|
||||
| 16 | PG-T2.8.1 | TODO | Depends on PG-T2.7 | Scheduler Guild | Write integration tests for job queue operations |
|
||||
| 17 | PG-T2.8.2 | TODO | Depends on PG-T2.8.1 | Scheduler Guild | Write determinism tests for trigger calculations |
|
||||
| 18 | PG-T2.8.3 | TODO | Depends on PG-T2.8.1 | Scheduler Guild | Write concurrency tests for distributed locking |
|
||||
| 19 | PG-T2.9 | TODO | Depends on PG-T2.8 | Scheduler Guild | Run backfill from MongoDB to PostgreSQL |
|
||||
| 20 | PG-T2.10 | TODO | Depends on PG-T2.9 | Scheduler Guild | Verify data integrity and trigger timing |
|
||||
| 21 | PG-T2.11 | TODO | Depends on PG-T2.10 | Scheduler Guild | Switch Scheduler to PostgreSQL-only |
|
||||
|
||||
## Execution Log
|
||||
| Date (UTC) | Update | Owner |
|
||||
| --- | --- | --- |
|
||||
| 2025-11-28 | Sprint file created | Planning |
|
||||
|
||||
## Decisions & Risks
|
||||
- PostgreSQL advisory locks replace MongoDB distributed locks.
|
||||
- `FOR UPDATE SKIP LOCKED` for efficient job claiming without contention.
|
||||
- Cron expressions stored as TEXT; next-fire computed in application.
|
||||
- Job payload stored as JSONB for flexibility.
|
||||
- Risk: advisory lock key collision; use tenant-scoped hash values.
|
||||
|
||||
## Exit Criteria
|
||||
- [ ] All repository interfaces implemented
|
||||
- [ ] Distributed locking working with advisory locks
|
||||
- [ ] Trigger calculations deterministic
|
||||
- [ ] All integration and concurrency tests pass
|
||||
- [ ] Scheduler running on PostgreSQL in staging
|
||||
|
||||
## Next Checkpoints
|
||||
- Validate job throughput matches MongoDB performance.
|
||||
- Coordinate with Orchestrator for any job handoff patterns.
|
||||
|
||||
---
|
||||
*Reference: docs/db/tasks/PHASE_2_SCHEDULER.md*
|
||||
76
docs/implplan/SPRINT_3403_0001_0001_postgres_notify.md
Normal file
76
docs/implplan/SPRINT_3403_0001_0001_postgres_notify.md
Normal file
@@ -0,0 +1,76 @@
|
||||
# Sprint 3403 · PostgreSQL Conversion: Phase 3 - Notify Module
|
||||
|
||||
## Topic & Scope
|
||||
- Phase 3 of MongoDB to PostgreSQL conversion: Notify module.
|
||||
- Create `StellaOps.Notify.Storage.Postgres` project.
|
||||
- Implement 15 repository interfaces for notification delivery and escalation.
|
||||
- Handle delivery tracking, digest aggregation, and escalation state.
|
||||
- **Working directory:** src/Notify/__Libraries/StellaOps.Notify.Storage.Postgres
|
||||
|
||||
## Dependencies & Concurrency
|
||||
- Upstream: Sprint 3400 (Phase 0 - Foundations) must be DONE.
|
||||
- Concurrency: Can run in parallel with Phase 1, 2, 4 after foundations complete.
|
||||
- Reference: `docs/db/tasks/PHASE_3_NOTIFY.md`
|
||||
|
||||
## Documentation Prerequisites
|
||||
- docs/db/README.md
|
||||
- docs/db/SPECIFICATION.md (Section 5.5 - Notify Schema)
|
||||
- docs/db/RULES.md
|
||||
- src/Notify/AGENTS.md (if exists)
|
||||
|
||||
## Delivery Tracker
|
||||
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| 1 | PG-T3.1 | TODO | Depends on PG-T0.7 | Notify Guild | Create `StellaOps.Notify.Storage.Postgres` project structure |
|
||||
| 2 | PG-T3.2.1 | TODO | Depends on PG-T3.1 | Notify Guild | Create schema migration for `notify` schema |
|
||||
| 3 | PG-T3.2.2 | TODO | Depends on PG-T3.2.1 | Notify Guild | Create `channels` table (email, slack, teams, webhook) |
|
||||
| 4 | PG-T3.2.3 | TODO | Depends on PG-T3.2.1 | Notify Guild | Create `rules`, `templates` tables |
|
||||
| 5 | PG-T3.2.4 | TODO | Depends on PG-T3.2.1 | Notify Guild | Create `deliveries` table with status tracking |
|
||||
| 6 | PG-T3.2.5 | TODO | Depends on PG-T3.2.1 | Notify Guild | Create `digests`, `quiet_hours`, `maintenance_windows` tables |
|
||||
| 7 | PG-T3.2.6 | TODO | Depends on PG-T3.2.1 | Notify Guild | Create `escalation_policies`, `escalation_states` tables |
|
||||
| 8 | PG-T3.2.7 | TODO | Depends on PG-T3.2.1 | Notify Guild | Create `on_call_schedules`, `inbox`, `incidents` tables |
|
||||
| 9 | PG-T3.3 | TODO | Depends on PG-T3.2 | Notify Guild | Implement `NotifyDataSource` class |
|
||||
| 10 | PG-T3.4.1 | TODO | Depends on PG-T3.3 | Notify Guild | Implement `IChannelRepository` |
|
||||
| 11 | PG-T3.4.2 | TODO | Depends on PG-T3.3 | Notify Guild | Implement `IRuleRepository` with filter JSONB |
|
||||
| 12 | PG-T3.4.3 | TODO | Depends on PG-T3.3 | Notify Guild | Implement `ITemplateRepository` with localization |
|
||||
| 13 | PG-T3.5.1 | TODO | Depends on PG-T3.3 | Notify Guild | Implement `IDeliveryRepository` with status transitions |
|
||||
| 14 | PG-T3.5.2 | TODO | Depends on PG-T3.3 | Notify Guild | Implement retry logic for failed deliveries |
|
||||
| 15 | PG-T3.6.1 | TODO | Depends on PG-T3.3 | Notify Guild | Implement `IDigestRepository` |
|
||||
| 16 | PG-T3.6.2 | TODO | Depends on PG-T3.3 | Notify Guild | Implement `IQuietHoursRepository` |
|
||||
| 17 | PG-T3.6.3 | TODO | Depends on PG-T3.3 | Notify Guild | Implement `IMaintenanceWindowRepository` |
|
||||
| 18 | PG-T3.7.1 | TODO | Depends on PG-T3.3 | Notify Guild | Implement `IEscalationPolicyRepository` |
|
||||
| 19 | PG-T3.7.2 | TODO | Depends on PG-T3.3 | Notify Guild | Implement `IEscalationStateRepository` |
|
||||
| 20 | PG-T3.7.3 | TODO | Depends on PG-T3.3 | Notify Guild | Implement `IOnCallScheduleRepository` |
|
||||
| 21 | PG-T3.8.1 | TODO | Depends on PG-T3.3 | Notify Guild | Implement `IInboxRepository` |
|
||||
| 22 | PG-T3.8.2 | TODO | Depends on PG-T3.3 | Notify Guild | Implement `IIncidentRepository` |
|
||||
| 23 | PG-T3.8.3 | TODO | Depends on PG-T3.3 | Notify Guild | Implement `IAuditRepository` |
|
||||
| 24 | PG-T3.9 | TODO | Depends on PG-T3.4-8 | Notify Guild | Add configuration switch in `ServiceCollectionExtensions` |
|
||||
| 25 | PG-T3.10.1 | TODO | Depends on PG-T3.9 | Notify Guild | Write integration tests for all repositories |
|
||||
| 26 | PG-T3.10.2 | TODO | Depends on PG-T3.10.1 | Notify Guild | Test notification delivery flow end-to-end |
|
||||
| 27 | PG-T3.10.3 | TODO | Depends on PG-T3.10.1 | Notify Guild | Test escalation handling |
|
||||
| 28 | PG-T3.10.4 | TODO | Depends on PG-T3.10.1 | Notify Guild | Test digest aggregation |
|
||||
| 29 | PG-T3.11 | TODO | Depends on PG-T3.10 | Notify Guild | Switch Notify to PostgreSQL-only |
|
||||
|
||||
## Execution Log
|
||||
| Date (UTC) | Update | Owner |
|
||||
| --- | --- | --- |
|
||||
| 2025-11-28 | Sprint file created | Planning |
|
||||
|
||||
## Decisions & Risks
|
||||
- Channel configurations stored as JSONB for flexibility across channel types.
|
||||
- Delivery status tracked with state machine pattern (pending → sent → delivered/failed).
|
||||
- Escalation states may need frequent updates; index accordingly.
|
||||
- Digest aggregation queries may be complex; consider materialized views.
|
||||
|
||||
## Exit Criteria
|
||||
- [ ] All 15 repository interfaces implemented
|
||||
- [ ] Delivery tracking working end-to-end
|
||||
- [ ] Escalation logic verified
|
||||
- [ ] All integration tests pass
|
||||
- [ ] Notify running on PostgreSQL in staging
|
||||
|
||||
## Next Checkpoints
|
||||
- Coordinate with Scheduler for notification trigger integration.
|
||||
|
||||
---
|
||||
*Reference: docs/db/tasks/PHASE_3_NOTIFY.md*
|
||||
73
docs/implplan/SPRINT_3404_0001_0001_postgres_policy.md
Normal file
73
docs/implplan/SPRINT_3404_0001_0001_postgres_policy.md
Normal file
@@ -0,0 +1,73 @@
|
||||
# Sprint 3404 · PostgreSQL Conversion: Phase 4 - Policy Module
|
||||
|
||||
## Topic & Scope
|
||||
- Phase 4 of MongoDB to PostgreSQL conversion: Policy module.
|
||||
- Create `StellaOps.Policy.Storage.Postgres` project.
|
||||
- Implement policy pack versioning and risk profile management.
|
||||
- Handle OPA/Rego policy storage and evaluation run tracking.
|
||||
- **Working directory:** src/Policy/__Libraries/StellaOps.Policy.Storage.Postgres
|
||||
|
||||
## Dependencies & Concurrency
|
||||
- Upstream: Sprint 3400 (Phase 0 - Foundations) must be DONE.
|
||||
- Concurrency: Can run in parallel with Phase 1-3 after foundations complete.
|
||||
- Reference: `docs/db/tasks/PHASE_4_POLICY.md`
|
||||
|
||||
## Documentation Prerequisites
|
||||
- docs/db/README.md
|
||||
- docs/db/SPECIFICATION.md (Section 5.6 - Policy Schema)
|
||||
- docs/db/RULES.md
|
||||
- src/Policy/AGENTS.md (if exists)
|
||||
|
||||
## Delivery Tracker
|
||||
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| 1 | PG-T4.1 | TODO | Depends on PG-T0.7 | Policy Guild | Create `StellaOps.Policy.Storage.Postgres` project structure |
|
||||
| 2 | PG-T4.2.1 | TODO | Depends on PG-T4.1 | Policy Guild | Create schema migration for `policy` schema |
|
||||
| 3 | PG-T4.2.2 | TODO | Depends on PG-T4.2.1 | Policy Guild | Create `packs`, `pack_versions` tables |
|
||||
| 4 | PG-T4.2.3 | TODO | Depends on PG-T4.2.1 | Policy Guild | Create `rules` table with Rego content |
|
||||
| 5 | PG-T4.2.4 | TODO | Depends on PG-T4.2.1 | Policy Guild | Create `risk_profiles` table with version history |
|
||||
| 6 | PG-T4.2.5 | TODO | Depends on PG-T4.2.1 | Policy Guild | Create `evaluation_runs`, `explanations` tables |
|
||||
| 7 | PG-T4.2.6 | TODO | Depends on PG-T4.2.1 | Policy Guild | Create `exceptions`, `audit` tables |
|
||||
| 8 | PG-T4.3 | TODO | Depends on PG-T4.2 | Policy Guild | Implement `PolicyDataSource` class |
|
||||
| 9 | PG-T4.4.1 | TODO | Depends on PG-T4.3 | Policy Guild | Implement `IPackRepository` with CRUD |
|
||||
| 10 | PG-T4.4.2 | TODO | Depends on PG-T4.3 | Policy Guild | Implement version management for packs |
|
||||
| 11 | PG-T4.4.3 | TODO | Depends on PG-T4.3 | Policy Guild | Implement active version promotion |
|
||||
| 12 | PG-T4.5.1 | TODO | Depends on PG-T4.3 | Policy Guild | Implement `IRiskProfileRepository` |
|
||||
| 13 | PG-T4.5.2 | TODO | Depends on PG-T4.3 | Policy Guild | Implement version history for risk profiles |
|
||||
| 14 | PG-T4.5.3 | TODO | Depends on PG-T4.3 | Policy Guild | Implement `GetVersionAsync` and `ListVersionsAsync` |
|
||||
| 15 | PG-T4.6.1 | TODO | Depends on PG-T4.3 | Policy Guild | Implement `IEvaluationRunRepository` |
|
||||
| 16 | PG-T4.6.2 | TODO | Depends on PG-T4.3 | Policy Guild | Implement `IExplanationRepository` |
|
||||
| 17 | PG-T4.6.3 | TODO | Depends on PG-T4.3 | Policy Guild | Implement `IExceptionRepository` |
|
||||
| 18 | PG-T4.6.4 | TODO | Depends on PG-T4.3 | Policy Guild | Implement `IAuditRepository` |
|
||||
| 19 | PG-T4.7 | TODO | Depends on PG-T4.4-6 | Policy Guild | Add configuration switch in `ServiceCollectionExtensions` |
|
||||
| 20 | PG-T4.8.1 | TODO | Depends on PG-T4.7 | Policy Guild | Write integration tests for all repositories |
|
||||
| 21 | PG-T4.8.2 | TODO | Depends on PG-T4.8.1 | Policy Guild | Test pack versioning workflow |
|
||||
| 22 | PG-T4.8.3 | TODO | Depends on PG-T4.8.1 | Policy Guild | Test risk profile version history |
|
||||
| 23 | PG-T4.9 | TODO | Depends on PG-T4.8 | Policy Guild | Export active packs from MongoDB |
|
||||
| 24 | PG-T4.10 | TODO | Depends on PG-T4.9 | Policy Guild | Import packs to PostgreSQL |
|
||||
| 25 | PG-T4.11 | TODO | Depends on PG-T4.10 | Policy Guild | Verify version numbers and active version settings |
|
||||
| 26 | PG-T4.12 | TODO | Depends on PG-T4.11 | Policy Guild | Switch Policy to PostgreSQL-only |
|
||||
|
||||
## Execution Log
|
||||
| Date (UTC) | Update | Owner |
|
||||
| --- | --- | --- |
|
||||
| 2025-11-28 | Sprint file created | Planning |
|
||||
|
||||
## Decisions & Risks
|
||||
- Pack versions are immutable once published; new versions create new rows.
|
||||
- Rego content stored as TEXT; consider compression for large policies.
|
||||
- Evaluation results may grow rapidly; consider partitioning or archival.
|
||||
- Risk profile versioning critical for audit trail; never delete old versions.
|
||||
|
||||
## Exit Criteria
|
||||
- [ ] All repository interfaces implemented
|
||||
- [ ] Pack versioning working correctly
|
||||
- [ ] Risk profile version history maintained
|
||||
- [ ] All integration tests pass
|
||||
- [ ] Policy running on PostgreSQL in staging
|
||||
|
||||
## Next Checkpoints
|
||||
- Coordinate with Excititor for VEX policy integration.
|
||||
|
||||
---
|
||||
*Reference: docs/db/tasks/PHASE_4_POLICY.md*
|
||||
@@ -0,0 +1,90 @@
|
||||
# Sprint 3405 · PostgreSQL Conversion: Phase 5 - Vulnerabilities (Concelier)
|
||||
|
||||
## Topic & Scope
|
||||
- Phase 5 of MongoDB to PostgreSQL conversion: Concelier vulnerability index.
|
||||
- Create `StellaOps.Concelier.Storage.Postgres` project.
|
||||
- Implement full advisory schema with PURL matching and full-text search.
|
||||
- Critical: maintain deterministic vulnerability matching.
|
||||
- **Working directory:** src/Concelier/__Libraries/StellaOps.Concelier.Storage.Postgres
|
||||
|
||||
## Dependencies & Concurrency
|
||||
- Upstream: Sprint 3400 (Phase 0 - Foundations) must be DONE.
|
||||
- Concurrency: Should run after Phase 1-4; Excititor depends on this.
|
||||
- Reference: `docs/db/tasks/PHASE_5_VULNERABILITIES.md`
|
||||
|
||||
## Documentation Prerequisites
|
||||
- docs/db/README.md
|
||||
- docs/db/SPECIFICATION.md (Section 5.2 - Vulnerability Schema)
|
||||
- docs/db/RULES.md
|
||||
- src/Concelier/AGENTS.md
|
||||
|
||||
## Delivery Tracker
|
||||
|
||||
### Sprint 5a: Schema & Repositories
|
||||
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| 1 | PG-T5a.1 | TODO | Depends on PG-T0.7 | Concelier Guild | Create `StellaOps.Concelier.Storage.Postgres` project structure |
|
||||
| 2 | PG-T5a.2.1 | TODO | Depends on PG-T5a.1 | Concelier Guild | Create schema migration for `vuln` schema |
|
||||
| 3 | PG-T5a.2.2 | TODO | Depends on PG-T5a.2.1 | Concelier Guild | Create `sources`, `feed_snapshots` tables |
|
||||
| 4 | PG-T5a.2.3 | TODO | Depends on PG-T5a.2.1 | Concelier Guild | Create `advisories`, `advisory_snapshots` tables |
|
||||
| 5 | PG-T5a.2.4 | TODO | Depends on PG-T5a.2.1 | Concelier Guild | Create `advisory_aliases`, `advisory_cvss` tables |
|
||||
| 6 | PG-T5a.2.5 | TODO | Depends on PG-T5a.2.1 | Concelier Guild | Create `advisory_affected` with PURL matching indexes |
|
||||
| 7 | PG-T5a.2.6 | TODO | Depends on PG-T5a.2.1 | Concelier Guild | Create `advisory_references`, `advisory_credits`, `advisory_weaknesses` tables |
|
||||
| 8 | PG-T5a.2.7 | TODO | Depends on PG-T5a.2.1 | Concelier Guild | Create `kev_flags`, `source_states`, `merge_events` tables |
|
||||
| 9 | PG-T5a.2.8 | TODO | Depends on PG-T5a.2.1 | Concelier Guild | Add full-text search index on advisories |
|
||||
| 10 | PG-T5a.3 | TODO | Depends on PG-T5a.2 | Concelier Guild | Implement `ConcelierDataSource` class |
|
||||
| 11 | PG-T5a.4.1 | TODO | Depends on PG-T5a.3 | Concelier Guild | Implement `ISourceRepository` |
|
||||
| 12 | PG-T5a.4.2 | TODO | Depends on PG-T5a.3 | Concelier Guild | Implement `IAdvisoryRepository.GetByKeyAsync` |
|
||||
| 13 | PG-T5a.4.3 | TODO | Depends on PG-T5a.3 | Concelier Guild | Implement `IAdvisoryRepository.GetByAliasAsync` (CVE lookup) |
|
||||
| 14 | PG-T5a.4.4 | TODO | Depends on PG-T5a.3 | Concelier Guild | Implement `IAdvisoryRepository.SearchAsync` with full-text search |
|
||||
| 15 | PG-T5a.4.5 | TODO | Depends on PG-T5a.3 | Concelier Guild | Implement `IAdvisoryRepository.UpsertAsync` with all child tables |
|
||||
| 16 | PG-T5a.4.6 | TODO | Depends on PG-T5a.3 | Concelier Guild | Implement `IAdvisoryRepository.GetAffectingPackageAsync` (PURL match) |
|
||||
| 17 | PG-T5a.4.7 | TODO | Depends on PG-T5a.3 | Concelier Guild | Implement `IAdvisoryRepository.GetAffectingPackageNameAsync` |
|
||||
| 18 | PG-T5a.5.1 | TODO | Depends on PG-T5a.3 | Concelier Guild | Implement child table repositories (Alias, CVSS, Affected) |
|
||||
| 19 | PG-T5a.5.2 | TODO | Depends on PG-T5a.3 | Concelier Guild | Implement child table repositories (Reference, Credit, Weakness) |
|
||||
| 20 | PG-T5a.5.3 | TODO | Depends on PG-T5a.3 | Concelier Guild | Implement KEV and SourceState repositories |
|
||||
| 21 | PG-T5a.6 | TODO | Depends on PG-T5a.5 | Concelier Guild | Write integration tests for all repositories |
|
||||
|
||||
### Sprint 5b: Conversion & Verification
|
||||
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| 22 | PG-T5b.1.1 | TODO | Depends on PG-T5a.6 | Concelier Guild | Build `AdvisoryConverter` to parse MongoDB documents |
|
||||
| 23 | PG-T5b.1.2 | TODO | Depends on PG-T5b.1.1 | Concelier Guild | Map to relational structure with child tables |
|
||||
| 24 | PG-T5b.1.3 | TODO | Depends on PG-T5b.1.2 | Concelier Guild | Preserve provenance JSONB |
|
||||
| 25 | PG-T5b.1.4 | TODO | Depends on PG-T5b.1.2 | Concelier Guild | Handle version ranges (keep as JSONB) |
|
||||
| 26 | PG-T5b.2.1 | TODO | Depends on PG-T5b.1 | Concelier Guild | Update NVD importer to write to PostgreSQL |
|
||||
| 27 | PG-T5b.2.2 | TODO | Depends on PG-T5b.1 | Concelier Guild | Update OSV importer to write to PostgreSQL |
|
||||
| 28 | PG-T5b.2.3 | TODO | Depends on PG-T5b.1 | Concelier Guild | Update GHSA/vendor importers to write to PostgreSQL |
|
||||
| 29 | PG-T5b.3.1 | TODO | Depends on PG-T5b.2 | Concelier Guild | Configure dual-import mode |
|
||||
| 30 | PG-T5b.3.2 | TODO | Depends on PG-T5b.3.1 | Concelier Guild | Run import cycle and compare record counts |
|
||||
| 31 | PG-T5b.4.1 | TODO | Depends on PG-T5b.3 | Concelier Guild | Select sample SBOMs for verification |
|
||||
| 32 | PG-T5b.4.2 | TODO | Depends on PG-T5b.4.1 | Concelier Guild | Run matching with MongoDB backend |
|
||||
| 33 | PG-T5b.4.3 | TODO | Depends on PG-T5b.4.2 | Concelier Guild | Run matching with PostgreSQL backend |
|
||||
| 34 | PG-T5b.4.4 | TODO | Depends on PG-T5b.4.3 | Concelier Guild | Compare findings (must be identical) |
|
||||
| 35 | PG-T5b.5 | TODO | Depends on PG-T5b.4 | Concelier Guild | Performance optimization with EXPLAIN ANALYZE |
|
||||
| 36 | PG-T5b.6 | TODO | Depends on PG-T5b.5 | Concelier Guild | Switch Scanner/Concelier to PostgreSQL-only |
|
||||
|
||||
## Execution Log
|
||||
| Date (UTC) | Update | Owner |
|
||||
| --- | --- | --- |
|
||||
| 2025-11-28 | Sprint file created | Planning |
|
||||
|
||||
## Decisions & Risks
|
||||
- PURL stored as TEXT with GIN trigram index for efficient matching.
|
||||
- Version ranges stored as JSONB; too complex for relational decomposition.
|
||||
- Full-text search using `tsvector` column with GIN index.
|
||||
- Risk: matching discrepancies between backends; extensive comparison testing required.
|
||||
- Expected data volume: 300K+ advisories, 2M+ affected entries.
|
||||
|
||||
## Exit Criteria
|
||||
- [ ] All repository interfaces implemented
|
||||
- [ ] Advisory conversion pipeline working
|
||||
- [ ] Vulnerability matching produces identical results
|
||||
- [ ] Feed imports working on PostgreSQL
|
||||
- [ ] Concelier running on PostgreSQL in staging
|
||||
|
||||
## Next Checkpoints
|
||||
- Phase 6 (Excititor) depends on this completing successfully.
|
||||
|
||||
---
|
||||
*Reference: docs/db/tasks/PHASE_5_VULNERABILITIES.md*
|
||||
102
docs/implplan/SPRINT_3406_0001_0001_postgres_vex_graph.md
Normal file
102
docs/implplan/SPRINT_3406_0001_0001_postgres_vex_graph.md
Normal file
@@ -0,0 +1,102 @@
|
||||
# Sprint 3406 · PostgreSQL Conversion: Phase 6 - VEX & Graph (Excititor)
|
||||
|
||||
## Topic & Scope
|
||||
- Phase 6 of MongoDB to PostgreSQL conversion: Excititor VEX and graph storage.
|
||||
- Create `StellaOps.Excititor.Storage.Postgres` project.
|
||||
- Implement graph node/edge storage with efficient bulk operations.
|
||||
- **CRITICAL:** Preserve graph_revision_id stability (determinism required).
|
||||
- **Working directory:** src/Excititor/__Libraries/StellaOps.Excititor.Storage.Postgres
|
||||
|
||||
## Dependencies & Concurrency
|
||||
- Upstream: Sprint 3400 (Phase 0) and Sprint 3405 (Phase 5 - Vulnerabilities) must be DONE.
|
||||
- Concurrency: Must follow Phase 5 due to VEX-vulnerability relationships.
|
||||
- Reference: `docs/db/tasks/PHASE_6_VEX_GRAPH.md`
|
||||
|
||||
## Documentation Prerequisites
|
||||
- docs/db/README.md
|
||||
- docs/db/SPECIFICATION.md (Section 5.3 - VEX Schema)
|
||||
- docs/db/RULES.md
|
||||
- src/Excititor/AGENTS.md (if exists)
|
||||
|
||||
## Delivery Tracker
|
||||
|
||||
### Sprint 6a: Core Schema & Repositories
|
||||
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| 1 | PG-T6a.1 | TODO | Depends on PG-T5b.6 | Excititor Guild | Create `StellaOps.Excititor.Storage.Postgres` project structure |
|
||||
| 2 | PG-T6a.2.1 | TODO | Depends on PG-T6a.1 | Excititor Guild | Create schema migration for `vex` schema |
|
||||
| 3 | PG-T6a.2.2 | TODO | Depends on PG-T6a.2.1 | Excititor Guild | Create `projects`, `graph_revisions` tables |
|
||||
| 4 | PG-T6a.2.3 | TODO | Depends on PG-T6a.2.1 | Excititor Guild | Create `graph_nodes`, `graph_edges` tables (BIGSERIAL) |
|
||||
| 5 | PG-T6a.2.4 | TODO | Depends on PG-T6a.2.1 | Excititor Guild | Create `statements`, `observations` tables |
|
||||
| 6 | PG-T6a.2.5 | TODO | Depends on PG-T6a.2.1 | Excititor Guild | Create `linksets`, `linkset_events` tables |
|
||||
| 7 | PG-T6a.2.6 | TODO | Depends on PG-T6a.2.1 | Excititor Guild | Create `consensus`, `consensus_holds` tables |
|
||||
| 8 | PG-T6a.2.7 | TODO | Depends on PG-T6a.2.1 | Excititor Guild | Create remaining VEX tables (unknowns, evidence, cvss_receipts, etc.) |
|
||||
| 9 | PG-T6a.2.8 | TODO | Depends on PG-T6a.2.1 | Excititor Guild | Add indexes for graph traversal |
|
||||
| 10 | PG-T6a.3 | TODO | Depends on PG-T6a.2 | Excititor Guild | Implement `ExcititorDataSource` class |
|
||||
| 11 | PG-T6a.4.1 | TODO | Depends on PG-T6a.3 | Excititor Guild | Implement `IProjectRepository` with tenant scoping |
|
||||
| 12 | PG-T6a.4.2 | TODO | Depends on PG-T6a.3 | Excititor Guild | Implement `IVexStatementRepository` |
|
||||
| 13 | PG-T6a.4.3 | TODO | Depends on PG-T6a.3 | Excititor Guild | Implement `IVexObservationRepository` |
|
||||
| 14 | PG-T6a.5.1 | TODO | Depends on PG-T6a.3 | Excititor Guild | Implement `ILinksetRepository` |
|
||||
| 15 | PG-T6a.5.2 | TODO | Depends on PG-T6a.3 | Excititor Guild | Implement `IConsensusRepository` |
|
||||
| 16 | PG-T6a.6 | TODO | Depends on PG-T6a.5 | Excititor Guild | Write integration tests for core repositories |
|
||||
|
||||
### Sprint 6b: Graph Storage
|
||||
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| 17 | PG-T6b.1.1 | TODO | Depends on PG-T6a.6 | Excititor Guild | Implement `IGraphRevisionRepository.GetByIdAsync` |
|
||||
| 18 | PG-T6b.1.2 | TODO | Depends on PG-T6a.6 | Excititor Guild | Implement `IGraphRevisionRepository.GetByRevisionIdAsync` |
|
||||
| 19 | PG-T6b.1.3 | TODO | Depends on PG-T6a.6 | Excititor Guild | Implement `IGraphRevisionRepository.GetLatestByProjectAsync` |
|
||||
| 20 | PG-T6b.1.4 | TODO | Depends on PG-T6a.6 | Excititor Guild | Implement `IGraphRevisionRepository.CreateAsync` |
|
||||
| 21 | PG-T6b.2.1 | TODO | Depends on PG-T6b.1 | Excititor Guild | Implement `IGraphNodeRepository.GetByKeyAsync` |
|
||||
| 22 | PG-T6b.2.2 | TODO | Depends on PG-T6b.1 | Excititor Guild | Implement `IGraphNodeRepository.BulkInsertAsync` using COPY |
|
||||
| 23 | PG-T6b.2.3 | TODO | Depends on PG-T6b.2.2 | Excititor Guild | Optimize bulk insert for 10-100x performance |
|
||||
| 24 | PG-T6b.3.1 | TODO | Depends on PG-T6b.2 | Excititor Guild | Implement `IGraphEdgeRepository.GetByRevisionAsync` |
|
||||
| 25 | PG-T6b.3.2 | TODO | Depends on PG-T6b.2 | Excititor Guild | Implement `IGraphEdgeRepository.BulkInsertAsync` using COPY |
|
||||
| 26 | PG-T6b.3.3 | TODO | Depends on PG-T6b.2 | Excititor Guild | Implement traversal queries (GetOutgoingAsync, GetIncomingAsync) |
|
||||
| 27 | PG-T6b.4.1 | TODO | Depends on PG-T6b.3 | Excititor Guild | **CRITICAL:** Document revision_id computation algorithm |
|
||||
| 28 | PG-T6b.4.2 | TODO | Depends on PG-T6b.4.1 | Excititor Guild | **CRITICAL:** Verify nodes inserted in deterministic order |
|
||||
| 29 | PG-T6b.4.3 | TODO | Depends on PG-T6b.4.2 | Excititor Guild | **CRITICAL:** Verify edges inserted in deterministic order |
|
||||
| 30 | PG-T6b.4.4 | TODO | Depends on PG-T6b.4.3 | Excititor Guild | **CRITICAL:** Write stability tests (5x computation must match) |
|
||||
|
||||
### Sprint 6c: Migration & Verification
|
||||
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| 31 | PG-T6c.1.1 | TODO | Depends on PG-T6b.4 | Excititor Guild | Build graph conversion service for MongoDB documents |
|
||||
| 32 | PG-T6c.1.2 | TODO | Depends on PG-T6c.1.1 | Excititor Guild | Extract and insert nodes in deterministic order |
|
||||
| 33 | PG-T6c.1.3 | TODO | Depends on PG-T6c.1.2 | Excititor Guild | Extract and insert edges in deterministic order |
|
||||
| 34 | PG-T6c.2.1 | TODO | Depends on PG-T6c.1 | Excititor Guild | Build VEX statement conversion service |
|
||||
| 35 | PG-T6c.2.2 | TODO | Depends on PG-T6c.2.1 | Excititor Guild | Preserve provenance and evidence |
|
||||
| 36 | PG-T6c.3.1 | TODO | Depends on PG-T6c.2 | Excititor Guild | Select sample projects for dual pipeline comparison |
|
||||
| 37 | PG-T6c.3.2 | TODO | Depends on PG-T6c.3.1 | Excititor Guild | Compute graphs with MongoDB backend |
|
||||
| 38 | PG-T6c.3.3 | TODO | Depends on PG-T6c.3.2 | Excititor Guild | Compute graphs with PostgreSQL backend |
|
||||
| 39 | PG-T6c.3.4 | TODO | Depends on PG-T6c.3.3 | Excititor Guild | **CRITICAL:** Compare revision_ids (must match) |
|
||||
| 40 | PG-T6c.3.5 | TODO | Depends on PG-T6c.3.4 | Excititor Guild | Compare node/edge counts and VEX statements |
|
||||
| 41 | PG-T6c.4 | TODO | Depends on PG-T6c.3 | Excititor Guild | Migrate active projects |
|
||||
| 42 | PG-T6c.5 | TODO | Depends on PG-T6c.4 | Excititor Guild | Switch Excititor to PostgreSQL-only |
|
||||
|
||||
## Execution Log
|
||||
| Date (UTC) | Update | Owner |
|
||||
| --- | --- | --- |
|
||||
| 2025-11-28 | Sprint file created | Planning |
|
||||
|
||||
## Decisions & Risks
|
||||
- Graph nodes/edges use BIGSERIAL for high-volume IDs.
|
||||
- Bulk insert using PostgreSQL COPY for 10-100x performance.
|
||||
- **CRITICAL RISK:** Revision ID instability would break reproducibility guarantees.
|
||||
- Graph traversal indexes on `(from_node_id)` and `(to_node_id)`.
|
||||
- Estimated volumes: 10M+ nodes, 20M+ edges, 1M+ VEX statements.
|
||||
|
||||
## Exit Criteria
|
||||
- [ ] All repository interfaces implemented
|
||||
- [ ] Graph storage working efficiently with bulk operations
|
||||
- [ ] **Graph revision IDs stable (deterministic)** - CRITICAL
|
||||
- [ ] VEX statements preserved correctly
|
||||
- [ ] All comparison tests pass
|
||||
- [ ] Excititor running on PostgreSQL in staging
|
||||
|
||||
## Next Checkpoints
|
||||
- This is the most complex phase; allocate extra time for determinism verification.
|
||||
- Phase 7 (Cleanup) follows after successful cutover.
|
||||
|
||||
---
|
||||
*Reference: docs/db/tasks/PHASE_6_VEX_GRAPH.md*
|
||||
153
docs/implplan/SPRINT_3407_0001_0001_postgres_cleanup.md
Normal file
153
docs/implplan/SPRINT_3407_0001_0001_postgres_cleanup.md
Normal file
@@ -0,0 +1,153 @@
|
||||
# Sprint 3407 · PostgreSQL Conversion: Phase 7 - Cleanup & Optimization
|
||||
|
||||
## Topic & Scope
|
||||
- Phase 7 of MongoDB to PostgreSQL conversion: Final cleanup and optimization.
|
||||
- Remove MongoDB dependencies from all converted modules.
|
||||
- Archive MongoDB data and decommission infrastructure.
|
||||
- Optimize PostgreSQL performance and update documentation.
|
||||
- **Working directory:** Multiple (cleanup across all modules)
|
||||
|
||||
## Dependencies & Concurrency
|
||||
- Upstream: ALL previous phases (3400-3406) must be DONE.
|
||||
- Concurrency: Must run sequentially after all modules converted.
|
||||
- Reference: `docs/db/tasks/PHASE_7_CLEANUP.md`
|
||||
|
||||
## Documentation Prerequisites
|
||||
- docs/db/README.md
|
||||
- docs/db/SPECIFICATION.md
|
||||
- docs/db/RULES.md
|
||||
- docs/db/VERIFICATION.md
|
||||
- All module AGENTS.md files
|
||||
|
||||
## Delivery Tracker
|
||||
|
||||
### T7.1: Remove MongoDB Dependencies
|
||||
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| 1 | PG-T7.1.1 | TODO | All phases complete | Infrastructure Guild | Remove `StellaOps.Authority.Storage.Mongo` project |
|
||||
| 2 | PG-T7.1.2 | TODO | Depends on PG-T7.1.1 | Infrastructure Guild | Remove `StellaOps.Scheduler.Storage.Mongo` project |
|
||||
| 3 | PG-T7.1.3 | TODO | Depends on PG-T7.1.1 | Infrastructure Guild | Remove `StellaOps.Notify.Storage.Mongo` project |
|
||||
| 4 | PG-T7.1.4 | TODO | Depends on PG-T7.1.1 | Infrastructure Guild | Remove `StellaOps.Policy.Storage.Mongo` project |
|
||||
| 5 | PG-T7.1.5 | TODO | Depends on PG-T7.1.1 | Infrastructure Guild | Remove `StellaOps.Concelier.Storage.Mongo` project |
|
||||
| 6 | PG-T7.1.6 | TODO | Depends on PG-T7.1.1 | Infrastructure Guild | Remove `StellaOps.Excititor.Storage.Mongo` project |
|
||||
| 7 | PG-T7.1.7 | TODO | Depends on PG-T7.1.6 | Infrastructure Guild | Update solution files |
|
||||
| 8 | PG-T7.1.8 | TODO | Depends on PG-T7.1.7 | Infrastructure Guild | Remove dual-write wrappers |
|
||||
| 9 | PG-T7.1.9 | TODO | Depends on PG-T7.1.8 | Infrastructure Guild | Remove MongoDB configuration options |
|
||||
| 10 | PG-T7.1.10 | TODO | Depends on PG-T7.1.9 | Infrastructure Guild | Run full build to verify no broken references |
|
||||
|
||||
### T7.2: Archive MongoDB Data
|
||||
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| 11 | PG-T7.2.1 | TODO | Depends on PG-T7.1.10 | DevOps Guild | Take final MongoDB backup |
|
||||
| 12 | PG-T7.2.2 | TODO | Depends on PG-T7.2.1 | DevOps Guild | Export to BSON/JSON archives |
|
||||
| 13 | PG-T7.2.3 | TODO | Depends on PG-T7.2.2 | DevOps Guild | Store archives in secure location |
|
||||
| 14 | PG-T7.2.4 | TODO | Depends on PG-T7.2.3 | DevOps Guild | Document archive contents and structure |
|
||||
| 15 | PG-T7.2.5 | TODO | Depends on PG-T7.2.4 | DevOps Guild | Set retention policy for archives |
|
||||
| 16 | PG-T7.2.6 | TODO | Depends on PG-T7.2.5 | DevOps Guild | Schedule MongoDB cluster decommission |
|
||||
|
||||
### T7.3: PostgreSQL Performance Optimization
|
||||
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| 17 | PG-T7.3.1 | TODO | Depends on PG-T7.2.6 | DBA Guild | Enable `pg_stat_statements` extension |
|
||||
| 18 | PG-T7.3.2 | TODO | Depends on PG-T7.3.1 | DBA Guild | Identify slow queries |
|
||||
| 19 | PG-T7.3.3 | TODO | Depends on PG-T7.3.2 | DBA Guild | Analyze query plans with EXPLAIN ANALYZE |
|
||||
| 20 | PG-T7.3.4 | TODO | Depends on PG-T7.3.3 | DBA Guild | Add missing indexes |
|
||||
| 21 | PG-T7.3.5 | TODO | Depends on PG-T7.3.4 | DBA Guild | Remove unused indexes |
|
||||
| 22 | PG-T7.3.6 | TODO | Depends on PG-T7.3.5 | DBA Guild | Tune PostgreSQL configuration |
|
||||
| 23 | PG-T7.3.7 | TODO | Depends on PG-T7.3.6 | Observability Guild | Set up query monitoring dashboard |
|
||||
| 24 | PG-T7.3.8 | TODO | Depends on PG-T7.3.7 | DBA Guild | Document performance baselines |
|
||||
|
||||
### T7.4: Update Documentation
|
||||
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| 25 | PG-T7.4.1 | TODO | Depends on PG-T7.3.8 | Docs Guild | Update `docs/07_HIGH_LEVEL_ARCHITECTURE.md` |
|
||||
| 26 | PG-T7.4.2 | TODO | Depends on PG-T7.4.1 | Docs Guild | Update module architecture docs |
|
||||
| 27 | PG-T7.4.3 | TODO | Depends on PG-T7.4.2 | Docs Guild | Update deployment guides |
|
||||
| 28 | PG-T7.4.4 | TODO | Depends on PG-T7.4.3 | Docs Guild | Update operations runbooks |
|
||||
| 29 | PG-T7.4.5 | TODO | Depends on PG-T7.4.4 | Docs Guild | Update troubleshooting guides |
|
||||
| 30 | PG-T7.4.6 | TODO | Depends on PG-T7.4.5 | Docs Guild | Update `CLAUDE.md` technology stack |
|
||||
| 31 | PG-T7.4.7 | TODO | Depends on PG-T7.4.6 | Docs Guild | Create `docs/operations/postgresql-guide.md` |
|
||||
| 32 | PG-T7.4.8 | TODO | Depends on PG-T7.4.7 | Docs Guild | Document backup/restore procedures |
|
||||
| 33 | PG-T7.4.9 | TODO | Depends on PG-T7.4.8 | Docs Guild | Document scaling recommendations |
|
||||
|
||||
### T7.5: Update Air-Gap Kit
|
||||
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| 34 | PG-T7.5.1 | TODO | Depends on PG-T7.4.9 | DevOps Guild | Add PostgreSQL container image to kit |
|
||||
| 35 | PG-T7.5.2 | TODO | Depends on PG-T7.5.1 | DevOps Guild | Update kit scripts for PostgreSQL setup |
|
||||
| 36 | PG-T7.5.3 | TODO | Depends on PG-T7.5.2 | DevOps Guild | Include schema migrations in kit |
|
||||
| 37 | PG-T7.5.4 | TODO | Depends on PG-T7.5.3 | DevOps Guild | Update kit documentation |
|
||||
| 38 | PG-T7.5.5 | TODO | Depends on PG-T7.5.4 | DevOps Guild | Test kit installation in air-gapped environment |
|
||||
| 39 | PG-T7.5.6 | TODO | Depends on PG-T7.5.5 | Docs Guild | Update `docs/24_OFFLINE_KIT.md` |
|
||||
|
||||
### T7.6: Final Verification
|
||||
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| 40 | PG-T7.6.1 | TODO | Depends on PG-T7.5.6 | QA Guild | Run full integration test suite |
|
||||
| 41 | PG-T7.6.2 | TODO | Depends on PG-T7.6.1 | QA Guild | Run performance benchmark suite |
|
||||
| 42 | PG-T7.6.3 | TODO | Depends on PG-T7.6.2 | QA Guild | Verify all modules on PostgreSQL |
|
||||
| 43 | PG-T7.6.4 | TODO | Depends on PG-T7.6.3 | QA Guild | **Verify determinism tests pass** |
|
||||
| 44 | PG-T7.6.5 | TODO | Depends on PG-T7.6.4 | QA Guild | Verify air-gap kit works |
|
||||
| 45 | PG-T7.6.6 | TODO | Depends on PG-T7.6.5 | QA Guild | Generate final verification report |
|
||||
| 46 | PG-T7.6.7 | TODO | Depends on PG-T7.6.6 | Management | Get sign-off from stakeholders |
|
||||
|
||||
### T7.7: Decommission MongoDB
|
||||
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| 47 | PG-T7.7.1 | TODO | Depends on PG-T7.6.7 | DevOps Guild | Verify no services using MongoDB |
|
||||
| 48 | PG-T7.7.2 | TODO | Depends on PG-T7.7.1 | DevOps Guild | Stop MongoDB instances |
|
||||
| 49 | PG-T7.7.3 | TODO | Depends on PG-T7.7.2 | DevOps Guild | Archive final state |
|
||||
| 50 | PG-T7.7.4 | TODO | Depends on PG-T7.7.3 | DevOps Guild | Remove MongoDB from infrastructure |
|
||||
| 51 | PG-T7.7.5 | TODO | Depends on PG-T7.7.4 | Observability Guild | Update monitoring/alerting |
|
||||
| 52 | PG-T7.7.6 | TODO | Depends on PG-T7.7.5 | Finance | Update cost projections |
|
||||
|
||||
## Execution Log
|
||||
| Date (UTC) | Update | Owner |
|
||||
| --- | --- | --- |
|
||||
| 2025-11-28 | Sprint file created | Planning |
|
||||
|
||||
## Decisions & Risks
|
||||
- MongoDB archives are read-only backup; rollback to MongoDB after this phase is complex.
|
||||
- Any new data created after cutover is PostgreSQL-only.
|
||||
- Full rollback would require data export/import.
|
||||
- PostgreSQL configuration tuning recommendations in PHASE_7_CLEANUP.md.
|
||||
|
||||
## Success Metrics
|
||||
| Metric | Target | Measurement |
|
||||
| --- | --- | --- |
|
||||
| Query latency (p95) | < 100ms | pg_stat_statements |
|
||||
| Error rate | < 0.01% | Application logs |
|
||||
| Storage efficiency | < 120% of MongoDB | Disk usage |
|
||||
| Test coverage | 100% | CI reports |
|
||||
| Documentation coverage | 100% | Manual review |
|
||||
|
||||
## Exit Criteria
|
||||
- [ ] All MongoDB code removed from converted modules
|
||||
- [ ] MongoDB data archived
|
||||
- [ ] PostgreSQL performance optimized
|
||||
- [ ] All documentation updated
|
||||
- [ ] Air-gap kit updated and tested
|
||||
- [ ] Final verification report approved
|
||||
- [ ] MongoDB infrastructure decommissioned
|
||||
|
||||
## Post-Conversion Monitoring
|
||||
### First Week
|
||||
- Monitor error rates closely
|
||||
- Track query performance
|
||||
- Watch for any data inconsistencies
|
||||
- Have rollback plan ready (restore MongoDB)
|
||||
|
||||
### First Month
|
||||
- Review query statistics weekly
|
||||
- Optimize any slow queries found
|
||||
- Monitor storage growth
|
||||
- Adjust vacuum settings if needed
|
||||
|
||||
### Ongoing
|
||||
- Regular performance reviews
|
||||
- Index maintenance
|
||||
- Backup verification
|
||||
- Capacity planning
|
||||
|
||||
---
|
||||
*Reference: docs/db/tasks/PHASE_7_CLEANUP.md*
|
||||
@@ -75,3 +75,122 @@ CLI mirrors these endpoints (`stella findings list|view|update|export`). Console
|
||||
- `reports/` (generated PDFs/CSVs).
|
||||
- `signatures/` (DSSE envelopes).
|
||||
- Bundles produced deterministically; Export Center consumes them for mirror profiles.
|
||||
|
||||
## 8) VEX-First Triage UX
|
||||
|
||||
> Reference: Product advisory `28-Nov-2025 - Vulnerability Triage UX & VEX-First Decisioning.md`
|
||||
|
||||
### 8.1 Evidence-First Finding Cards
|
||||
|
||||
Each vulnerability finding is displayed as an evidence-first card showing:
|
||||
- CVE/vulnerability identifier with severity badge
|
||||
- Package name, version, ecosystem
|
||||
- Location (file path, container layer, function, call path)
|
||||
- Scanner and database date
|
||||
- Status badges: `New`, `VEX: Not affected`, `Policy: blocked`
|
||||
|
||||
Primary actions per card:
|
||||
- **VEX: Set status** - Opens VEX decision modal
|
||||
- **Fix PR / View Fix** - When available from connected scanners (Snyk/GitLab)
|
||||
- **Attach Evidence** - Link PRs, tickets, docs, commits
|
||||
- **Copy audit reference** - findingId + attestation digest
|
||||
|
||||
### 8.2 VEX Decision Model
|
||||
|
||||
VEX decisions follow the `VexDecision` schema (`docs/schemas/vex-decision.schema.json`):
|
||||
|
||||
**Status values:**
|
||||
- `NOT_AFFECTED` - Vulnerability does not apply to this artifact
|
||||
- `AFFECTED_MITIGATED` - Vulnerable but mitigations in place
|
||||
- `AFFECTED_UNMITIGATED` - Vulnerable without mitigations
|
||||
- `FIXED` - Vulnerability has been remediated
|
||||
|
||||
**Justification types (CSAF/VEX aligned):**
|
||||
- `CODE_NOT_PRESENT`
|
||||
- `CODE_NOT_REACHABLE`
|
||||
- `VULNERABLE_CODE_NOT_IN_EXECUTE_PATH`
|
||||
- `CONFIGURATION_NOT_AFFECTED`
|
||||
- `OS_NOT_AFFECTED`
|
||||
- `RUNTIME_MITIGATION_PRESENT`
|
||||
- `COMPENSATING_CONTROLS`
|
||||
- `ACCEPTED_BUSINESS_RISK`
|
||||
- `OTHER`
|
||||
|
||||
**Scope and validity:**
|
||||
- Decisions can be scoped to specific environments and projects
|
||||
- Validity windows with `notBefore` and `notAfter` timestamps
|
||||
- Expired decisions are surfaced with warnings
|
||||
|
||||
### 8.3 Explainability Panel
|
||||
|
||||
Right-side panel with tabs for each finding:
|
||||
|
||||
**Overview tab:**
|
||||
- Title, severity, package/version
|
||||
- Scanner + DB date
|
||||
- Finding history timeline
|
||||
- Current VEX decision summary
|
||||
|
||||
**Reachability tab:**
|
||||
- Call path visualization
|
||||
- Module dependency list
|
||||
- Runtime usage indicators (when available)
|
||||
|
||||
**Policy tab:**
|
||||
- Policy evaluation result (PASS/WARN/FAIL)
|
||||
- Gate details with "this gate failed because..." explanations
|
||||
- Links to gate definitions
|
||||
|
||||
**Attestations tab:**
|
||||
- Attestations mentioning this artifact/vulnerability/scan
|
||||
- Type, subject, predicate, signer, verified status
|
||||
- "Signed evidence" pill linking to attestation detail
|
||||
|
||||
### 8.4 VEX Decision APIs
|
||||
|
||||
New endpoints for VEX decisions:
|
||||
|
||||
- `POST /v1/vex-decisions` - Create new VEX decision with optional attestation
|
||||
- `PATCH /v1/vex-decisions/{id}` - Update existing decision (creates superseding record)
|
||||
- `GET /v1/vex-decisions` - List decisions with filters
|
||||
- `GET /v1/vex-decisions/{id}` - Get decision detail
|
||||
|
||||
Request/response follows `VexDecisionDto` per schema.
|
||||
|
||||
### 8.5 Audit Bundle Export
|
||||
|
||||
Immutable audit bundles follow the `AuditBundleIndex` schema (`docs/schemas/audit-bundle-index.schema.json`):
|
||||
|
||||
**Bundle contents:**
|
||||
- Vulnerability reports (scanner outputs)
|
||||
- SBOM (CycloneDX/SPDX)
|
||||
- VEX decisions
|
||||
- Policy evaluations
|
||||
- Raw attestations (DSSE envelopes)
|
||||
- `audit-bundle-index.json` manifest with integrity hashes
|
||||
|
||||
**APIs:**
|
||||
- `POST /v1/audit-bundles` - Create new bundle (async generation)
|
||||
- `GET /v1/audit-bundles/{bundleId}` - Download bundle (ZIP or OCI)
|
||||
- `GET /v1/audit-bundles` - List previously created bundles
|
||||
|
||||
### 8.6 Industry Pattern Alignment
|
||||
|
||||
The triage UX aligns with industry patterns from:
|
||||
|
||||
| Tool | Pattern Adopted |
|
||||
|------|-----------------|
|
||||
| **Snyk** | PR checks, Fix PRs, ignore with reasons |
|
||||
| **GitLab SCA** | Vulnerability Report, status workflow, activity log |
|
||||
| **Harbor/Trivy** | Artifact-centric navigation, attestation accessories |
|
||||
| **Anchore** | Policy gates with trigger explanations, allowlists |
|
||||
|
||||
## 9) Schemas
|
||||
|
||||
The following JSON schemas define the data contracts for VEX and audit functionality:
|
||||
|
||||
- `docs/schemas/vex-decision.schema.json` - VEX decision form and persistence
|
||||
- `docs/schemas/attestation-vuln-scan.schema.json` - Vulnerability scan attestation predicate
|
||||
- `docs/schemas/audit-bundle-index.schema.json` - Audit bundle manifest
|
||||
|
||||
These schemas are referenced by both backend DTOs and frontend TypeScript interfaces.
|
||||
|
||||
@@ -0,0 +1,523 @@
|
||||
# Vulnerability Triage UX & VEX-First Decisioning
|
||||
|
||||
**Version:** 1.0
|
||||
**Date:** 2025-11-28
|
||||
**Status:** Canonical
|
||||
|
||||
This advisory defines the **end-to-end UX and data contracts** for vulnerability triage, VEX decisioning, evidence/explainability views, and audit export in Stella Ops. It synthesizes patterns from Snyk, GitLab SCA, Harbor/Trivy, and Anchore Enterprise into a converged UX layer.
|
||||
|
||||
---
|
||||
|
||||
## 1. Scope
|
||||
|
||||
This spec covers:
|
||||
|
||||
1. **Vulnerability triage** (first touch)
|
||||
2. **Suppression / "Not Affected"** (VEX-aligned)
|
||||
3. **Evidence & explainability views**
|
||||
4. **Audit export** (immutable bundles)
|
||||
5. **Attestations** as the backbone of evidence and gating
|
||||
|
||||
Stella Ops is the **converged UX layer** over scanner backends (Snyk, Trivy, GitLab, Anchore, or others).
|
||||
|
||||
---
|
||||
|
||||
## 2. Industry Pattern Analysis
|
||||
|
||||
### 2.1 Triage (First Touch)
|
||||
|
||||
| Tool | Pattern | Stella Ops Mirror |
|
||||
|------|---------|-------------------|
|
||||
| **Snyk** | PR checks show before/after diffs; Fix PRs directly from Issues list | Evidence-first cards with "Fix PR" CTA |
|
||||
| **GitLab SCA** | Vulnerability Report with `Needs triage` default state | Status workflow starting at `DETECTED` |
|
||||
| **Harbor/Trivy** | Project -> Artifacts -> Vulnerabilities panel with Rescan CTA | Artifact-centric navigation with scan badges |
|
||||
| **Anchore** | Images -> Vulnerabilities aligned to Policies (pass/fail) | Policy gate indicators on all finding views |
|
||||
|
||||
**UI pattern to reuse:** An **evidence-first card** per finding (CVE, package, version, path) with primary actions (Fix PR, Dismiss/Not Affected, View Evidence).
|
||||
|
||||
### 2.2 Suppression / "Not Affected" (VEX-Aligned)
|
||||
|
||||
| Tool | Pattern | Stella Ops Mirror |
|
||||
|------|---------|-------------------|
|
||||
| **Snyk** | "Ignore" with reason + expiry; org-restricted; PR checks skip ignored | VEX `statusJustification` with validity window |
|
||||
| **GitLab** | `Dismissed` status with required comment; activity log | VEX decisions with actor/timestamp/audit trail |
|
||||
| **Anchore** | Allowlists + Policy Gates + VEX annotations | Allowlist integration + VEX buttons |
|
||||
| **Harbor/Trivy** | No native VEX; store as in-toto attestation | Attestation-backed VEX decisions |
|
||||
|
||||
**UI pattern to reuse:** An **Actionable VEX** button (`Not Affected`, `Affected - mitigated`, `Fixed`) that opens a compact form: justification, evidence links, scope, expiry -> generates/updates a signed VEX note.
|
||||
|
||||
### 2.3 Evidence View (Explainability)
|
||||
|
||||
| Tool | Pattern | Stella Ops Mirror |
|
||||
|------|---------|-------------------|
|
||||
| **Snyk** | PR context + Fix PR evidence + ignore policy display | Explainability panel with PR/commit links |
|
||||
| **GitLab** | Vulnerability Report hub with lifecycle activity | Decision history timeline |
|
||||
| **Anchore** | Policy Gates breakdown showing which trigger caused fail/pass | Gate evaluation with trigger explanations |
|
||||
| **Harbor/Trivy** | Scanner DB date, version, attestation links | Scanner metadata + attestation digest |
|
||||
|
||||
**UI pattern to reuse:** An **Explainability panel** on the right: "Why this is flagged / Why it passed" with timestamps, rule IDs, feed freshness, and the **Attestation digest**.
|
||||
|
||||
### 2.4 Audit Export (Immutable)
|
||||
|
||||
| Tool | Export Contents |
|
||||
|------|-----------------|
|
||||
| **Snyk** | PR check results + Ignore ledger + Fix PRs |
|
||||
| **GitLab** | Vulnerability Report with status history |
|
||||
| **Anchore** | Policy Bundle eval JSON as primary audit unit |
|
||||
| **Harbor/Trivy** | Trivy report + signed attestation |
|
||||
|
||||
**UI pattern to reuse:** **"Create immutable audit bundle"** CTA that writes a ZIP/OCI artifact containing reports, VEX, policy evals, and attestations, plus a top-level manifest with hashes.
|
||||
|
||||
---
|
||||
|
||||
## 3. Core Data Model
|
||||
|
||||
### 3.1 Artifact
|
||||
|
||||
```text
|
||||
Artifact
|
||||
- id (string, stable)
|
||||
- type (IMAGE | REPO | SBOM | FUNCTION | HOST)
|
||||
- displayName
|
||||
- coordinates (registry/repo URL, tag, branch, env, etc.)
|
||||
- digests[] (e.g. sha256 for OCI images, commit SHA for repos)
|
||||
- latestScanAttestations[] (AttestationRef)
|
||||
- riskSummary (openCount, totalCount, maxSeverity, lastScanAt)
|
||||
```
|
||||
|
||||
### 3.2 VulnerabilityFinding
|
||||
|
||||
```text
|
||||
VulnerabilityFinding
|
||||
- id (string, internal stable ID)
|
||||
- sourceFindingId (string, from Snyk/Trivy/etc.)
|
||||
- scanner (name, version)
|
||||
- artifactId
|
||||
- vulnerabilityId (CVE, GHSA, etc.)
|
||||
- title
|
||||
- severity (CRITICAL | HIGH | MEDIUM | LOW | INFO)
|
||||
- package (name, version, ecosystem)
|
||||
- location (filePath, containerLayer, function, callPath[])
|
||||
- introducedBy (commitId?, imageDigest?, buildId?)
|
||||
- firstSeenAt
|
||||
- lastSeenAt
|
||||
- status (DETECTED | RESOLVED | NO_LONGER_DETECTED)
|
||||
- currentVexDecisionId? (if a VEX decision is attached)
|
||||
- evidenceAttestationRefs[] (AttestationRef[])
|
||||
```
|
||||
|
||||
### 3.3 VEXDecision
|
||||
|
||||
Represents a **VEX-style statement** attached to a finding + subject.
|
||||
|
||||
```text
|
||||
VEXDecision
|
||||
- id
|
||||
- vulnerabilityId (CVE, etc.)
|
||||
- subject (ArtifactRef / SBOM node ref)
|
||||
- status (NOT_AFFECTED | AFFECTED_MITIGATED | AFFECTED_UNMITIGATED | FIXED)
|
||||
- justificationType (enum; see section 7.3)
|
||||
- justificationText (free text)
|
||||
- evidenceRefs[] (links to PRs, commits, tickets, docs, etc.)
|
||||
- scope (envs/projects where this decision applies)
|
||||
- validFor (notBefore, notAfter?)
|
||||
- attestationRef? (AttestationRef)
|
||||
- supersedesDecisionId?
|
||||
- createdBy (id, displayName)
|
||||
- createdAt
|
||||
- updatedAt
|
||||
```
|
||||
|
||||
### 3.4 Attestation / AttestationRef
|
||||
|
||||
```text
|
||||
AttestationRef
|
||||
- id
|
||||
- type (VULN_SCAN | SBOM | VEX | POLICY_EVAL | OTHER)
|
||||
- statementId (if DSSE/Intoto)
|
||||
- subjectName
|
||||
- subjectDigest (e.g. sha256)
|
||||
- predicateType (URI)
|
||||
- createdAt
|
||||
- signer (name, keyId)
|
||||
- storage (ociRef | bundlePath | url)
|
||||
```
|
||||
|
||||
### 3.5 PolicyEvaluation
|
||||
|
||||
```text
|
||||
PolicyEvaluation
|
||||
- id
|
||||
- subject (ArtifactRef)
|
||||
- policyBundleVersion
|
||||
- overallResult (PASS | WARN | FAIL)
|
||||
- gates[] (GateResult)
|
||||
- attestationRef? (AttestationRef)
|
||||
- evaluatedAt
|
||||
```
|
||||
|
||||
### 3.6 AuditBundle
|
||||
|
||||
Represents a **downloadable immutable bundle** (ZIP or OCI artifact).
|
||||
|
||||
```text
|
||||
AuditBundle
|
||||
- bundleId
|
||||
- version
|
||||
- createdAt
|
||||
- createdBy
|
||||
- subject (ArtifactRef)
|
||||
- index (AuditBundleIndex) <- JSON index inside the bundle
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Primary UX Surfaces
|
||||
|
||||
### 4.1 Artifacts List
|
||||
|
||||
**Goal:** High-level "what's risky?" view and entry point into triage.
|
||||
|
||||
**Columns:**
|
||||
- Artifact
|
||||
- Type
|
||||
- Environment(s)
|
||||
- Open / Total vulns
|
||||
- Max severity
|
||||
- **Attestations** (badge w/ count)
|
||||
- Last scan (timestamp + scanner)
|
||||
|
||||
**Actions:**
|
||||
- View vulnerabilities (primary)
|
||||
- View attestations
|
||||
- Create audit bundle
|
||||
|
||||
### 4.2 Vulnerability Workspace (per Artifact)
|
||||
|
||||
**Split layout:**
|
||||
|
||||
**Left: Vulnerability list**
|
||||
- Filters: severity, status, VEX status, scanner, package, introducedBy, env
|
||||
- Sort: severity, recency, package, path
|
||||
- Badges for:
|
||||
- `New` (first seen in last N scans)
|
||||
- `VEX: Not affected`
|
||||
- `Policy: blocked` / `Policy: allowed`
|
||||
|
||||
**Right: Evidence / Explainability panel**
|
||||
|
||||
Tabs:
|
||||
1. **Overview**
|
||||
- Title, severity, package, version, path
|
||||
- Scanner + db date
|
||||
- Finding history timeline
|
||||
- Current VEX decision summary (if any)
|
||||
2. **Reachability**
|
||||
- Call path, modules, runtime usage info (when available)
|
||||
3. **Policy**
|
||||
- Policy evaluation: which gate caused pass/fail
|
||||
- Links to gate definitions
|
||||
4. **Attestations**
|
||||
- All attestations that mention:
|
||||
- this artifact
|
||||
- this vulnerabilityId
|
||||
- this scan result
|
||||
|
||||
**Primary actions per finding:**
|
||||
- **VEX: Set status** -> opens VEX Modal (see 4.3)
|
||||
- **Open Fix PR / View Fix** (if available from Snyk/GitLab)
|
||||
- **Attach Evidence** (link tickets / docs)
|
||||
- **Copy audit reference** (findingId + attestation digest)
|
||||
|
||||
### 4.3 VEX Modal - "Affect & Justification"
|
||||
|
||||
**Entry points:**
|
||||
- From a finding row ("VEX" button)
|
||||
- From a policy failure explanation
|
||||
- From a bulk action on multiple findings
|
||||
|
||||
**Fields (backed by `VEXDecision`):**
|
||||
- Status (radio buttons):
|
||||
- `Not affected`
|
||||
- `Affected - mitigated`
|
||||
- `Affected - not mitigated`
|
||||
- `Fixed`
|
||||
- Justification type (select - see section 7.3)
|
||||
- Justification text (multi-line)
|
||||
- Scope:
|
||||
- Environments (multi-select)
|
||||
- Projects / services (multi-select)
|
||||
- Validity:
|
||||
- Start (defaults now)
|
||||
- Optional expiry (recommended)
|
||||
- Evidence:
|
||||
- Add links (PR, ticket, doc, commit)
|
||||
- Attach attestation (optional; pick from list)
|
||||
- Review:
|
||||
- Summary of what will be written to the VEX statement
|
||||
- "Will generate signed attestation" note (if enabled)
|
||||
|
||||
**Actions:**
|
||||
- Save (creates or updates VEXDecision, writes VEX attestation)
|
||||
- Cancel
|
||||
- View raw JSON (for power users)
|
||||
|
||||
### 4.4 Attestations View
|
||||
|
||||
Per artifact, tab: **Attestations**
|
||||
|
||||
Table of attestations:
|
||||
- Type (vuln scan, SBOM, VEX, policy)
|
||||
- Subject name (shortened)
|
||||
- Predicate type (URI)
|
||||
- Scanner / policy engine (derived from predicate)
|
||||
- Signer (keyId, trusted/not-trusted badge)
|
||||
- Created at
|
||||
- Verified (yes/no)
|
||||
|
||||
Click to open:
|
||||
- Header: statement id, subject, signer
|
||||
- Predicate preview:
|
||||
- For vuln scan: counts, scanner version, db date
|
||||
- For SBOM: bomRef, component counts
|
||||
- For VEX: decision status, vulnerabilityId, scope
|
||||
|
||||
### 4.5 Policy & Gating View
|
||||
|
||||
Per environment / pipeline:
|
||||
- Matrix of **gates** vs **subject types**:
|
||||
- e.g. `CI Build`, `Registry Admission`, `Runtime Admission`
|
||||
- Each gate shows:
|
||||
- Rule description (severity thresholds, allowlist usage, required attestations)
|
||||
- Last evaluation stats (pass/fail counts)
|
||||
- Clicking a gate shows:
|
||||
- Recent evaluations (with link to artifact & policy attestation)
|
||||
- Which condition failed
|
||||
|
||||
### 4.6 Audit Export - Bundle Creation
|
||||
|
||||
**From:**
|
||||
- Artifact page (button: "Create immutable audit bundle")
|
||||
- Pipeline run detail
|
||||
- Policy evaluation detail
|
||||
|
||||
**Workflow:**
|
||||
1. User selects:
|
||||
- Subject artifact + digest
|
||||
- Time window (e.g. "last 7 days of scans & decisions")
|
||||
- Included content (checklist):
|
||||
- Vuln reports
|
||||
- SBOM
|
||||
- VEX decisions
|
||||
- Policy evaluations
|
||||
- Raw attestations
|
||||
2. Backend generates:
|
||||
- ZIP or OCI artifact
|
||||
- `audit-bundle-index.json` at root
|
||||
3. UI shows:
|
||||
- Bundle ID & hash
|
||||
- Download button
|
||||
- OCI reference (if pushed to registry)
|
||||
|
||||
---
|
||||
|
||||
## 5. State Model
|
||||
|
||||
### 5.1 Finding Status vs VEX Status
|
||||
|
||||
Two separate but related states:
|
||||
|
||||
**Finding.status:**
|
||||
- `DETECTED` - currently reported by at least one scanner
|
||||
- `NO_LONGER_DETECTED` - was present, not in latest scan for this subject
|
||||
- `RESOLVED` - confirmed removed (e.g. package upgraded, image replaced)
|
||||
|
||||
**VEXDecision.status:**
|
||||
- `NOT_AFFECTED`
|
||||
- `AFFECTED_MITIGATED`
|
||||
- `AFFECTED_UNMITIGATED`
|
||||
- `FIXED`
|
||||
|
||||
**UI rules:**
|
||||
- If `Finding.status = NO_LONGER_DETECTED` and a VEXDecision still exists:
|
||||
- Show badge: "Historical VEX decision (finding no longer detected)"
|
||||
- If `VEXDecision.status = NOT_AFFECTED`:
|
||||
- Policy engines may treat this as **non-blocking** (configurable)
|
||||
|
||||
---
|
||||
|
||||
## 6. Interaction Patterns to Mirror
|
||||
|
||||
### 6.1 From Snyk
|
||||
|
||||
- PR checks show **before/after** and don't fail on ignored issues
|
||||
- Action: "Fix PR" from a finding
|
||||
- Mapping:
|
||||
- Stella Ops should show "Fix PR" and "Compare before/after" where data exists
|
||||
- VEX `NOT_AFFECTED` should make **future checks ignore** that finding for that subject/scope
|
||||
|
||||
### 6.2 From GitLab SCA
|
||||
|
||||
- `Dismissed` with reasons and activity log
|
||||
- Mapping:
|
||||
- VEX decisions must have reason + actor + timestamp
|
||||
- The activity log should show a full **decision history**
|
||||
|
||||
### 6.3 From Anchore
|
||||
|
||||
- Policy gates & allowlists
|
||||
- Mapping:
|
||||
- Gate evaluation screen with clear "this gate failed because..." explanation
|
||||
|
||||
---
|
||||
|
||||
## 7. Enumerations & Conventions
|
||||
|
||||
### 7.1 VEX Status
|
||||
|
||||
```text
|
||||
NOT_AFFECTED
|
||||
AFFECTED_MITIGATED
|
||||
AFFECTED_UNMITIGATED
|
||||
FIXED
|
||||
```
|
||||
|
||||
### 7.2 VEX Scope
|
||||
|
||||
- `envs[]`: e.g. `["prod", "staging"]`
|
||||
- `projects[]`: service / app names
|
||||
- Default: applies to **all** unless restricted
|
||||
|
||||
### 7.3 Justification Type (inspired by CSAF/VEX)
|
||||
|
||||
```text
|
||||
CODE_NOT_PRESENT
|
||||
CODE_NOT_REACHABLE
|
||||
VULNERABLE_CODE_NOT_IN_EXECUTE_PATH
|
||||
CONFIGURATION_NOT_AFFECTED
|
||||
OS_NOT_AFFECTED
|
||||
RUNTIME_MITIGATION_PRESENT
|
||||
COMPENSATING_CONTROLS
|
||||
ACCEPTED_BUSINESS_RISK
|
||||
OTHER
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8. Attestation Placement
|
||||
|
||||
### 8.1 Trivy + Cosign
|
||||
|
||||
Generate **vulnerability-scan attestation** and SBOM attestation; attach to image via OCI referrers. These attestations become the source of truth for evidence and audit export.
|
||||
|
||||
### 8.2 Harbor
|
||||
|
||||
Treat attestations as first-class accessories/refs to the image. Surface them next to the Vulnerabilities tab. Link them into the explainability panel.
|
||||
|
||||
### 8.3 Anchore
|
||||
|
||||
Reference attestation digests inside **Policy evaluation** output so pass/fail is traceable to signed inputs.
|
||||
|
||||
### 8.4 Snyk/GitLab
|
||||
|
||||
Surface attestation presence in PR/Security dashboards to prove findings came from a **signed** scan; link out to the OCI digest.
|
||||
|
||||
**UI pattern:** Small **"Signed evidence"** pill on each finding; clicking opens the attestation JSON (human-readable view) + verify command snippet.
|
||||
|
||||
---
|
||||
|
||||
## 9. Gating Controls
|
||||
|
||||
| Tool | Mechanism | Stella Ops Mirror |
|
||||
|------|-----------|-------------------|
|
||||
| **Anchore** | Policy Gates/Triggers model for hard gates | Gates per environment with trigger explainability |
|
||||
| **Snyk** | PR checks + Auto Fix PRs as soft gates | PR integration with soft/hard gate toggles |
|
||||
| **GitLab** | MR approvals + Security Policies; auto-resolve on no-longer-detected | Status-aware policies with auto-resolution |
|
||||
| **Harbor** | External policy engines (Kyverno/OPA) verify signatures/attestations | Admission controller integration |
|
||||
|
||||
---
|
||||
|
||||
## 10. Minimal UI Wireframe
|
||||
|
||||
### 10.1 Artifacts List
|
||||
|
||||
| Image | Tag | Risk (open/total) | Attestations | Last scan |
|
||||
|-------|-----|-------------------|--------------|-----------|
|
||||
| app/service | v1.2.3 | 3/47 | 4 | 2h ago (Trivy) |
|
||||
|
||||
### 10.2 Artifact -> Vulnerabilities Tab (Evidence-First)
|
||||
|
||||
```
|
||||
+----------------------------------+-----------------------------------+
|
||||
| Finding Cards (scrollable) | Explainability Panel |
|
||||
| | |
|
||||
| [CVE-2024-1234] CRITICAL | Overview | Reachability | Policy |
|
||||
| openssl 3.0.14 -> 3.0.15 | |
|
||||
| [Fix PR] [VEX: Not Affected] | Scanner: Trivy 0.53.0 |
|
||||
| [Attach Evidence] | DB: 2025-11-27 |
|
||||
| | Attestation: sha256:2e61... |
|
||||
| [CVE-2024-5678] HIGH | |
|
||||
| log4j 2.17.0 | [Why flagged] |
|
||||
| [VEX: Mitigated] | - version.match: 2.17.0 < 2.17.1 |
|
||||
| | - gate: severity >= HIGH |
|
||||
+----------------------------------+-----------------------------------+
|
||||
```
|
||||
|
||||
### 10.3 Policy View
|
||||
|
||||
Gate rules (like Anchore) with preview + dry-run; show which triggers cause failure.
|
||||
|
||||
### 10.4 Audit
|
||||
|
||||
**"Create immutable audit bundle"** -> produces ZIP/OCI artifact with reports, VEX JSON, policy evals, and in-toto/DSSE attestations.
|
||||
|
||||
### 10.5 Registry/Admission
|
||||
|
||||
"Ready to deploy" badge when all gates met and required attestations verified.
|
||||
|
||||
---
|
||||
|
||||
## 11. API Endpoints (High-Level)
|
||||
|
||||
```text
|
||||
GET /artifacts
|
||||
GET /artifacts/{id}/vulnerabilities
|
||||
GET /vulnerabilities/{id}
|
||||
POST /vex-decisions
|
||||
PATCH /vex-decisions/{id}
|
||||
GET /artifacts/{id}/attestations
|
||||
POST /audit-bundles
|
||||
GET /audit-bundles/{bundleId}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 12. JSON Schema Locations
|
||||
|
||||
The following schemas should be created/maintained:
|
||||
|
||||
- `docs/schemas/vex-decision.schema.json` - VEX decision form schema
|
||||
- `docs/schemas/attestation-vuln-scan.schema.json` - Vulnerability scan attestation
|
||||
- `docs/schemas/audit-bundle-index.schema.json` - Audit bundle manifest
|
||||
|
||||
---
|
||||
|
||||
## 13. Related Advisories
|
||||
|
||||
- `27-Nov-2025 - Explainability Layer for Vulnerability Verdicts.md` - Evidence chain model
|
||||
- `27-Nov-2025 - Making Graphs Understandable to Humans.md` - Graph navigation UX
|
||||
- `25-Nov-2025 - Define Safe VEX 'Not Affected' Claims with Proofs.md` - VEX proof requirements
|
||||
|
||||
---
|
||||
|
||||
## 14. Sprint Integration
|
||||
|
||||
This advisory maps to:
|
||||
|
||||
- **SPRINT_0215_0001_0001_vuln_triage_ux.md** (NEW) - UI triage workspace implementation
|
||||
- **SPRINT_210_ui_ii.md** - VEX tab tasks (UI-LNM-22-003)
|
||||
- **SPRINT_0334_docs_modules_vuln_explorer.md** - Module documentation updates
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2025-11-28*
|
||||
@@ -64,6 +64,22 @@ These are the authoritative advisories to reference for implementation:
|
||||
- **Sprint:** Multiple sprints (0186, 0401, 0512)
|
||||
- **Status:** High-level roadmap document
|
||||
|
||||
### Vulnerability Triage UX & VEX-First Decisioning
|
||||
- **Canonical:** `28-Nov-2025 - Vulnerability Triage UX & VEX-First Decisioning.md`
|
||||
- **Sprint:** SPRINT_0215_0001_0001_vuln_triage_ux.md (NEW)
|
||||
- **Related Sprints:**
|
||||
- SPRINT_210_ui_ii.md (UI-LNM-22-003 VEX tab)
|
||||
- SPRINT_0334_docs_modules_vuln_explorer.md (docs)
|
||||
- **Related Advisories:**
|
||||
- `27-Nov-2025 - Explainability Layer for Vulnerability Verdicts.md` (evidence chain)
|
||||
- `27-Nov-2025 - Making Graphs Understandable to Humans.md` (graph UX)
|
||||
- `25-Nov-2025 - Define Safe VEX 'Not Affected' Claims with Proofs.md` (VEX proofs)
|
||||
- **Status:** New - defines converged triage UX across Snyk/GitLab/Harbor/Anchore patterns
|
||||
- **Schemas:**
|
||||
- `docs/schemas/vex-decision.schema.json`
|
||||
- `docs/schemas/attestation-vuln-scan.schema.json`
|
||||
- `docs/schemas/audit-bundle-index.schema.json`
|
||||
|
||||
## Files to Archive
|
||||
|
||||
The following files should be moved to `archived/` as they are superseded:
|
||||
@@ -95,6 +111,7 @@ The following files should be moved to `archived/` as they are superseded:
|
||||
| Unknowns Registry | SPRINT_0140_0001_0001 | EXISTING (implemented) |
|
||||
| Graph Revision IDs | SPRINT_0401_0001_0001 | EXISTING |
|
||||
| DSSE/Rekor Batching | SPRINT_0401_0001_0001 | EXISTING |
|
||||
| Vuln Triage UX / VEX | SPRINT_0215_0001_0001 | NEW |
|
||||
|
||||
## Implementation Priority
|
||||
|
||||
@@ -103,8 +120,9 @@ Based on gap analysis:
|
||||
1. **P0 - CVSS v4.0** (Sprint 0190) - Industry moving to v4.0, genuine gap
|
||||
2. **P1 - SPDX 3.0.1** (Sprint 0186 tasks 15a-15f) - Standards compliance
|
||||
3. **P1 - Public Benchmark** (Sprint 0513) - Differentiation/marketing value
|
||||
4. **P2 - Explainability** (Sprint 0401) - UX enhancement, existing tasks
|
||||
5. **P3 - Already Implemented** - Unknowns, Graph IDs, DSSE batching
|
||||
4. **P1 - Vuln Triage UX** (Sprint 0215) - Industry-aligned UX for competitive parity
|
||||
5. **P2 - Explainability** (Sprint 0401) - UX enhancement, existing tasks
|
||||
6. **P3 - Already Implemented** - Unknowns, Graph IDs, DSSE batching
|
||||
|
||||
## Implementer Quick Reference
|
||||
|
||||
@@ -124,7 +142,10 @@ For each topic, the implementer should read:
|
||||
| Sbomer | `docs/modules/sbomer/architecture.md` | `src/Sbomer/*/AGENTS.md` |
|
||||
| Signals | `docs/modules/signals/architecture.md` | `src/Signals/*/AGENTS.md` |
|
||||
| Attestor | `docs/modules/attestor/architecture.md` | `src/Attestor/*/AGENTS.md` |
|
||||
| Vuln Explorer | `docs/modules/vuln-explorer/architecture.md` | `src/VulnExplorer/*/AGENTS.md` |
|
||||
| VEX-Lens | `docs/modules/vex-lens/architecture.md` | `src/Excititor/*/AGENTS.md` |
|
||||
| UI | `docs/modules/ui/architecture.md` | `src/UI/*/AGENTS.md` |
|
||||
|
||||
---
|
||||
*Index created: 2025-11-27*
|
||||
*Last updated: 2025-11-27*
|
||||
*Last updated: 2025-11-28*
|
||||
|
||||
226
docs/schemas/attestation-vuln-scan.schema.json
Normal file
226
docs/schemas/attestation-vuln-scan.schema.json
Normal file
@@ -0,0 +1,226 @@
|
||||
{
|
||||
"$id": "https://stella.ops/schema/attestation-vuln-scan.json",
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"title": "VulnScanAttestation",
|
||||
"description": "In-toto style attestation for vulnerability scan results",
|
||||
"type": "object",
|
||||
"required": ["_type", "predicateType", "subject", "predicate", "attestationMeta"],
|
||||
"properties": {
|
||||
"_type": {
|
||||
"type": "string",
|
||||
"const": "https://in-toto.io/Statement/v0.1",
|
||||
"description": "In-toto statement type URI"
|
||||
},
|
||||
"predicateType": {
|
||||
"type": "string",
|
||||
"const": "https://stella.ops/predicates/vuln-scan/v1",
|
||||
"description": "Predicate type URI for Stella Ops vulnerability scans"
|
||||
},
|
||||
"subject": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/$defs/AttestationSubject"
|
||||
},
|
||||
"minItems": 1,
|
||||
"description": "Artifacts that were scanned"
|
||||
},
|
||||
"predicate": {
|
||||
"$ref": "#/$defs/VulnScanPredicate",
|
||||
"description": "Vulnerability scan result predicate"
|
||||
},
|
||||
"attestationMeta": {
|
||||
"$ref": "#/$defs/AttestationMeta",
|
||||
"description": "Attestation metadata including signer info"
|
||||
}
|
||||
},
|
||||
"$defs": {
|
||||
"AttestationSubject": {
|
||||
"type": "object",
|
||||
"required": ["name", "digest"],
|
||||
"properties": {
|
||||
"name": {
|
||||
"type": "string",
|
||||
"description": "Subject name (e.g. image reference)",
|
||||
"examples": ["registry.internal/stella/app-service@sha256:7d9c..."]
|
||||
},
|
||||
"digest": {
|
||||
"type": "object",
|
||||
"additionalProperties": {
|
||||
"type": "string"
|
||||
},
|
||||
"description": "Algorithm -> digest map",
|
||||
"examples": [{"sha256": "7d9cd5f1a2a0dd9a41a2c43a5b7d8a0bcd9e34cf39b3f43a70595c834f0a4aee"}]
|
||||
}
|
||||
}
|
||||
},
|
||||
"VulnScanPredicate": {
|
||||
"type": "object",
|
||||
"required": ["scanner", "scanStartedAt", "scanCompletedAt", "severityCounts", "findingReport"],
|
||||
"properties": {
|
||||
"scanner": {
|
||||
"$ref": "#/$defs/ScannerInfo",
|
||||
"description": "Scanner that produced this result"
|
||||
},
|
||||
"scannerDb": {
|
||||
"$ref": "#/$defs/ScannerDbInfo",
|
||||
"description": "Vulnerability database info"
|
||||
},
|
||||
"scanStartedAt": {
|
||||
"type": "string",
|
||||
"format": "date-time",
|
||||
"description": "ISO-8601 timestamp when scan started"
|
||||
},
|
||||
"scanCompletedAt": {
|
||||
"type": "string",
|
||||
"format": "date-time",
|
||||
"description": "ISO-8601 timestamp when scan completed"
|
||||
},
|
||||
"severityCounts": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"CRITICAL": { "type": "integer", "minimum": 0 },
|
||||
"HIGH": { "type": "integer", "minimum": 0 },
|
||||
"MEDIUM": { "type": "integer", "minimum": 0 },
|
||||
"LOW": { "type": "integer", "minimum": 0 }
|
||||
},
|
||||
"description": "Count of findings by severity"
|
||||
},
|
||||
"findingReport": {
|
||||
"$ref": "#/$defs/FindingReport",
|
||||
"description": "Reference to the full findings report"
|
||||
}
|
||||
}
|
||||
},
|
||||
"ScannerInfo": {
|
||||
"type": "object",
|
||||
"required": ["name", "version"],
|
||||
"properties": {
|
||||
"name": {
|
||||
"type": "string",
|
||||
"description": "Scanner name",
|
||||
"examples": ["Trivy", "Snyk", "Grype"]
|
||||
},
|
||||
"version": {
|
||||
"type": "string",
|
||||
"description": "Scanner version",
|
||||
"examples": ["0.53.0"]
|
||||
}
|
||||
}
|
||||
},
|
||||
"ScannerDbInfo": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"lastUpdatedAt": {
|
||||
"type": "string",
|
||||
"format": "date-time",
|
||||
"description": "ISO-8601 timestamp when vulnerability DB was last updated"
|
||||
}
|
||||
}
|
||||
},
|
||||
"FindingReport": {
|
||||
"type": "object",
|
||||
"required": ["mediaType", "location", "digest"],
|
||||
"properties": {
|
||||
"mediaType": {
|
||||
"type": "string",
|
||||
"default": "application/json",
|
||||
"description": "Media type of the report",
|
||||
"examples": ["application/json", "application/vnd.cyclonedx+json"]
|
||||
},
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "Path or URI to the report file",
|
||||
"examples": ["reports/trivy/app-service-7d9c-vulns.json"]
|
||||
},
|
||||
"digest": {
|
||||
"type": "object",
|
||||
"additionalProperties": {
|
||||
"type": "string"
|
||||
},
|
||||
"description": "Content digest of the report"
|
||||
}
|
||||
}
|
||||
},
|
||||
"AttestationMeta": {
|
||||
"type": "object",
|
||||
"required": ["statementId", "createdAt", "signer"],
|
||||
"properties": {
|
||||
"statementId": {
|
||||
"type": "string",
|
||||
"description": "Unique identifier for this attestation statement"
|
||||
},
|
||||
"createdAt": {
|
||||
"type": "string",
|
||||
"format": "date-time",
|
||||
"description": "ISO-8601 timestamp when attestation was created"
|
||||
},
|
||||
"signer": {
|
||||
"$ref": "#/$defs/AttestationSigner",
|
||||
"description": "Entity that signed this attestation"
|
||||
}
|
||||
}
|
||||
},
|
||||
"AttestationSigner": {
|
||||
"type": "object",
|
||||
"required": ["name", "keyId"],
|
||||
"properties": {
|
||||
"name": {
|
||||
"type": "string",
|
||||
"description": "Signer name/identity",
|
||||
"examples": ["ci/trivy-signer"]
|
||||
},
|
||||
"keyId": {
|
||||
"type": "string",
|
||||
"description": "Key identifier (fingerprint)",
|
||||
"examples": ["SHA256:ae12c8d1..."]
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"examples": [
|
||||
{
|
||||
"_type": "https://in-toto.io/Statement/v0.1",
|
||||
"predicateType": "https://stella.ops/predicates/vuln-scan/v1",
|
||||
"subject": [
|
||||
{
|
||||
"name": "registry.internal/stella/app-service@sha256:7d9c...",
|
||||
"digest": {
|
||||
"sha256": "7d9cd5f1a2a0dd9a41a2c43a5b7d8a0bcd9e34cf39b3f43a70595c834f0a4aee"
|
||||
}
|
||||
}
|
||||
],
|
||||
"predicate": {
|
||||
"scanner": {
|
||||
"name": "Trivy",
|
||||
"version": "0.53.0"
|
||||
},
|
||||
"scannerDb": {
|
||||
"lastUpdatedAt": "2025-11-20T09:32:00Z"
|
||||
},
|
||||
"scanStartedAt": "2025-11-21T09:00:00Z",
|
||||
"scanCompletedAt": "2025-11-21T09:01:05Z",
|
||||
"severityCounts": {
|
||||
"CRITICAL": 1,
|
||||
"HIGH": 7,
|
||||
"MEDIUM": 13,
|
||||
"LOW": 4
|
||||
},
|
||||
"findingReport": {
|
||||
"mediaType": "application/json",
|
||||
"location": "reports/trivy/app-service-7d9c-vulns.json",
|
||||
"digest": {
|
||||
"sha256": "db569aa8a1b847a922b7d61d276cc2a0ccf99efad0879500b56854b43265c09a"
|
||||
}
|
||||
}
|
||||
},
|
||||
"attestationMeta": {
|
||||
"statementId": "att-vuln-trivy-app-service-7d9c",
|
||||
"createdAt": "2025-11-21T09:01:05Z",
|
||||
"signer": {
|
||||
"name": "ci/trivy-signer",
|
||||
"keyId": "SHA256:ae12c8d1..."
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
312
docs/schemas/audit-bundle-index.schema.json
Normal file
312
docs/schemas/audit-bundle-index.schema.json
Normal file
@@ -0,0 +1,312 @@
|
||||
{
|
||||
"$id": "https://stella.ops/schema/audit-bundle-index.json",
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"title": "AuditBundleIndex",
|
||||
"description": "Root manifest for an immutable audit bundle containing vulnerability reports, VEX decisions, policy evaluations, and attestations",
|
||||
"type": "object",
|
||||
"required": ["apiVersion", "kind", "bundleId", "createdAt", "createdBy", "subject", "artifacts"],
|
||||
"properties": {
|
||||
"apiVersion": {
|
||||
"type": "string",
|
||||
"const": "stella.ops/v1",
|
||||
"description": "API version for this bundle format"
|
||||
},
|
||||
"kind": {
|
||||
"type": "string",
|
||||
"const": "AuditBundleIndex",
|
||||
"description": "Resource kind identifier"
|
||||
},
|
||||
"bundleId": {
|
||||
"type": "string",
|
||||
"description": "Unique identifier for this bundle",
|
||||
"examples": ["bndl-6f6b0c94-9c5b-4bbf-9a77-a5d8a83da4a2"]
|
||||
},
|
||||
"createdAt": {
|
||||
"type": "string",
|
||||
"format": "date-time",
|
||||
"description": "ISO-8601 timestamp when bundle was created"
|
||||
},
|
||||
"createdBy": {
|
||||
"$ref": "#/$defs/BundleActorRef",
|
||||
"description": "User who created this bundle"
|
||||
},
|
||||
"subject": {
|
||||
"$ref": "#/$defs/BundleSubjectRef",
|
||||
"description": "Primary artifact this bundle documents"
|
||||
},
|
||||
"timeWindow": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"from": {
|
||||
"type": "string",
|
||||
"format": "date-time",
|
||||
"description": "Start of time window for included artifacts"
|
||||
},
|
||||
"to": {
|
||||
"type": "string",
|
||||
"format": "date-time",
|
||||
"description": "End of time window for included artifacts"
|
||||
}
|
||||
},
|
||||
"description": "Optional time window filter for included content"
|
||||
},
|
||||
"artifacts": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/$defs/BundleArtifact"
|
||||
},
|
||||
"description": "List of artifacts included in this bundle"
|
||||
},
|
||||
"vexDecisions": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/$defs/BundleVexDecisionEntry"
|
||||
},
|
||||
"description": "Summary of VEX decisions included in this bundle"
|
||||
},
|
||||
"integrity": {
|
||||
"$ref": "#/$defs/BundleIntegrity",
|
||||
"description": "Integrity verification data for the entire bundle"
|
||||
}
|
||||
},
|
||||
"$defs": {
|
||||
"BundleActorRef": {
|
||||
"type": "object",
|
||||
"required": ["id", "displayName"],
|
||||
"properties": {
|
||||
"id": {
|
||||
"type": "string",
|
||||
"description": "User identifier"
|
||||
},
|
||||
"displayName": {
|
||||
"type": "string",
|
||||
"description": "Human-readable display name"
|
||||
}
|
||||
}
|
||||
},
|
||||
"BundleSubjectRef": {
|
||||
"type": "object",
|
||||
"required": ["type", "name", "digest"],
|
||||
"properties": {
|
||||
"type": {
|
||||
"type": "string",
|
||||
"enum": ["IMAGE", "REPO", "SBOM", "OTHER"],
|
||||
"description": "Type of subject artifact"
|
||||
},
|
||||
"name": {
|
||||
"type": "string",
|
||||
"description": "Human-readable subject name"
|
||||
},
|
||||
"digest": {
|
||||
"type": "object",
|
||||
"additionalProperties": {
|
||||
"type": "string"
|
||||
},
|
||||
"description": "Algorithm -> digest map"
|
||||
}
|
||||
}
|
||||
},
|
||||
"BundleArtifact": {
|
||||
"type": "object",
|
||||
"required": ["id", "type", "source", "path", "mediaType", "digest"],
|
||||
"properties": {
|
||||
"id": {
|
||||
"type": "string",
|
||||
"description": "Internal identifier for this artifact within the bundle"
|
||||
},
|
||||
"type": {
|
||||
"type": "string",
|
||||
"enum": ["VULN_REPORT", "SBOM", "VEX", "POLICY_EVAL", "OTHER"],
|
||||
"description": "Type of artifact"
|
||||
},
|
||||
"source": {
|
||||
"type": "string",
|
||||
"description": "Tool/service that produced this artifact",
|
||||
"examples": ["Trivy@0.53.0", "Syft@1.0.0", "StellaOps", "StellaPolicyEngine@2.1.0"]
|
||||
},
|
||||
"path": {
|
||||
"type": "string",
|
||||
"description": "Relative path within the bundle",
|
||||
"examples": ["reports/trivy/app-service-7d9c-vulns.json"]
|
||||
},
|
||||
"mediaType": {
|
||||
"type": "string",
|
||||
"description": "Media type of the artifact",
|
||||
"examples": ["application/json", "application/vnd.cyclonedx+json"]
|
||||
},
|
||||
"digest": {
|
||||
"type": "object",
|
||||
"additionalProperties": {
|
||||
"type": "string"
|
||||
},
|
||||
"description": "Content digest of the artifact"
|
||||
},
|
||||
"attestation": {
|
||||
"$ref": "#/$defs/BundleArtifactAttestationRef",
|
||||
"description": "Optional reference to attestation for this artifact"
|
||||
}
|
||||
}
|
||||
},
|
||||
"BundleArtifactAttestationRef": {
|
||||
"type": "object",
|
||||
"required": ["path", "digest"],
|
||||
"properties": {
|
||||
"path": {
|
||||
"type": "string",
|
||||
"description": "Relative path to attestation within the bundle"
|
||||
},
|
||||
"digest": {
|
||||
"type": "object",
|
||||
"additionalProperties": {
|
||||
"type": "string"
|
||||
},
|
||||
"description": "Content digest of the attestation"
|
||||
}
|
||||
}
|
||||
},
|
||||
"BundleVexDecisionEntry": {
|
||||
"type": "object",
|
||||
"required": ["decisionId", "vulnerabilityId", "status", "path", "digest"],
|
||||
"properties": {
|
||||
"decisionId": {
|
||||
"type": "string",
|
||||
"format": "uuid",
|
||||
"description": "VEX decision ID"
|
||||
},
|
||||
"vulnerabilityId": {
|
||||
"type": "string",
|
||||
"description": "CVE or vulnerability identifier"
|
||||
},
|
||||
"status": {
|
||||
"type": "string",
|
||||
"enum": ["NOT_AFFECTED", "AFFECTED_MITIGATED", "AFFECTED_UNMITIGATED", "FIXED"],
|
||||
"description": "VEX status"
|
||||
},
|
||||
"path": {
|
||||
"type": "string",
|
||||
"description": "Relative path to VEX decision file"
|
||||
},
|
||||
"digest": {
|
||||
"type": "object",
|
||||
"additionalProperties": {
|
||||
"type": "string"
|
||||
},
|
||||
"description": "Content digest of the decision file"
|
||||
}
|
||||
}
|
||||
},
|
||||
"BundleIntegrity": {
|
||||
"type": "object",
|
||||
"required": ["rootHash", "hashAlgorithm"],
|
||||
"properties": {
|
||||
"rootHash": {
|
||||
"type": "string",
|
||||
"description": "Root hash covering all artifacts in the bundle"
|
||||
},
|
||||
"hashAlgorithm": {
|
||||
"type": "string",
|
||||
"default": "sha256",
|
||||
"description": "Hash algorithm used for integrity verification"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"examples": [
|
||||
{
|
||||
"apiVersion": "stella.ops/v1",
|
||||
"kind": "AuditBundleIndex",
|
||||
"bundleId": "bndl-6f6b0c94-9c5b-4bbf-9a77-a5d8a83da4a2",
|
||||
"createdAt": "2025-11-21T09:05:30Z",
|
||||
"createdBy": {
|
||||
"id": "user-123",
|
||||
"displayName": "Alice Johnson"
|
||||
},
|
||||
"subject": {
|
||||
"type": "IMAGE",
|
||||
"name": "registry.internal/stella/app-service@sha256:7d9c...",
|
||||
"digest": {
|
||||
"sha256": "7d9cd5f1a2a0dd9a41a2c43a5b7d8a0bcd9e34cf39b3f43a70595c834f0a4aee"
|
||||
}
|
||||
},
|
||||
"timeWindow": {
|
||||
"from": "2025-11-14T00:00:00Z",
|
||||
"to": "2025-11-21T09:05:00Z"
|
||||
},
|
||||
"artifacts": [
|
||||
{
|
||||
"id": "vuln-report-trivy",
|
||||
"type": "VULN_REPORT",
|
||||
"source": "Trivy@0.53.0",
|
||||
"path": "reports/trivy/app-service-7d9c-vulns.json",
|
||||
"mediaType": "application/json",
|
||||
"digest": {
|
||||
"sha256": "db569aa8a1b847a922b7d61d276cc2a0ccf99efad0879500b56854b43265c09a"
|
||||
},
|
||||
"attestation": {
|
||||
"path": "attestations/vuln-scan-trivy.dsse.json",
|
||||
"digest": {
|
||||
"sha256": "2e613df97fe2aa9baf7a8dac9cfaa407e60c808a8af8e7d5e50c029f6c51a54b"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "sbom-cyclonedx",
|
||||
"type": "SBOM",
|
||||
"source": "Syft@1.0.0",
|
||||
"path": "sbom/app-service-7d9c-cyclonedx.json",
|
||||
"mediaType": "application/vnd.cyclonedx+json",
|
||||
"digest": {
|
||||
"sha256": "9477b3a9410423b37c39076678a936d5854aa2d905e72a2222c153e3e51ab150"
|
||||
},
|
||||
"attestation": {
|
||||
"path": "attestations/sbom-syft.dsse.json",
|
||||
"digest": {
|
||||
"sha256": "3ebf5dc03f862b4b2fdef201130f5c6a9bde7cb0bcf4f57e7686adbc83c9c897"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "vex-decisions",
|
||||
"type": "VEX",
|
||||
"source": "StellaOps",
|
||||
"path": "vex/app-service-7d9c-vex.json",
|
||||
"mediaType": "application/json",
|
||||
"digest": {
|
||||
"sha256": "b56f0d05af5dc4ba79ccc1d228dba27a0d9607eef17fa7faf569e3020c39da83"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "policy-eval-prod-admission",
|
||||
"type": "POLICY_EVAL",
|
||||
"source": "StellaPolicyEngine@2.1.0",
|
||||
"path": "policy-evals/prod-admission.json",
|
||||
"mediaType": "application/json",
|
||||
"digest": {
|
||||
"sha256": "cf8617dd3a63b953f31501045bb559c7095fa2b6965643b64a4b463756cfa9c3"
|
||||
},
|
||||
"attestation": {
|
||||
"path": "attestations/policy-prod-admission.dsse.json",
|
||||
"digest": {
|
||||
"sha256": "a7ea883ffa1100a62f0f89f455b659017864c65a4fad0af0ac3d8b989e1a6ff3"
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"vexDecisions": [
|
||||
{
|
||||
"decisionId": "8a3d0b5a-1e07-4b57-b6a1-1a29ce6c889e",
|
||||
"vulnerabilityId": "CVE-2023-12345",
|
||||
"status": "NOT_AFFECTED",
|
||||
"path": "vex/CVE-2023-12345-app-service.json",
|
||||
"digest": {
|
||||
"sha256": "b56f0d05af5dc4ba79ccc1d228dba27a0d9607eef17fa7faf569e3020c39da83"
|
||||
}
|
||||
}
|
||||
],
|
||||
"integrity": {
|
||||
"rootHash": "f4ede91c4396f9dfdacaf15fe0293c6349f467701f4ef7af6a2ecd4f5bf42254",
|
||||
"hashAlgorithm": "sha256"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
257
docs/schemas/vex-decision.schema.json
Normal file
257
docs/schemas/vex-decision.schema.json
Normal file
@@ -0,0 +1,257 @@
|
||||
{
|
||||
"$id": "https://stella.ops/schema/vex-decision.json",
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"title": "VexDecision",
|
||||
"description": "VEX-style statement attached to a finding + subject, representing a vulnerability exploitability decision",
|
||||
"type": "object",
|
||||
"required": [
|
||||
"id",
|
||||
"vulnerabilityId",
|
||||
"subject",
|
||||
"status",
|
||||
"justificationType",
|
||||
"createdBy",
|
||||
"createdAt"
|
||||
],
|
||||
"properties": {
|
||||
"id": {
|
||||
"type": "string",
|
||||
"format": "uuid",
|
||||
"description": "Internal stable ID for this decision"
|
||||
},
|
||||
"vulnerabilityId": {
|
||||
"type": "string",
|
||||
"description": "CVE, GHSA, or other vulnerability identifier",
|
||||
"examples": ["CVE-2023-12345", "GHSA-xxxx-yyyy-zzzz"]
|
||||
},
|
||||
"subject": {
|
||||
"$ref": "#/$defs/SubjectRef",
|
||||
"description": "The artifact or SBOM component this decision applies to"
|
||||
},
|
||||
"status": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"NOT_AFFECTED",
|
||||
"AFFECTED_MITIGATED",
|
||||
"AFFECTED_UNMITIGATED",
|
||||
"FIXED"
|
||||
],
|
||||
"description": "VEX status following OpenVEX semantics"
|
||||
},
|
||||
"justificationType": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"CODE_NOT_PRESENT",
|
||||
"CODE_NOT_REACHABLE",
|
||||
"VULNERABLE_CODE_NOT_IN_EXECUTE_PATH",
|
||||
"CONFIGURATION_NOT_AFFECTED",
|
||||
"OS_NOT_AFFECTED",
|
||||
"RUNTIME_MITIGATION_PRESENT",
|
||||
"COMPENSATING_CONTROLS",
|
||||
"ACCEPTED_BUSINESS_RISK",
|
||||
"OTHER"
|
||||
],
|
||||
"description": "Justification type inspired by CSAF/VEX specifications"
|
||||
},
|
||||
"justificationText": {
|
||||
"type": "string",
|
||||
"maxLength": 4000,
|
||||
"description": "Free-form explanation supporting the justification type"
|
||||
},
|
||||
"evidenceRefs": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/$defs/EvidenceRef"
|
||||
},
|
||||
"description": "Links to PRs, commits, tickets, docs supporting this decision"
|
||||
},
|
||||
"scope": {
|
||||
"$ref": "#/$defs/VexScope",
|
||||
"description": "Environments and projects where this decision applies"
|
||||
},
|
||||
"validFor": {
|
||||
"$ref": "#/$defs/ValidFor",
|
||||
"description": "Time window during which this decision is valid"
|
||||
},
|
||||
"attestationRef": {
|
||||
"$ref": "#/$defs/AttestationRef",
|
||||
"description": "Reference to the signed attestation for this decision"
|
||||
},
|
||||
"supersedesDecisionId": {
|
||||
"type": "string",
|
||||
"format": "uuid",
|
||||
"description": "ID of a previous decision this one supersedes"
|
||||
},
|
||||
"createdBy": {
|
||||
"$ref": "#/$defs/ActorRef",
|
||||
"description": "User who created this decision"
|
||||
},
|
||||
"createdAt": {
|
||||
"type": "string",
|
||||
"format": "date-time",
|
||||
"description": "ISO-8601 timestamp when decision was created"
|
||||
},
|
||||
"updatedAt": {
|
||||
"type": "string",
|
||||
"format": "date-time",
|
||||
"description": "ISO-8601 timestamp when decision was last updated"
|
||||
}
|
||||
},
|
||||
"$defs": {
|
||||
"SubjectRef": {
|
||||
"type": "object",
|
||||
"required": ["type", "name", "digest"],
|
||||
"properties": {
|
||||
"type": {
|
||||
"type": "string",
|
||||
"enum": ["IMAGE", "REPO", "SBOM_COMPONENT", "OTHER"],
|
||||
"description": "Type of artifact this subject represents"
|
||||
},
|
||||
"name": {
|
||||
"type": "string",
|
||||
"description": "Human-readable subject name (e.g. image ref, package name)",
|
||||
"examples": ["registry.internal/stella/app-service@sha256:7d9c..."]
|
||||
},
|
||||
"digest": {
|
||||
"type": "object",
|
||||
"additionalProperties": {
|
||||
"type": "string"
|
||||
},
|
||||
"description": "Algorithm -> digest map (e.g. sha256 -> hex string)",
|
||||
"examples": [{"sha256": "7d9cd5f1a2a0dd9a41a2c43a5b7d8a0bcd9e34cf39b3f43a70595c834f0a4aee"}]
|
||||
},
|
||||
"sbomNodeId": {
|
||||
"type": "string",
|
||||
"description": "Optional SBOM node/bomRef identifier for SBOM_COMPONENT subjects"
|
||||
}
|
||||
}
|
||||
},
|
||||
"EvidenceRef": {
|
||||
"type": "object",
|
||||
"required": ["type", "url"],
|
||||
"properties": {
|
||||
"type": {
|
||||
"type": "string",
|
||||
"enum": ["PR", "TICKET", "DOC", "COMMIT", "OTHER"],
|
||||
"description": "Type of evidence link"
|
||||
},
|
||||
"title": {
|
||||
"type": "string",
|
||||
"description": "Human-readable title for the evidence"
|
||||
},
|
||||
"url": {
|
||||
"type": "string",
|
||||
"format": "uri",
|
||||
"description": "URL to the evidence resource"
|
||||
}
|
||||
}
|
||||
},
|
||||
"VexScope": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"environments": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "string"
|
||||
},
|
||||
"description": "Environment names where decision applies (e.g. prod, staging)",
|
||||
"examples": [["prod", "staging"]]
|
||||
},
|
||||
"projects": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "string"
|
||||
},
|
||||
"description": "Project/service names where decision applies"
|
||||
}
|
||||
},
|
||||
"description": "If empty/null, decision applies to all environments and projects"
|
||||
},
|
||||
"ValidFor": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"notBefore": {
|
||||
"type": "string",
|
||||
"format": "date-time",
|
||||
"description": "Decision is not valid before this timestamp (defaults to creation time)"
|
||||
},
|
||||
"notAfter": {
|
||||
"type": "string",
|
||||
"format": "date-time",
|
||||
"description": "Decision expires after this timestamp (recommended to set)"
|
||||
}
|
||||
}
|
||||
},
|
||||
"AttestationRef": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"id": {
|
||||
"type": "string",
|
||||
"description": "Internal attestation identifier"
|
||||
},
|
||||
"digest": {
|
||||
"type": "object",
|
||||
"additionalProperties": {
|
||||
"type": "string"
|
||||
},
|
||||
"description": "Content digest of the attestation"
|
||||
},
|
||||
"storage": {
|
||||
"type": "string",
|
||||
"description": "Storage location (OCI ref, bundle path, or URL)",
|
||||
"examples": ["oci://registry.internal/stella/attestations@sha256:2e61..."]
|
||||
}
|
||||
}
|
||||
},
|
||||
"ActorRef": {
|
||||
"type": "object",
|
||||
"required": ["id", "displayName"],
|
||||
"properties": {
|
||||
"id": {
|
||||
"type": "string",
|
||||
"description": "User identifier"
|
||||
},
|
||||
"displayName": {
|
||||
"type": "string",
|
||||
"description": "Human-readable display name"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"examples": [
|
||||
{
|
||||
"id": "8a3d0b5a-1e07-4b57-b6a1-1a29ce6c889e",
|
||||
"vulnerabilityId": "CVE-2023-12345",
|
||||
"subject": {
|
||||
"type": "IMAGE",
|
||||
"name": "registry.internal/stella/app-service@sha256:7d9c...",
|
||||
"digest": {
|
||||
"sha256": "7d9cd5f1a2a0dd9a41a2c43a5b7d8a0bcd9e34cf39b3f43a70595c834f0a4aee"
|
||||
}
|
||||
},
|
||||
"status": "NOT_AFFECTED",
|
||||
"justificationType": "VULNERABLE_CODE_NOT_IN_EXECUTE_PATH",
|
||||
"justificationText": "Vulnerable CLI helper is present in the image but never invoked in the running service.",
|
||||
"evidenceRefs": [
|
||||
{
|
||||
"type": "PR",
|
||||
"title": "Document non-usage of CLI helper",
|
||||
"url": "https://git.example.com/stella/app-service/merge_requests/42"
|
||||
}
|
||||
],
|
||||
"scope": {
|
||||
"environments": ["prod", "staging"],
|
||||
"projects": ["app-service"]
|
||||
},
|
||||
"validFor": {
|
||||
"notBefore": "2025-11-21T10:15:00Z",
|
||||
"notAfter": "2026-05-21T10:15:00Z"
|
||||
},
|
||||
"createdBy": {
|
||||
"id": "user-123",
|
||||
"displayName": "Alice Johnson"
|
||||
},
|
||||
"createdAt": "2025-11-21T10:15:00Z"
|
||||
}
|
||||
]
|
||||
}
|
||||
Reference in New Issue
Block a user