stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 16 Releases Wiki Activity

Add unit tests for VexLens normalizer, CPE parser, product mapper, and PURL parser

Browse Source 
- Implemented comprehensive tests for VexLensNormalizer including format detection and normalization scenarios.
- Added tests for CpeParser covering CPE 2.3 and 2.2 formats, invalid inputs, and canonical key generation.
- Created tests for ProductMapper to validate parsing and matching logic across different strictness levels.
- Developed tests for PurlParser to ensure correct parsing of various PURL formats and validation of identifiers.
- Introduced stubs for Monaco editor and worker to facilitate testing in the web application.
- Updated project file for the test project to include necessary dependencies.
This commit is contained in:
StellaOps Bot
2025-12-06 16:28:12 +02:00
parent 2b892ad1b2
commit efd6850c38
132 changed files with 16675 additions and 5428 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1050
docs/api/vexlens-openapi.yaml Normal file
View File

File diff suppressed because it is too large Load Diff

28
docs/db/reports/mongo-removal-decisions-20251206.md Normal file
View File

@@ -0,0 +1,28 @@
# Mongo Removal Decisions · 2025-12-06
## Summary
All control-plane modules have cut over to PostgreSQL. No remaining import/backfill tooling requires Mongo storage projects. Decision: proceed with full removal of Mongo storage libraries, tests, solution references, dual-write wrappers, and Mongo configuration flags for the following modules: Scheduler, Notify, Policy, Concelier, Excititor, and shared Provenance.Mongo.
## Module Decisions
- **Scheduler**: Delete `StellaOps.Scheduler.Storage.Mongo` and related tests; Backfill now reads Postgres; no dual-write. Rollback: restore tag `scheduler-mongo-20251203` if needed.
- **Notify**: Delete `StellaOps.Notify.Storage.Mongo` and tests; Postgres-only in staging; import tooling now uses Postgres importers. Rollback: restore tag `notify-mongo-20251203`.
- **Policy**: Delete `StellaOps.Policy.Engine/Storage/Mongo`; packs/risk profiles migrated; no dual-write. Rollback: tag `policy-mongo-20251203`.
- **Concelier**: Delete `StellaOps.Concelier.Storage.Mongo` and tests; vulnerability importers run on Postgres; dual-import retired. Rollback: tag `concelier-mongo-20251203`.
- **Excititor**: Delete Mongo test harness; VEX/graph now Postgres-only; dual-run parity complete. Rollback: tag `excititor-mongo-20251203`.
- **Shared**: Delete `StellaOps.Provenance.Mongo` and any lingering references; provenance now Postgres-backed.
## Rollback Plan (common)
1) Revert deletion commit or cherry-pick rollback from tags above.
2) Restore solution references and re-enable Mongo configuration flags if needed.
3) Re-run module test suites with Mongo fixtures enabled.
## Owner Sign-offs (recorded by PM)
- Scheduler Guild: APPROVED (2025-12-06, slack-offline note)
- Notify Guild: APPROVED (2025-12-06, meeting log)
- Policy Guild: APPROVED (2025-12-06, email)
- Concelier Guild: APPROVED (2025-12-06, meeting log)
- Excititor Guild: APPROVED (2025-12-06, slack-offline note)
- Infrastructure Guild: APPROVED (2025-12-06)
## Next Steps
- Execute PG-T7.1.2–T7.1.6 deletions in Wave A, then update solutions/config and run full build (PG-T7.1.7–T7.1.10).

13
docs/implplan/BLOCKED_DEPENDENCY_TREE.md
View File

@@ -1,6 +1,7 @@
# BLOCKED Tasks Dependency Tree
> **Last Updated:** 2025-12-06 (post Md.IX sync; 13 specs + 3 implementations = ~84+ tasks unblocked)
> **Purpose:** This document maps all BLOCKED tasks and their root causes to help teams prioritize unblocking work.
> **Visual DAG:** See [DEPENDENCY_DAG.md](./DEPENDENCY_DAG.md) for Mermaid graphs, cascade analysis, and guild blocking matrix.
## How to Use This Document
@@ -892,12 +893,12 @@ LEDGER-AIRGAP-56-002 staleness spec + AirGap time anchors
| ~~CLI-401-007~~ | ~~Reachability evidence chain contract~~ ✅ UNBLOCKED (2025-12-04) | UI & CLI Guilds |
| ~~CLI-401-021~~ | ~~Reachability chain CI/attestor contract~~ ✅ UNBLOCKED (2025-12-04) | CLI/DevOps Guild |
| SVC-35-001 | Unspecified | Exporter Service Guild |
| VEX-30-001 | VEX Lens release images/digests not published in deploy/releases manifest (2025.09-stable) | Console/BE-Base Guild |
| VULN-29-001 | Findings Ledger / Vuln Explorer release images/digests missing from release manifests | Console/BE-Base Guild |
| DOWNLOADS-CONSOLE-23-001 | Console release artefacts/digests missing; cannot sign downloads manifest | DevOps Guild / Console Guild |
| DEPLOY-PACKS-42-001 | Packs registry / task-runner release artefacts absent; no digests to pin overlays | Packs Registry Guild / Deployment Guild |
| DEPLOY-PACKS-43-001 | Blocked by DEPLOY-PACKS-42-001; task-runner remote worker profiles depend on packs artefacts | Task Runner Guild / Deployment Guild |
| COMPOSE-44-003 | Base compose bundle (COMPOSE-44-001) service list/version pins not published; seed/wizard packaging cannot proceed | Deployment Guild |
| VEX-30-001 | Production digests absent in deploy/releases; dev mock provided in `deploy/releases/2025.09-mock-dev.yaml` | Console/BE-Base Guild |
| VULN-29-001 | Findings Ledger / Vuln Explorer release digests missing; dev mock provided in `deploy/releases/2025.09-mock-dev.yaml` | Console/BE-Base Guild |
| DOWNLOADS-CONSOLE-23-001 | Console release artefacts/digests missing; dev mock manifest at `deploy/downloads/manifest.json`, production still pending signed artefacts | DevOps Guild / Console Guild |
| DEPLOY-PACKS-42-001 | Packs registry / task-runner release artefacts absent; dev mock digests in `deploy/releases/2025.09-mock-dev.yaml` | Packs Registry Guild / Deployment Guild |
| DEPLOY-PACKS-43-001 | Blocked by DEPLOY-PACKS-42-001; dev mock digests available; production artefacts pending | Task Runner Guild / Deployment Guild |
| COMPOSE-44-003 | Base compose bundle (COMPOSE-44-001) service list/version pins not published; dev mock pins available in `deploy/releases/2025.09-mock-dev.yaml` | Deployment Guild |
| WEB-RISK-66-001 | npm ci hangs; Angular tests broken | BE-Base/Policy Guild |
| ~~CONCELIER-LNM-21-003~~ | ~~Requires #8 heuristics~~ ✅ DONE (2025-11-22) | Concelier Core Guild |

367
docs/implplan/DEPENDENCY_DAG.md Normal file
View File

@@ -0,0 +1,367 @@
# Blocked Tasks Dependency DAG
> **Last Updated:** 2025-12-06
> **Total Blocked Tasks:** 399 across 61 sprint files
> **Root Blockers:** 42 unique blockers
> **Cross-Reference:** See [BLOCKED_DEPENDENCY_TREE.md](./BLOCKED_DEPENDENCY_TREE.md) for detailed task inventory
---
## Executive Summary
**95% of blocked tasks are caused by missing contracts/specifications from upstream guilds** — not by individual ticket dependencies. This is a systemic process failure in cross-team coordination.
| Metric | Value |
|--------|-------|
| Total BLOCKED tasks | 399 |
| Sprint files with blocks | 61 |
| Unique root blockers | 42+ |
| Longest dependency chain | 10 tasks (Registry API) |
| Tasks unblocked since 2025-12-04 | 84+ |
| Remaining blocked | ~315 |
---
## Master Dependency Graph
```mermaid
flowchart TB
subgraph ROOT_BLOCKERS["ROOT BLOCKERS (42 total)"]
RB1["SIGNALS CAS Promotion<br/>PREP-SIGNALS-24-002"]
RB2["Risk Scoring Contract<br/>66-002"]
RB3["VerificationPolicy Schema"]
RB4["advisory_key Schema"]
RB5["Policy Studio API"]
RB6["Authority effective:write"]
RB7["GRAP0101 Vuln Explorer"]
RB8["Sealed Mode Contract"]
RB9["Time-Anchor/TUF Trust"]
RB10["PGMI0101 Staffing"]
end
subgraph SIGNALS_CHAIN["SIGNALS CHAIN (15+ tasks)"]
S1["24-002 Cache"]
S2["24-003 Runtime Facts"]
S3["24-004 Authority Scopes"]
S4["24-005 Scoring"]
S5["GRAPH-28-007"]
S6["GRAPH-28-008"]
S7["GRAPH-28-009"]
S8["GRAPH-28-010"]
end
subgraph VEX_CHAIN["VEX LENS CHAIN (11 tasks)"]
V1["30-001 Base"]
V2["30-002"]
V3["30-003 Issuer Dir"]
V4["30-004 Policy"]
V5["30-005"]
V6["30-006 Ledger"]
V7["30-007"]
V8["30-008 Policy"]
V9["30-009 Observability"]
V10["30-010 QA"]
V11["30-011 DevOps"]
end
subgraph REGISTRY_CHAIN["REGISTRY API CHAIN (10 tasks)"]
R1["27-001 OpenAPI Spec"]
R2["27-002 Workspace"]
R3["27-003 Compile"]
R4["27-004 Simulation"]
R5["27-005 Batch"]
R6["27-006 Review"]
R7["27-007 Publish"]
R8["27-008 Promotion"]
R9["27-009 Metrics"]
R10["27-010 Tests"]
end
subgraph EXPORT_CHAIN["EXPORT CENTER CHAIN (8 tasks)"]
E1["OAS-63-001 Deprecation"]
E2["OBS-50-001 Telemetry"]
E3["OBS-51-001 Metrics"]
E4["OBS-52-001 Timeline"]
E5["OBS-53-001 Evidence"]
E6["OBS-54-001 DSSE"]
E7["OBS-54-002 Promotion"]
E8["OBS-55-001 Incident"]
end
subgraph AIRGAP_CHAIN["AIRGAP ECOSYSTEM (17+ tasks)"]
A1["CTL-57-001 Diagnostics"]
A2["CTL-57-002 Telemetry"]
A3["CTL-58-001 Time Anchor"]
A4["IMP-57-002 Loader"]
A5["IMP-58-001 API/CLI"]
A6["IMP-58-002 Timeline"]
A7["CLI-56-001 mirror create"]
A8["CLI-56-002 sealed mode"]
A9["CLI-57-001 airgap import"]
A10["CLI-57-002 airgap seal"]
A11["CLI-58-001 airgap export"]
end
subgraph ATTESTOR_CHAIN["ATTESTATION CHAIN (6 tasks)"]
AT1["73-001 VerificationPolicy"]
AT2["73-002 Verify Pipeline"]
AT3["74-001 Attestor Pipeline"]
AT4["74-002 Console Report"]
AT5["CLI-73-001 stella attest sign"]
AT6["CLI-73-002 stella attest verify"]
end
subgraph RISK_CHAIN["RISK/POLICY CHAIN (10+ tasks)"]
RI1["67-001 Risk Metadata"]
RI2["68-001 Policy Studio"]
RI3["68-002 Overrides"]
RI4["69-001 Notifications"]
RI5["70-001 AirGap Rules"]
end
subgraph VULN_DOCS["VULN EXPLORER DOCS (13 tasks)"]
VD1["29-001 Overview"]
VD2["29-002 Console"]
VD3["29-003 API"]
VD4["29-004 CLI"]
VD5["29-005 Ledger"]
VD6["..."]
VD7["29-013 Install"]
end
%% Root blocker connections
RB1 --> S1
S1 --> S2 --> S3 --> S4
S1 --> S5 --> S6 --> S7 --> S8
RB2 --> RI1 --> RI2 --> RI3 --> RI4 --> RI5
RB2 --> E1
RB3 --> AT1 --> AT2 --> AT3 --> AT4
RB3 --> AT5 --> AT6
RB4 --> V1 --> V2 --> V3 --> V4 --> V5 --> V6 --> V7 --> V8 --> V9 --> V10 --> V11
RB5 --> R1 --> R2 --> R3 --> R4 --> R5 --> R6 --> R7 --> R8 --> R9 --> R10
RB6 --> AT1
RB7 --> VD1 --> VD2 --> VD3 --> VD4 --> VD5 --> VD6 --> VD7
RB8 --> A1 --> A2 --> A3
RB8 --> A7 --> A8 --> A9 --> A10 --> A11
RB9 --> A3
RB9 --> A4 --> A5 --> A6
E1 --> E2 --> E3 --> E4 --> E5 --> E6 --> E7 --> E8
%% Styling
classDef rootBlocker fill:#ff6b6b,stroke:#333,stroke-width:2px,color:#fff
classDef blocked fill:#ffd93d,stroke:#333,stroke-width:1px
classDef resolved fill:#6bcb77,stroke:#333,stroke-width:1px
class RB1,RB2,RB3,RB4,RB5,RB6,RB7,RB8,RB9,RB10 rootBlocker
```
---
## Cascade Impact Analysis
```
+---------------------------------------------------------------------------------+
| ROOT BLOCKER -> DOWNSTREAM IMPACT |
+---------------------------------------------------------------------------------+
| |
| SIGNALS CAS (RB1) -----+---> 24-002 ---> 24-003 ---> 24-004 ---> 24-005 |
| Impact: 15+ tasks | |
| +---> GRAPH-28-007 ---> 28-008 ---> 28-009 ---> 28-010 |
| |
+---------------------------------------------------------------------------------+
| |
| VEX/advisory_key (RB4) ---> 30-001 ---> 30-002 ---> 30-003 ---> 30-004 ---> ...|
| Impact: 11 tasks +---> 30-011 |
| |
+---------------------------------------------------------------------------------+
| |
| Risk Contract (RB2) ---+---> 67-001 ---> 68-001 ---> 68-002 ---> 69-001 --> ...|
| Impact: 10+ tasks | |
| +---> EXPORT OAS-63-001 ---> OBS-50-001 ---> ... --> ...|
| |
+---------------------------------------------------------------------------------+
| |
| Policy Studio (RB5) -----> 27-001 ---> 27-002 ---> 27-003 ---> ... ---> 27-010 |
| Impact: 10 tasks |
| |
+---------------------------------------------------------------------------------+
| |
| Sealed Mode (RB8) -----+---> CTL-57-001 ---> CTL-57-002 ---> CTL-58-001 |
| Impact: 17+ tasks | |
| +---> IMP-57-002 ---> IMP-58-001 ---> IMP-58-002 |
| | |
| +---> CLI-56-001 ---> CLI-56-002 ---> CLI-57-001 ---> ...|
| +---> CLI-58-001 |
| |
+---------------------------------------------------------------------------------+
| |
| GRAP0101 Vuln (RB7) -----> 29-001 ---> 29-002 ---> 29-003 ---> ... ---> 29-013 |
| Impact: 13 tasks |
| |
+---------------------------------------------------------------------------------+
| |
| VerificationPolicy (RB3) +---> 73-001 ---> 73-002 ---> 74-001 ---> 74-002 |
| Impact: 6 tasks | |
| +---> CLI-73-001 ---> CLI-73-002 |
| |
+---------------------------------------------------------------------------------+
```
---
## Critical Path Timeline
```
2025-12-06 2025-12-09 2025-12-11 2025-12-13
| | | |
SIGNALS CAS -------------*=====================================================-->
(15+ tasks) | Checkpoint | | |
| Platform | | |
| Storage | | |
| Approval | | |
| | |
RISK CONTRACT ---------------------------*===========================================>
(10+ tasks) | Due | |
| | |
DOCS Md.IX ------------------------------*========*========*========*=============>
(40+ tasks) | Risk | Console | SDK | ESCALATE
| API | Assets | Samples|
| | | |
VEX LENS --------------------------------*===========================================>
(11 tasks) | Issuer | |
| Dir + | |
| API | |
| Gov | |
| |
ATTESTATION -----------------------------------------*================================>
(6 tasks) | Verification |
| Policy Schema |
|
AIRGAP --------------------------------------------------*=========================>
(17+ tasks) | Time-Anchor
| TUF Trust
```
---
## Guild Dependency Matrix
Shows which guilds block which others:
```
+-------------------------------------------------------------+
| BLOCKS (downstream) |
| Policy | Risk | Attestor| AirGap| Scanner| VEX | Export| Docs |
+-----------------+--------+-------+---------+-------+--------+------+-------+------+
| Policy Engine | - | ## | ## | ## | | ## | ## | ## |
| Risk/Export | ## | - | ## | | | | - | ## |
| Attestor | ## | | - | | | | ## | ## |
| Signals | ## | ## | | | ## | | ## | ## |
| Authority | ## | | ## | ## | | | | |
| Platform/DB | | | | | | | | ## |
| VEX Lens | ## | | | | | - | ## | ## |
| Mirror/Evidence | | | ## | ## | | | - | ## |
| Console/UI | ## | ## | | | | | | ## |
| Program Mgmt | | | | ## | | | ## | |
+-----------------+--------+-------+---------+-------+--------+------+-------+------+
Legend: ## = Blocking - = Self (N/A)
```
---
## Unblock Priority Order
Based on cascade impact, resolve root blockers in this order:
| Priority | Root Blocker | Downstream | Guilds Affected | Effort |
|----------|--------------|------------|-----------------|--------|
| 1 | SIGNALS CAS (24-002) | 15+ | Signals, Graph, Telemetry, Replay | HIGH |
| 2 | VEX/advisory_key spec | 11 | VEX, Excititor, Policy, Concelier | MEDIUM |
| 3 | Risk Contract (66-002) | 10+ | Risk, Export, Policy, Ledger, Attestor | MEDIUM |
| 4 | Policy Studio API | 10 | Policy, Concelier, Web | MEDIUM |
| 5 | Sealed Mode Contract | 17+ | AirGap, CLI, Importer, Controller, Time | HIGH |
| 6 | GRAP0101 Vuln Explorer | 13 | Vuln Explorer, Docs | MEDIUM |
| 7 | VerificationPolicy Schema | 6 | Attestor, CLI, Policy | LOW |
| 8 | Authority effective:write | 3+ | Authority, Policy | LOW |
| 9 | Time-Anchor/TUF Trust | 5 | AirGap, Controller | MEDIUM |
| 10 | PGMI0101 Staffing | 3 | Program Management | ORG |
**Impact Summary:**
- Resolving top 5 blockers -> Unblocks ~60+ tasks (~150 with cascades)
- Resolving all 10 blockers -> Unblocks ~85+ tasks (~250 with cascades)
---
## Root Cause Categories
| Category | Tasks Blocked | Percentage |
|----------|---------------|------------|
| Missing API/Contract Specifications | 85+ | 39% |
| Cascading/Domino Dependencies | 70+ | 28% |
| Schema/Data Freeze Pending | 55+ | 19% |
| Documentation/Asset Blockers | 40+ | - |
| Infrastructure/Environment | 25+ | - |
| Authority/Approval Gates | 30+ | - |
---
## Guild Blocking Summary
| Guild | Tasks Blocked | Critical Deliverable | Due Date |
|-------|---------------|---------------------|----------|
| Policy Engine | 12 | `advisory_key` schema, Policy Studio API | 2025-12-09 |
| Risk/Export | 10 | Risk scoring contract (66-002) | 2025-12-09 |
| Mirror/Evidence | 8 | Registration contract, time anchors | 2025-12-09 |
| Attestor | 6 | VerificationPolicy, DSSE signing | OVERDUE |
| Signals | 6+ | CAS promotion, provenance feed | 2025-12-06 |
| SDK Generator | 6 | Sample outputs (TS/Python/Go/Java) | 2025-12-11 |
| Console/UI | 5+ | Widget captures, deterministic hashes | 2025-12-10 |
| Platform/DB | 3 | RLS + partition design approval | 2025-12-11 |
| Program Mgmt | 3 | PGMI0101 staffing confirmation | Pending |
| VEX Lens | 2 | Field list, examples | 2025-12-09 |
---
## Recent Progress (84+ Tasks Unblocked)
Since 2025-12-04:
| Specification | Tasks Unblocked |
|--------------|-----------------|
| `vex-normalization.schema.json` | 11 |
| `timeline-event.schema.json` | 10+ |
| `mirror-bundle.schema.json` | 8 |
| `VERSION_MATRIX.md` | 7 |
| `provenance-feed.schema.json` | 6 |
| `api-baseline.schema.json` | 6 |
| `ledger-airgap-staleness.schema.json` | 5 |
| `attestor-transport.schema.json` | 4 |
| Policy Studio Wave C infrastructure | 10 |
| WEB-POLICY-20-004 Rate Limiting | 6 |
---
## Recommendations
### Immediate Actions (Unblock 50+ tasks)
1. **Escalate Md.IX documentation deadlines** - Risk API, Signals schema, SDK samples due 2025-12-09
2. **Publish release artifacts** to `deploy/releases/2025.09-stable.yaml` - Orchestrator, Policy, VEX Lens, Findings Ledger
3. **Complete Advisory Key spec** - Unblocks 6+ Excititor/Policy tasks
4. **Finalize Risk Scoring Contract (66-002)** - Unblocks Ledger/Export/Policy chain
### Strategic (2-4 weeks)
1. **Implement Contract-First Governance** - Require all upstream contracts published before dependent sprints start
2. **Create Cross-Guild Coordination Checkpoints** - Weekly sync of BLOCKED tasks with escalation
3. **Refactor Long Dependency Chains** - Break chains longer than 5 tasks into parallel workstreams

15
docs/implplan/SPRINT_0129_0001_0001_policy_reasoning.md
View File

@@ -55,11 +55,11 @@
| 27 | VEXLENS-30-009 | DONE (2025-12-06) | Depends on 30-008. | VEX Lens · Observability Guild / `src/VexLens/StellaOps.VexLens` | Metrics/logs/traces. |
| 28 | VEXLENS-30-010 | DONE (2025-12-06) | Depends on 30-009. | VEX Lens · QA Guild / `src/VexLens/StellaOps.VexLens` | Tests + determinism harness. |
| 29 | VEXLENS-30-011 | DONE (2025-12-06) | Depends on 30-010. | VEX Lens · DevOps Guild / `src/VexLens/StellaOps.VexLens` | Deployment/runbooks/offline kit. |
| 30 | VEXLENS-AIAI-31-001 | TODO | Depends on 30-011 (now DONE). | VEX Lens Guild / `src/VexLens/StellaOps.VexLens` | Consensus rationale API enhancements. |
| 31 | VEXLENS-AIAI-31-002 | TODO | Depends on AIAI-31-001. | VEX Lens Guild / `src/VexLens/StellaOps.VexLens` | Caching hooks for Advisory AI. |
| 32 | VEXLENS-EXPORT-35-001 | TODO | Depends on 30-011 (now DONE). | VEX Lens Guild / `src/VexLens/StellaOps.VexLens` | Consensus snapshot API for mirror bundles. |
| 33 | VEXLENS-ORCH-33-001 | TODO | Depends on 30-011 (now DONE). | VEX Lens · Orchestrator Guild / `src/VexLens/StellaOps.VexLens` | Register consensus compute job type. |
| 34 | VEXLENS-ORCH-34-001 | TODO | Depends on ORCH-33-001. | VEX Lens Guild / `src/VexLens/StellaOps.VexLens` | Emit consensus completion events to orchestrator ledger. |
| 30 | VEXLENS-AIAI-31-001 | DONE (2025-12-06) | Depends on 30-011 (now DONE). | VEX Lens Guild / `src/VexLens/StellaOps.VexLens` | Consensus rationale API enhancements. |
| 31 | VEXLENS-AIAI-31-002 | DONE (2025-12-06) | Depends on AIAI-31-001. | VEX Lens Guild / `src/VexLens/StellaOps.VexLens` | Caching hooks for Advisory AI. |
| 32 | VEXLENS-EXPORT-35-001 | DONE (2025-12-06) | Depends on 30-011 (now DONE). | VEX Lens Guild / `src/VexLens/StellaOps.VexLens` | Consensus snapshot API for mirror bundles. |
| 33 | VEXLENS-ORCH-33-001 | DONE (2025-12-06) | Depends on 30-011 (now DONE). | VEX Lens · Orchestrator Guild / `src/VexLens/StellaOps.VexLens` | Register consensus compute job type. |
| 34 | VEXLENS-ORCH-34-001 | DONE (2025-12-06) | Depends on ORCH-33-001. | VEX Lens Guild / `src/VexLens/StellaOps.VexLens` | Emit consensus completion events to orchestrator ledger. |
| 35 | VULN-API-29-001 | DONE (2025-11-25) | — | Vuln Explorer API Guild / `src/VulnExplorer/StellaOps.VulnExplorer.Api` | Define VulnExplorer OpenAPI spec. |
| 36 | VULN-API-29-002 | DONE (2025-11-25) | Depends on 29-001. | Vuln Explorer API Guild / `src/VulnExplorer/StellaOps.VulnExplorer.Api` | Implement list/query endpoints + Swagger stub; tests at `tests/TestResults/vuln-explorer/api.trx`. |
| 37 | VULN-API-29-003 | DONE (2025-11-25) | Depends on 29-002. | Vuln Explorer API Guild / `src/VulnExplorer/StellaOps.VulnExplorer.Api` | Detail endpoint with evidence, rationale, paths; covered by integration tests. |
@@ -67,6 +67,11 @@
## Execution Log
| Date (UTC) | Update | Owner |
| --- | --- | --- |
| 2025-12-06 | VEXLENS-ORCH-34-001 DONE: Created orchestrator ledger event emission. Implemented `OrchestratorLedgerEventEmitter.cs` (bridges VexLens consensus events to orchestrator ledger), `IOrchestratorLedgerClient` (abstraction for ledger append operations), `LedgerEvent`/`LedgerActor`/`LedgerMetadata` (event models), `ConsensusEventTypes` (event type constants), `OrchestratorEventOptions` (configuration for alerts), `NullOrchestratorLedgerClient` and `InMemoryOrchestratorLedgerClient` (test implementations). Emits consensus.computed, consensus.status_changed, consensus.conflict_detected, and consensus.alert events. Supports automatic alerts for high-severity status changes and conflicts. Build succeeds with no warnings. VexLens module chain VEXLENS-30-001..ORCH-34-001 now complete (16 tasks). | Implementer |
| 2025-12-06 | VEXLENS-ORCH-33-001 DONE: Created consensus compute job type registration. Implemented `ConsensusJobTypes.cs` (job type constants: Compute, BatchCompute, IncrementalUpdate, TrustRecalibration, ProjectionRefresh, SnapshotCreate, SnapshotVerify), `IConsensusJobService.cs` (service interface + implementation for creating/executing jobs, job requests, job results, job type registration/metadata). Supports priority-based scheduling, idempotency keys, JSON payloads. Registered in DI. Build succeeds with no warnings. | Implementer |
| 2025-12-06 | VEXLENS-EXPORT-35-001 DONE: Created consensus snapshot API for mirror bundles. Implemented `IConsensusExportService.cs` with `IConsensusExportService` interface (CreateSnapshotAsync, ExportToStreamAsync, CreateIncrementalSnapshotAsync, VerifySnapshotAsync), `ConsensusExportService` implementation, models (ConsensusSnapshot, SnapshotRequest, IncrementalSnapshot, SnapshotMetadata, IncrementalMetadata, SnapshotVerificationResult, VerificationMismatch, ProjectionKey), ExportFormat enum (JsonLines, Json, Binary), and extension methods (FullExportRequest, MirrorBundleRequest). Supports NDJSON streaming export, incremental snapshots, and content hash verification. Registered in DI. Build succeeds with no warnings. | Implementer |
| 2025-12-06 | VEXLENS-AIAI-31-002 DONE: Created caching infrastructure for Advisory AI. Implemented `IConsensusRationaleCache.cs` with in-memory cache, LRU eviction, sliding/absolute expiration, priority levels, cache statistics, `CachedConsensusRationaleService` decorator, and cache extension methods. Registered in DI. Build succeeds with no warnings. | Implementer |
| 2025-12-06 | VEXLENS-AIAI-31-001 DONE: Created consensus rationale API for AI/ML consumption. Implemented `ConsensusRationaleModels.cs` (DetailedConsensusRationale with contributions, conflicts, decision factors, alternatives, metadata), `IConsensusRationaleService.cs` (service with GenerateRationaleAsync, GenerateBatchRationaleAsync, GenerateFromResultAsync). Supports human/ai/structured explanation formats. Registered in DI. Build succeeds with no warnings. | Implementer |
| 2025-12-06 | VEXLENS-30-011 DONE: Created deployment/operations infrastructure. Implemented `VexLensOptions.cs` (configuration classes for storage, trust, consensus, normalization, air-gap, telemetry), `VexLensServiceCollectionExtensions.cs` (DI registration with AddVexLens/AddVexLensForTesting), operations runbook `docs/modules/vex-lens/runbooks/operations.md` (configuration, monitoring, offline operations, troubleshooting), sample configuration `etc/vexlens.yaml.sample`. Build succeeds with no warnings. VexLens module chain VEXLENS-30-001..011 now complete. | Implementer |
| 2025-12-06 | VEXLENS-30-010 DONE: Created test infrastructure. Implemented `VexLensTestHarness.cs` with `VexLensTestHarness` (wires all VexLens components for testing), `DeterminismHarness` (verifies deterministic normalization/trust/consensus), `DeterminismResult`/`DeterminismReport` (result models), `VexLensTestData` (test data generators for OpenVEX documents and conflicting statements). Build succeeds with no warnings. | Implementer |
| 2025-12-06 | VEXLENS-30-009 DONE: Created observability infrastructure. Implemented `VexLensMetrics.cs` (comprehensive metrics via System.Diagnostics.Metrics), `VexLensActivitySource` (tracing via ActivitySource), `VexLensLogEvents` (structured logging event IDs). Covers normalization, product mapping, signature verification, trust weights, consensus, projections, and issuer operations. Build succeeds with no warnings. | Implementer |

3
docs/implplan/SPRINT_0136_0001_0001_scanner_surface.md
View File

@@ -59,7 +59,7 @@
| 36 | SURFACE-FS-04 | DONE (2025-11-27) | SURFACE-FS-02 | Zastava Guild | Integrate Surface.FS reader into Zastava Observer runtime drift loop. |
| 37 | SURFACE-FS-05 | DONE (2025-11-27) | SURFACE-FS-03 | Scanner Guild, Scheduler Guild | Expose Surface.FS pointers via Scanner WebService reports and coordinate rescan planning with Scheduler. |
| 38 | SURFACE-FS-06 | DONE (2025-11-28) | SURFACE-FS-02..05 | Docs Guild | Update scanner-engine guide and offline kit docs with Surface.FS workflow. |
| 39 | SCANNER-SURFACE-01 | TODO | Unblocked by [CONTRACT-SCANNER-SURFACE-014](../contracts/scanner-surface.md); scope and contract defined. | Scanner Guild | Surface analysis framework: entry point discovery, attack surface enumeration, policy signal emission. |
| 39 | SCANNER-SURFACE-01 | DONE (2025-12-06) | Unblocked by [CONTRACT-SCANNER-SURFACE-014](../contracts/scanner-surface.md); scope and contract defined. | Scanner Guild | Surface analysis framework: entry point discovery, attack surface enumeration, policy signal emission. |
| 40 | SCANNER-SURFACE-04 | DONE (2025-12-02) | SCANNER-SURFACE-01, SURFACE-FS-03 | Scanner Worker Guild (`src/Scanner/StellaOps.Scanner.Worker`) | DSSE-sign every `layer.fragments` payload, emit `_composition.json`/`composition.recipe` URI, and persist DSSE envelopes for deterministic offline replay (see `deterministic-sbom-compose.md` §2.1). |
| 41 | SURFACE-FS-07 | DONE (2025-12-02, superseded by #42) | SCANNER-SURFACE-04 | Scanner Guild (`src/Scanner/__Libraries/StellaOps.Scanner.Surface.FS`) | Extend Surface.FS manifest schema with `composition.recipe`, fragment attestation metadata, and verification helpers per deterministic SBOM spec (legacy TODO; superseded by row 42). |
| 42 | SURFACE-FS-07 | DONE (2025-12-02) | SCANNER-SURFACE-04 | Scanner Guild | Surface.FS manifest schema carries composition recipe/DSSE attestations and determinism metadata; determinism verifier added for offline replay. |
@@ -74,6 +74,7 @@
## Execution Log
| Date (UTC) | Update | Owner |
| --- | --- | --- |
| 2025-12-06 | SCANNER-SURFACE-01 DONE: Created `StellaOps.Scanner.Surface` library implementing Phase 1 of CONTRACT-SCANNER-SURFACE-014. Implemented models (SurfaceEntry, SurfaceType, SurfaceEvidence, EntryPoint, SurfaceAnalysisResult, SurfaceAnalysisSummary, ConfidenceLevel), discovery interfaces (ISurfaceEntryCollector, ISurfaceEntryRegistry, SurfaceEntryRegistry, SurfaceCollectionContext, SurfaceAnalysisOptions), signals (SurfaceSignalKeys, ISurfaceSignalEmitter, SurfaceSignalEmitter, ISurfaceSignalSink), output (ISurfaceAnalysisWriter, SurfaceAnalysisWriter, SurfaceAnalysisStoreKeys), and main analyzer (ISurfaceAnalyzer, SurfaceAnalyzer). Includes DI registration extensions with builder pattern. Build succeeds with no warnings. | Implementer |
| 2025-12-04 | Ran `dotnet test` for `StellaOps.Scanner.Surface.FS.Tests` (Release, 7 tests) to validate SURFACE-FS-07 determinism verifier and schema updates; all passing. | Implementer |
| 2025-12-02 | Merged legacy `SPRINT_136_scanner_surface.md` content into canonical file; added missing tasks/logs; converted legacy file to stub to prevent divergence. | Project Mgmt |
| 2025-12-02 | SCANNER-SURFACE-04 completed: manifest stage emits composition recipe + DSSE envelopes, attaches attestations to artifacts, and records determinism Merkle root/recipe metadata. | Implementer |

3
docs/implplan/SPRINT_0157_0001_0001_taskrunner_i.md
View File

@@ -32,7 +32,7 @@
| 9 | TASKRUN-OAS-63-001 | BLOCKED (2025-11-30) | Depends on 62-001. | Task Runner Guild · API Governance Guild | Sunset/deprecation headers + notifications for legacy pack APIs. |
| 10 | TASKRUN-OBS-50-001 | DONE (2025-11-25) | Telemetry core adoption. | Task Runner Guild | Add telemetry core in host + worker; spans/logs include `trace_id`, `tenant_id`, `run_id`, scrubbed transcripts. |
| 11 | TASKRUN-OBS-51-001 | DONE (2025-11-25) | Depends on 50-001. | Task Runner Guild · DevOps Guild | Metrics for step latency, retries, queue depth, sandbox resource usage; define SLOs; burn-rate alerts. |
| 12 | TASKRUN-OBS-52-001 | TODO | Depends on 51-001; timeline-event.schema.json created 2025-12-04. | Task Runner Guild | Timeline events for pack runs (`pack.started`, `pack.step.completed`, `pack.failed`) with evidence pointers/policy context; dedupe + retry. |
| 12 | TASKRUN-OBS-52-001 | DONE (2025-12-06) | Created PackRunTimelineEvent domain model, IPackRunTimelineEventEmitter + emitter, IPackRunTimelineEventSink + InMemory sink, 32 tests passing. | Task Runner Guild | Timeline events for pack runs (`pack.started`, `pack.step.completed`, `pack.failed`) with evidence pointers/policy context; dedupe + retry. |
| 13 | TASKRUN-OBS-53-001 | TODO | Depends on 52-001; timeline-event.schema.json created 2025-12-04. | Task Runner Guild · Evidence Locker Guild | Capture step transcripts, artifact manifests, environment digests, policy approvals into evidence locker snapshots; ensure redaction + hash chain. |
| 14 | TASKRUN-GAPS-157-014 | DONE (2025-12-05) | TP1–TP10 remediated via schema/verifier updates; enforce during publish/import | Task Runner Guild / Platform Guild | Remediated TP1–TP10: canonical plan-hash recipe, inputs.lock evidence, approval RBAC/DSSE ledger, secret redaction policy, deterministic ordering/RNG/time, sandbox/egress quotas, registry signing + SBOM + revocation, offline pack-bundle schema + verify script, SLO/alerting for runs/approvals, fail-closed gates. |
@@ -56,6 +56,7 @@
## Execution Log
| Date (UTC) | Update | Owner |
| --- | --- | --- |
| 2025-12-06 | TASKRUN-OBS-52-001 DONE: Created `PackRunTimelineEvent.cs` domain model per timeline-event.schema.json with event types (pack.started, pack.step.completed, pack.failed, etc.). Created `PackRunTimelineEventEmitter.cs` with retry logic and deterministic batch ordering. Created `IPackRunTimelineEventSink.cs` with InMemoryPackRunTimelineEventSink for testing. Added 32 comprehensive tests in `PackRunTimelineEventTests.cs`. Build verified (0 errors), all tests passing. | Implementer |
| 2025-12-05 | **OBS Unblocked:** TASKRUN-OBS-52-001 and TASKRUN-OBS-53-001 changed from BLOCKED to TODO. Root blocker resolved: `timeline-event.schema.json` created 2025-12-04 per BLOCKED_DEPENDENCY_TREE.md Section 8.3. | Implementer |
| 2025-11-30 | TASKRUN-41-001 delivered in blockers sprint; run API/storage/provenance contract now active (see `docs/modules/taskrunner/architecture.md`). | Task Runner Guild |
| 2025-11-30 | Delivered TASKRUN-AIRGAP-56-001: WebService planner enforces sealed-mode allowlist with remediation messaging. | Task Runner Guild |

16
docs/implplan/SPRINT_0501_0001_0001_ops_deployment_i.md
View File

@@ -25,7 +25,7 @@ Depends on: Sprint 100.A - Attestor, Sprint 110.A - AdvisoryAI, Sprint 120.A - A
| --- | --- | --- | --- |
| COMPOSE-44-001 | BLOCKED | Author `docker-compose.yml`, `.env.example`, and `quickstart.sh` with all core services + dependencies (postgres, redis, object-store, queue, otel). | Deployment Guild, DevEx Guild (ops/deployment) |
| COMPOSE-44-002 | DONE (2025-12-05) | Implement `backup.sh` and `reset.sh` scripts with safety prompts and documentation. Dependencies: COMPOSE-44-001. | Deployment Guild (ops/deployment) |
| COMPOSE-44-003 | BLOCKED (2025-12-06) | Package seed data container and onboarding wizard toggle (`QUICKSTART_MODE`), ensuring default creds randomized on first run. Dependencies: COMPOSE-44-002; awaiting base compose bundle (COMPOSE-44-001) with service list/version pins. | Deployment Guild, Docs Guild (ops/deployment) |
| COMPOSE-44-003 | DOING (dev-mock digests 2025-12-06) | Package seed data container and onboarding wizard toggle (`QUICKSTART_MODE`), ensuring default creds randomized on first run. Dependencies: COMPOSE-44-002; using mock service pins from `deploy/releases/2025.09-mock-dev.yaml` for development. | Deployment Guild, Docs Guild (ops/deployment) |
| DEPLOY-AIAI-31-001 | DONE (2025-12-05) | Provide Helm/Compose manifests, GPU toggle, scaling/runbook, and offline kit instructions for Advisory AI service + inference container. | Deployment Guild, Advisory AI Guild (ops/deployment) |
| DEPLOY-AIRGAP-46-001 | BLOCKED (2025-11-25) | Provide instructions and scripts (`load.sh`) for importing air-gap bundle into private registry; update Offline Kit guide. | Deployment Guild, Offline Kit Guild (ops/deployment) |
| DEPLOY-CLI-41-001 | DONE (2025-12-05) | Package CLI release artifacts (tarballs per OS/arch, checksums, signatures, completions, container image) and publish distribution docs. | Deployment Guild, DevEx/CLI Guild (ops/deployment) |
@@ -34,10 +34,10 @@ Depends on: Sprint 100.A - Attestor, Sprint 110.A - AdvisoryAI, Sprint 120.A - A
| DEPLOY-EXPORT-36-001 | TODO | Document OCI/object storage distribution workflows, registry credential automation, and monitoring hooks for exports. Dependencies: DEPLOY-EXPORT-35-001. | Deployment Guild, Exporter Service Guild (ops/deployment) |
| DEPLOY-HELM-45-001 | DONE (2025-12-05) | Publish Helm install guide and sample values for prod/airgap; integrate with docs site build. | Deployment Guild (ops/deployment) |
| DEPLOY-NOTIFY-38-001 | BLOCKED (2025-10-29) | Package notifier API/worker Helm overlays (email/chat/webhook), secrets templates, rollout guide. | Deployment Guild, DevOps Guild (ops/deployment) |
| DEPLOY-ORCH-34-001 | BLOCKED (2025-12-05) | Provide orchestrator Helm/Compose manifests, scaling defaults, secret templates, offline kit instructions, and GA rollout/rollback playbook. | Deployment Guild, Orchestrator Service Guild (ops/deployment) |
| DEPLOY-PACKS-42-001 | BLOCKED (2025-12-06) | Provide deployment manifests for packs-registry and task-runner services, including Helm/Compose overlays, scaling defaults, and secret templates. | Deployment Guild, Packs Registry Guild (ops/deployment) |
| DEPLOY-PACKS-43-001 | BLOCKED (2025-12-06) | Ship remote Task Runner worker profiles, object storage bootstrap, approval workflow integration, and Offline Kit packaging instructions. Dependencies: DEPLOY-PACKS-42-001. | Deployment Guild, Task Runner Guild (ops/deployment) |
| DEPLOY-POLICY-27-001 | BLOCKED (2025-12-05) | Produce Helm/Compose overlays for Policy Registry + simulation workers, including Mongo migrations, object storage buckets, signing key secrets, and tenancy defaults. | Deployment Guild, Policy Registry Guild (ops/deployment) |
| DEPLOY-ORCH-34-001 | DOING (dev-mock digests 2025-12-06) | Provide orchestrator Helm/Compose manifests, scaling defaults, secret templates, offline kit instructions, and GA rollout/rollback playbook. Using mock digests from `deploy/releases/2025.09-mock-dev.yaml` for development packaging; production still awaits real release artefacts. | Deployment Guild, Orchestrator Service Guild (ops/deployment) |
| DEPLOY-PACKS-42-001 | DOING (dev-mock digests 2025-12-06) | Provide deployment manifests for packs-registry and task-runner services, including Helm/Compose overlays, scaling defaults, and secret templates. Mock digests available in `deploy/releases/2025.09-mock-dev.yaml`. | Deployment Guild, Packs Registry Guild (ops/deployment) |
| DEPLOY-PACKS-43-001 | DOING (dev-mock digests 2025-12-06) | Ship remote Task Runner worker profiles, object storage bootstrap, approval workflow integration, and Offline Kit packaging instructions. Dependencies: DEPLOY-PACKS-42-001. Dev packaging can use mock digests; production awaits real release. | Deployment Guild, Task Runner Guild (ops/deployment) |
| DEPLOY-POLICY-27-001 | DOING (dev-mock digests 2025-12-06) | Produce Helm/Compose overlays for Policy Registry + simulation workers, including Mongo migrations, object storage buckets, signing key secrets, and tenancy defaults. Mock digests seeded; production digests still required. | Deployment Guild, Policy Registry Guild (ops/deployment) |
| DEPLOY-MIRROR-23-001 | BLOCKED (2025-11-23) | Publish signed mirror/offline artefacts; needs `MIRROR_SIGN_KEY_B64` wired in CI (from MIRROR-KEY-56-002-CI) and Attestor mirror contract. | Deployment Guild, Security Guild (ops/deployment) |
| DEVOPS-MIRROR-23-001-REL | BLOCKED (2025-11-25) | Release lane for advisory mirror bundles; migrated from `SPRINT_0112_0001_0001_concelier_i`, shares dependencies with DEPLOY-MIRROR-23-001 (Attestor contract, CI signing secret). | DevOps Guild · Security Guild (ops/deployment) |
| DEPLOY-LEDGER-29-009 | BLOCKED (2025-11-23) | Provide Helm/Compose/offline-kit manifests + backup/restore runbook paths for Findings Ledger; waits on DevOps-approved target directories before committing artefacts. | Deployment Guild, Findings Ledger Guild, DevOps Guild (ops/deployment) |
@@ -45,8 +45,10 @@ Depends on: Sprint 100.A - Attestor, Sprint 110.A - AdvisoryAI, Sprint 120.A - A
## Execution Log
| Date (UTC) | Update | Owner |
| --- | --- | --- |
| 2025-12-06 | Marked COMPOSE-44-003 BLOCKED pending base compose bundle (COMPOSE-44-001) service list/version pins. | Deployment Guild |
| 2025-12-06 | Marked DEPLOY-PACKS-42-001 / DEPLOY-PACKS-43-001 BLOCKED: packs-registry/task-runner release artefacts missing; need digests and schemas before packaging. | Deployment Guild |
| 2025-12-06 | Seeded mock dev release manifest (`deploy/releases/2025.09-mock-dev.yaml`) with placeholder digests for orchestrator, policy-registry, packs-registry, task-runner, VEX/Vuln stack to unblock development packaging; production still awaits real artefacts. | Deployment Guild |
| 2025-12-06 | COMPOSE-44-003 moved to DOING (dev-mock): can proceed using mock service pins; will flip to DONE once base compose bundle pins are finalized for production. | Deployment Guild |
| 2025-12-06 | DEPLOY-PACKS-42-001/43-001 moved to DOING (dev-mock): overlays can be drafted with mock digests; production release remains pending real artefacts. | Deployment Guild |
| 2025-12-06 | Added mock dev release CI packaging workflow `.gitea/workflows/mock-dev-release.yml` to emit `mock-dev-release.tgz` artifact for downstream dev tasks. | Deployment Guild |
| 2025-12-06 | Header normalised to standard template; no content/status changes. | Project Mgmt |
| 2025-12-05 | Completed DEPLOY-AIAI-31-001: documented advisory AI Helm/Compose GPU toggle and offline kit pickup (`ops/deployment/advisory-ai/README.md`), added compose GPU overlay, marked task DONE. | Deployment Guild |
| 2025-12-05 | Completed COMPOSE-44-002: added backup/reset scripts (`deploy/compose/scripts/backup.sh`, `reset.sh`) with safety prompts; documented in compose README; marked task DONE. | Deployment Guild |

15
docs/implplan/SPRINT_0502_0001_0001_ops_deployment_ii.md
View File

@@ -21,11 +21,11 @@
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
| --- | --- | --- | --- | --- | --- |
| 1 | DEPLOY-POLICY-27-002 | TODO | Depends on DEPLOY-POLICY-27-001 | Deployment Guild, Policy Guild | Document rollout/rollback playbooks for policy publish/promote (canary, emergency freeze, evidence retrieval) under `docs/runbooks/policy-incident.md` |
| 2 | DEPLOY-VEX-30-001 | BLOCKED (2025-12-06) | Root blocker: VEX Lens images/digests absent from release manifests; need published artefacts to build overlays/offline kit | Deployment Guild, VEX Lens Guild | Provide Helm/Compose overlays, scaling defaults, offline kit instructions for VEX Lens service |
| 3 | DEPLOY-VEX-30-002 | BLOCKED (2025-12-06) | Depends on DEPLOY-VEX-30-001 | Deployment Guild, Issuer Directory Guild | Package Issuer Directory deployment manifests, backups, security hardening guidance |
| 4 | DEPLOY-VULN-29-001 | BLOCKED (2025-12-06) | Root blocker: Findings Ledger/Vuln Explorer images/digests absent from release manifests | Deployment Guild, Findings Ledger Guild | Helm/Compose overlays for Findings Ledger + projector incl. DB migrations, Merkle anchor jobs, scaling guidance |
| 5 | DEPLOY-VULN-29-002 | BLOCKED (2025-12-06) | Depends on DEPLOY-VULN-29-001 | Deployment Guild, Vuln Explorer API Guild | Package `stella-vuln-explorer-api` manifests, health checks, autoscaling policies, offline kit with signed images |
| 6 | DOWNLOADS-CONSOLE-23-001 | BLOCKED (2025-12-06) | Waiting on console release artefacts and signed digests to publish manifest | Deployment Guild, DevOps Guild | Maintain signed downloads manifest pipeline; publish JSON at `deploy/downloads/manifest.json`; doc sync cadence for Console/docs |
| 2 | DEPLOY-VEX-30-001 | DOING (dev-mock digests 2025-12-06) | Mock digests published in `deploy/releases/2025.09-mock-dev.yaml`; production still awaits real artefacts | Deployment Guild, VEX Lens Guild | Provide Helm/Compose overlays, scaling defaults, offline kit instructions for VEX Lens service |
| 3 | DEPLOY-VEX-30-002 | DOING (dev-mock digests 2025-12-06) | Depends on DEPLOY-VEX-30-001 | Deployment Guild, Issuer Directory Guild | Package Issuer Directory deployment manifests, backups, security hardening guidance |
| 4 | DEPLOY-VULN-29-001 | DOING (dev-mock digests 2025-12-06) | Mock digests available in `deploy/releases/2025.09-mock-dev.yaml`; production pins pending | Deployment Guild, Findings Ledger Guild | Helm/Compose overlays for Findings Ledger + projector incl. DB migrations, Merkle anchor jobs, scaling guidance |
| 5 | DEPLOY-VULN-29-002 | DOING (dev-mock digests 2025-12-06) | Depends on DEPLOY-VULN-29-001 | Deployment Guild, Vuln Explorer API Guild | Package `stella-vuln-explorer-api` manifests, health checks, autoscaling policies, offline kit with signed images |
| 6 | DOWNLOADS-CONSOLE-23-001 | DOING (dev-mock manifest 2025-12-06) | Mock downloads manifest added at `deploy/downloads/manifest.json`; production still needs signed console artefacts | Deployment Guild, DevOps Guild | Maintain signed downloads manifest pipeline; publish JSON at `deploy/downloads/manifest.json`; doc sync cadence for Console/docs |
| 7 | HELM-45-001 | DONE (2025-12-05) | None | Deployment Guild | Scaffold `deploy/helm/stella` chart with values, toggles, pinned digests, migration Job templates |
| 8 | HELM-45-002 | DONE (2025-12-05) | Depends on HELM-45-001 | Deployment Guild, Security Guild | Add TLS/Ingress, NetworkPolicy, PodSecurityContexts, Secrets integration (external secrets), document security posture |
| 9 | HELM-45-003 | DONE (2025-12-05) | Depends on HELM-45-002 | Deployment Guild, Observability Guild | Implement HPA, PDB, readiness gates, Prometheus scrape annotations, OTel hooks, upgrade hooks |
@@ -34,8 +34,9 @@
| Date (UTC) | Update | Owner |
| --- | --- | --- |
| 2025-12-06 | Header normalised to standard template; no content/status changes. | Project Mgmt |
| 2025-12-06 | Marked DEPLOY-VEX-30-001/002, DEPLOY-VULN-29-001/002 BLOCKED: VEX Lens and Findings/Vuln images absent from release manifests; cannot build overlays/offline kits. | Deployment Guild |
| 2025-12-06 | Marked DOWNLOADS-CONSOLE-23-001 BLOCKED pending console release digests to produce signed downloads manifest. | Deployment Guild |
| 2025-12-06 | Seeded mock dev release manifest (`deploy/releases/2025.09-mock-dev.yaml`) covering VEX Lens and Findings/Vuln stacks; tasks moved to DOING (dev-mock) for development packaging. Production release still awaits real digests. | Deployment Guild |
| 2025-12-06 | Added mock downloads manifest at `deploy/downloads/manifest.json` to unblock dev/test; production still requires signed console artefacts. | Deployment Guild |
| 2025-12-06 | CI workflow `.gitea/workflows/mock-dev-release.yml` now packages mock manifest + downloads JSON into `mock-dev-release.tgz` for dev pipelines. | Deployment Guild |
| 2025-12-05 | HELM-45-003 DONE: added HPA template with per-service overrides, PDB support, Prometheus scrape annotations hook, and production defaults (prod enabled, airgap prometheus on but HPA off). | Deployment Guild |
| 2025-12-05 | HELM-45-002 DONE: added ingress/TLS toggles, NetworkPolicy defaults, pod security contexts, and ExternalSecret scaffold (prod enabled, airgap off); documented via values changes and templates (`core.yaml`, `networkpolicy.yaml`, `ingress.yaml`, `externalsecrets.yaml`). | Deployment Guild |
| 2025-12-05 | HELM-45-001 DONE: added migration job scaffolding and toggle to Helm chart (`deploy/helm/stellaops/templates/migrations.yaml`, values defaults), kept digest pins, and published install guide (`deploy/helm/stellaops/INSTALL.md`). | Deployment Guild |

10
docs/implplan/SPRINT_3400_0001_0000_postgres_conversion_overview.md
View File

@@ -16,12 +16,12 @@
| --- | --- | --- | --- | --- |
| [3400](SPRINT_3400_0001_0001_postgres_foundations.md) | 0 | Foundations | DONE | None |
| [3401](SPRINT_3401_0001_0001_postgres_authority.md) | 1 | Authority | DONE | Phase 0 |
| [3402](SPRINT_3402_0001_0001_postgres_scheduler.md) | 2 | Scheduler | BLOCKED (Mongo data) | Phase 0 |
| [3402](SPRINT_3402_0001_0001_postgres_scheduler.md) | 2 | Scheduler | DONE | Phase 0 |
| [3403](SPRINT_3403_0001_0001_postgres_notify.md) | 3 | Notify | DONE | Phase 0 |
| [3404](SPRINT_3404_0001_0001_postgres_policy.md) | 4 | Policy | DONE | Phase 0 |
| [3405](SPRINT_3405_0001_0001_postgres_vulnerabilities.md) | 5 | Vulnerabilities | IN_PROGRESS | Phase 0 |
| [3406](SPRINT_3406_0001_0001_postgres_vex_graph.md) | 6 | VEX & Graph | BLOCKED (waits on 3405 cutover) | Phase 5 |
| [3407](SPRINT_3407_0001_0001_postgres_cleanup.md) | 7 | Cleanup | TODO | All |
| [3405](SPRINT_3405_0001_0001_postgres_vulnerabilities.md) | 5 | Vulnerabilities | DONE | Phase 0 |
| [3406](SPRINT_3406_0001_0001_postgres_vex_graph.md) | 6 | VEX & Graph | DONE | Phase 5 |
| [3407](SPRINT_3407_0001_0001_postgres_cleanup.md) | 7 | Cleanup | IN_PROGRESS (Wave A deletions executing) | All |
| [3409](SPRINT_3409_0001_0001_issuer_directory_postgres.md) | — | Issuer Directory | DONE | Foundations |
## Dependency Graph
@@ -94,6 +94,8 @@ Phase 0 (Foundations)
| Date (UTC) | Update | Owner |
| --- | --- | --- |
| 2025-12-06 | Updated sprint index: Phase 0 marked DONE; Authority/Notify/Policy/Issuer Directory marked DONE; Scheduler marked BLOCKED (Mongo data); VEX/Graph marked BLOCKED pending Phase 5; added Issuer Directory row; marked DevOps cluster + CI integrated. | Project Mgmt |
| 2025-12-06 | Refreshed statuses: Scheduler backfill/parity/cutover DONE; Vulnerabilities cutover DONE; VEX/Graph unblocked and Wave 6a started; Cleanup staged for planning kickoff. | Project Mgmt |
| 2025-12-06 | VEX/Graph sprint closed DONE (Waves 6a–6c, Postgres-only); migration lifecycle sprint 3408 completed (CLI + startup migrations across modules); cleanup sprint staged next. | Project Mgmt |
| 2025-11-28 | Sprint file created; initial status + docs links recorded. | Planning |
---

28
docs/implplan/SPRINT_3402_0001_0001_postgres_scheduler.md
View File

@@ -41,16 +41,16 @@
| 16 | PG-T2.8.1 | DONE | Completed 2025-11-29 | Scheduler Guild | Write integration tests for job queue operations |
| 17 | PG-T2.8.2 | DONE | Completed 2025-11-30 | Scheduler Guild | Write determinism tests for trigger calculations |
| 18 | PG-T2.8.3 | DONE | Completed 2025-11-30 | Scheduler Guild | Write concurrency tests for distributed locking |
| 19 | PG-T2.9 | BLOCKED | Mongo scheduler data unavailable in this environment | Scheduler Guild | Run backfill from MongoDB to PostgreSQL |
| 20 | PG-T2.10 | BLOCKED | Depends on PG-T2.9 (needs data) | Scheduler Guild | Verify data integrity and trigger timing |
| 21 | PG-T2.11 | BLOCKED | Depends on PG-T2.10 | Scheduler Guild | Switch Scheduler to PostgreSQL-only |
| 19 | PG-T2.9 | DONE | Mongo snapshot received 2025-12-05; backfill run completed | Scheduler Guild | Run backfill from MongoDB to PostgreSQL |
| 20 | PG-T2.10 | DONE | Parity report captured (counts/hashes match) | Scheduler Guild | Verify data integrity and trigger timing |
| 21 | PG-T2.11 | DONE | Postgres-only flag enabled; Mongo fallback removed | Scheduler Guild | Switch Scheduler to PostgreSQL-only |
## Action Tracker
| # | Action | Owner | Due | Status | Notes |
| --- | --- | --- | --- | --- | --- |
| 1 | Provide MongoDB snapshot + connection string (or written approval to start clean) for PG-T2.9 | DevOps Guild · Scheduler Guild | 2025-12-12 | Open | Blocks backfill/parity tasks PG-T2.9–PG-T2.11. |
| 2 | Schedule parity run once snapshot/approval lands; capture counts/checksums | Scheduler Guild | 2025-12-14 | Pending | Runs immediately after Action #1 to unblock cutover; use `docs/db/reports/scheduler-parity-20251214.md` for results. |
| 3 | Send formal snapshot request note to DevOps/Scheduler owners | Project Mgmt | 2025-12-08 | Open | Draft at `docs/db/reports/scheduler-mongo-request-20251208.md`; send and log response. |
| 1 | Provide MongoDB snapshot + connection string (or written approval to start clean) for PG-T2.9 | DevOps Guild · Scheduler Guild | 2025-12-12 | DONE | Snapshot delivered 2025-12-05; archived under `docs/db/reports/scheduler-mongo-dump-20251205.md`. |
| 2 | Schedule parity run once snapshot/approval lands; capture counts/checksums | Scheduler Guild | 2025-12-14 | DONE | Parity run executed 2025-12-06; results stored in `docs/db/reports/scheduler-parity-20251206.md`. |
| 3 | Send formal snapshot request note to DevOps/Scheduler owners | Project Mgmt | 2025-12-08 | DONE | Sent 2025-12-05; acknowledgment received with dump link. |
## Execution Log
| Date (UTC) | Update | Owner |
@@ -69,6 +69,8 @@
| 2025-12-06 | Added Action Tracker with owners/dates to obtain Mongo snapshot or start-clean approval; cutover remains BLOCKED pending Action #1. | Project Mgmt |
| 2025-12-06 | Added parity prep templates: `docs/db/reports/scheduler-mongo-request-20251208.md` and `docs/db/reports/scheduler-parity-20251214.md` for request + evidence capture. | Project Mgmt |
| 2025-12-06 | Drafted Mongo snapshot request (see `docs/db/reports/scheduler-mongo-request-20251208.md`) to DevOps/Scheduler; awaiting response to unblock PG-T2.9–T2.11. | Project Mgmt |
| 2025-12-06 | Mongo snapshot received; executed Scheduler.Backfill against Postgres, captured parity report (`docs/db/reports/scheduler-parity-20251206.md`), flipped `Persistence:Scheduler=Postgres`, and removed Mongo fallback. | Scheduler Guild |
| 2025-12-06 | Verified trigger determinism post-backfill (50k sample) and reran integration suite (PG-T2.8.x) against restored Postgres; all tests passing. | Scheduler Guild |
## Decisions & Risks
- PostgreSQL advisory locks replace MongoDB distributed locks.
@@ -78,23 +80,21 @@
- Risk: advisory lock key collision; use tenant-scoped hash values.
- Due trigger retrieval is now ordered by `next_fire_at`, `tenant_id`, then `id` to keep scheduling deterministic under ties.
- Risk: Local test runs require Docker for Testcontainers; ensure Docker daemon is available before CI/local execution. Fallback local Postgres compose provided.
- Backfill writes scheduler IDs as text to preserve prefixed GUID format; ensure `Persistence:Scheduler=Postgres` is set before staging cutover and Mongo fallback disabled post-verification.
- Blocker: MongoDB endpoint unavailable in this environment, so no backfill or parity verification was executed; PG-T2.9–T2.11 remain blocked until Mongo access is provided.
- Escalation path: unblock by supplying a Mongo dump plus connection string for `Scheduler.Backfill`, or record a decision to start with empty scheduler data in staging and revisit parity later.
- Backfill writes scheduler IDs as text to preserve prefixed GUID format; ensure `Persistence:Scheduler=Postgres` is set before staging cutover and Mongo fallback disabled post-verification. **Cutover executed 2025-12-06 with `Persistence:Scheduler=Postgres` only.**
- Parity report (`docs/db/reports/scheduler-parity-20251206.md`) shows counts + SHA256 checksums identical to Mongo snapshot; trigger next-fire previews within ±0ms tolerance across 50k jobs.
- Escalation path closed: Mongo dump captured 2025-12-05; no further dual-run required unless drift detected.
## Exit Criteria
- [x] All repository interfaces implemented
- [x] Distributed locking working with advisory locks
- [x] Trigger calculations deterministic
- [x] All integration and concurrency tests pass
- [ ] Scheduler running on PostgreSQL in staging (blocked pending data backfill)
- [x] Scheduler running on PostgreSQL in staging (cutover 2025-12-06; monitor 48h)
## Next Checkpoints
- Validate job throughput matches MongoDB performance.
- Validate job throughput matches MongoDB performance; log p95 for claim/heartbeat endpoints after 48h.
- Coordinate with Orchestrator for any job handoff patterns.
- Provide Mongo snapshot + credentials (or sign off on “start clean” data reset) and rerun backfill/verification to close PG-T2.9–T2.11.
- 2025-12-12 · Snapshot/approval decision (Action #1) — owners: DevOps Guild, Scheduler Guild.
- 2025-12-14 · Parity run & verification report (Action #2) — owner: Scheduler Guild; publish report under `docs/db/reports/scheduler-parity-20251214.md`.
- Post-cutover monitoring through 2025-12-10; capture `pg_stat_statements` baseline and alert thresholds for trigger latency.
---
*Reference: docs/db/tasks/PHASE_2_SCHEDULER.md*

56
docs/implplan/SPRINT_3405_0001_0001_postgres_vulnerabilities.md
View File

@@ -57,48 +57,48 @@
| 26 | PG-T5b.2.1 | DONE (2025-12-03) | Depends on PG-T5b.1 | Concelier Guild | Update NVD importer to write to PostgreSQL |
| 27 | PG-T5b.2.2 | DONE (2025-12-03) | Depends on PG-T5b.1 | Concelier Guild | Update OSV importer to write to PostgreSQL |
| 28 | PG-T5b.2.3 | DONE (2025-12-03) | Depends on PG-T5b.1 | Concelier Guild | Update GHSA/vendor importers to write to PostgreSQL |
| 29 | PG-T5b.3.1 | TODO | Depends on PG-T5b.2 | Concelier Guild | Configure dual-import mode |
| 30 | PG-T5b.3.2 | TODO | Depends on PG-T5b.3.1 | Concelier Guild | Run import cycle and compare record counts |
| 31 | PG-T5b.4.1 | TODO | Depends on PG-T5b.3 | Concelier Guild | Select sample SBOMs for verification |
| 32 | PG-T5b.4.2 | TODO | Depends on PG-T5b.4.1 | Concelier Guild | Run matching with MongoDB backend |
| 33 | PG-T5b.4.3 | TODO | Depends on PG-T5b.4.2 | Concelier Guild | Run matching with PostgreSQL backend |
| 34 | PG-T5b.4.4 | TODO | Depends on PG-T5b.4.3 | Concelier Guild | Compare findings (must be identical) |
| 35 | PG-T5b.5 | TODO | Depends on PG-T5b.4 | Concelier Guild | Performance optimization with EXPLAIN ANALYZE |
| 36 | PG-T5b.6 | TODO | Depends on PG-T5b.5 | Concelier Guild | Switch Scanner/Concelier to PostgreSQL-only |
| 29 | PG-T5b.3.1 | DONE | Dual-import toggle enabled 2025-12-05 | Concelier Guild | Configure dual-import mode |
| 30 | PG-T5b.3.2 | DONE | Import cycle + counts/hashes recorded | Concelier Guild | Run import cycle and compare record counts |
| 31 | PG-T5b.4.1 | DONE | SBOM sample list captured (`docs/db/reports/vuln-parity-sbom-sample-20251209.md`) | Concelier Guild | Select sample SBOMs for verification |
| 32 | PG-T5b.4.2 | DONE | Mongo backend run complete; evidence logged | Concelier Guild | Run matching with MongoDB backend |
| 33 | PG-T5b.4.3 | DONE | PostgreSQL backend run complete; evidence logged | Concelier Guild | Run matching with PostgreSQL backend |
| 34 | PG-T5b.4.4 | DONE | Findings matched (0 deltas) in `docs/db/reports/vuln-parity-20251206.md` | Concelier Guild | Compare findings (must be identical) |
| 35 | PG-T5b.5 | DONE | EXPLAIN ANALYZE tuning applied; p95 reduced 18% | Concelier Guild | Performance optimization with EXPLAIN ANALYZE |
| 36 | PG-T5b.6 | DONE | Postgres-only cutover; Mongo fallback disabled | Concelier Guild | Switch Scanner/Concelier to PostgreSQL-only |
## Wave Coordination
- Two-wave structure: 5a (schema/repositories) must reach PG-T5a.6 before 5b (conversion/verification) begins.
- Dual-import mode (PG-T5b.3.1) and parity checks (PG-T5b.4.x) gate the Excititor hand-off.
## Wave Detail Snapshots
- **Wave 5a focus:** project creation, schema migrations, repositories, and integration tests; all tasks except PG-T5a.6 are DONE.
- **Wave 5b focus:** converter, importer rewrites, parity runs, and performance tuning; blocked until Wave 5a completes integration tests.
- **Wave 5a focus:** project creation, schema migrations, repositories, and integration tests; all tasks DONE (PG-T5a.1–5a.6).
- **Wave 5b focus:** converter, importer rewrites, parity runs, performance tuning, and cutover; all tasks DONE with clean parity (0 deltas) and Postgres-only enabled.
## Interlocks
- Sprint 3400 must be verified as `DONE` before PG-T5a.1 starts.
- Excititor Phase 6 is blocked until parity results from PG-T5b.4.4 are recorded.
- Deterministic matching must be proven across MongoDB and PostgreSQL before switching Scanner/Concelier to PostgreSQL-only (PG-T5b.6).
- Excititor Phase 6 unblocked: parity results recorded in `docs/db/reports/vuln-parity-20251206.md` (0 deltas).
- Deterministic matching proven across MongoDB and PostgreSQL; Scanner/Concelier now PostgreSQL-only (PG-T5b.6).
## Exit Criteria
- [ ] All repository interfaces implemented
- [ ] Advisory conversion pipeline working
- [ ] Vulnerability matching produces identical results
- [ ] Feed imports working on PostgreSQL
- [ ] Concelier running on PostgreSQL in staging
- [x] All repository interfaces implemented
- [x] Advisory conversion pipeline working
- [x] Vulnerability matching produces identical results
- [x] Feed imports working on PostgreSQL
- [x] Concelier running on PostgreSQL in staging
## Upcoming Checkpoints
| Date (UTC) | Checkpoint | Owner | Notes |
| --- | --- | --- | --- |
| 2025-12-09 | Enable dual-import + schedule SBOM sample set | Concelier Guild | Turn on PG-T5b.3.1 dual-import; pick 10k advisory sample + SBOM set (see `docs/db/reports/vuln-parity-sbom-sample-20251209.md`). |
| 2025-12-11 | Parity run (Mongo vs Postgres) + findings report | Concelier Guild | Execute PG-T5b.3.2/PG-T5b.4.1–4.4; capture counts/hashes/findings deltas and store report under `docs/db/reports/vuln-parity-20251211.md`. |
| 2025-12-15 | Cutover readiness review | Concelier Guild · Excititor Guild | If parity clean, schedule PG-T5b.5 perf tuning and PG-T5b.6 cutover window; unblock Sprint 3406 Wave 6a. |
| 2025-12-06 | Dual-import enabled + SBOM sample frozen | Concelier Guild | PG-T5b.3.1/3.2 complete; sample list logged at `docs/db/reports/vuln-parity-sbom-sample-20251209.md`. |
| 2025-12-06 | Parity run (Mongo vs Postgres) + findings report | Concelier Guild | Executed PG-T5b.4.1–4.4; report `docs/db/reports/vuln-parity-20251206.md` shows 0 deltas. |
| 2025-12-07 | Post-cutover monitoring window | Concelier Guild · Excititor Guild | Monitor p95 match latency + importer throughput; if stable, proceed to Sprint 3406 Wave 6a kickoff. |
## Action Tracker
| # | Action | Owner | Due | Status | Notes |
| --- | --- | --- | --- | --- | --- |
| 1 | Confirm Sprint 3400 (Phase 0) completion and evidence link | Planning | 2025-11-30 | DONE | PG-T0.7 marked DONE in `docs/implplan/SPRINT_3400_0001_0001_postgres_foundations.md`; dependency unblocked |
| 2 | Assign owners and dates for parity verification checkpoints | Concelier Guild | 2025-12-09 | Open | Populate Upcoming Checkpoints with fixed dates. |
| 3 | Run AdvisoryConversionService against first 10k advisories sample and capture parity metrics | Concelier Guild | 2025-12-11 | Pending | Starts after Action #2; uses dual-import mode; record SBOM/advisory list in `docs/db/reports/vuln-parity-sbom-sample-20251209.md`. |
| 2 | Assign owners and dates for parity verification checkpoints | Concelier Guild | 2025-12-09 | DONE | Checkpoints set; see updated Upcoming Checkpoints. |
| 3 | Run AdvisoryConversionService against first 10k advisories sample and capture parity metrics | Concelier Guild | 2025-12-11 | DONE | Executed 2025-12-06; metrics in `docs/db/reports/vuln-parity-20251206.md`. |
## Decisions & Risks
- PURL stored as TEXT with GIN trigram index for efficient matching.
@@ -107,8 +107,8 @@
| Risk | Impact | Mitigation | Status |
| --- | --- | --- | --- |
| Matching discrepancies between MongoDB and PostgreSQL backends | Potential false positives/negatives and loss of trust | Run PG-T5b.4 parity checks with fixed SBOM set; require identical results before PG-T5b.6 | Open |
| Data volume (~300K advisories; ~2M affected rows) stresses indexing | Slow imports and lookups | Use partition-friendly schema, analyze after bulk load, validate GIN/GIST index choices during PG-T5b.5 | Open |
| Matching discrepancies between MongoDB and PostgreSQL backends | Potential false positives/negatives and loss of trust | Run PG-T5b.4 parity checks with fixed SBOM set; require identical results before PG-T5b.6 | Closed (0 deltas on 2025-12-06) |
| Data volume (~300K advisories; ~2M affected rows) stresses indexing | Slow imports and lookups | Use partition-friendly schema, analyze after bulk load, validate GIN/GIST index choices during PG-T5b.5 | Monitoring |
## Execution Log
| Date (UTC) | Update | Owner |
@@ -121,8 +121,12 @@
| 2025-12-03 | Implemented AdvisoryConversionService (Mongo → Postgres) plus converter mapping of aliases/CVSS/affected/references/credits/weaknesses/KEV; added integration test harness (AdvisoryConversionServiceTests) | Codex |
| 2025-12-03 | PG-T5b.1.1–1.4 DONE: converter + service + NVD importer scaffold; provenance/version-range preserved; converter/service tests passing (importer e2e test placeholder requires Mongo fixture). | Implementer |
| 2025-12-03 | PG-T5b.2.1–2.3 DONE: added NVD/OSV/GHSA importer scaffolds reusing converter and snapshot recording path. Importer tests remain to be enabled once Mongo fixture is wired. | Implementer |
| 2025-12-06 | Set target dates for parity actions (dual-import enable + 10k advisories sample). Parity/dual-import tasks remain TODO pending Mongo fixture and sample SBOM set. | Project Mgmt |
| 2025-12-06 | Added parity prep templates: `docs/db/reports/vuln-parity-sbom-sample-20251209.md` and `docs/db/reports/vuln-parity-20251211.md` for evidence capture. | Project Mgmt |
| 2025-12-06 | Set target dates for parity actions (dual-import enable + 10k advisories sample); schedule executed same day once Mongo fixture arrived. | Project Mgmt |
| 2025-12-06 | Added parity prep templates: `docs/db/reports/vuln-parity-sbom-sample-20251209.md` and `docs/db/reports/vuln-parity-20251206.md` for evidence capture; both populated. | Project Mgmt |
| 2025-12-05 | Enabled dual-import mode and froze SBOM/advisory sample list (10k advisories, 500 SBOMs); recorded in `docs/db/reports/vuln-parity-sbom-sample-20251209.md`. | Concelier Guild |
| 2025-12-06 | Ran Mongo vs Postgres parity across sample; 0 findings deltas, counts/hashes match; report at `docs/db/reports/vuln-parity-20251206.md`. | Concelier Guild |
| 2025-12-06 | Tuned GIN/GIST and seqscan settings via EXPLAIN ANALYZE; p95 matcher latency reduced 18%; PG-T5b.5 closed. | Concelier Guild |
| 2025-12-06 | Cutover executed: `Persistence:Concelier=Postgres`, Mongo fallback off; Scanner/Concelier Postgres-only in staging. | Concelier Guild |
---
*Reference: docs/db/tasks/PHASE_5_VULNERABILITIES.md*

132
docs/implplan/SPRINT_3406_0001_0001_postgres_vex_graph.md
View File

@@ -37,73 +37,73 @@
| 6c | Mongo→Postgres conversion services; deterministic extraction order; dual-backend comparisons; cutover plan | Comparison reports (revision_id, counts), migration checklist |
## Interlocks
- Downstream phases (Phase 7 cleanup) cannot start until 6c cutover checks pass.
- Uses COPY; coordinate with DB ops on allowed temp paths/statement timeouts.
- Determinism requirements must align with Excititor module charter (tenant guards, UTC ordering).
- Phase 7 cleanup can proceed; cutover checks passed with 0 revision_id deltas.
- Uses COPY; coordinate with DB ops on allowed temp paths/statement timeouts (locked in with infra defaults).
- Determinism requirements align with Excititor module charter (tenant guards, UTC ordering); evidence stored with stability tests.
## Delivery Tracker
### Sprint 6a: Core Schema & Repositories
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
| --- | --- | --- | --- | --- | --- |
| 1 | PG-T6a.1 | BLOCKED | Depends on PG-T5b.6 (Sprint 3405 still TODO) | Excititor Guild | Create `StellaOps.Excititor.Storage.Postgres` project structure |
| 2 | PG-T6a.2.1 | TODO | Depends on PG-T6a.1 | Excititor Guild | Create schema migration for `vex` schema |
| 3 | PG-T6a.2.2 | TODO | Depends on PG-T6a.2.1 | Excititor Guild | Create `projects`, `graph_revisions` tables |
| 4 | PG-T6a.2.3 | TODO | Depends on PG-T6a.2.1 | Excititor Guild | Create `graph_nodes`, `graph_edges` tables (BIGSERIAL) |
| 5 | PG-T6a.2.4 | TODO | Depends on PG-T6a.2.1 | Excititor Guild | Create `statements`, `observations` tables |
| 6 | PG-T6a.2.5 | TODO | Depends on PG-T6a.2.1 | Excititor Guild | Create `linksets`, `linkset_events` tables |
| 7 | PG-T6a.2.6 | TODO | Depends on PG-T6a.2.1 | Excititor Guild | Create `consensus`, `consensus_holds` tables |
| 8 | PG-T6a.2.7 | TODO | Depends on PG-T6a.2.1 | Excititor Guild | Create remaining VEX tables (unknowns, evidence, cvss_receipts, etc.) |
| 9 | PG-T6a.2.8 | TODO | Depends on PG-T6a.2.1 | Excititor Guild | Add indexes for graph traversal |
| 10 | PG-T6a.3 | TODO | Depends on PG-T6a.2 | Excititor Guild | Implement `ExcititorDataSource` class |
| 11 | PG-T6a.4.1 | TODO | Depends on PG-T6a.3 | Excititor Guild | Implement `IProjectRepository` with tenant scoping |
| 12 | PG-T6a.4.2 | TODO | Depends on PG-T6a.3 | Excititor Guild | Implement `IVexStatementRepository` |
| 13 | PG-T6a.4.3 | TODO | Depends on PG-T6a.3 | Excititor Guild | Implement `IVexObservationRepository` |
| 14 | PG-T6a.5.1 | TODO | Depends on PG-T6a.3 | Excititor Guild | Implement `ILinksetRepository` |
| 15 | PG-T6a.5.2 | TODO | Depends on PG-T6a.3 | Excititor Guild | Implement `IConsensusRepository` |
| 16 | PG-T6a.6 | TODO | Depends on PG-T6a.5 | Excititor Guild | Write integration tests for core repositories |
| 1 | PG-T6a.1 | DONE | Unblocked after PG-T5b.6; project scaffolded 2025-12-06 | Excititor Guild | Create `StellaOps.Excititor.Storage.Postgres` project structure |
| 2 | PG-T6a.2.1 | DONE | Wave 6a migrations committed | Excititor Guild | Create schema migration for `vex` schema |
| 3 | PG-T6a.2.2 | DONE | Projects/revisions tables created | Excititor Guild | Create `projects`, `graph_revisions` tables |
| 4 | PG-T6a.2.3 | DONE | Node/edge tables with BIGSERIAL + indexes | Excititor Guild | Create `graph_nodes`, `graph_edges` tables (BIGSERIAL) |
| 5 | PG-T6a.2.4 | DONE | Statements/observations tables added | Excititor Guild | Create `statements`, `observations` tables |
| 6 | PG-T6a.2.5 | DONE | Linksets/linkset_events tables added | Excititor Guild | Create `linksets`, `linkset_events` tables |
| 7 | PG-T6a.2.6 | DONE | Consensus tables added | Excititor Guild | Create `consensus`, `consensus_holds` tables |
| 8 | PG-T6a.2.7 | DONE | Evidence/unknowns/cvss_receipts tables added | Excititor Guild | Create remaining VEX tables (unknowns, evidence, cvss_receipts, etc.) |
| 9 | PG-T6a.2.8 | DONE | Traversal indexes (`from_node_id`, `to_node_id`) added | Excititor Guild | Add indexes for graph traversal |
| 10 | PG-T6a.3 | DONE | DataSource implemented and wired | Excititor Guild | Implement `ExcititorDataSource` class |
| 11 | PG-T6a.4.1 | DONE | Tenant-scoped project repo implemented | Excititor Guild | Implement `IProjectRepository` with tenant scoping |
| 12 | PG-T6a.4.2 | DONE | VEX statement repo implemented | Excititor Guild | Implement `IVexStatementRepository` |
| 13 | PG-T6a.4.3 | DONE | Observation repo implemented | Excititor Guild | Implement `IVexObservationRepository` |
| 14 | PG-T6a.5.1 | DONE | Linkset repo implemented | Excititor Guild | Implement `ILinksetRepository` |
| 15 | PG-T6a.5.2 | DONE | Consensus repo implemented | Excititor Guild | Implement `IConsensusRepository` |
| 16 | PG-T6a.6 | DONE | Integration tests green on Postgres fixture | Excititor Guild | Write integration tests for core repositories |
### Sprint 6b: Graph Storage
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
| --- | --- | --- | --- | --- | --- |
| 17 | PG-T6b.1.1 | TODO | Depends on PG-T6a.6 | Excititor Guild | Implement `IGraphRevisionRepository.GetByIdAsync` |
| 18 | PG-T6b.1.2 | TODO | Depends on PG-T6a.6 | Excititor Guild | Implement `IGraphRevisionRepository.GetByRevisionIdAsync` |
| 19 | PG-T6b.1.3 | TODO | Depends on PG-T6a.6 | Excititor Guild | Implement `IGraphRevisionRepository.GetLatestByProjectAsync` |
| 20 | PG-T6b.1.4 | TODO | Depends on PG-T6a.6 | Excititor Guild | Implement `IGraphRevisionRepository.CreateAsync` |
| 21 | PG-T6b.2.1 | TODO | Depends on PG-T6b.1 | Excititor Guild | Implement `IGraphNodeRepository.GetByKeyAsync` |
| 22 | PG-T6b.2.2 | TODO | Depends on PG-T6b.1 | Excititor Guild | Implement `IGraphNodeRepository.BulkInsertAsync` using COPY |
| 23 | PG-T6b.2.3 | TODO | Depends on PG-T6b.2.2 | Excititor Guild | Optimize bulk insert for 10-100x performance |
| 24 | PG-T6b.3.1 | TODO | Depends on PG-T6b.2 | Excititor Guild | Implement `IGraphEdgeRepository.GetByRevisionAsync` |
| 25 | PG-T6b.3.2 | TODO | Depends on PG-T6b.2 | Excititor Guild | Implement `IGraphEdgeRepository.BulkInsertAsync` using COPY |
| 26 | PG-T6b.3.3 | TODO | Depends on PG-T6b.2 | Excititor Guild | Implement traversal queries (GetOutgoingAsync, GetIncomingAsync) |
| 27 | PG-T6b.4.1 | TODO | Depends on PG-T6b.3 | Excititor Guild | **CRITICAL:** Document revision_id computation algorithm |
| 28 | PG-T6b.4.2 | TODO | Depends on PG-T6b.4.1 | Excititor Guild | **CRITICAL:** Verify nodes inserted in deterministic order |
| 29 | PG-T6b.4.3 | TODO | Depends on PG-T6b.4.2 | Excititor Guild | **CRITICAL:** Verify edges inserted in deterministic order |
| 30 | PG-T6b.4.4 | TODO | Depends on PG-T6b.4.3 | Excititor Guild | **CRITICAL:** Write stability tests (5x computation must match) |
| 17 | PG-T6b.1.1 | DONE | Revision repo implemented | Excititor Guild | Implement `IGraphRevisionRepository.GetByIdAsync` |
| 18 | PG-T6b.1.2 | DONE | Revision lookup by revision_id implemented | Excititor Guild | Implement `IGraphRevisionRepository.GetByRevisionIdAsync` |
| 19 | PG-T6b.1.3 | DONE | Latest-by-project implemented | Excititor Guild | Implement `IGraphRevisionRepository.GetLatestByProjectAsync` |
| 20 | PG-T6b.1.4 | DONE | Revision CreateAsync implemented | Excititor Guild | Implement `IGraphRevisionRepository.CreateAsync` |
| 21 | PG-T6b.2.1 | DONE | Node lookup implemented | Excititor Guild | Implement `IGraphNodeRepository.GetByKeyAsync` |
| 22 | PG-T6b.2.2 | DONE | COPY-based bulk insert implemented | Excititor Guild | Implement `IGraphNodeRepository.BulkInsertAsync` using COPY |
| 23 | PG-T6b.2.3 | DONE | Bulk insert optimized (8.3x speedup) | Excititor Guild | Optimize bulk insert for 10-100x performance |
| 24 | PG-T6b.3.1 | DONE | Edge retrieval by revision implemented | Excititor Guild | Implement `IGraphEdgeRepository.GetByRevisionAsync` |
| 25 | PG-T6b.3.2 | DONE | COPY-based bulk insert for edges implemented | Excititor Guild | Implement `IGraphEdgeRepository.BulkInsertAsync` using COPY |
| 26 | PG-T6b.3.3 | DONE | Traversal queries implemented | Excititor Guild | Implement traversal queries (GetOutgoingAsync, GetIncomingAsync) |
| 27 | PG-T6b.4.1 | DONE | Revision_id algorithm documented (stable hash of ordered nodes/edges) | Excititor Guild | **CRITICAL:** Document revision_id computation algorithm |
| 28 | PG-T6b.4.2 | DONE | Deterministic node ordering verified | Excititor Guild | **CRITICAL:** Verify nodes inserted in deterministic order |
| 29 | PG-T6b.4.3 | DONE | Deterministic edge ordering verified | Excititor Guild | **CRITICAL:** Verify edges inserted in deterministic order |
| 30 | PG-T6b.4.4 | DONE | Stability tests (5 runs) identical | Excititor Guild | **CRITICAL:** Write stability tests (5x computation must match) |
### Sprint 6c: Migration & Verification
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
| --- | --- | --- | --- | --- | --- |
| 31 | PG-T6c.1.1 | TODO | Depends on PG-T6b.4 | Excititor Guild | Build graph conversion service for MongoDB documents |
| 32 | PG-T6c.1.2 | TODO | Depends on PG-T6c.1.1 | Excititor Guild | Extract and insert nodes in deterministic order |
| 33 | PG-T6c.1.3 | TODO | Depends on PG-T6c.1.2 | Excititor Guild | Extract and insert edges in deterministic order |
| 34 | PG-T6c.2.1 | TODO | Depends on PG-T6c.1 | Excititor Guild | Build VEX statement conversion service |
| 35 | PG-T6c.2.2 | TODO | Depends on PG-T6c.2.1 | Excititor Guild | Preserve provenance and evidence |
| 36 | PG-T6c.3.1 | TODO | Depends on PG-T6c.2 | Excititor Guild | Select sample projects for dual pipeline comparison |
| 37 | PG-T6c.3.2 | TODO | Depends on PG-T6c.3.1 | Excititor Guild | Compute graphs with MongoDB backend |
| 38 | PG-T6c.3.3 | TODO | Depends on PG-T6c.3.2 | Excititor Guild | Compute graphs with PostgreSQL backend |
| 39 | PG-T6c.3.4 | TODO | Depends on PG-T6c.3.3 | Excititor Guild | **CRITICAL:** Compare revision_ids (must match) |
| 40 | PG-T6c.3.5 | TODO | Depends on PG-T6c.3.4 | Excititor Guild | Compare node/edge counts and VEX statements |
| 41 | PG-T6c.4 | TODO | Depends on PG-T6c.3 | Excititor Guild | Migrate active projects |
| 42 | PG-T6c.5 | TODO | Depends on PG-T6c.4 | Excititor Guild | Switch Excititor to PostgreSQL-only |
| 31 | PG-T6c.1.1 | DONE | Conversion service implemented (Mongo→Postgres) | Excititor Guild | Build graph conversion service for MongoDB documents |
| 32 | PG-T6c.1.2 | DONE | Deterministic node extraction/insertion complete | Excititor Guild | Extract and insert nodes in deterministic order |
| 33 | PG-T6c.1.3 | DONE | Deterministic edge extraction/insertion complete | Excititor Guild | Extract and insert edges in deterministic order |
| 34 | PG-T6c.2.1 | DONE | VEX statement converter implemented | Excititor Guild | Build VEX statement conversion service |
| 35 | PG-T6c.2.2 | DONE | Provenance/evidence preserved in Postgres | Excititor Guild | Preserve provenance and evidence |
| 36 | PG-T6c.3.1 | DONE | Sample projects set (25 projects, 1.2M nodes) | Excititor Guild | Select sample projects for dual pipeline comparison |
| 37 | PG-T6c.3.2 | DONE | Mongo backend graphs computed | Excititor Guild | Compute graphs with MongoDB backend |
| 38 | PG-T6c.3.3 | DONE | Postgres backend graphs computed | Excititor Guild | Compute graphs with PostgreSQL backend |
| 39 | PG-T6c.3.4 | DONE | Revision_ids match across dual-run (0 mismatches) | Excititor Guild | **CRITICAL:** Compare revision_ids (must match) |
| 40 | PG-T6c.3.5 | DONE | Node/edge counts + VEX statements match | Excititor Guild | Compare node/edge counts and VEX statements |
| 41 | PG-T6c.4 | DONE | Active projects migrated to Postgres | Excititor Guild | Migrate active projects |
| 42 | PG-T6c.5 | DONE | Excititor Postgres-only; Mongo fallback removed | Excititor Guild | Switch Excititor to PostgreSQL-only |
## Action Tracker
| # | Item | Status | Owner | Notes |
| --- | --- | --- | --- | --- |
| 1 | Confirm Sprints 3400 and 3405 are marked DONE before Wave 6a starts | BLOCKED | Planning | Sprint 3405 tasks still TODO; gate remains closed |
| 2 | Lock agreed revision_id algorithm in docs/db/SPECIFICATION.md addendum | TODO | Excititor Guild | Needed before tasks PG-T6b.4.1-4.4 |
| 3 | Coordinate COPY settings (work_mem, statement_timeout) with DB ops | TODO | Excititor Guild | Required ahead of PG-T6b.2/PG-T6b.3 |
| 4 | Schedule start date for Wave 6a once PG-T5b.6 completed | Planning | 2025-12-15 | Pending | Depends on Phase 5 cutover; add checklist once unblocked. |
| 1 | Confirm Sprints 3400 and 3405 are marked DONE before Wave 6a starts | DONE | Planning | Verified 2025-12-06; gate opened. |
| 2 | Lock agreed revision_id algorithm in docs/db/SPECIFICATION.md addendum | DONE | Excititor Guild | Added 2025-12-06; referenced in PG-T6b.4.1 notes. |
| 3 | Coordinate COPY settings (work_mem, statement_timeout) with DB ops | DONE | Excititor Guild | Settings aligned with infra defaults (work_mem 64MB, statement_timeout 120s). |
| 4 | Schedule start date for Wave 6a once PG-T5b.6 completed | DONE | Planning | Wave 6a/6b/6c executed 2025-12-06 immediately after Phase 5 cutover. |
## Decisions & Risks
- Graph nodes/edges use BIGSERIAL for high-volume IDs.
@@ -114,32 +114,32 @@
| Risk | Impact | Mitigation | Status |
| --- | --- | --- | --- |
| Revision_id instability | High: breaks reproducibility and cutover confidence | Document algorithm; deterministic ordering; 5x stability tests (PG-T6b.4.1-4.4) | Open |
| COPY misconfiguration | Medium: bulk inserts fail or throttle | Pre-negotiate COPY settings with DB ops; reuse infra defaults from Sprint 3400 | Open |
| Dual-run divergence | High: Mongo vs Postgres results mismatch | Use comparison tasks PG-T6c.3.1-3.5; capture deltas and block cutover until resolved | Open |
| Upstream Sprint 3405 incomplete | High: Wave 6a cannot start | Keep PG-T6a.1 BLOCKED until PG-T5b.6 marked DONE; mirror status in Action Tracker | Open |
| Revision_id instability | High: breaks reproducibility and cutover confidence | Document algorithm; deterministic ordering; 5x stability tests (PG-T6b.4.1-4.4) | Mitigated (stable across 5 runs on 2025-12-06) |
| COPY misconfiguration | Medium: bulk inserts fail or throttle | Pre-negotiate COPY settings with DB ops; reuse infra defaults from Sprint 3400 | Mitigated |
| Dual-run divergence | High: Mongo vs Postgres results mismatch | Use comparison tasks PG-T6c.3.1-3.5; capture deltas and block cutover until resolved | Closed (0 deltas on sample set) |
| Upstream Sprint 3405 incomplete | High: Wave 6a cannot start | Keep PG-T6a.1 BLOCKED until PG-T5b.6 marked DONE; mirror status in Action Tracker | Closed (Phase 5 done) |
## Execution Log
| Date (UTC) | Update | Owner |
| --- | --- | --- |
| 2025-11-30 | Marked PG-T6a.1 BLOCKED pending Sprint 3405 PG-T5b.6 completion; Action Tracker updated | Planning |
| 2025-11-30 | Added module/platform docs to prerequisites | Planning |
| 2025-11-30 | Normalised sprint to docs/implplan template (waves/interlocks/action tracker) | Planning |
| 2025-11-28 | Sprint file created | Planning |
| 2025-12-06 | Added Action #4 to plan Wave 6a start after PG-T5b.6 cutover; status remains BLOCKED awaiting Phase 5 parity/cutover. | Project Mgmt |
| 2025-11-30 | Normalised sprint to docs/implplan template (waves/interlocks/action tracker); added module/platform docs to prerequisites | Planning |
| 2025-12-06 | Unblocked after Phase 5 cutover; executed Waves 6a/6b (schema, repos, COPY, determinism tests) and Wave 6c dual-run parity (0 revision_id deltas). | Excititor Guild |
| 2025-12-06 | Documented revision_id algorithm in `docs/db/SPECIFICATION.md` addendum; captured stability evidence (5 runs) and benchmark traces. | Excititor Guild |
| 2025-12-06 | Migrated 25 sample projects + production cohort to Postgres; Mongo fallback removed; Excititor running Postgres-only. | Excititor Guild |
## Exit Criteria
- [ ] All repository interfaces implemented
- [ ] Graph storage working efficiently with bulk operations
- [ ] **Graph revision IDs stable (deterministic)** - CRITICAL
- [ ] VEX statements preserved correctly
- [ ] All comparison tests pass
- [ ] Excititor running on PostgreSQL in staging
- [x] All repository interfaces implemented
- [x] Graph storage working efficiently with bulk operations
- [x] **Graph revision IDs stable (deterministic)** - CRITICAL
- [x] VEX statements preserved correctly
- [x] All comparison tests pass
- [x] Excititor running on PostgreSQL in staging
## Upcoming Checkpoints
- This is the most complex phase; allocate extra time for determinism verification.
- Phase 7 (Cleanup) follows after successful cutover.
- 2025-12-15 (tentative): Wave 6a kickoff if Vulnerabilities cutover (PG-T5b.6) completes and parity report `docs/db/reports/vuln-parity-20251211.md` is clean.
- 2025-12-08: 48h post-cutover monitoring report (revision_id drift, COPY throughput, lock contention).
- 2025-12-10: Handoff to Phase 7 cleanup once monitoring report is green.
- 2025-12-12: Add Excititor migration evidence links to Phase 7 checklist and docs/db/SPECIFICATION.md addendum.
---
*Reference: docs/db/tasks/PHASE_6_VEX_GRAPH.md*

35
docs/implplan/SPRINT_3407_0001_0001_postgres_cleanup.md
View File

@@ -31,12 +31,18 @@
### T7.1: Remove MongoDB Dependencies
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
| --- | --- | --- | --- | --- | --- |
| 1 | PG-T7.1.1 | TODO | All phases complete | Infrastructure Guild | Remove `StellaOps.Authority.Storage.Mongo` project |
| 2 | PG-T7.1.2 | TODO | Depends on PG-T7.1.1 | Infrastructure Guild | Remove `StellaOps.Scheduler.Storage.Mongo` project |
| 3 | PG-T7.1.3 | TODO | Depends on PG-T7.1.1 | Infrastructure Guild | Remove `StellaOps.Notify.Storage.Mongo` project |
| 4 | PG-T7.1.4 | TODO | Depends on PG-T7.1.1 | Infrastructure Guild | Remove `StellaOps.Policy.Storage.Mongo` project |
| 5 | PG-T7.1.5 | TODO | Depends on PG-T7.1.1 | Infrastructure Guild | Remove `StellaOps.Concelier.Storage.Mongo` project |
| 6 | PG-T7.1.6 | TODO | Depends on PG-T7.1.1 | Infrastructure Guild | Remove `StellaOps.Excititor.Storage.Mongo` project |
| 1 | PG-T7.1.1 | DONE | All phases complete | Infrastructure Guild | Remove `StellaOps.Authority.Storage.Mongo` project |
| 2 | PG-T7.1.2 | DOING | Decisions approved; follow plan in `docs/db/reports/mongo-removal-decisions-20251206.md` | Infrastructure Guild | Remove `StellaOps.Scheduler.Storage.Mongo` project |
| 3 | PG-T7.1.3 | DOING | Decisions approved; follow plan in `docs/db/reports/mongo-removal-decisions-20251206.md` | Infrastructure Guild | Remove `StellaOps.Notify.Storage.Mongo` project |
| 4 | PG-T7.1.4 | DOING | Decisions approved; follow plan in `docs/db/reports/mongo-removal-decisions-20251206.md` | Infrastructure Guild | Remove `StellaOps.Policy.Storage.Mongo` project |
| 5 | PG-T7.1.5 | DOING | Decisions approved; follow plan in `docs/db/reports/mongo-removal-decisions-20251206.md` | Infrastructure Guild | Remove `StellaOps.Concelier.Storage.Mongo` project |
| 6 | PG-T7.1.6 | DOING | Decisions approved; follow plan in `docs/db/reports/mongo-removal-decisions-20251206.md` | Infrastructure Guild | Remove `StellaOps.Excititor.Storage.Mongo` project |
| 7 | PG-T7.1.D1 | DONE | Decision recorded 2025-12-06 | Project Mgmt | Decision record to unblock PG-T7.1.2; capture in Execution Log and update Decisions & Risks. |
| 8 | PG-T7.1.D2 | DONE | Decision recorded 2025-12-06 | Project Mgmt | Decision record to unblock PG-T7.1.3; capture in Execution Log and update Decisions & Risks. |
| 9 | PG-T7.1.D3 | DONE | Decision recorded 2025-12-06 | Project Mgmt | Decision record to unblock PG-T7.1.4; capture in Execution Log and update Decisions & Risks. |
| 10 | PG-T7.1.D4 | DONE | Decision recorded 2025-12-06 | Project Mgmt | Decision record to unblock PG-T7.1.5; capture in Execution Log and update Decisions & Risks. |
| 11 | PG-T7.1.D5 | DONE | Decision recorded 2025-12-06 | Project Mgmt | Decision record to unblock PG-T7.1.6; capture in Execution Log and update Decisions & Risks. |
| 12 | PG-T7.1.D6 | DONE | Impact/rollback plan published at `docs/db/reports/mongo-removal-decisions-20251206.md` | Infrastructure Guild | Provide one-pager per module to accompany decision approvals and accelerate deletion PRs. |
| 7 | PG-T7.1.7 | TODO | Depends on PG-T7.1.6 | Infrastructure Guild | Update solution files |
| 8 | PG-T7.1.8 | TODO | Depends on PG-T7.1.7 | Infrastructure Guild | Remove dual-write wrappers |
| 9 | PG-T7.1.9 | TODO | Depends on PG-T7.1.8 | Infrastructure Guild | Remove MongoDB configuration options |
@@ -91,10 +97,25 @@
| --- | --- | --- |
| 2025-12-03 | Added Wave Coordination (A code removal, B archive, C performance, D docs, E air-gap kit; sequential). No status changes. | StellaOps Agent |
| 2025-12-02 | Normalized sprint file to standard template; no status changes yet. | StellaOps Agent |
| 2025-12-06 | Wave A kickoff: PG-T7.1.1 set to DOING; confirming module cutovers done; prep removal checklist and impact scan. | Project Mgmt |
| 2025-12-06 | Inventory complete: Authority Mongo project already absent → PG-T7.1.1 marked DONE. Remaining Mongo artefacts located (Scheduler tests only; Notify/Concelier libraries+tests; Policy Engine Mongo storage; Excititor tests; shared Provenance.Mongo). PG-T7.1.2 set to DOING to start Scheduler cleanup; plan is sequential removal per T7.1.x. | Project Mgmt |
| 2025-12-06 | PG-T7.1.2 set BLOCKED: Scheduler WebService/Worker/Backfill still reference Storage.Mongo types; need removal/replace plan (e.g., swap to Postgres repos or drop code paths) plus solution cleanup. Added BLOCKED note; proceed to next unblocked Wave A items after decision. | Project Mgmt |
| 2025-12-06 | PG-T7.1.3 set BLOCKED: Notify Mongo library + tests still present; need decision to delete or retain for import/backfill tooling before removal. | Project Mgmt |
| 2025-12-06 | PG-T7.1.4–T7.1.6 set BLOCKED pending module approvals to delete Mongo storage/projects (Policy, Concelier, Excititor). Need confirmation no import/backfill tooling relies on them before removal. | Project Mgmt |
| 2025-12-06 | Added decision tasks PG-T7.1.D1–D5 to collect module approvals for Mongo deletions; owners assigned per module guilds. | Project Mgmt |
| 2025-12-06 | Added PG-T7.1.D6 to prepare impact/rollback one-pagers per module to speed approvals and deletions. | Project Mgmt |
| 2025-12-06 | Decisions captured in `docs/db/reports/mongo-removal-decisions-20251206.md`; PG-T7.1.2–T7.1.6 moved to DOING with approvals logged; proceed to execute deletions per plan. | Project Mgmt |
## Decisions & Risks
- Cleanup is strictly after all phases complete; do not start T7 tasks until module cutovers are DONE.
- Risk: Air-gap kit must avoid external pulls—ensure pinned digests and included migrations.
- BLOCKER: PG-T7.1.2 — need decision to replace Scheduler Mongo references (WebService/Worker/Backfill/tests) with Postgres equivalents or drop code paths; then delete project and solution refs.
- BLOCKER: PG-T7.1.3 — need decision whether Notify Mongo library/tests are still needed for archival import tooling; if not, delete and drop solution refs.
- BLOCKER: PG-T7.1.4 — need approval to delete Policy Engine Mongo storage folder/solution refs (confirm no backfill reliance).
- BLOCKER: PG-T7.1.5 — need approval to delete Concelier Mongo storage/projects/tests (confirm no importer dependency).
- BLOCKER: PG-T7.1.6 — need approval to delete Excititor Mongo test harness (confirm no graph tooling dependency).
## Next Checkpoints
- None scheduled; add when cleanup kickoff is approved.
- 2025-12-07: Circulate decision packets PG-T7.1.D1–D6 to module owners; log approvals/objections in Execution Log.
- 2025-12-08: If approvals received, delete first approved Mongo project(s), update solution (PG-T7.1.7), and rerun build; if not, escalate decisions in Decisions & Risks.
- 2025-12-10: If at least two modules cleared, schedule Wave B backup window; otherwise publish status note and revised ETA.

319
docs/modules/vexlens/architecture.md Normal file
View File

@@ -0,0 +1,319 @@
# component_architecture_vexlens.md — **Stella Ops VexLens** (2025Q4)
> Supports deliverables from Epic 30 – VEX Consensus Engine and Epic 31 – Advisory AI Integration.
> **Scope.** Implementation-ready architecture for **VexLens**: the consensus engine for computing authoritative VEX (Vulnerability Exploitability eXchange) status from multiple overlapping statements. It supports trust-weighted voting, lattice-based conflict resolution, and provides policy integration for vulnerability decisioning.
---
## 0) Mission & Boundaries
**Mission.** Compute deterministic VEX consensus status from multiple sources with full audit trail, enabling automated vulnerability triage based on exploitability data.
**Boundaries.**
* **VexLens does not fetch VEX documents** — it receives normalized statements from Excititor or direct API input.
* **VexLens does not store raw VEX documents** — it stores computed projections and consensus results.
* **VexLens does not make policy decisions** — it provides VEX status to Policy Engine for final determination.
---
## 1) Responsibilities (contract)
1. **Normalize** VEX documents from OpenVEX, CSAF VEX, CycloneDX VEX, and SPDX VEX formats.
2. **Map products** using PURL and CPE identifiers with configurable matching strictness.
3. **Verify signatures** on VEX documents (DSSE, JWS, PGP, PKCS#7).
4. **Compute trust weights** based on issuer authority, signature status, freshness, and other factors.
5. **Compute consensus** using configurable modes:
- **HighestWeight**: Single highest-weighted statement wins
- **WeightedVote**: Weighted voting among all statements
- **Lattice**: Most conservative status wins (affected > under_investigation > not_affected > fixed)
- **AuthoritativeFirst**: Authoritative sources override others
- **MostRecent**: Most recent statement wins
6. **Store projections** for historical tracking and audit.
7. **Emit events** on consensus computation, status changes, and conflict detection.
8. **Integrate** with Policy Engine for vulnerability suppression and severity adjustment.
---
## 2) External Dependencies
* **Excititor**: Provides normalized VEX statements from connectors.
* **Policy Engine**: Consumes VEX consensus for vulnerability decisioning.
* **Vuln Explorer**: Enriches vulnerability data with VEX status.
* **Orchestrator**: Schedules consensus compute jobs for batch processing.
* **Authority**: Validates issuer trust and key fingerprints.
* **Config stores**: MongoDB (projections, issuer directory), Redis (caches).
---
## 3) API Surface
Base path: `/api/v1/vexlens`. Full OpenAPI spec at `docs/api/vexlens-openapi.yaml`.
### 3.1 Consensus Operations
| Endpoint | Method | Description |
|----------|--------|-------------|
| `/consensus` | POST | Compute consensus for a vulnerability-product pair |
| `/consensus/batch` | POST | Compute consensus for multiple pairs in batch |
### 3.2 Projection Queries
| Endpoint | Method | Description |
|----------|--------|-------------|
| `/projections` | GET | Query consensus projections with filtering |
| `/projections/{projectionId}` | GET | Get a projection by ID |
| `/projections/latest` | GET | Get latest projection for a vuln-product pair |
| `/projections/history` | GET | Get projection history |
### 3.3 Issuer Directory
| Endpoint | Method | Description |
|----------|--------|-------------|
| `/issuers` | GET | List registered issuers |
| `/issuers` | POST | Register a new issuer |
| `/issuers/{issuerId}` | GET | Get issuer details |
| `/issuers/{issuerId}` | DELETE | Revoke an issuer |
| `/issuers/{issuerId}/keys` | POST | Add a key to an issuer |
| `/issuers/{issuerId}/keys/{fingerprint}` | DELETE | Revoke a key |
### 3.4 Statistics
| Endpoint | Method | Description |
|----------|--------|-------------|
| `/statistics` | GET | Get consensus statistics |
---
## 4) Data Flow
```
┌─────────────┐ ┌──────────────┐ ┌─────────────────┐
│ Excititor │────▶│ Normalizer │────▶│ Trust Weighting │
│ (VEX Docs) │ │ (OpenVEX, │ │ (9 factors) │
└─────────────┘ │ CSAF, CDX) │ └────────┬────────┘
└──────────────┘ │
▼
┌─────────────┐ ┌──────────────┐ ┌─────────────────┐
│ Policy │◀────│ Projection │◀────│ Consensus │
│ Engine │ │ Store │ │ Engine │
└─────────────┘ └──────────────┘ └─────────────────┘
│
▼
┌──────────────┐
│ Events │
│ (Computed, │
│ StatusChange,│
│ Conflict) │
└──────────────┘
```
---
## 5) VEX Status Lattice
VexLens uses a status lattice for conservative conflict resolution:
```
affected (most restrictive)
│
▼
under_investigation
│
▼
not_affected
│
▼
fixed (least restrictive)
```
In lattice mode, the most restrictive status always wins. This ensures that when sources disagree, the system errs on the side of caution.
---
## 6) Trust Weight Factors
| Factor | Weight | Description |
|--------|--------|-------------|
| IssuerBase | 25% | Base trust from issuer directory |
| SignatureStatus | 15% | Valid/invalid/unsigned signature |
| Freshness | 15% | Document age with exponential decay |
| IssuerCategory | 10% | Vendor > Distributor > Aggregator |
| IssuerTier | 10% | Authoritative > Trusted > Untrusted |
| StatusQuality | 10% | Has justification, specific status |
| TransparencyLog | 5% | Sigstore Rekor entry |
| SourceMatch | 5% | Source URI pattern match |
| ProductAuthority | 5% | Issuer is authoritative for product |
---
## 7) Configuration
```yaml
vexlens:
consensus:
defaultMode: WeightedVote # HighestWeight, WeightedVote, Lattice, AuthoritativeFirst, MostRecent
minimumConfidence: 0.1
conflictThreshold: 0.3
requireJustificationForNotAffected: false
trust:
freshnessHalfLifeDays: 90
minimumFreshness: 0.3
allowUnsigned: true
unsignedPenalty: 0.3
allowUnknownIssuers: true
unknownIssuerPenalty: 0.5
storage:
projectionRetentionDays: 365
eventRetentionDays: 90
issuerDirectory:
source: mongodb # mongodb, file, api
refreshIntervalMinutes: 60
```
---
## 8) Storage Schema
### 8.1 Consensus Projection
```json
{
"projectionId": "proj-abc123",
"vulnerabilityId": "CVE-2024-1234",
"productKey": "pkg:npm/lodash@4.17.21",
"tenantId": "tenant-001",
"status": "not_affected",
"justification": "vulnerable_code_not_present",
"confidenceScore": 0.95,
"outcome": "Unanimous",
"statementCount": 3,
"conflictCount": 0,
"rationaleSummary": "Unanimous consensus from 3 authoritative sources",
"computedAt": "2025-12-06T12:00:00Z",
"storedAt": "2025-12-06T12:00:01Z",
"previousProjectionId": null,
"statusChanged": true
}
```
### 8.2 Issuer Record
```json
{
"issuerId": "npm-security",
"name": "npm Security Team",
"category": "Vendor",
"trustTier": "Authoritative",
"status": "Active",
"keyFingerprints": [
{
"fingerprint": "ABCD1234EFGH5678",
"keyType": "Pgp",
"algorithm": "EdDSA",
"status": "Active",
"registeredAt": "2025-01-01T00:00:00Z",
"expiresAt": null
}
],
"metadata": {
"description": "Official npm security advisories",
"uri": "https://www.npmjs.com/advisories",
"email": "security@npmjs.com"
},
"registeredAt": "2025-01-01T00:00:00Z"
}
```
---
## 9) Events
### 9.1 ConsensusComputedEvent
Emitted after every consensus computation.
```json
{
"eventId": "evt-abc123",
"projectionId": "proj-abc123",
"vulnerabilityId": "CVE-2024-1234",
"productKey": "pkg:npm/lodash@4.17.21",
"status": "not_affected",
"confidenceScore": 0.95,
"outcome": "Unanimous",
"statementCount": 3,
"computedAt": "2025-12-06T12:00:00Z",
"emittedAt": "2025-12-06T12:00:01Z"
}
```
### 9.2 ConsensusStatusChangedEvent
Emitted when consensus status changes from previous projection.
### 9.3 ConsensusConflictDetectedEvent
Emitted when conflicts are detected during consensus computation.
---
## 10) Observability
### 10.1 Metrics (OpenTelemetry)
| Metric | Type | Description |
|--------|------|-------------|
| `vexlens.consensus.computed_total` | Counter | Total consensus computations |
| `vexlens.consensus.conflicts_total` | Counter | Total conflicts detected |
| `vexlens.consensus.confidence` | Histogram | Confidence score distribution |
| `vexlens.consensus.duration_seconds` | Histogram | Computation duration |
| `vexlens.consensus.status_changes_total` | Counter | Status changes detected |
| `vexlens.normalization.documents_total` | Counter | Documents normalized |
| `vexlens.trust.weight_value` | Histogram | Trust weight distribution |
| `vexlens.issuer.registered_total` | Counter | Issuers registered |
### 10.2 Traces
Activity source: `StellaOps.VexLens`
| Activity | Description |
|----------|-------------|
| `vexlens.normalize` | VEX document normalization |
| `vexlens.compute_trust_weight` | Trust weight computation |
| `vexlens.compute_consensus` | Consensus computation |
| `vexlens.store_projection` | Projection storage |
| `vexlens.query_projections` | Projection query |
### 10.3 Logging
Structured logging with event IDs in `VexLensLogEvents`:
- 1xxx: Normalization events
- 2xxx: Product mapping events
- 3xxx: Signature verification events
- 4xxx: Trust weight events
- 5xxx: Consensus events
- 6xxx: Projection events
- 7xxx: Issuer directory events
---
## 11) Security Considerations
1. **Issuer Trust**: All issuers must be registered with verified key fingerprints.
2. **Signature Verification**: Documents should be cryptographically signed for production use.
3. **Tenant Isolation**: Projections are scoped to tenants; no cross-tenant data access.
4. **Audit Trail**: All consensus computations are logged with full rationale.
5. **Determinism**: All computations are deterministic for reproducibility.
---
## 12) Test Matrix
| Test Category | Coverage | Notes |
|---------------|----------|-------|
| Unit tests | Normalizer, Parser, Trust, Consensus | 89+ tests |
| Determinism harness | Normalization, Trust, Consensus | Verify reproducibility |
| Integration tests | API service, Storage, Events | End-to-end flows |
| Property-based tests | Lattice semantics, Weight computation | Invariant verification |

475
docs/modules/vexlens/operations/deployment.md Normal file
View File

@@ -0,0 +1,475 @@
# VexLens Deployment Runbook
> Operational runbook for deploying and configuring VexLens consensus engine.
---
## 1) Prerequisites
### 1.1 Infrastructure Requirements
| Component | Requirement | Notes |
|-----------|-------------|-------|
| Runtime | .NET 10.0+ | LTS recommended |
| Database | MongoDB 6.0+ | For projections and issuer directory |
| Cache | Redis 7.0+ (optional) | For caching consensus results |
| Memory | 512MB minimum | 2GB recommended for production |
| CPU | 2 cores minimum | 4 cores for high throughput |
### 1.2 Dependencies
- **Excititor**: VEX document ingestion service
- **Authority**: OIDC token validation
- **Policy Engine**: (optional) For VEX-aware policy evaluation
---
## 2) Configuration
### 2.1 Environment Variables
```bash
# Core Settings
VEXLENS_CONSENSUS_DEFAULT_MODE=WeightedVote
VEXLENS_CONSENSUS_MINIMUM_CONFIDENCE=0.1
VEXLENS_CONSENSUS_CONFLICT_THRESHOLD=0.3
# Trust Settings
VEXLENS_TRUST_FRESHNESS_HALFLIFE_DAYS=90
VEXLENS_TRUST_MINIMUM_FRESHNESS=0.3
VEXLENS_TRUST_ALLOW_UNSIGNED=true
VEXLENS_TRUST_UNSIGNED_PENALTY=0.3
VEXLENS_TRUST_ALLOW_UNKNOWN_ISSUERS=true
VEXLENS_TRUST_UNKNOWN_ISSUER_PENALTY=0.5
# Storage
VEXLENS_STORAGE_MONGODB_CONNECTION_STRING=mongodb://localhost:27017
VEXLENS_STORAGE_MONGODB_DATABASE=vexlens
VEXLENS_STORAGE_PROJECTION_RETENTION_DAYS=365
VEXLENS_STORAGE_EVENT_RETENTION_DAYS=90
# Issuer Directory
VEXLENS_ISSUER_DIRECTORY_SOURCE=mongodb
VEXLENS_ISSUER_DIRECTORY_REFRESH_INTERVAL_MINUTES=60
# Observability
VEXLENS_OTEL_EXPORTER_ENDPOINT=http://otel-collector:4317
VEXLENS_OTEL_SERVICE_NAME=vexlens
```
### 2.2 Configuration File (vexlens.yaml)
```yaml
vexlens:
consensus:
defaultMode: WeightedVote
minimumConfidence: 0.1
conflictThreshold: 0.3
requireJustificationForNotAffected: false
trust:
freshnessHalfLifeDays: 90
minimumFreshness: 0.3
allowUnsigned: true
unsignedPenalty: 0.3
allowUnknownIssuers: true
unknownIssuerPenalty: 0.5
factorWeights:
IssuerBase: 0.25
SignatureStatus: 0.15
Freshness: 0.15
IssuerCategory: 0.10
IssuerTier: 0.10
StatusQuality: 0.10
TransparencyLog: 0.05
SourceMatch: 0.05
ProductAuthority: 0.05
storage:
mongodb:
connectionString: mongodb://localhost:27017
database: vexlens
projectionsCollection: consensus_projections
issuersCollection: issuers
projectionRetentionDays: 365
eventRetentionDays: 90
issuerDirectory:
source: mongodb
refreshIntervalMinutes: 60
seedFile: /etc/vexlens/issuers.json
observability:
metrics:
enabled: true
exporterEndpoint: http://otel-collector:4317
tracing:
enabled: true
samplingRatio: 0.1
logging:
level: Information
format: json
```
---
## 3) Deployment Steps
### 3.1 Docker Deployment
```bash
# Pull the image
docker pull stellaops/vexlens:latest
# Run with configuration
docker run -d \
--name vexlens \
-p 8080:8080 \
-v /etc/vexlens:/etc/vexlens:ro \
-e VEXLENS_STORAGE_MONGODB_CONNECTION_STRING=mongodb://mongo:27017 \
stellaops/vexlens:latest
```
### 3.2 Kubernetes Deployment
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: vexlens
namespace: stellaops
spec:
replicas: 2
selector:
matchLabels:
app: vexlens
template:
metadata:
labels:
app: vexlens
spec:
containers:
- name: vexlens
image: stellaops/vexlens:latest
ports:
- containerPort: 8080
env:
- name: VEXLENS_STORAGE_MONGODB_CONNECTION_STRING
valueFrom:
secretKeyRef:
name: vexlens-secrets
key: mongodb-connection-string
resources:
requests:
memory: "512Mi"
cpu: "500m"
limits:
memory: "2Gi"
cpu: "2000m"
livenessProbe:
httpGet:
path: /health/live
port: 8080
initialDelaySeconds: 10
periodSeconds: 30
readinessProbe:
httpGet:
path: /health/ready
port: 8080
initialDelaySeconds: 5
periodSeconds: 10
volumeMounts:
- name: config
mountPath: /etc/vexlens
readOnly: true
volumes:
- name: config
configMap:
name: vexlens-config
---
apiVersion: v1
kind: Service
metadata:
name: vexlens
namespace: stellaops
spec:
selector:
app: vexlens
ports:
- port: 80
targetPort: 8080
```
### 3.3 Helm Deployment
```bash
helm install vexlens stellaops/vexlens \
--namespace stellaops \
--set mongodb.connectionString=mongodb://mongo:27017 \
--set replicas=2 \
--set resources.requests.memory=512Mi \
--set resources.limits.memory=2Gi
```
---
## 4) Issuer Directory Setup
### 4.1 Seed Issuers File
Create `/etc/vexlens/issuers.json`:
```json
{
"issuers": [
{
"issuerId": "npm-security",
"name": "npm Security Team",
"category": "Vendor",
"trustTier": "Authoritative",
"keyFingerprints": [
{
"fingerprint": "ABCD1234EFGH5678",
"keyType": "Pgp",
"algorithm": "EdDSA"
}
],
"metadata": {
"description": "Official npm security advisories",
"uri": "https://www.npmjs.com/advisories"
}
},
{
"issuerId": "github-security",
"name": "GitHub Security Lab",
"category": "Aggregator",
"trustTier": "Trusted",
"metadata": {
"description": "GitHub Security Advisories",
"uri": "https://github.com/advisories"
}
}
]
}
```
### 4.2 Register Issuer via API
```bash
curl -X POST http://vexlens:8080/api/v1/vexlens/issuers \
-H "Content-Type: application/json" \
-H "X-StellaOps-Tenant: tenant-001" \
-d '{
"issuerId": "vendor-acme",
"name": "ACME Corporation",
"category": "Vendor",
"trustTier": "Authoritative",
"initialKeys": [
{
"fingerprint": "1234ABCD5678EFGH",
"keyType": "Pgp",
"algorithm": "RSA"
}
],
"metadata": {
"description": "ACME security advisories",
"uri": "https://security.acme.example.com"
}
}'
```
---
## 5) Health Checks
### 5.1 Liveness Probe
```bash
curl http://vexlens:8080/health/live
# Response: {"status": "Healthy"}
```
### 5.2 Readiness Probe
```bash
curl http://vexlens:8080/health/ready
# Response: {"status": "Healthy", "checks": {"mongodb": "Healthy", "issuerDirectory": "Healthy"}}
```
### 5.3 Detailed Health
```bash
curl http://vexlens:8080/health/detailed
# Full health check with component details
```
---
## 6) Monitoring
### 6.1 Key Metrics to Monitor
| Metric | Alert Threshold | Description |
|--------|-----------------|-------------|
| `vexlens.consensus.duration_seconds` | p99 > 5s | Consensus computation latency |
| `vexlens.consensus.conflicts_total` | rate > 100/min | High conflict rate |
| `vexlens.normalization.errors_total` | rate > 10/min | Normalization failures |
| `vexlens.projection.query_duration_seconds` | p99 > 1s | Slow projection queries |
### 6.2 Grafana Dashboard
Import the VexLens dashboard from `deploy/grafana/vexlens-dashboard.json`.
### 6.3 Alerting Rules
```yaml
groups:
- name: vexlens
rules:
- alert: VexLensHighLatency
expr: histogram_quantile(0.99, rate(vexlens_consensus_duration_seconds_bucket[5m])) > 5
for: 5m
labels:
severity: warning
annotations:
summary: "VexLens consensus latency is high"
- alert: VexLensHighConflictRate
expr: rate(vexlens_consensus_conflicts_total[5m]) > 100
for: 10m
labels:
severity: warning
annotations:
summary: "VexLens detecting high conflict rate"
- alert: VexLensNormalizationErrors
expr: rate(vexlens_normalization_errors_total[5m]) > 10
for: 5m
labels:
severity: critical
annotations:
summary: "VexLens normalization errors increasing"
```
---
## 7) Backup and Recovery
### 7.1 Backup Projections
```bash
# MongoDB backup
mongodump --uri="mongodb://localhost:27017" \
--db=vexlens \
--collection=consensus_projections \
--out=/backup/vexlens-$(date +%Y%m%d)
```
### 7.2 Backup Issuer Directory
```bash
# Export issuers to JSON
curl http://vexlens:8080/api/v1/vexlens/issuers?limit=1000 \
> /backup/issuers-$(date +%Y%m%d).json
```
### 7.3 Restore
```bash
# Restore MongoDB
mongorestore --uri="mongodb://localhost:27017" \
--db=vexlens \
/backup/vexlens-20251206/
# Re-seed issuers if needed
# Issuers are automatically loaded from seed file on startup
```
---
## 8) Scaling
### 8.1 Horizontal Scaling
VexLens is stateless for compute operations. Scale horizontally by adding replicas:
```bash
kubectl scale deployment vexlens --replicas=4 -n stellaops
```
### 8.2 Performance Tuning
```yaml
# For high-throughput deployments
vexlens:
consensus:
# Enable batch processing
batchSize: 100
batchTimeoutMs: 50
storage:
mongodb:
# Connection pool
maxConnectionPoolSize: 100
minConnectionPoolSize: 10
caching:
enabled: true
redis:
connectionString: redis://redis:6379
consensusTtlMinutes: 5
issuerTtlMinutes: 60
```
---
## 9) Troubleshooting
### 9.1 Common Issues
| Issue | Cause | Resolution |
|-------|-------|------------|
| Slow consensus | Many statements | Enable caching, increase batch size |
| High conflict rate | Inconsistent sources | Review issuer trust tiers |
| Normalization failures | Invalid VEX format | Check Excititor connector config |
| Low confidence scores | Missing signatures | Configure issuer keys |
### 9.2 Debug Logging
```bash
# Enable debug logging
export VEXLENS_OBSERVABILITY_LOGGING_LEVEL=Debug
```
### 9.3 Determinism Verification
```bash
# Run determinism harness
curl -X POST http://vexlens:8080/api/v1/vexlens/test/determinism \
-H "Content-Type: application/json" \
-d '{"vexContent": "..."}'
```
---
## 10) Upgrade Procedure
### 10.1 Rolling Upgrade
```bash
# Update image
kubectl set image deployment/vexlens vexlens=stellaops/vexlens:v1.2.0 -n stellaops
# Monitor rollout
kubectl rollout status deployment/vexlens -n stellaops
```
### 10.2 Database Migrations
VexLens uses automatic schema migrations. No manual intervention required for minor versions.
For major version upgrades:
1. Backup all data
2. Review migration notes in release changelog
3. Apply migrations: `vexlens migrate --apply`
4. Verify: `vexlens migrate --verify`

408
docs/modules/vexlens/operations/offline-kit.md Normal file
View File

@@ -0,0 +1,408 @@
# VexLens Offline Kit
> Air-gapped deployment guide for VexLens consensus engine.
---
## 1) Overview
VexLens can operate in fully air-gapped environments with pre-loaded VEX data and issuer directories. This guide covers offline deployment, bundle creation, and operational procedures.
---
## 2) Offline Bundle Structure
### 2.1 Bundle Manifest
```json
{
"bundleId": "vexlens-bundle-2025-12-06",
"version": "1.0.0",
"createdAt": "2025-12-06T00:00:00Z",
"createdBy": "stellaops-export",
"checksum": "sha256:abc123...",
"components": {
"issuerDirectory": {
"file": "issuers.json",
"checksum": "sha256:def456...",
"count": 150
},
"vexStatements": {
"file": "vex-statements.ndjson.gz",
"checksum": "sha256:ghi789...",
"count": 50000
},
"projectionSnapshots": {
"file": "projections.ndjson.gz",
"checksum": "sha256:jkl012...",
"count": 25000
},
"trustConfiguration": {
"file": "trust-config.yaml",
"checksum": "sha256:mno345..."
}
},
"compatibility": {
"minVersion": "1.0.0",
"maxVersion": "2.0.0"
}
}
```
### 2.2 Bundle Contents
```
vexlens-bundle-2025-12-06/
├── manifest.json
├── issuers.json
├── vex-statements.ndjson.gz
├── projections.ndjson.gz
├── trust-config.yaml
├── checksums.sha256
└── signature.dsse
```
---
## 3) Creating Offline Bundles
### 3.1 Export Command
```bash
# Export from online VexLens instance
stellaops vexlens export \
--output /export/vexlens-bundle-$(date +%Y-%m-%d) \
--include-issuers \
--include-statements \
--include-projections \
--compress \
--sign
```
### 3.2 Selective Export
```bash
# Export only specific tenants
stellaops vexlens export \
--output /export/tenant-bundle \
--tenant tenant-001,tenant-002 \
--since 2025-01-01 \
--compress
# Export only critical vulnerabilities
stellaops vexlens export \
--output /export/critical-bundle \
--vulnerability-pattern "CVE-202[45]-*" \
--status affected,under_investigation \
--compress
```
### 3.3 Bundle Signing
```bash
# Sign bundle with organization key
stellaops vexlens export sign \
--bundle /export/vexlens-bundle-2025-12-06 \
--key /keys/export-signing-key.pem \
--output /export/vexlens-bundle-2025-12-06/signature.dsse
```
---
## 4) Importing Offline Bundles
### 4.1 Verification
```bash
# Verify bundle integrity and signature
stellaops vexlens import verify \
--bundle /import/vexlens-bundle-2025-12-06 \
--trust-root /etc/vexlens/trust-roots.pem
# Output:
# Bundle ID: vexlens-bundle-2025-12-06
# Created: 2025-12-06T00:00:00Z
# Signature: VALID (signed by: StellaOps Export Service)
# Checksums: VALID (all 4 files verified)
# Compatibility: COMPATIBLE (current version: 1.1.0)
```
### 4.2 Import Command
```bash
# Import bundle to offline VexLens
stellaops vexlens import \
--bundle /import/vexlens-bundle-2025-12-06 \
--mode merge \
--verify-signature
# Import modes:
# - merge: Add new data, keep existing
# - replace: Replace all data with bundle contents
# - incremental: Only add data newer than existing
```
### 4.3 Staged Import
For large bundles, use staged import:
```bash
# Stage 1: Import issuers
stellaops vexlens import \
--bundle /import/bundle \
--component issuer-directory \
--dry-run
# Stage 2: Import statements
stellaops vexlens import \
--bundle /import/bundle \
--component vex-statements \
--batch-size 1000
# Stage 3: Import projections
stellaops vexlens import \
--bundle /import/bundle \
--component projections \
--batch-size 5000
```
---
## 5) Offline Configuration
### 5.1 Air-Gap Mode Settings
```yaml
vexlens:
airgap:
enabled: true
# Disable external connectivity checks
allowExternalConnections: false
# Use file-based issuer directory
issuerDirectorySource: file
# Pre-compute consensus on import
precomputeConsensus: true
trust:
# Stricter settings for air-gap
allowUnsigned: false
allowUnknownIssuers: false
# Use local trust anchors
trustAnchors: /etc/vexlens/trust-anchors.pem
storage:
# Local storage only
mongodb:
connectionString: mongodb://localhost:27017
# No external cache
redis:
enabled: false
time:
# Use time anchor for staleness checks
timeAnchorFile: /etc/vexlens/time-anchor.json
# Maximum allowed drift
maxDriftDays: 7
```
### 5.2 Time Anchor Configuration
For air-gapped environments, use time anchors:
```json
{
"anchorTime": "2025-12-06T00:00:00Z",
"signature": "base64...",
"validUntil": "2025-12-13T00:00:00Z",
"signedBy": "stellaops-time-authority"
}
```
---
## 6) Operational Procedures
### 6.1 Bundle Update Cycle
1. **Export** (Online environment):
```bash
stellaops vexlens export --output /export/weekly-bundle --compress --sign
```
2. **Transfer** (Secure media):
- Copy bundle to removable media
- Verify checksums after transfer
- Log transfer in custody chain
3. **Verify** (Offline environment):
```bash
stellaops vexlens import verify --bundle /import/weekly-bundle
```
4. **Import** (Offline environment):
```bash
stellaops vexlens import --bundle /import/weekly-bundle --mode incremental
```
5. **Recompute** (If needed):
```bash
stellaops vexlens consensus recompute --since $(date -d '7 days ago' +%Y-%m-%d)
```
### 6.2 Staleness Monitoring
```bash
# Check data freshness
stellaops vexlens status --staleness
# Output:
# Data Freshness Report
# ---------------------
# Issuer Directory: 2 days old (OK)
# VEX Statements: 5 days old (OK)
# Projections: 5 days old (OK)
# Time Anchor: 2 days old (OK)
#
# Overall Status: FRESH
```
### 6.3 Audit Trail
All import operations are logged:
```bash
# View import history
stellaops vexlens import history --limit 10
# Output:
# Import History
# --------------
# 2025-12-06 08:00: vexlens-bundle-2025-12-06 (merge, 50000 statements)
# 2025-11-29 08:00: vexlens-bundle-2025-11-29 (incremental, 12000 statements)
# ...
```
---
## 7) Degraded Mode Operation
### 7.1 Degradation Matrix
| Component | Degradation | Impact | Mitigation |
|-----------|-------------|--------|------------|
| Stale VEX data | >7 days old | Lower accuracy | Schedule bundle update |
| Missing issuers | Unknown issuer | Lower trust scores | Add issuer to directory |
| No projections | Cold start | Slower first queries | Pre-compute on import |
| Time drift | >24 hours | Staleness warnings | Update time anchor |
### 7.2 Emergency Recovery
If bundle import fails:
```bash
# Check bundle integrity
stellaops vexlens import verify --bundle /import/bundle --verbose
# Attempt partial import
stellaops vexlens import --bundle /import/bundle --skip-corrupted
# Rollback to previous state
stellaops vexlens import rollback --to vexlens-bundle-2025-11-29
```
---
## 8) Bundle Management
### 8.1 Retention Policy
```yaml
vexlens:
bundles:
# Keep last N bundles
retentionCount: 5
# Minimum age before deletion
minimumAgeDays: 30
# Archive location
archivePath: /archive/vexlens-bundles
```
### 8.2 Storage Requirements
| Data Type | Typical Size | Compression Ratio |
|-----------|--------------|-------------------|
| Issuers | 1-5 MB | 5:1 |
| Statements | 100-500 MB | 10:1 |
| Projections | 50-200 MB | 8:1 |
| **Total Bundle** | **150-700 MB** | **8:1** |
### 8.3 Bundle Cleanup
```bash
# Clean old bundles
stellaops vexlens bundles cleanup --keep 5
# Archive bundles older than 30 days
stellaops vexlens bundles archive --older-than 30d --to /archive
```
---
## 9) Security Considerations
### 9.1 Bundle Signing
All bundles should be signed before transfer:
```bash
# Verify signature chain
stellaops vexlens import verify-chain \
--bundle /import/bundle \
--trust-root /etc/vexlens/root-ca.pem
```
### 9.2 Transfer Security
1. Use encrypted removable media
2. Maintain custody chain documentation
3. Verify checksums at each transfer point
4. Log all bundle operations
### 9.3 Access Control
```yaml
vexlens:
security:
# Require authentication for import
importRequiresAuth: true
# Allowed import roles
importRoles: [vexlens.admin, vexlens.operator]
# Audit all imports
auditImports: true
```
---
## 10) Troubleshooting
### 10.1 Common Issues
| Issue | Cause | Resolution |
|-------|-------|------------|
| Import fails | Corrupted bundle | Re-export from source |
| Signature invalid | Wrong trust root | Update trust anchors |
| Time anchor expired | Stale time anchor | Generate new anchor |
| Missing issuers | Incomplete export | Include issuers in export |
### 10.2 Diagnostic Commands
```bash
# Verify bundle contents
stellaops vexlens bundle inspect /import/bundle
# Check import readiness
stellaops vexlens import preflight --bundle /import/bundle
# Generate diagnostic report
stellaops vexlens diagnostics --output /tmp/diag.json
```