up

docs/implplan/archived/SPRINT_0110_0001_0001_ingestion_evidence.md

@@ -99,6 +99,9 @@
 ## Execution Log
 | Date (UTC) | Update | Owner |
 | --- | --- | --- |
+| 2025-11-25 | Relocated remaining ops-track items (EXPORT-OBS-51-001, OBS-51/53-001, TIME-57-001, CLI-AIRGAP-56-001, MIRROR-CRT-56/57/58) to Ops sprints 503/0506 per “no ops in dev sprint” rule; sprint now has zero open TODO/BLOCKED rows. | Project Mgmt |
+| 2025-11-25 | Added `tools/run-airgap-bundle-tests.sh` to run the Airgap bundle determinism slice with TRX output (`TestResults/airgap-bundle.trx`) for CI runners with warmed NuGet cache; local runs still stall on this host. | Implementer |
+| 2025-11-25 | Attempted local build/test via `tools/run-airgap-bundle-tests.sh`; restore/build stalled and was cancelled (~12s). Action: execute on CI runner with warmed NuGet cache to produce `TestResults/airgap-bundle.trx`. | Implementer |
 | 2025-11-25 | Finalised air-gap bundle determinism: `AirgapBundleBuilder` now accepts injected `createdUtc` (default Unix epoch) and manifests/entry-traces are bit-for-bit stable across runs; CONCELIER-AIRGAP-56-001..58-001 dependencies (LNM schema + Evidence Locker contract) closed out. | Implementer |
 | 2025-11-23 | Moved CI runner + mirror assembler promotion actions to `SPRINT_506_ops_devops_iv.md`; Sprint 0110 now tracks development deliverables only. | Project Mgmt |
 | 2025-11-23 | Normalised sections to template (added Wave Coordination/Detail Snapshots/Interlocks/Action Tracker; renamed Upcoming Checkpoints; no status changes.) | Project Mgmt |

docs/implplan/archived/SPRINT_0119_0001_0002_excititor_ii.md

@@ -0,0 +1,93 @@
+# Sprint 0119_0001_0002 · Excititor Ingestion & Evidence (Phase II)
+
+## Topic & Scope
+- Harden ingestion/linkset storage and connector trust provenance so Excititor stays aggregation-only while downstream consumers build consensus.
+- Deliver Console VEX aggregation/search views plus Graph/Vuln Explorer feeds without embedding verdict logic.
+- Enforce idempotent raw VEX upserts and remove legacy consensus paths.
+- **Working directory:** `src/Excititor` (WebService, Core, Storage, Connectors); keep changes inside module boundaries.
+
+## Dependencies & Concurrency
+- Upstream: Sprint 0119_0001_0001 (Excititor I) projection work; Policy contracts (EXCITITOR-POLICY-01-001); Attestor DSSE readiness.
+- Concurrency: Console APIs can progress alongside connector provenance DONE items; Graph overlay tasks blocked pending inspector linkouts; storage idempotency must precede consensus removal.
+- Peers: No CC-decade conflicts; coordinate with Cartographer/Vuln Explorer for API shapes.
+
+## Documentation Prerequisites
+- `docs/modules/excititor/architecture.md`
+- `docs/modules/excititor/README.md`
+- `docs/modules/excititor/mirrors.md`
+- `docs/modules/excititor/operations/*`
+- `docs/modules/excititor/implementation_plan.md`
+- Excititor component `AGENTS.md` files (WebService, Core, Storage, Connectors).
+
+## Delivery Tracker
+| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
+| --- | --- | --- | --- | --- | --- |
+| P1 | PREP-EXCITITOR-CONSOLE-23-001-AWAITING-CONCRE | DONE (2025-11-20) | Prep note at `docs/modules/excititor/prep/2025-11-20-console-vex-contract-prep.md`; awaiting LNM view spec + SSE envelopes. | Excititor WebService Guild · BE-Base Platform Guild | Awaiting concrete `/console/vex` API contract and grouping schema. |
+| P2 | PREP-EXCITITOR-CONSOLE-23-002-DEPENDS-ON-23-0 | DONE (2025-11-20) | Prep note at `docs/modules/excititor/prep/2025-11-20-console-counters-prep.md`; depends on 23-001 buckets. | Excititor WebService Guild | Counters contract. |
+| P3 | PREP-EXCITITOR-CONSOLE-23-003-DEPENDS-ON-23-0 | DONE (2025-11-20) | Prep note at `docs/modules/excititor/prep/2025-11-20-console-cache-rbac-prep.md`; awaits cache TTL/precedence traces. | Excititor WebService Guild | Caching/RBAC/precedence context. |
+| P4 | PREP-EXCITITOR-CORE-AOC-19-002-LINKSET-EXTRAC | DONE (2025-11-20) | Prep note at `docs/modules/excititor/prep/2025-11-20-linkset-extraction-prep.md`. | Excititor Core Guild | Linkset extraction rules/ordering. |
+| P5 | PREP-EXCITITOR-CORE-AOC-19-003-BLOCKED-ON-19 | DONE (2025-11-20) | Prep note at `docs/modules/excititor/prep/2025-11-20-raw-upsert-idempotency-prep.md`. | Excititor Core Guild | Idempotent upsert supersede chains. |
+| P6 | PREP-EXCITITOR-CORE-AOC-19-004-REMOVE-CONSENS | DONE (2025-11-20) | Prep doc at `docs/modules/excititor/prep/2025-11-20-consensus-removal-prep.md`. | Excititor Core Guild | Remove consensus after idempotency. |
+| P7 | PREP-EXCITITOR-CORE-AOC-19-013-SEED-TENANT-AW | DONE (2025-11-20) | Prep doc at `docs/modules/excititor/prep/2025-11-20-tenant-authority-prep.md`. | Excititor Core Guild | Tenant-aware Authority clients. |
+| P8 | PREP-EXCITITOR-GRAPH-21-001-NEEDS-CARTOGRAPHE | DONE (2025-11-20) | Prep doc at `docs/modules/excititor/prep/2025-11-20-graph-21-001-prep.md`. | Excititor Core · Cartographer Guild | Cartographer API contract. |
+| P9 | PREP-EXCITITOR-GRAPH-21-002-BLOCKED-ON-21-001 | DONE (2025-11-20) | Prep doc at `docs/modules/excititor/prep/2025-11-20-graph-21-002-prep.md`. | Excititor Core Guild | Overlay payload. |
+| P10 | PREP-EXCITITOR-GRAPH-21-005-BLOCKED-ON-21-002 | DONE (2025-11-20) | Prep doc at `docs/modules/excititor/prep/2025-11-20-graph-21-005-prep.md`. | Excititor Storage Guild | Index plan. |
+| 1 | EXCITITOR-CONN-SUSE-01-003 | DONE (2025-11-09) | Trust metadata flowing; monitor. | Connectors – SUSE | Emit provider trust configuration. |
+| 2 | EXCITITOR-CONN-UBUNTU-01-003 | DONE (2025-11-09) | Trust metadata flowing; monitor. | Connectors – Ubuntu | Emit Ubuntu signing metadata. |
+| 3 | EXCITITOR-CONSOLE-23-001 | DONE (2025-11-23) | Endpoint `/console/vex` grouped statements live; tenant filters enforced | Excititor WebService Guild · BE-Base | Grouped VEX statements with traces/tenant filters. |
+| 4 | EXCITITOR-CONSOLE-23-002 | DONE (2025-11-23) | Counters emitted via `ConsoleTelemetry`; status buckets returned in response | Excititor WebService Guild | Delta counts + metrics. |
+| 5 | EXCITITOR-CONSOLE-23-003 | DONE (2025-11-23) | Response caching added (30s per query key); RBAC via required tenant header | Excititor WebService Guild | Rapid VEX lookups with precedence/caching/RBAC. |
+| 6 | EXCITITOR-CORE-AOC-19-002 | DONE (2025-11-23) | Core unit extractor landed; tests green | Excititor Core Guild | Linkset extraction. |
+| 7 | EXCITITOR-CORE-AOC-19-003 | DONE (2025-11-23) | Append-only enforcement landed in Mongo raw store; duplicates short-circuit | Excititor Core Guild | Raw VEX append-only uniqueness. |
+| 8 | EXCITITOR-CORE-AOC-19-004 | DONE (2025-11-23) | Consensus refresh hosted service disabled when Aggregation-Only flag set; scheduler no-ops under DisableConsensus | Excititor Core Guild | Excise consensus/merge/severity logic. |
+| 9 | EXCITITOR-CORE-AOC-19-013 | DONE (2025-11-23) | Tenant Authority client factory + options validator added; tests authored | Excititor Core Guild | Tenant-aware Authority clients/tests. |
+| 10 | EXCITITOR-GRAPH-21-001 | DONE (2025-11-23) | `/internal/graph/linkouts` implemented per prep (batched linkouts) | Excititor Core · Cartographer | Batched linkouts. |
+| 11 | EXCITITOR-GRAPH-21-002 | DONE (2025-11-23) | PREP-EXCITITOR-GRAPH-21-002-BLOCKED-ON-21-001 | Excititor Core Guild | Overlays. |
+| 12 | EXCITITOR-GRAPH-21-005 | DONE (2025-11-23) | PREP-EXCITITOR-GRAPH-21-005-BLOCKED-ON-21-002 | Excititor Storage Guild | Index/materialized overlays. |
+| 13 | EXCITITOR-GRAPH-24-101 | DONE (2025-11-25) | PREP-EXCITITOR-GRAPH-24-101-WAIT-FOR-21-005-I | Excititor WebService Guild | VEX status summaries. |
+| 14 | EXCITITOR-GRAPH-24-102 | DONE (2025-11-25) | PREP-EXCITITOR-GRAPH-24-102-DEPENDS-ON-24-101 | Excititor WebService Guild | Batch retrieval for overlays/tooltips. |
+
+## Execution Log
+| Date (UTC) | Update | Owner |
+| --- | --- | --- |
+| 2025-11-19 | Normalized PREP-EXCITITOR-CORE-AOC-19-003 Task ID. | Project Mgmt |
+| 2025-11-19 | Marked PREP tasks P1–P17 BLOCKED (missing console contract, linkset schema, Cartographer API, orchestrator inputs). | Project Mgmt |
+| 2025-11-23 | PREP artifacts delivered; moved EXCITITOR-CONSOLE-23-001/002/003 and EXCITITOR-CORE-AOC-19-002/003 from BLOCKED to TODO to begin implementation. | Project Mgmt |
+| 2025-11-23 | Implemented `/console/vex` with tenant enforcement, status/purl/advisory filters, stable paging + cursor, in-memory caching, and status counters + telemetry; set console tasks 23-001/002/003 to DONE. | Implementer |
+| 2025-11-23 | Updated console prep doc with counters + caching notes; SSE still pending final view spec. | Implementer |
+| 2025-11-23 | Enforced append-only raw VEX ingest: Mongo raw store now short-circuits when digest exists (no rewrites) and leaves GridFS untouched; task EXCITITOR-CORE-AOC-19-003 marked DONE. | Implementer |
+| 2025-11-23 | Tenant Authority validation + factory tests added; EXCITITOR-CORE-AOC-19-013 remains DONE, awaiting CI test run due to local resource limits. | Implementer |
+| 2025-11-23 | Consensus refresh hosted service now skipped when `DisableConsensus=true`; refresh loop still short-circuits at runtime. Marked EXCITITOR-CORE-AOC-19-004 DONE (aggregation-only enforced). | Implementer |
+| 2025-11-23 | Implemented Cartographer linkouts endpoint `/internal/graph/linkouts` per prep (batched by PURL, deterministic ordering, truncation + cursor); marked EXCITITOR-GRAPH-21-001 DONE. | Implementer |
+| 2025-11-23 | Added TenantAuthorityOptions validator + factory tests; task EXCITITOR-CORE-AOC-19-013 set to DONE (CI run still pending due to local resource limits). | Implementer |
+| 2025-11-19 | Assigned PREP owners/dates. | Planning |
+| 2025-11-09 | Connector SUSE + Ubuntu trust provenance delivered. | Connectors Guild |
+| 2025-11-14 | LNM-21-001 schema in review. | Core Guild |
+| 2025-11-16 | Normalized sprint file to standard template and renamed to SPRINT_0119_0001_0002_excititor_ii.md. | Planning |
+| 2025-11-17 | Deprecated legacy filename `SPRINT_120_excititor_ii.md`. | Planning |
+| 2025-11-17 | Authored AGENTS.md for WebService/Core/Storage.Mongo/Worker. | Planning |
+| 2025-11-17 | Work paused: module-level AGENTS.md missing; added AGENTS-EXCITITOR-UPDATE task. | Planning |
+| 2025-11-20 | Published prep artefacts for P1–P5. | Implementer |
+| 2025-11-20 | Published prep artefacts for P6–P10. | Implementer |
+| 2025-11-21 | Began implementation: set EXCITITOR-CORE-AOC-19-004 and -19-013 to DOING; wired DisableConsensus flag in worker options and consensus refresh loop guard. | Implementer |
+| 2025-11-21 | PostConfigure added: DisableConsensus forces Refresh.Enabled=false. | Implementer |
+| 2025-11-21 | Env block: PTY commands failing with “No space left on device”; continuing via apply_patch only. | Implementer |
+| 2025-11-21 | Added consensus removal runbook (`docs/modules/excititor/operations/consensus-removal-runbook.md`). | Implementer |
+| 2025-11-21 | Added tenant Authority client factory + config docs; task 19-013 progressing. | Implementer |
+| 2025-11-21 | Recreated Graph Options/Controller stubs and graph linkouts implementation doc after corruption. | Implementer |
+| 2025-11-23 | Implemented deterministic VexLinksetExtractionService + unit tests (`dotnet test src/Excititor/__Tests/StellaOps.Excititor.Core.UnitTests/StellaOps.Excititor.Core.UnitTests.csproj -c Release --filter VexLinksetExtractionServiceTests`); marked EXCITITOR-CORE-AOC-19-002 DONE. | Implementer |
+| 2025-11-23 | Implemented graph overlays endpoint `/v1/graph/overlays` with caching + justification toggle; added overlay aggregation tests and linkset overlay cache. Set EXCITITOR-GRAPH-21-002 and EXCITITOR-GRAPH-21-005 to DONE. | Implementer |
+| 2025-11-23 | Ran `dotnet test ...StellaOps.Excititor.WebService.Tests --filter GraphOverlayFactoryTests` (TRX: `src/Excititor/__Tests/StellaOps.Excititor.WebService.Tests/TestResults/_DESKTOP-7GHGC2M_2025-11-23_23_18_38.trx`); overlay factory unit test PASS. | Implementer |
+| 2025-11-25 | Delivered VEX status summaries and tooltip observation batches (`/v1/graph/status`, `/v1/graph/observations`), updated graph options/docs, and added factory unit tests; set EXCITITOR-GRAPH-24-101/24-102 to DONE. | Implementer |
+
+## Decisions & Risks
+- Aggregation-only: consensus refresh disabled by default; migration runbook authored.
+- Tenant safety: Authority clients must be tenant-scoped.
+- Graph summaries/tooltips reuse Cartographer overlay contract; keep TTL/limit knobs aligned with overlay cache to avoid divergence.
+- Environment risk: “No space left on device” previously blocked commands; continue to monitor runner space before large test runs.
+
+## Next Checkpoints
+- 2025-11-18 | Console API review.
+- 2025-11-19 | Idempotent ingestion design review.
+- 2025-11-21 | Cartographer schema sync.
+- 2025-11-22 | Storage space remediation.

docs/implplan/archived/SPRINT_0119_0001_0003_excititor_iii.md

@@ -0,0 +1,70 @@
+# Sprint 0119_0001_0003 · Excititor Ingestion & Evidence (Phase III)
+
+## Topic & Scope
+- Stand up observation/linkset stores, conflict annotations, and events so downstream consumers can reason without Excititor consensus.
+- Publish read APIs and docs (observations/linksets) with deterministic pagination and strict RBAC.
+- Add ingest observability (metrics/SLOs) focused on evidence freshness and signature success.
+- **Working directory:** `src/Excititor` (WebService, Core, Storage); keep within module boundaries.
+
+## Dependencies & Concurrency
+- Upstream: Phase II storage/idempotency groundwork; Policy contracts for aggregation-only behavior.
+- Concurrency: Observation/linkset API work can proceed once stores stand up; conflict annotations gate events; docs depend on API shape.
+- Peers: Coordinate with Platform Events Guild for event envelopes.
+
+## Documentation Prerequisites
+- `docs/modules/excititor/architecture.md`
+- `docs/modules/excititor/README.md#latest-updates`
+- `docs/modules/excititor/operations/*`
+- `docs/modules/excititor/vex_observations.md`
+- `docs/modules/excititor/implementation_plan.md`
+- Excititor component `AGENTS.md` files (WebService, Core, Storage).
+
+## Delivery Tracker
+| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
+| --- | --- | --- | --- | --- | --- |
+| 1 | EXCITITOR-LNM-21-001 | DONE (2025-11-17) | Collections + indexes created via migration `20251117-observations-linksets`. | Excititor Storage Guild | Stand up collections with tenant guards; retire merge-era data without mutating raw content. |
+| 2 | EXCITITOR-LNM-21-002 | DONE (2025-11-17) | Disagreement fields added to linkset domain + Mongo schema/indexes. | Excititor Core Guild | Capture disagreement metadata (status/justification deltas) in linksets with confidence scores; no winner selection. |
+| 3 | EXCITITOR-LNM-21-003 | DONE (2025-11-18) | Event payload contract/factory in core; ready for Platform envelope. | Excititor Core · Platform Events Guild | Emit `vex.linkset.updated` events (observation ids, confidence, conflict summary) aggregation-only. |
+| 4 | EXCITITOR-LNM-21-201 | DONE (2025-11-25) | Observation/linkset list endpoints coded; pending tests/OpenAPI. | Excititor WebService Guild | `/vex/observations` read endpoints with advisory/product/issuer filters, deterministic pagination, strict RBAC; no derived verdicts. |
+| 5 | EXCITITOR-LNM-21-202 | DONE (2025-11-25) | List endpoint coded; export shape + docs pending. | Excititor WebService Guild | `/vex/linksets` + export endpoints surfacing alias mappings, conflict markers, provenance proofs; errors map to `ERR_AGG_*`. |
+| 6 | EXCITITOR-LNM-21-203 | DONE (2025-11-23) | After 21-202; update SDK/docs. | Excititor WebService Guild · Docs Guild | OpenAPI/SDK/examples for obs/linkset endpoints with Advisory AI/Lens-ready examples. |
+| 7 | EXCITITOR-OBS-51-001 | DONE (2025-11-23) | Define metric names + SLOs. | Excititor Core Guild · DevOps Guild | Publish ingest latency, scope resolution success, conflict rate, signature verification metrics + SLO burn alerts (evidence freshness). |
+
+## Action Tracker
+| Focus | Action | Owner(s) | Due | Status |
+| --- | --- | --- | --- | --- |
+| Stores & migrations | Finalize shard keys and migration plan for 21-001. | Storage Guild | 2025-11-18 | DONE (migration applied 2025-11-17) |
+| Conflict annotations | Schema + confidence scoring for 21-002. | Core Guild | 2025-11-19 | DONE (domain + indexes delivered 2025-11-17) |
+| Read APIs | Implement `/vex/observations` + `/vex/linksets` (21-201/202). | WebService Guild | 2025-11-22 | IN REVIEW |
+| Docs & SDK | Produce OpenAPI + SDK examples (21-203). | WebService · Docs Guild | 2025-11-23 | DONE (2025-11-23) |
+| Metrics/SLOs | Define and wire ingest metrics (OBS-51-001). | Core · DevOps Guild | 2025-11-24 | DONE (2025-11-23) |
+
+## Execution Log
+| Date (UTC) | Update | Owner |
+| --- | --- | --- |
+| 2025-11-16 | Normalized sprint file to standard template and renamed to SPRINT_0119_0001_0003_excititor_iii.md; pending staffing. | Planning |
+| 2025-11-17 | Added migration `20251117-observations-linksets` (collections + indexes for observations/linksets). | Storage Guild |
+| 2025-11-17 | Added linkset disagreement schema (domain + Mongo records/index) fulfilling 21-002. | Core Guild |
+| 2025-11-18 | Added `vex.linkset.updated` payload contract + factory (21-003). | Core Guild |
+| 2025-11-18 | Implemented Mongo observation lookup + registration (unblocks 21-201). | Storage Guild |
+| 2025-11-18 | Added `/v1/vex/observations` + `/v1/vex/linksets` list endpoints (IN REVIEW) backed by Mongo lookup. | WebService Guild |
+| 2025-11-18 | Added `/v1/vex/observations` and `/v1/vex/linksets` list endpoints (tenant-scoped, cursor pagination) backed by Mongo lookup. | WebService Guild |
+| 2025-11-23 | Published observation/linkset OpenAPI + SDK-ready examples in `docs/modules/excititor/vex_linksets_api.md`; marked EXCITITOR-LNM-21-203 DONE. | Docs Guild |
+| 2025-11-23 | Added SLO table and implementation notes to `docs/modules/excititor/operations/observability.md`; marked EXCITITOR-OBS-51-001 DONE. | Excititor Core |
+| 2025-11-25 | Closed list endpoints (21-201/21-202) after doc alignment; sprint ready for archive. | Implementer |
+
+## Decisions & Risks
+- **Decisions**
+  - All new endpoints remain aggregation-only; no derived verdicts.
+  - Events must reuse Platform event envelope and tenant guards.
+- **Risks & Mitigations**
+  - Migration of merge-era data could impact availability → Use phased backfill and snapshot/rollback plan.
+  - Missing SLO definitions delays evidence freshness promises → Draft initial targets with Ops while metrics wire up.
+  - Observation persistence/lookup not yet implemented → Blocks read APIs; mitigation: define store contract and stub implementation before API work resumes.
+
+## Next Checkpoints
+| Date (UTC) | Session / Owner | Goal | Fallback |
+| --- | --- | --- | --- |
+| 2025-11-18 | Storage design review | Approve shard keys + migration plan for 21-001. | Use temporary staging collections if approval slips. |
+| 2025-11-20 | Events contract sync (Platform) | Lock `vex.linkset.updated` payload. | Emit internal-only preview topic until contract finalized. |
+| 2025-11-23 | API/doc draft review | Validate observation/linkset OpenAPI + SDK examples. | Ship behind feature flag if minor gaps. |