prep docs and service updates

docs/modules/excititor/operations/consensus-removal-runbook.md

@@ -0,0 +1,31 @@
+# Excititor Consensus Removal Runbook (AOC-19-004)
+
+- **Date:** 2025-11-21
+- **Scope:** EXCITITOR-CORE-AOC-19-004
+- **Goal:** Eliminate legacy consensus/merged severity fields so Excititor remains aggregation-only.
+
+## Cutover steps
+1) **Freeze consensus refresh** — `DisableConsensus=true` (default) forces refresh loop off. Keep this enabled during migration.
+2) **Schema cleanup** — migrate collections to remove or null legacy fields:
+   - `vex_consensus` / `vex_consensus_holds`: drop/ignore fields `consensusDigest`, `policyVersion`, `policyRevisionId`, `policyDigest`, `summary`, `signals`, `status` (merged) once Policy takes over.
+   - `vex_observations` / materialized exports: ensure no merged severity/status fields are written.
+   - `vex_mirror` exports: stop emitting consensus JSON; retain raw observations only.
+3) **Telemetry:** emit counter `excititor.ingest.consensus.disabled` (tags `tenant`, `source`, `connectorId`) once per batch to prove cutover.
+4) **Guards:** AOC guards reject any incoming/derived field in `{mergedSeverity, consensusScore, computedStatus}`.
+5) **Backfill:** run one-off job to set `consensusDisabled=true` on legacy records and remove merged fields without touching raw observations.
+6) **Verification:** regression checklist (per tenant):
+   - No writes to `vex_consensus*` collections after cutover.
+   - Ingest + export fixtures show only raw observations/linksets; snapshots deterministic.
+   - Telemetry counter present; absence of consensus refresh logs.
+
+## Config
+```
+Excititor:Worker:
+  DisableConsensus: true   # keep true post-cutover
+```
+
+## Test plan (after disk space is restored)
+- Unit: AOC guard rejects merged fields.
+- Integration (Mongo2Go): ingest batch containing merged fields → rejected; telemetry counter increments.
+- Worker: start with DisableConsensus=true → consensus refresh loop does not schedule; log once at startup.
+

docs/modules/excititor/operations/graph-linkouts-implementation.md

@@ -0,0 +1,42 @@
+# Excititor · Graph Linkouts & Overlays — Implementation Notes (Graph 21-001/002/005)
+
+- **Date:** 2025-11-21
+- **Scope:** EXCITITOR-GRAPH-21-001, EXCITITOR-GRAPH-21-002, EXCITITOR-GRAPH-21-005
+- **Status:** Implementation guidance (storage wiring pending).
+
+## Endpoints
+1) **Linkouts (21-001)**  
+   - `POST /internal/graph/linkouts`  
+   - Body: `tenant`, `purls[]` (max 500), `includeJustifications?`, `includeProvenance?`  
+   - Response: ordered by input `purls`; each item includes `advisories[]` (`advisoryId`, `source`, `status`, `justification?`, `modifiedAt`, `evidenceHash`, `connectorId`, `dsseEnvelopeHash?`) plus `conflicts[]`; `notFound[]`.
+
+2) **Overlays (21-002)**  
+   - `GET /v1/graph/overlays?purl=<purl>&purl=<purl>&includeJustifications=true|false`  
+   - Response per PURL: `summary` counts (`open`, `not_affected`, `under_investigation`, `no_statement`), `latestModifiedAt`, `justifications[]` (unique, sorted), `provenance` (`sources[]`, `lastEvidenceHash`), `cached`, `cacheAgeMs`.
+
+## Storage & Indexes (21-005)
+- `vex_observations` indexes:  
+  - `{ tenant: 1, component.purl: 1, advisoryId: 1, source: 1, modifiedAt: -1 }`  
+  - Sparse `{ tenant: 1, component.purl: 1, status: 1 }`
+- Optional materialized `vex_overlays` cache: unique `{ tenant: 1, purl: 1 }`, TTL on `cachedAt` driven by `excititor:graph:overlayTtlSeconds` (default 300s).
+
+## Determinism
+- Ordering: input PURL order → `advisoryId` → `source` for linkouts; overlays follow input order.
+- Truncation: max 200 advisories per PURL; when truncated, include `truncated: true` and `nextCursor` (`advisoryId`, `source`).
+
+## Config knobs
+- `excititor:graph:overlayTtlSeconds` (default 300)
+- `excititor:graph:maxPurls` (default 500)
+- `excititor:graph:maxAdvisoriesPerPurl` (default 200)
+
+## Telemetry
+- Counter `excititor.graph.linkouts.requests` tags: `tenant`, `includeJustifications`, `includeProvenance`.
+- Counter `excititor.graph.overlays.cache` tags: `tenant`, `hit` (`true|false`).
+- Histogram `excititor.graph.linkouts.latency.ms` tags: `tenant`.
+
+## Steps to implement
+- Bind `GraphOptions` to `Excititor:Graph`.
+- Add endpoints to WebService with tenant guard; enforce limits.
+- Implement overlay cache with deterministic sort; respect TTL; surface `cached` + `cacheAgeMs`.
+- Backfill Mongo indexes above.
+- Integration tests (WebApplicationFactory + Mongo2Go) for ordering, truncation, cache metadata, tenant isolation.

docs/modules/excititor/operations/tenant-authority-client.md

@@ -0,0 +1,39 @@
+# Excititor Tenant Authority Client (AOC-19-013)
+
+- **Date:** 2025-11-21
+- **Scope:** EXCITITOR-CORE-AOC-19-013
+- **Files:** `src/Excititor/StellaOps.Excititor.Worker/Auth/TenantAuthorityClientFactory.cs`
+
+## Contract
+- Every outbound Authority call must carry `X-Tenant` header and use tenant-specific base URL.
+- Base URLs and optional client credentials are configured under `Excititor:Authority:` with per-tenant keys.
+- Factory throws when tenant is missing or not configured to prevent cross-tenant leakage.
+
+## Configuration shape
+```json
+{
+  "Excititor": {
+    "Authority": {
+      "BaseUrls": {
+        "alpha": "https://authority.alpha.local/",
+        "bravo": "https://authority.bravo.local/"
+      },
+      "ClientIds": {
+        "alpha": "alpha-client-id"
+      },
+      "ClientSecrets": {
+        "alpha": "alpha-secret"
+      }
+    }
+  }
+}
+```
+
+## Implementation notes
+- `TenantAuthorityClientFactory` (worker) enforces tenant presence and configured base URL; adds `Accept: application/json` and `X-Tenant` headers.
+- Registered in DI via `Program.cs` with options binding to `Excititor:Authority`.
+- Intended to be reused by WebService/Worker components once disk space block is resolved.
+
+## Next steps
+- Wire factory into services that call Authority (WebService + Worker jobs), replacing any tenant-agnostic HttpClient usages.
+- Add integration tests to ensure cross-tenant calls reject when config missing or header mismatched.

docs/modules/excititor/prep/2025-11-20-consensus-removal-prep.md

@@ -0,0 +1,27 @@
+# Excititor · Consensus Removal Prep (AOC-19-004)
+
+- **Date:** 2025-11-20
+- **Working directory:** `src/Excititor/__Libraries/StellaOps.Excititor.Core` + `src/Excititor/StellaOps.Excititor.WebService`
+- **Scope:** PREP-EXCITITOR-CORE-AOC-19-004-REMOVE-CONSENS
+
+## Objective
+Define the cutover plan to remove legacy consensus/severity merge logic so Excititor remains aggregation-only and emits raw facts for downstream Policy/Concelier consumers.
+
+## Required changes (contract)
+- **API/Storage:**
+  - Deprecate/disable any fields representing merged severity/status (`mergedSeverity`, `consensusScore`, `computedStatus`).
+  - Retain raw source fields: `status`, `justification`, `impact`, `affects`, `references`, `notes`, `provenance`, `reconciledFrom`.
+  - Add boolean `consensusDisabled: true` to existing documents during migration for audit.
+- **Ingestion pipeline:**
+  - When dual/conflicting statuses arrive, store both observations; no reconciliation beyond stable ordering.
+  - Maintain deterministic ordering when multiple observations share `(tenant, advisoryId, component)` — sort by `ingestedAt`, then `source`, then `evidenceHash`.
+- **Feature flag:** `excititor:aoc:disableConsensus` default `true`; only temporary `false` allowed for rollback during migration.
+- **Telemetry:** counter `excititor.ingest.consensus.disabled` tagged by `tenant`, `source`, `connectorId`; increment once per batch after flag applied.
+
+## Migration outline
+- Backfill step sets `consensusDisabled=true` where merged fields exist, and clears merged fields without touching raw observations.
+- Tests must assert merged fields are absent/null after migration and ingestion flows do not write them.
+
+## Acceptance for prep completion
+- Cutover rules, telemetry, and migration outline frozen here; implementation tasks must follow or update this note and sprint risks.
+

docs/modules/excititor/prep/2025-11-20-console-cache-rbac-prep.md

@@ -0,0 +1,24 @@
+# Console Cache & RBAC Prep — PREP-EXCITITOR-CONSOLE-23-003-DEPENDS-ON-23-001
+
+Status: Draft (2025-11-20)
+Owners: Excititor WebService Guild
+Scope: Capture caching, RBAC, and precedence-context requirements for console VEX lookups once the base contract (23-001) is defined.
+
+## Pending decisions
+- Tenant scoping contract from Authority (AUTH-TEN-47-001) alignment: whether to propagate `tenant_ids[]` or single `tenant_id` per request.
+- Caching TTLs and cache key shape: proposed key = hash of `(tenant_id, advisory_id, component_purl, version_range, include_precedence)`; TTL to follow Policy overlay freshness once defined.
+- Precedence trace payload (links to Policy Engine overlays) depends on POLICY-ENGINE-30-001/002.
+
+## Proposed endpoints (draft)
+- `GET /console/vex/cache/entries?tenant_id=&component_purl=&advisory_id=` → returns cache metadata (`ttl_seconds`, `hits`, `last_refresh_at`, `materialization_version`).
+- `DELETE /console/vex/cache/entries/{materialization_version}` → force eviction for specific tenant/advisory/component.
+
+## RBAC sketch
+- Roles: `console.viewer`, `console.operator`, `console.admin`.
+- Permissions:
+  - viewer: read-only to `/console/vex` + counters.
+  - operator: can invalidate cache and request refresh.
+  - admin: can set cache policy per tenant/project.
+
+## Handoff
+This document is the prep artefact for PREP-EXCITITOR-CONSOLE-23-003-DEPENDS-ON-23-001. Fill in TTLs, cache key fields, and precedence trace format once 23-001 and Policy overlay schemas land, then finalize and move task to DONE.

docs/modules/excititor/prep/2025-11-20-console-counters-prep.md

@@ -0,0 +1,23 @@
+# Console Counters Prep — PREP-EXCITITOR-CONSOLE-23-002-DEPENDS-ON-23-001
+
+Status: Draft (2025-11-20)
+Owners: Excititor WebService Guild
+Scope: Define the counter surfaces required for console delta cards, pending the `/console/vex` contract.
+
+## Inputs still pending
+- Final `/console/vex` contract (23-001) including status buckets and justification categories.
+- Source-of-truth metrics/telemetry names from Policy Engine overlays (POLICY-ENGINE-30-001 once available).
+
+## Proposed counter contract (to validate once 23-001 lands)
+- Endpoint: `GET /console/vex/counters?tenant_id=&component_purl=&advisory_id=&since=`
+- Response fields:
+  - `total`, `affected`, `not_affected`, `under_investigation`, `mitigated`, `unknown`
+  - `delta_since` (ISO-8601) and `window_seconds`
+  - `evidence_refs[]` (DSSE hashes or linkset ids) optional
+- Metrics to emit:
+  - Gauge `console_vex_active_total{tenant,status}`
+  - Counter `console_vex_delta_total{tenant,status}` with `delta_since` label
+- Determinism: counters computed from immutable materialized views keyed by `(tenant, advisory_id, component_purl)`; avoid wall-clock beyond `since` parameter.
+
+## Handoff
+Treat this as the prep artefact for PREP-EXCITITOR-CONSOLE-23-002-DEPENDS-ON-23-001. Update once status buckets are frozen in 23-001 and Policy metrics are published; then finalize endpoints and samples.

docs/modules/excititor/prep/2025-11-20-console-vex-contract-prep.md

@@ -0,0 +1,23 @@
+# Console / VEX Contract Prep — PREP-EXCITITOR-CONSOLE-23-001-AWAITING-CONCRE
+
+Status: Draft (2025-11-20)
+Owners: Excititor WebService Guild · BE-Base Platform Guild
+Scope: Capture the required `/console/vex` API contract inputs so downstream tasks can proceed once the concrete spec lands.
+
+## Missing inputs blocking final contract
+- LNM 21-* view specification (grouping, sorting, pagination) to align with Console UI cards.
+- Final status chip taxonomy and precedence rules from Policy/Concelier overlays.
+- SSE channel naming + retry/heartbeat semantics shared with Scheduler/Policy streams.
+
+## Expectations for the final artefact
+- OpenAPI snippet covering endpoints:
+  - `GET /console/vex` with filters: `component_purl`, `advisory_id`, `tenant_id`, `status`, `justification`, `page`, `page_size`, `sort` (stable ordering by `(tenant_id, component_purl, advisory_id, status, updated_at)`).
+  - `GET /console/vex/{advisory_id}` returning grouped statements, precedence trace pointer, provenance links (DSSE hash + linkset id), and tenant scoping.
+- Response envelope: standard console error schema once WEB-OAS-61-002 is frozen; until then use draft shape with `error`, `message`, `trace_id`.
+- Determinism: results ordered by `(tenant_id, advisory_id, component_purl, version_range)`; pagination stable under new data.
+
+## Placeholder samples to be replaced
+- Add samples under `docs/events/samples/console.vex@draft.json` once view spec is provided.
+
+## Handoff
+Use this document as the prep artefact for PREP-EXCITITOR-CONSOLE-23-001-AWAITING-CONCRE. Update once LNM view spec and SSE envelope land; then freeze the OpenAPI excerpt and move the sprint task to DONE.

docs/modules/excititor/prep/2025-11-20-graph-21-001-prep.md

@@ -0,0 +1,32 @@
+# Excititor · Graph Linkouts Prep (GRAPH-21-001)
+
+- **Date:** 2025-11-20
+- **Scope:** PREP-EXCITITOR-GRAPH-21-001-NEEDS-CARTOGRAPHE
+- **Working directory:** `src/Excititor/__Libraries/StellaOps.Excititor.Core` + `src/Excititor/StellaOps.Excititor.WebService`
+
+## Goal
+Define the Cartographer-facing contract for batched VEX/advisory reference fetches by PURL to unblock inspector linkouts.
+
+## Batch request
+- Endpoint (to be hosted in Excititor WebService): `POST /internal/graph/linkouts`
+- Body:
+  - `tenant` (string, required)
+  - `purls` (array, required, max 500) — normalized PURL strings.
+  - `includeJustifications` (bool, default false)
+  - `includeProvenance` (bool, default true)
+- Idempotency key: `tenant` + SHA256 of sorted `purls` list.
+
+## Response shape
+- `items[]` ordered by input PURL list:
+  - `purl`
+  - `advisories[]` — entries with `advisoryId`, `source`, `status`, `justification?`, `modifiedAt`, `evidenceHash`, `connectorId`, `dsseEnvelopeHash?`.
+  - `conflicts[]` — optional disagreements (status/justification) with `source`, `observedAt`, `evidenceHash`.
+- `notFound[]` — PURLs with no VEX observations.
+
+## Determinism & limits
+- Response ordering stable: by input PURL order, then `advisoryId`, then `source`.
+- Max rows: cap `advisories` to 200 per PURL; truncate with `truncated: true` flag and `nextCursor` (advisoryId, source).
+
+## Acceptance for prep completion
+- Request/response contract frozen; Cartographer can stub to this interface. Downstream GRAPH-21-001 implementation must adhere or update doc + sprint risks.
+

docs/modules/excititor/prep/2025-11-20-graph-21-002-prep.md

@@ -0,0 +1,23 @@
+# Excititor · Graph Overlay Prep (GRAPH-21-002)
+
+- **Date:** 2025-11-20
+- **Depends on:** GRAPH-21-001 linkout contract
+- **Working directory:** `src/Excititor/StellaOps.Excititor.WebService`
+
+## Overlay payload
+- Aggregates output of GRAPH-21-001 into overlay items for inspectors:
+  - `purl`
+  - `summary`: `open`, `not_affected`, `under_investigation`, `no_statement` counts
+  - `latestModifiedAt` (ISO-8601 UTC)
+  - `justifications[]` (optional) — unique justification codes present for the PURL
+  - `provenance` — `sources[]` (unique source IDs), `lastEvidenceHash`
+- Endpoint: `GET /v1/graph/overlays?purl=<purl>[&purl=...]&includeJustifications=true|false`
+- Sorting: results ordered by input PURL list; within overlays, `justifications` sorted ascending.
+
+## Caching
+- Cache key: tenant + sorted PURL list + `includeJustifications` flag; ttl 5 minutes default, configurable `excititor:graph:overlayTtlSeconds`.
+- Cache metadata returned: `cached: true|false`, `cacheAgeMs`.
+
+## Acceptance for prep completion
+- Overlay shape and caching contract defined; implementation can proceed once GRAPH-21-001 is available.
+

docs/modules/excititor/prep/2025-11-20-graph-21-005-prep.md

@@ -0,0 +1,21 @@
+# Excititor · Graph Indexes Prep (GRAPH-21-005)
+
+- **Date:** 2025-11-20
+- **Depends on:** GRAPH-21-002 overlays
+- **Working directory:** `src/Excititor/__Libraries/StellaOps.Excititor.Storage.Mongo`
+
+## Index plan
+- Collection: `vex_observations`
+  - Compound index `{ tenant: 1, component.purl: 1, advisoryId: 1, source: 1, modifiedAt: -1 }` (supports overlay queries and truncation cursor).
+  - Sparse index `{ tenant: 1, component.purl: 1, status: 1 }` for summary counts.
+- Collection: `vex_overlays` (materialized cache, optional)
+  - Index `{ tenant: 1, purl: 1 }` unique.
+  - TTL index on `cachedAt` configurable via `excititor:graph:overlayTtlSeconds`.
+
+## Determinism
+- Materialization job must sort observations as per GRAPH-21-001 ordering before writing overlays so pagination/cursors align.
+- TTL applied identically across tenants; default 300s, override allowed via config but must be documented.
+
+## Acceptance for prep completion
+- Index keys and TTL knobs defined; downstream storage tasks can implement without further contract churn.
+

docs/modules/excititor/prep/2025-11-20-linkset-extraction-prep.md

@@ -0,0 +1,18 @@
+# Linkset Extraction Prep — PREP-EXCITITOR-CORE-AOC-19-002-LINKSET-EXTRAC
+
+Status: Draft (2025-11-20)
+Owners: Excititor Core Guild
+Scope: Identify the extraction rules and ordering needed to produce linksets from VEX/advisory inputs before idempotent raw upsert work starts.
+
+## Required content to unblock
+- Canonical linkset schema version (pending Cartographer/Concelier alignment); need field list and conflict markers.
+- Source ranking/precedence table shared with Concelier LNM 21-002 fixtures.
+
+## Proposed extraction rules (draft)
+- Inputs: advisory documents (component PURLs, version ranges, references, severities, CVSS vectors); output: linkset entries with `advisory_id`, `component_purl`, `version_range`, `references[]`, `severity`, `cvss`.
+- Ordering: sort entries by `(component_purl, advisory_id, version_range)`; within references, sort lexicographically.
+- Conflict handling: if multiple sources disagree, emit `conflicts[]` with `source`, `field`, `reason`; never collapse values.
+- Determinism: no wall-clock; timestamps only from source payloads (UTC ISO-8601) and preserved as-is.
+
+## Handoff
+Treat this as the prep artefact for PREP-EXCITITOR-CORE-AOC-19-002-LINKSET-EXTRAC. Once the shared linkset schema and precedence table land, finalize the rules and move the sprint task to DONE.

docs/modules/excititor/prep/2025-11-20-raw-upsert-idempotency-prep.md

@@ -0,0 +1,18 @@
+# Raw Upsert Idempotency Prep — PREP-EXCITITOR-CORE-AOC-19-003-BLOCKED-ON-19
+
+Status: Draft (2025-11-20)
+Owners: Excititor Core Guild
+Scope: Document the idempotent raw upsert and versioning requirements once linkset extraction (19-002) is defined.
+
+## Pending inputs
+- Linkset schema and conflict markers from 19-002.
+- Storage model choice (Mongo vs Postgres) and required unique keys per tenant/advisory/component/version_range.
+
+## Proposed rules (draft)
+- Unique key: `(tenant_id, advisory_id, component_purl, version_range, source)`; store a monotonic `revision` and `ingested_at` (UTC) for traceability.
+- Idempotency: compute content hash over canonicalized payload; if identical, no-op; otherwise append new revision with `supersedes` pointer.
+- Append-only log: keep prior revisions for audit; consumers read latest by hash or highest revision per key.
+- Determinism: canonical JSON ordering; stable sorting by `(tenant_id, advisory_id, component_purl, version_range, revision)`.
+
+## Handoff
+Use this as the prep artefact for PREP-EXCITITOR-CORE-AOC-19-003-BLOCKED-ON-19. Finalize once 19-002 freezes schema and storage choice; then wire migrations/indexes accordingly.

docs/modules/excititor/prep/2025-11-20-tenant-authority-prep.md

@@ -0,0 +1,23 @@
+# Excititor · Tenant-Aware Authority Prep (AOC-19-013)
+
+- **Date:** 2025-11-20
+- **Scope:** PREP-EXCITITOR-CORE-AOC-19-013-SEED-TENANT-AW
+- **Working directory:** `src/Excititor/StellaOps.Excititor.WebService`, `src/Excititor/StellaOps.Excititor.Worker`, `src/Excititor/__Libraries/StellaOps.Excititor.Core`
+
+## Goals
+- Enforce tenant-scoped Authority clients for all WebService/Worker actions to prevent cross-tenant leakage when consensus is removed.
+- Provide deterministic fixture/seed guidance for e2e tests.
+
+## Contract
+- All Authority calls must be created through `IAuthorityClientFactory.Create(tenantId)`; factories that lack tenant must throw.
+- Configuration: `excititor:authority:baseUrl`, `excititor:authority:audience`, per-tenant `clientId/clientSecret` retrieved via internal secret resolver (no cross-tenant cache).
+- Headers: include `X-Tenant` on every outbound request; reject response lacking matching `tenant` claim.
+- Telemetry: meter `StellaOps.Excititor.Auth` counters `authority.call` tagged `tenant`, `operation`, `result` (`ok|unauthorized|forbidden|error`).
+
+## Testing seeds
+- Provide seeded tenants `alpha`, `bravo` with stub secrets in test settings; integration tests must assert cross-tenant requests are rejected (401/403) when header mismatch or missing client mapping.
+- Fake Authority server returns tenant claim; tests validate enforcement and logs.
+
+## Acceptance for prep completion
+- Tenant-scoped client contract, config keys, and test seeds documented; downstream tasks 19-013 can proceed using this as authority.
+