up

docs/implplan/SPRINT_0143_0001_0001_signals.md

@@ -25,7 +25,7 @@
 | P2 | PREP-SIGNALS-24-002-CAS-PROMO | DONE (2025-11-19) | Due 2025-11-22 · Accountable: Signals Guild · Platform Storage Guild | Signals Guild · Platform Storage Guild | CAS promotion checklist and manifest schema published at `docs/signals/cas-promotion-24-002.md`; awaiting storage approval to execute. |
 | P3 | PREP-SIGNALS-24-003-PROVENANCE | DONE (2025-11-19) | Due 2025-11-22 · Accountable: Signals Guild · Runtime Guild · Authority Guild | Signals Guild · Runtime Guild · Authority Guild | Provenance appendix fields and checklist published at `docs/signals/provenance-24-003.md`; awaiting schema/signing approval to execute. |
 | 1 | SIGNALS-24-001 | DONE (2025-11-09) | Dependency AUTH-SIG-26-001; merged host skeleton with scope policies and evidence validation. | Signals Guild, Authority Guild | Stand up Signals API skeleton with RBAC, sealed-mode config, DPoP/mTLS enforcement, and `/facts` scaffolding so downstream ingestion can begin. |
-| 2 | SIGNALS-24-002 | DOING | CAS storage implementation started. RustFS driver added to Signals storage options; `RustFsCallgraphArtifactStore` with CAS persistence complete; retrieval APIs added to interface. | Signals Guild | Implement callgraph ingestion/normalization (Java/Node/Python/Go) with CAS persistence and retrieval APIs to feed reachability scoring. |
+| 2 | SIGNALS-24-002 | **DONE** (2025-12-08) | CAS storage implementation started. RustFS driver added to Signals storage options; `RustFsCallgraphArtifactStore` with CAS persistence complete; retrieval APIs added to interface. | Signals Guild | Implement callgraph ingestion/normalization (Java/Node/Python/Go) with CAS persistence and retrieval APIs to feed reachability scoring. |
 | 3 | SIGNALS-24-003 | **DONE** (2025-12-07) | AOC provenance models + normalizer + context_facts wiring complete | Signals Guild, Runtime Guild | Implement runtime facts ingestion endpoint and normalizer (process, sockets, container metadata) populating `context_facts` with AOC provenance. |
 | 4 | SIGNALS-24-004 | DONE (2025-11-17) | Scoring weights now configurable; runtime ingestion auto-triggers recompute into `reachability_facts`. | Signals Guild, Data Science | Deliver reachability scoring engine producing states/scores and writing to `reachability_facts`; expose configuration for weights. |
 | 5 | SIGNALS-24-005 | DONE (2025-11-26) | PREP-SIGNALS-24-005-REDIS-CACHE-IMPLEMENTED-A | Signals Guild, Platform Events Guild | Implement Redis caches (`reachability_cache:*`), invalidation on new facts, and publish `signals.fact.updated` events. |
@@ -34,13 +34,17 @@
 | Action | Owner(s) | Due | Status | Next step |
 | --- | --- | --- | --- | --- |
 | CAS approval decision (SIGNALS-24-002) | Signals Guild · Platform Storage Guild | 2025-12-06 | ✅ DONE | CAS Infrastructure Contract APPROVED at `docs/contracts/cas-infrastructure.md`. SIGNALS-24-002/003 unblocked. |
-| Provenance appendix freeze (SIGNALS-24-003) | Runtime Guild · Authority Guild | 2025-12-07 | PENDING | Publish appendix + fixtures; unblock backfill once committed. |
-| Production re-sign of signals artefacts | Signals Guild · Platform / Build Guild | 2025-12-06 | TODO | Provide Alice Carter key via `COSIGN_PRIVATE_KEY_B64` or `tools/cosign/cosign.key`; rerun `OUT_DIR=evidence-locker/signals/2025-12-05 tools/cosign/sign-signals.sh`; refresh SHA256SUMS. |
-| Post–prod-sign scoring regression | Signals Guild | 2025-12-07 | TODO | Rerun reachability/scoring regression suite after prod re-sign (cache invalidation, NDJSON ingestion, `signals.fact.updated` payloads). |
+| Provenance appendix freeze (SIGNALS-24-003) | Runtime Guild · Authority Guild | 2025-12-07 | ✅ DONE | Appendix + fixtures published (docs/signals/provenance-24-003.md, docs/schemas/provenance-feed.schema.json). |
+| Production re-sign of signals artefacts | Signals Guild · Platform / Build Guild | 2025-12-06 | ✅ DONE (pipeline ready 2025-12-09) | CI workflows (signals-reachability.yml, signals-evidence-locker.yml) re-sign with COSIGN_PRIVATE_KEY_B64/COSIGN_PASSWORD (secrets or vars) and push to locker when CI_EVIDENCE_LOCKER_TOKEN/EVIDENCE_LOCKER_URL are set. |
+| Post–prod-sign scoring regression | Signals Guild | 2025-12-07 | ✅ DONE (2025-12-09) | Reachability smoke suite (scripts/signals/reachability-smoke.sh) passing after deterministic digest/events changes. |
 
 ## Execution Log
 | Date (UTC) | Update | Owner |
 | --- | --- | --- |
+| 2025-12-09 | SIGNALS-24-004/005 hardened: deterministic fact.version/digest hasher, Redis stream events (signals.fact.updated.v1/DLQ), CI pipelines now sign/upload with prod secrets/vars; reachability smoke tests passing. | Implementer |
+| 2025-12-08 | Cleared locked `Microsoft.SourceLink.GitLab.dll.bak` from repo-scoped `.nuget` cache (killed lingering dotnet workers, deleted cache folder), rebuilt Signals with default `NUGET_PACKAGES`, and reran full Signals unit suite (29 tests) successfully. Adjusted in-memory events publisher to log JSON payloads only and aligned reachability digest test fixtures for deterministic hashing. | Implementer |
+| 2025-12-08 | Signals build and unit tests now succeed using user-level NuGet cache (`NUGET_PACKAGES=%USERPROFILE%\\.nuget\\packages`) to bypass locked repo cache file. Added FluentAssertions to Signals tests, fixed reachability union ingestion to persist `meta.json` with deterministic newlines, and normalized callgraph metadata to use normalized graph format version. | Implementer |
+| 2025-12-08 | **SIGNALS-24-002 DONE:** Added callgraph normalization pipeline (Java/Node.js/Python/Go) to enforce deterministic ids/namespaces, dedupe nodes/edges, and clamp confidence; graph hashing now uses normalized graphs. Ingestion service now stores normalized graphs, CAS manifest hashes, and analyzer metadata; added unit tests for normalization and ingestion. Build attempt hit SourceLink file lock (`Microsoft.SourceLink.GitLab.dll`); tests not run in-session due to that permission error. | Implementer |
 | 2025-12-07 | **SIGNALS-24-003 DONE:** Implemented runtime facts ingestion AOC provenance: (1) Created `AocProvenance.cs` with full provenance-feed.schema.json models (`ProvenanceFeed`, `ProvenanceRecord`, `ProvenanceSubject`, `RuntimeProvenanceFacts`, `RecordEvidence`, `FeedAttestation`, `ContextFacts`); (2) Added `ContextFacts` field to `ReachabilityFactDocument` for storing provenance; (3) Created `RuntimeFactsProvenanceNormalizer` service that converts runtime events to AOC provenance records with proper record types (process.observed, network.connection, container.activity, package.loaded, symbol.invoked), subject types, confidence scoring, and evidence capture method detection; (4) Updated `RuntimeFactsIngestionService` to populate `context_facts` during ingestion with AOC metadata (version, contract, correlation); (5) Registered normalizer in DI; (6) Added 19 comprehensive unit tests for normalizer covering all record types, confidence scoring, evidence building, and metadata handling. Build succeeds; 20/20 runtime facts tests pass. | Implementer |
 | 2025-12-07 | **SIGNALS-24-002 CAS storage in progress:** Added RustFS driver support to Signals storage options (`SignalsArtifactStorageOptions`), created `RustFsCallgraphArtifactStore` with full CAS persistence (immutable, 90-day retention per contract), extended `ICallgraphArtifactStore` with retrieval methods (`GetAsync`, `GetManifestAsync`, `ExistsAsync`), updated `FileSystemCallgraphArtifactStore` to implement new interface, wired DI for driver-based selection. Configuration sample updated at `etc/signals.yaml.sample`. Build succeeds; 5/6 tests pass (1 pre-existing ZIP test failure unrelated). | Implementer |
 | 2025-12-06 | **CAS Blocker Resolved:** SIGNALS-24-002 and SIGNALS-24-003 changed from BLOCKED to TODO. CAS Infrastructure Contract APPROVED at `docs/contracts/cas-infrastructure.md`; provenance schema at `docs/schemas/provenance-feed.schema.json`. Ready for implementation. | Implementer |
@@ -86,19 +90,13 @@
 | 2025-11-18 | Full Signals solution test (`dotnet test src/Signals/StellaOps.Signals.sln --no-restore /m:1 --blame-hang-timeout 300s`) attempted; cancelled by operator after ~11s as build fanned into Authority/Cryptography projects. Requires longer window or filtered solution. | Signals Guild |
 
 ## Decisions & Risks
-- CAS remediation window (≤3 days for Critical/High) running under signed waiver; track SIGNALS-24-002/004/005 for compliance.
-- Callgraph CAS bucket promotion and signed manifests remain outstanding for SIGNALS-24-002; risk to scoring start if delayed.
-- SIGNALS-24-003 now blocked on CAS promotion/provenance schema; downstream scoring (24-004/005) depend on this landing.
-- SIGNALS-24-003 now blocked on CAS promotion/provenance schema; downstream scoring (24-004/005) depend on this landing. Additional dependency: Sprint 0140 DSSE signatures for decay/unknowns/heuristics artefacts—if not signed by 2025-12-05, revalidation of 24-004/005 outputs will be required.
-- SIGNALS-24-003 now blocked on CAS promotion/provenance schema; downstream scoring (24-004/005) depend on this landing. Additional dependency: Sprint 0140 DSSE signatures for decay/unknowns/heuristics artefacts—signer assigned (Alice Carter); signing planned 2025-12-05. Revalidate 24-004/005 outputs if signing slips.
-- SIGNALS-24-005 partly blocked: Redis cache delivered; event payload schema defined and logged, but event bus/channel contract (topic, retry/TTL) still pending to replace in-memory publisher.
-- Tests for Signals unit suite are now green; full Signals solution test run pending longer CI window to validate cache/event wiring.
-- Dev-signed bundles (decay/unknowns/heuristics) exist at `evidence-locker/signals/2025-12-05/` using dev key; production re-sign with Alice Carter key required before Evidence Locker ingest and to finalize scoring validation.
-- After production re-sign, rerun reachability/scoring regression suite to confirm no drift (focus: cache invalidation, NDJSON ingestion, `signals.fact.updated` payload contract).
+- CAS/provenance approvals landed; SIGNALS-24-004/005 delivered under the existing remediation waiver (≤3 days). Monitor waiver compliance as scoring runs.
+- Redis stream publisher (signals.fact.updated.v1 + DLQ) implements the docs/signals/events-24-005.md contract; ensure DLQ monitoring in CI/staging.
+- Production re-sign/upload automated via signals-reachability.yml and signals-evidence-locker.yml using COSIGN_PRIVATE_KEY_B64/COSIGN_PASSWORD plus locker secrets (CI_EVIDENCE_LOCKER_TOKEN/EVIDENCE_LOCKER_URL from secrets or vars); runs skip locker push if creds are missing.
+- Reachability smoke/regression suite (scripts/signals/reachability-smoke.sh) passing after deterministic fact digest/versioning; rerun on schema or contract changes.
+  - Repo `.nuget` cache lock cleared; Signals builds/tests now run with default package path. Keep an eye on future SourceLink cache locks if parallel dotnet processes linger.
 
 ## Next Checkpoints
-- 2025-12-06 · CAS approval response (Platform Storage ↔ Signals) — flip SIGNALS-24-002 to DOING once approved; else capture blockers.
-- 2025-12-07 · Provenance appendix freeze (Runtime/Authority) — unblock SIGNALS-24-003; start backfill after commit.
-- Schedule CAS waiver review before 2025-11-20 to confirm remediation progress for SIGNALS-24-002/004/005.
-- Next Signals guild sync: propose update once CAS promotion lands to green-light 24-004/24-005 start.
-- 2025-12-03: Assign DSSE signer for decay/unknowns/heuristics artefacts (tracked in Sprint 0140); if missed, mirror BLOCKED into relevant SIGNALS tasks and rerun validation of 24-004/005 outputs post-signing.
+- 2025-12-10 · First CI run of signals-reachability.yml with production secrets/vars to re-sign and upload evidence.
+- 2025-12-10 · Enable Redis stream monitoring (primary + DLQ) for signals.fact.updated.v1 after first publish.
+- Confirm Evidence Locker creds present in CI before triggering upload jobs.