579236bfce
- Created StellaOps.Notify.Storage.Mongo project with initial configuration. - Added expected output files for acceptance tests (at1.txt to at10.txt). - Added fixture input files for acceptance tests (at1 to at10). - Created input and signature files for test cases fc1 to fc5.
12 KiB
12 KiB
Sprint 0165_0001_0001 · Timeline Indexer (Export & Evidence 160.C)
Topic & Scope
- Bootstrap Timeline Indexer service: migrations/RLS, ingestion, query APIs, and evidence linkage.
- Keep ordering deterministic and tenant-scoped; link timeline events to evidence bundle digests/attestations.
- Working directory:
src/TimelineIndexer/StellaOps.TimelineIndexer.
Dependencies & Concurrency
- Upstream: AdvisoryAI (110.A), AirGap (120.A), Scanner (130.A), Orchestrator (150.A) schemas required for event payloads.
- Concurrency: execute tasks in listed order; evidence linkage follows ingestion and API/RLS work.
Documentation Prerequisites
- docs/README.md
- docs/07_HIGH_LEVEL_ARCHITECTURE.md
- docs/modules/platform/architecture-overview.md
- docs/modules/export-center/architecture.md (for evidence linkage)
- src/TimelineIndexer/StellaOps.TimelineIndexer/AGENTS.md (if present)
BLOCKED Tasks: Before working on BLOCKED tasks, review BLOCKED_DEPENDENCY_TREE.md for root blockers and dependencies.
Delivery Tracker
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
|---|---|---|---|---|---|
| 1 | TIMELINE-OBS-52-001 | DONE (2025-11-30) | Postgres schema + RLS committed; evidence linkage table aligned to bundle contract | Timeline Indexer Guild (src/TimelineIndexer/StellaOps.TimelineIndexer) |
Bootstrap service; Postgres migrations for timeline_events, timeline_event_details, timeline_event_digests; enable RLS scaffolding and deterministic migration scripts. |
| 2 | TIMELINE-OBS-52-002 | DONE (2025-12-03) | NATS/Redis subscribers + orchestrator envelope parser wired; ingestion worker records lag metrics and dedupes (tenant,event_id) |
Timeline Indexer Guild | Implement event ingestion pipeline (NATS/Redis consumers) with ordering guarantees, dedupe (event_id, tenant_id), trace-ID correlation, backpressure metrics. |
| 3 | TIMELINE-OBS-52-003 | DONE (2025-12-03) | REST timeline APIs return tenant-scoped listings and detail views (payload/digests) with filters/pagination | Timeline Indexer Guild | Expose REST/gRPC APIs for timeline queries (GET /timeline, /timeline/{id}) with filters, pagination, tenant enforcement; provide OpenAPI + contract tests. |
| 4 | TIMELINE-OBS-52-004 | DONE (2025-12-03) | RLS enforced via tenant session; timeline:read/timeline:write scopes enforced with audit sink logging auth events; payload hash constraint aligned |
Timeline Indexer Guild · Security Guild | Finalize RLS policies, scope checks (timeline:read), audit logging; integration tests for cross-tenant isolation and legal hold markers. |
| 5 | TIMELINE-OBS-53-001 | DOING (2025-12-05) | EvidenceLocker EB1 manifest + checksums schemas landed 2025-12-04 (docs/modules/evidence-locker/schemas/bundle.manifest.schema.json); begin wiring linkage tests. |
Timeline Indexer Guild · Evidence Locker Guild | Link timeline events to evidence bundle digests + attestation subjects; expose /timeline/{id}/evidence returning signed manifest references. |
Wave Coordination
- Wave 1: TIMELINE-OBS-52 chain (service bootstrap → ingestion → APIs → RLS/policies).
- Wave 2: Evidence linkage (TIMELINE-OBS-53-001) after digest schema lands and RLS is approved.
Wave Detail Snapshots
- Wave 1 deliverable: tenant-scoped timeline service with deterministic ingestion, pagination, and RLS/audit logging ready for Security review.
- Wave 2 deliverable: evidence linkage endpoint returning signed manifest references tied to EvidenceLocker digests/attestations.
Interlocks
| Dependency | Impacts | Status / Next signal |
|---|---|---|
| Orchestrator/Notifications event schema | Tasks 2–4 | Mitigated: parser bound to docs/events/*@1.json orchestrator envelopes; tolerant to additive fields. Monitor doc updates. |
| EvidenceLocker bundle digest schema | Tasks 1, 5 | Available (2025-12-04): EB1 manifest + checksums schemas published; align TIMELINE-OBS-53-001 linkage with Merkle root + DSSE subject. Monitor 2025-12-06 AdvisoryAI/Orch ETA for payload note impacts. |
| Security/Compliance RLS review | Task 4 | Implemented RLS/audit; ready for Security review once scheduled. |
Action Tracker
| # | Action | Owner | Due (UTC) | Status |
|---|---|---|---|---|
| 1 | Attach orchestrator/notification event schema sample to sprint doc. | Timeline Indexer Guild | 2025-12-02 | CLOSED (bound to docs/events/scanner.event.*@1.json) |
| 2 | Obtain EvidenceLocker digest schema/sample manifest for linkage design. | Timeline Indexer Guild · Evidence Locker Guild | 2025-12-06 | DONE (2025-12-05) — EB1 manifest + checksums schemas published; fixtures available under tests/EvidenceLocker/Bundles/Golden. |
| 3 | Draft RLS/migration proposal and route to Security/Compliance for approval. | Timeline Indexer Guild | 2025-12-04 | CLOSED (RLS + audit sink implemented; ready for review) |
| 4 | Add CI gate for EB1 evidence linkage integration test (TIMELINE-OBS-53-001) in TimelineIndexer pipeline. | Timeline Indexer Guild | 2025-12-07 | DONE (2025-12-05) — build-test-deploy adds timelineindexer test step with TRX output. |
Upcoming Checkpoints
- 2025-12-06 — Schema ETA sync (AdvisoryAI + Orchestrator/Notifications leads) to unblock evidence linkage; escalate to steering on 2025-12-07 if silent.
- 2025-12-10 — Wave 160 snapshot refresh to align EvidenceLocker digest schema and ExportCenter handoff; extend to 2025-12-13 if still blocked.
Decisions & Risks
| Risk / Decision | Impact | Mitigation / Next step | Status |
|---|---|---|---|
| Orchestrator/notification schemas not yet published. | Blocks ingestion and API field definitions (TIMELINE-OBS-52-002/003). | Parser now bound to docs/events/*@1.json envelopes; tolerant to additive fields. Monitor doc updates. |
CLOSED |
| EvidenceLocker digest schema pending. | Blocks digest table shape and evidence linkage (TIMELINE-OBS-53-001). | EB1 manifest + checksums schemas landed 2025-12-04; proceed with linkage using published Merkle subject and DSSE requirements. | CLOSED |
| RLS review not scheduled. | Could delay production readiness of policies (TIMELINE-OBS-52-004). | RLS + audit sink implemented; ready for Security review scheduling. | CLOSED |
Baseline docs may change (docs/modules/orchestrator/event-envelope.md, docs/modules/evidence-locker/prep/2025-11-24-evidence-locker-contract.md). |
Schema drift could invalidate migrations. | Monitor upstream doc updates; re-run schema diff before coding resumes. | OPEN |
Workspace disk full prevents running dotnet test. |
Tests for timeline ingestion/query remain unverified. | Cleared; dotnet test for TimelineIndexer now passes. |
CLOSED |
Risk table
| Risk | Severity | Mitigation / Owner |
|---|---|---|
| Orchestrator/notification schema slip. | Medium | Parser bound to docs/events/*@1.json; monitor 2025-12-06 ETA sync. Owner: Timeline Indexer Guild. |
| AdvisoryAI payload note drift post-ETA. | Medium | Re-run EB1 integration + manifest fallback tests after 2025-12-06 sync; adjust linkage mapping if predicates change. Owner: Timeline Indexer Guild · AdvisoryAI Guild. |
| EvidenceLocker digest schema slip. | Medium | Schema delivered 2025-12-04; continue to monitor for payload note changes after 2025-12-06 sync. Owner: Timeline Indexer Guild · Evidence Locker Guild. |
| RLS review delayed. | Medium | Action 3 to draft and schedule review with Security/Compliance. Owner: Timeline Indexer Guild. |
| Schema drift after migrations drafted. | Medium | Re-run schema diff against upstream docs before coding resumes. Owner: Timeline Indexer Guild. |
Execution Log
| Date (UTC) | Update | Owner |
|---|---|---|
| 2025-12-03 | TIMELINE-OBS-52-002: wired NATS/Redis subscribers with orchestrator envelope parser, ingestion lag histogram, and deterministic payload hashing; fixed payload hash regex + appsettings for Postgres/ingestion. | Implementer |
| 2025-12-03 | TIMELINE-OBS-52-003/004: REST timeline endpoints return payload/digest detail with tenant filters; timeline:read/timeline:write scopes enforced with audit sink; dotnet test on StellaOps.TimelineIndexer.sln passing (10 tests). |
Implementer |
| 2025-12-01 | Testing blocked: runner reports “No space left on device”; timeline module tests not executed. | Implementer |
| 2025-11-30 | Ran TimelineIndexer unit tests (TimelineIndexer.Tests) successfully after ingestion worker + query option fixes; still waiting on transport schema to wire NATS/Redis. | Implementer |
| 2025-11-30 | Implemented TimelineIngestionWorker with subscriber abstraction, session dedupe, and metrics counters; awaiting NATS/Redis subject schema to wire real transports. | Implementer |
| 2025-11-30 | Started TIMELINE-OBS-52-002: added ingestion service, Postgres store, and deterministic payload hashing; queue bindings pending schema alignment. | Implementer |
| 2025-11-30 | Normalized sprint to AGENTS template (Wave/Interlocks/Action tracker) while keeping prior content intact. | Implementer |
| 2025-11-30 | Completed TIMELINE-OBS-52-001: added Postgres schema/RLS migrations, DataSource + migration runner wiring; test run attempted for module but cancelled due to long solution restore—manual rerun needed. | Implementer |
| 2025-11-30 | Located orchestrator event envelope draft and Evidence Locker bundle contract; unblocked migrations and RLS design for TIMELINE-OBS-52-001 and started implementation. | Implementer |
| 2025-11-30 | Built TimelineIndexer solution successfully after query options fix; dotnet test on TimelineIndexer.Tests now passing (9 tests). |
Implementer |
| 2025-11-30 | Re-checked for orchestrator/notification schema and EvidenceLocker bundle digest; none landed in docs/events or docs/modules/evidence-locker, so keeping all tasks blocked. |
Implementer |
| 2025-11-25 | Marked TIMELINE-OBS-52-001 BLOCKED: missing orchestrator/notification event schema and EvidenceLocker digest schema prevent drafting migrations/RLS. | Implementer |
| 2025-11-12 | Captured task snapshot and blockers; waiting on orchestrator/notifications schema and EvidenceLocker digest schema. | Planning |
| 2025-11-19 | Normalized sprint to standard template and renamed from SPRINT_165_timelineindexer.md to SPRINT_0165_0001_0001_timelineindexer.md; content preserved. |
Implementer |
| 2025-11-19 | Added legacy-file redirect stub to prevent divergent updates. | Implementer |
| 2025-12-04 | Synced checkpoints with Sprint 160: added 2025-12-06 schema ETA sync and 2025-12-10 refresh; updated Action 2 due date/status and risk severities. | Project PM |
| 2025-12-05 | EB1 manifest + checksums schemas landed (EvidenceLocker); moved TIMELINE-OBS-53-001 to DOING, closed Action 2, and set linkage work to use Merkle root/DSSE subject from schema. | Implementer |
| 2025-12-05 | Implemented /timeline/{id}/evidence endpoint + query/store plumbing; added evidence parsing + ingestion/query coverage; dotnet test (TimelineIndexer.sln) passing (16 tests). |
Implementer |
| 2025-12-05 | Added ingestion-path evidence metadata tests in service + worker to guard bundle/attestation/manifest capture for EB1 linkage; added offline EB1 integration test using golden sealed bundle fixtures. | Implementer |
| 2025-12-05 | EB1 golden sealed bundle integration test passing (16/16 tests) after fixture path fix; evidence linkage validated end-to-end for TIMELINE-OBS-53-001 pending AdvisoryAI/Orch payload notes. | Implementer |
| 2025-12-05 | Added manifest URI fallback (bundleId→bundles/{id}/manifest.dsse.json) in query/service to guarantee evidence endpoint returns manifest path even when absent; covered by new fallback unit test. |
Implementer |
| 2025-12-05 | Added CI-gate action for EB1 integration test (TIMELINE-OBS-53-001) to timeline pipeline. | Implementer |
| 2025-12-05 | Action 4 completed: build-test-deploy now runs TimelineIndexer.sln (EB1 gate) with TRX output. | Implementer |
| 2025-12-05 | CI updated (.gitea/workflows/build-test-deploy.yml) to run TimelineIndexer solution (EB1 linkage gate); Action 4 marked DONE. |
Implementer |
| 2025-12-05 | Updated tests to 16/16 green (includes EB1 integration + manifest fallback); TimelineIndexer evidence linkage snapshot remains DOING pending 2025-12-06 payload note sync. | Implementer |
| 2025-12-05 | Post-CI-gate validation: reran TimelineIndexer.sln locally; suite remains green (16/16). | Implementer |