semi implemented and features implemented save checkpoint

This commit is contained in:
master
2026-02-08 18:00:49 +02:00
parent 04360dff63
commit 1bf6bbf395
20895 changed files with 716795 additions and 64 deletions

View File

@@ -0,0 +1,30 @@
# Hybrid Logical Clock (HLC) Audit-Safe Job Queue Ordering
## Module
Timeline
## Status
IMPLEMENTED
## Description
HLC-based global job ordering for distributed deployments, replacing wall-clock timestamps. Includes HLC core library (PhysicalTime+NodeId+LogicalCounter), Scheduler queue chain integration with chain-linked audit logs, offline merge protocol for air-gapped job synchronization with deterministic merge and conflict resolution, and cross-module integration tests.
## Implementation Details
- **TimelineQueryService**: `src/Timeline/__Libraries/StellaOps.Timeline.Core/TimelineQueryService.cs` -- queries events by correlation ID with HLC range filtering (FromHlc/ToHlc); GetByCorrelationIdAsync supports limit/offset pagination, service/kind filtering; HLC-based cursor pagination via ToSortableString(); CountByCorrelationIdAsync for total counts
- **ITimelineQueryService**: `src/Timeline/__Libraries/StellaOps.Timeline.Core/ITimelineQueryService.cs` -- interface: GetByCorrelationIdAsync, GetCriticalPathAsync, GetByServiceAsync
- **TimelineEndpoints**: `src/Timeline/StellaOps.Timeline.WebService/Endpoints/TimelineEndpoints.cs` -- REST API at `/api/v1/timeline`: GET /{correlationId} (query with limit, offset, fromHlc, toHlc, services, kinds filters; returns TimelineResponse with events, totalCount, hasMore, nextCursor), GET /{correlationId}/critical-path (returns stages sorted by duration descending)
- **HlcTimestamp**: referenced from `StellaOps.HybridLogicalClock` namespace -- Parse, TryParse, ToSortableString for HLC values
- **TimelineEvent**: referenced from `StellaOps.Eventing.Models` -- EventId, THlc (HlcTimestamp), TsWall (wall-clock), Service, Kind, Payload, PayloadDigest, EngineVersion (EngineName/Version/SourceDigest), CorrelationId, SchemaVersion
- **ITimelineEventStore**: referenced from `StellaOps.Eventing.Storage` -- persistence: GetByCorrelationIdAsync, GetByHlcRangeAsync, GetByServiceAsync, CountByCorrelationIdAsync
- **TimelineMetrics**: `src/Timeline/__Libraries/StellaOps.Timeline.Core/Telemetry/TimelineMetrics.cs` -- OpenTelemetry metrics for timeline operations
- **Tests**: `src/Timeline/__Tests/StellaOps.Timeline.Core.Tests/TimelineQueryServiceTests.cs`, `src/Timeline/__Tests/StellaOps.Timeline.WebService.Tests/TimelineApiIntegrationTests.cs`
- **Source**: SPRINT_20260105_002_000_INDEX_hlc_audit_safe_ordering.md
## E2E Test Plan
- [ ] GET /api/v1/timeline/{correlationId} returns HLC-ordered events with correct pagination
- [ ] Verify HLC range filtering (fromHlc/toHlc) returns only events within the specified range
- [ ] Test service and kind filters narrow results correctly
- [ ] Verify cursor-based pagination using nextCursor (HLC sortable string)
- [ ] Verify events are ordered by HLC timestamp, not wall-clock time
- [ ] Test critical path analysis returns stages sorted by duration descending with percentage
- [ ] Verify deterministic event IDs are consistent across queries

View File

@@ -0,0 +1,30 @@
# Immutable Audit Log (Timeline)
## Module
Timeline
## Status
IMPLEMENTED
## Description
Immutable timeline audit log with a dedicated web service and indexer for recording all scan, attestation, and verdict events.
## Implementation Details
- **TimelineQueryService**: `src/Timeline/__Libraries/StellaOps.Timeline.Core/TimelineQueryService.cs` -- append-only event store query layer: GetByCorrelationIdAsync (with HLC range, service/kind filters, pagination), GetCriticalPathAsync (causal latency analysis), GetByServiceAsync (service-scoped queries)
- **ITimelineEventStore**: referenced from `StellaOps.Eventing.Storage` -- append-only persistence interface: events stored with deterministic EventId (SHA-256 of correlation_id+t_hlc+service+kind), HLC timestamps, payload digests, engine version fingerprints
- **TimelineEndpoints**: `src/Timeline/StellaOps.Timeline.WebService/Endpoints/TimelineEndpoints.cs` -- REST API at `/api/v1/timeline`: GET /{correlationId} returns immutable event chain, GET /{correlationId}/critical-path for latency analysis
- **ExportEndpoints**: `src/Timeline/StellaOps.Timeline.WebService/Endpoints/ExportEndpoints.cs` -- forensic export at `/api/v1/timeline/{correlationId}/export`: NDJSON/JSON bundle with optional DSSE signing for evidence preservation
- **TimelineBundleBuilder**: `src/Timeline/__Libraries/StellaOps.Timeline.Core/Export/TimelineBundleBuilder.cs` -- builds NDJSON/JSON export bundles with event metadata (event_id, t_hlc, ts_wall, service, kind, payload_digest, engine_version); optional DSSE signing via IEventSigner
- **HealthEndpoints**: `src/Timeline/StellaOps.Timeline.WebService/Endpoints/HealthEndpoints.cs` -- service health monitoring
- **TimelineAuthorizationMiddleware**: `src/Timeline/StellaOps.Timeline.WebService/Authorization/TimelineAuthorizationMiddleware.cs` -- authorization for timeline access
- **Tests**: `src/Timeline/__Tests/StellaOps.Timeline.WebService.Tests/TimelineApiIntegrationTests.cs`
- **Source**: Feature matrix scan
## E2E Test Plan
- [ ] Verify events stored are immutable (no update/delete operations exposed)
- [ ] Verify event IDs are deterministic based on correlation_id + t_hlc + service + kind
- [ ] Test export endpoint produces valid NDJSON bundle with all event metadata
- [ ] Verify DSSE-signed export bundles can be verified with the signing key
- [ ] Test JSON export format includes event metadata section with count and export timestamp
- [ ] Verify payload digests in exported events match original payloads
- [ ] Test authorization middleware restricts timeline access to authorized users

View File

@@ -0,0 +1,32 @@
# Timeline Indexer Service
## Module
Timeline
## Status
IMPLEMENTED
## Description
Dedicated service for ingesting, indexing, and querying timeline events across all platform modules, with Postgres-backed storage (RLS), REST APIs for event retrieval, and evidence linkage to correlate events with attestation artifacts.
## Implementation Details
- **TimelineQueryService**: `src/Timeline/__Libraries/StellaOps.Timeline.Core/TimelineQueryService.cs` -- central query service: GetByCorrelationIdAsync (HLC range, service/kind filters, limit/offset pagination, cursor-based paging via HLC sortable strings), GetCriticalPathAsync (builds stage list from consecutive event pairs, sorts by duration descending), GetByServiceAsync (service-scoped queries with HLC cursor)
- **ITimelineQueryService**: `src/Timeline/__Libraries/StellaOps.Timeline.Core/ITimelineQueryService.cs` -- query interface
- **ITimelineEventStore**: referenced from `StellaOps.Eventing.Storage` -- PostgreSQL-backed event store: GetByCorrelationIdAsync, GetByHlcRangeAsync, GetByServiceAsync, CountByCorrelationIdAsync; append-only with RLS for tenant isolation
- **TimelineEndpoints**: `src/Timeline/StellaOps.Timeline.WebService/Endpoints/TimelineEndpoints.cs` -- REST API: GET /api/v1/timeline/{correlationId} (with fromHlc, toHlc, services, kinds, limit, offset query parameters; returns events, totalCount, hasMore, nextCursor), GET /{correlationId}/critical-path
- **ExportEndpoints**: `src/Timeline/StellaOps.Timeline.WebService/Endpoints/ExportEndpoints.cs` -- export API: POST /{correlationId}/export (NDJSON/JSON format, optional DSSE signing), GET /export/{exportId} (status), GET /export/{exportId}/download (bundle download)
- **TimelineBundleBuilder**: `src/Timeline/__Libraries/StellaOps.Timeline.Core/Export/TimelineBundleBuilder.cs` -- asynchronous bundle building with progress tracking, NDJSON/JSON serialization, optional DSSE signing via IEventSigner
- **ServiceCollectionExtensions**: `src/Timeline/__Libraries/StellaOps.Timeline.Core/ServiceCollectionExtensions.cs` -- DI registration for timeline services
- **TimelineMetrics**: `src/Timeline/__Libraries/StellaOps.Timeline.Core/Telemetry/TimelineMetrics.cs` -- OpenTelemetry metrics: replay and export operation tracking
- **Tests**: `src/Timeline/__Tests/StellaOps.Timeline.Core.Tests/TimelineQueryServiceTests.cs`, `src/Timeline/__Tests/StellaOps.Timeline.WebService.Tests/TimelineApiIntegrationTests.cs`
- **Source**: SPRINT_0165_0001_0001_timelineindexer.md
## E2E Test Plan
- [ ] Verify GET /api/v1/timeline/{correlationId} returns indexed events with correct HLC ordering
- [ ] Test service and kind filters narrow indexed results
- [ ] Verify HLC range queries (fromHlc/toHlc) return correct event subsets
- [ ] Test cursor-based pagination produces consistent results across pages
- [ ] Verify critical path endpoint computes stage durations and percentages correctly
- [ ] Test export API: initiate -> check status -> download bundle
- [ ] Verify NDJSON export includes all event fields (event_id, t_hlc, ts_wall, service, kind, payload_digest, engine_version)
- [ ] Test evidence linkage: events with attestation references are queryable by correlation

View File

@@ -0,0 +1,31 @@
# Timeline Replay API
## Module
Timeline
## Status
IMPLEMENTED
## Description
REST API endpoints for querying and replaying HLC-ordered events: GET /timeline/{correlationId} with service/kind/HLC-range/pagination filters, critical path analysis endpoint, and integration with StellaOps.Replay.Core for deterministic replay at a specific HLC timestamp.
## Implementation Details
- **ReplayEndpoints**: `src/Timeline/StellaOps.Timeline.WebService/Endpoints/ReplayEndpoints.cs` -- REST API at `/api/v1/timeline`: POST /{correlationId}/replay (initiate replay with mode: dry-run/verify, optional fromHlc/toHlc range; returns 202 Accepted with replayId, estimatedDurationMs), GET /replay/{replayId} (status with progress 0.0-1.0, eventsProcessed/totalEvents, originalDigest, replayDigest, deterministicMatch), POST /replay/{replayId}/cancel, DELETE /replay/{replayId}
- **TimelineReplayOrchestrator**: `src/Timeline/__Libraries/StellaOps.Timeline.Core/Replay/TimelineReplayOrchestrator.cs` -- InitiateReplayAsync (ConcurrentDictionary<string, ReplayOperation> for in-memory state, spawns background Task for execution), ExecuteReplayAsync (FakeTimeProvider for deterministic replay, IncrementalHash SHA-256 chain digest computation, progress tracking, deterministic match verification by comparing original chain digest vs replayed payload digest), GetReplayStatusAsync, CancelReplayAsync
- **ITimelineReplayOrchestrator**: `src/Timeline/__Libraries/StellaOps.Timeline.Core/Replay/ITimelineReplayOrchestrator.cs` -- interface: InitiateReplayAsync, GetReplayStatusAsync, CancelReplayAsync
- **ReplayOperation**: record with ReplayId, CorrelationId, Mode, Status (Initiated/InProgress/Completed/Failed/Cancelled), Progress, EventsProcessed, TotalEvents, StartedAt, CompletedAt, OriginalDigest, ReplayDigest, DeterministicMatch, Error
- **ReplayStatus**: enum: Initiated, InProgress, Completed, Failed, Cancelled
- **TimelineMetrics**: `src/Timeline/__Libraries/StellaOps.Timeline.Core/Telemetry/TimelineMetrics.cs` -- RecordReplay(mode, outcome, eventCount, durationSeconds)
- **Tests**: `src/Timeline/__Tests/StellaOps.Timeline.WebService.Tests/ReplayOrchestratorIntegrationTests.cs`
- **Source**: SPRINT_20260107_003_002_BE_timeline_replay_api.md
## E2E Test Plan
- [ ] POST /api/v1/timeline/{correlationId}/replay returns 202 Accepted with replayId and estimatedDurationMs
- [ ] GET /replay/{replayId} returns progress from 0.0 to 1.0 with eventsProcessed and totalEvents
- [ ] Verify completed replay includes originalDigest and replayDigest (SHA-256 chain hashes)
- [ ] Verify deterministicMatch is true when replayed output matches original event chain
- [ ] Test dry-run mode processes all events without side effects
- [ ] POST /replay/{replayId}/cancel stops an in-progress replay
- [ ] Verify cancelled replay cannot be restarted
- [ ] Test replay with HLC range (fromHlc/toHlc) replays only events within the range
- [ ] Verify replay of non-existent correlationId returns appropriate error

View File

@@ -0,0 +1,34 @@
# Unified Event Timeline Service
## Module
Timeline
## Status
IMPLEMENTED
## Description
Cross-service event timeline with HLC-ordered events, deterministic event IDs (SHA-256 of correlation_id+t_hlc+service+kind), W3C Trace Context integration, PostgreSQL append-only storage with materialized critical-path views. Provides event SDK for Scheduler/AirGap/Attestor/Policy/VexLens integration, timeline query API with HLC range filtering, causal latency measurement, and forensic event export with DSSE attestation.
## Implementation Details
- **TimelineQueryService**: `src/Timeline/__Libraries/StellaOps.Timeline.Core/TimelineQueryService.cs` -- central query engine: GetByCorrelationIdAsync (HLC range, service/kind filters, limit/offset, cursor paging), GetCriticalPathAsync (stage duration analysis: consecutive event pairs with percentage of total, sorted by duration descending), GetByServiceAsync (cross-correlation service queries)
- **TimelineEndpoints**: `src/Timeline/StellaOps.Timeline.WebService/Endpoints/TimelineEndpoints.cs` -- `/api/v1/timeline`: GET /{correlationId} returns TimelineResponse (events with EventId, THlc, TsWall, Service, Kind, Payload, EngineVersion; totalCount, hasMore, nextCursor); GET /{correlationId}/critical-path returns CriticalPathResponse (totalDurationMs, stages with durationMs/percentage/fromHlc/toHlc)
- **ReplayEndpoints**: `src/Timeline/StellaOps.Timeline.WebService/Endpoints/ReplayEndpoints.cs` -- deterministic replay: POST /{correlationId}/replay (dry-run/verify modes, HLC range), GET /replay/{replayId} (progress, deterministic match verification via SHA-256 chain digest comparison)
- **ExportEndpoints**: `src/Timeline/StellaOps.Timeline.WebService/Endpoints/ExportEndpoints.cs` -- forensic export: POST /{correlationId}/export (NDJSON/JSON, optional DSSE signing), GET /export/{exportId}, GET /export/{exportId}/download
- **TimelineReplayOrchestrator**: `src/Timeline/__Libraries/StellaOps.Timeline.Core/Replay/TimelineReplayOrchestrator.cs` -- background replay execution with FakeTimeProvider for determinism, IncrementalHash chain digest, progress tracking, cancellation support
- **TimelineBundleBuilder**: `src/Timeline/__Libraries/StellaOps.Timeline.Core/Export/TimelineBundleBuilder.cs` -- NDJSON/JSON bundle building with IEventSigner integration for DSSE-attested exports; includes event_id, t_hlc, ts_wall, correlation_id, service, kind, payload_digest, engine_version, schema_version
- **ServiceCollectionExtensions**: `src/Timeline/__Libraries/StellaOps.Timeline.Core/ServiceCollectionExtensions.cs` -- DI registration for all timeline services
- **TimelineMetrics**: `src/Timeline/__Libraries/StellaOps.Timeline.Core/Telemetry/TimelineMetrics.cs` -- OpenTelemetry metrics: RecordReplay, RecordExport
- **Program.cs**: `src/Timeline/StellaOps.Timeline.WebService/Program.cs` -- maps TimelineEndpoints, ReplayEndpoints, ExportEndpoints, HealthEndpoints
- **Tests**: `src/Timeline/__Tests/StellaOps.Timeline.Core.Tests/TimelineQueryServiceTests.cs`, `src/Timeline/__Tests/StellaOps.Timeline.WebService.Tests/TimelineApiIntegrationTests.cs`, `ReplayOrchestratorIntegrationTests.cs`
- **Source**: SPRINT_20260107_003_000_INDEX_unified_event_timeline.md
## E2E Test Plan
- [ ] GET /api/v1/timeline/{correlationId} returns cross-service events ordered by HLC timestamp
- [ ] Verify deterministic event IDs are SHA-256 hashes of correlation_id+t_hlc+service+kind
- [ ] Test HLC range filtering returns only events within the specified window
- [ ] Verify critical path analysis computes correct stage durations and percentages
- [ ] Test deterministic replay: initiate -> poll status -> verify deterministicMatch=true
- [ ] Verify forensic export produces NDJSON bundle with all event fields
- [ ] Test DSSE-signed export bundles include valid signature attestation
- [ ] Verify service and kind filters work correctly across multiple source services
- [ ] Test pagination with cursor returns consistent ordered results