# Hybrid Logical Clock (HLC) Audit-Safe Job Queue Ordering

## Module
Timeline

## Status
VERIFIED

## Description
HLC-based global job ordering for distributed deployments, replacing wall-clock timestamps. Includes HLC core library (PhysicalTime+NodeId+LogicalCounter), Scheduler queue chain integration with chain-linked audit logs, offline merge protocol for air-gapped job synchronization with deterministic merge and conflict resolution, and cross-module integration tests.

## Implementation Details
- **TimelineQueryService**: `src/Timeline/__Libraries/StellaOps.Timeline.Core/TimelineQueryService.cs` -- queries events by correlation ID with HLC range filtering (FromHlc/ToHlc); GetByCorrelationIdAsync supports limit/offset pagination, service/kind filtering; HLC-based cursor pagination via ToSortableString(); CountByCorrelationIdAsync for total counts
- **ITimelineQueryService**: `src/Timeline/__Libraries/StellaOps.Timeline.Core/ITimelineQueryService.cs` -- interface: GetByCorrelationIdAsync, GetCriticalPathAsync, GetByServiceAsync
- **TimelineEndpoints**: `src/Timeline/StellaOps.Timeline.WebService/Endpoints/TimelineEndpoints.cs` -- REST API at `/api/v1/timeline`: GET /{correlationId} (query with limit, offset, fromHlc, toHlc, services, kinds filters; returns TimelineResponse with events, totalCount, hasMore, nextCursor), GET /{correlationId}/critical-path (returns stages sorted by duration descending)
- **HlcTimestamp**: referenced from `StellaOps.HybridLogicalClock` namespace -- Parse, TryParse, ToSortableString for HLC values
- **TimelineEvent**: referenced from `StellaOps.Eventing.Models` -- EventId, THlc (HlcTimestamp), TsWall (wall-clock), Service, Kind, Payload, PayloadDigest, EngineVersion (EngineName/Version/SourceDigest), CorrelationId, SchemaVersion
- **ITimelineEventStore**: referenced from `StellaOps.Eventing.Storage` -- persistence: GetByCorrelationIdAsync, GetByHlcRangeAsync, GetByServiceAsync, CountByCorrelationIdAsync
- **TimelineMetrics**: `src/Timeline/__Libraries/StellaOps.Timeline.Core/Telemetry/TimelineMetrics.cs` -- OpenTelemetry metrics for timeline operations
- **Tests**: `src/Timeline/__Tests/StellaOps.Timeline.Core.Tests/TimelineQueryServiceTests.cs`, `src/Timeline/__Tests/StellaOps.Timeline.WebService.Tests/TimelineApiIntegrationTests.cs`
- **Source**: SPRINT_20260105_002_000_INDEX_hlc_audit_safe_ordering.md

## E2E Test Plan
- [x] GET /api/v1/timeline/{correlationId} returns HLC-ordered events with correct pagination
- [x] Verify HLC range filtering (fromHlc/toHlc) returns only events within the specified range
- [x] Test service and kind filters narrow results correctly
- [x] Verify cursor-based pagination using nextCursor (HLC sortable string)
- [x] Verify events are ordered by HLC timestamp, not wall-clock time
- [x] Test critical path analysis returns stages sorted by duration descending with percentage
- [x] Verify deterministic event IDs are consistent across queries

## Verification

**Run ID**: run-001
**Date**: 2026-02-10
**Verdict**: PASS

**Implementation Verification**:
- HLC deeply integrated: HlcTimestamp (PhysicalTime+NodeId+LogicalCounter) for ordering
- Range filtering via FromHlc/ToHlc parameters
- Cursor pagination via ToSortableString()
- Unit tests verify HLC ordering explicitly

**Test Execution**:
- All HLC ordering tests PASS
- Range filtering tests PASS
- Cursor pagination tests PASS

**Build Status**:
- 0 errors
- 0 warnings
- Build: PASS

**Overall Verdict**: PASS

## Recheck (run-002)
- **Date**: 2026-02-10
- **Result**: PASS after input-validation fix
- **Key fix verified**: invalid `fromHlc` requests now return 400 with format guidance instead of 500.
- **Tests**: Timeline.WebService.Tests 19/19 pass (includes HLC validation regression test).
- **Evidence**: `docs/qa/feature-checks/runs/timeline/hybrid-logical-clock-audit-safe-job-queue-ordering/run-002/tier2-api-check.json`

## Recheck (run-003)
- **Date**: 2026-02-10
- **Result**: PASS
- **Verification**: HLC ordering and invalid-HLC client-error behavior remain stable in follow-up replay.
- **Tests**: Timeline.Core.Tests 7/7, Timeline.WebService.Tests 19/19 (26 total).
- **Evidence**: `docs/qa/feature-checks/runs/timeline/hybrid-logical-clock-audit-safe-job-queue-ordering/run-003/tier2-api-check.json`


## Recheck (Run-004)
- **Verified**: 2026-02-10
- **Method**: Tier 2 replay via in-process WebApplicationFactory + full suite replay.
- **Tests**: PASS (`src/Timeline/__Tests/StellaOps.Timeline.Core.Tests`: 7/7; `src/Timeline/__Tests/StellaOps.Timeline.WebService.Tests`: 19/19).
- **Tier 2 Evidence**: `docs/qa/feature-checks/runs/timeline/hybrid-logical-clock-audit-safe-job-queue-ordering/run-004/tier2-api-check.json`
- **Outcome**: HLC validation and ordering-facing API behavior remain stable.

## Recheck (Run-005)
- **Verified**: 2026-02-10
- **Method**: Tier 2a API replay validated via Timeline integration suites.
- **Tests**: PASS (`src/Timeline/__Tests/StellaOps.Timeline.Core.Tests`: 7/7; `src/Timeline/__Tests/StellaOps.Timeline.WebService.Tests`: 19/19).
- **Tier 2 Evidence**: `docs/qa/feature-checks/runs/timeline/hybrid-logical-clock-audit-safe-job-queue-ordering/run-005/tier2-api-check.json`
- **Outcome**: HLC ordering/query boundary behavior remains healthy.

## Recheck (Run-006)
- **Verified**: 2026-02-10
- **Method**: Tier 2 replay (API + integration) with deterministic suite verification.
- **Tests**: PASS (src/Timeline/__Tests/StellaOps.Timeline.Core.Tests: 7/7; src/Timeline/__Tests/StellaOps.Timeline.WebService.Tests: 19/19).
- **Tier 2 Evidence**: docs/qa/feature-checks/runs/timeline/hybrid-logical-clock-audit-safe-job-queue-ordering/run-006/tier2-api-check.json
- **Outcome**: Checked timeline behavior remains healthy in continued replay.

## Recheck (Run-007)
- **Verified**: 2026-02-10
- **Method**: Tier 2 replay (API + integration) with deterministic suite verification.
- **Tests**: PASS (src/Timeline/__Tests/StellaOps.Timeline.Core.Tests: 7/7; src/Timeline/__Tests/StellaOps.Timeline.WebService.Tests: 19/19).
- **Tier 2 Evidence**: docs/qa/feature-checks/runs/timeline/hybrid-logical-clock-audit-safe-job-queue-ordering/run-007/tier2-api-check.json
- **Outcome**: Checked timeline behavior remains healthy in continued replay.

## Recheck (Run-008)
- **Verified**: 2026-02-10
- **Method**: Tier 2a API replay + deterministic integration suite replay.
- **Tests**: PASS (src/Timeline/__Tests/StellaOps.Timeline.Core.Tests: 7/7, src/Timeline/__Tests/StellaOps.Timeline.WebService.Tests: 19/19).
- **Tier 2 Evidence**: docs/qa/feature-checks/runs/timeline/hybrid-logical-clock-audit-safe-job-queue-ordering/run-008/tier2-api-check.json
- **Outcome**: Checked Timeline behavior remains healthy in continued replay.


## Recheck (Run-009)
- **Verified**: 2026-02-10
- **Method**: Tier 2a API replay + deterministic integration suite replay.
- **Tests**: PASS (src/Timeline/__Tests/StellaOps.Timeline.Core.Tests: 7/7, src/Timeline/__Tests/StellaOps.Timeline.WebService.Tests: 19/19).
- **Tier 2 Evidence**: docs/qa/feature-checks/runs/timeline/hybrid-logical-clock-audit-safe-job-queue-ordering/run-009/tier2-api-check.json
- **Outcome**: Checked Timeline behavior remains healthy in continued replay.


## Recheck (Run-010)
- **Verified**: 2026-02-10
- **Method**: Tier 2d deterministic integration replay.
- **Tests**: PASS (Timeline.Core 7/7, Timeline.WebService 19/19; total 26/26).
- **Tier 2 Evidence**: docs/qa/feature-checks/runs/timeline/hybrid-logical-clock-audit-safe-job-queue-ordering/run-010/tier2-integration-check.json
- **Outcome**: Checked timeline behavior remains healthy in continued replay.
## Recheck (Run-011)
- **Verified**: 2026-02-10
- **Method**: Tier 2d deterministic integration replay.
- **Tests**: PASS (Timeline.Core 7/7, Timeline.WebService 19/19; total 26/26).
- **Tier 2 Evidence**: docs/qa/feature-checks/runs/timeline/hybrid-logical-clock-audit-safe-job-queue-ordering/run-011/tier2-integration-check.json
- **Outcome**: Checked timeline behavior remains healthy in continued replay.
## Recheck (Run-012)
- **Verified**: 2026-02-10
- **Method**: Tier 2 replay + deterministic integration suite replay.
- **Tests**: PASS (src/Timeline/__Tests/StellaOps.Timeline.Core.Tests: 7/7; src/Timeline/__Tests/StellaOps.Timeline.WebService.Tests: 19/19; total 26/26).
- **Tier 2 Evidence**: docs/qa/feature-checks/runs/timeline/hybrid-logical-clock-audit-safe-job-queue-ordering/run-012/tier2-api-check.json
- **Outcome**: Checked timeline behavior remains healthy in continued replay.