git.stella-ops.org/src/__Libraries/StellaOps.HybridLogicalClock/README.md

StellaOps.HybridLogicalClock

A Hybrid Logical Clock (HLC) library for deterministic, monotonic job ordering across distributed nodes. HLC combines physical (wall-clock) time with logical counters to provide causally-ordered timestamps even under clock skew.

Overview

Problem Statement

Distributed systems face challenges with event ordering:

• Wall-clock timestamps are susceptible to clock skew between nodes
• Logical clocks alone don't provide real-time context
• Concurrent events from different nodes need deterministic tie-breaking

Solution

HLC addresses these by combining:

• Physical time (Unix milliseconds UTC) for real-time context
• Logical counter for events at the same millisecond
• Node ID for deterministic tie-breaking across nodes

Installation

Reference the project in your .csproj:

<ProjectReference Include="..\__Libraries\StellaOps.HybridLogicalClock\StellaOps.HybridLogicalClock.csproj" />

Quick Start

Basic Usage

using StellaOps.HybridLogicalClock;

// Create a clock instance
var clock = new HybridLogicalClock(
    TimeProvider.System,
    nodeId: "scheduler-east-1",
    stateStore: new InMemoryHlcStateStore(),
    logger: logger);

// Generate timestamps for local events
var ts1 = clock.Tick();  // e.g., 1704067200000-scheduler-east-1-000000
var ts2 = clock.Tick();  // e.g., 1704067200000-scheduler-east-1-000001

// Timestamps are always monotonically increasing
Debug.Assert(ts2 > ts1);

// When receiving a message from another node
var remoteTs = HlcTimestamp.Parse("1704067200100-scheduler-west-1-000005");
var mergedTs = clock.Receive(remoteTs);  // Merges clocks, returns new timestamp > both

Dependency Injection

// Program.cs or Startup.cs

// Option 1: In-memory state (development/testing)
services.AddHybridLogicalClock(
    nodeId: Environment.MachineName,
    maxClockSkew: TimeSpan.FromMinutes(1));

// Option 2: PostgreSQL persistence (production)
services.AddHybridLogicalClock<PostgresHlcStateStore>(
    nodeId: Environment.MachineName,
    maxClockSkew: TimeSpan.FromMinutes(1));

// Option 3: Custom state store factory
services.AddHybridLogicalClock(
    nodeId: Environment.MachineName,
    stateStoreFactory: sp => new PostgresHlcStateStore(
        sp.GetRequiredService<NpgsqlDataSource>(),
        sp.GetRequiredService<ILogger<PostgresHlcStateStore>>()),
    maxClockSkew: TimeSpan.FromMinutes(1));

Then inject the clock:

public class JobScheduler(IHybridLogicalClock clock)
{
    public void EnqueueJob(Job job)
    {
        job.EnqueuedAt = clock.Tick();
        // Jobs are now globally ordered across all scheduler nodes
    }
}

Core Types

HlcTimestamp

A readonly record struct representing an HLC timestamp:

public readonly record struct HlcTimestamp : IComparable<HlcTimestamp>
{
    public required long PhysicalTime { get; init; }    // Unix milliseconds UTC
    public required string NodeId { get; init; }        // e.g., "scheduler-east-1"
    public required int LogicalCounter { get; init; }   // Events at same millisecond
}

Key Methods:

Method	Description
ToSortableString()	Returns "1704067200000-scheduler-east-1-000042"
Parse(string)	Parses from sortable string format
TryParse(string, out HlcTimestamp)	Safe parsing without exceptions
ToDateTimeOffset()	Converts physical time to DateTimeOffset
CompareTo(HlcTimestamp)	Total ordering comparison
IsBefore(HlcTimestamp)	Returns true if causally before
IsAfter(HlcTimestamp)	Returns true if causally after
IsConcurrent(HlcTimestamp)	True if same time/counter, different nodes

Comparison Operators:

if (ts1 < ts2) { /* ts1 happened before ts2 */ }
if (ts1 > ts2) { /* ts1 happened after ts2 */ }
if (ts1 <= ts2) { /* ts1 happened at or before ts2 */ }
if (ts1 >= ts2) { /* ts1 happened at or after ts2 */ }

IHybridLogicalClock

The main clock interface:

public interface IHybridLogicalClock
{
    HlcTimestamp Tick();                      // Generate timestamp for local event
    HlcTimestamp Receive(HlcTimestamp remote); // Merge with remote timestamp
    HlcTimestamp Current { get; }             // Current clock state
    string NodeId { get; }                    // This node's identifier
}

IHlcStateStore

Persistence interface for clock state:

public interface IHlcStateStore
{
    Task<HlcTimestamp?> LoadAsync(string nodeId, CancellationToken ct = default);
    Task SaveAsync(HlcTimestamp timestamp, CancellationToken ct = default);
}

Implementations:

• InMemoryHlcStateStore - For development and testing
• PostgresHlcStateStore - For production with durable persistence

Persistence

PostgreSQL Schema

CREATE TABLE scheduler.hlc_state (
    node_id TEXT PRIMARY KEY,
    physical_time BIGINT NOT NULL,
    logical_counter INT NOT NULL,
    updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

CREATE INDEX idx_hlc_state_updated ON scheduler.hlc_state(updated_at DESC);

PostgresHlcStateStore

Uses atomic upsert with conditional update to maintain monotonicity:

var stateStore = new PostgresHlcStateStore(
    dataSource,
    logger,
    schemaName: "scheduler",  // default
    tableName: "hlc_state"    // default
);

Serialization

JSON (System.Text.Json)

Two converters are provided:

1. String format (default) - Compact, sortable:

   "1704067200000-scheduler-east-1-000042"
   

2. Object format - Explicit properties:

   {
     "physicalTime": 1704067200000,
     "nodeId": "scheduler-east-1",
     "logicalCounter": 42
   }
   

Register converters:

var options = new JsonSerializerOptions();
options.Converters.Add(new HlcTimestampJsonConverter());        // String format
// or
options.Converters.Add(new HlcTimestampObjectJsonConverter());  // Object format

Database (Npgsql/Dapper)

Extension methods for reading/writing HLC timestamps:

// NpgsqlCommand extension
await using var cmd = dataSource.CreateCommand();
cmd.CommandText = "INSERT INTO events (timestamp) VALUES (@ts)";
cmd.AddHlcTimestamp("ts", timestamp);
await cmd.ExecuteNonQueryAsync();

// NpgsqlDataReader extension
await using var reader = await cmd.ExecuteReaderAsync();
var ts = reader.GetHlcTimestamp("timestamp");
var nullableTs = reader.GetHlcTimestampOrNull("timestamp");

Dapper type handlers:

// Register handlers at startup
HlcTypeHandlerRegistration.Register(services);

// Then use normally with Dapper
var results = await connection.QueryAsync<MyEntity>(
    "SELECT * FROM events WHERE timestamp > @since",
    new { since = sinceTimestamp });

Clock Skew Handling

The clock detects and rejects excessive clock skew:

var clock = new HybridLogicalClock(
    timeProvider,
    nodeId,
    stateStore,
    logger,
    maxClockSkew: TimeSpan.FromMinutes(1));  // Default: 1 minute

try
{
    var merged = clock.Receive(remoteTimestamp);
}
catch (HlcClockSkewException ex)
{
    // Remote clock differs by more than maxClockSkew
    logger.LogWarning(
        "Clock skew detected: {Actual} exceeds threshold {Max}",
        ex.ActualSkew, ex.MaxAllowedSkew);
}

Recovery from Restart

After a node restart, initialize the clock from persisted state:

var clock = new HybridLogicalClock(timeProvider, nodeId, stateStore, logger);

// Load last persisted state
bool recovered = await clock.InitializeFromStateAsync();
if (recovered)
{
    logger.LogInformation("Clock recovered from state: {Current}", clock.Current);
}

// First tick after restart is guaranteed > last persisted tick
var ts = clock.Tick();

Testing

FakeTimeProvider

For deterministic testing, use a fake time provider:

public class FakeTimeProvider : TimeProvider
{
    private DateTimeOffset _now = DateTimeOffset.UtcNow;

    public override DateTimeOffset GetUtcNow() => _now;

    public void SetUtcNow(DateTimeOffset value) => _now = value;
    public void Advance(TimeSpan duration) => _now = _now.Add(duration);
}

[Fact]
public void Tick_Advances_Counter()
{
    var timeProvider = new FakeTimeProvider();
    var clock = new HybridLogicalClock(timeProvider, "test", new InMemoryHlcStateStore(), logger);

    var ts1 = clock.Tick();
    var ts2 = clock.Tick();

    Assert.Equal(0, ts1.LogicalCounter);
    Assert.Equal(1, ts2.LogicalCounter);
}

Algorithm

On Local Event (Tick)

l' = l
l = max(l, physical_clock())
if l == l':
    c = c + 1
else:
    c = 0
return (l, node_id, c)

On Receive

l' = l
l = max(l', m_l, physical_clock())
if l == l' == m_l:
    c = max(c, m_c) + 1
elif l == l':
    c = c + 1
elif l == m_l:
    c = m_c + 1
else:
    c = 0
return (l, node_id, c)

Performance

Benchmarks on typical hardware:

Operation	Throughput
Tick (single-thread)	> 100,000/sec
Tick (multi-thread)	> 50,000/sec
Receive	> 50,000/sec
Parse	> 500,000/sec
ToSortableString	> 500,000/sec
CompareTo	> 10,000,000/sec

Memory: HlcTimestamp is a value type (struct) with minimal allocation.

References

• Logical Physical Clocks and Consistent Snapshots in Globally Distributed Databases - Kulkarni et al.
• Time, Clocks, and the Ordering of Events in a Distributed System - Lamport