# Scheduler HLC Ordering Architecture

This document describes the Hybrid Logical Clock (HLC) based ordering system used by the StellaOps Scheduler for audit-safe job queue operations.

## Overview

The Scheduler uses HLC timestamps instead of wall-clock time to ensure:

1. **Total ordering** of jobs across distributed nodes
2. **Audit-safe sequencing** with cryptographic chain linking
3. **Deterministic merge** when offline nodes reconnect
4. **Clock skew tolerance** in distributed deployments

## HLC Timestamp Format

An HLC timestamp consists of three components:

```
(PhysicalTime, LogicalCounter, NodeId)
```

| Component | Description | Example |
|-----------|-------------|---------|
| PhysicalTime | Unix milliseconds (UTC) | `1704585600000` |
| LogicalCounter | Monotonic counter for same-millisecond events | `0`, `1`, `2`... |
| NodeId | Unique identifier for the node | `scheduler-prod-01` |

**String format:** `{physical}:{logical}:{nodeId}`
Example: `1704585600000:0:scheduler-prod-01`

## Database Schema

### scheduler_log Table

```sql
CREATE TABLE scheduler.scheduler_log (
    id                  BIGSERIAL PRIMARY KEY,
    t_hlc               TEXT NOT NULL,           -- HLC timestamp
    job_id              TEXT NOT NULL,           -- Job identifier
    action              TEXT NOT NULL,           -- ENQUEUE, DEQUEUE, EXECUTE, COMPLETE, FAIL
    prev_chain_link     TEXT,                    -- Hash of previous entry
    chain_link          TEXT NOT NULL,           -- Hash of this entry
    payload             JSONB NOT NULL,          -- Job metadata
    tenant_id           TEXT NOT NULL,
    created_at          TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

CREATE INDEX idx_scheduler_log_hlc ON scheduler.scheduler_log (t_hlc);
CREATE INDEX idx_scheduler_log_tenant_hlc ON scheduler.scheduler_log (tenant_id, t_hlc);
CREATE INDEX idx_scheduler_log_job ON scheduler.scheduler_log (job_id);
```

### batch_snapshot Table

```sql
CREATE TABLE scheduler.batch_snapshot (
    id                  BIGSERIAL PRIMARY KEY,
    snapshot_hlc        TEXT NOT NULL,           -- HLC at snapshot time
    from_chain_link     TEXT NOT NULL,           -- First entry in batch
    to_chain_link       TEXT NOT NULL,           -- Last entry in batch
    entry_count         INTEGER NOT NULL,
    merkle_root         TEXT NOT NULL,           -- Merkle root of entries
    dsse_envelope       JSONB,                   -- DSSE-signed attestation
    created_at          TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
```

### chain_heads Table

```sql
CREATE TABLE scheduler.chain_heads (
    tenant_id           TEXT PRIMARY KEY,
    head_chain_link     TEXT NOT NULL,           -- Current chain head
    head_hlc            TEXT NOT NULL,           -- HLC of chain head
    updated_at          TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
```

## Chain Link Computation

Each log entry is cryptographically linked to its predecessor:

```csharp
public static string ComputeChainLink(
    string tHlc,
    string jobId,
    string action,
    string? prevChainLink,
    string payloadDigest)
{
    using var hasher = IncrementalHash.CreateHash(HashAlgorithmName.SHA256);
    hasher.AppendData(Encoding.UTF8.GetBytes(tHlc));
    hasher.AppendData(Encoding.UTF8.GetBytes(jobId));
    hasher.AppendData(Encoding.UTF8.GetBytes(action));
    hasher.AppendData(Encoding.UTF8.GetBytes(prevChainLink ?? "genesis"));
    hasher.AppendData(Encoding.UTF8.GetBytes(payloadDigest));
    return Convert.ToHexString(hasher.GetHashAndReset()).ToLowerInvariant();
}
```

## Configuration Options

```yaml
# etc/scheduler.yaml
scheduler:
  hlc:
    enabled: true                    # Enable HLC ordering (default: true)
    nodeId: "scheduler-prod-01"      # Unique node identifier
    maxClockSkew: "00:00:05"         # Maximum tolerable clock skew (5 seconds)
    persistenceInterval: "00:01:00"  # HLC state persistence interval
    
  chain:
    enabled: true                    # Enable chain linking (default: true)
    batchSize: 1000                  # Entries per batch snapshot
    batchInterval: "00:05:00"        # Batch snapshot interval
    signSnapshots: true              # DSSE-sign batch snapshots
    keyId: "scheduler-signing-key"   # Key for snapshot signing
```

## Operational Considerations

### Clock Skew Handling

The HLC algorithm tolerates clock skew by:

1. Advancing logical counter when physical time hasn't progressed
2. Rejecting events with excessive clock skew (> `maxClockSkew`)
3. Emitting `hlc_clock_skew_rejections_total` metric for monitoring

**Alert:** `HlcClockSkewExceeded` triggers when skew > tolerance.

### Chain Verification

Verify chain integrity on startup and periodically:

```bash
# CLI command
stella scheduler chain verify --tenant-id <tenant>

# API endpoint
GET /api/v1/scheduler/chain/verify?tenantId=<tenant>
```

### Offline Merge

When offline nodes reconnect:

1. Export local job log as bundle
2. Import on connected node
3. HLC-based merge produces deterministic ordering
4. Chain is extended with merged entries

See `docs/operations/airgap-operations-runbook.md` for details.

## Metrics

| Metric | Type | Description |
|--------|------|-------------|
| `hlc_ticks_total` | Counter | Total HLC tick operations |
| `hlc_clock_skew_rejections_total` | Counter | Events rejected due to clock skew |
| `hlc_physical_offset_seconds` | Gauge | Current physical time offset |
| `scheduler_chain_entries_total` | Counter | Total chain log entries |
| `scheduler_chain_verifications_total` | Counter | Chain verification operations |
| `scheduler_chain_verification_failures_total` | Counter | Failed verifications |
| `scheduler_batch_snapshots_total` | Counter | Batch snapshots created |

## Grafana Dashboard

See `devops/observability/grafana/hlc-queue-metrics.json` for the HLC monitoring dashboard.

## Related Documentation

- [HLC Core Library](../../../src/__Libraries/StellaOps.HybridLogicalClock/README.md)
- [HLC Migration Guide](./hlc-migration-guide.md)
- [Air-Gap Operations Runbook](../../operations/airgap-operations-runbook.md)
- [HLC Troubleshooting](../../operations/runbooks/hlc-troubleshooting.md)