docs consolidation, big sln build fixes, new advisories and sprints/tasks

This commit is contained in:
master
2026-01-05 18:37:04 +02:00
parent d0a7b88398
commit d7bdca6d97
175 changed files with 10322 additions and 307 deletions

View File

@@ -6,7 +6,7 @@
## Required Reading (treat as read before DOING)
- `docs/README.md`
- `docs/07_HIGH_LEVEL_ARCHITECTURE.md`
- `docs/ARCHITECTURE_OVERVIEW.md`
- `docs/modules/platform/architecture-overview.md`
- `docs/implplan` sprint template rules (see Section “Naming & Structure” below)
- Any sprint-specific upstream docs linked from the current sprint file (e.g., crypto audit, replay runbooks, module architecture dossiers referenced in Dependencies/Prereqs sections)

View File

@@ -10,7 +10,7 @@
- Parallel execution is safe across modules with per-project ownership.
## Documentation Prerequisites
- docs/README.md
- docs/07_HIGH_LEVEL_ARCHITECTURE.md
- docs/ARCHITECTURE_OVERVIEW.md
- docs/modules/platform/architecture-overview.md
- Module dossier for each project under review (docs/modules/<module>/architecture.md).
## Delivery Tracker

View File

@@ -14,7 +14,7 @@
## Documentation Prerequisites
- docs/README.md
- docs/07_HIGH_LEVEL_ARCHITECTURE.md
- docs/ARCHITECTURE_OVERVIEW.md
- AGENTS.md § 8.2 (Deterministic Time & ID Generation)
- Module dossier for each project under refactoring.

View File

@@ -0,0 +1,541 @@
# Sprint 20260105_001_001_BINDEX - Semantic Diffing Phase 1: IR-Level Semantic Analysis
## Topic & Scope
Enhance the BinaryIndex module to leverage B2R2's Intermediate Representation (IR) for semantic-level function comparison, moving beyond instruction-byte normalization to true semantic matching that is resilient to compiler optimizations, instruction reordering, and register allocation differences.
**Advisory Reference:** Product advisory on semantic diffing breakthrough capabilities (Jan 2026)
**Key Insight:** Current implementation normalizes instruction bytes and computes CFG hashes, but does not lift to B2R2's LowUIR/SSA form for semantic analysis. This limits accuracy on optimized/obfuscated binaries by ~15-20%.
**Working directory:** `src/BinaryIndex/`
**Evidence:** New `StellaOps.BinaryIndex.Semantic` library, updated fingerprint generators, integration tests.
---
## Dependencies & Concurrency
| Dependency | Type | Status |
|------------|------|--------|
| B2R2 v0.9.1+ | Package | Available |
| StellaOps.BinaryIndex.Disassembly | Internal | Stable |
| StellaOps.BinaryIndex.Fingerprints | Internal | Stable |
| StellaOps.BinaryIndex.DeltaSig | Internal | Stable |
**Parallel Execution:** Tasks SEMD-001 through SEMD-004 can proceed in parallel. SEMD-005+ depend on foundation work.
---
## Documentation Prerequisites
- `docs/modules/binary-index/architecture.md`
- `docs/modules/binary-index/README.md`
- B2R2 documentation: https://b2r2.org/
- SemDiff paper: https://arxiv.org/abs/2308.01463
---
## Problem Analysis
### Current State
```
Binary Input
|
v
B2R2 Disassembly --> Raw Instructions
|
v
Normalization Pipeline --> Normalized Bytes (position-independent)
|
v
Hash Generation --> BasicBlockHash, CfgHash, StringRefsHash
|
v
Fingerprint Matching --> Similarity Score
```
**Limitations:**
1. **Instruction-level comparison** - Sensitive to register allocation changes
2. **No semantic lifting** - Cannot detect equivalent operations with different instructions
3. **Optimization blindness** - Loop unrolling, inlining, constant propagation break matches
4. **Basic CFG hashing** - Edge counts/hashes miss semantic equivalence
### Target State
```
Binary Input
|
v
B2R2 Disassembly --> Raw Instructions
|
v
B2R2 IR Lifting --> LowUIR Statements
|
v
SSA Transformation --> SSA Form (optional)
|
v
Semantic Graph Extraction --> Key-Semantics Graph (KSG)
|
v
Graph Fingerprinting --> Semantic Fingerprint
|
v
Graph Isomorphism Check --> Semantic Similarity Score
```
---
## Architecture Design
### New Components
#### 1. IR Lifting Service
```csharp
// src/BinaryIndex/__Libraries/StellaOps.BinaryIndex.Semantic/IrLiftingService.cs
namespace StellaOps.BinaryIndex.Semantic;
public interface IIrLiftingService
{
/// <summary>
/// Lift disassembled instructions to B2R2 LowUIR.
/// </summary>
Task<LiftedFunction> LiftToIrAsync(
DisassembledFunction function,
LiftOptions? options = null,
CancellationToken ct = default);
/// <summary>
/// Transform IR to SSA form for dataflow analysis.
/// </summary>
Task<SsaFunction> TransformToSsaAsync(
LiftedFunction lifted,
CancellationToken ct = default);
}
public sealed record LiftedFunction(
string Name,
ulong Address,
ImmutableArray<IrStatement> Statements,
ImmutableArray<IrBasicBlock> BasicBlocks,
ControlFlowGraph Cfg);
public sealed record SsaFunction(
string Name,
ulong Address,
ImmutableArray<SsaStatement> Statements,
ImmutableArray<SsaBasicBlock> BasicBlocks,
DefUseChains DefUse);
```
#### 2. Semantic Graph Extractor
```csharp
// src/BinaryIndex/__Libraries/StellaOps.BinaryIndex.Semantic/SemanticGraphExtractor.cs
namespace StellaOps.BinaryIndex.Semantic;
public interface ISemanticGraphExtractor
{
/// <summary>
/// Extract key-semantics graph from lifted IR.
/// Captures: data dependencies, control dependencies, memory operations.
/// </summary>
Task<KeySemanticsGraph> ExtractGraphAsync(
LiftedFunction function,
GraphExtractionOptions? options = null,
CancellationToken ct = default);
}
public sealed record KeySemanticsGraph(
string FunctionName,
ImmutableArray<SemanticNode> Nodes,
ImmutableArray<SemanticEdge> Edges,
GraphProperties Properties);
public sealed record SemanticNode(
int Id,
SemanticNodeType Type, // Compute, Load, Store, Branch, Call, Return
string Operation, // add, mul, cmp, etc.
ImmutableArray<string> Operands);
public sealed record SemanticEdge(
int SourceId,
int TargetId,
SemanticEdgeType Type); // DataDep, ControlDep, MemoryDep
public enum SemanticNodeType { Compute, Load, Store, Branch, Call, Return, Phi }
public enum SemanticEdgeType { DataDependency, ControlDependency, MemoryDependency }
```
#### 3. Semantic Fingerprint Generator
```csharp
// src/BinaryIndex/__Libraries/StellaOps.BinaryIndex.Semantic/SemanticFingerprintGenerator.cs
namespace StellaOps.BinaryIndex.Semantic;
public interface ISemanticFingerprintGenerator
{
/// <summary>
/// Generate semantic fingerprint from key-semantics graph.
/// </summary>
Task<SemanticFingerprint> GenerateAsync(
KeySemanticsGraph graph,
SemanticFingerprintOptions? options = null,
CancellationToken ct = default);
}
public sealed record SemanticFingerprint(
string FunctionName,
byte[] GraphHash, // 32-byte SHA-256 of canonical graph
byte[] OperationHash, // Hash of operation sequence
byte[] DataFlowHash, // Hash of data dependency patterns
int NodeCount,
int EdgeCount,
int CyclomaticComplexity,
ImmutableArray<string> ApiCalls, // External calls (semantic anchors)
SemanticFingerprintAlgorithm Algorithm);
public enum SemanticFingerprintAlgorithm
{
KsgV1, // Key-Semantics Graph v1
WeisfeilerLehman, // WL graph hashing
GraphletCounting // Graphlet-based similarity
}
```
#### 4. Semantic Matcher
```csharp
// src/BinaryIndex/__Libraries/StellaOps.BinaryIndex.Semantic/SemanticMatcher.cs
namespace StellaOps.BinaryIndex.Semantic;
public interface ISemanticMatcher
{
/// <summary>
/// Compute semantic similarity between two functions.
/// </summary>
Task<SemanticMatchResult> MatchAsync(
SemanticFingerprint a,
SemanticFingerprint b,
MatchOptions? options = null,
CancellationToken ct = default);
/// <summary>
/// Find best matches for a function in a corpus.
/// </summary>
Task<ImmutableArray<SemanticMatchResult>> FindMatchesAsync(
SemanticFingerprint query,
IAsyncEnumerable<SemanticFingerprint> corpus,
decimal minSimilarity = 0.7m,
int maxResults = 10,
CancellationToken ct = default);
}
public sealed record SemanticMatchResult(
string FunctionA,
string FunctionB,
decimal OverallSimilarity,
decimal GraphSimilarity,
decimal DataFlowSimilarity,
decimal ApiCallSimilarity,
MatchConfidence Confidence,
ImmutableArray<MatchDelta> Deltas); // What changed
public enum MatchConfidence { VeryHigh, High, Medium, Low, VeryLow }
public sealed record MatchDelta(
DeltaType Type,
string Description,
decimal Impact);
public enum DeltaType { NodeAdded, NodeRemoved, EdgeAdded, EdgeRemoved, OperationChanged }
```
---
## Delivery Tracker
| # | Task ID | Status | Dependency | Owners | Task Definition |
|---|---------|--------|------------|--------|-----------------|
| 1 | SEMD-001 | TODO | - | Guild | Create `StellaOps.BinaryIndex.Semantic` project structure |
| 2 | SEMD-002 | TODO | - | Guild | Define IR model types (IrStatement, IrBasicBlock, IrOperand) |
| 3 | SEMD-003 | TODO | - | Guild | Define semantic graph model types (KeySemanticsGraph, SemanticNode, SemanticEdge) |
| 4 | SEMD-004 | TODO | - | Guild | Define SemanticFingerprint and matching result types |
| 5 | SEMD-005 | TODO | SEMD-001,002 | Guild | Implement B2R2 IR lifting adapter (LowUIR extraction) |
| 6 | SEMD-006 | TODO | SEMD-005 | Guild | Implement SSA transformation (optional dataflow analysis) |
| 7 | SEMD-007 | TODO | SEMD-003,005 | Guild | Implement KeySemanticsGraph extractor from IR |
| 8 | SEMD-008 | TODO | SEMD-004,007 | Guild | Implement graph canonicalization for deterministic hashing |
| 9 | SEMD-009 | TODO | SEMD-008 | Guild | Implement Weisfeiler-Lehman graph hashing |
| 10 | SEMD-010 | TODO | SEMD-009 | Guild | Implement SemanticFingerprintGenerator |
| 11 | SEMD-011 | TODO | SEMD-010 | Guild | Implement SemanticMatcher with weighted similarity |
| 12 | SEMD-012 | TODO | SEMD-011 | Guild | Integrate semantic fingerprints into PatchDiffEngine |
| 13 | SEMD-013 | TODO | SEMD-012 | Guild | Integrate semantic fingerprints into DeltaSignatureGenerator |
| 14 | SEMD-014 | TODO | SEMD-010 | Guild | Unit tests: IR lifting correctness |
| 15 | SEMD-015 | TODO | SEMD-010 | Guild | Unit tests: Graph extraction determinism |
| 16 | SEMD-016 | TODO | SEMD-011 | Guild | Unit tests: Semantic matching accuracy |
| 17 | SEMD-017 | TODO | SEMD-013 | Guild | Integration tests: End-to-end semantic diffing |
| 18 | SEMD-018 | TODO | SEMD-017 | Guild | Golden corpus: Create test binaries with known semantic equivalences |
| 19 | SEMD-019 | TODO | SEMD-018 | Guild | Benchmark: Compare accuracy vs. instruction-level matching |
| 20 | SEMD-020 | TODO | SEMD-019 | Guild | Documentation: Update architecture.md with semantic diffing |
---
## Task Details
### SEMD-001: Create Project Structure
Create new library project for semantic analysis:
```
src/BinaryIndex/__Libraries/StellaOps.BinaryIndex.Semantic/
StellaOps.BinaryIndex.Semantic.csproj
IrLiftingService.cs
SemanticGraphExtractor.cs
SemanticFingerprintGenerator.cs
SemanticMatcher.cs
Models/
IrModels.cs
GraphModels.cs
FingerprintModels.cs
MatchModels.cs
Internal/
B2R2IrAdapter.cs
GraphCanonicalizer.cs
WeisfeilerLehmanHasher.cs
```
**Acceptance Criteria:**
- [ ] Project builds successfully
- [ ] References StellaOps.BinaryIndex.Disassembly
- [ ] References B2R2.FrontEnd.BinLifter
---
### SEMD-005: Implement B2R2 IR Lifting Adapter
Leverage B2R2's BinLifter to lift raw instructions to LowUIR:
```csharp
internal sealed class B2R2IrAdapter : IIrLiftingService
{
public async Task<LiftedFunction> LiftToIrAsync(
DisassembledFunction function,
LiftOptions? options = null,
CancellationToken ct = default)
{
var handle = BinHandle.FromBytes(
function.Architecture.ToB2R2Isa(),
function.RawBytes);
var lifter = LowUIRHelper.init(handle);
var statements = new List<IrStatement>();
foreach (var instr in function.Instructions)
{
ct.ThrowIfCancellationRequested();
var stmts = LowUIRHelper.translateInstr(lifter, instr.Address);
statements.AddRange(ConvertStatements(stmts));
}
var cfg = BuildControlFlowGraph(statements, function.StartAddress);
return new LiftedFunction(
function.Name,
function.StartAddress,
[.. statements],
ExtractBasicBlocks(cfg),
cfg);
}
}
```
**Acceptance Criteria:**
- [ ] Successfully lifts x64 instructions to IR
- [ ] Successfully lifts ARM64 instructions to IR
- [ ] CFG is correctly constructed
- [ ] Memory operations are properly modeled
---
### SEMD-007: Implement Key-Semantics Graph Extractor
Extract semantic graph capturing:
- **Computation nodes**: Arithmetic, logic, comparison operations
- **Memory nodes**: Load/store operations with abstract addresses
- **Control nodes**: Branches, calls, returns
- **Data dependency edges**: Def-use chains
- **Control dependency edges**: Branch->target relationships
```csharp
internal sealed class KeySemanticsGraphExtractor : ISemanticGraphExtractor
{
public async Task<KeySemanticsGraph> ExtractGraphAsync(
LiftedFunction function,
GraphExtractionOptions? options = null,
CancellationToken ct = default)
{
var nodes = new List<SemanticNode>();
var edges = new List<SemanticEdge>();
var defMap = new Dictionary<string, int>(); // Variable -> defining node
var nodeId = 0;
foreach (var stmt in function.Statements)
{
ct.ThrowIfCancellationRequested();
var node = CreateNode(ref nodeId, stmt);
nodes.Add(node);
// Add data dependency edges
foreach (var use in GetUses(stmt))
{
if (defMap.TryGetValue(use, out var defNode))
{
edges.Add(new SemanticEdge(defNode, node.Id, SemanticEdgeType.DataDependency));
}
}
// Track definitions
foreach (var def in GetDefs(stmt))
{
defMap[def] = node.Id;
}
}
// Add control dependency edges from CFG
AddControlDependencies(function.Cfg, nodes, edges);
return new KeySemanticsGraph(
function.Name,
[.. nodes],
[.. edges],
ComputeProperties(nodes, edges));
}
}
```
---
### SEMD-009: Implement Weisfeiler-Lehman Graph Hashing
WL hashing provides stable graph fingerprints:
```csharp
internal sealed class WeisfeilerLehmanHasher
{
private readonly int _iterations;
public WeisfeilerLehmanHasher(int iterations = 3)
{
_iterations = iterations;
}
public byte[] ComputeHash(KeySemanticsGraph graph)
{
// Initialize labels from node types
var labels = graph.Nodes.ToDictionary(
n => n.Id,
n => ComputeNodeLabel(n));
// WL iteration
for (var i = 0; i < _iterations; i++)
{
var newLabels = new Dictionary<int, string>();
foreach (var node in graph.Nodes)
{
var neighbors = graph.Edges
.Where(e => e.SourceId == node.Id || e.TargetId == node.Id)
.Select(e => e.SourceId == node.Id ? e.TargetId : e.SourceId)
.OrderBy(id => labels[id])
.ToList();
var multiset = string.Join(",", neighbors.Select(id => labels[id]));
var newLabel = ComputeLabel(labels[node.Id], multiset);
newLabels[node.Id] = newLabel;
}
labels = newLabels;
}
// Compute final hash from sorted labels
var sortedLabels = labels.Values.OrderBy(l => l).ToList();
var combined = string.Join("|", sortedLabels);
return SHA256.HashData(Encoding.UTF8.GetBytes(combined));
}
}
```
---
## Testing Strategy
### Unit Tests
| Test Class | Coverage |
|------------|----------|
| `IrLiftingServiceTests` | IR lifting correctness per architecture |
| `SemanticGraphExtractorTests` | Graph construction, edge types, node types |
| `GraphCanonicalizerTests` | Deterministic ordering |
| `WeisfeilerLehmanHasherTests` | Hash stability, collision resistance |
| `SemanticMatcherTests` | Similarity scoring accuracy |
### Integration Tests
| Test Class | Coverage |
|------------|----------|
| `EndToEndSemanticDiffTests` | Full pipeline from binary to match result |
| `OptimizationResilienceTests` | Same source, different optimization levels |
| `CompilerVariantTests` | Same source, GCC vs Clang |
### Golden Corpus
Create test binaries from known C source with variations:
- `test_func_O0.o` - No optimization
- `test_func_O2.o` - Standard optimization
- `test_func_O3.o` - Aggressive optimization
- `test_func_clang.o` - Different compiler
All should match semantically despite instruction differences.
---
## Success Metrics
| Metric | Current | Target |
|--------|---------|--------|
| Semantic match accuracy (optimized binaries) | ~65% | 85%+ |
| False positive rate | ~5% | <2% |
| Match latency (per function) | N/A | <50ms |
| Memory per function | N/A | <10MB |
---
## Execution Log
| Date (UTC) | Update | Owner |
|------------|--------|-------|
| 2026-01-05 | Sprint created from product advisory analysis | Planning |
---
## Decisions & Risks
| Decision/Risk | Type | Mitigation |
|---------------|------|------------|
| B2R2 IR coverage may be incomplete for some instructions | Risk | Fallback to instruction-level matching for unsupported operations |
| WL hashing may produce collisions for small functions | Risk | Combine with operation hash and API call hash |
| SSA transformation adds latency | Trade-off | Make SSA optional, use for high-confidence matching only |
| Graph size explosion for large functions | Risk | Limit node count, use sampling for very large functions |
---
## Next Checkpoints
- 2026-01-10: SEMD-001 through SEMD-004 (project structure, models) complete
- 2026-01-17: SEMD-005 through SEMD-010 (core implementation) complete
- 2026-01-24: SEMD-011 through SEMD-020 (integration, testing, benchmarks) complete

View File

@@ -0,0 +1,592 @@
# Sprint 20260105_001_002_BINDEX - Semantic Diffing Phase 2: Function Behavior Corpus
## Topic & Scope
Build a comprehensive function behavior corpus (similar to Ghidra's BSim/FunctionID) containing fingerprints of known library functions across multiple versions and architectures. This enables identification of functions in stripped binaries by matching against a large corpus of pre-indexed function behaviors.
**Advisory Reference:** Product advisory on semantic diffing - BSim behavioral similarity against large signature sets.
**Key Insight:** Current delta signatures are CVE-specific. A large pre-built corpus of "known good" function behaviors enables identifying functions like "this is `memcpy` from glibc 2.31" even in stripped binaries, which is critical for accurate vulnerability attribution.
**Working directory:** `src/BinaryIndex/`
**Evidence:** New `StellaOps.BinaryIndex.Corpus` library, corpus ingestion pipeline, PostgreSQL corpus schema.
---
## Dependencies & Concurrency
| Dependency | Type | Status |
|------------|------|--------|
| SPRINT_20260105_001_001 (IR Semantics) | Sprint | Required for semantic fingerprints |
| StellaOps.BinaryIndex.Semantic | Internal | From Phase 1 |
| PostgreSQL | Infrastructure | Available |
| Package mirrors (Debian, Alpine, RHEL) | External | Available |
**Parallel Execution:** Corpus connector development (CORP-005-007) can proceed in parallel after CORP-004.
---
## Documentation Prerequisites
- `docs/modules/binary-index/architecture.md`
- Phase 1 sprint: `docs/implplan/SPRINT_20260105_001_001_BINDEX_semdiff_ir_semantics.md`
- Ghidra BSim documentation: https://ghidra.re/ghidra_docs/api/ghidra/features/bsim/BSimServerAPI.html
---
## Problem Analysis
### Current State
- Delta signatures are generated on-demand for specific CVEs
- No pre-built corpus of common library functions
- Cannot identify functions by behavior alone (requires symbols or prior CVE signature)
- Stripped binaries fall back to weaker Build-ID/hash matching
### Target State
```
┌─────────────────────────────────────────────────────────────────────────────┐
│ Function Behavior Corpus │
│ │
│ ┌───────────────────────────────────────────────────────────────────────┐ │
│ │ Corpus Ingestion Layer │ │
│ │ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ │
│ │ │ GlibcCorpus │ │ OpenSSLCorpus│ │ zlibCorpus │ ... │ │
│ │ │ Connector │ │ Connector │ │ Connector │ │ │
│ │ └──────────────┘ └──────────────┘ └──────────────┘ │ │
│ └───────────────────────────────────────────────────────────────────────┘ │
│ │ │
│ v │
│ ┌───────────────────────────────────────────────────────────────────────┐ │
│ │ Fingerprint Generation │ │
│ │ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ │
│ │ │ Instruction │ │ Semantic │ │ API Call │ │ │
│ │ │ Fingerprint │ │ Fingerprint │ │ Fingerprint │ │ │
│ │ └──────────────┘ └──────────────┘ └──────────────┘ │ │
│ └───────────────────────────────────────────────────────────────────────┘ │
│ │ │
│ v │
│ ┌───────────────────────────────────────────────────────────────────────┐ │
│ │ Corpus Storage (PostgreSQL) │ │
│ │ │ │
│ │ corpus.libraries - Known libraries (glibc, openssl, etc.) │ │
│ │ corpus.library_versions - Version snapshots │ │
│ │ corpus.functions - Function metadata │ │
│ │ corpus.fingerprints - Fingerprint index (semantic + instruction) │ │
│ │ corpus.function_clusters - Similar function groups │ │
│ │ │ │
│ └───────────────────────────────────────────────────────────────────────┘ │
│ │ │
│ v │
│ ┌───────────────────────────────────────────────────────────────────────┐ │
│ │ Query Layer │ │
│ │ │ │
│ │ ICorpusQueryService.IdentifyFunctionAsync(fingerprint) │ │
│ │ -> Returns: [{library: "glibc", version: "2.31", name: "memcpy"}] │ │
│ │ │ │
│ └───────────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────┘
```
---
## Architecture Design
### Database Schema
```sql
-- Corpus schema for function behavior database
CREATE SCHEMA IF NOT EXISTS corpus;
-- Known libraries tracked in corpus
CREATE TABLE corpus.libraries (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
name TEXT NOT NULL UNIQUE, -- glibc, openssl, zlib, curl
description TEXT,
homepage_url TEXT,
source_repo TEXT, -- git URL
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
-- Library versions indexed
CREATE TABLE corpus.library_versions (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
library_id UUID NOT NULL REFERENCES corpus.libraries(id),
version TEXT NOT NULL, -- 2.31, 1.1.1n, 1.2.13
release_date DATE,
is_security_release BOOLEAN DEFAULT false,
source_archive_sha256 TEXT, -- Hash of source tarball
indexed_at TIMESTAMPTZ NOT NULL DEFAULT now(),
UNIQUE (library_id, version)
);
-- Architecture variants
CREATE TABLE corpus.build_variants (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
library_version_id UUID NOT NULL REFERENCES corpus.library_versions(id),
architecture TEXT NOT NULL, -- x86_64, aarch64, armv7
abi TEXT, -- gnu, musl, msvc
compiler TEXT, -- gcc, clang
compiler_version TEXT,
optimization_level TEXT, -- O0, O2, O3, Os
build_id TEXT, -- ELF Build-ID if available
binary_sha256 TEXT NOT NULL,
indexed_at TIMESTAMPTZ NOT NULL DEFAULT now(),
UNIQUE (library_version_id, architecture, abi, compiler, optimization_level)
);
-- Functions in corpus
CREATE TABLE corpus.functions (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
build_variant_id UUID NOT NULL REFERENCES corpus.build_variants(id),
name TEXT NOT NULL, -- Function name (may be mangled)
demangled_name TEXT, -- Demangled C++ name
address BIGINT NOT NULL,
size_bytes INTEGER NOT NULL,
is_exported BOOLEAN DEFAULT false,
is_inline BOOLEAN DEFAULT false,
source_file TEXT, -- Source file if debug info
source_line INTEGER,
UNIQUE (build_variant_id, name, address)
);
-- Function fingerprints (multiple algorithms per function)
CREATE TABLE corpus.fingerprints (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
function_id UUID NOT NULL REFERENCES corpus.functions(id),
algorithm TEXT NOT NULL, -- semantic_ksg, instruction_bb, cfg_wl
fingerprint BYTEA NOT NULL, -- Variable length depending on algorithm
fingerprint_hex TEXT GENERATED ALWAYS AS (encode(fingerprint, 'hex')) STORED,
metadata JSONB, -- Algorithm-specific metadata
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
UNIQUE (function_id, algorithm)
);
-- Index for fast fingerprint lookup
CREATE INDEX idx_fingerprints_algorithm_hex ON corpus.fingerprints(algorithm, fingerprint_hex);
CREATE INDEX idx_fingerprints_bytea ON corpus.fingerprints USING hash (fingerprint);
-- Function clusters (similar functions across versions)
CREATE TABLE corpus.function_clusters (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
library_id UUID NOT NULL REFERENCES corpus.libraries(id),
canonical_name TEXT NOT NULL, -- e.g., "memcpy" across all versions
description TEXT,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
UNIQUE (library_id, canonical_name)
);
-- Cluster membership
CREATE TABLE corpus.cluster_members (
cluster_id UUID NOT NULL REFERENCES corpus.function_clusters(id),
function_id UUID NOT NULL REFERENCES corpus.functions(id),
similarity_to_centroid DECIMAL(5,4),
PRIMARY KEY (cluster_id, function_id)
);
-- CVE associations (which functions are affected by which CVEs)
CREATE TABLE corpus.function_cves (
function_id UUID NOT NULL REFERENCES corpus.functions(id),
cve_id TEXT NOT NULL,
affected_state TEXT NOT NULL, -- vulnerable, fixed, not_affected
patch_commit TEXT, -- Git commit that fixed
confidence DECIMAL(3,2) NOT NULL,
evidence_type TEXT, -- changelog, commit, advisory
PRIMARY KEY (function_id, cve_id)
);
-- Ingestion job tracking
CREATE TABLE corpus.ingestion_jobs (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
library_id UUID NOT NULL REFERENCES corpus.libraries(id),
job_type TEXT NOT NULL, -- full_ingest, incremental, cve_update
status TEXT NOT NULL DEFAULT 'pending',
started_at TIMESTAMPTZ,
completed_at TIMESTAMPTZ,
functions_indexed INTEGER,
errors JSONB,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
```
### Core Interfaces
```csharp
// src/BinaryIndex/__Libraries/StellaOps.BinaryIndex.Corpus/ICorpusIngestionService.cs
namespace StellaOps.BinaryIndex.Corpus;
public interface ICorpusIngestionService
{
/// <summary>
/// Ingest all functions from a library binary.
/// </summary>
Task<IngestionResult> IngestLibraryAsync(
LibraryMetadata metadata,
Stream binaryStream,
IngestionOptions? options = null,
CancellationToken ct = default);
/// <summary>
/// Ingest a specific version range.
/// </summary>
Task<ImmutableArray<IngestionResult>> IngestVersionRangeAsync(
string libraryName,
VersionRange range,
IAsyncEnumerable<LibraryBinary> binaries,
CancellationToken ct = default);
}
public sealed record LibraryMetadata(
string Name,
string Version,
string Architecture,
string? Abi,
string? Compiler,
string? OptimizationLevel);
public sealed record IngestionResult(
Guid JobId,
string LibraryName,
string Version,
int FunctionsIndexed,
int FingerprintsGenerated,
ImmutableArray<string> Errors);
```
```csharp
// src/BinaryIndex/__Libraries/StellaOps.BinaryIndex.Corpus/ICorpusQueryService.cs
namespace StellaOps.BinaryIndex.Corpus;
public interface ICorpusQueryService
{
/// <summary>
/// Identify a function by its fingerprint.
/// </summary>
Task<ImmutableArray<FunctionMatch>> IdentifyFunctionAsync(
FunctionFingerprints fingerprints,
IdentifyOptions? options = null,
CancellationToken ct = default);
/// <summary>
/// Get all functions associated with a CVE.
/// </summary>
Task<ImmutableArray<CorpusFunction>> GetFunctionsForCveAsync(
string cveId,
CancellationToken ct = default);
/// <summary>
/// Get function evolution across versions.
/// </summary>
Task<FunctionEvolution> GetFunctionEvolutionAsync(
string libraryName,
string functionName,
CancellationToken ct = default);
}
public sealed record FunctionFingerprints(
byte[]? SemanticHash,
byte[]? InstructionHash,
byte[]? CfgHash,
ImmutableArray<string>? ApiCalls);
public sealed record FunctionMatch(
string LibraryName,
string Version,
string FunctionName,
decimal Similarity,
MatchConfidence Confidence,
string? CveStatus, // null if not CVE-affected
ImmutableArray<string> AffectedCves);
public sealed record FunctionEvolution(
string LibraryName,
string FunctionName,
ImmutableArray<VersionSnapshot> Versions);
public sealed record VersionSnapshot(
string Version,
int SizeBytes,
string FingerprintHex,
ImmutableArray<string> CveChanges); // CVEs fixed/introduced in this version
```
### Library Connectors
```csharp
// src/BinaryIndex/__Libraries/StellaOps.BinaryIndex.Corpus/Connectors/IGlibcCorpusConnector.cs
namespace StellaOps.BinaryIndex.Corpus.Connectors;
public interface ILibraryCorpusConnector
{
string LibraryName { get; }
string[] SupportedArchitectures { get; }
/// <summary>
/// Get available versions from source.
/// </summary>
Task<ImmutableArray<string>> GetAvailableVersionsAsync(CancellationToken ct);
/// <summary>
/// Download and extract library binary for a version.
/// </summary>
Task<LibraryBinary> FetchBinaryAsync(
string version,
string architecture,
string? abi = null,
CancellationToken ct = default);
}
// Implementations:
// - GlibcCorpusConnector (GNU C Library)
// - OpenSslCorpusConnector (OpenSSL/LibreSSL/BoringSSL)
// - ZlibCorpusConnector (zlib/zlib-ng)
// - CurlCorpusConnector (libcurl)
// - SqliteCorpusConnector (SQLite)
// - LibpngCorpusConnector (libpng)
// - LibjpegCorpusConnector (libjpeg-turbo)
// - LibxmlCorpusConnector (libxml2)
// - OpenJpegCorpusConnector (OpenJPEG)
// - ExpatCorpusConnector (Expat XML parser)
```
---
## Delivery Tracker
| # | Task ID | Status | Dependency | Owners | Task Definition |
|---|---------|--------|------------|--------|-----------------|
| 1 | CORP-001 | TODO | Phase 1 | Guild | Create `StellaOps.BinaryIndex.Corpus` project structure |
| 2 | CORP-002 | TODO | CORP-001 | Guild | Define corpus model types (LibraryMetadata, FunctionMatch, etc.) |
| 3 | CORP-003 | TODO | CORP-001 | Guild | Create PostgreSQL corpus schema (corpus.* tables) |
| 4 | CORP-004 | TODO | CORP-003 | Guild | Implement PostgreSQL corpus repository |
| 5 | CORP-005 | TODO | CORP-004 | Guild | Implement GlibcCorpusConnector |
| 6 | CORP-006 | TODO | CORP-004 | Guild | Implement OpenSslCorpusConnector |
| 7 | CORP-007 | TODO | CORP-004 | Guild | Implement ZlibCorpusConnector |
| 8 | CORP-008 | TODO | CORP-004 | Guild | Implement CurlCorpusConnector |
| 9 | CORP-009 | TODO | CORP-005-008 | Guild | Implement CorpusIngestionService |
| 10 | CORP-010 | TODO | CORP-009 | Guild | Implement batch fingerprint generation pipeline |
| 11 | CORP-011 | TODO | CORP-010 | Guild | Implement function clustering (group similar functions) |
| 12 | CORP-012 | TODO | CORP-011 | Guild | Implement CorpusQueryService |
| 13 | CORP-013 | TODO | CORP-012 | Guild | Implement CVE-to-function mapping updater |
| 14 | CORP-014 | TODO | CORP-012 | Guild | Integrate corpus queries into BinaryVulnerabilityService |
| 15 | CORP-015 | TODO | CORP-009 | Guild | Initial corpus ingestion: glibc (5 major versions x 3 archs) |
| 16 | CORP-016 | TODO | CORP-015 | Guild | Initial corpus ingestion: OpenSSL (10 versions x 3 archs) |
| 17 | CORP-017 | TODO | CORP-016 | Guild | Initial corpus ingestion: zlib, curl, sqlite |
| 18 | CORP-018 | TODO | CORP-012 | Guild | Unit tests: Corpus ingestion correctness |
| 19 | CORP-019 | TODO | CORP-012 | Guild | Unit tests: Query service accuracy |
| 20 | CORP-020 | TODO | CORP-017 | Guild | Integration tests: End-to-end function identification |
| 21 | CORP-021 | TODO | CORP-020 | Guild | Benchmark: Query latency at scale (100K+ functions) |
| 22 | CORP-022 | TODO | CORP-021 | Guild | Documentation: Corpus management guide |
---
## Task Details
### CORP-005: Implement GlibcCorpusConnector
Fetch glibc binaries from GNU mirrors and Debian/Ubuntu packages:
```csharp
internal sealed class GlibcCorpusConnector : ILibraryCorpusConnector
{
private readonly IHttpClientFactory _httpClientFactory;
private readonly ILogger<GlibcCorpusConnector> _logger;
public string LibraryName => "glibc";
public string[] SupportedArchitectures => ["x86_64", "aarch64", "armv7", "i686"];
public async Task<ImmutableArray<string>> GetAvailableVersionsAsync(CancellationToken ct)
{
// Query GNU FTP mirror for available versions
// https://ftp.gnu.org/gnu/glibc/
var client = _httpClientFactory.CreateClient("GnuMirror");
var html = await client.GetStringAsync("https://ftp.gnu.org/gnu/glibc/", ct);
// Parse directory listing for glibc-X.Y.tar.gz files
var versions = ParseVersionsFromListing(html);
return [.. versions.OrderByDescending(v => Version.Parse(v))];
}
public async Task<LibraryBinary> FetchBinaryAsync(
string version,
string architecture,
string? abi = null,
CancellationToken ct = default)
{
// Strategy 1: Try Debian/Ubuntu package (pre-built)
var debBinary = await TryFetchDebianPackageAsync(version, architecture, ct);
if (debBinary is not null)
return debBinary;
// Strategy 2: Download source and compile with specific flags
var sourceTarball = await DownloadSourceAsync(version, ct);
return await CompileForArchitecture(sourceTarball, architecture, abi, ct);
}
private async Task<LibraryBinary?> TryFetchDebianPackageAsync(
string version,
string architecture,
CancellationToken ct)
{
// Map glibc version to Debian package version
// e.g., glibc 2.31 -> libc6_2.31-13+deb11u5_amd64.deb
var packages = await QueryDebianPackagesAsync(version, architecture, ct);
foreach (var pkg in packages)
{
var binary = await DownloadAndExtractDebAsync(pkg, ct);
if (binary is not null)
return binary;
}
return null;
}
}
```
### CORP-011: Implement Function Clustering
Group semantically similar functions across versions:
```csharp
internal sealed class FunctionClusteringService
{
private readonly ICorpusRepository _repository;
private readonly ISemanticMatcher _matcher;
public async Task ClusterFunctionsAsync(
Guid libraryId,
ClusteringOptions options,
CancellationToken ct)
{
// Get all functions with semantic fingerprints
var functions = await _repository.GetFunctionsWithFingerprintsAsync(libraryId, ct);
// Group by canonical name (demangled, normalized)
var groups = functions
.GroupBy(f => NormalizeCanonicalName(f.DemangledName ?? f.Name))
.ToList();
foreach (var group in groups)
{
ct.ThrowIfCancellationRequested();
// Create or update cluster
var clusterId = await _repository.EnsureClusterAsync(
libraryId,
group.Key,
ct);
// Compute centroid (most common fingerprint)
var centroid = ComputeCentroid(group);
// Add members with similarity scores
foreach (var function in group)
{
var similarity = await _matcher.MatchAsync(
function.SemanticFingerprint,
centroid,
ct: ct);
await _repository.AddClusterMemberAsync(
clusterId,
function.Id,
similarity.OverallSimilarity,
ct);
}
}
}
private static string NormalizeCanonicalName(string name)
{
// Strip version suffixes, GLIBC_2.X annotations
// Demangle C++ names
// Normalize to base function name
return CppDemangler.Demangle(name)
.Replace("@GLIBC_", "")
.TrimEnd("@@".ToCharArray());
}
}
```
---
## Initial Corpus Coverage
### Priority Libraries (Phase 2a)
| Library | Versions | Architectures | Est. Functions | CVE Coverage |
|---------|----------|---------------|----------------|--------------|
| glibc | 2.17, 2.28, 2.31, 2.35, 2.38 | x64, arm64, armv7 | ~15,000 | 50+ CVEs |
| OpenSSL | 1.0.2, 1.1.0, 1.1.1, 3.0, 3.1 | x64, arm64 | ~8,000 | 100+ CVEs |
| zlib | 1.2.8, 1.2.11, 1.2.13, 1.3 | x64, arm64 | ~200 | 5+ CVEs |
| libcurl | 7.50-7.88 (select) | x64, arm64 | ~2,000 | 80+ CVEs |
| SQLite | 3.30-3.44 (select) | x64, arm64 | ~1,500 | 30+ CVEs |
### Extended Coverage (Phase 2b)
| Library | Est. Functions | Priority |
|---------|----------------|----------|
| libpng | ~300 | Medium |
| libjpeg-turbo | ~400 | Medium |
| libxml2 | ~1,200 | High |
| expat | ~150 | High |
| OpenJPEG | ~600 | Medium |
| freetype | ~800 | Medium |
| harfbuzz | ~500 | Low |
**Total estimated corpus size:** ~30,000 unique functions, ~100,000 fingerprints (including variants)
---
## Storage Estimates
| Component | Size Estimate |
|-----------|---------------|
| PostgreSQL tables | ~2 GB |
| Fingerprint index | ~500 MB |
| Full corpus with metadata | ~5 GB |
| Query cache (Valkey) | ~100 MB |
---
## Success Metrics
| Metric | Target |
|--------|--------|
| Function identification accuracy | 90%+ on stripped binaries |
| Query latency (p99) | <100ms |
| Corpus coverage (top 20 libs) | 80%+ of security-critical functions |
| CVE attribution accuracy | 95%+ |
| False positive rate | <3% |
---
## Execution Log
| Date (UTC) | Update | Owner |
|------------|--------|-------|
| 2026-01-05 | Sprint created from product advisory analysis | Planning |
---
## Decisions & Risks
| Decision/Risk | Type | Mitigation |
|---------------|------|------------|
| Corpus size may grow large | Risk | Implement tiered storage, archive old versions |
| Package version mapping is complex | Risk | Maintain distro-version mapping tables |
| Compilation variants create explosion | Risk | Prioritize common optimization levels (O2, O3) |
| CVE mapping requires manual curation | Risk | Start with high-impact CVEs, automate with NVD data |
---
## Next Checkpoints
- 2026-01-20: CORP-001 through CORP-008 (infrastructure, connectors) complete
- 2026-01-31: CORP-009 through CORP-014 (services, integration) complete
- 2026-02-15: CORP-015 through CORP-022 (corpus ingestion, testing) complete

View File

@@ -0,0 +1,772 @@
# Sprint 20260105_001_003_BINDEX - Semantic Diffing Phase 3: Ghidra Integration
## Topic & Scope
Integrate Ghidra as a secondary analysis backend for cases where B2R2 provides insufficient coverage or accuracy. Leverage Ghidra's mature Version Tracking, BSim, and FunctionID capabilities via headless analysis and the ghidriff Python bridge.
**Advisory Reference:** Product advisory on semantic diffing - Ghidra Version Tracking correlators, BSim behavioral similarity, ghidriff for automated patch diff workflows.
**Key Insight:** Ghidra has 15+ years of refinement in binary diffing. Rather than reimplementing, we should integrate Ghidra as a fallback/enhancement layer for:
1. Architectures B2R2 handles poorly
2. Complex obfuscation scenarios
3. Version Tracking with multiple correlators
4. BSim database queries
**Working directory:** `src/BinaryIndex/`
**Evidence:** New `StellaOps.BinaryIndex.Ghidra` library, Ghidra Headless integration, ghidriff bridge.
---
## Dependencies & Concurrency
| Dependency | Type | Status |
|------------|------|--------|
| SPRINT_20260105_001_001 (IR Semantics) | Sprint | Should be complete |
| SPRINT_20260105_001_002 (Corpus) | Sprint | Can run in parallel |
| Ghidra 11.x | External | Available |
| Java 17+ | Runtime | Required for Ghidra |
| Python 3.10+ | Runtime | Required for ghidriff |
| ghidriff | External | Available (pip) |
**Parallel Execution:** Ghidra Headless setup (GHID-001-004) and ghidriff integration (GHID-005-008) can proceed in parallel.
---
## Documentation Prerequisites
- `docs/modules/binary-index/architecture.md`
- Ghidra documentation: https://ghidra.re/ghidra_docs/
- Ghidra Version Tracking: https://cve-north-stars.github.io/docs/Ghidra-Patch-Diffing
- ghidriff repository: https://github.com/clearbluejar/ghidriff
- BSim documentation: https://ghidra.re/ghidra_docs/api/ghidra/features/bsim/
---
## Problem Analysis
### Current State
- B2R2 is the sole disassembly/analysis backend
- B2R2 coverage varies by architecture (excellent x64/ARM64, limited others)
- No access to Ghidra's mature correlators and similarity engines
- Cannot leverage BSim's pre-built signature databases
### B2R2 vs Ghidra Trade-offs
| Capability | B2R2 | Ghidra |
|------------|------|--------|
| Speed | Fast (native .NET) | Slower (Java, headless startup) |
| Architecture coverage | 12+ (some limited) | 20+ (mature) |
| IR quality | Good (LowUIR) | Excellent (P-Code) |
| Decompiler | None | Excellent |
| Version Tracking | None | Mature (multiple correlators) |
| BSim | None | Full support |
| Integration | Native .NET | Process/API bridge |
### Target Architecture
```
┌─────────────────────────────────────────────────────────────────────────────┐
│ Unified Disassembly/Analysis Layer │
│ │
│ ┌───────────────────────────────────────────────────────────────────────┐ │
│ │ IDisassemblyPlugin Selection Logic │ │
│ │ │ │
│ │ Primary: B2R2 (fast, deterministic) │ │
│ │ Fallback: Ghidra (complex cases, low B2R2 confidence) │ │
│ │ │ │
│ └───────────────────────────────────────────────────────────────────────┘ │
│ │ │ │
│ v v │
│ ┌──────────────────────────┐ ┌──────────────────────────────────────┐ │
│ │ B2R2 Backend │ │ Ghidra Backend │ │
│ │ │ │ │ │
│ │ - Native .NET │ │ ┌────────────────────────────────┐ │ │
│ │ - LowUIR lifting │ │ │ Ghidra Headless Server │ │ │
│ │ - CFG recovery │ │ │ │ │ │
│ │ - Fast fingerprinting │ │ │ - P-Code decompilation │ │ │
│ │ │ │ │ - Version Tracking │ │ │
│ └──────────────────────────┘ │ │ - BSim queries │ │ │
│ │ │ - FunctionID matching │ │ │
│ │ └────────────────────────────────┘ │ │
│ │ │ │ │
│ │ v │ │
│ │ ┌────────────────────────────────┐ │ │
│ │ │ ghidriff Bridge │ │ │
│ │ │ │ │ │
│ │ │ - Automated patch diffing │ │ │
│ │ │ - JSON/Markdown output │ │ │
│ │ │ - CI/CD integration │ │ │
│ │ └────────────────────────────────┘ │ │
│ └──────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────┘
```
---
## Architecture Design
### Ghidra Headless Service
```csharp
// src/BinaryIndex/__Libraries/StellaOps.BinaryIndex.Ghidra/IGhidraService.cs
namespace StellaOps.BinaryIndex.Ghidra;
public interface IGhidraService
{
/// <summary>
/// Analyze a binary using Ghidra headless.
/// </summary>
Task<GhidraAnalysisResult> AnalyzeAsync(
Stream binaryStream,
GhidraAnalysisOptions? options = null,
CancellationToken ct = default);
/// <summary>
/// Run Version Tracking between two binaries.
/// </summary>
Task<VersionTrackingResult> CompareVersionsAsync(
Stream oldBinary,
Stream newBinary,
VersionTrackingOptions? options = null,
CancellationToken ct = default);
/// <summary>
/// Query BSim for function matches.
/// </summary>
Task<ImmutableArray<BSimMatch>> QueryBSimAsync(
GhidraFunction function,
BSimQueryOptions? options = null,
CancellationToken ct = default);
/// <summary>
/// Check if Ghidra backend is available and healthy.
/// </summary>
Task<bool> IsAvailableAsync(CancellationToken ct = default);
}
public sealed record GhidraAnalysisResult(
string BinaryHash,
ImmutableArray<GhidraFunction> Functions,
ImmutableArray<GhidraImport> Imports,
ImmutableArray<GhidraExport> Exports,
ImmutableArray<GhidraString> Strings,
GhidraMetadata Metadata);
public sealed record GhidraFunction(
string Name,
ulong Address,
int Size,
string? Signature, // Decompiled signature
string? DecompiledCode, // Decompiled C code
byte[] PCodeHash, // P-Code semantic hash
ImmutableArray<string> CalledFunctions,
ImmutableArray<string> CallingFunctions);
```
### Version Tracking Integration
```csharp
// src/BinaryIndex/__Libraries/StellaOps.BinaryIndex.Ghidra/IVersionTrackingService.cs
namespace StellaOps.BinaryIndex.Ghidra;
public interface IVersionTrackingService
{
/// <summary>
/// Run Ghidra Version Tracking with multiple correlators.
/// </summary>
Task<VersionTrackingResult> TrackVersionsAsync(
Stream oldBinary,
Stream newBinary,
VersionTrackingOptions options,
CancellationToken ct = default);
}
public sealed record VersionTrackingOptions
{
public ImmutableArray<CorrelatorType> Correlators { get; init; } =
[CorrelatorType.ExactBytes, CorrelatorType.ExactMnemonics,
CorrelatorType.SymbolName, CorrelatorType.DataReference,
CorrelatorType.CombinedReference];
public decimal MinSimilarity { get; init; } = 0.5m;
public bool IncludeDecompilation { get; init; } = false;
}
public enum CorrelatorType
{
ExactBytes, // Identical byte sequences
ExactMnemonics, // Identical instruction mnemonics
SymbolName, // Matching symbol names
DataReference, // Similar data references
CombinedReference, // Combined reference scoring
BSim // Behavioral similarity
}
public sealed record VersionTrackingResult(
ImmutableArray<FunctionMatch> Matches,
ImmutableArray<FunctionAdded> AddedFunctions,
ImmutableArray<FunctionRemoved> RemovedFunctions,
ImmutableArray<FunctionModified> ModifiedFunctions,
VersionTrackingStats Statistics);
public sealed record FunctionMatch(
string OldName,
ulong OldAddress,
string NewName,
ulong NewAddress,
decimal Similarity,
CorrelatorType MatchedBy,
ImmutableArray<MatchDifference> Differences);
public sealed record MatchDifference(
DifferenceType Type,
string Description,
string? OldValue,
string? NewValue);
public enum DifferenceType
{
InstructionAdded,
InstructionRemoved,
InstructionChanged,
BranchTargetChanged,
CallTargetChanged,
ConstantChanged,
SizeChanged
}
```
### ghidriff Bridge
```csharp
// src/BinaryIndex/__Libraries/StellaOps.BinaryIndex.Ghidra/IGhidriffBridge.cs
namespace StellaOps.BinaryIndex.Ghidra;
public interface IGhidriffBridge
{
/// <summary>
/// Run ghidriff to compare two binaries.
/// </summary>
Task<GhidriffResult> DiffAsync(
string oldBinaryPath,
string newBinaryPath,
GhidriffOptions? options = null,
CancellationToken ct = default);
/// <summary>
/// Generate patch diff report.
/// </summary>
Task<string> GenerateReportAsync(
GhidriffResult result,
ReportFormat format,
CancellationToken ct = default);
}
public sealed record GhidriffOptions
{
public string? GhidraPath { get; init; }
public string? ProjectPath { get; init; }
public bool IncludeDecompilation { get; init; } = true;
public bool IncludeDisassembly { get; init; } = true;
public ImmutableArray<string> ExcludeFunctions { get; init; } = [];
}
public sealed record GhidriffResult(
string OldBinaryHash,
string NewBinaryHash,
ImmutableArray<GhidriffFunction> AddedFunctions,
ImmutableArray<GhidriffFunction> RemovedFunctions,
ImmutableArray<GhidriffDiff> ModifiedFunctions,
GhidriffStats Statistics,
string RawJsonOutput);
public sealed record GhidriffDiff(
string FunctionName,
string OldSignature,
string NewSignature,
decimal Similarity,
string? OldDecompiled,
string? NewDecompiled,
ImmutableArray<string> InstructionChanges);
public enum ReportFormat { Json, Markdown, Html }
```
### BSim Integration
```csharp
// src/BinaryIndex/__Libraries/StellaOps.BinaryIndex.Ghidra/IBSimService.cs
namespace StellaOps.BinaryIndex.Ghidra;
public interface IBSimService
{
/// <summary>
/// Generate BSim signatures for functions.
/// </summary>
Task<ImmutableArray<BSimSignature>> GenerateSignaturesAsync(
GhidraAnalysisResult analysis,
BSimGenerationOptions? options = null,
CancellationToken ct = default);
/// <summary>
/// Query BSim database for similar functions.
/// </summary>
Task<ImmutableArray<BSimMatch>> QueryAsync(
BSimSignature signature,
BSimQueryOptions? options = null,
CancellationToken ct = default);
/// <summary>
/// Ingest functions into BSim database.
/// </summary>
Task IngestAsync(
string libraryName,
string version,
ImmutableArray<BSimSignature> signatures,
CancellationToken ct = default);
}
public sealed record BSimSignature(
string FunctionName,
ulong Address,
byte[] FeatureVector, // BSim feature extraction
int VectorLength,
double SelfSignificance); // How distinctive is this function
public sealed record BSimMatch(
string MatchedLibrary,
string MatchedVersion,
string MatchedFunction,
double Similarity,
double Significance,
double Confidence);
public sealed record BSimQueryOptions
{
public double MinSimilarity { get; init; } = 0.7;
public double MinSignificance { get; init; } = 0.0;
public int MaxResults { get; init; } = 10;
public ImmutableArray<string> TargetLibraries { get; init; } = [];
}
```
---
## Delivery Tracker
| # | Task ID | Status | Dependency | Owners | Task Definition |
|---|---------|--------|------------|--------|-----------------|
| 1 | GHID-001 | TODO | - | Guild | Create `StellaOps.BinaryIndex.Ghidra` project structure |
| 2 | GHID-002 | TODO | GHID-001 | Guild | Define Ghidra model types (GhidraFunction, VersionTrackingResult, etc.) |
| 3 | GHID-003 | TODO | GHID-001 | Guild | Implement Ghidra Headless launcher/manager |
| 4 | GHID-004 | TODO | GHID-003 | Guild | Implement GhidraService (headless analysis wrapper) |
| 5 | GHID-005 | TODO | GHID-001 | Guild | Set up ghidriff Python environment |
| 6 | GHID-006 | TODO | GHID-005 | Guild | Implement GhidriffBridge (Python interop) |
| 7 | GHID-007 | TODO | GHID-006 | Guild | Implement GhidriffReportGenerator |
| 8 | GHID-008 | TODO | GHID-004,006 | Guild | Implement VersionTrackingService |
| 9 | GHID-009 | TODO | GHID-004 | Guild | Implement BSim signature generation |
| 10 | GHID-010 | TODO | GHID-009 | Guild | Implement BSim query service |
| 11 | GHID-011 | TODO | GHID-010 | Guild | Set up BSim PostgreSQL database |
| 12 | GHID-012 | TODO | GHID-008,010 | Guild | Implement GhidraDisassemblyPlugin (IDisassemblyPlugin) |
| 13 | GHID-013 | TODO | GHID-012 | Guild | Integrate Ghidra into DisassemblyService as fallback |
| 14 | GHID-014 | TODO | GHID-013 | Guild | Implement fallback selection logic (B2R2 -> Ghidra) |
| 15 | GHID-015 | TODO | GHID-008 | Guild | Unit tests: Version Tracking correlators |
| 16 | GHID-016 | TODO | GHID-010 | Guild | Unit tests: BSim signature generation |
| 17 | GHID-017 | TODO | GHID-014 | Guild | Integration tests: Fallback scenarios |
| 18 | GHID-018 | TODO | GHID-017 | Guild | Benchmark: Ghidra vs B2R2 accuracy comparison |
| 19 | GHID-019 | TODO | GHID-018 | Guild | Documentation: Ghidra deployment guide |
| 20 | GHID-020 | TODO | GHID-019 | Guild | Docker image: Ghidra Headless service |
---
## Task Details
### GHID-003: Implement Ghidra Headless Launcher
Manage Ghidra Headless process lifecycle:
```csharp
internal sealed class GhidraHeadlessManager : IAsyncDisposable
{
private readonly GhidraOptions _options;
private readonly ILogger<GhidraHeadlessManager> _logger;
private Process? _ghidraProcess;
private readonly SemaphoreSlim _lock = new(1, 1);
public GhidraHeadlessManager(
IOptions<GhidraOptions> options,
ILogger<GhidraHeadlessManager> logger)
{
_options = options.Value;
_logger = logger;
}
public async Task<string> AnalyzeAsync(
string binaryPath,
string scriptName,
string[] scriptArgs,
CancellationToken ct)
{
await _lock.WaitAsync(ct);
try
{
var projectDir = Path.Combine(_options.WorkDir, Guid.NewGuid().ToString("N"));
Directory.CreateDirectory(projectDir);
var args = BuildAnalyzeArgs(projectDir, binaryPath, scriptName, scriptArgs);
var result = await RunGhidraAsync(args, ct);
return result;
}
finally
{
_lock.Release();
}
}
private string[] BuildAnalyzeArgs(
string projectDir,
string binaryPath,
string scriptName,
string[] scriptArgs)
{
var args = new List<string>
{
projectDir, // Project location
"TempProject", // Project name
"-import", binaryPath,
"-postScript", scriptName
};
if (scriptArgs.Length > 0)
{
args.AddRange(scriptArgs);
}
// Add standard options
args.AddRange([
"-noanalysis", // We'll run analysis explicitly
"-scriptPath", _options.ScriptsDir,
"-max-cpu", _options.MaxCpu.ToString(CultureInfo.InvariantCulture)
]);
return [.. args];
}
private async Task<string> RunGhidraAsync(string[] args, CancellationToken ct)
{
var startInfo = new ProcessStartInfo
{
FileName = Path.Combine(_options.GhidraHome, "support", "analyzeHeadless"),
Arguments = string.Join(" ", args.Select(QuoteArg)),
RedirectStandardOutput = true,
RedirectStandardError = true,
UseShellExecute = false,
CreateNoWindow = true
};
// Set Java options
startInfo.EnvironmentVariables["JAVA_HOME"] = _options.JavaHome;
startInfo.EnvironmentVariables["MAXMEM"] = _options.MaxMemory;
using var process = Process.Start(startInfo)
?? throw new InvalidOperationException("Failed to start Ghidra");
var output = await process.StandardOutput.ReadToEndAsync(ct);
var error = await process.StandardError.ReadToEndAsync(ct);
await process.WaitForExitAsync(ct);
if (process.ExitCode != 0)
{
throw new GhidraException($"Ghidra failed: {error}");
}
return output;
}
}
```
### GHID-006: Implement ghidriff Bridge
Python interop for ghidriff:
```csharp
internal sealed class GhidriffBridge : IGhidriffBridge
{
private readonly GhidriffOptions _options;
private readonly ILogger<GhidriffBridge> _logger;
public async Task<GhidriffResult> DiffAsync(
string oldBinaryPath,
string newBinaryPath,
GhidriffOptions? options = null,
CancellationToken ct = default)
{
options ??= _options;
var outputDir = Path.Combine(Path.GetTempPath(), $"ghidriff_{Guid.NewGuid():N}");
Directory.CreateDirectory(outputDir);
try
{
var args = BuildGhidriffArgs(oldBinaryPath, newBinaryPath, outputDir, options);
var result = await RunPythonAsync("ghidriff", args, ct);
// Parse JSON output
var jsonPath = Path.Combine(outputDir, "diff.json");
if (!File.Exists(jsonPath))
{
throw new GhidriffException($"ghidriff did not produce output: {result}");
}
var json = await File.ReadAllTextAsync(jsonPath, ct);
return ParseGhidriffOutput(json);
}
finally
{
if (Directory.Exists(outputDir))
{
Directory.Delete(outputDir, recursive: true);
}
}
}
private static string[] BuildGhidriffArgs(
string oldPath,
string newPath,
string outputDir,
GhidriffOptions options)
{
var args = new List<string>
{
oldPath,
newPath,
"--output-dir", outputDir,
"--output-format", "json"
};
if (!string.IsNullOrEmpty(options.GhidraPath))
{
args.AddRange(["--ghidra-path", options.GhidraPath]);
}
if (options.IncludeDecompilation)
{
args.Add("--include-decompilation");
}
if (options.ExcludeFunctions.Length > 0)
{
args.AddRange(["--exclude", string.Join(",", options.ExcludeFunctions)]);
}
return [.. args];
}
private async Task<string> RunPythonAsync(
string module,
string[] args,
CancellationToken ct)
{
var startInfo = new ProcessStartInfo
{
FileName = _options.PythonPath ?? "python3",
Arguments = $"-m {module} {string.Join(" ", args.Select(QuoteArg))}",
RedirectStandardOutput = true,
RedirectStandardError = true,
UseShellExecute = false,
CreateNoWindow = true
};
using var process = Process.Start(startInfo)
?? throw new InvalidOperationException("Failed to start Python");
var output = await process.StandardOutput.ReadToEndAsync(ct);
await process.WaitForExitAsync(ct);
return output;
}
}
```
### GHID-014: Implement Fallback Selection Logic
Smart routing between B2R2 and Ghidra:
```csharp
internal sealed class HybridDisassemblyService : IDisassemblyService
{
private readonly B2R2DisassemblyPlugin _b2r2;
private readonly GhidraDisassemblyPlugin _ghidra;
private readonly ILogger<HybridDisassemblyService> _logger;
public async Task<DisassemblyResult> DisassembleAsync(
Stream binaryStream,
DisassemblyOptions? options = null,
CancellationToken ct = default)
{
options ??= new DisassemblyOptions();
// Try B2R2 first (faster, native)
var b2r2Result = await TryB2R2Async(binaryStream, options, ct);
if (b2r2Result is not null && MeetsQualityThreshold(b2r2Result, options))
{
_logger.LogDebug("Using B2R2 result (confidence: {Confidence})",
b2r2Result.Confidence);
return b2r2Result;
}
// Fallback to Ghidra for:
// 1. Low B2R2 confidence
// 2. Unsupported architecture
// 3. Explicit Ghidra preference
if (!await _ghidra.IsAvailableAsync(ct))
{
_logger.LogWarning("Ghidra unavailable, returning B2R2 result");
return b2r2Result ?? throw new DisassemblyException("No backend available");
}
_logger.LogInformation("Falling back to Ghidra (B2R2 confidence: {Confidence})",
b2r2Result?.Confidence ?? 0);
binaryStream.Position = 0;
return await _ghidra.DisassembleAsync(binaryStream, options, ct);
}
private static bool MeetsQualityThreshold(
DisassemblyResult result,
DisassemblyOptions options)
{
// Confidence threshold
if (result.Confidence < options.MinConfidence)
return false;
// Function discovery threshold
if (result.Functions.Length < options.MinFunctions)
return false;
// Instruction decoding success rate
var decodeRate = (double)result.DecodedInstructions / result.TotalInstructions;
if (decodeRate < options.MinDecodeRate)
return false;
return true;
}
}
```
---
## Deployment Architecture
### Container Setup
```yaml
# docker-compose.ghidra.yml
services:
ghidra-headless:
image: stellaops/ghidra-headless:11.2
build:
context: ./devops/docker/ghidra
dockerfile: Dockerfile.headless
volumes:
- ghidra-projects:/projects
- ghidra-scripts:/scripts
environment:
JAVA_HOME: /opt/java/openjdk
MAXMEM: 4G
deploy:
resources:
limits:
cpus: '4'
memory: 8G
bsim-postgres:
image: postgres:16
volumes:
- bsim-data:/var/lib/postgresql/data
environment:
POSTGRES_DB: bsim
POSTGRES_USER: bsim
POSTGRES_PASSWORD: ${BSIM_DB_PASSWORD}
volumes:
ghidra-projects:
ghidra-scripts:
bsim-data:
```
### Dockerfile
```dockerfile
# devops/docker/ghidra/Dockerfile.headless
FROM eclipse-temurin:17-jdk-jammy
ARG GHIDRA_VERSION=11.2
ARG GHIDRA_SHA256=abc123...
# Download and extract Ghidra
RUN curl -fsSL https://github.com/NationalSecurityAgency/ghidra/releases/download/Ghidra_${GHIDRA_VERSION}_build/ghidra_${GHIDRA_VERSION}_PUBLIC_*.zip \
-o /tmp/ghidra.zip \
&& echo "${GHIDRA_SHA256} /tmp/ghidra.zip" | sha256sum -c - \
&& unzip /tmp/ghidra.zip -d /opt \
&& rm /tmp/ghidra.zip \
&& ln -s /opt/ghidra_* /opt/ghidra
# Install Python for ghidriff
RUN apt-get update && apt-get install -y python3 python3-pip \
&& pip3 install ghidriff \
&& apt-get clean
ENV GHIDRA_HOME=/opt/ghidra
ENV PATH="${GHIDRA_HOME}/support:${PATH}"
WORKDIR /projects
ENTRYPOINT ["analyzeHeadless"]
```
---
## Success Metrics
| Metric | Current | Target |
|--------|---------|--------|
| Architecture coverage | 12 (B2R2) | 20+ (with Ghidra) |
| Complex binary accuracy | ~70% | 90%+ |
| Version tracking precision | N/A | 85%+ |
| BSim identification rate | N/A | 80%+ on known libs |
| Fallback latency overhead | N/A | <30s per binary |
---
## Execution Log
| Date (UTC) | Update | Owner |
|------------|--------|-------|
| 2026-01-05 | Sprint created from product advisory analysis | Planning |
---
## Decisions & Risks
| Decision/Risk | Type | Mitigation |
|---------------|------|------------|
| Ghidra adds Java dependency | Trade-off | Containerize Ghidra, keep optional |
| ghidriff Python interop adds complexity | Trade-off | Use subprocess, avoid embedding |
| Ghidra startup time is slow (~10-30s) | Risk | Keep B2R2 primary, Ghidra fallback only |
| BSim database grows large | Risk | Prune old versions, tier storage |
| License considerations (Apache 2.0) | Compliance | Ghidra is Apache 2.0, compatible with AGPL |
---
## Next Checkpoints
- 2026-02-01: GHID-001 through GHID-007 (project setup, bridges) complete
- 2026-02-15: GHID-008 through GHID-014 (services, integration) complete
- 2026-02-28: GHID-015 through GHID-020 (testing, deployment) complete

View File

@@ -0,0 +1,906 @@
# Sprint 20260105_001_004_BINDEX - Semantic Diffing Phase 4: Decompiler Integration & ML Similarity
## Topic & Scope
Implement advanced semantic analysis capabilities including decompiled pseudo-code comparison and machine learning-based function embeddings. This phase addresses the highest-impact but most complex enhancements for detecting semantic equivalence in heavily optimized and obfuscated binaries.
**Advisory Reference:** Product advisory on semantic diffing - SEI Carnegie Mellon semantic equivalence checking of decompiled binaries, ML-based similarity models.
**Key Insight:** Comparing decompiled C-like code provides the highest semantic fidelity, as it abstracts away instruction-level details. ML embeddings capture functional behavior patterns that resist obfuscation.
**Working directory:** `src/BinaryIndex/`
**Evidence:** New `StellaOps.BinaryIndex.Decompiler` and `StellaOps.BinaryIndex.ML` libraries, model training pipeline.
---
## Dependencies & Concurrency
| Dependency | Type | Status |
|------------|------|--------|
| SPRINT_20260105_001_001 (IR Semantics) | Sprint | Required |
| SPRINT_20260105_001_002 (Corpus) | Sprint | Required for training data |
| SPRINT_20260105_001_003 (Ghidra) | Sprint | Required for decompiler |
| Ghidra Decompiler | External | Via Phase 3 |
| ONNX Runtime | Package | Available |
| ML.NET | Package | Available |
**Parallel Execution:** Decompiler integration (DCML-001-010) and ML pipeline (DCML-011-020) can proceed in parallel.
---
## Documentation Prerequisites
- Phase 1-3 sprint documents
- `docs/modules/binary-index/architecture.md`
- SEI paper: https://www.sei.cmu.edu/annual-reviews/2022-research-review/semantic-equivalence-checking-of-decompiled-binaries/
- Code similarity research: https://arxiv.org/abs/2308.01463
---
## Problem Analysis
### Current State
After Phases 1-3:
- B2R2 IR-level semantic fingerprints (Phase 1)
- Function behavior corpus (Phase 2)
- Ghidra fallback with Version Tracking (Phase 3)
**Remaining Gaps:**
1. No decompiled code comparison (highest semantic fidelity)
2. No ML-based similarity (robustness to obfuscation)
3. Cannot detect functionally equivalent code with radically different structure
### Target Capabilities
```
┌─────────────────────────────────────────────────────────────────────────────┐
│ Advanced Semantic Analysis Stack │
│ │
│ ┌───────────────────────────────────────────────────────────────────────┐ │
│ │ Decompilation Layer │ │
│ │ │ │
│ │ Binary -> Ghidra P-Code -> Decompiled C -> AST -> Semantic Hash │ │
│ │ │ │
│ │ Comparison methods: │ │
│ │ - AST structural similarity │ │
│ │ - Control flow equivalence │ │
│ │ - Data flow equivalence │ │
│ │ - Normalized code text similarity │ │
│ │ │ │
│ └───────────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌───────────────────────────────────────────────────────────────────────┐ │
│ │ ML Embedding Layer │ │
│ │ │ │
│ │ Function Code -> Tokenization -> Transformer -> Embedding Vector │ │
│ │ │ │
│ │ Models: │ │
│ │ - CodeBERT variant for binary code │ │
│ │ - Graph Neural Network for CFG │ │
│ │ - Contrastive learning for similarity │ │
│ │ │ │
│ │ Vector similarity: cosine, euclidean, learned metric │ │
│ │ │ │
│ └───────────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌───────────────────────────────────────────────────────────────────────┐ │
│ │ Ensemble Decision Layer │ │
│ │ │ │
│ │ Combine signals: │ │
│ │ - Instruction fingerprint (Phase 1) : 15% weight │ │
│ │ - Semantic graph (Phase 1) : 25% weight │ │
│ │ - Decompiled AST similarity : 35% weight │ │
│ │ - ML embedding similarity : 25% weight │ │
│ │ │ │
│ │ Output: Confidence-weighted similarity score │ │
│ │ │ │
│ └───────────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────┘
```
---
## Architecture Design
### Decompiler Integration
```csharp
// src/BinaryIndex/__Libraries/StellaOps.BinaryIndex.Decompiler/IDecompilerService.cs
namespace StellaOps.BinaryIndex.Decompiler;
public interface IDecompilerService
{
/// <summary>
/// Decompile a function to C-like pseudo-code.
/// </summary>
Task<DecompiledFunction> DecompileAsync(
GhidraFunction function,
DecompileOptions? options = null,
CancellationToken ct = default);
/// <summary>
/// Parse decompiled code into AST.
/// </summary>
Task<DecompiledAst> ParseToAstAsync(
string decompiledCode,
CancellationToken ct = default);
/// <summary>
/// Compare two decompiled functions for semantic equivalence.
/// </summary>
Task<DecompiledComparisonResult> CompareAsync(
DecompiledFunction a,
DecompiledFunction b,
ComparisonOptions? options = null,
CancellationToken ct = default);
}
public sealed record DecompiledFunction(
string FunctionName,
string Signature,
string Code, // Decompiled C code
DecompiledAst? Ast,
ImmutableArray<LocalVariable> Locals,
ImmutableArray<string> CalledFunctions);
public sealed record DecompiledAst(
AstNode Root,
int NodeCount,
int Depth,
ImmutableArray<AstPattern> Patterns); // Recognized code patterns
public abstract record AstNode(AstNodeType Type, ImmutableArray<AstNode> Children);
public enum AstNodeType
{
Function, Block, If, While, For, DoWhile, Switch,
Return, Break, Continue, Goto,
Assignment, BinaryOp, UnaryOp, Call, Cast,
Variable, Constant, ArrayAccess, FieldAccess, Deref
}
```
### AST Comparison Engine
```csharp
// src/BinaryIndex/__Libraries/StellaOps.BinaryIndex.Decompiler/AstComparisonEngine.cs
namespace StellaOps.BinaryIndex.Decompiler;
public interface IAstComparisonEngine
{
/// <summary>
/// Compute structural similarity between ASTs.
/// </summary>
decimal ComputeStructuralSimilarity(DecompiledAst a, DecompiledAst b);
/// <summary>
/// Compute edit distance between ASTs.
/// </summary>
AstEditDistance ComputeEditDistance(DecompiledAst a, DecompiledAst b);
/// <summary>
/// Find semantic equivalent patterns.
/// </summary>
ImmutableArray<SemanticEquivalence> FindEquivalences(
DecompiledAst a,
DecompiledAst b);
}
public sealed record AstEditDistance(
int Insertions,
int Deletions,
int Modifications,
int TotalOperations,
decimal NormalizedDistance); // 0.0 = identical, 1.0 = completely different
public sealed record SemanticEquivalence(
AstNode NodeA,
AstNode NodeB,
EquivalenceType Type,
decimal Confidence);
public enum EquivalenceType
{
Identical, // Exact match
Renamed, // Same structure, different names
Reordered, // Same operations, different order
Optimized, // Compiler optimization variant
Semantically, // Different structure, same behavior
}
```
### Decompiled Code Normalizer
```csharp
// src/BinaryIndex/__Libraries/StellaOps.BinaryIndex.Decompiler/CodeNormalizer.cs
namespace StellaOps.BinaryIndex.Decompiler;
public interface ICodeNormalizer
{
/// <summary>
/// Normalize decompiled code for comparison.
/// </summary>
string Normalize(string code, NormalizationOptions? options = null);
/// <summary>
/// Generate canonical form hash.
/// </summary>
byte[] ComputeCanonicalHash(string code);
}
internal sealed class CodeNormalizer : ICodeNormalizer
{
public string Normalize(string code, NormalizationOptions? options = null)
{
options ??= NormalizationOptions.Default;
var normalized = code;
// 1. Normalize variable names (var1, var2, ...)
if (options.NormalizeVariables)
{
normalized = NormalizeVariableNames(normalized);
}
// 2. Normalize function calls (func1, func2, ... or keep known names)
if (options.NormalizeFunctionCalls)
{
normalized = NormalizeFunctionCalls(normalized, options.KnownFunctions);
}
// 3. Normalize constants (replace magic numbers with placeholders)
if (options.NormalizeConstants)
{
normalized = NormalizeConstants(normalized);
}
// 4. Normalize whitespace
if (options.NormalizeWhitespace)
{
normalized = NormalizeWhitespace(normalized);
}
// 5. Sort independent statements (where order doesn't matter)
if (options.SortIndependentStatements)
{
normalized = SortIndependentStatements(normalized);
}
return normalized;
}
private static string NormalizeVariableNames(string code)
{
// Replace all local variable names with canonical names
// var_0, var_1, ... in order of first appearance
var varIndex = 0;
var varMap = new Dictionary<string, string>();
// Regex to find variable declarations and uses
return Regex.Replace(code, @"\b([a-zA-Z_][a-zA-Z0-9_]*)\b", match =>
{
var name = match.Value;
// Skip keywords and known types
if (IsKeywordOrType(name))
return name;
if (!varMap.TryGetValue(name, out var canonical))
{
canonical = $"var_{varIndex++}";
varMap[name] = canonical;
}
return canonical;
});
}
}
```
### ML Embedding Service
```csharp
// src/BinaryIndex/__Libraries/StellaOps.BinaryIndex.ML/IEmbeddingService.cs
namespace StellaOps.BinaryIndex.ML;
public interface IEmbeddingService
{
/// <summary>
/// Generate embedding vector for a function.
/// </summary>
Task<FunctionEmbedding> GenerateEmbeddingAsync(
EmbeddingInput input,
EmbeddingOptions? options = null,
CancellationToken ct = default);
/// <summary>
/// Compute similarity between embeddings.
/// </summary>
decimal ComputeSimilarity(
FunctionEmbedding a,
FunctionEmbedding b,
SimilarityMetric metric = SimilarityMetric.Cosine);
/// <summary>
/// Find similar functions in embedding index.
/// </summary>
Task<ImmutableArray<EmbeddingMatch>> FindSimilarAsync(
FunctionEmbedding query,
int topK = 10,
decimal minSimilarity = 0.7m,
CancellationToken ct = default);
}
public sealed record EmbeddingInput(
string? DecompiledCode, // Preferred
KeySemanticsGraph? SemanticGraph, // Fallback
byte[]? InstructionBytes, // Last resort
EmbeddingInputType PreferredInput);
public enum EmbeddingInputType { DecompiledCode, SemanticGraph, Instructions }
public sealed record FunctionEmbedding(
string FunctionName,
float[] Vector, // 768-dimensional
EmbeddingModel Model,
EmbeddingInputType InputType);
public enum EmbeddingModel
{
CodeBertBinary, // Fine-tuned CodeBERT for binary code
GraphSageFunction, // GNN for CFG/call graph
ContrastiveFunction // Contrastive learning model
}
public enum SimilarityMetric { Cosine, Euclidean, Manhattan, LearnedMetric }
```
### Model Training Pipeline
```csharp
// src/BinaryIndex/__Libraries/StellaOps.BinaryIndex.ML/IModelTrainingService.cs
namespace StellaOps.BinaryIndex.ML;
public interface IModelTrainingService
{
/// <summary>
/// Train embedding model on function pairs.
/// </summary>
Task<TrainingResult> TrainAsync(
IAsyncEnumerable<TrainingPair> trainingData,
TrainingOptions options,
IProgress<TrainingProgress>? progress = null,
CancellationToken ct = default);
/// <summary>
/// Evaluate model on test set.
/// </summary>
Task<EvaluationResult> EvaluateAsync(
IAsyncEnumerable<TrainingPair> testData,
CancellationToken ct = default);
/// <summary>
/// Export trained model for inference.
/// </summary>
Task ExportModelAsync(
string outputPath,
ModelExportFormat format = ModelExportFormat.Onnx,
CancellationToken ct = default);
}
public sealed record TrainingPair(
EmbeddingInput FunctionA,
EmbeddingInput FunctionB,
bool IsSimilar, // Ground truth: same function?
decimal? SimilarityScore); // Optional: how similar (0-1)
public sealed record TrainingOptions
{
public EmbeddingModel Model { get; init; } = EmbeddingModel.CodeBertBinary;
public int EmbeddingDimension { get; init; } = 768;
public int BatchSize { get; init; } = 32;
public int Epochs { get; init; } = 10;
public double LearningRate { get; init; } = 1e-5;
public double MarginLoss { get; init; } = 0.5; // Contrastive margin
public string? PretrainedModelPath { get; init; }
}
public sealed record TrainingResult(
string ModelPath,
int TotalPairs,
int Epochs,
double FinalLoss,
double ValidationAccuracy,
TimeSpan TrainingTime);
public sealed record EvaluationResult(
double Accuracy,
double Precision,
double Recall,
double F1Score,
double AucRoc,
ImmutableArray<ConfusionEntry> ConfusionMatrix);
```
### ONNX Inference Engine
```csharp
// src/BinaryIndex/__Libraries/StellaOps.BinaryIndex.ML/OnnxInferenceEngine.cs
namespace StellaOps.BinaryIndex.ML;
internal sealed class OnnxInferenceEngine : IEmbeddingService, IAsyncDisposable
{
private readonly InferenceSession _session;
private readonly ITokenizer _tokenizer;
private readonly ILogger<OnnxInferenceEngine> _logger;
public OnnxInferenceEngine(
string modelPath,
ITokenizer tokenizer,
ILogger<OnnxInferenceEngine> logger)
{
var options = new SessionOptions
{
GraphOptimizationLevel = GraphOptimizationLevel.ORT_ENABLE_ALL,
ExecutionMode = ExecutionMode.ORT_PARALLEL
};
_session = new InferenceSession(modelPath, options);
_tokenizer = tokenizer;
_logger = logger;
}
public async Task<FunctionEmbedding> GenerateEmbeddingAsync(
EmbeddingInput input,
EmbeddingOptions? options = null,
CancellationToken ct = default)
{
var text = input.PreferredInput switch
{
EmbeddingInputType.DecompiledCode => input.DecompiledCode
?? throw new ArgumentException("DecompiledCode required"),
EmbeddingInputType.SemanticGraph => SerializeGraph(input.SemanticGraph
?? throw new ArgumentException("SemanticGraph required")),
EmbeddingInputType.Instructions => SerializeInstructions(input.InstructionBytes
?? throw new ArgumentException("InstructionBytes required")),
_ => throw new ArgumentOutOfRangeException()
};
// Tokenize
var tokens = _tokenizer.Tokenize(text, maxLength: 512);
// Run inference
var inputTensor = new DenseTensor<long>(tokens, [1, tokens.Length]);
var inputs = new List<NamedOnnxValue>
{
NamedOnnxValue.CreateFromTensor("input_ids", inputTensor)
};
using var results = await Task.Run(() => _session.Run(inputs), ct);
var outputTensor = results.First().AsTensor<float>();
var embedding = outputTensor.ToArray();
return new FunctionEmbedding(
input.DecompiledCode?.GetHashCode().ToString() ?? "unknown",
embedding,
EmbeddingModel.CodeBertBinary,
input.PreferredInput);
}
public decimal ComputeSimilarity(
FunctionEmbedding a,
FunctionEmbedding b,
SimilarityMetric metric = SimilarityMetric.Cosine)
{
return metric switch
{
SimilarityMetric.Cosine => CosineSimilarity(a.Vector, b.Vector),
SimilarityMetric.Euclidean => EuclideanSimilarity(a.Vector, b.Vector),
SimilarityMetric.Manhattan => ManhattanSimilarity(a.Vector, b.Vector),
_ => throw new ArgumentOutOfRangeException(nameof(metric))
};
}
private static decimal CosineSimilarity(float[] a, float[] b)
{
var dotProduct = 0.0;
var normA = 0.0;
var normB = 0.0;
for (var i = 0; i < a.Length; i++)
{
dotProduct += a[i] * b[i];
normA += a[i] * a[i];
normB += b[i] * b[i];
}
if (normA == 0 || normB == 0)
return 0;
return (decimal)(dotProduct / (Math.Sqrt(normA) * Math.Sqrt(normB)));
}
}
```
### Ensemble Decision Engine
```csharp
// src/BinaryIndex/__Libraries/StellaOps.BinaryIndex.Ensemble/IEnsembleDecisionEngine.cs
namespace StellaOps.BinaryIndex.Ensemble;
public interface IEnsembleDecisionEngine
{
/// <summary>
/// Compute final similarity using all available signals.
/// </summary>
Task<EnsembleResult> ComputeSimilarityAsync(
FunctionAnalysis a,
FunctionAnalysis b,
EnsembleOptions? options = null,
CancellationToken ct = default);
}
public sealed record FunctionAnalysis(
string FunctionName,
byte[]? InstructionFingerprint, // Phase 1
SemanticFingerprint? SemanticGraph, // Phase 1
DecompiledFunction? Decompiled, // Phase 4
FunctionEmbedding? Embedding); // Phase 4
public sealed record EnsembleOptions
{
// Weight configuration (must sum to 1.0)
public decimal InstructionWeight { get; init; } = 0.15m;
public decimal SemanticGraphWeight { get; init; } = 0.25m;
public decimal DecompiledWeight { get; init; } = 0.35m;
public decimal EmbeddingWeight { get; init; } = 0.25m;
// Confidence thresholds
public decimal MinConfidence { get; init; } = 0.6m;
public bool RequireAllSignals { get; init; } = false;
}
public sealed record EnsembleResult(
decimal OverallSimilarity,
MatchConfidence Confidence,
ImmutableArray<SignalContribution> Contributions,
string? Explanation);
public sealed record SignalContribution(
string SignalName,
decimal RawSimilarity,
decimal Weight,
decimal WeightedContribution,
bool WasAvailable);
```
---
## Delivery Tracker
| # | Task ID | Status | Dependency | Owners | Task Definition |
|---|---------|--------|------------|--------|-----------------|
| **Decompiler Integration** |
| 1 | DCML-001 | TODO | Phase 3 | Guild | Create `StellaOps.BinaryIndex.Decompiler` project |
| 2 | DCML-002 | TODO | DCML-001 | Guild | Define decompiled code model types |
| 3 | DCML-003 | TODO | DCML-002 | Guild | Implement Ghidra decompiler adapter |
| 4 | DCML-004 | TODO | DCML-003 | Guild | Implement C code parser (AST generation) |
| 5 | DCML-005 | TODO | DCML-004 | Guild | Implement AST comparison engine |
| 6 | DCML-006 | TODO | DCML-005 | Guild | Implement code normalizer |
| 7 | DCML-007 | TODO | DCML-006 | Guild | Implement semantic equivalence detector |
| 8 | DCML-008 | TODO | DCML-007 | Guild | Unit tests: Decompiler adapter |
| 9 | DCML-009 | TODO | DCML-007 | Guild | Unit tests: AST comparison |
| 10 | DCML-010 | TODO | DCML-009 | Guild | Integration tests: End-to-end decompiled comparison |
| **ML Embedding Pipeline** |
| 11 | DCML-011 | TODO | Phase 2 | Guild | Create `StellaOps.BinaryIndex.ML` project |
| 12 | DCML-012 | TODO | DCML-011 | Guild | Define embedding model types |
| 13 | DCML-013 | TODO | DCML-012 | Guild | Implement code tokenizer (binary-aware BPE) |
| 14 | DCML-014 | TODO | DCML-013 | Guild | Set up ONNX Runtime inference engine |
| 15 | DCML-015 | TODO | DCML-014 | Guild | Implement embedding service |
| 16 | DCML-016 | TODO | DCML-015 | Guild | Create training data from corpus (positive/negative pairs) |
| 17 | DCML-017 | TODO | DCML-016 | Guild | Train CodeBERT-Binary model |
| 18 | DCML-018 | TODO | DCML-017 | Guild | Export model to ONNX format |
| 19 | DCML-019 | TODO | DCML-015 | Guild | Unit tests: Embedding generation |
| 20 | DCML-020 | TODO | DCML-018 | Guild | Evaluation: Model accuracy metrics |
| **Ensemble Integration** |
| 21 | DCML-021 | TODO | DCML-010,020 | Guild | Create `StellaOps.BinaryIndex.Ensemble` project |
| 22 | DCML-022 | TODO | DCML-021 | Guild | Implement ensemble decision engine |
| 23 | DCML-023 | TODO | DCML-022 | Guild | Implement weight tuning (grid search) |
| 24 | DCML-024 | TODO | DCML-023 | Guild | Integrate ensemble into PatchDiffEngine |
| 25 | DCML-025 | TODO | DCML-024 | Guild | Integrate ensemble into DeltaSignatureMatcher |
| 26 | DCML-026 | TODO | DCML-025 | Guild | Unit tests: Ensemble decision logic |
| 27 | DCML-027 | TODO | DCML-026 | Guild | Integration tests: Full semantic diffing pipeline |
| 28 | DCML-028 | TODO | DCML-027 | Guild | Benchmark: Accuracy vs. baseline (Phase 1 only) |
| 29 | DCML-029 | TODO | DCML-028 | Guild | Benchmark: Latency impact |
| 30 | DCML-030 | TODO | DCML-029 | Guild | Documentation: ML model training guide |
---
## Task Details
### DCML-004: Implement C Code Parser
Parse Ghidra's decompiled C output into AST:
```csharp
internal sealed class DecompiledCodeParser
{
public DecompiledAst Parse(string code)
{
// Use Tree-sitter or Roslyn-based C parser
// Ghidra output is C-like but not standard C
var tokens = Tokenize(code);
var ast = BuildAst(tokens);
return new DecompiledAst(
ast,
CountNodes(ast),
ComputeDepth(ast),
ExtractPatterns(ast));
}
private AstNode BuildAst(IList<Token> tokens)
{
var parser = new RecursiveDescentParser(tokens);
return parser.ParseFunction();
}
private ImmutableArray<AstPattern> ExtractPatterns(AstNode root)
{
var patterns = new List<AstPattern>();
// Detect common patterns
patterns.AddRange(DetectLoopPatterns(root));
patterns.AddRange(DetectBranchPatterns(root));
patterns.AddRange(DetectAllocationPatterns(root));
patterns.AddRange(DetectErrorHandlingPatterns(root));
return [.. patterns];
}
private static IEnumerable<AstPattern> DetectLoopPatterns(AstNode root)
{
// Find: for loops, while loops, do-while
// Classify: counted loop, sentinel loop, infinite loop
foreach (var node in TraverseNodes(root))
{
if (node.Type == AstNodeType.For)
{
yield return new AstPattern(
PatternType.CountedLoop,
node,
AnalyzeForLoop(node));
}
else if (node.Type == AstNodeType.While)
{
yield return new AstPattern(
PatternType.ConditionalLoop,
node,
AnalyzeWhileLoop(node));
}
}
}
}
```
### DCML-017: Train CodeBERT-Binary Model
Training pipeline for function similarity:
```python
# tools/ml/train_codebert_binary.py
import torch
from transformers import RobertaTokenizer, RobertaModel
from torch.utils.data import DataLoader
import onnx
class CodeBertBinaryModel(torch.nn.Module):
def __init__(self, pretrained_model="microsoft/codebert-base"):
super().__init__()
self.encoder = RobertaModel.from_pretrained(pretrained_model)
self.projection = torch.nn.Linear(768, 768)
def forward(self, input_ids, attention_mask):
outputs = self.encoder(input_ids, attention_mask=attention_mask)
pooled = outputs.last_hidden_state[:, 0, :] # [CLS] token
projected = self.projection(pooled)
return torch.nn.functional.normalize(projected, p=2, dim=1)
class ContrastiveLoss(torch.nn.Module):
def __init__(self, margin=0.5):
super().__init__()
self.margin = margin
def forward(self, embedding_a, embedding_b, label):
distance = torch.nn.functional.pairwise_distance(embedding_a, embedding_b)
# label=1: similar, label=0: dissimilar
loss = label * distance.pow(2) + \
(1 - label) * torch.clamp(self.margin - distance, min=0).pow(2)
return loss.mean()
def train_model(train_dataloader, val_dataloader, epochs=10):
model = CodeBertBinaryModel()
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-5)
criterion = ContrastiveLoss(margin=0.5)
for epoch in range(epochs):
model.train()
total_loss = 0
for batch in train_dataloader:
optimizer.zero_grad()
emb_a = model(batch['input_ids_a'], batch['attention_mask_a'])
emb_b = model(batch['input_ids_b'], batch['attention_mask_b'])
loss = criterion(emb_a, emb_b, batch['label'])
loss.backward()
optimizer.step()
total_loss += loss.item()
# Validation
model.eval()
val_accuracy = evaluate(model, val_dataloader)
print(f"Epoch {epoch+1}: Loss={total_loss:.4f}, Val Acc={val_accuracy:.4f}")
return model
def export_to_onnx(model, output_path):
model.eval()
dummy_input = torch.randint(0, 50000, (1, 512))
dummy_mask = torch.ones(1, 512)
torch.onnx.export(
model,
(dummy_input, dummy_mask),
output_path,
input_names=['input_ids', 'attention_mask'],
output_names=['embedding'],
dynamic_axes={
'input_ids': {0: 'batch', 1: 'seq'},
'attention_mask': {0: 'batch', 1: 'seq'},
'embedding': {0: 'batch'}
}
)
```
### DCML-023: Implement Weight Tuning
Grid search for optimal ensemble weights:
```csharp
internal sealed class EnsembleWeightTuner
{
public async Task<EnsembleOptions> TuneWeightsAsync(
IAsyncEnumerable<LabeledPair> validationData,
CancellationToken ct)
{
var bestOptions = EnsembleOptions.Default;
var bestF1 = 0.0;
// Grid search over weight combinations
var weightCombinations = GenerateWeightCombinations(step: 0.05m);
foreach (var weights in weightCombinations)
{
ct.ThrowIfCancellationRequested();
var options = new EnsembleOptions
{
InstructionWeight = weights[0],
SemanticGraphWeight = weights[1],
DecompiledWeight = weights[2],
EmbeddingWeight = weights[3]
};
var metrics = await EvaluateAsync(validationData, options, ct);
if (metrics.F1Score > bestF1)
{
bestF1 = metrics.F1Score;
bestOptions = options;
}
}
return bestOptions;
}
private static IEnumerable<decimal[]> GenerateWeightCombinations(decimal step)
{
for (var w1 = 0m; w1 <= 1m; w1 += step)
for (var w2 = 0m; w2 <= 1m - w1; w2 += step)
for (var w3 = 0m; w3 <= 1m - w1 - w2; w3 += step)
{
var w4 = 1m - w1 - w2 - w3;
if (w4 >= 0)
{
yield return [w1, w2, w3, w4];
}
}
}
}
```
---
## Training Data Requirements
### Positive Pairs (Similar Functions)
| Source | Count | Description |
|--------|-------|-------------|
| Same function, different optimization | ~50,000 | O0 vs O2 vs O3 |
| Same function, different compiler | ~30,000 | GCC vs Clang |
| Same function, different version | ~100,000 | From corpus (Phase 2) |
| Same function, with patches | ~20,000 | Vulnerable vs fixed |
### Negative Pairs (Dissimilar Functions)
| Source | Count | Description |
|--------|-------|-------------|
| Random function pairs | ~100,000 | Random sampling |
| Similar-named different functions | ~50,000 | Hard negatives |
| Same library, different functions | ~50,000 | Medium negatives |
**Total training data:** ~400,000 labeled pairs
---
## Success Metrics
| Metric | Phase 1 Only | With Phase 4 | Target |
|--------|--------------|--------------|--------|
| Accuracy (optimized binaries) | 70% | 92% | 90%+ |
| Accuracy (obfuscated binaries) | 40% | 75% | 70%+ |
| False positive rate | 5% | 1.5% | <2% |
| False negative rate | 25% | 8% | <10% |
| Latency (per comparison) | 10ms | 150ms | <200ms |
---
## Resource Requirements
| Resource | Training | Inference |
|----------|----------|-----------|
| GPU | 1x V100 (32GB) or 4x T4 | Optional (CPU viable) |
| Memory | 64GB | 16GB |
| Storage | 100GB (training data) | 5GB (model) |
| Time | ~24 hours | <200ms per function |
---
## Execution Log
| Date (UTC) | Update | Owner |
|------------|--------|-------|
| 2026-01-05 | Sprint created from product advisory analysis | Planning |
---
## Decisions & Risks
| Decision/Risk | Type | Mitigation |
|---------------|------|------------|
| ML model requires significant training data | Risk | Leverage corpus from Phase 2 |
| ONNX inference adds latency | Trade-off | Make ML optional, use for high-value comparisons |
| Decompiler output varies by Ghidra version | Risk | Pin Ghidra version, normalize output |
| Model may overfit to training library set | Risk | Diverse training data, regularization |
| GPU dependency for training | Constraint | Use cloud GPU, document CPU-only option |
---
## Next Checkpoints
- 2026-03-01: DCML-001 through DCML-010 (decompiler integration) complete
- 2026-03-15: DCML-011 through DCML-020 (ML pipeline) complete
- 2026-03-31: DCML-021 through DCML-030 (ensemble, benchmarks) complete

View File

@@ -0,0 +1,203 @@
# Sprint Series 20260105_002 - HLC: Audit-Safe Job Queue Ordering
## Executive Summary
This sprint series implements the "Audit-safe job queue ordering" product advisory, adding Hybrid Logical Clock (HLC) based ordering with cryptographic sequence proofs to the StellaOps Scheduler. This closes the ~30% compliance gap identified in the advisory analysis.
## Problem Statement
Current StellaOps architecture relies on:
- Wall-clock timestamps (`TimeProvider.GetUtcNow()`) for job ordering
- Per-module sequence numbers (local ordering, not global)
- Hash chains only in downstream ledgers (Findings, Orchestrator Audit)
This creates risks in:
- **Distributed deployments** with clock skew between nodes
- **Offline/air-gap scenarios** where jobs enqueued offline must merge deterministically
- **Audit forensics** where "prove job A preceded job B" requires global ordering
## Solution Architecture
```
┌─────────────────────────────────────────────────┐
│ HLC Core Library │
│ (PhysicalTime, NodeId, LogicalCounter) │
└──────────────────────┬──────────────────────────┘
┌───────────────────────────────────┼───────────────────────────────────┐
│ │ │
▼ ▼ ▼
┌───────────────┐ ┌───────────────────┐ ┌───────────────┐
│ Scheduler │ │ Offline Merge │ │ Integration │
│ Queue Chain │ │ Protocol │ │ Tests │
│ │ │ │ │ │
│ - HLC at │ │ - Local HLC │ │ - E2E tests │
│ enqueue │ │ persistence │ │ - Benchmarks │
│ - Chain link │ │ - Bundle export │ │ - Alerts │
│ computation │ │ - Deterministic │ │ - Docs │
│ - Batch │ │ merge │ │ │
│ snapshots │ │ - Conflict │ │ │
│ │ │ resolution │ │ │
└───────────────┘ └───────────────────┘ └───────────────┘
```
## Sprint Breakdown
| Sprint | Module | Scope | Est. Effort |
|--------|--------|-------|-------------|
| [002_001](SPRINT_20260105_002_001_LB_hlc_core_library.md) | Library | HLC core implementation | 3 days |
| [002_002](SPRINT_20260105_002_002_SCHEDULER_hlc_queue_chain.md) | Scheduler | Queue chain integration | 4 days |
| [002_003](SPRINT_20260105_002_003_ROUTER_hlc_offline_merge.md) | Router/AirGap | Offline merge protocol | 4 days |
| [002_004](SPRINT_20260105_002_004_BE_hlc_integration_tests.md) | Testing | Integration & E2E tests | 3 days |
**Total Estimated Effort:** ~14 days (2-3 weeks with buffer)
## Dependency Graph
```
SPRINT_20260104_001_BE (TimeProvider injection)
SPRINT_20260105_002_001_LB (HLC core library)
SPRINT_20260105_002_002_SCHEDULER (Queue chain)
SPRINT_20260105_002_003_ROUTER (Offline merge)
SPRINT_20260105_002_004_BE (Integration tests)
Production Rollout
```
## Task Summary
### Sprint 002_001: HLC Core Library (12 tasks)
- HLC timestamp struct with comparison
- Tick/Receive algorithm implementation
- State persistence (PostgreSQL, in-memory)
- JSON/Npgsql serialization
- Unit tests and benchmarks
### Sprint 002_002: Scheduler Queue Chain (22 tasks)
- Database schema: `scheduler_log`, `batch_snapshot`, `chain_heads`
- Chain link computation
- HLC-based enqueue/dequeue services
- Redis/NATS adapter updates
- Batch snapshot with DSSE signing
- Chain verification
- Feature flags for gradual rollout
### Sprint 002_003: Offline Merge Protocol (21 tasks)
- Offline HLC manager
- File-based job log store
- Merge algorithm with total ordering
- Conflict resolution
- Air-gap bundle format
- CLI command updates (`stella airgap export/import`)
- Integration with Router transport
### Sprint 002_004: Integration Tests (22 tasks)
- HLC propagation tests
- Chain integrity tests
- Batch snapshot + Attestor integration
- Offline sync tests
- Replay determinism tests
- Performance benchmarks
- Grafana dashboard and alerts
- Documentation updates
## Key Design Decisions
| Decision | Rationale |
|----------|-----------|
| HLC over Lamport | Physical time component improves debuggability |
| Separate `scheduler_log` table | Avoid breaking changes to existing `jobs` table |
| Chain link at enqueue | Ensures ordering proof exists before execution |
| Feature flags | Gradual rollout; easy rollback |
| DSSE signing optional | Not all deployments need attestation |
## Risk Register
| Risk | Probability | Impact | Mitigation |
|------|-------------|--------|------------|
| Performance regression | Medium | Medium | Benchmarks; feature flag for rollback |
| Clock skew exceeds tolerance | Low | High | NTP hardening; pre-sync validation |
| Migration complexity | Medium | Medium | Dual-write mode; gradual rollout |
| Chain corruption | Low | Critical | Verification alerts; immutable logs |
## Success Criteria
1. **Determinism:** Same inputs produce same HLC order across restarts/nodes
2. **Chain Integrity:** 100% tampering detection in verification tests
3. **Offline Merge:** Jobs from multiple offline nodes merge in correct HLC order
4. **Performance:** HLC tick > 100K/sec; chain verification < 100ms/1K entries
5. **Replay:** HLC-ordered replay produces identical results
## Rollout Plan
### Phase 1: Shadow Mode (Week 1)
- Deploy with `EnableHlcOrdering = false`, `DualWriteMode = true`
- HLC timestamps recorded but not used for ordering
- Verify chain integrity on shadow writes
### Phase 2: Canary (Week 2)
- Enable `EnableHlcOrdering = true` for 5% of tenants
- Monitor metrics: latency, errors, chain verifications
- Compare results between HLC and legacy ordering
### Phase 3: General Availability (Week 3)
- Gradual rollout to all tenants
- Disable `DualWriteMode` after 1 week of stable GA
- Deprecate legacy ordering path
### Phase 4: Offline Features (Week 4+)
- Enable air-gap bundle export/import with HLC
- Test multi-node merge scenarios
- Document operational procedures
## Metrics to Monitor
```
# HLC Health
hlc_ticks_total
hlc_clock_skew_rejections_total
hlc_physical_time_offset_seconds
# Scheduler Chain
scheduler_hlc_enqueues_total
scheduler_chain_verifications_total
scheduler_chain_verification_failures_total
scheduler_batch_snapshots_total
# Offline Sync
airgap_bundles_exported_total
airgap_bundles_imported_total
airgap_jobs_synced_total
airgap_merge_conflicts_total
airgap_sync_duration_seconds
```
## Documentation Deliverables
- [ ] `docs/ARCHITECTURE_REFERENCE.md` - HLC section
- [ ] `docs/modules/scheduler/architecture.md` - HLC ordering
- [ ] `docs/airgap/OFFLINE_KIT.md` - HLC merge protocol
- [ ] `docs/observability/observability.md` - HLC metrics
- [ ] `docs/operations/runbooks/hlc-troubleshooting.md`
- [ ] `CLAUDE.md` Section 8.19 - HLC guidelines
## Contact & Ownership
- **Sprint Owner:** Guild
- **Technical Lead:** TBD
- **Review:** Architecture Board
## References
- Product Advisory: "Audit-safe job queue ordering using monotonic timestamps"
- Gap Analysis: StellaOps implementation vs. advisory (2026-01-05)
- HLC Paper: "Logical Physical Clocks and Consistent Snapshots" (Kulkarni et al.)

View File

@@ -0,0 +1,343 @@
# Sprint 20260105_002_001_LB - HLC: Hybrid Logical Clock Core Library
## Topic & Scope
Implement a Hybrid Logical Clock (HLC) library for deterministic, monotonic job ordering across distributed nodes. This addresses the gap identified in the "Audit-safe job queue ordering" product advisory where StellaOps currently uses wall-clock timestamps susceptible to clock skew.
- **Working directory:** `src/__Libraries/StellaOps.HybridLogicalClock/`
- **Evidence:** NuGet package, unit tests, integration tests, benchmark results
## Problem Statement
Current StellaOps architecture uses:
- `TimeProvider.GetUtcNow()` for wall-clock time (deterministic but not skew-resistant)
- Per-module sequence numbers (local ordering, not global)
- Hash chains only in downstream ledgers (Findings, Orchestrator Audit)
The advisory prescribes:
- HLC `(T, NodeId, Ctr)` tuples for global logical time
- Total ordering via `(T_hlc, PartitionKey?, JobId)` sort key
- Hash chain at enqueue time, not just downstream
## Dependencies & Concurrency
- **Depends on:** SPRINT_20260104_001_BE (TimeProvider injection complete)
- **Blocks:** SPRINT_20260105_002_002_SCHEDULER (HLC queue chain)
- **Parallel safe:** Library development independent of other modules
## Documentation Prerequisites
- docs/README.md
- docs/ARCHITECTURE_REFERENCE.md
- CLAUDE.md Section 8.2 (Deterministic Time & ID Generation)
- Product Advisory: "Audit-safe job queue ordering using monotonic timestamps"
## Technical Design
### HLC Algorithm (Lamport + Physical Clock Hybrid)
```
On local event or send:
l' = l
l = max(l, physical_clock())
if l == l':
c = c + 1
else:
c = 0
return (l, node_id, c)
On receive(m_l, m_c):
l' = l
l = max(l', m_l, physical_clock())
if l == l' == m_l:
c = max(c, m_c) + 1
elif l == l':
c = c + 1
elif l == m_l:
c = m_c + 1
else:
c = 0
return (l, node_id, c)
```
### Data Model
```csharp
/// <summary>
/// Hybrid Logical Clock timestamp providing monotonic, causally-ordered time
/// across distributed nodes even under clock skew.
/// </summary>
public readonly record struct HlcTimestamp : IComparable<HlcTimestamp>
{
/// <summary>Physical time component (Unix milliseconds UTC).</summary>
public required long PhysicalTime { get; init; }
/// <summary>Unique node identifier (e.g., "scheduler-east-1").</summary>
public required string NodeId { get; init; }
/// <summary>Logical counter for events at same physical time.</summary>
public required int LogicalCounter { get; init; }
/// <summary>String representation for storage: "1704067200000-scheduler-east-1-42"</summary>
public string ToSortableString() => $"{PhysicalTime:D13}-{NodeId}-{LogicalCounter:D6}";
/// <summary>Parse from sortable string format.</summary>
public static HlcTimestamp Parse(string value);
/// <summary>Compare for total ordering.</summary>
public int CompareTo(HlcTimestamp other);
}
```
### Interfaces
```csharp
/// <summary>
/// Hybrid Logical Clock for monotonic timestamp generation.
/// </summary>
public interface IHybridLogicalClock
{
/// <summary>Generate next timestamp for local event.</summary>
HlcTimestamp Tick();
/// <summary>Update clock on receiving remote timestamp, return merged result.</summary>
HlcTimestamp Receive(HlcTimestamp remote);
/// <summary>Current clock state (for persistence/recovery).</summary>
HlcTimestamp Current { get; }
/// <summary>Node identifier for this clock instance.</summary>
string NodeId { get; }
}
/// <summary>
/// Persistent storage for HLC state (survives restarts).
/// </summary>
public interface IHlcStateStore
{
/// <summary>Load last persisted HLC state for node.</summary>
Task<HlcTimestamp?> LoadAsync(string nodeId, CancellationToken ct = default);
/// <summary>Persist HLC state (called after each tick).</summary>
Task SaveAsync(HlcTimestamp timestamp, CancellationToken ct = default);
}
```
### PostgreSQL Schema
```sql
-- HLC state persistence (one row per node)
CREATE TABLE scheduler.hlc_state (
node_id TEXT PRIMARY KEY,
physical_time BIGINT NOT NULL,
logical_counter INT NOT NULL,
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
-- Index for recovery queries
CREATE INDEX idx_hlc_state_updated ON scheduler.hlc_state(updated_at DESC);
```
## Delivery Tracker
| # | Task ID | Status | Dependency | Owner | Task Definition |
|---|---------|--------|------------|-------|-----------------|
| 1 | HLC-001 | TODO | - | Guild | Create `StellaOps.HybridLogicalClock` project with Directory.Build.props integration |
| 2 | HLC-002 | TODO | HLC-001 | Guild | Implement `HlcTimestamp` record with comparison, parsing, serialization |
| 3 | HLC-003 | TODO | HLC-002 | Guild | Implement `HybridLogicalClock` class with Tick/Receive/Current |
| 4 | HLC-004 | TODO | HLC-003 | Guild | Implement `IHlcStateStore` interface and `InMemoryHlcStateStore` |
| 5 | HLC-005 | TODO | HLC-004 | Guild | Implement `PostgresHlcStateStore` with atomic update semantics |
| 6 | HLC-006 | TODO | HLC-003 | Guild | Add `HlcTimestampJsonConverter` for System.Text.Json serialization |
| 7 | HLC-007 | TODO | HLC-003 | Guild | Add `HlcTimestampTypeHandler` for Npgsql/Dapper |
| 8 | HLC-008 | TODO | HLC-005 | Guild | Write unit tests: tick monotonicity, receive merge, clock skew handling |
| 9 | HLC-009 | TODO | HLC-008 | Guild | Write integration tests: concurrent ticks, node restart recovery |
| 10 | HLC-010 | TODO | HLC-009 | Guild | Write benchmarks: tick throughput, memory allocation |
| 11 | HLC-011 | TODO | HLC-010 | Guild | Create `HlcServiceCollectionExtensions` for DI registration |
| 12 | HLC-012 | TODO | HLC-011 | Guild | Documentation: README.md, API docs, usage examples |
## Implementation Details
### Clock Skew Tolerance
```csharp
public class HybridLogicalClock : IHybridLogicalClock
{
private readonly TimeProvider _timeProvider;
private readonly string _nodeId;
private readonly IHlcStateStore _stateStore;
private readonly TimeSpan _maxClockSkew;
private long _lastPhysicalTime;
private int _logicalCounter;
private readonly object _lock = new();
public HybridLogicalClock(
TimeProvider timeProvider,
string nodeId,
IHlcStateStore stateStore,
TimeSpan? maxClockSkew = null)
{
_timeProvider = timeProvider;
_nodeId = nodeId;
_stateStore = stateStore;
_maxClockSkew = maxClockSkew ?? TimeSpan.FromMinutes(1);
}
public HlcTimestamp Tick()
{
lock (_lock)
{
var physicalNow = _timeProvider.GetUtcNow().ToUnixTimeMilliseconds();
if (physicalNow > _lastPhysicalTime)
{
_lastPhysicalTime = physicalNow;
_logicalCounter = 0;
}
else
{
_logicalCounter++;
}
var timestamp = new HlcTimestamp
{
PhysicalTime = _lastPhysicalTime,
NodeId = _nodeId,
LogicalCounter = _logicalCounter
};
// Persist state asynchronously (fire-and-forget with error logging)
_ = _stateStore.SaveAsync(timestamp);
return timestamp;
}
}
public HlcTimestamp Receive(HlcTimestamp remote)
{
lock (_lock)
{
var physicalNow = _timeProvider.GetUtcNow().ToUnixTimeMilliseconds();
// Validate clock skew
var skew = TimeSpan.FromMilliseconds(Math.Abs(remote.PhysicalTime - physicalNow));
if (skew > _maxClockSkew)
{
throw new HlcClockSkewException(skew, _maxClockSkew);
}
var maxPhysical = Math.Max(Math.Max(_lastPhysicalTime, remote.PhysicalTime), physicalNow);
if (maxPhysical == _lastPhysicalTime && maxPhysical == remote.PhysicalTime)
{
_logicalCounter = Math.Max(_logicalCounter, remote.LogicalCounter) + 1;
}
else if (maxPhysical == _lastPhysicalTime)
{
_logicalCounter++;
}
else if (maxPhysical == remote.PhysicalTime)
{
_logicalCounter = remote.LogicalCounter + 1;
}
else
{
_logicalCounter = 0;
}
_lastPhysicalTime = maxPhysical;
return new HlcTimestamp
{
PhysicalTime = _lastPhysicalTime,
NodeId = _nodeId,
LogicalCounter = _logicalCounter
};
}
}
}
```
### Comparison for Total Ordering
```csharp
public int CompareTo(HlcTimestamp other)
{
// Primary: physical time
var physicalCompare = PhysicalTime.CompareTo(other.PhysicalTime);
if (physicalCompare != 0) return physicalCompare;
// Secondary: logical counter
var counterCompare = LogicalCounter.CompareTo(other.LogicalCounter);
if (counterCompare != 0) return counterCompare;
// Tertiary: node ID (for stable tie-breaking)
return string.Compare(NodeId, other.NodeId, StringComparison.Ordinal);
}
```
## Test Cases
### Unit Tests
| Test | Description |
|------|-------------|
| `Tick_Monotonic` | Successive ticks always increase |
| `Tick_SamePhysicalTime_IncrementCounter` | Counter increments when physical time unchanged |
| `Tick_NewPhysicalTime_ResetCounter` | Counter resets when physical time advances |
| `Receive_MergesCorrectly` | Remote timestamp merged per HLC algorithm |
| `Receive_ClockSkewExceeded_Throws` | Excessive skew detected and rejected |
| `Parse_RoundTrip` | ToSortableString/Parse symmetry |
| `CompareTo_TotalOrdering` | All orderings follow spec |
### Integration Tests
| Test | Description |
|------|-------------|
| `ConcurrentTicks_AllUnique` | 1000 concurrent ticks produce unique timestamps |
| `NodeRestart_ResumesFromPersisted` | After restart, clock >= persisted state |
| `MultiNode_CausalOrdering` | Messages across nodes maintain causal order |
| `PostgresStateStore_AtomicUpdate` | Concurrent saves don't lose state |
## Metrics & Observability
```csharp
// Counters
hlc_ticks_total{node_id} // Total ticks generated
hlc_receives_total{node_id} // Total remote timestamps received
hlc_clock_skew_rejections_total{node_id} // Skew threshold exceeded
// Histograms
hlc_tick_duration_seconds{node_id} // Tick operation latency
hlc_logical_counter_value{node_id} // Counter distribution
// Gauges
hlc_physical_time_offset_seconds{node_id} // Drift from wall clock
```
## Decisions & Risks
| Decision | Rationale |
|----------|-----------|
| Store physical time as Unix milliseconds | Sufficient precision, compact storage |
| Use string node ID (not UUID) | Human-readable, stable across restarts |
| Fire-and-forget state persistence | Performance; recovery handles gaps |
| 1-minute default max skew | Balance between strictness and operability |
| Risk | Mitigation |
|------|------------|
| Clock skew exceeds threshold | Alert on `hlc_clock_skew_rejections_total`; NTP hardening |
| State store unavailable | In-memory continues; warns on recovery |
| Counter overflow (INT) | At 1M ticks/sec, 35 minutes to overflow; use long if needed |
## Execution Log
| Date (UTC) | Update | Owner |
|------------|--------|-------|
| 2026-01-05 | Sprint created from product advisory gap analysis | Planning |
## Next Checkpoints
- 2026-01-06: HLC-001 to HLC-003 complete (core implementation)
- 2026-01-07: HLC-004 to HLC-007 complete (persistence + serialization)
- 2026-01-08: HLC-008 to HLC-012 complete (tests, docs, DI)

View File

@@ -0,0 +1,528 @@
# Sprint 20260105_002_001_REPLAY - Complete Replay Infrastructure
## Topic & Scope
Complete the existing replay infrastructure to achieve full "Verifiable Policy Replay" as described in the product advisory. This sprint focuses on wiring existing stubs, completing DSSE verification, and adding the compact replay proof format.
**Advisory Reference:** Product advisory on deterministic replay - "Verifiable Policy Replay (deterministic time-travel)" section.
**Key Insight:** StellaOps has ~75% of the replay infrastructure built. This sprint closes the remaining gaps by integrating existing components (VerdictBuilder, Signer) into the CLI and API, and standardizing the replay proof output format.
**Working directory:** `src/Cli/`, `src/Replay/`, `src/__Libraries/StellaOps.Replay.Core/`
**Evidence:** Functional `stella verify --bundle` with full replay, `stella prove --at` command, DSSE signature verification, compact `replay-proof:<hash>` format.
---
## Dependencies & Concurrency
| Dependency | Type | Status |
|------------|------|--------|
| KnowledgeSnapshot model | Internal | Available |
| ReplayBundleWriter | Internal | Available |
| ReplayEngine | Internal | Available |
| VerdictBuilder | Internal | Stub exists, needs integration |
| ISigner/DSSE | Internal | Available in Attestor module |
| DsseHelper | Internal | Available |
**Parallel Execution:** Tasks RPL-001 through RPL-005 (VerdictBuilder wiring) must complete before RPL-006 (DSSE). RPL-007 through RPL-010 (CLI) can proceed in parallel once dependencies land.
---
## Documentation Prerequisites
- `docs/modules/replay/architecture.md` (if exists)
- `docs/modules/attestor/architecture.md`
- CLAUDE.md sections on determinism (8.1-8.18)
- Existing: `src/__Libraries/StellaOps.Replay.Core/Models/KnowledgeSnapshot.cs`
- Existing: `src/Policy/__Libraries/StellaOps.Policy/Replay/ReplayEngine.cs`
---
## Problem Analysis
### Current State
After prior implementation work:
- `KnowledgeSnapshot` model captures all inputs (SBOMs, VEX, feeds, policy, seeds)
- `ReplayBundleWriter` produces deterministic `.tar.zst` bundles
- `ReplayEngine` replays with frozen inputs and compares verdicts
- `VerdictReplayEndpoints` API exists with eligibility checking
- `stella verify --bundle` CLI exists but `ReplayVerdictAsync()` returns null (stub)
- DSSE signature verification marked "not implemented"
**Remaining Gaps:**
1. `stella verify --bundle` doesn't actually replay verdicts
2. No DSSE signature verification on bundles
3. No compact `replay-proof:<hash>` output format
4. No `stella prove --image <sha256> --at <timestamp>` command
### Target Capabilities
```
Replay Infrastructure Complete
+------------------------------------------------------------------+
| |
| stella verify --bundle B.dsig |
| ├── Load manifest.json |
| ├── Validate input hashes (SBOM, feeds, VEX, policy) |
| ├── Execute VerdictBuilder.ReplayAsync(manifest) <-- NEW |
| ├── Compare replayed verdict hash to expected |
| ├── Verify DSSE signature <-- NEW |
| └── Output: replay-proof:<hash> <-- NEW |
| |
| stella prove --image sha256:abc... --at 2025-12-15T10:00Z |
| ├── Query TimelineIndexer for snapshot at timestamp <-- NEW |
| ├── Fetch bundle from CAS |
| ├── Execute replay (same as verify) |
| └── Output: replay-proof:<hash> |
| |
+------------------------------------------------------------------+
```
---
## Architecture Design
### ReplayProof Schema
```csharp
// src/__Libraries/StellaOps.Replay.Core/Models/ReplayProof.cs
namespace StellaOps.Replay.Core.Models;
/// <summary>
/// Compact proof artifact for audit trails and ticket attachments.
/// </summary>
public sealed record ReplayProof
{
/// <summary>
/// SHA-256 of the replay bundle used.
/// </summary>
public required string BundleHash { get; init; }
/// <summary>
/// Policy version at replay time.
/// </summary>
public required string PolicyVersion { get; init; }
/// <summary>
/// Merkle root of verdict findings.
/// </summary>
public required string VerdictRoot { get; init; }
/// <summary>
/// Replay execution duration in milliseconds.
/// </summary>
public required long DurationMs { get; init; }
/// <summary>
/// Whether replayed verdict matches original.
/// </summary>
public required bool VerdictMatches { get; init; }
/// <summary>
/// UTC timestamp of replay execution.
/// </summary>
public required DateTimeOffset ReplayedAt { get; init; }
/// <summary>
/// Engine version that performed the replay.
/// </summary>
public required string EngineVersion { get; init; }
/// <summary>
/// Generate compact proof string for ticket/PR attachment.
/// Format: replay-proof:&lt;base64url(sha256(canonical_json))&gt;
/// </summary>
public string ToCompactString()
{
var canonical = CanonicalJsonSerializer.Serialize(this);
var hash = SHA256.HashData(Encoding.UTF8.GetBytes(canonical));
var b64 = Convert.ToBase64String(hash).Replace("+", "-").Replace("/", "_").TrimEnd('=');
return $"replay-proof:{b64}";
}
}
```
### VerdictBuilder Integration
```csharp
// src/Cli/StellaOps.Cli/Commands/CommandHandlers.VerifyBundle.cs
// Enhancement to existing ReplayVerdictAsync method
private static async Task<string?> ReplayVerdictAsync(
IServiceProvider services,
string bundleDir,
ReplayBundleManifest manifest,
List<BundleViolation> violations,
ILogger logger,
CancellationToken cancellationToken)
{
var verdictBuilder = services.GetService<IVerdictBuilder>();
if (verdictBuilder is null)
{
logger.LogWarning("VerdictBuilder not registered - replay skipped");
violations.Add(new BundleViolation(
"verdict.replay.service_unavailable",
"VerdictBuilder service not available in DI container"));
return null;
}
try
{
// Load frozen inputs from bundle
var sbomPath = Path.Combine(bundleDir, manifest.Inputs.Sbom.Path);
var feedsPath = manifest.Inputs.Feeds is not null
? Path.Combine(bundleDir, manifest.Inputs.Feeds.Path) : null;
var vexPath = manifest.Inputs.Vex is not null
? Path.Combine(bundleDir, manifest.Inputs.Vex.Path) : null;
var policyPath = manifest.Inputs.Policy is not null
? Path.Combine(bundleDir, manifest.Inputs.Policy.Path) : null;
var replayRequest = new VerdictReplayRequest
{
SbomPath = sbomPath,
FeedsPath = feedsPath,
VexPath = vexPath,
PolicyPath = policyPath,
ImageDigest = manifest.Scan.ImageDigest,
PolicyDigest = manifest.Scan.PolicyDigest,
FeedSnapshotDigest = manifest.Scan.FeedSnapshotDigest
};
var result = await verdictBuilder.ReplayAsync(replayRequest, cancellationToken)
.ConfigureAwait(false);
if (!result.Success)
{
violations.Add(new BundleViolation(
"verdict.replay.failed",
result.Error ?? "Verdict replay failed without error message"));
return null;
}
logger.LogInformation("Verdict replay completed: Hash={Hash}", result.VerdictHash);
return result.VerdictHash;
}
catch (Exception ex)
{
logger.LogError(ex, "Verdict replay threw exception");
violations.Add(new BundleViolation(
"verdict.replay.exception",
$"Replay exception: {ex.Message}"));
return null;
}
}
```
### DSSE Verification Integration
```csharp
// src/Cli/StellaOps.Cli/Commands/CommandHandlers.VerifyBundle.cs
// Enhancement to existing VerifyDsseSignatureAsync method
private static async Task<bool> VerifyDsseSignatureAsync(
IServiceProvider services,
string dssePath,
string bundleDir,
List<BundleViolation> violations,
ILogger logger,
CancellationToken cancellationToken)
{
var dsseVerifier = services.GetService<IDsseVerifier>();
if (dsseVerifier is null)
{
logger.LogWarning("DSSE verifier not registered - signature verification skipped");
violations.Add(new BundleViolation(
"signature.verify.service_unavailable",
"DSSE verifier service not available"));
return false;
}
try
{
var envelopeJson = await File.ReadAllTextAsync(dssePath, cancellationToken)
.ConfigureAwait(false);
// Look for public key in attestation folder
var pubKeyPath = Path.Combine(bundleDir, "attestation", "public-key.pem");
if (!File.Exists(pubKeyPath))
{
pubKeyPath = Path.Combine(bundleDir, "attestation", "signing-key.pub");
}
if (!File.Exists(pubKeyPath))
{
violations.Add(new BundleViolation(
"signature.key.missing",
"No public key found in attestation folder"));
return false;
}
var publicKeyPem = await File.ReadAllTextAsync(pubKeyPath, cancellationToken)
.ConfigureAwait(false);
var result = await dsseVerifier.VerifyAsync(
envelopeJson,
publicKeyPem,
cancellationToken).ConfigureAwait(false);
if (!result.IsValid)
{
violations.Add(new BundleViolation(
"signature.verify.invalid",
result.Error ?? "DSSE signature verification failed"));
return false;
}
logger.LogInformation("DSSE signature verified successfully");
return true;
}
catch (Exception ex)
{
logger.LogError(ex, "DSSE verification threw exception");
violations.Add(new BundleViolation(
"signature.verify.exception",
$"Verification exception: {ex.Message}"));
return false;
}
}
```
### stella prove Command
```csharp
// src/Cli/StellaOps.Cli/Commands/ProveCommandGroup.cs
namespace StellaOps.Cli.Commands;
/// <summary>
/// Command group for replay proof operations.
/// </summary>
internal static class ProveCommandGroup
{
public static Command CreateProveCommand()
{
var imageOption = new Option<string>(
"--image",
"Image digest (sha256:...) to generate proof for")
{ IsRequired = true };
var atOption = new Option<DateTimeOffset?>(
"--at",
"Point-in-time for snapshot lookup (ISO 8601)");
var snapshotOption = new Option<string?>(
"--snapshot",
"Explicit snapshot ID to use instead of time lookup");
var outputOption = new Option<string>(
"--output",
() => "compact",
"Output format: compact, json, full");
var command = new Command("prove", "Generate replay proof for an image verdict")
{
imageOption,
atOption,
snapshotOption,
outputOption
};
command.SetHandler(async (context) =>
{
var image = context.ParseResult.GetValueForOption(imageOption)!;
var at = context.ParseResult.GetValueForOption(atOption);
var snapshot = context.ParseResult.GetValueForOption(snapshotOption);
var output = context.ParseResult.GetValueForOption(outputOption)!;
var ct = context.GetCancellationToken();
await HandleProveAsync(
context.BindingContext.GetRequiredService<IServiceProvider>(),
image, at, snapshot, output, ct).ConfigureAwait(false);
});
return command;
}
private static async Task HandleProveAsync(
IServiceProvider services,
string imageDigest,
DateTimeOffset? at,
string? snapshotId,
string outputFormat,
CancellationToken ct)
{
var timelineService = services.GetRequiredService<ITimelineQueryService>();
var bundleStore = services.GetRequiredService<IReplayBundleStore>();
var replayExecutor = services.GetRequiredService<IReplayExecutor>();
// Step 1: Resolve snapshot
string resolvedSnapshotId;
if (!string.IsNullOrEmpty(snapshotId))
{
resolvedSnapshotId = snapshotId;
}
else if (at.HasValue)
{
var query = new TimelineQuery
{
ArtifactDigest = imageDigest,
PointInTime = at.Value,
EventType = TimelineEventType.VerdictComputed
};
var result = await timelineService.QueryAsync(query, ct).ConfigureAwait(false);
if (result.Events.Count == 0)
{
AnsiConsole.MarkupLine("[red]No verdict found for image at specified time[/]");
Environment.ExitCode = 1;
return;
}
resolvedSnapshotId = result.Events[0].SnapshotId;
}
else
{
// Use latest snapshot
var latest = await timelineService.GetLatestSnapshotAsync(imageDigest, ct)
.ConfigureAwait(false);
if (latest is null)
{
AnsiConsole.MarkupLine("[red]No snapshots found for image[/]");
Environment.ExitCode = 1;
return;
}
resolvedSnapshotId = latest.SnapshotId;
}
// Step 2: Fetch bundle
var bundle = await bundleStore.GetBundleAsync(resolvedSnapshotId, ct)
.ConfigureAwait(false);
if (bundle is null)
{
AnsiConsole.MarkupLine($"[red]Bundle not found for snapshot {resolvedSnapshotId}[/]");
Environment.ExitCode = 1;
return;
}
// Step 3: Execute replay
var replayResult = await replayExecutor.ExecuteAsync(bundle, ct).ConfigureAwait(false);
// Step 4: Generate proof
var proof = new ReplayProof
{
BundleHash = bundle.Sha256,
PolicyVersion = bundle.Manifest.PolicyVersion,
VerdictRoot = replayResult.VerdictRoot,
DurationMs = replayResult.DurationMs,
VerdictMatches = replayResult.VerdictMatches,
ReplayedAt = DateTimeOffset.UtcNow,
EngineVersion = replayResult.EngineVersion
};
// Step 5: Output
switch (outputFormat.ToLowerInvariant())
{
case "compact":
AnsiConsole.WriteLine(proof.ToCompactString());
break;
case "json":
var json = JsonSerializer.Serialize(proof, new JsonSerializerOptions { WriteIndented = true });
AnsiConsole.WriteLine(json);
break;
case "full":
OutputFullProof(proof, replayResult);
break;
}
}
private static void OutputFullProof(ReplayProof proof, ReplayExecutionResult result)
{
var table = new Table().AddColumns("Field", "Value");
table.AddRow("Bundle Hash", proof.BundleHash);
table.AddRow("Policy Version", proof.PolicyVersion);
table.AddRow("Verdict Root", proof.VerdictRoot);
table.AddRow("Duration", $"{proof.DurationMs}ms");
table.AddRow("Verdict Matches", proof.VerdictMatches ? "[green]Yes[/]" : "[red]No[/]");
table.AddRow("Engine Version", proof.EngineVersion);
table.AddRow("Replayed At", proof.ReplayedAt.ToString("O"));
AnsiConsole.Write(table);
AnsiConsole.WriteLine();
AnsiConsole.MarkupLine("[bold]Compact Proof:[/]");
AnsiConsole.WriteLine(proof.ToCompactString());
}
}
```
---
## Delivery Tracker
| # | Task ID | Status | Dependency | Owners | Task Definition |
|---|---------|--------|------------|--------|-----------------|
| **VerdictBuilder Integration** |
| 1 | RPL-001 | TODO | - | Replay Guild | Define `IVerdictBuilder.ReplayAsync()` contract in `StellaOps.Verdict` |
| 2 | RPL-002 | TODO | RPL-001 | Replay Guild | Implement `VerdictBuilder.ReplayAsync()` using frozen inputs |
| 3 | RPL-003 | TODO | RPL-002 | Replay Guild | Wire `VerdictBuilder` into CLI DI container |
| 4 | RPL-004 | TODO | RPL-003 | Replay Guild | Update `CommandHandlers.VerifyBundle.ReplayVerdictAsync()` to use service |
| 5 | RPL-005 | TODO | RPL-004 | Replay Guild | Unit tests: VerdictBuilder replay with fixtures |
| **DSSE Verification** |
| 6 | RPL-006 | TODO | - | Attestor Guild | Define `IDsseVerifier` interface in `StellaOps.Attestation` |
| 7 | RPL-007 | TODO | RPL-006 | Attestor Guild | Implement `DsseVerifier` using existing `DsseHelper` |
| 8 | RPL-008 | TODO | RPL-007 | CLI Guild | Wire `DsseVerifier` into CLI DI container |
| 9 | RPL-009 | TODO | RPL-008 | CLI Guild | Update `CommandHandlers.VerifyBundle.VerifyDsseSignatureAsync()` |
| 10 | RPL-010 | TODO | RPL-009 | Attestor Guild | Unit tests: DSSE verification with valid/invalid signatures |
| **ReplayProof Schema** |
| 11 | RPL-011 | TODO | - | Replay Guild | Create `ReplayProof` model in `StellaOps.Replay.Core` |
| 12 | RPL-012 | TODO | RPL-011 | Replay Guild | Implement `ToCompactString()` with canonical JSON + SHA-256 |
| 13 | RPL-013 | TODO | RPL-012 | Replay Guild | Update `stella verify --bundle` to output replay proof |
| 14 | RPL-014 | TODO | RPL-013 | Replay Guild | Unit tests: Replay proof generation and parsing |
| **stella prove Command** |
| 15 | RPL-015 | TODO | RPL-011 | CLI Guild | Create `ProveCommandGroup.cs` with command structure |
| 16 | RPL-016 | TODO | RPL-015 | CLI Guild | Implement `ITimelineQueryService` adapter for snapshot lookup |
| 17 | RPL-017 | TODO | RPL-016 | CLI Guild | Implement `IReplayBundleStore` adapter for bundle retrieval |
| 18 | RPL-018 | TODO | RPL-017 | CLI Guild | Wire `stella prove` into main command tree |
| 19 | RPL-019 | TODO | RPL-018 | CLI Guild | Integration tests: `stella prove` with test bundles |
| **Documentation & Polish** |
| 20 | RPL-020 | TODO | RPL-019 | Docs Guild | Update `docs/cli/admin-reference.md` with new commands |
| 21 | RPL-021 | TODO | RPL-020 | Docs Guild | Create `docs/modules/replay/replay-proof-schema.md` |
| 22 | RPL-022 | TODO | RPL-021 | QA Guild | E2E test: Full verify → prove workflow |
---
## Success Metrics
| Metric | Before | After | Target |
|--------|--------|-------|--------|
| `stella verify --bundle` actually replays | No | Yes | 100% |
| DSSE signature verification functional | No | Yes | 100% |
| Compact replay proof format available | No | Yes | 100% |
| `stella prove --at` command available | No | Yes | 100% |
| Replay latency (warm cache) | N/A | <5s | <5s (p50) |
---
## Execution Log
| Date (UTC) | Update | Owner |
|------------|--------|-------|
| 2026-01-05 | Sprint created from product advisory gap analysis | Planning |
---
## Decisions & Risks
| Decision/Risk | Type | Mitigation |
|---------------|------|------------|
| VerdictBuilder may not be ready | Risk | Fall back to existing ReplayEngine, document limitations |
| DSSE verification requires key management | Constraint | Use embedded public key in bundle, document key rotation |
| Timeline service may not support point-in-time queries | Risk | Add snapshot-by-timestamp index to Postgres |
| Compact proof format needs to be tamper-evident | Decision | Include SHA-256 of canonical JSON, not just fields |
---
## Next Checkpoints
- RPL-001 through RPL-005 (VerdictBuilder) target completion
- RPL-006 through RPL-010 (DSSE) target completion
- RPL-015 through RPL-019 (stella prove) target completion
- RPL-022 (E2E) sprint completion gate

View File

@@ -0,0 +1,676 @@
# Sprint 20260105_002_002_FACET - Facet Abstraction Layer
## Topic & Scope
Implement the foundational "Facet" abstraction layer that enables per-facet sealing and drift tracking. This sprint defines the core domain models (`IFacet`, `FacetSeal`, `FacetDrift`), facet taxonomy, and per-facet Merkle tree computation.
**Advisory Reference:** Product advisory on facet sealing - "Facet Sealing & Drift Quotas" section.
**Key Insight:** A "facet" is a declared slice of an image (OS packages, language dependencies, key binaries, config files). By sealing facets individually with Merkle roots, we can track drift at granular levels and apply different quotas to different component types.
**Working directory:** `src/__Libraries/StellaOps.Facet/`, `src/Scanner/`
**Evidence:** New `StellaOps.Facet` library with models, Merkle tree computation, and facet extractors integrated into Scanner surface manifest.
---
## Dependencies & Concurrency
| Dependency | Type | Status |
|------------|------|--------|
| SurfaceManifestDocument | Internal | Available |
| Scanner Analyzers (OS, Lang, Native) | Internal | Available |
| Merkle tree utilities | Internal | Partial (single root exists) |
| ICryptoHash | Internal | Available |
**Parallel Execution:** FCT-001 through FCT-010 (core models) can proceed independently. FCT-011 through FCT-020 (extractors) depend on models. FCT-021 through FCT-025 (integration) depend on extractors.
---
## Documentation Prerequisites
- `docs/modules/scanner/architecture.md`
- `src/Scanner/__Libraries/StellaOps.Scanner.Surface.FS/SurfaceManifestModels.cs`
- CLAUDE.md determinism rules
- Product advisory on facet sealing
---
## Problem Analysis
### Current State
StellaOps currently:
- Computes a single `DeterminismMerkleRoot` for the entire scan output
- Tracks drift at aggregate level via `FnDriftCalculator`
- Has language-specific analyzers (DotNet, Node, Python, etc.)
- Has native analyzer for ELF/PE/Mach-O binaries
- No concept of "facets" as distinct trackable units
**Gaps:**
1. No facet taxonomy or abstraction
2. No per-facet Merkle roots
3. No facet-specific file selectors
4. Cannot apply different drift quotas to different component types
### Target Capabilities
```
┌─────────────────────────────────────────────────────────────────────────────┐
│ Facet Abstraction Layer │
│ │
│ ┌─────────────────────────────────────────────────────────────────────────┐ │
│ │ Facet Taxonomy │ │
│ │ │ │
│ │ OS Packages Language Dependencies Binaries │ │
│ │ ├─ dpkg/deb ├─ npm/node_modules ├─ /usr/bin/* │ │
│ │ ├─ rpm ├─ pip/site-packages ├─ /usr/lib/*.so │ │
│ │ ├─ apk ├─ nuget/packages ├─ *.dll │ │
│ │ └─ pacman ├─ maven/m2 └─ *.dylib │ │
│ │ ├─ cargo/registry │ │
│ │ Config Files └─ go/pkg Certificates │ │
│ │ ├─ /etc/* ├─ /etc/ssl/certs │ │
│ │ ├─ *.conf Interpreters ├─ /etc/pki │ │
│ │ └─ *.yaml ├─ /usr/bin/python* └─ trust anchors │ │
│ │ ├─ /usr/bin/node │ │
│ │ └─ /usr/bin/ruby │ │
│ │ │ │
│ └─────────────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────────────────────┐ │
│ │ FacetSeal Structure │ │
│ │ │ │
│ │ { │ │
│ │ "imageDigest": "sha256:abc...", │ │
│ │ "createdAt": "2026-01-05T12:00:00Z", │ │
│ │ "facets": [ │ │
│ │ { │ │
│ │ "name": "os-packages", │ │
│ │ "selector": "/var/lib/dpkg/status", │ │
│ │ "merkleRoot": "sha256:...", │ │
│ │ "fileCount": 1247, │ │
│ │ "totalBytes": 15_000_000 │ │
│ │ }, │ │
│ │ { │ │
│ │ "name": "lang-deps-npm", │ │
│ │ "selector": "**/node_modules/**/package.json", │ │
│ │ "merkleRoot": "sha256:...", │ │
│ │ "fileCount": 523, │ │
│ │ "totalBytes": 45_000_000 │ │
│ │ } │ │
│ │ ], │ │
│ │ "signature": "DSSE envelope" │ │
│ │ } │ │
│ │ │ │
│ └─────────────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────┘
```
---
## Architecture Design
### Core Facet Models
```csharp
// src/__Libraries/StellaOps.Facet/IFacet.cs
namespace StellaOps.Facet;
/// <summary>
/// Represents a trackable slice of an image.
/// </summary>
public interface IFacet
{
/// <summary>
/// Unique identifier for this facet type.
/// </summary>
string FacetId { get; }
/// <summary>
/// Human-readable name.
/// </summary>
string Name { get; }
/// <summary>
/// Facet category (os-packages, lang-deps, binaries, config, certs).
/// </summary>
FacetCategory Category { get; }
/// <summary>
/// Glob patterns or path selectors for files in this facet.
/// </summary>
IReadOnlyList<string> Selectors { get; }
/// <summary>
/// Priority for conflict resolution when files match multiple facets.
/// Lower = higher priority.
/// </summary>
int Priority { get; }
}
public enum FacetCategory
{
OsPackages,
LanguageDependencies,
Binaries,
Configuration,
Certificates,
Interpreters,
Custom
}
```
### Facet Seal Model
```csharp
// src/__Libraries/StellaOps.Facet/FacetSeal.cs
namespace StellaOps.Facet;
/// <summary>
/// Sealed manifest of facets for an image at a point in time.
/// </summary>
public sealed record FacetSeal
{
/// <summary>
/// Schema version for forward compatibility.
/// </summary>
public string SchemaVersion { get; init; } = "1.0.0";
/// <summary>
/// Image this seal applies to.
/// </summary>
public required string ImageDigest { get; init; }
/// <summary>
/// When the seal was created.
/// </summary>
public required DateTimeOffset CreatedAt { get; init; }
/// <summary>
/// Build attestation reference (in-toto provenance).
/// </summary>
public string? BuildAttestationRef { get; init; }
/// <summary>
/// Individual facet seals.
/// </summary>
public required ImmutableArray<FacetEntry> Facets { get; init; }
/// <summary>
/// Quota configuration per facet.
/// </summary>
public ImmutableDictionary<string, FacetQuota>? Quotas { get; init; }
/// <summary>
/// Combined Merkle root of all facet roots (for single-value integrity check).
/// </summary>
public required string CombinedMerkleRoot { get; init; }
/// <summary>
/// DSSE signature over canonical form.
/// </summary>
public string? Signature { get; init; }
}
public sealed record FacetEntry
{
/// <summary>
/// Facet identifier (e.g., "os-packages-dpkg", "lang-deps-npm").
/// </summary>
public required string FacetId { get; init; }
/// <summary>
/// Human-readable name.
/// </summary>
public required string Name { get; init; }
/// <summary>
/// Category for grouping.
/// </summary>
public required FacetCategory Category { get; init; }
/// <summary>
/// Selectors used to identify files in this facet.
/// </summary>
public required ImmutableArray<string> Selectors { get; init; }
/// <summary>
/// Merkle root of all files in this facet.
/// </summary>
public required string MerkleRoot { get; init; }
/// <summary>
/// Number of files in this facet.
/// </summary>
public required int FileCount { get; init; }
/// <summary>
/// Total bytes across all files.
/// </summary>
public required long TotalBytes { get; init; }
/// <summary>
/// Optional: individual file entries (for detailed audit).
/// </summary>
public ImmutableArray<FacetFileEntry>? Files { get; init; }
}
public sealed record FacetFileEntry(
string Path,
string Digest,
long SizeBytes,
DateTimeOffset? ModifiedAt);
public sealed record FacetQuota
{
/// <summary>
/// Maximum allowed churn percentage (0-100).
/// </summary>
public decimal MaxChurnPercent { get; init; } = 10m;
/// <summary>
/// Maximum number of changed files before alert.
/// </summary>
public int MaxChangedFiles { get; init; } = 50;
/// <summary>
/// Glob patterns for files exempt from quota enforcement.
/// </summary>
public ImmutableArray<string> AllowlistGlobs { get; init; } = [];
/// <summary>
/// Action when quota exceeded: Warn, Block, RequireVex.
/// </summary>
public QuotaExceededAction Action { get; init; } = QuotaExceededAction.Warn;
}
public enum QuotaExceededAction
{
Warn,
Block,
RequireVex
}
```
### Facet Drift Model
```csharp
// src/__Libraries/StellaOps.Facet/FacetDrift.cs
namespace StellaOps.Facet;
/// <summary>
/// Drift detection result for a single facet.
/// </summary>
public sealed record FacetDrift
{
/// <summary>
/// Facet this drift applies to.
/// </summary>
public required string FacetId { get; init; }
/// <summary>
/// Files added since baseline.
/// </summary>
public required ImmutableArray<FacetFileEntry> Added { get; init; }
/// <summary>
/// Files removed since baseline.
/// </summary>
public required ImmutableArray<FacetFileEntry> Removed { get; init; }
/// <summary>
/// Files modified since baseline.
/// </summary>
public required ImmutableArray<FacetFileModification> Modified { get; init; }
/// <summary>
/// Drift score (0-100, higher = more drift).
/// </summary>
public required decimal DriftScore { get; init; }
/// <summary>
/// Quota evaluation result.
/// </summary>
public required QuotaVerdict QuotaVerdict { get; init; }
/// <summary>
/// Churn percentage = (added + removed + modified) / baseline count * 100.
/// </summary>
public decimal ChurnPercent => BaselineFileCount > 0
? (Added.Length + Removed.Length + Modified.Length) / (decimal)BaselineFileCount * 100
: 0;
/// <summary>
/// Number of files in baseline facet seal.
/// </summary>
public required int BaselineFileCount { get; init; }
}
public sealed record FacetFileModification(
string Path,
string PreviousDigest,
string CurrentDigest,
long PreviousSizeBytes,
long CurrentSizeBytes);
public enum QuotaVerdict
{
Ok,
Warning,
Blocked,
RequiresVex
}
```
### Facet Merkle Tree
```csharp
// src/__Libraries/StellaOps.Facet/FacetMerkleTree.cs
namespace StellaOps.Facet;
/// <summary>
/// Computes Merkle roots for facet file sets.
/// </summary>
public sealed class FacetMerkleTree
{
private readonly ICryptoHash _cryptoHash;
public FacetMerkleTree(ICryptoHash cryptoHash)
{
_cryptoHash = cryptoHash ?? throw new ArgumentNullException(nameof(cryptoHash));
}
/// <summary>
/// Compute Merkle root from file entries.
/// </summary>
public string ComputeRoot(IEnumerable<FacetFileEntry> files)
{
// Sort files by path for determinism
var sortedFiles = files
.OrderBy(f => f.Path, StringComparer.Ordinal)
.ToList();
if (sortedFiles.Count == 0)
{
// Empty tree root = SHA-256 of empty string
return "sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855";
}
// Build leaf nodes: hash of (path + digest + size)
var leaves = sortedFiles
.Select(f => ComputeLeafHash(f))
.ToList();
// Build tree bottom-up
return ComputeMerkleRoot(leaves);
}
private byte[] ComputeLeafHash(FacetFileEntry file)
{
// Canonical leaf format: "path|digest|size"
var canonical = $"{file.Path}|{file.Digest}|{file.SizeBytes}";
return _cryptoHash.ComputeHash(
System.Text.Encoding.UTF8.GetBytes(canonical),
"SHA256");
}
private string ComputeMerkleRoot(List<byte[]> nodes)
{
while (nodes.Count > 1)
{
var nextLevel = new List<byte[]>();
for (var i = 0; i < nodes.Count; i += 2)
{
if (i + 1 < nodes.Count)
{
// Combine two nodes
var combined = new byte[nodes[i].Length + nodes[i + 1].Length];
nodes[i].CopyTo(combined, 0);
nodes[i + 1].CopyTo(combined, nodes[i].Length);
nextLevel.Add(_cryptoHash.ComputeHash(combined, "SHA256"));
}
else
{
// Odd node: promote as-is
nextLevel.Add(nodes[i]);
}
}
nodes = nextLevel;
}
return $"sha256:{Convert.ToHexString(nodes[0]).ToLowerInvariant()}";
}
/// <summary>
/// Compute combined root from multiple facet roots.
/// </summary>
public string ComputeCombinedRoot(IEnumerable<FacetEntry> facets)
{
var facetRoots = facets
.OrderBy(f => f.FacetId, StringComparer.Ordinal)
.Select(f => HexToBytes(f.MerkleRoot.Replace("sha256:", "")))
.ToList();
return ComputeMerkleRoot(facetRoots);
}
private static byte[] HexToBytes(string hex)
{
return Convert.FromHexString(hex);
}
}
```
### Built-in Facet Definitions
```csharp
// src/__Libraries/StellaOps.Facet/BuiltInFacets.cs
namespace StellaOps.Facet;
/// <summary>
/// Built-in facet definitions for common image components.
/// </summary>
public static class BuiltInFacets
{
public static IReadOnlyList<IFacet> All { get; } = new IFacet[]
{
// OS Package Managers
new FacetDefinition("os-packages-dpkg", "Debian Packages", FacetCategory.OsPackages,
["/var/lib/dpkg/status", "/var/lib/dpkg/info/**"], priority: 10),
new FacetDefinition("os-packages-rpm", "RPM Packages", FacetCategory.OsPackages,
["/var/lib/rpm/**", "/usr/lib/sysimage/rpm/**"], priority: 10),
new FacetDefinition("os-packages-apk", "Alpine Packages", FacetCategory.OsPackages,
["/lib/apk/db/**"], priority: 10),
// Language Dependencies
new FacetDefinition("lang-deps-npm", "NPM Packages", FacetCategory.LanguageDependencies,
["**/node_modules/**/package.json", "**/package-lock.json"], priority: 20),
new FacetDefinition("lang-deps-pip", "Python Packages", FacetCategory.LanguageDependencies,
["**/site-packages/**/*.dist-info/METADATA", "**/requirements.txt"], priority: 20),
new FacetDefinition("lang-deps-nuget", "NuGet Packages", FacetCategory.LanguageDependencies,
["**/*.deps.json", "**/*.nuget/**"], priority: 20),
new FacetDefinition("lang-deps-maven", "Maven Packages", FacetCategory.LanguageDependencies,
["**/.m2/repository/**/*.pom"], priority: 20),
new FacetDefinition("lang-deps-cargo", "Cargo Packages", FacetCategory.LanguageDependencies,
["**/.cargo/registry/**", "**/Cargo.lock"], priority: 20),
new FacetDefinition("lang-deps-go", "Go Modules", FacetCategory.LanguageDependencies,
["**/go.sum", "**/go/pkg/mod/**"], priority: 20),
// Binaries
new FacetDefinition("binaries-usr", "System Binaries", FacetCategory.Binaries,
["/usr/bin/*", "/usr/sbin/*", "/bin/*", "/sbin/*"], priority: 30),
new FacetDefinition("binaries-lib", "Shared Libraries", FacetCategory.Binaries,
["/usr/lib/**/*.so*", "/lib/**/*.so*", "/usr/lib64/**/*.so*"], priority: 30),
// Interpreters
new FacetDefinition("interpreters", "Language Interpreters", FacetCategory.Interpreters,
["/usr/bin/python*", "/usr/bin/node*", "/usr/bin/ruby*", "/usr/bin/perl*"], priority: 15),
// Configuration
new FacetDefinition("config-etc", "System Configuration", FacetCategory.Configuration,
["/etc/**/*.conf", "/etc/**/*.cfg", "/etc/**/*.yaml", "/etc/**/*.json"], priority: 40),
// Certificates
new FacetDefinition("certs-system", "System Certificates", FacetCategory.Certificates,
["/etc/ssl/certs/**", "/etc/pki/**", "/usr/share/ca-certificates/**"], priority: 25),
};
public static IFacet? GetById(string facetId)
=> All.FirstOrDefault(f => f.FacetId == facetId);
}
internal sealed class FacetDefinition : IFacet
{
public string FacetId { get; }
public string Name { get; }
public FacetCategory Category { get; }
public IReadOnlyList<string> Selectors { get; }
public int Priority { get; }
public FacetDefinition(
string facetId,
string name,
FacetCategory category,
string[] selectors,
int priority)
{
FacetId = facetId;
Name = name;
Category = category;
Selectors = selectors;
Priority = priority;
}
}
```
### Facet Extractor Interface
```csharp
// src/__Libraries/StellaOps.Facet/IFacetExtractor.cs
namespace StellaOps.Facet;
/// <summary>
/// Extracts facet file entries from an image filesystem.
/// </summary>
public interface IFacetExtractor
{
/// <summary>
/// Extract files matching a facet's selectors.
/// </summary>
Task<FacetExtractionResult> ExtractAsync(
IFacet facet,
IImageFileSystem imageFs,
FacetExtractionOptions? options = null,
CancellationToken ct = default);
}
public sealed record FacetExtractionResult(
string FacetId,
ImmutableArray<FacetFileEntry> Files,
long TotalBytes,
TimeSpan Duration,
ImmutableArray<string>? Errors);
public sealed record FacetExtractionOptions
{
/// <summary>
/// Include file content hashes (slower but required for sealing).
/// </summary>
public bool ComputeHashes { get; init; } = true;
/// <summary>
/// Maximum files to extract per facet (0 = unlimited).
/// </summary>
public int MaxFiles { get; init; } = 0;
/// <summary>
/// Skip files larger than this size.
/// </summary>
public long MaxFileSizeBytes { get; init; } = 100 * 1024 * 1024; // 100MB
/// <summary>
/// Follow symlinks when extracting.
/// </summary>
public bool FollowSymlinks { get; init; } = false;
}
```
---
## Delivery Tracker
| # | Task ID | Status | Dependency | Owners | Task Definition |
|---|---------|--------|------------|--------|-----------------|
| **Core Models** |
| 1 | FCT-001 | TODO | - | Facet Guild | Create `StellaOps.Facet` project structure |
| 2 | FCT-002 | TODO | FCT-001 | Facet Guild | Define `IFacet` interface and `FacetCategory` enum |
| 3 | FCT-003 | TODO | FCT-002 | Facet Guild | Define `FacetSeal` model with entries and quotas |
| 4 | FCT-004 | TODO | FCT-003 | Facet Guild | Define `FacetDrift` model with change tracking |
| 5 | FCT-005 | TODO | FCT-004 | Facet Guild | Define `FacetQuota` model with actions |
| 6 | FCT-006 | TODO | FCT-005 | Facet Guild | Unit tests: Model serialization round-trips |
| **Merkle Tree** |
| 7 | FCT-007 | TODO | FCT-003 | Facet Guild | Implement `FacetMerkleTree` with leaf computation |
| 8 | FCT-008 | TODO | FCT-007 | Facet Guild | Implement combined root from multiple facets |
| 9 | FCT-009 | TODO | FCT-008 | Facet Guild | Unit tests: Merkle root determinism |
| 10 | FCT-010 | TODO | FCT-009 | Facet Guild | Golden tests: Known inputs → known roots |
| **Built-in Facets** |
| 11 | FCT-011 | TODO | FCT-002 | Facet Guild | Define OS package facets (dpkg, rpm, apk) |
| 12 | FCT-012 | TODO | FCT-011 | Facet Guild | Define language dependency facets (npm, pip, etc.) |
| 13 | FCT-013 | TODO | FCT-012 | Facet Guild | Define binary facets (usr/bin, libs) |
| 14 | FCT-014 | TODO | FCT-013 | Facet Guild | Define config and certificate facets |
| 15 | FCT-015 | TODO | FCT-014 | Facet Guild | Create `BuiltInFacets` registry |
| **Extraction** |
| 16 | FCT-016 | TODO | FCT-015 | Scanner Guild | Define `IFacetExtractor` interface |
| 17 | FCT-017 | TODO | FCT-016 | Scanner Guild | Implement `GlobFacetExtractor` for selector matching |
| 18 | FCT-018 | TODO | FCT-017 | Scanner Guild | Integrate with Scanner's `IImageFileSystem` |
| 19 | FCT-019 | TODO | FCT-018 | Scanner Guild | Unit tests: Extraction from mock FS |
| 20 | FCT-020 | TODO | FCT-019 | Scanner Guild | Integration tests: Extraction from real image layers |
| **Surface Manifest Integration** |
| 21 | FCT-021 | TODO | FCT-020 | Scanner Guild | Add `FacetSeals` property to `SurfaceManifestDocument` |
| 22 | FCT-022 | TODO | FCT-021 | Scanner Guild | Compute facet seals during scan surface publishing |
| 23 | FCT-023 | TODO | FCT-022 | Scanner Guild | Store facet seals in Postgres alongside surface manifest |
| 24 | FCT-024 | TODO | FCT-023 | Scanner Guild | Unit tests: Surface manifest with facets |
| 25 | FCT-025 | TODO | FCT-024 | QA Guild | E2E test: Scan → facet seal generation |
---
## Success Metrics
| Metric | Before | After | Target |
|--------|--------|-------|--------|
| Per-facet Merkle roots available | No | Yes | 100% |
| Facet taxonomy defined | No | Yes | 15+ facet types |
| Facet extraction from images | No | Yes | All built-in facets |
| Surface manifest includes facets | No | Yes | 100% |
| Merkle computation deterministic | N/A | Yes | 100% reproducible |
---
## Execution Log
| Date (UTC) | Update | Owner |
|------------|--------|-------|
| 2026-01-05 | Sprint created from product advisory gap analysis | Planning |
---
## Decisions & Risks
| Decision/Risk | Type | Mitigation |
|---------------|------|------------|
| Facet selectors may overlap | Decision | Use priority field, document conflict resolution |
| Large images may have many files | Risk | Add MaxFiles limit, streaming extraction |
| Merkle computation adds scan latency | Trade-off | Make facet sealing opt-in via config |
| Glob matching performance | Risk | Use optimized glob library (DotNet.Glob) |
| Symlink handling complexity | Decision | Default to not following, document rationale |
---
## Next Checkpoints
- FCT-001 through FCT-006 (core models) target completion
- FCT-007 through FCT-010 (Merkle) target completion
- FCT-016 through FCT-020 (extraction) target completion
- FCT-025 (E2E) sprint completion gate

View File

@@ -0,0 +1,427 @@
# Sprint 20260105_002_002_SCHEDULER - HLC: Scheduler Queue Chain Integration
## Topic & Scope
Integrate Hybrid Logical Clock (HLC) into the Scheduler queue with cryptographic sequence proofs at enqueue time. This implements the core advisory requirement: "derive order from deterministic, monotonic time inside your system and prove the sequence with hashes."
- **Working directory:** `src/Scheduler/`
- **Evidence:** Updated schema, queue implementations, chain verification tests
## Problem Statement
Current Scheduler queue implementation:
- Orders by `(priority DESC, created_at ASC, id)` - wall-clock based
- No hash chain at enqueue - chains only exist in downstream ledgers
- Redis/NATS sequences provide local ordering, not global HLC ordering
Advisory requires:
- HLC timestamp assigned at enqueue: `t_hlc = hlc.Tick()`
- Chain link computed: `link = Hash(prev_link || job_id || t_hlc || payload_hash)`
- Total order persisted at enqueue, not dequeue
## Dependencies & Concurrency
- **Depends on:** SPRINT_20260105_002_001_LB (HLC core library)
- **Blocks:** SPRINT_20260105_002_003_ROUTER (offline merge protocol)
- **Parallel safe:** Scheduler-only changes; no cross-module conflicts
## Documentation Prerequisites
- docs/modules/scheduler/architecture.md
- src/Scheduler/AGENTS.md
- SPRINT_20260105_002_001_LB (HLC library design)
- Product Advisory: scheduler_log table specification
## Technical Design
### Database Schema Changes
```sql
-- New: Scheduler log table for HLC-ordered, chain-linked jobs
CREATE TABLE scheduler.scheduler_log (
seq_bigint BIGSERIAL PRIMARY KEY, -- Storage order (not authoritative)
tenant_id TEXT NOT NULL,
t_hlc TEXT NOT NULL, -- HLC timestamp string
partition_key TEXT, -- Optional queue partition
job_id UUID NOT NULL,
payload_hash BYTEA NOT NULL, -- SHA-256 of canonical payload
prev_link BYTEA, -- Previous chain link (null for first)
link BYTEA NOT NULL, -- Hash(prev_link || job_id || t_hlc || payload_hash)
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
CONSTRAINT uq_scheduler_log_order UNIQUE (tenant_id, t_hlc, partition_key, job_id)
);
CREATE INDEX idx_scheduler_log_tenant_hlc ON scheduler.scheduler_log(tenant_id, t_hlc);
CREATE INDEX idx_scheduler_log_partition ON scheduler.scheduler_log(tenant_id, partition_key, t_hlc);
-- New: Batch snapshot table for audit anchors
CREATE TABLE scheduler.batch_snapshot (
batch_id UUID PRIMARY KEY,
tenant_id TEXT NOT NULL,
range_start_t TEXT NOT NULL, -- HLC range start
range_end_t TEXT NOT NULL, -- HLC range end
head_link BYTEA NOT NULL, -- Chain head at snapshot
job_count INT NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
signed_by TEXT, -- Optional key ID for DSSE
signature BYTEA -- Optional DSSE signature
);
CREATE INDEX idx_batch_snapshot_tenant ON scheduler.batch_snapshot(tenant_id, created_at DESC);
-- New: Per-partition chain head tracking
CREATE TABLE scheduler.chain_heads (
tenant_id TEXT NOT NULL,
partition_key TEXT NOT NULL DEFAULT '',
last_link BYTEA NOT NULL,
last_t_hlc TEXT NOT NULL,
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
PRIMARY KEY (tenant_id, partition_key)
);
```
### Chain Link Computation
```csharp
public static class SchedulerChainLinking
{
/// <summary>
/// Compute chain link per advisory specification:
/// link_i = Hash(link_{i-1} || job_id || t_hlc || payload_hash)
/// </summary>
public static byte[] ComputeLink(
byte[]? prevLink,
Guid jobId,
HlcTimestamp tHlc,
byte[] payloadHash)
{
using var hasher = IncrementalHash.CreateHash(HashAlgorithmName.SHA256);
// Previous link (or 32 zero bytes for first entry)
hasher.AppendData(prevLink ?? new byte[32]);
// Job ID as bytes (big-endian for consistency)
hasher.AppendData(jobId.ToByteArray());
// HLC timestamp as UTF-8 bytes
hasher.AppendData(Encoding.UTF8.GetBytes(tHlc.ToSortableString()));
// Payload hash
hasher.AppendData(payloadHash);
return hasher.GetHashAndReset();
}
/// <summary>
/// Compute deterministic payload hash from canonical JSON.
/// </summary>
public static byte[] ComputePayloadHash(object payload)
{
var canonical = CanonicalJsonSerializer.Serialize(payload);
return SHA256.HashData(Encoding.UTF8.GetBytes(canonical));
}
}
```
### Enqueue Flow Enhancement
```csharp
public sealed class HlcSchedulerEnqueueService
{
private readonly IHybridLogicalClock _hlc;
private readonly ISchedulerLogRepository _logRepository;
private readonly IChainHeadRepository _chainHeadRepository;
private readonly IGuidProvider _guidProvider;
public async Task<SchedulerEnqueueResult> EnqueueAsync(
SchedulerJobPayload payload,
CancellationToken ct = default)
{
// 1. Generate HLC timestamp
var tHlc = _hlc.Tick();
// 2. Compute deterministic job ID from payload (idempotency)
var jobId = ComputeDeterministicJobId(payload);
// 3. Compute payload hash
var payloadHash = SchedulerChainLinking.ComputePayloadHash(payload);
// 4. Get previous chain link
var prevLink = await _chainHeadRepository.GetLastLinkAsync(
payload.TenantId,
payload.PartitionKey,
ct);
// 5. Compute new chain link
var link = SchedulerChainLinking.ComputeLink(prevLink, jobId, tHlc, payloadHash);
// 6. Insert log entry (atomic with chain head update)
await _logRepository.InsertWithChainUpdateAsync(
new SchedulerLogEntry
{
TenantId = payload.TenantId,
THlc = tHlc.ToSortableString(),
PartitionKey = payload.PartitionKey,
JobId = jobId,
PayloadHash = payloadHash,
PrevLink = prevLink,
Link = link
},
ct);
return new SchedulerEnqueueResult(tHlc, jobId, link);
}
private Guid ComputeDeterministicJobId(SchedulerJobPayload payload)
{
// GUID v5 (SHA-1 based) over canonical JSON for determinism
var canonical = CanonicalJsonSerializer.Serialize(payload);
return GuidUtility.Create(
SchedulerNamespaces.JobPayload,
canonical,
version: 5);
}
}
```
### Dequeue with HLC Ordering
```csharp
public sealed class HlcSchedulerDequeueService
{
private readonly ISchedulerLogRepository _logRepository;
private readonly IJobVerdictRepository _verdictRepository;
public async Task<IReadOnlyList<SchedulerJob>> DequeueAsync(
string tenantId,
string? partitionKey,
int limit,
CancellationToken ct = default)
{
// Query by HLC order (ascending) for deterministic dequeue
var logEntries = await _logRepository.GetByHlcOrderAsync(
tenantId,
partitionKey,
limit,
ct);
var jobs = new List<SchedulerJob>();
foreach (var entry in logEntries)
{
// Check idempotency: skip if verdict already exists
var verdict = await _verdictRepository.GetAsync(entry.JobId, ct);
if (verdict is not null)
{
continue; // Already processed
}
jobs.Add(MapToJob(entry));
}
return jobs;
}
}
```
### Batch Snapshot Creation
```csharp
public sealed class BatchSnapshotService
{
private readonly ISchedulerLogRepository _logRepository;
private readonly IBatchSnapshotRepository _snapshotRepository;
private readonly IAttestationSigningService? _signingService;
public async Task<BatchSnapshot> CreateSnapshotAsync(
string tenantId,
HlcTimestamp startT,
HlcTimestamp endT,
CancellationToken ct = default)
{
// 1. Select jobs in HLC range
var jobs = await _logRepository.GetByHlcRangeAsync(
tenantId,
startT.ToSortableString(),
endT.ToSortableString(),
ct);
if (jobs.Count == 0)
{
throw new InvalidOperationException("No jobs in specified HLC range");
}
// 2. Get chain head (last link in range)
var headLink = jobs[^1].Link;
// 3. Create snapshot
var snapshot = new BatchSnapshot
{
BatchId = Guid.NewGuid(),
TenantId = tenantId,
RangeStartT = startT.ToSortableString(),
RangeEndT = endT.ToSortableString(),
HeadLink = headLink,
JobCount = jobs.Count,
CreatedAt = DateTimeOffset.UtcNow
};
// 4. Optional: Sign snapshot with DSSE
if (_signingService is not null)
{
var digest = ComputeSnapshotDigest(snapshot, jobs);
var signed = await _signingService.SignAsync(digest, ct);
snapshot = snapshot with
{
SignedBy = signed.KeyId,
Signature = signed.Signature
};
}
// 5. Persist
await _snapshotRepository.InsertAsync(snapshot, ct);
return snapshot;
}
}
```
## Delivery Tracker
| # | Task ID | Status | Dependency | Owner | Task Definition |
|---|---------|--------|------------|-------|-----------------|
| 1 | SQC-001 | TODO | HLC lib | Guild | Add StellaOps.HybridLogicalClock reference to Scheduler projects |
| 2 | SQC-002 | TODO | SQC-001 | Guild | Create migration: `scheduler.scheduler_log` table |
| 3 | SQC-003 | TODO | SQC-002 | Guild | Create migration: `scheduler.batch_snapshot` table |
| 4 | SQC-004 | TODO | SQC-002 | Guild | Create migration: `scheduler.chain_heads` table |
| 5 | SQC-005 | TODO | SQC-004 | Guild | Implement `ISchedulerLogRepository` interface |
| 6 | SQC-006 | TODO | SQC-005 | Guild | Implement `PostgresSchedulerLogRepository` |
| 7 | SQC-007 | TODO | SQC-004 | Guild | Implement `IChainHeadRepository` and Postgres implementation |
| 8 | SQC-008 | TODO | SQC-006 | Guild | Implement `SchedulerChainLinking` static class |
| 9 | SQC-009 | TODO | SQC-008 | Guild | Implement `HlcSchedulerEnqueueService` |
| 10 | SQC-010 | TODO | SQC-009 | Guild | Implement `HlcSchedulerDequeueService` |
| 11 | SQC-011 | TODO | SQC-010 | Guild | Update Redis queue adapter to include HLC in message |
| 12 | SQC-012 | TODO | SQC-010 | Guild | Update NATS queue adapter to include HLC in message |
| 13 | SQC-013 | TODO | SQC-006 | Guild | Implement `BatchSnapshotService` |
| 14 | SQC-014 | TODO | SQC-013 | Guild | Add DSSE signing integration for batch snapshots |
| 15 | SQC-015 | TODO | SQC-008 | Guild | Implement chain verification: `VerifyChainIntegrity()` |
| 16 | SQC-016 | TODO | SQC-015 | Guild | Write unit tests: chain linking, HLC ordering |
| 17 | SQC-017 | TODO | SQC-016 | Guild | Write integration tests: enqueue/dequeue with chain |
| 18 | SQC-018 | TODO | SQC-017 | Guild | Write determinism tests: same input -> same chain |
| 19 | SQC-019 | TODO | SQC-018 | Guild | Update existing JobRepository to use HLC ordering optionally |
| 20 | SQC-020 | TODO | SQC-019 | Guild | Feature flag: `SchedulerOptions.EnableHlcOrdering` |
| 21 | SQC-021 | TODO | SQC-020 | Guild | Migration guide: enabling HLC on existing deployments |
| 22 | SQC-022 | TODO | SQC-021 | Guild | Metrics: `scheduler_hlc_enqueues_total`, `scheduler_chain_verifications_total` |
## Chain Verification
```csharp
public sealed class SchedulerChainVerifier
{
public async Task<ChainVerificationResult> VerifyAsync(
string tenantId,
string? partitionKey,
HlcTimestamp? startT = null,
HlcTimestamp? endT = null,
CancellationToken ct = default)
{
var entries = await _logRepository.GetByHlcRangeAsync(
tenantId,
startT?.ToSortableString(),
endT?.ToSortableString(),
ct);
byte[]? expectedPrevLink = null;
var issues = new List<ChainVerificationIssue>();
foreach (var entry in entries)
{
// Verify prev_link matches expected
if (!ByteArrayEquals(entry.PrevLink, expectedPrevLink))
{
issues.Add(new ChainVerificationIssue(
entry.JobId,
entry.THlc,
"PrevLinkMismatch",
$"Expected {ToHex(expectedPrevLink)}, got {ToHex(entry.PrevLink)}"));
}
// Recompute link and verify
var computed = SchedulerChainLinking.ComputeLink(
entry.PrevLink,
entry.JobId,
HlcTimestamp.Parse(entry.THlc),
entry.PayloadHash);
if (!ByteArrayEquals(entry.Link, computed))
{
issues.Add(new ChainVerificationIssue(
entry.JobId,
entry.THlc,
"LinkMismatch",
$"Stored link doesn't match computed"));
}
expectedPrevLink = entry.Link;
}
return new ChainVerificationResult(
IsValid: issues.Count == 0,
EntriesChecked: entries.Count,
Issues: issues);
}
}
```
## Backward Compatibility
### Feature Flag
```csharp
public sealed class SchedulerOptions
{
/// <summary>
/// Enable HLC-based ordering with chain linking.
/// When false, uses legacy (priority, created_at) ordering.
/// </summary>
public bool EnableHlcOrdering { get; set; } = false;
/// <summary>
/// When true, writes to both legacy and HLC tables during migration.
/// </summary>
public bool DualWriteMode { get; set; } = false;
}
```
### Migration Path
1. **Phase 1:** Deploy with `DualWriteMode = true` - writes to both tables
2. **Phase 2:** Backfill `scheduler_log` from existing `scheduler.jobs`
3. **Phase 3:** Enable `EnableHlcOrdering = true` for reads
4. **Phase 4:** Disable `DualWriteMode`, deprecate legacy ordering
## Decisions & Risks
| Decision | Rationale |
|----------|-----------|
| Separate `scheduler_log` table | Avoid schema changes to existing `jobs` table |
| Store HLC as TEXT | Human-readable, sortable, avoids custom types |
| SHA-256 for chain links | Consistent with existing hash usage; pluggable |
| Optional DSSE signing | Not all deployments need attestation |
| Risk | Mitigation |
|------|------------|
| Performance regression | Benchmark; index optimization; optional feature |
| Chain corruption | Verification function; alerts on mismatch |
| Migration complexity | Dual-write mode; gradual rollout |
## Execution Log
| Date (UTC) | Update | Owner |
|------------|--------|-------|
| 2026-01-05 | Sprint created from product advisory gap analysis | Planning |
## Next Checkpoints
- 2026-01-09: SQC-001 to SQC-008 complete (schema + core)
- 2026-01-10: SQC-009 to SQC-014 complete (services)
- 2026-01-11: SQC-015 to SQC-022 complete (verification, tests, docs)

View File

@@ -0,0 +1,701 @@
# Sprint 20260105_002_003_FACET - Per-Facet Drift Quotas
## Topic & Scope
Implement per-facet drift quota enforcement that tracks changes against sealed baselines and applies configurable thresholds with WARN/BLOCK/RequireVex actions. This sprint extends the existing `FnDriftCalculator` to support facet-level granularity.
**Advisory Reference:** Product advisory on facet sealing - "Track drift" and "Quotas & actions" sections.
**Key Insight:** Different facets should have different drift tolerances. OS package updates are expected during patching, but binary changes outside of known patches are suspicious. Per-facet quotas enable nuanced enforcement.
**Working directory:** `src/__Libraries/StellaOps.Facet/`, `src/Policy/__Libraries/StellaOps.Policy/Gates/`
**Evidence:** `IFacetQuotaEnforcer` with per-facet drift tracking, quota breach actions integrated into policy gates, auto-VEX draft generation for authorized drift.
---
## Dependencies & Concurrency
| Dependency | Type | Status |
|------------|------|--------|
| SPRINT_20260105_002_002_FACET (models) | Sprint | Required |
| FnDriftCalculator | Internal | Available |
| BudgetConstraintEnforcer | Internal | Available |
| DeltaSigVexEmitter | Internal | Available |
| FacetSeal model | Sprint 002 | Required |
**Parallel Execution:** QTA-001 through QTA-008 (drift engine) can proceed independently. QTA-009 through QTA-015 (enforcement) depend on drift engine. QTA-016 through QTA-020 (auto-VEX) can proceed in parallel with enforcement.
---
## Documentation Prerequisites
- SPRINT_20260105_002_002_FACET models
- `src/Scanner/__Libraries/StellaOps.Scanner.Storage/Services/FnDriftCalculator.cs`
- `src/Policy/__Libraries/StellaOps.Policy/Gates/BudgetConstraintEnforcer.cs`
- `src/Scanner/__Libraries/StellaOps.Scanner.Evidence/DeltaSigVexEmitter.cs`
---
## Problem Analysis
### Current State
StellaOps currently:
- Tracks aggregate FN-Drift with cause attribution (Feed, Rule, Lattice, Reachability, Engine)
- Has budget enforcement with Risk Points (RP) and gate levels
- Generates auto-VEX from delta signature detection
- No per-facet drift tracking
- No facet-specific quota configuration
- No quota-based BLOCK actions
**Gaps:**
1. `FnDriftCalculator` operates at artifact level, not facet level
2. No `FacetDriftEngine` to compare current vs sealed baseline
3. No quota enforcement per facet
4. No facet-aware auto-VEX generation
### Target Capabilities
```
┌─────────────────────────────────────────────────────────────────────────────┐
│ Per-Facet Quota Enforcement │
│ │
│ ┌─────────────────────────────────────────────────────────────────────────┐ │
│ │ FacetDriftEngine │ │
│ │ │ │
│ │ Input: Current Image + Sealed Baseline │ │
│ │ │ │
│ │ For each facet: │ │
│ │ 1. Extract current files via IFacetExtractor │ │
│ │ 2. Load baseline FacetEntry from FacetSeal │ │
│ │ 3. Compute diff: added, removed, modified │ │
│ │ 4. Calculate drift score and churn % │ │
│ │ 5. Evaluate quota: MaxChurnPercent, MaxChangedFiles │ │
│ │ 6. Determine verdict: OK / Warning / Block / RequiresVex │ │
│ │ │ │
│ │ Output: FacetDriftReport with per-facet FacetDrift entries │ │
│ │ │ │
│ └─────────────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────────────────────┐ │
│ │ Quota Configuration Example │ │
│ │ │ │
│ │ facet_quotas: │ │
│ │ os-packages-dpkg: │ │
│ │ max_churn_percent: 15 │ │
│ │ max_changed_files: 100 │ │
│ │ action: warn │ │
│ │ allowlist: │ │
│ │ - "/var/lib/dpkg/status" # Expected to change │ │
│ │ │ │
│ │ binaries-usr: │ │
│ │ max_churn_percent: 5 │ │
│ │ max_changed_files: 10 │ │
│ │ action: block # Binaries shouldn't change unexpectedly │ │
│ │ │ │
│ │ lang-deps-npm: │ │
│ │ max_churn_percent: 25 │ │
│ │ max_changed_files: 200 │ │
│ │ action: require_vex │ │
│ │ │ │
│ └─────────────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────────────────────┐ │
│ │ Auto-VEX from Drift │ │
│ │ │ │
│ │ When drift is detected and action = require_vex: │ │
│ │ 1. Generate draft VEX statement with status "under_investigation" │ │
│ │ 2. Include drift context: facet, files changed, churn % │ │
│ │ 3. Queue for human review in VEX workflow │ │
│ │ 4. If approved, drift is "authorized" and quota resets │ │
│ │ │ │
│ └─────────────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────┘
```
---
## Architecture Design
### Facet Drift Engine
```csharp
// src/__Libraries/StellaOps.Facet/Drift/IFacetDriftEngine.cs
namespace StellaOps.Facet.Drift;
/// <summary>
/// Computes drift between current image state and sealed baseline.
/// </summary>
public interface IFacetDriftEngine
{
/// <summary>
/// Compute drift for all facets in a seal.
/// </summary>
Task<FacetDriftReport> ComputeDriftAsync(
FacetSeal baseline,
IImageFileSystem currentImage,
FacetDriftOptions? options = null,
CancellationToken ct = default);
/// <summary>
/// Compute drift for a single facet.
/// </summary>
Task<FacetDrift> ComputeFacetDriftAsync(
FacetEntry baselineEntry,
IImageFileSystem currentImage,
FacetQuota? quota = null,
CancellationToken ct = default);
}
public sealed record FacetDriftReport
{
/// <summary>
/// Image being analyzed.
/// </summary>
public required string ImageDigest { get; init; }
/// <summary>
/// Baseline seal used for comparison.
/// </summary>
public required string BaselineSealId { get; init; }
/// <summary>
/// When drift analysis was performed.
/// </summary>
public required DateTimeOffset AnalyzedAt { get; init; }
/// <summary>
/// Per-facet drift results.
/// </summary>
public required ImmutableArray<FacetDrift> FacetDrifts { get; init; }
/// <summary>
/// Overall verdict (worst of all facets).
/// </summary>
public required QuotaVerdict OverallVerdict { get; init; }
/// <summary>
/// Facets that exceeded quota.
/// </summary>
public ImmutableArray<string> QuotaBreaches =>
FacetDrifts.Where(d => d.QuotaVerdict != QuotaVerdict.Ok)
.Select(d => d.FacetId)
.ToImmutableArray();
/// <summary>
/// Total files changed across all facets.
/// </summary>
public int TotalChangedFiles =>
FacetDrifts.Sum(d => d.Added.Length + d.Removed.Length + d.Modified.Length);
}
public sealed record FacetDriftOptions
{
/// <summary>
/// Custom quota overrides per facet.
/// </summary>
public ImmutableDictionary<string, FacetQuota>? QuotaOverrides { get; init; }
/// <summary>
/// Skip drift computation for these facets.
/// </summary>
public ImmutableArray<string> SkipFacets { get; init; } = [];
/// <summary>
/// Include detailed file lists (slower but useful for debugging).
/// </summary>
public bool IncludeFileDetails { get; init; } = true;
}
```
### Facet Drift Engine Implementation
```csharp
// src/__Libraries/StellaOps.Facet/Drift/FacetDriftEngine.cs
namespace StellaOps.Facet.Drift;
internal sealed class FacetDriftEngine : IFacetDriftEngine
{
private readonly IFacetExtractor _extractor;
private readonly FacetMerkleTree _merkleTree;
private readonly TimeProvider _timeProvider;
private readonly ILogger<FacetDriftEngine> _logger;
public FacetDriftEngine(
IFacetExtractor extractor,
FacetMerkleTree merkleTree,
TimeProvider? timeProvider = null,
ILogger<FacetDriftEngine>? logger = null)
{
_extractor = extractor;
_merkleTree = merkleTree;
_timeProvider = timeProvider ?? TimeProvider.System;
_logger = logger ?? NullLogger<FacetDriftEngine>.Instance;
}
public async Task<FacetDriftReport> ComputeDriftAsync(
FacetSeal baseline,
IImageFileSystem currentImage,
FacetDriftOptions? options = null,
CancellationToken ct = default)
{
options ??= new FacetDriftOptions();
var drifts = new List<FacetDrift>();
var worstVerdict = QuotaVerdict.Ok;
foreach (var facetEntry in baseline.Facets)
{
ct.ThrowIfCancellationRequested();
if (options.SkipFacets.Contains(facetEntry.FacetId))
{
_logger.LogDebug("Skipping facet {FacetId} per options", facetEntry.FacetId);
continue;
}
// Get quota (override or from baseline)
var quota = options.QuotaOverrides?.GetValueOrDefault(facetEntry.FacetId)
?? baseline.Quotas?.GetValueOrDefault(facetEntry.FacetId);
var drift = await ComputeFacetDriftAsync(facetEntry, currentImage, quota, ct)
.ConfigureAwait(false);
drifts.Add(drift);
if (drift.QuotaVerdict > worstVerdict)
{
worstVerdict = drift.QuotaVerdict;
}
}
return new FacetDriftReport
{
ImageDigest = baseline.ImageDigest,
BaselineSealId = baseline.CombinedMerkleRoot,
AnalyzedAt = _timeProvider.GetUtcNow(),
FacetDrifts = [.. drifts],
OverallVerdict = worstVerdict
};
}
public async Task<FacetDrift> ComputeFacetDriftAsync(
FacetEntry baselineEntry,
IImageFileSystem currentImage,
FacetQuota? quota = null,
CancellationToken ct = default)
{
// Get facet definition
var facet = BuiltInFacets.GetById(baselineEntry.FacetId)
?? throw new InvalidOperationException($"Unknown facet: {baselineEntry.FacetId}");
// Extract current files
var extraction = await _extractor.ExtractAsync(facet, currentImage, ct: ct)
.ConfigureAwait(false);
// Build lookup maps
var baselineFiles = baselineEntry.Files?.ToDictionary(f => f.Path) ?? new();
var currentFiles = extraction.Files.ToDictionary(f => f.Path);
// Compute diff
var added = new List<FacetFileEntry>();
var removed = new List<FacetFileEntry>();
var modified = new List<FacetFileModification>();
// Files in current but not baseline = added
foreach (var file in extraction.Files)
{
if (!baselineFiles.ContainsKey(file.Path))
{
added.Add(file);
}
}
// Files in baseline
foreach (var (path, baselineFile) in baselineFiles)
{
if (!currentFiles.TryGetValue(path, out var currentFile))
{
// In baseline but not current = removed
removed.Add(baselineFile);
}
else if (baselineFile.Digest != currentFile.Digest)
{
// In both but different = modified
modified.Add(new FacetFileModification(
path,
baselineFile.Digest,
currentFile.Digest,
baselineFile.SizeBytes,
currentFile.SizeBytes));
}
}
// Apply allowlist filtering
if (quota?.AllowlistGlobs.Length > 0)
{
var allowedPaths = new HashSet<string>();
foreach (var glob in quota.AllowlistGlobs)
{
// Simple glob matching (would use DotNet.Glob in production)
allowedPaths.UnionWith(FilterByGlob(
added.Select(f => f.Path)
.Concat(removed.Select(f => f.Path))
.Concat(modified.Select(m => m.Path)),
glob));
}
added = added.Where(f => !allowedPaths.Contains(f.Path)).ToList();
removed = removed.Where(f => !allowedPaths.Contains(f.Path)).ToList();
modified = modified.Where(m => !allowedPaths.Contains(m.Path)).ToList();
}
// Calculate metrics
var totalChanges = added.Count + removed.Count + modified.Count;
var churnPercent = baselineEntry.FileCount > 0
? totalChanges / (decimal)baselineEntry.FileCount * 100
: 0;
var driftScore = ComputeDriftScore(added.Count, removed.Count, modified.Count, churnPercent);
// Evaluate quota
var verdict = EvaluateQuota(quota, churnPercent, totalChanges);
return new FacetDrift
{
FacetId = baselineEntry.FacetId,
Added = [.. added],
Removed = [.. removed],
Modified = [.. modified],
DriftScore = driftScore,
QuotaVerdict = verdict,
BaselineFileCount = baselineEntry.FileCount
};
}
private static decimal ComputeDriftScore(int added, int removed, int modified, decimal churnPercent)
{
// Weighted score: removals and modifications are more significant than additions
const decimal addWeight = 1.0m;
const decimal removeWeight = 2.0m;
const decimal modifyWeight = 1.5m;
var weightedChanges = added * addWeight + removed * removeWeight + modified * modifyWeight;
// Normalize to 0-100 scale based on churn
return Math.Min(100, churnPercent + weightedChanges / 10);
}
private static QuotaVerdict EvaluateQuota(FacetQuota? quota, decimal churnPercent, int totalChanges)
{
if (quota is null)
{
return QuotaVerdict.Ok; // No quota = no enforcement
}
var breached = churnPercent > quota.MaxChurnPercent || totalChanges > quota.MaxChangedFiles;
if (!breached)
{
return QuotaVerdict.Ok;
}
return quota.Action switch
{
QuotaExceededAction.Warn => QuotaVerdict.Warning,
QuotaExceededAction.Block => QuotaVerdict.Blocked,
QuotaExceededAction.RequireVex => QuotaVerdict.RequiresVex,
_ => QuotaVerdict.Warning
};
}
private static IEnumerable<string> FilterByGlob(IEnumerable<string> paths, string glob)
{
// Simplified glob matching - use DotNet.Glob in production
var pattern = "^" + Regex.Escape(glob)
.Replace("\\*\\*", ".*")
.Replace("\\*", "[^/]*") + "$";
var regex = new Regex(pattern, RegexOptions.Compiled);
return paths.Where(p => regex.IsMatch(p));
}
}
```
### Quota Enforcer Gate
```csharp
// src/Policy/__Libraries/StellaOps.Policy/Gates/FacetQuotaGate.cs
namespace StellaOps.Policy.Gates;
/// <summary>
/// Policy gate that enforces facet drift quotas.
/// </summary>
public sealed class FacetQuotaGate : IGateEvaluator
{
private readonly IFacetDriftEngine _driftEngine;
private readonly IFacetSealStore _sealStore;
private readonly ILogger<FacetQuotaGate> _logger;
public string GateId => "facet-quota";
public string DisplayName => "Facet Drift Quota";
public int Priority => 50; // After evidence freshness, before budget
public FacetQuotaGate(
IFacetDriftEngine driftEngine,
IFacetSealStore sealStore,
ILogger<FacetQuotaGate>? logger = null)
{
_driftEngine = driftEngine;
_sealStore = sealStore;
_logger = logger ?? NullLogger<FacetQuotaGate>.Instance;
}
public async Task<GateResult> EvaluateAsync(
GateContext context,
CancellationToken ct = default)
{
// Check if facet quota enforcement is enabled
if (!context.PolicyOptions.FacetQuotaEnabled)
{
return GateResult.Pass("Facet quota enforcement disabled");
}
// Load baseline seal
var baseline = await _sealStore.GetLatestSealAsync(context.ImageDigest, ct)
.ConfigureAwait(false);
if (baseline is null)
{
_logger.LogWarning("No baseline seal found for {Image}, skipping quota check",
context.ImageDigest);
return GateResult.Pass("No baseline seal - quota check skipped");
}
// Compute drift
var driftReport = await _driftEngine.ComputeDriftAsync(
baseline,
context.ImageFileSystem,
ct: ct).ConfigureAwait(false);
// Evaluate result
return driftReport.OverallVerdict switch
{
QuotaVerdict.Ok => GateResult.Pass(
$"Facet quotas OK: {driftReport.TotalChangedFiles} files changed"),
QuotaVerdict.Warning => GateResult.Warn(
$"Facet quota warning: {FormatBreaches(driftReport)}",
new GateWarning("facet.quota.warning", driftReport.QuotaBreaches)),
QuotaVerdict.Blocked => GateResult.Block(
$"Facet quota BLOCKED: {FormatBreaches(driftReport)}",
BlockReason.QuotaExceeded),
QuotaVerdict.RequiresVex => GateResult.RequiresAction(
$"Facet drift requires VEX: {FormatBreaches(driftReport)}",
RequiredAction.SubmitVex,
GenerateVexContext(driftReport)),
_ => GateResult.Pass("Unknown verdict - defaulting to pass")
};
}
private static string FormatBreaches(FacetDriftReport report)
{
return string.Join(", ", report.FacetDrifts
.Where(d => d.QuotaVerdict != QuotaVerdict.Ok)
.Select(d => $"{d.FacetId}({d.ChurnPercent:F1}%)"));
}
private static VexContext GenerateVexContext(FacetDriftReport report)
{
return new VexContext
{
ContextType = "facet-drift",
ArtifactDigest = report.ImageDigest,
FacetBreaches = report.QuotaBreaches.ToList(),
TotalChangedFiles = report.TotalChangedFiles,
AnalyzedAt = report.AnalyzedAt
};
}
}
```
### Auto-VEX from Drift
```csharp
// src/__Libraries/StellaOps.Facet/Vex/FacetDriftVexEmitter.cs
namespace StellaOps.Facet.Vex;
/// <summary>
/// Generates draft VEX statements from facet drift requiring authorization.
/// </summary>
public sealed class FacetDriftVexEmitter
{
private readonly IVexDraftStore _draftStore;
private readonly TimeProvider _timeProvider;
private readonly IGuidProvider _guidProvider;
public FacetDriftVexEmitter(
IVexDraftStore draftStore,
TimeProvider? timeProvider = null,
IGuidProvider? guidProvider = null)
{
_draftStore = draftStore;
_timeProvider = timeProvider ?? TimeProvider.System;
_guidProvider = guidProvider ?? SystemGuidProvider.Instance;
}
/// <summary>
/// Generate draft VEX statements for facets requiring authorization.
/// </summary>
public async Task<ImmutableArray<VexDraft>> GenerateDraftsAsync(
FacetDriftReport report,
CancellationToken ct = default)
{
var drafts = new List<VexDraft>();
foreach (var drift in report.FacetDrifts.Where(d => d.QuotaVerdict == QuotaVerdict.RequiresVex))
{
var draft = CreateDraft(report, drift);
await _draftStore.SaveAsync(draft, ct).ConfigureAwait(false);
drafts.Add(draft);
}
return [.. drafts];
}
private VexDraft CreateDraft(FacetDriftReport report, FacetDrift drift)
{
var now = _timeProvider.GetUtcNow();
return new VexDraft
{
DraftId = $"vex-draft:{_guidProvider.NewGuid()}",
Status = VexStatus.UnderInvestigation,
Category = "facet-drift-authorization",
CreatedAt = now,
ArtifactDigest = report.ImageDigest,
FacetId = drift.FacetId,
Justification = GenerateJustification(drift),
Context = new VexDraftContext
{
BaselineSealId = report.BaselineSealId,
ChurnPercent = drift.ChurnPercent,
FilesAdded = drift.Added.Length,
FilesRemoved = drift.Removed.Length,
FilesModified = drift.Modified.Length,
SampleChanges = GetSampleChanges(drift, maxSamples: 10)
},
RequiresReview = true,
ReviewDeadline = now.AddDays(7) // 7-day SLA for drift review
};
}
private static string GenerateJustification(FacetDrift drift)
{
var sb = new StringBuilder();
sb.AppendLine($"Facet '{drift.FacetId}' exceeded drift quota.");
sb.AppendLine($"Churn: {drift.ChurnPercent:F2}% ({drift.Added.Length} added, {drift.Removed.Length} removed, {drift.Modified.Length} modified)");
sb.AppendLine();
sb.AppendLine("Review required to authorize this drift. Possible reasons:");
sb.AppendLine("- Planned security patch deployment");
sb.AppendLine("- Dependency update");
sb.AppendLine("- Build reproducibility variance");
sb.AppendLine("- Unauthorized modification (investigate)");
return sb.ToString();
}
private static ImmutableArray<string> GetSampleChanges(FacetDrift drift, int maxSamples)
{
var samples = new List<string>();
foreach (var added in drift.Added.Take(maxSamples / 3))
samples.Add($"+ {added.Path}");
foreach (var removed in drift.Removed.Take(maxSamples / 3))
samples.Add($"- {removed.Path}");
foreach (var modified in drift.Modified.Take(maxSamples / 3))
samples.Add($"~ {modified.Path}");
return [.. samples];
}
}
```
---
## Delivery Tracker
| # | Task ID | Status | Dependency | Owners | Task Definition |
|---|---------|--------|------------|--------|-----------------|
| **Drift Engine** |
| 1 | QTA-001 | TODO | FCT models | Facet Guild | Define `IFacetDriftEngine` interface |
| 2 | QTA-002 | TODO | QTA-001 | Facet Guild | Define `FacetDriftReport` model |
| 3 | QTA-003 | TODO | QTA-002 | Facet Guild | Implement file diff computation (added/removed/modified) |
| 4 | QTA-004 | TODO | QTA-003 | Facet Guild | Implement allowlist glob filtering |
| 5 | QTA-005 | TODO | QTA-004 | Facet Guild | Implement drift score calculation |
| 6 | QTA-006 | TODO | QTA-005 | Facet Guild | Implement quota evaluation logic |
| 7 | QTA-007 | TODO | QTA-006 | Facet Guild | Unit tests: Drift computation with fixtures |
| 8 | QTA-008 | TODO | QTA-007 | Facet Guild | Unit tests: Quota evaluation edge cases |
| **Quota Enforcement** |
| 9 | QTA-009 | TODO | QTA-006 | Policy Guild | Create `FacetQuotaGate` class |
| 10 | QTA-010 | TODO | QTA-009 | Policy Guild | Integrate with `IGateEvaluator` pipeline |
| 11 | QTA-011 | TODO | QTA-010 | Policy Guild | Add `FacetQuotaEnabled` to policy options |
| 12 | QTA-012 | TODO | QTA-011 | Policy Guild | Create `IFacetSealStore` for baseline lookups |
| 13 | QTA-013 | TODO | QTA-012 | Policy Guild | Implement Postgres storage for facet seals |
| 14 | QTA-014 | TODO | QTA-013 | Policy Guild | Unit tests: Gate evaluation scenarios |
| 15 | QTA-015 | TODO | QTA-014 | Policy Guild | Integration tests: Full gate pipeline |
| **Auto-VEX Generation** |
| 16 | QTA-016 | TODO | QTA-006 | VEX Guild | Create `FacetDriftVexEmitter` class |
| 17 | QTA-017 | TODO | QTA-016 | VEX Guild | Define `VexDraft` and `VexDraftContext` models |
| 18 | QTA-018 | TODO | QTA-017 | VEX Guild | Implement draft storage and retrieval |
| 19 | QTA-019 | TODO | QTA-018 | VEX Guild | Wire into Excititor VEX workflow |
| 20 | QTA-020 | TODO | QTA-019 | VEX Guild | Unit tests: Draft generation and justification |
| **Configuration & Documentation** |
| 21 | QTA-021 | TODO | QTA-015 | Ops Guild | Create facet quota YAML schema |
| 22 | QTA-022 | TODO | QTA-021 | Ops Guild | Add default quota profiles (strict, moderate, permissive) |
| 23 | QTA-023 | TODO | QTA-022 | Docs Guild | Document quota configuration in ops guide |
| 24 | QTA-024 | TODO | QTA-023 | QA Guild | E2E test: Quota breach → VEX draft → approval |
---
## Success Metrics
| Metric | Before | After | Target |
|--------|--------|-------|--------|
| Per-facet drift tracking | No | Yes | All facets |
| Quota enforcement per facet | No | Yes | Configurable |
| BLOCK action on quota breach | No | Yes | Functional |
| Auto-VEX draft from drift | No | Yes | Queued for review |
| 50% fewer noisy regressions | Baseline | Measured | 50% reduction |
---
## Execution Log
| Date (UTC) | Update | Owner |
|------------|--------|-------|
| 2026-01-05 | Sprint created from product advisory gap analysis | Planning |
---
## Decisions & Risks
| Decision/Risk | Type | Mitigation |
|---------------|------|------------|
| Drift computation adds latency | Trade-off | Make optional, cache baselines |
| Allowlist globs may be too broad | Risk | Document best practices, provide linting |
| VEX draft SLA enforcement | Decision | 7-day default, configurable per tenant |
| Storage growth from drift history | Risk | Retention policy, aggregate old data |
---
## Next Checkpoints
- QTA-001 through QTA-008 (drift engine) target completion
- QTA-009 through QTA-015 (enforcement) target completion
- QTA-016 through QTA-020 (auto-VEX) target completion
- QTA-024 (E2E) sprint completion gate

View File

@@ -0,0 +1,444 @@
# Sprint 20260105_002_003_ROUTER - HLC: Offline Merge Protocol
## Topic & Scope
Implement HLC-based deterministic merge protocol for offline/air-gap scenarios. When disconnected nodes sync, jobs must merge by HLC order key to maintain global ordering without conflicts.
- **Working directory:** `src/Router/`, `src/AirGap/`
- **Evidence:** Merge algorithm implementation, conflict resolution tests, air-gap sync integration
## Problem Statement
Current air-gap handling:
- Staleness validation gates job scheduling
- No deterministic merge protocol for offline-enqueued jobs
- Wall-clock based ordering causes drift on reconnection
Advisory requires:
- Enqueue locally with HLC rules and chain links
- On sync, merge by order key `(T_hlc, PartitionKey?, JobId)`
- Merges are conflict-free because keys are deterministic
## Dependencies & Concurrency
- **Depends on:** SPRINT_20260105_002_002_SCHEDULER (HLC queue chain)
- **Blocks:** SPRINT_20260105_002_004_BE (integration tests)
- **Parallel safe:** Router/AirGap changes isolated from other modules
## Documentation Prerequisites
- docs/modules/router/architecture.md
- docs/airgap/OFFLINE_KIT.md
- src/AirGap/AGENTS.md
- Product Advisory: offline + replay section
## Technical Design
### Offline HLC Persistence
When operating in air-gap/offline mode, each node maintains its own HLC state:
```csharp
public sealed class OfflineHlcManager
{
private readonly IHybridLogicalClock _hlc;
private readonly IOfflineJobLogStore _jobLogStore;
private readonly string _nodeId;
public OfflineHlcManager(
IHybridLogicalClock hlc,
IOfflineJobLogStore jobLogStore,
string nodeId)
{
_hlc = hlc;
_jobLogStore = jobLogStore;
_nodeId = nodeId;
}
/// <summary>
/// Enqueue job locally while offline. Maintains local chain.
/// </summary>
public async Task<OfflineEnqueueResult> EnqueueOfflineAsync(
SchedulerJobPayload payload,
CancellationToken ct = default)
{
var tHlc = _hlc.Tick();
var jobId = ComputeDeterministicJobId(payload);
var payloadHash = SchedulerChainLinking.ComputePayloadHash(payload);
var prevLink = await _jobLogStore.GetLastLinkAsync(_nodeId, ct);
var link = SchedulerChainLinking.ComputeLink(prevLink, jobId, tHlc, payloadHash);
var entry = new OfflineJobLogEntry
{
NodeId = _nodeId,
THlc = tHlc,
JobId = jobId,
Payload = payload,
PayloadHash = payloadHash,
PrevLink = prevLink,
Link = link,
EnqueuedAt = DateTimeOffset.UtcNow
};
await _jobLogStore.AppendAsync(entry, ct);
return new OfflineEnqueueResult(tHlc, jobId, link, _nodeId);
}
}
```
### Merge Algorithm
The merge algorithm combines job logs from multiple nodes while preserving HLC ordering:
```csharp
public sealed class HlcMergeService
{
/// <summary>
/// Merge job logs from multiple offline nodes into unified, HLC-ordered stream.
/// </summary>
public async Task<MergeResult> MergeAsync(
IReadOnlyList<NodeJobLog> nodeLogs,
CancellationToken ct = default)
{
// 1. Collect all entries from all nodes
var allEntries = nodeLogs
.SelectMany(log => log.Entries.Select(e => (log.NodeId, Entry: e)))
.ToList();
// 2. Sort by HLC total order: (PhysicalTime, LogicalCounter, NodeId, JobId)
var sorted = allEntries
.OrderBy(x => x.Entry.THlc.PhysicalTime)
.ThenBy(x => x.Entry.THlc.LogicalCounter)
.ThenBy(x => x.Entry.THlc.NodeId, StringComparer.Ordinal)
.ThenBy(x => x.Entry.JobId)
.ToList();
// 3. Detect duplicates (same JobId = same deterministic payload)
var seen = new HashSet<Guid>();
var deduplicated = new List<MergedJobEntry>();
var duplicates = new List<DuplicateEntry>();
foreach (var (nodeId, entry) in sorted)
{
if (seen.Contains(entry.JobId))
{
duplicates.Add(new DuplicateEntry(entry.JobId, nodeId, entry.THlc));
continue;
}
seen.Add(entry.JobId);
deduplicated.Add(new MergedJobEntry
{
SourceNodeId = nodeId,
THlc = entry.THlc,
JobId = entry.JobId,
Payload = entry.Payload,
PayloadHash = entry.PayloadHash,
OriginalLink = entry.Link
});
}
// 4. Recompute unified chain
byte[]? prevLink = null;
foreach (var entry in deduplicated)
{
entry.MergedLink = SchedulerChainLinking.ComputeLink(
prevLink,
entry.JobId,
entry.THlc,
entry.PayloadHash);
prevLink = entry.MergedLink;
}
return new MergeResult
{
MergedEntries = deduplicated,
Duplicates = duplicates,
MergedChainHead = prevLink,
SourceNodes = nodeLogs.Select(l => l.NodeId).ToList()
};
}
}
```
### Sync Protocol
```csharp
public sealed class AirGapSyncService
{
private readonly IHlcMergeService _mergeService;
private readonly ISchedulerLogRepository _schedulerLogRepo;
private readonly IHybridLogicalClock _hlc;
/// <summary>
/// Sync offline jobs from air-gap bundle to central scheduler.
/// </summary>
public async Task<SyncResult> SyncFromBundleAsync(
AirGapBundle bundle,
CancellationToken ct = default)
{
var nodeLogs = bundle.JobLogs;
// 1. Merge all offline logs
var merged = await _mergeService.MergeAsync(nodeLogs, ct);
// 2. Get current scheduler chain head
var currentHead = await _schedulerLogRepo.GetChainHeadAsync(
bundle.TenantId,
ct);
// 3. For each merged entry, update HLC clock (receive)
// This ensures central clock advances past all offline timestamps
foreach (var entry in merged.MergedEntries)
{
_hlc.Receive(entry.THlc);
}
// 4. Append merged entries to scheduler log
// Chain links recomputed to extend from current head
byte[]? prevLink = currentHead?.Link;
var appended = new List<SchedulerLogEntry>();
foreach (var entry in merged.MergedEntries)
{
// Check if job already exists (idempotency)
var existing = await _schedulerLogRepo.GetByJobIdAsync(
bundle.TenantId,
entry.JobId,
ct);
if (existing is not null)
{
continue; // Already synced
}
var newLink = SchedulerChainLinking.ComputeLink(
prevLink,
entry.JobId,
entry.THlc,
entry.PayloadHash);
var logEntry = new SchedulerLogEntry
{
TenantId = bundle.TenantId,
THlc = entry.THlc.ToSortableString(),
PartitionKey = entry.Payload.PartitionKey,
JobId = entry.JobId,
PayloadHash = entry.PayloadHash,
PrevLink = prevLink,
Link = newLink,
SourceNodeId = entry.SourceNodeId,
SyncedFromBundle = bundle.BundleId
};
await _schedulerLogRepo.InsertAsync(logEntry, ct);
appended.Add(logEntry);
prevLink = newLink;
}
return new SyncResult
{
BundleId = bundle.BundleId,
TotalInBundle = merged.MergedEntries.Count,
Appended = appended.Count,
Duplicates = merged.Duplicates.Count,
NewChainHead = prevLink
};
}
}
```
### Air-Gap Bundle Format
```csharp
public sealed record AirGapBundle
{
public required Guid BundleId { get; init; }
public required string TenantId { get; init; }
public required DateTimeOffset CreatedAt { get; init; }
public required string CreatedByNodeId { get; init; }
/// <summary>Job logs from each offline node.</summary>
public required IReadOnlyList<NodeJobLog> JobLogs { get; init; }
/// <summary>Bundle manifest digest for integrity.</summary>
public required string ManifestDigest { get; init; }
/// <summary>Optional DSSE signature over manifest.</summary>
public string? Signature { get; init; }
}
public sealed record NodeJobLog
{
public required string NodeId { get; init; }
public required HlcTimestamp LastHlc { get; init; }
public required byte[] ChainHead { get; init; }
public required IReadOnlyList<OfflineJobLogEntry> Entries { get; init; }
}
```
### Conflict Resolution
HLC ensures conflicts are rare, but when they occur:
```csharp
public sealed class ConflictResolver
{
/// <summary>
/// Resolve conflicts when same JobId has different payloads.
/// This should NOT happen with deterministic JobId computation.
/// </summary>
public ConflictResolution Resolve(
Guid jobId,
IReadOnlyList<(string NodeId, OfflineJobLogEntry Entry)> conflicting)
{
// Verify payloads are actually different
var uniquePayloads = conflicting
.Select(c => Convert.ToHexString(c.Entry.PayloadHash))
.Distinct()
.ToList();
if (uniquePayloads.Count == 1)
{
// Same payload, different HLC - not a real conflict
// Take earliest HLC (preserves causality)
var earliest = conflicting
.OrderBy(c => c.Entry.THlc)
.First();
return new ConflictResolution
{
Type = ConflictType.DuplicateTimestamp,
Resolution = ResolutionStrategy.TakeEarliest,
SelectedEntry = earliest.Entry,
DroppedEntries = conflicting
.Where(c => c.Entry != earliest.Entry)
.Select(c => c.Entry)
.ToList()
};
}
// Actual conflict: same JobId, different payloads
// This indicates a bug in deterministic ID computation
return new ConflictResolution
{
Type = ConflictType.PayloadMismatch,
Resolution = ResolutionStrategy.Error,
Error = $"JobId {jobId} has conflicting payloads from nodes: " +
string.Join(", ", conflicting.Select(c => c.NodeId))
};
}
}
```
## Delivery Tracker
| # | Task ID | Status | Dependency | Owner | Task Definition |
|---|---------|--------|------------|-------|-----------------|
| 1 | OMP-001 | TODO | SQC lib | Guild | Create `StellaOps.AirGap.Sync` library project |
| 2 | OMP-002 | TODO | OMP-001 | Guild | Implement `OfflineHlcManager` for local offline enqueue |
| 3 | OMP-003 | TODO | OMP-002 | Guild | Implement `IOfflineJobLogStore` and file-based store |
| 4 | OMP-004 | TODO | OMP-003 | Guild | Implement `HlcMergeService` with total order merge |
| 5 | OMP-005 | TODO | OMP-004 | Guild | Implement `ConflictResolver` for edge cases |
| 6 | OMP-006 | TODO | OMP-005 | Guild | Implement `AirGapSyncService` for bundle import |
| 7 | OMP-007 | TODO | OMP-006 | Guild | Define `AirGapBundle` format (JSON schema) |
| 8 | OMP-008 | TODO | OMP-007 | Guild | Implement bundle export: `AirGapBundleExporter` |
| 9 | OMP-009 | TODO | OMP-008 | Guild | Implement bundle import: `AirGapBundleImporter` |
| 10 | OMP-010 | TODO | OMP-009 | Guild | Add DSSE signing for bundle integrity |
| 11 | OMP-011 | TODO | OMP-006 | Guild | Integrate with Router transport layer |
| 12 | OMP-012 | TODO | OMP-011 | Guild | Update `stella airgap export` CLI command |
| 13 | OMP-013 | TODO | OMP-012 | Guild | Update `stella airgap import` CLI command |
| 14 | OMP-014 | TODO | OMP-004 | Guild | Write unit tests: merge algorithm correctness |
| 15 | OMP-015 | TODO | OMP-014 | Guild | Write unit tests: duplicate detection |
| 16 | OMP-016 | TODO | OMP-015 | Guild | Write unit tests: conflict resolution |
| 17 | OMP-017 | TODO | OMP-016 | Guild | Write integration tests: offline -> online sync |
| 18 | OMP-018 | TODO | OMP-017 | Guild | Write integration tests: multi-node merge |
| 19 | OMP-019 | TODO | OMP-018 | Guild | Write determinism tests: same bundles -> same result |
| 20 | OMP-020 | TODO | OMP-019 | Guild | Metrics: `airgap_sync_total`, `airgap_merge_conflicts_total` |
| 21 | OMP-021 | TODO | OMP-020 | Guild | Documentation: offline operations guide |
## Test Scenarios
### Scenario 1: Simple Two-Node Merge
```
Node A (offline): Node B (offline):
Job1 @ T=100 Job2 @ T=101
Job3 @ T=102 Job4 @ T=103
After merge (HLC order):
Job1 @ T=100 (from A)
Job2 @ T=101 (from B)
Job3 @ T=102 (from A)
Job4 @ T=103 (from B)
```
### Scenario 2: Same Payload, Different Nodes
```
Node A: Job(payload=X) @ T=100 -> JobId=abc123
Node B: Job(payload=X) @ T=105 -> JobId=abc123
Result: Single entry with T=100 (earliest), duplicate at T=105 dropped
```
### Scenario 3: Clock Skew During Offline
```
Node A (clock +5min): Job1 @ T=300 (actually T=0)
Node B (clock correct): Job2 @ T=100
After merge with HLC receive():
Central clock advances to max(local, 300)
Order: Job2 @ T=100, Job1 @ T=300 (logical order preserved)
```
## Metrics & Observability
```
# Counters
airgap_bundles_exported_total{node_id}
airgap_bundles_imported_total{node_id}
airgap_jobs_synced_total{node_id}
airgap_duplicates_dropped_total{node_id}
airgap_merge_conflicts_total{conflict_type}
# Histograms
airgap_bundle_size_bytes{node_id}
airgap_sync_duration_seconds{node_id}
airgap_merge_entries_count{node_id}
# Gauges
airgap_pending_sync_bundles{node_id}
airgap_last_sync_timestamp{node_id}
```
## Decisions & Risks
| Decision | Rationale |
|----------|-----------|
| Merge sorts by full HLC tuple | Ensures total order even with identical physical time |
| Recompute chain on merge | Central chain must be contiguous; original links preserved for audit |
| Store source node ID | Traceability for sync origin |
| Error on payload mismatch | Same JobId must have same payload (determinism invariant) |
| Risk | Mitigation |
|------|------------|
| Large bundle sizes | Compression; chunked sync; incremental bundles |
| Clock skew exceeds HLC tolerance | Pre-sync clock validation; NTP enforcement |
| Merge performance with many nodes | Parallel sort; streaming merge; batch processing |
| Bundle corruption during transfer | DSSE signature; checksum validation |
## Execution Log
| Date (UTC) | Update | Owner |
|------------|--------|-------|
| 2026-01-05 | Sprint created from product advisory gap analysis | Planning |
## Next Checkpoints
- 2026-01-12: OMP-001 to OMP-007 complete (core merge)
- 2026-01-13: OMP-008 to OMP-013 complete (bundle + CLI)
- 2026-01-14: OMP-014 to OMP-021 complete (tests, docs)

View File

@@ -0,0 +1,515 @@
# Sprint 20260105_002_004_BE - HLC: Cross-Module Integration & Testing
## Topic & Scope
Comprehensive integration testing and cross-module wiring for the HLC-based audit-safe job queue ordering implementation. Ensures end-to-end determinism from job enqueue through verdict replay.
- **Working directory:** `src/__Tests/Integration/`, cross-module
- **Evidence:** Integration test suite, E2E tests, performance benchmarks, documentation
## Problem Statement
Individual HLC components (library, scheduler chain, offline merge) must work together seamlessly:
- HLC timestamps must flow through Scheduler -> Timeline -> Ledgers
- Chain links must be verifiable across module boundaries
- Batch snapshots must integrate with Attestor for DSSE signing
- Replay must produce identical results with HLC-ordered inputs
## Dependencies & Concurrency
- **Depends on:** All previous sprints (002_001, 002_002, 002_003)
- **Blocks:** Production rollout
- **Parallel safe:** Test development can proceed once interfaces are defined
## Documentation Prerequisites
- All previous sprint documentation
- docs/modules/attestor/proof-chain-specification.md
- docs/modules/replay/architecture.md
- src/__Tests/AGENTS.md
## Technical Design
### Cross-Module HLC Flow
```
┌─────────────┐ HLC Tick ┌─────────────┐ Chain Link ┌─────────────┐
│ Client │ ───────────────▶│ Scheduler │ ─────────────────▶│ Scheduler │
│ Request │ │ Queue │ │ Log │
└─────────────┘ └─────────────┘ └─────────────┘
│ │
│ Job Execution │
▼ │
┌─────────────┐ │
│ Orchestrator│ │
│ Job │ │
└─────────────┘ │
│ │
│ Evidence │
▼ │
┌─────────────┐ Batch Snap ┌─────────────┐ DSSE Sign ┌─────────────┐
│ Replay │ ◀──────────────│ Findings │ ◀────────────────│ Attestor │
│ Engine │ │ Ledger │ │ │
└─────────────┘ └─────────────┘ └─────────────┘
```
### Integration Test Categories
#### 1. HLC Propagation Tests
```csharp
[Trait("Category", "Integration")]
[Trait("Category", "HLC")]
public sealed class HlcPropagationTests : IClassFixture<HlcTestFixture>
{
[Fact]
public async Task Enqueue_HlcTimestamp_PropagatedToTimeline()
{
// Arrange
var job = CreateTestJob();
// Act
var enqueueResult = await _scheduler.EnqueueAsync(job);
await WaitForTimelineEventAsync(enqueueResult.JobId);
// Assert
var timelineEvent = await _timeline.GetByCorrelationIdAsync(enqueueResult.JobId);
Assert.NotNull(timelineEvent);
Assert.Equal(enqueueResult.THlc.ToSortableString(), timelineEvent.HlcTimestamp);
}
[Fact]
public async Task Enqueue_HlcTimestamp_PropagatedToFindingsLedger()
{
// Arrange
var job = CreateScanJob();
// Act
var enqueueResult = await _scheduler.EnqueueAsync(job);
await WaitForJobCompletionAsync(enqueueResult.JobId);
// Assert
var ledgerEvent = await _findingsLedger.GetBySourceRunIdAsync(enqueueResult.JobId);
Assert.NotNull(ledgerEvent);
Assert.True(HlcTimestamp.Parse(ledgerEvent.HlcTimestamp) >= enqueueResult.THlc);
}
}
```
#### 2. Chain Integrity Tests
```csharp
[Trait("Category", "Integration")]
[Trait("Category", "HLC")]
public sealed class ChainIntegrityTests : IClassFixture<HlcTestFixture>
{
[Fact]
public async Task EnqueueMultiple_ChainLinksValid()
{
// Arrange
var jobs = Enumerable.Range(0, 100).Select(i => CreateTestJob(i)).ToList();
// Act
foreach (var job in jobs)
{
await _scheduler.EnqueueAsync(job);
}
// Assert
var verificationResult = await _scheduler.VerifyChainIntegrityAsync(_tenantId);
Assert.True(verificationResult.IsValid);
Assert.Equal(100, verificationResult.EntriesChecked);
Assert.Empty(verificationResult.Issues);
}
[Fact]
public async Task ChainVerification_DetectsTampering()
{
// Arrange
var jobs = Enumerable.Range(0, 10).Select(i => CreateTestJob(i)).ToList();
foreach (var job in jobs)
{
await _scheduler.EnqueueAsync(job);
}
// Act - Tamper with middle entry
await TamperWithSchedulerLogEntryAsync(jobs[5].Id);
// Assert
var verificationResult = await _scheduler.VerifyChainIntegrityAsync(_tenantId);
Assert.False(verificationResult.IsValid);
Assert.Contains(verificationResult.Issues, i => i.JobId == jobs[5].Id);
}
}
```
#### 3. Batch Snapshot Integration
```csharp
[Trait("Category", "Integration")]
[Trait("Category", "HLC")]
public sealed class BatchSnapshotIntegrationTests : IClassFixture<HlcTestFixture>
{
[Fact]
public async Task CreateSnapshot_SignedByAttestor()
{
// Arrange
var jobs = await EnqueueMultipleJobsAsync(50);
var startT = jobs.First().THlc;
var endT = jobs.Last().THlc;
// Act
var snapshot = await _batchService.CreateSnapshotAsync(_tenantId, startT, endT);
// Assert
Assert.NotNull(snapshot.Signature);
Assert.NotNull(snapshot.SignedBy);
var verified = await _attestor.VerifySnapshotSignatureAsync(snapshot);
Assert.True(verified);
}
[Fact]
public async Task Snapshot_HeadLinkMatchesChain()
{
// Arrange
var jobs = await EnqueueMultipleJobsAsync(25);
// Act
var snapshot = await _batchService.CreateSnapshotAsync(
_tenantId,
jobs.First().THlc,
jobs.Last().THlc);
// Assert
var chainHead = await _schedulerLog.GetChainHeadAsync(_tenantId);
Assert.Equal(chainHead.Link, snapshot.HeadLink);
}
}
```
#### 4. Offline Sync Integration
```csharp
[Trait("Category", "Integration")]
[Trait("Category", "HLC")]
[Trait("Category", "AirGap")]
public sealed class OfflineSyncIntegrationTests : IClassFixture<AirGapTestFixture>
{
[Fact]
public async Task OfflineEnqueue_SyncsWithCorrectOrder()
{
// Arrange - Simulate two offline nodes
var nodeA = CreateOfflineNode("node-a");
var nodeB = CreateOfflineNode("node-b");
// Enqueue interleaved jobs
await nodeA.EnqueueAsync(CreateJob("A1")); // T=100
await nodeB.EnqueueAsync(CreateJob("B1")); // T=101
await nodeA.EnqueueAsync(CreateJob("A2")); // T=102
// Act - Export and sync
var bundleA = await nodeA.ExportBundleAsync();
var bundleB = await nodeB.ExportBundleAsync();
var syncResult = await _syncService.SyncFromBundlesAsync([bundleA, bundleB]);
// Assert - Merged in HLC order
var merged = await _schedulerLog.GetByHlcOrderAsync(_tenantId, limit: 10);
Assert.Equal(3, merged.Count);
Assert.Equal("A1", merged[0].JobName); // T=100
Assert.Equal("B1", merged[1].JobName); // T=101
Assert.Equal("A2", merged[2].JobName); // T=102
}
[Fact]
public async Task OfflineSync_DeduplicatesSamePayload()
{
// Arrange - Same job enqueued on two nodes
var nodeA = CreateOfflineNode("node-a");
var nodeB = CreateOfflineNode("node-b");
var samePayload = CreateJob("shared");
await nodeA.EnqueueAsync(samePayload);
await nodeB.EnqueueAsync(samePayload); // Same payload = same JobId
// Act
var bundleA = await nodeA.ExportBundleAsync();
var bundleB = await nodeB.ExportBundleAsync();
var syncResult = await _syncService.SyncFromBundlesAsync([bundleA, bundleB]);
// Assert
Assert.Equal(1, syncResult.Appended);
Assert.Equal(1, syncResult.Duplicates);
}
}
```
#### 5. Replay Determinism Tests
```csharp
[Trait("Category", "Integration")]
[Trait("Category", "HLC")]
[Trait("Category", "Replay")]
public sealed class HlcReplayDeterminismTests : IClassFixture<ReplayTestFixture>
{
[Fact]
public async Task Replay_SameHlcOrder_SameResults()
{
// Arrange
var jobs = await EnqueueAndExecuteJobsAsync(20);
var snapshot = await _batchService.CreateSnapshotAsync(
_tenantId,
jobs.First().THlc,
jobs.Last().THlc);
var originalResults = await GetJobResultsAsync(jobs);
// Act - Replay with same HLC-ordered inputs
var replayResults = await _replayEngine.ReplayFromSnapshotAsync(snapshot);
// Assert
Assert.Equal(originalResults.Count, replayResults.Count);
for (int i = 0; i < originalResults.Count; i++)
{
Assert.Equal(originalResults[i].VerdictDigest, replayResults[i].VerdictDigest);
}
}
[Fact]
public async Task Replay_HlcOrderPreserved_AcrossRestarts()
{
// Arrange
var jobs = await EnqueueJobsAsync(10);
var hlcOrder = jobs.Select(j => j.THlc.ToSortableString()).ToList();
// Act - Simulate restart
await RestartSchedulerServiceAsync();
var recoveredJobs = await _schedulerLog.GetByHlcOrderAsync(_tenantId, limit: 10);
// Assert
var recoveredOrder = recoveredJobs.Select(j => j.THlc).ToList();
Assert.Equal(hlcOrder, recoveredOrder);
}
}
```
## Delivery Tracker
| # | Task ID | Status | Dependency | Owner | Task Definition |
|---|---------|--------|------------|-------|-----------------|
| 1 | INT-001 | TODO | All sprints | Guild | Create `StellaOps.Integration.HLC` test project |
| 2 | INT-002 | TODO | INT-001 | Guild | Implement `HlcTestFixture` with full stack setup |
| 3 | INT-003 | TODO | INT-002 | Guild | Write HLC propagation tests (Scheduler -> Timeline) |
| 4 | INT-004 | TODO | INT-003 | Guild | Write HLC propagation tests (Scheduler -> Ledger) |
| 5 | INT-005 | TODO | INT-004 | Guild | Write chain integrity tests (valid chain) |
| 6 | INT-006 | TODO | INT-005 | Guild | Write chain integrity tests (tampering detection) |
| 7 | INT-007 | TODO | INT-006 | Guild | Write batch snapshot + Attestor integration tests |
| 8 | INT-008 | TODO | INT-007 | Guild | Create `AirGapTestFixture` for offline simulation |
| 9 | INT-009 | TODO | INT-008 | Guild | Write offline sync integration tests (order) |
| 10 | INT-010 | TODO | INT-009 | Guild | Write offline sync integration tests (dedup) |
| 11 | INT-011 | TODO | INT-010 | Guild | Write offline sync integration tests (multi-node) |
| 12 | INT-012 | TODO | INT-011 | Guild | Write replay determinism tests |
| 13 | INT-013 | TODO | INT-012 | Guild | Write E2E test: full job lifecycle with HLC |
| 14 | INT-014 | TODO | INT-013 | Guild | Write performance benchmarks: HLC tick throughput |
| 15 | INT-015 | TODO | INT-014 | Guild | Write performance benchmarks: chain verification |
| 16 | INT-016 | TODO | INT-015 | Guild | Write performance benchmarks: offline merge |
| 17 | INT-017 | TODO | INT-016 | Guild | Create Grafana dashboard for HLC metrics |
| 18 | INT-018 | TODO | INT-017 | Guild | Create alerts for HLC anomalies |
| 19 | INT-019 | TODO | INT-018 | Guild | Update Architecture documentation |
| 20 | INT-020 | TODO | INT-019 | Guild | Create Operations runbook for HLC |
| 21 | INT-021 | TODO | INT-020 | Guild | Create Migration guide for existing deployments |
| 22 | INT-022 | TODO | INT-021 | Guild | Final review and sign-off |
## Performance Benchmarks
### Benchmark Targets
| Metric | Target | Rationale |
|--------|--------|-----------|
| HLC tick throughput | > 100K/sec | Support high-volume job queues |
| Chain link computation | < 10us | Minimal overhead per enqueue |
| Chain verification (1K entries) | < 100ms | Fast audit checks |
| Offline merge (10K entries) | < 1s | Reasonable sync time |
| Batch snapshot creation | < 500ms | Interactive batch operations |
### Benchmark Suite
```csharp
[MemoryDiagnoser]
[SimpleJob(RuntimeMoniker.Net100)]
public class HlcBenchmarks
{
private IHybridLogicalClock _hlc = null!;
[GlobalSetup]
public void Setup()
{
_hlc = new HybridLogicalClock(
TimeProvider.System,
"bench-node",
new InMemoryHlcStateStore());
}
[Benchmark]
public HlcTimestamp Tick() => _hlc.Tick();
[Benchmark]
public byte[] ComputeChainLink()
{
return SchedulerChainLinking.ComputeLink(
_prevLink,
_jobId,
_hlc.Tick(),
_payloadHash);
}
[Benchmark]
[Arguments(100)]
[Arguments(1000)]
[Arguments(10000)]
public async Task VerifyChain(int entries)
{
await _verifier.VerifyAsync(_tenantId, entries);
}
}
```
## Observability Integration
### Grafana Dashboard Panels
```json
{
"panels": [
{
"title": "HLC Ticks per Second",
"expr": "rate(hlc_ticks_total[1m])"
},
{
"title": "HLC Clock Skew Rejections",
"expr": "rate(hlc_clock_skew_rejections_total[5m])"
},
{
"title": "Scheduler Chain Verifications",
"expr": "rate(scheduler_chain_verifications_total[5m])"
},
{
"title": "Chain Verification Failures",
"expr": "scheduler_chain_verification_failures_total"
},
{
"title": "AirGap Sync Duration P99",
"expr": "histogram_quantile(0.99, airgap_sync_duration_seconds_bucket)"
},
{
"title": "Batch Snapshots Created",
"expr": "rate(scheduler_batch_snapshots_total[1h])"
}
]
}
```
### Alerts
```yaml
groups:
- name: hlc-alerts
rules:
- alert: HlcClockSkewExcessive
expr: rate(hlc_clock_skew_rejections_total[5m]) > 0
for: 1m
labels:
severity: warning
annotations:
summary: "HLC clock skew rejections detected"
description: "Node {{ $labels.node_id }} is rejecting timestamps due to clock skew"
- alert: SchedulerChainCorruption
expr: scheduler_chain_verification_failures_total > 0
for: 0m
labels:
severity: critical
annotations:
summary: "Scheduler chain corruption detected"
description: "Chain verification failed for tenant {{ $labels.tenant_id }}"
- alert: AirGapSyncBacklog
expr: airgap_pending_sync_bundles > 10
for: 10m
labels:
severity: warning
annotations:
summary: "AirGap sync backlog growing"
```
## Documentation Updates
### Files to Update
| File | Changes |
|------|---------|
| `docs/ARCHITECTURE_REFERENCE.md` | Add HLC section |
| `docs/modules/scheduler/architecture.md` | Document HLC ordering |
| `docs/airgap/OFFLINE_KIT.md` | Add HLC merge protocol |
| `docs/observability/observability.md` | Add HLC metrics |
| `docs/operations/runbooks/` | Create `hlc-troubleshooting.md` |
| `CLAUDE.md` | Add HLC guidelines to Section 8 |
### CLAUDE.md Update (Section 8.19)
```markdown
### 8.19) HLC Usage for Audit-Safe Ordering
| Rule | Guidance |
|------|----------|
| **Use HLC for distributed ordering** | When ordering must be deterministic across distributed nodes or offline scenarios, use `IHybridLogicalClock.Tick()` instead of `TimeProvider.GetUtcNow()`. |
```csharp
// BAD - wall-clock ordering, susceptible to skew
var timestamp = _timeProvider.GetUtcNow();
var job = new Job { CreatedAt = timestamp };
// GOOD - HLC ordering, skew-resistant
var hlcTimestamp = _hlc.Tick();
var job = new Job { THlc = hlcTimestamp };
```
```
## Decisions & Risks
| Decision | Rationale |
|----------|-----------|
| Separate integration test project | Isolation from unit tests; different dependencies |
| Testcontainers for Postgres | Realistic integration without mocking |
| Benchmark suite in main repo | Track performance over time; CI integration |
| Grafana dashboard as code | Version-controlled observability |
| Risk | Mitigation |
|------|------------|
| Integration test flakiness | Retry logic; deterministic test data; container health checks |
| Performance regression | Benchmark baselines; CI gates on performance |
| Documentation drift | Doc updates required for PR merge |
## Execution Log
| Date (UTC) | Update | Owner |
|------------|--------|-------|
| 2026-01-05 | Sprint created from product advisory gap analysis | Planning |
## Next Checkpoints
- 2026-01-15: INT-001 to INT-007 complete (basic integration)
- 2026-01-16: INT-008 to INT-013 complete (offline + replay)
- 2026-01-17: INT-014 to INT-022 complete (perf, docs, rollout)
## Final Acceptance Criteria
- [ ] All integration tests pass in CI
- [ ] Chain verification detects 100% of tampering attempts
- [ ] Offline merge produces deterministic results
- [ ] Replay with HLC inputs produces identical outputs
- [ ] Performance benchmarks meet targets
- [ ] Grafana dashboard deployed
- [ ] Alerts configured and tested
- [ ] Documentation complete
- [ ] Migration guide validated on staging

File diff suppressed because it is too large Load Diff