git.stella-ops.org/docs/db/schemas/sync-ledger.md

# Sync Ledger Schema

Sprint: `SPRINT_8200_0014_0001_DB_sync_ledger_schema`
Migration: `008_sync_ledger.sql`
Working Directory: `src/Concelier/__Libraries/StellaOps.Concelier.Storage.Postgres/`

## Overview

The sync ledger schema provides federation cursor tracking for multi-site vulnerability advisory synchronization. It enables air-gapped and distributed deployments to:

1. Track sync state per remote site using monotonic cursors
2. Prevent duplicate bundle imports via hash-based deduplication
3. Enforce per-site policies (source filtering, size limits, signature requirements)

This is critical infrastructure for the Concelier module's federation capabilities.

## Tables

### `vuln.sync_ledger`

Tracks federation sync state per remote site. Each record represents a successfully imported bundle.

| Column | Type | Description |
|--------|------|-------------|
| `id` | UUID | Primary key (auto-generated) |
| `site_id` | TEXT | Remote site identifier (e.g., `site-us-west`, `airgap-dc2`) |
| `cursor` | TEXT | Position marker for incremental sync (ISO8601#sequence format) |
| `bundle_hash` | TEXT | SHA256 hash of imported bundle (for deduplication) |
| `items_count` | INT | Number of advisory items in the bundle |
| `signed_at` | TIMESTAMPTZ | Timestamp when bundle was signed by remote site |
| `imported_at` | TIMESTAMPTZ | Timestamp when bundle was imported locally (auto-generated) |

**Constraints:**
- `uq_sync_ledger_site_cursor`: Unique constraint on `(site_id, cursor)` - prevents duplicate cursor positions
- `uq_sync_ledger_bundle`: Unique constraint on `bundle_hash` - prevents reimporting same bundle

### `vuln.site_policy`

Per-site federation governance policies. Controls what data can be imported from each remote site.

| Column | Type | Description |
|--------|------|-------------|
| `id` | UUID | Primary key (auto-generated) |
| `site_id` | TEXT | Site identifier (unique) |
| `display_name` | TEXT | Human-readable site name |
| `allowed_sources` | TEXT[] | Source allow list (empty = allow all, supports wildcards) |
| `denied_sources` | TEXT[] | Source deny list (takes precedence over allow list) |
| `max_bundle_size_mb` | INT | Maximum bundle size in MB (default: 100) |
| `max_items_per_bundle` | INT | Maximum items per bundle (default: 10000) |
| `require_signature` | BOOLEAN | Require cryptographic signature on bundles (default: TRUE) |
| `allowed_signers` | TEXT[] | Allowed signer key IDs or issuer names |
| `enabled` | BOOLEAN | Policy enabled status (default: TRUE) |
| `created_at` | TIMESTAMPTZ | Record creation time |
| `updated_at` | TIMESTAMPTZ | Last modification time (auto-updated via trigger) |

## Indexes

| Index | Columns | Purpose |
|-------|---------|---------|
| `idx_sync_ledger_site` | `site_id` | Fast site-based lookups |
| `idx_sync_ledger_site_time` | `site_id, signed_at DESC` | Get latest entries per site |
| `idx_site_policy_enabled` | `enabled` (partial, WHERE enabled = TRUE) | Filter active policies |

## Cursor Format

Cursors use a combined timestamp + sequence format for total ordering:

```
2025-01-15T10:30:00.000Z#0042
```

**Components:**
- ISO8601 timestamp with timezone (UTC)
- `#` separator
- 4-digit sequence number (allows multiple bundles per timestamp)

**Ordering rules:**
1. Compare timestamps first
2. If equal, compare sequence numbers
3. Higher values indicate later positions

### CursorFormat Utility

```csharp
// Create cursor
string cursor = CursorFormat.Create(DateTimeOffset.UtcNow, sequence: 0);
// Result: "2025-01-15T10:30:00.0000000+00:00#0000"

// Parse cursor
var (timestamp, sequence) = CursorFormat.Parse(cursor);

// Compare cursors
bool isNewer = CursorFormat.IsAfter(cursor1, cursor2);
```

## Repository Operations

### ISyncLedgerRepository Interface

```csharp
public interface ISyncLedgerRepository
{
    // Ledger CRUD
    Task<SyncLedgerEntity?> GetLatestAsync(string siteId, CancellationToken ct = default);
    Task<IReadOnlyList<SyncLedgerEntity>> GetHistoryAsync(string siteId, int limit = 10, CancellationToken ct = default);
    Task<SyncLedgerEntity?> GetByBundleHashAsync(string bundleHash, CancellationToken ct = default);
    Task<Guid> InsertAsync(SyncLedgerEntity entry, CancellationToken ct = default);

    // Cursor operations
    Task<string?> GetCursorAsync(string siteId, CancellationToken ct = default);
    Task AdvanceCursorAsync(string siteId, string newCursor, string bundleHash,
                            int itemsCount, DateTimeOffset signedAt, CancellationToken ct = default);
    Task<bool> IsCursorConflictAsync(string siteId, string cursor, CancellationToken ct = default);

    // Policy operations
    Task<SitePolicyEntity?> GetPolicyAsync(string siteId, CancellationToken ct = default);
    Task UpsertPolicyAsync(SitePolicyEntity policy, CancellationToken ct = default);
    Task<IReadOnlyList<SitePolicyEntity>> GetAllPoliciesAsync(bool enabledOnly = true, CancellationToken ct = default);

    // Statistics
    Task<SyncStatistics> GetStatisticsAsync(CancellationToken ct = default);
}
```

## Policy Enforcement

The `SitePolicyEnforcementService` validates imports against site policies.

### Source Validation

Sources are validated against allow/deny lists with wildcard support:

```csharp
// Pattern matching
"github.com/*"     // Matches github.com/advisories, github.com/osv
"*.redhat.com"     // Matches access.redhat.com, bugzilla.redhat.com
"nvd.nist.gov"     // Exact match only
```

**Evaluation order:**
1. Check deny list first (if matched, reject)
2. If allow list is empty, allow all
3. If allow list has entries, source must match at least one

### Bundle Size Validation

```csharp
var result = await policyService.ValidateBundleSizeAsync(
    siteId: "site-us-west",
    bundleSizeMb: 50,
    itemsCount: 5000,
    ct: cancellationToken);

if (!result.IsValid)
{
    // result.SizeExceedsLimit or result.ItemsExceedLimit
}
```

### Budget Tracking

```csharp
var budget = await policyService.GetBudgetInfoAsync("site-us-west", ct);
// Returns: TotalImported, ItemsImported, LastImportAt, RemainingBudgetMb
```

## Usage Examples

### Check for duplicate bundle

```sql
SELECT EXISTS(
    SELECT 1 FROM vuln.sync_ledger
    WHERE bundle_hash = :hash
) AS already_imported;
```

### Get latest cursor for incremental sync

```sql
SELECT cursor
FROM vuln.sync_ledger
WHERE site_id = :site_id
ORDER BY signed_at DESC
LIMIT 1;
```

### Import history with pagination

```sql
SELECT id, cursor, bundle_hash, items_count, signed_at, imported_at
FROM vuln.sync_ledger
WHERE site_id = :site_id
ORDER BY signed_at DESC
LIMIT :limit OFFSET :offset;
```

### Get active site policies

```sql
SELECT site_id, display_name, allowed_sources, denied_sources,
       max_bundle_size_mb, max_items_per_bundle
FROM vuln.site_policy
WHERE enabled = TRUE;
```

### Site sync statistics

```sql
SELECT
    site_id,
    COUNT(*) as bundle_count,
    SUM(items_count) as total_items,
    MAX(signed_at) as latest_bundle,
    MIN(imported_at) as first_import
FROM vuln.sync_ledger
GROUP BY site_id;
```

## Error Handling

### Duplicate Bundle Import

If a bundle with the same hash is imported twice, the unique constraint prevents insertion. The repository returns the existing entry ID:

```csharp
var existingEntry = await repository.GetByBundleHashAsync(bundleHash, ct);
if (existingEntry != null)
{
    // Bundle already imported, skip
    return existingEntry.Id;
}
```

### Cursor Conflict Detection

Out-of-order imports are detected via cursor comparison:

```csharp
bool hasConflict = await repository.IsCursorConflictAsync(siteId, newCursor, ct);
if (hasConflict)
{
    // newCursor is not newer than current position
    throw new CursorConflictException(...);
}
```

## Migration

Migration file: `src/Concelier/__Libraries/StellaOps.Concelier.Storage.Postgres/Migrations/008_sync_ledger.sql`

**Prerequisites:**
- `vuln` schema must exist
- `vuln.update_timestamp()` trigger function must exist (from canonical schema migration)

Apply with:
```bash
psql -d stellaops -f 008_sync_ledger.sql
```

Rollback (if needed):
```sql
DROP TRIGGER IF EXISTS trg_site_policy_updated ON vuln.site_policy;
DROP TABLE IF EXISTS vuln.site_policy;
DROP TABLE IF EXISTS vuln.sync_ledger;
```

## Related

- [Database Specification](../SPECIFICATION.md)
- [Concelier Architecture](../../modules/concelier/architecture.md)
- [Federation Export Sprint](../../implplan/SPRINT_8200_0014_0002_federation_export.md)
- [Federation Import Sprint](../../implplan/SPRINT_8200_0014_0003_federation_import.md)
- [Air-Gap Operation Guide](../../24_OFFLINE_KIT.md)