save development progress

2025-12-25 23:09:58 +02:00
parent d71853ad7e
commit aa70af062e
351 changed files with 37683 additions and 150156 deletions
--- a/docs/db/schemas/sync-ledger.md
+++ b/docs/db/schemas/sync-ledger.md
@@ -0,0 +1,274 @@
+# Sync Ledger Schema
+
+Sprint: `SPRINT_8200_0014_0001_DB_sync_ledger_schema`
+Migration: `008_sync_ledger.sql`
+Working Directory: `src/Concelier/__Libraries/StellaOps.Concelier.Storage.Postgres/`
+
+## Overview
+
+The sync ledger schema provides federation cursor tracking for multi-site vulnerability advisory synchronization. It enables air-gapped and distributed deployments to:
+
+1. Track sync state per remote site using monotonic cursors
+2. Prevent duplicate bundle imports via hash-based deduplication
+3. Enforce per-site policies (source filtering, size limits, signature requirements)
+
+This is critical infrastructure for the Concelier module's federation capabilities.
+
+## Tables
+
+### `vuln.sync_ledger`
+
+Tracks federation sync state per remote site. Each record represents a successfully imported bundle.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | UUID | Primary key (auto-generated) |
+| `site_id` | TEXT | Remote site identifier (e.g., `site-us-west`, `airgap-dc2`) |
+| `cursor` | TEXT | Position marker for incremental sync (ISO8601#sequence format) |
+| `bundle_hash` | TEXT | SHA256 hash of imported bundle (for deduplication) |
+| `items_count` | INT | Number of advisory items in the bundle |
+| `signed_at` | TIMESTAMPTZ | Timestamp when bundle was signed by remote site |
+| `imported_at` | TIMESTAMPTZ | Timestamp when bundle was imported locally (auto-generated) |
+
+**Constraints:**
+- `uq_sync_ledger_site_cursor`: Unique constraint on `(site_id, cursor)` - prevents duplicate cursor positions
+- `uq_sync_ledger_bundle`: Unique constraint on `bundle_hash` - prevents reimporting same bundle
+
+### `vuln.site_policy`
+
+Per-site federation governance policies. Controls what data can be imported from each remote site.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | UUID | Primary key (auto-generated) |
+| `site_id` | TEXT | Site identifier (unique) |
+| `display_name` | TEXT | Human-readable site name |
+| `allowed_sources` | TEXT[] | Source allow list (empty = allow all, supports wildcards) |
+| `denied_sources` | TEXT[] | Source deny list (takes precedence over allow list) |
+| `max_bundle_size_mb` | INT | Maximum bundle size in MB (default: 100) |
+| `max_items_per_bundle` | INT | Maximum items per bundle (default: 10000) |
+| `require_signature` | BOOLEAN | Require cryptographic signature on bundles (default: TRUE) |
+| `allowed_signers` | TEXT[] | Allowed signer key IDs or issuer names |
+| `enabled` | BOOLEAN | Policy enabled status (default: TRUE) |
+| `created_at` | TIMESTAMPTZ | Record creation time |
+| `updated_at` | TIMESTAMPTZ | Last modification time (auto-updated via trigger) |
+
+## Indexes
+
+| Index | Columns | Purpose |
+|-------|---------|---------|
+| `idx_sync_ledger_site` | `site_id` | Fast site-based lookups |
+| `idx_sync_ledger_site_time` | `site_id, signed_at DESC` | Get latest entries per site |
+| `idx_site_policy_enabled` | `enabled` (partial, WHERE enabled = TRUE) | Filter active policies |
+
+## Cursor Format
+
+Cursors use a combined timestamp + sequence format for total ordering:
+
+```
+2025-01-15T10:30:00.000Z#0042
+```
+
+**Components:**
+- ISO8601 timestamp with timezone (UTC)
+- `#` separator
+- 4-digit sequence number (allows multiple bundles per timestamp)
+
+**Ordering rules:**
+1. Compare timestamps first
+2. If equal, compare sequence numbers
+3. Higher values indicate later positions
+
+### CursorFormat Utility
+
+```csharp
+// Create cursor
+string cursor = CursorFormat.Create(DateTimeOffset.UtcNow, sequence: 0);
+// Result: "2025-01-15T10:30:00.0000000+00:00#0000"
+
+// Parse cursor
+var (timestamp, sequence) = CursorFormat.Parse(cursor);
+
+// Compare cursors
+bool isNewer = CursorFormat.IsAfter(cursor1, cursor2);
+```
+
+## Repository Operations
+
+### ISyncLedgerRepository Interface
+
+```csharp
+public interface ISyncLedgerRepository
+{
+    // Ledger CRUD
+    Task<SyncLedgerEntity?> GetLatestAsync(string siteId, CancellationToken ct = default);
+    Task<IReadOnlyList<SyncLedgerEntity>> GetHistoryAsync(string siteId, int limit = 10, CancellationToken ct = default);
+    Task<SyncLedgerEntity?> GetByBundleHashAsync(string bundleHash, CancellationToken ct = default);
+    Task<Guid> InsertAsync(SyncLedgerEntity entry, CancellationToken ct = default);
+
+    // Cursor operations
+    Task<string?> GetCursorAsync(string siteId, CancellationToken ct = default);
+    Task AdvanceCursorAsync(string siteId, string newCursor, string bundleHash,
+                            int itemsCount, DateTimeOffset signedAt, CancellationToken ct = default);
+    Task<bool> IsCursorConflictAsync(string siteId, string cursor, CancellationToken ct = default);
+
+    // Policy operations
+    Task<SitePolicyEntity?> GetPolicyAsync(string siteId, CancellationToken ct = default);
+    Task UpsertPolicyAsync(SitePolicyEntity policy, CancellationToken ct = default);
+    Task<IReadOnlyList<SitePolicyEntity>> GetAllPoliciesAsync(bool enabledOnly = true, CancellationToken ct = default);
+
+    // Statistics
+    Task<SyncStatistics> GetStatisticsAsync(CancellationToken ct = default);
+}
+```
+
+## Policy Enforcement
+
+The `SitePolicyEnforcementService` validates imports against site policies.
+
+### Source Validation
+
+Sources are validated against allow/deny lists with wildcard support:
+
+```csharp
+// Pattern matching
+"github.com/*"     // Matches github.com/advisories, github.com/osv
+"*.redhat.com"     // Matches access.redhat.com, bugzilla.redhat.com
+"nvd.nist.gov"     // Exact match only
+```
+
+**Evaluation order:**
+1. Check deny list first (if matched, reject)
+2. If allow list is empty, allow all
+3. If allow list has entries, source must match at least one
+
+### Bundle Size Validation
+
+```csharp
+var result = await policyService.ValidateBundleSizeAsync(
+    siteId: "site-us-west",
+    bundleSizeMb: 50,
+    itemsCount: 5000,
+    ct: cancellationToken);
+
+if (!result.IsValid)
+{
+    // result.SizeExceedsLimit or result.ItemsExceedLimit
+}
+```
+
+### Budget Tracking
+
+```csharp
+var budget = await policyService.GetBudgetInfoAsync("site-us-west", ct);
+// Returns: TotalImported, ItemsImported, LastImportAt, RemainingBudgetMb
+```
+
+## Usage Examples
+
+### Check for duplicate bundle
+
+```sql
+SELECT EXISTS(
+    SELECT 1 FROM vuln.sync_ledger
+    WHERE bundle_hash = :hash
+) AS already_imported;
+```
+
+### Get latest cursor for incremental sync
+
+```sql
+SELECT cursor
+FROM vuln.sync_ledger
+WHERE site_id = :site_id
+ORDER BY signed_at DESC
+LIMIT 1;
+```
+
+### Import history with pagination
+
+```sql
+SELECT id, cursor, bundle_hash, items_count, signed_at, imported_at
+FROM vuln.sync_ledger
+WHERE site_id = :site_id
+ORDER BY signed_at DESC
+LIMIT :limit OFFSET :offset;
+```
+
+### Get active site policies
+
+```sql
+SELECT site_id, display_name, allowed_sources, denied_sources,
+       max_bundle_size_mb, max_items_per_bundle
+FROM vuln.site_policy
+WHERE enabled = TRUE;
+```
+
+### Site sync statistics
+
+```sql
+SELECT
+    site_id,
+    COUNT(*) as bundle_count,
+    SUM(items_count) as total_items,
+    MAX(signed_at) as latest_bundle,
+    MIN(imported_at) as first_import
+FROM vuln.sync_ledger
+GROUP BY site_id;
+```
+
+## Error Handling
+
+### Duplicate Bundle Import
+
+If a bundle with the same hash is imported twice, the unique constraint prevents insertion. The repository returns the existing entry ID:
+
+```csharp
+var existingEntry = await repository.GetByBundleHashAsync(bundleHash, ct);
+if (existingEntry != null)
+{
+    // Bundle already imported, skip
+    return existingEntry.Id;
+}
+```
+
+### Cursor Conflict Detection
+
+Out-of-order imports are detected via cursor comparison:
+
+```csharp
+bool hasConflict = await repository.IsCursorConflictAsync(siteId, newCursor, ct);
+if (hasConflict)
+{
+    // newCursor is not newer than current position
+    throw new CursorConflictException(...);
+}
+```
+
+## Migration
+
+Migration file: `src/Concelier/__Libraries/StellaOps.Concelier.Storage.Postgres/Migrations/008_sync_ledger.sql`
+
+**Prerequisites:**
+- `vuln` schema must exist
+- `vuln.update_timestamp()` trigger function must exist (from canonical schema migration)
+
+Apply with:
+```bash
+psql -d stellaops -f 008_sync_ledger.sql
+```
+
+Rollback (if needed):
+```sql
+DROP TRIGGER IF EXISTS trg_site_policy_updated ON vuln.site_policy;
+DROP TABLE IF EXISTS vuln.site_policy;
+DROP TABLE IF EXISTS vuln.sync_ledger;
+```
+
+## Related
+
+- [Database Specification](../SPECIFICATION.md)
+- [Concelier Architecture](../../modules/concelier/architecture.md)
+- [Federation Export Sprint](../../implplan/SPRINT_8200_0014_0002_federation_export.md)
+- [Federation Import Sprint](../../implplan/SPRINT_8200_0014_0003_federation_import.md)
+- [Air-Gap Operation Guide](../../24_OFFLINE_KIT.md)