# Provcache Architecture Guide

> Detailed architecture documentation for the Provenance Cache module

## Overview

Provcache provides a caching layer that maximizes "provenance density" — the amount of trustworthy evidence retained per byte. This document covers the internal architecture, invalidation mechanisms, air-gap support, and replay capabilities.

## Table of Contents

1. [Cache Architecture](#cache-architecture)
2. [Invalidation Mechanisms](#invalidation-mechanisms)
3. [Evidence Chunk Storage](#evidence-chunk-storage)
4. [Air-Gap Export/Import](#air-gap-exportimport)
5. [Lazy Evidence Fetching](#lazy-evidence-fetching)
6. [Revocation Ledger](#revocation-ledger)
7. [API Reference](#api-reference)

---

## Cache Architecture

### Storage Layers

```
┌───────────────────────────────────────────────────────────────┐
│                    Application Layer                          │
│  ┌─────────────┐    ┌─────────────┐    ┌─────────────────┐   │
│  │ VeriKey     │───▶│ Provcache   │───▶│ Policy Engine   │   │
│  │ Builder     │    │ Service     │    │ (cache miss)    │   │
│  └─────────────┘    └─────────────┘    └─────────────────┘   │
└───────────────────────────────────────────────────────────────┘
                             │
                             ▼
┌───────────────────────────────────────────────────────────────┐
│                     Caching Layer                             │
│  ┌─────────────────┐         ┌──────────────────────────┐    │
│  │     Valkey      │◀───────▶│      PostgreSQL          │    │
│  │  (read-through) │         │   (write-behind queue)   │    │
│  │                 │         │                          │    │
│  │  • Hot cache    │         │  • provcache_items       │    │
│  │  • Sub-ms reads │         │  • prov_evidence_chunks  │    │
│  │  • TTL-based    │         │  • prov_revocations      │    │
│  └─────────────────┘         └──────────────────────────┘    │
└───────────────────────────────────────────────────────────────┘
```

### Key Components

| Component | Purpose |
|-----------|---------|
| `IProvcacheService` | Main service interface for cache operations |
| `IProvcacheStore` | Storage abstraction (Valkey + Postgres) |
| `WriteBehindQueue` | Async persistence to Postgres |
| `IEvidenceChunker` | Splits large evidence into Merkle-verified chunks |
| `IRevocationLedger` | Audit trail for all invalidation events |

---

## Invalidation Mechanisms

Provcache supports multiple invalidation triggers to ensure cache consistency when upstream data changes.

### Automatic Invalidation

#### 1. Signer Revocation

When a signing key is compromised or rotated:

```
┌─────────────┐     SignerRevokedEvent     ┌──────────────────┐
│  Authority  │ ──────────────────────────▶│ SignerSet        │
│  Module     │                            │ Invalidator      │
└─────────────┘                            └────────┬─────────┘
                                                    │
                                                    ▼
                                           DELETE FROM provcache_items
                                           WHERE signer_set_hash = ?
```

**Implementation**: `SignerSetInvalidator` subscribes to `SignerRevokedEvent` and invalidates all entries signed by the revoked key.

#### 2. Feed Epoch Advancement

When vulnerability feeds are updated:

```
┌─────────────┐    FeedEpochAdvancedEvent   ┌──────────────────┐
│  Concelier  │ ───────────────────────────▶│ FeedEpoch        │
│  Module     │                             │ Invalidator      │
└─────────────┘                             └────────┬─────────┘
                                                     │
                                                     ▼
                                            DELETE FROM provcache_items
                                            WHERE feed_epoch < ?
```

**Implementation**: `FeedEpochInvalidator` compares epochs using semantic versioning or ISO timestamps.

#### 3. Policy Updates

When policy bundles change:

```
┌─────────────┐      PolicyUpdatedEvent     ┌──────────────────┐
│   Policy    │ ───────────────────────────▶│ PolicyHash       │
│   Engine    │                             │ Invalidator      │
└─────────────┘                             └────────┬─────────┘
                                                     │
                                                     ▼
                                            DELETE FROM provcache_items
                                            WHERE policy_hash = ?
```

### Invalidation Recording

All invalidation events are recorded in the revocation ledger for audit and replay:

```csharp
public interface IProvcacheInvalidator
{
    Task<int> InvalidateAsync(
        InvalidationCriteria criteria,
        string reason,
        string? correlationId = null,
        CancellationToken cancellationToken = default);
}
```

The ledger entry includes:
- Revocation type (signer, feed_epoch, policy, explicit)
- The revoked key
- Number of entries invalidated
- Timestamp and correlation ID for tracing

---

## Evidence Chunk Storage

Large evidence (SBOMs, VEX documents, call graphs) is stored in fixed-size chunks with Merkle tree verification.

### Chunking Process

```
┌─────────────────────────────────────────────────────────────────┐
│                    Original Evidence                             │
│    [ 2.3 MB SPDX SBOM JSON ]                                    │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼ IEvidenceChunker.ChunkAsync()
┌─────────────────────────────────────────────────────────────────┐
│  Chunk 0 (64KB)  │  Chunk 1 (64KB)  │ ... │  Chunk N (partial)  │
│  hash: abc123    │  hash: def456    │     │  hash: xyz789       │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼ Merkle tree construction
┌─────────────────────────────────────────────────────────────────┐
│                        Proof Root                                │
│              sha256:merkle_root_of_all_chunks                    │
└─────────────────────────────────────────────────────────────────┘
```

### Database Schema

```sql
CREATE TABLE provcache.prov_evidence_chunks (
    chunk_id        UUID PRIMARY KEY,
    proof_root      VARCHAR(128) NOT NULL,
    chunk_index     INTEGER NOT NULL,
    chunk_hash      VARCHAR(128) NOT NULL,
    blob            BYTEA NOT NULL,
    blob_size       INTEGER NOT NULL,
    content_type    VARCHAR(64) NOT NULL,
    created_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
    
    CONSTRAINT uk_proof_chunk UNIQUE (proof_root, chunk_index)
);

CREATE INDEX idx_evidence_proof_root ON provcache.prov_evidence_chunks(proof_root);
```

### Paging API

Evidence can be retrieved in pages to manage memory:

```http
GET /api/v1/proofs/{proofRoot}?page=0&pageSize=10
```

Response includes chunk metadata without blob data, allowing clients to fetch specific chunks on demand.

---

## Air-Gap Export/Import

Provcache supports air-gapped environments through minimal proof bundles.

### Bundle Format (v1)

```json
{
    "version": "v1",
    "exportedAt": "2025-01-15T10:30:00Z",
    "density": "standard",
    "digest": {
        "veriKey": "sha256:...",
        "verdictHash": "sha256:...",
        "proofRoot": "sha256:...",
        "trustScore": 85
    },
    "manifest": {
        "proofRoot": "sha256:...",
        "totalChunks": 42,
        "totalSize": 2752512,
        "chunks": [...]
    },
    "chunks": [
        {
            "index": 0,
            "data": "base64...",
            "hash": "sha256:..."
        }
    ],
    "signature": {
        "algorithm": "ECDSA-P256",
        "signature": "base64...",
        "signedAt": "2025-01-15T10:30:01Z"
    }
}
```

### Density Levels

| Level | Contents | Typical Size | Use Case |
|-------|----------|--------------|----------|
| **Lite** | Digest + ProofRoot + Manifest | ~2 KB | Quick verification, requires lazy fetch for full evidence |
| **Standard** | + First 10% of chunks | ~200 KB | Normal audits, balance of size vs completeness |
| **Strict** | + All chunks | Variable | Full compliance, no network needed |

### Export Example

```csharp
var exporter = serviceProvider.GetRequiredService<IMinimalProofExporter>();

// Lite export (manifest only)
var liteBundle = await exporter.ExportAsync(
    veriKey: "sha256:abc123",
    new MinimalProofExportOptions { Density = ProofDensity.Lite });

// Signed strict export
var strictBundle = await exporter.ExportAsync(
    veriKey: "sha256:abc123",
    new MinimalProofExportOptions 
    { 
        Density = ProofDensity.Strict,
        SignBundle = true,
        Signer = signerInstance
    });
```

### Import and Verification

```csharp
var result = await exporter.ImportAsync(bundle);

if (result.DigestVerified && result.ChunksVerified)
{
    // Bundle is authentic
    await provcache.UpsertAsync(result.Entry);
}
```

---

## Lazy Evidence Fetching

For lite bundles, missing chunks can be fetched on-demand from connected or file sources.

### Fetcher Architecture

```
┌────────────────────┐
│ ILazyEvidenceFetcher│
└─────────┬──────────┘
          │
    ┌─────┴─────┐
    │           │
    ▼           ▼
┌─────────┐  ┌──────────┐
│  HTTP   │  │   File   │
│ Fetcher │  │ Fetcher  │
└─────────┘  └──────────┘
```

### HTTP Fetcher (Connected Mode)

```csharp
var fetcher = new HttpChunkFetcher(
    new Uri("https://api.stellaops.com"),
    logger);

var orchestrator = new LazyFetchOrchestrator(repository, logger);

var result = await orchestrator.FetchAndStoreAsync(
    proofRoot: "sha256:...",
    fetcher,
    new LazyFetchOptions
    {
        VerifyOnFetch = true,
        BatchSize = 100
    });
```

### File Fetcher (Sneakernet Mode)

For fully air-gapped environments:

1. Export full evidence to USB drive
2. Transport to isolated network
3. Import using file fetcher

```csharp
var fetcher = new FileChunkFetcher(
    basePath: "/mnt/usb/evidence",
    logger);

var result = await orchestrator.FetchAndStoreAsync(proofRoot, fetcher);
```

---

## Revocation Ledger

The revocation ledger provides a complete audit trail of all invalidation events.

### Schema

```sql
CREATE TABLE provcache.prov_revocations (
    seq_no              BIGSERIAL PRIMARY KEY,
    revocation_id       UUID NOT NULL,
    revocation_type     VARCHAR(32) NOT NULL,
    revoked_key         VARCHAR(512) NOT NULL,
    reason              VARCHAR(1024),
    entries_invalidated INTEGER NOT NULL,
    source              VARCHAR(128) NOT NULL,
    correlation_id      VARCHAR(128),
    revoked_at          TIMESTAMPTZ NOT NULL,
    metadata            JSONB
);
```

### Replay for Catch-Up

After node restart or network partition, nodes can replay missed revocations:

```csharp
var replayService = serviceProvider.GetRequiredService<IRevocationReplayService>();

// Get last checkpoint
var checkpoint = await replayService.GetCheckpointAsync();

// Replay from checkpoint
var result = await replayService.ReplayFromAsync(
    sinceSeqNo: checkpoint,
    new RevocationReplayOptions
    {
        BatchSize = 1000,
        SaveCheckpointPerBatch = true
    });

Console.WriteLine($"Replayed {result.EntriesReplayed} revocations, {result.TotalInvalidations} entries invalidated");
```

### Statistics

```csharp
var ledger = serviceProvider.GetRequiredService<IRevocationLedger>();
var stats = await ledger.GetStatsAsync();

// stats.TotalEntries - total revocation events
// stats.EntriesByType - breakdown by type (signer, feed_epoch, etc.)
// stats.TotalEntriesInvalidated - sum of all invalidated cache entries
```

---

## API Reference

### Evidence Endpoints

| Endpoint | Method | Description |
|----------|--------|-------------|
| `/api/v1/proofs/{proofRoot}` | GET | Get paged evidence chunks |
| `/api/v1/proofs/{proofRoot}/manifest` | GET | Get chunk manifest |
| `/api/v1/proofs/{proofRoot}/chunks/{index}` | GET | Get specific chunk |
| `/api/v1/proofs/{proofRoot}/verify` | POST | Verify Merkle proof |

### Invalidation Endpoints

| Endpoint | Method | Description |
|----------|--------|-------------|
| `/api/v1/provcache/invalidate` | POST | Manual invalidation |
| `/api/v1/provcache/revocations` | GET | List revocation history |
| `/api/v1/provcache/stats` | GET | Cache statistics |

### CLI Commands

```bash
# Export commands
stella prov export --verikey <key> --density <lite|standard|strict> [--output <file>] [--sign]

# Import commands
stella prov import <file> [--lazy-fetch] [--backend <url>] [--chunks-dir <path>]

# Verify commands
stella prov verify <file> [--signer-cert <cert>]
```

---

## Configuration

Key settings in `appsettings.json`:

```json
{
    "Provcache": {
        "ChunkSize": 65536,
        "MaxChunksPerEntry": 1000,
        "DefaultTtl": "24:00:00",
        "EnableWriteBehind": true,
        "WriteBehindFlushInterval": "00:00:05"
    }
}
```

See [README.md](README.md) for full configuration reference.