save development progress
This commit is contained in:
StellaOps Bot
438
docs/modules/provcache/architecture.md
Normal file
438
docs/modules/provcache/architecture.md
Normal file
@@ -0,0 +1,438 @@
|
||||
# Provcache Architecture Guide
|
||||
|
||||
> Detailed architecture documentation for the Provenance Cache module
|
||||
|
||||
## Overview
|
||||
|
||||
Provcache provides a caching layer that maximizes "provenance density" — the amount of trustworthy evidence retained per byte. This document covers the internal architecture, invalidation mechanisms, air-gap support, and replay capabilities.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [Cache Architecture](#cache-architecture)
|
||||
2. [Invalidation Mechanisms](#invalidation-mechanisms)
|
||||
3. [Evidence Chunk Storage](#evidence-chunk-storage)
|
||||
4. [Air-Gap Export/Import](#air-gap-exportimport)
|
||||
5. [Lazy Evidence Fetching](#lazy-evidence-fetching)
|
||||
6. [Revocation Ledger](#revocation-ledger)
|
||||
7. [API Reference](#api-reference)
|
||||
|
||||
---
|
||||
|
||||
## Cache Architecture
|
||||
|
||||
### Storage Layers
|
||||
|
||||
```
|
||||
┌───────────────────────────────────────────────────────────────┐
|
||||
│ Application Layer │
|
||||
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────────┐ │
|
||||
│ │ VeriKey │───▶│ Provcache │───▶│ Policy Engine │ │
|
||||
│ │ Builder │ │ Service │ │ (cache miss) │ │
|
||||
│ └─────────────┘ └─────────────┘ └─────────────────┘ │
|
||||
└───────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌───────────────────────────────────────────────────────────────┐
|
||||
│ Caching Layer │
|
||||
│ ┌─────────────────┐ ┌──────────────────────────┐ │
|
||||
│ │ Valkey │◀───────▶│ PostgreSQL │ │
|
||||
│ │ (read-through) │ │ (write-behind queue) │ │
|
||||
│ │ │ │ │ │
|
||||
│ │ • Hot cache │ │ • provcache_items │ │
|
||||
│ │ • Sub-ms reads │ │ • prov_evidence_chunks │ │
|
||||
│ │ • TTL-based │ │ • prov_revocations │ │
|
||||
│ └─────────────────┘ └──────────────────────────┘ │
|
||||
└───────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Key Components
|
||||
|
||||
| Component | Purpose |
|
||||
|-----------|---------|
|
||||
| `IProvcacheService` | Main service interface for cache operations |
|
||||
| `IProvcacheStore` | Storage abstraction (Valkey + Postgres) |
|
||||
| `WriteBehindQueue` | Async persistence to Postgres |
|
||||
| `IEvidenceChunker` | Splits large evidence into Merkle-verified chunks |
|
||||
| `IRevocationLedger` | Audit trail for all invalidation events |
|
||||
|
||||
---
|
||||
|
||||
## Invalidation Mechanisms
|
||||
|
||||
Provcache supports multiple invalidation triggers to ensure cache consistency when upstream data changes.
|
||||
|
||||
### Automatic Invalidation
|
||||
|
||||
#### 1. Signer Revocation
|
||||
|
||||
When a signing key is compromised or rotated:
|
||||
|
||||
```
|
||||
┌─────────────┐ SignerRevokedEvent ┌──────────────────┐
|
||||
│ Authority │ ──────────────────────────▶│ SignerSet │
|
||||
│ Module │ │ Invalidator │
|
||||
└─────────────┘ └────────┬─────────┘
|
||||
│
|
||||
▼
|
||||
DELETE FROM provcache_items
|
||||
WHERE signer_set_hash = ?
|
||||
```
|
||||
|
||||
**Implementation**: `SignerSetInvalidator` subscribes to `SignerRevokedEvent` and invalidates all entries signed by the revoked key.
|
||||
|
||||
#### 2. Feed Epoch Advancement
|
||||
|
||||
When vulnerability feeds are updated:
|
||||
|
||||
```
|
||||
┌─────────────┐ FeedEpochAdvancedEvent ┌──────────────────┐
|
||||
│ Concelier │ ───────────────────────────▶│ FeedEpoch │
|
||||
│ Module │ │ Invalidator │
|
||||
└─────────────┘ └────────┬─────────┘
|
||||
│
|
||||
▼
|
||||
DELETE FROM provcache_items
|
||||
WHERE feed_epoch < ?
|
||||
```
|
||||
|
||||
**Implementation**: `FeedEpochInvalidator` compares epochs using semantic versioning or ISO timestamps.
|
||||
|
||||
#### 3. Policy Updates
|
||||
|
||||
When policy bundles change:
|
||||
|
||||
```
|
||||
┌─────────────┐ PolicyUpdatedEvent ┌──────────────────┐
|
||||
│ Policy │ ───────────────────────────▶│ PolicyHash │
|
||||
│ Engine │ │ Invalidator │
|
||||
└─────────────┘ └────────┬─────────┘
|
||||
│
|
||||
▼
|
||||
DELETE FROM provcache_items
|
||||
WHERE policy_hash = ?
|
||||
```
|
||||
|
||||
### Invalidation Recording
|
||||
|
||||
All invalidation events are recorded in the revocation ledger for audit and replay:
|
||||
|
||||
```csharp
|
||||
public interface IProvcacheInvalidator
|
||||
{
|
||||
Task<int> InvalidateAsync(
|
||||
InvalidationCriteria criteria,
|
||||
string reason,
|
||||
string? correlationId = null,
|
||||
CancellationToken cancellationToken = default);
|
||||
}
|
||||
```
|
||||
|
||||
The ledger entry includes:
|
||||
- Revocation type (signer, feed_epoch, policy, explicit)
|
||||
- The revoked key
|
||||
- Number of entries invalidated
|
||||
- Timestamp and correlation ID for tracing
|
||||
|
||||
---
|
||||
|
||||
## Evidence Chunk Storage
|
||||
|
||||
Large evidence (SBOMs, VEX documents, call graphs) is stored in fixed-size chunks with Merkle tree verification.
|
||||
|
||||
### Chunking Process
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ Original Evidence │
|
||||
│ [ 2.3 MB SPDX SBOM JSON ] │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼ IEvidenceChunker.ChunkAsync()
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ Chunk 0 (64KB) │ Chunk 1 (64KB) │ ... │ Chunk N (partial) │
|
||||
│ hash: abc123 │ hash: def456 │ │ hash: xyz789 │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼ Merkle tree construction
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ Proof Root │
|
||||
│ sha256:merkle_root_of_all_chunks │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Database Schema
|
||||
|
||||
```sql
|
||||
CREATE TABLE provcache.prov_evidence_chunks (
|
||||
chunk_id UUID PRIMARY KEY,
|
||||
proof_root VARCHAR(128) NOT NULL,
|
||||
chunk_index INTEGER NOT NULL,
|
||||
chunk_hash VARCHAR(128) NOT NULL,
|
||||
blob BYTEA NOT NULL,
|
||||
blob_size INTEGER NOT NULL,
|
||||
content_type VARCHAR(64) NOT NULL,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
|
||||
CONSTRAINT uk_proof_chunk UNIQUE (proof_root, chunk_index)
|
||||
);
|
||||
|
||||
CREATE INDEX idx_evidence_proof_root ON provcache.prov_evidence_chunks(proof_root);
|
||||
```
|
||||
|
||||
### Paging API
|
||||
|
||||
Evidence can be retrieved in pages to manage memory:
|
||||
|
||||
```http
|
||||
GET /api/v1/proofs/{proofRoot}?page=0&pageSize=10
|
||||
```
|
||||
|
||||
Response includes chunk metadata without blob data, allowing clients to fetch specific chunks on demand.
|
||||
|
||||
---
|
||||
|
||||
## Air-Gap Export/Import
|
||||
|
||||
Provcache supports air-gapped environments through minimal proof bundles.
|
||||
|
||||
### Bundle Format (v1)
|
||||
|
||||
```json
|
||||
{
|
||||
"version": "v1",
|
||||
"exportedAt": "2025-01-15T10:30:00Z",
|
||||
"density": "standard",
|
||||
"digest": {
|
||||
"veriKey": "sha256:...",
|
||||
"verdictHash": "sha256:...",
|
||||
"proofRoot": "sha256:...",
|
||||
"trustScore": 85
|
||||
},
|
||||
"manifest": {
|
||||
"proofRoot": "sha256:...",
|
||||
"totalChunks": 42,
|
||||
"totalSize": 2752512,
|
||||
"chunks": [...]
|
||||
},
|
||||
"chunks": [
|
||||
{
|
||||
"index": 0,
|
||||
"data": "base64...",
|
||||
"hash": "sha256:..."
|
||||
}
|
||||
],
|
||||
"signature": {
|
||||
"algorithm": "ECDSA-P256",
|
||||
"signature": "base64...",
|
||||
"signedAt": "2025-01-15T10:30:01Z"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Density Levels
|
||||
|
||||
| Level | Contents | Typical Size | Use Case |
|
||||
|-------|----------|--------------|----------|
|
||||
| **Lite** | Digest + ProofRoot + Manifest | ~2 KB | Quick verification, requires lazy fetch for full evidence |
|
||||
| **Standard** | + First 10% of chunks | ~200 KB | Normal audits, balance of size vs completeness |
|
||||
| **Strict** | + All chunks | Variable | Full compliance, no network needed |
|
||||
|
||||
### Export Example
|
||||
|
||||
```csharp
|
||||
var exporter = serviceProvider.GetRequiredService<IMinimalProofExporter>();
|
||||
|
||||
// Lite export (manifest only)
|
||||
var liteBundle = await exporter.ExportAsync(
|
||||
veriKey: "sha256:abc123",
|
||||
new MinimalProofExportOptions { Density = ProofDensity.Lite });
|
||||
|
||||
// Signed strict export
|
||||
var strictBundle = await exporter.ExportAsync(
|
||||
veriKey: "sha256:abc123",
|
||||
new MinimalProofExportOptions
|
||||
{
|
||||
Density = ProofDensity.Strict,
|
||||
SignBundle = true,
|
||||
Signer = signerInstance
|
||||
});
|
||||
```
|
||||
|
||||
### Import and Verification
|
||||
|
||||
```csharp
|
||||
var result = await exporter.ImportAsync(bundle);
|
||||
|
||||
if (result.DigestVerified && result.ChunksVerified)
|
||||
{
|
||||
// Bundle is authentic
|
||||
await provcache.UpsertAsync(result.Entry);
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Lazy Evidence Fetching
|
||||
|
||||
For lite bundles, missing chunks can be fetched on-demand from connected or file sources.
|
||||
|
||||
### Fetcher Architecture
|
||||
|
||||
```
|
||||
┌────────────────────┐
|
||||
│ ILazyEvidenceFetcher│
|
||||
└─────────┬──────────┘
|
||||
│
|
||||
┌─────┴─────┐
|
||||
│ │
|
||||
▼ ▼
|
||||
┌─────────┐ ┌──────────┐
|
||||
│ HTTP │ │ File │
|
||||
│ Fetcher │ │ Fetcher │
|
||||
└─────────┘ └──────────┘
|
||||
```
|
||||
|
||||
### HTTP Fetcher (Connected Mode)
|
||||
|
||||
```csharp
|
||||
var fetcher = new HttpChunkFetcher(
|
||||
new Uri("https://api.stellaops.com"),
|
||||
logger);
|
||||
|
||||
var orchestrator = new LazyFetchOrchestrator(repository, logger);
|
||||
|
||||
var result = await orchestrator.FetchAndStoreAsync(
|
||||
proofRoot: "sha256:...",
|
||||
fetcher,
|
||||
new LazyFetchOptions
|
||||
{
|
||||
VerifyOnFetch = true,
|
||||
BatchSize = 100
|
||||
});
|
||||
```
|
||||
|
||||
### File Fetcher (Sneakernet Mode)
|
||||
|
||||
For fully air-gapped environments:
|
||||
|
||||
1. Export full evidence to USB drive
|
||||
2. Transport to isolated network
|
||||
3. Import using file fetcher
|
||||
|
||||
```csharp
|
||||
var fetcher = new FileChunkFetcher(
|
||||
basePath: "/mnt/usb/evidence",
|
||||
logger);
|
||||
|
||||
var result = await orchestrator.FetchAndStoreAsync(proofRoot, fetcher);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Revocation Ledger
|
||||
|
||||
The revocation ledger provides a complete audit trail of all invalidation events.
|
||||
|
||||
### Schema
|
||||
|
||||
```sql
|
||||
CREATE TABLE provcache.prov_revocations (
|
||||
seq_no BIGSERIAL PRIMARY KEY,
|
||||
revocation_id UUID NOT NULL,
|
||||
revocation_type VARCHAR(32) NOT NULL,
|
||||
revoked_key VARCHAR(512) NOT NULL,
|
||||
reason VARCHAR(1024),
|
||||
entries_invalidated INTEGER NOT NULL,
|
||||
source VARCHAR(128) NOT NULL,
|
||||
correlation_id VARCHAR(128),
|
||||
revoked_at TIMESTAMPTZ NOT NULL,
|
||||
metadata JSONB
|
||||
);
|
||||
```
|
||||
|
||||
### Replay for Catch-Up
|
||||
|
||||
After node restart or network partition, nodes can replay missed revocations:
|
||||
|
||||
```csharp
|
||||
var replayService = serviceProvider.GetRequiredService<IRevocationReplayService>();
|
||||
|
||||
// Get last checkpoint
|
||||
var checkpoint = await replayService.GetCheckpointAsync();
|
||||
|
||||
// Replay from checkpoint
|
||||
var result = await replayService.ReplayFromAsync(
|
||||
sinceSeqNo: checkpoint,
|
||||
new RevocationReplayOptions
|
||||
{
|
||||
BatchSize = 1000,
|
||||
SaveCheckpointPerBatch = true
|
||||
});
|
||||
|
||||
Console.WriteLine($"Replayed {result.EntriesReplayed} revocations, {result.TotalInvalidations} entries invalidated");
|
||||
```
|
||||
|
||||
### Statistics
|
||||
|
||||
```csharp
|
||||
var ledger = serviceProvider.GetRequiredService<IRevocationLedger>();
|
||||
var stats = await ledger.GetStatsAsync();
|
||||
|
||||
// stats.TotalEntries - total revocation events
|
||||
// stats.EntriesByType - breakdown by type (signer, feed_epoch, etc.)
|
||||
// stats.TotalEntriesInvalidated - sum of all invalidated cache entries
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## API Reference
|
||||
|
||||
### Evidence Endpoints
|
||||
|
||||
| Endpoint | Method | Description |
|
||||
|----------|--------|-------------|
|
||||
| `/api/v1/proofs/{proofRoot}` | GET | Get paged evidence chunks |
|
||||
| `/api/v1/proofs/{proofRoot}/manifest` | GET | Get chunk manifest |
|
||||
| `/api/v1/proofs/{proofRoot}/chunks/{index}` | GET | Get specific chunk |
|
||||
| `/api/v1/proofs/{proofRoot}/verify` | POST | Verify Merkle proof |
|
||||
|
||||
### Invalidation Endpoints
|
||||
|
||||
| Endpoint | Method | Description |
|
||||
|----------|--------|-------------|
|
||||
| `/api/v1/provcache/invalidate` | POST | Manual invalidation |
|
||||
| `/api/v1/provcache/revocations` | GET | List revocation history |
|
||||
| `/api/v1/provcache/stats` | GET | Cache statistics |
|
||||
|
||||
### CLI Commands
|
||||
|
||||
```bash
|
||||
# Export commands
|
||||
stella prov export --verikey <key> --density <lite|standard|strict> [--output <file>] [--sign]
|
||||
|
||||
# Import commands
|
||||
stella prov import <file> [--lazy-fetch] [--backend <url>] [--chunks-dir <path>]
|
||||
|
||||
# Verify commands
|
||||
stella prov verify <file> [--signer-cert <cert>]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
Key settings in `appsettings.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"Provcache": {
|
||||
"ChunkSize": 65536,
|
||||
"MaxChunksPerEntry": 1000,
|
||||
"DefaultTtl": "24:00:00",
|
||||
"EnableWriteBehind": true,
|
||||
"WriteBehindFlushInterval": "00:00:05"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
See [README.md](README.md) for full configuration reference.
|
||||
Reference in New Issue
Block a user