383 lines
12 KiB
Markdown
383 lines
12 KiB
Markdown
# Provcache Module
|
|
|
|
> Provenance Cache — Maximizing Trust Evidence Density
|
|
|
|
## Overview
|
|
|
|
Provcache is a caching layer that maximizes "provenance density" — the amount of trustworthy evidence retained per byte — enabling faster security decisions, offline replays, and smaller air-gap bundles.
|
|
|
|
### Key Benefits
|
|
|
|
- **Trust Latency**: Warm cache lookups return in single-digit milliseconds
|
|
- **Bandwidth Efficiency**: Avoid re-fetching bulky SBOMs/attestations
|
|
- **Offline Operation**: Decisions usable without full SBOM/VEX payloads
|
|
- **Audit Transparency**: Full evidence chain verifiable via Merkle proofs
|
|
|
|
## Architecture
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────┐
|
|
│ Policy Evaluator │
|
|
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────┐ │
|
|
│ │ VeriKey │───▶│ Provcache │───▶│ TrustLatticeEngine │ │
|
|
│ │ Builder │ │ Service │ │ (if cache miss) │ │
|
|
│ └─────────────┘ └─────────────┘ └─────────────────────┘ │
|
|
└─────────────────────────────────────────────────────────────────┘
|
|
│
|
|
▼
|
|
┌─────────────────────────────────────────┐
|
|
│ Provcache Store │
|
|
│ ┌─────────────┐ ┌────────────────┐ │
|
|
│ │ Valkey │◀──▶│ Postgres │ │
|
|
│ │ (read-thru) │ │ (write-behind) │ │
|
|
│ └─────────────┘ └────────────────┘ │
|
|
└─────────────────────────────────────────┘
|
|
│
|
|
▼
|
|
┌─────────────────────────────────────────┐
|
|
│ Evidence Chunk Store │
|
|
│ ┌─────────────────────────────────────┐│
|
|
│ │ prov_evidence_chunks (Postgres) ││
|
|
│ │ - Chunked SBOM/VEX/CallGraph ││
|
|
│ │ - Merkle tree verification ││
|
|
│ └─────────────────────────────────────┘│
|
|
└─────────────────────────────────────────┘
|
|
```
|
|
|
|
## Core Concepts
|
|
|
|
### VeriKey (Provenance Identity Key)
|
|
|
|
A composite hash that uniquely identifies a provenance decision context:
|
|
|
|
```
|
|
VeriKey = SHA256(
|
|
source_hash || // Image/artifact digest
|
|
sbom_hash || // Canonical SBOM hash
|
|
vex_hash_set_hash || // Sorted VEX statement hashes
|
|
merge_policy_hash || // PolicyBundle hash
|
|
signer_set_hash || // Signer certificate hashes
|
|
time_window // Epoch bucket
|
|
)
|
|
```
|
|
|
|
**Why each component?**
|
|
|
|
| Component | Purpose |
|
|
|-----------|---------|
|
|
| `source_hash` | Different artifacts → different keys |
|
|
| `sbom_hash` | SBOM changes (new packages) → new key |
|
|
| `vex_hash_set` | VEX updates → new key |
|
|
| `policy_hash` | Policy changes → new key |
|
|
| `signer_set_hash` | Key rotation → new key (security) |
|
|
| `time_window` | Temporal bucketing → controlled expiry |
|
|
|
|
### DecisionDigest
|
|
|
|
Canonicalized representation of an evaluation result:
|
|
|
|
```json
|
|
{
|
|
"digestVersion": "v1",
|
|
"veriKey": "sha256:abc123...",
|
|
"verdictHash": "sha256:def456...",
|
|
"proofRoot": "sha256:789abc...",
|
|
"replaySeed": {
|
|
"feedIds": ["cve-2024", "ghsa-2024"],
|
|
"ruleIds": ["default-policy-v2"]
|
|
},
|
|
"trustScore": 85,
|
|
"createdAt": "2025-12-24T12:00:00Z",
|
|
"expiresAt": "2025-12-25T12:00:00Z"
|
|
}
|
|
```
|
|
|
|
### Trust Score
|
|
|
|
A composite score (0-100) indicating decision confidence:
|
|
|
|
| Component | Weight | Calculation |
|
|
|-----------|--------|-------------|
|
|
| Reachability | 25% | Call graph coverage, entry points analyzed |
|
|
| SBOM Completeness | 20% | Package count, license data presence |
|
|
| VEX Coverage | 20% | Vendor statements, justifications |
|
|
| Policy Freshness | 15% | Time since last policy update |
|
|
| Signer Trust | 20% | Key age, reputation, chain validity |
|
|
|
|
### Evidence Chunks
|
|
|
|
Large evidence (SBOM, VEX, call graphs) is stored in fixed-size chunks:
|
|
|
|
- **Default size**: 64 KB per chunk
|
|
- **Merkle verification**: Each chunk is a Merkle leaf
|
|
- **Lazy fetch**: Only fetch chunks needed for audit
|
|
- **LRU eviction**: Old chunks evicted under storage pressure
|
|
|
|
## API Reference
|
|
|
|
### Endpoints
|
|
|
|
| Method | Path | Description |
|
|
|--------|------|-------------|
|
|
| GET | `/v1/provcache/{veriKey}` | Lookup cached decision |
|
|
| POST | `/v1/provcache` | Store decision (idempotent) |
|
|
| POST | `/v1/provcache/invalidate` | Invalidate by pattern |
|
|
| GET | `/v1/proofs/{proofRoot}` | List evidence chunks |
|
|
| GET | `/v1/proofs/{proofRoot}/chunks/{index}` | Download chunk |
|
|
|
|
### Cache Lookup Flow
|
|
|
|
```mermaid
|
|
sequenceDiagram
|
|
participant Client
|
|
participant PolicyEngine
|
|
participant Provcache
|
|
participant Valkey
|
|
participant Postgres
|
|
participant TrustLattice
|
|
|
|
Client->>PolicyEngine: Evaluate(artifact)
|
|
PolicyEngine->>Provcache: Get(VeriKey)
|
|
Provcache->>Valkey: GET verikey
|
|
alt Cache Hit
|
|
Valkey-->>Provcache: DecisionDigest
|
|
Provcache-->>PolicyEngine: CacheResult(hit)
|
|
PolicyEngine-->>Client: Decision (cached)
|
|
else Cache Miss
|
|
Valkey-->>Provcache: null
|
|
Provcache->>Postgres: SELECT * FROM provcache_items
|
|
alt DB Hit
|
|
Postgres-->>Provcache: ProvcacheEntry
|
|
Provcache->>Valkey: SET (backfill)
|
|
Provcache-->>PolicyEngine: CacheResult(hit, source=postgres)
|
|
else DB Miss
|
|
Postgres-->>Provcache: null
|
|
Provcache-->>PolicyEngine: CacheResult(miss)
|
|
PolicyEngine->>TrustLattice: Evaluate
|
|
TrustLattice-->>PolicyEngine: EvaluationResult
|
|
PolicyEngine->>Provcache: Set(VeriKey, DecisionDigest)
|
|
Provcache->>Valkey: SET
|
|
Provcache->>Postgres: INSERT (async)
|
|
PolicyEngine-->>Client: Decision (computed)
|
|
end
|
|
end
|
|
```
|
|
|
|
## Invalidation
|
|
|
|
### Automatic Invalidation Triggers
|
|
|
|
| Trigger | Event | Scope |
|
|
|---------|-------|-------|
|
|
| Signer Revocation | `SignerRevokedEvent` | All entries with matching `signer_set_hash` |
|
|
| Feed Epoch Advance | `FeedEpochAdvancedEvent` | Entries with older `feed_epoch` |
|
|
| Policy Update | `PolicyUpdatedEvent` | Entries with matching `policy_hash` |
|
|
| TTL Expiry | Background job | Entries past `expires_at` |
|
|
|
|
### Manual Invalidation
|
|
|
|
```bash
|
|
# Invalidate by signer
|
|
POST /v1/provcache/invalidate
|
|
{
|
|
"by": "signer_set_hash",
|
|
"value": "sha256:revoked-signer...",
|
|
"reason": "key-compromise"
|
|
}
|
|
|
|
# Invalidate by policy
|
|
POST /v1/provcache/invalidate
|
|
{
|
|
"by": "policy_hash",
|
|
"value": "sha256:old-policy...",
|
|
"reason": "policy-update"
|
|
}
|
|
```
|
|
|
|
## Air-Gap Integration
|
|
|
|
### Export Workflow
|
|
|
|
```bash
|
|
# Export minimal proof (digest only)
|
|
stella prov export --verikey sha256:abc123 --density lite
|
|
|
|
# Export with evidence chunks
|
|
stella prov export --verikey sha256:abc123 --density standard
|
|
|
|
# Export full evidence
|
|
stella prov export --verikey sha256:abc123 --density strict --sign
|
|
```
|
|
|
|
### Import Workflow
|
|
|
|
```bash
|
|
# Import and verify Merkle root
|
|
stella prov import --input proof.bundle
|
|
|
|
# Import with lazy chunk fetch
|
|
stella prov import --input proof-lite.json --lazy-fetch --backend https://api.stellaops.com
|
|
```
|
|
|
|
### Density Levels
|
|
|
|
| Level | Contents | Size | Use Case |
|
|
|-------|----------|------|----------|
|
|
| `lite` | DecisionDigest + ProofRoot | ~2 KB | Quick verification |
|
|
| `standard` | + First N chunks | ~200 KB | Normal audit |
|
|
| `strict` | + All chunks | Variable | Full compliance |
|
|
|
|
## Configuration
|
|
|
|
```yaml
|
|
provcache:
|
|
# TTL configuration
|
|
defaultTtl: 24h
|
|
maxTtl: 168h # 7 days
|
|
timeWindowBucket: 1h
|
|
|
|
# Storage
|
|
valkeyKeyPrefix: "stellaops:prov:"
|
|
enableWriteBehind: true
|
|
writeBehindFlushInterval: 5s
|
|
writeBehindMaxBatchSize: 100
|
|
|
|
# Evidence chunking
|
|
chunkSize: 65536 # 64 KB
|
|
maxChunksPerEntry: 1000
|
|
|
|
# Behavior
|
|
allowCacheBypass: true
|
|
digestVersion: "v1"
|
|
```
|
|
|
|
## Observability
|
|
|
|
### Metrics
|
|
|
|
| Metric | Type | Description |
|
|
|--------|------|-------------|
|
|
| `provcache_requests_total` | Counter | Total cache requests |
|
|
| `provcache_hits_total` | Counter | Cache hits |
|
|
| `provcache_misses_total` | Counter | Cache misses |
|
|
| `provcache_latency_seconds` | Histogram | Operation latency |
|
|
| `provcache_items_count` | Gauge | Current item count |
|
|
| `provcache_invalidations_total` | Counter | Invalidation count |
|
|
|
|
### Alerts
|
|
|
|
```yaml
|
|
# Low cache hit rate
|
|
- alert: ProvcacheLowHitRate
|
|
expr: rate(provcache_hits_total[5m]) / rate(provcache_requests_total[5m]) < 0.5
|
|
for: 10m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Provcache hit rate below 50%"
|
|
|
|
# High invalidation rate
|
|
- alert: ProvcacheHighInvalidationRate
|
|
expr: rate(provcache_invalidations_total[5m]) > 100
|
|
for: 5m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "High cache invalidation rate"
|
|
```
|
|
|
|
## Security Considerations
|
|
|
|
### Signer-Aware Caching
|
|
|
|
The `signer_set_hash` is part of the VeriKey, ensuring:
|
|
|
|
- Key rotation → new cache entries
|
|
- Key revocation → immediate invalidation
|
|
- No stale decisions from compromised signers
|
|
|
|
### Merkle Verification
|
|
|
|
All evidence chunks are Merkle-verified:
|
|
|
|
- `ProofRoot` = Merkle root of all chunks
|
|
- Individual chunks verifiable without full tree
|
|
- Tamper detection on import
|
|
|
|
### Audit Trail
|
|
|
|
All invalidations are logged to `prov_revocations` table:
|
|
|
|
```sql
|
|
SELECT * FROM provcache.prov_revocations
|
|
WHERE created_at > NOW() - INTERVAL '24 hours'
|
|
ORDER BY created_at DESC;
|
|
```
|
|
|
|
## Database Schema
|
|
|
|
### provcache_items
|
|
|
|
```sql
|
|
CREATE TABLE provcache.provcache_items (
|
|
verikey TEXT PRIMARY KEY,
|
|
digest_version TEXT NOT NULL,
|
|
verdict_hash TEXT NOT NULL,
|
|
proof_root TEXT NOT NULL,
|
|
replay_seed JSONB NOT NULL,
|
|
policy_hash TEXT NOT NULL,
|
|
signer_set_hash TEXT NOT NULL,
|
|
feed_epoch TEXT NOT NULL,
|
|
trust_score INTEGER NOT NULL,
|
|
hit_count BIGINT DEFAULT 0,
|
|
created_at TIMESTAMPTZ NOT NULL,
|
|
expires_at TIMESTAMPTZ NOT NULL,
|
|
updated_at TIMESTAMPTZ NOT NULL
|
|
);
|
|
```
|
|
|
|
### prov_evidence_chunks
|
|
|
|
```sql
|
|
CREATE TABLE provcache.prov_evidence_chunks (
|
|
chunk_id UUID PRIMARY KEY,
|
|
proof_root TEXT NOT NULL REFERENCES provcache_items(proof_root),
|
|
chunk_index INTEGER NOT NULL,
|
|
chunk_hash TEXT NOT NULL,
|
|
blob BYTEA NOT NULL,
|
|
blob_size INTEGER NOT NULL,
|
|
content_type TEXT NOT NULL,
|
|
created_at TIMESTAMPTZ NOT NULL
|
|
);
|
|
```
|
|
|
|
### prov_revocations
|
|
|
|
```sql
|
|
CREATE TABLE provcache.prov_revocations (
|
|
revocation_id UUID PRIMARY KEY,
|
|
revocation_type TEXT NOT NULL,
|
|
target_hash TEXT NOT NULL,
|
|
reason TEXT,
|
|
actor TEXT,
|
|
entries_affected BIGINT NOT NULL,
|
|
created_at TIMESTAMPTZ NOT NULL
|
|
);
|
|
```
|
|
|
|
## Implementation Sprints
|
|
|
|
| Sprint | Focus | Key Deliverables |
|
|
|--------|-------|------------------|
|
|
| [8200.0001.0001](../../implplan/SPRINT_8200_0001_0001_provcache_core_backend.md) | Core Backend | VeriKey, DecisionDigest, Valkey+Postgres, API |
|
|
| [8200.0001.0002](../../implplan/SPRINT_8200_0001_0002_provcache_invalidation_airgap.md) | Invalidation & Air-Gap | Signer revocation, feed epochs, CLI export/import |
|
|
| [8200.0001.0003](../../implplan/SPRINT_8200_0001_0003_provcache_ux_observability.md) | UX & Observability | UI badges, proof tree, Grafana, OCI attestation |
|
|
|
|
## Related Documentation
|
|
|
|
- [Policy Engine Architecture](../policy/README.md)
|
|
- [TrustLattice Engine](../policy/design/policy-deterministic-evaluator.md)
|
|
- [Offline Kit Documentation](../../24_OFFLINE_KIT.md)
|
|
- [Air-Gap Controller](../airgap/README.md)
|
|
- [Authority Key Rotation](../authority/README.md)
|