docs(ops): Complete operations runbooks for Epic 3500
Sprint 3500.0004.0004 (Documentation & Handoff) - T2 DONE Operations Runbooks Added: - score-replay-runbook.md: Deterministic replay procedures - proof-verification-runbook.md: DSSE/Merkle verification ops - airgap-operations-runbook.md: Offline kit management CLI Reference Docs: - reachability-cli-reference.md - score-proofs-cli-reference.md - unknowns-cli-reference.md Air-Gap Guides: - score-proofs-reachability-airgap-runbook.md Training Materials: - score-proofs-concept-guide.md UI API Clients: - proof.client.ts - reachability.client.ts - unknowns.client.ts All 5 operations runbooks now complete (reachability, unknowns-queue, score-replay, proof-verification, airgap-operations).
This commit is contained in:
StellaOps Bot
630
docs/operations/proof-verification-runbook.md
Normal file
630
docs/operations/proof-verification-runbook.md
Normal file
@@ -0,0 +1,630 @@
|
||||
# Proof Verification Operations Runbook
|
||||
|
||||
> **Version**: 1.0.0
|
||||
> **Sprint**: 3500.0004.0004
|
||||
> **Last Updated**: 2025-12-20
|
||||
|
||||
This runbook covers operational procedures for Proof Verification, including DSSE signature validation, Merkle tree verification, transparency log checks, and offline verification workflows.
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [Overview](#1-overview)
|
||||
2. [Verification Operations](#2-verification-operations)
|
||||
3. [Offline Verification](#3-offline-verification)
|
||||
4. [Transparency Log Integration](#4-transparency-log-integration)
|
||||
5. [Troubleshooting](#5-troubleshooting)
|
||||
6. [Monitoring & Alerting](#6-monitoring--alerting)
|
||||
7. [Escalation Procedures](#7-escalation-procedures)
|
||||
|
||||
---
|
||||
|
||||
## 1. Overview
|
||||
|
||||
### What is Proof Verification?
|
||||
|
||||
Proof Verification is the process of cryptographically validating that a scan result has not been tampered with and was produced by an authorized StellaOps instance. It involves:
|
||||
|
||||
- **DSSE Signature Verification**: Validate the signing envelope
|
||||
- **Merkle Tree Verification**: Confirm the root hash matches the proof
|
||||
- **Certificate Chain Validation**: Verify the signing certificate
|
||||
- **Transparency Log Check**: Optional Rekor/Sigstore verification
|
||||
|
||||
### Verification Components
|
||||
|
||||
| Component | Purpose | Verification Type |
|
||||
|-----------|---------|-------------------|
|
||||
| DSSE Envelope | Contains signed payload | Signature validation |
|
||||
| Merkle Proof | Cryptographic proof of inclusion | Hash verification |
|
||||
| Certificate | Signing identity | Chain validation |
|
||||
| Rekor Entry | Transparency log record | Log inclusion proof |
|
||||
|
||||
### Trust Model
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Trust Hierarchy │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ Root CA (Offline) │
|
||||
│ └── Intermediate CA │
|
||||
│ └── Signing Certificate (Scanner Instance) │
|
||||
│ └── DSSE Envelope │
|
||||
│ └── Proof Bundle │
|
||||
│ └── Manifest + Score │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Verification Operations
|
||||
|
||||
### 2.1 Basic Proof Verification
|
||||
|
||||
#### Via CLI
|
||||
|
||||
```bash
|
||||
# Verify a proof bundle file
|
||||
stella proof verify --bundle bundle.tar.gz
|
||||
|
||||
# Verify with verbose output
|
||||
stella proof verify --bundle bundle.tar.gz --verbose
|
||||
|
||||
# Verify and output as JSON
|
||||
stella proof verify --bundle bundle.tar.gz --output json
|
||||
```
|
||||
|
||||
#### Expected Output (Success)
|
||||
|
||||
```
|
||||
Proof Verification Result
|
||||
══════════════════════════════════════════
|
||||
✓ DSSE Signature VALID
|
||||
✓ Merkle Root VALID
|
||||
✓ Certificate Chain VALID
|
||||
✓ Not Expired VALID
|
||||
──────────────────────────────────────────
|
||||
Overall: VERIFIED
|
||||
|
||||
Root Hash: sha256:abc123...
|
||||
Signed By: scanner-prod-01.stellaops.local
|
||||
Signed At: 2025-01-15T10:30:00Z
|
||||
Valid Until: 2026-01-15T10:30:00Z
|
||||
```
|
||||
|
||||
#### Expected Output (Failure)
|
||||
|
||||
```
|
||||
Proof Verification Result
|
||||
══════════════════════════════════════════
|
||||
✓ DSSE Signature VALID
|
||||
✗ Merkle Root INVALID
|
||||
✓ Certificate Chain VALID
|
||||
✓ Not Expired VALID
|
||||
──────────────────────────────────────────
|
||||
Overall: FAILED
|
||||
|
||||
Error: Merkle root mismatch
|
||||
Expected: sha256:abc123...
|
||||
Actual: sha256:def456...
|
||||
```
|
||||
|
||||
### 2.2 Verification via API
|
||||
|
||||
```bash
|
||||
# Verify by scan ID
|
||||
curl -X POST "https://scanner.stellaops.local/api/v1/scanner/scans/$SCAN_ID/proofs/$ROOT_HASH/verify" \
|
||||
-H "Authorization: Bearer $TOKEN" \
|
||||
-H "Content-Type: application/json"
|
||||
|
||||
# Response
|
||||
{
|
||||
"valid": true,
|
||||
"rootHash": "sha256:abc123...",
|
||||
"checks": [
|
||||
{"name": "dsse_signature", "passed": true, "message": "Signature valid"},
|
||||
{"name": "merkle_root", "passed": true, "message": "Root hash matches"},
|
||||
{"name": "certificate_chain", "passed": true, "message": "Chain valid"},
|
||||
{"name": "not_expired", "passed": true, "message": "Certificate not expired"}
|
||||
],
|
||||
"verifiedAt": "2025-01-16T10:30:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
### 2.3 Viewing Merkle Spine
|
||||
|
||||
The Merkle spine shows the path from leaf nodes to the root:
|
||||
|
||||
```bash
|
||||
stella proof spine --bundle bundle.tar.gz
|
||||
```
|
||||
|
||||
Output:
|
||||
```
|
||||
Merkle Tree Spine
|
||||
══════════════════════════════════════════
|
||||
Root: sha256:abc123...
|
||||
├── sha256:node1... (sbom_hash)
|
||||
├── sha256:node2... (rules_hash)
|
||||
├── sha256:node3... (policy_hash)
|
||||
└── sha256:node4... (feed_hash)
|
||||
|
||||
Depth: 3
|
||||
Leaves: 4
|
||||
Algorithm: SHA-256
|
||||
```
|
||||
|
||||
### 2.4 Certificate Inspection
|
||||
|
||||
```bash
|
||||
# Extract and inspect certificate
|
||||
tar -xzf bundle.tar.gz
|
||||
openssl x509 -in bundle/certificate.pem -noout -text
|
||||
|
||||
# Check validity period
|
||||
openssl x509 -in bundle/certificate.pem -noout -dates
|
||||
|
||||
# Verify against CA bundle
|
||||
openssl verify -CAfile /etc/stellaops/ca-bundle.pem bundle/certificate.pem
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Offline Verification
|
||||
|
||||
### 3.1 When to Use Offline Verification
|
||||
|
||||
- Air-gapped environments
|
||||
- Network-restricted systems
|
||||
- Compliance audits without API access
|
||||
- Disaster recovery scenarios
|
||||
|
||||
### 3.2 Prerequisites for Offline Verification
|
||||
|
||||
Required files:
|
||||
- Proof bundle (`.tar.gz`)
|
||||
- CA certificate bundle (`ca-bundle.pem`)
|
||||
- Trust root configuration (`trust-roots.json`)
|
||||
|
||||
```bash
|
||||
# Prepare offline verification kit
|
||||
stella proof offline-kit create \
|
||||
--output /path/to/offline-kit/ \
|
||||
--include-ca \
|
||||
--include-trust-roots
|
||||
```
|
||||
|
||||
Kit contents:
|
||||
```
|
||||
offline-kit/
|
||||
├── ca-bundle.pem # Certificate authority chain
|
||||
├── trust-roots.json # Trusted signing keys
|
||||
├── verify.sh # Standalone verification script
|
||||
└── README.md # Instructions
|
||||
```
|
||||
|
||||
### 3.3 Running Offline Verification
|
||||
|
||||
```bash
|
||||
# Using CLI with offline flag
|
||||
stella proof verify --bundle bundle.tar.gz --offline
|
||||
|
||||
# Using standalone script
|
||||
./verify.sh bundle.tar.gz
|
||||
|
||||
# Manual verification with OpenSSL
|
||||
./verify.sh bundle.tar.gz --ca-bundle ./ca-bundle.pem
|
||||
```
|
||||
|
||||
### 3.4 Offline Verification Checks
|
||||
|
||||
| Check | Online | Offline | Notes |
|
||||
|-------|--------|---------|-------|
|
||||
| DSSE Signature | ✓ | ✓ | Local crypto |
|
||||
| Merkle Root | ✓ | ✓ | Local hash computation |
|
||||
| Certificate Chain | ✓ | ✓ | Requires CA bundle |
|
||||
| Certificate Revocation | ✓ | ✗ | Needs CRL/OCSP |
|
||||
| Rekor Transparency | ✓ | ✗ | Needs network |
|
||||
|
||||
### 3.5 Air-Gap Considerations
|
||||
|
||||
For fully air-gapped environments:
|
||||
|
||||
1. **Pre-stage CA bundle**:
|
||||
```bash
|
||||
# On connected system
|
||||
stella ca export --output ca-bundle.pem
|
||||
|
||||
# Transfer to air-gapped system
|
||||
scp ca-bundle.pem airgap:/etc/stellaops/
|
||||
```
|
||||
|
||||
2. **Pre-stage CRL (optional)**:
|
||||
```bash
|
||||
# Download latest CRL
|
||||
curl -o crl.pem https://ca.stellaops.io/crl/latest.pem
|
||||
|
||||
# Transfer and use
|
||||
stella proof verify --bundle bundle.tar.gz --offline --crl crl.pem
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Transparency Log Integration
|
||||
|
||||
### 4.1 Rekor Overview
|
||||
|
||||
StellaOps optionally publishes proof attestations to Sigstore Rekor for immutable transparency logging.
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Transparency Flow │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ Proof Bundle │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ DSSE Envelope ──────► Rekor ──────► Inclusion Proof │
|
||||
│ │ │ │
|
||||
│ ▼ ▼ │
|
||||
│ Local Verify Log Entry ID │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### 4.2 Checking Rekor Entry
|
||||
|
||||
```bash
|
||||
# Verify with Rekor check
|
||||
stella proof verify --bundle bundle.tar.gz --check-rekor
|
||||
|
||||
# Get Rekor entry details
|
||||
stella proof rekor-entry --bundle bundle.tar.gz
|
||||
```
|
||||
|
||||
Output:
|
||||
```
|
||||
Rekor Entry
|
||||
══════════════════════════════════════════
|
||||
Log Index: 12345678
|
||||
Entry UUID: 24296fb24b8ad77a...
|
||||
Log ID: c0d23d6ad406973f...
|
||||
Integrated: 2025-01-15T10:30:05Z
|
||||
|
||||
Inclusion Proof:
|
||||
Root Hash: sha256:rekor-root...
|
||||
Tree Size: 98765432
|
||||
Hashes: [sha256:a1b2..., sha256:c3d4...]
|
||||
|
||||
Verification: ✓ INCLUDED
|
||||
```
|
||||
|
||||
### 4.3 Manual Rekor Verification
|
||||
|
||||
```bash
|
||||
# Using rekor-cli
|
||||
rekor-cli verify --artifact bundle.tar.gz \
|
||||
--signature bundle/dsse-envelope.json \
|
||||
--public-key bundle/certificate.pem
|
||||
|
||||
# Search for entries
|
||||
rekor-cli search --sha sha256:abc123...
|
||||
```
|
||||
|
||||
### 4.4 When Rekor is Unavailable
|
||||
|
||||
If Rekor is temporarily unavailable:
|
||||
|
||||
1. Verification still succeeds for DSSE and Merkle checks
|
||||
2. Rekor check is marked as "SKIPPED"
|
||||
3. Re-verify later when Rekor is available
|
||||
|
||||
```bash
|
||||
# Skip Rekor check
|
||||
stella proof verify --bundle bundle.tar.gz --skip-rekor
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Troubleshooting
|
||||
|
||||
### 5.1 DSSE Signature Invalid
|
||||
|
||||
**Symptoms**: `DSSE signature verification failed`
|
||||
|
||||
**Diagnostic Steps**:
|
||||
|
||||
1. Extract and inspect envelope:
|
||||
```bash
|
||||
tar -xzf bundle.tar.gz
|
||||
cat bundle/dsse-envelope.json | jq .
|
||||
```
|
||||
|
||||
2. Check payload type:
|
||||
```bash
|
||||
cat bundle/dsse-envelope.json | jq -r '.payloadType'
|
||||
# Expected: application/vnd.stellaops.proof+json
|
||||
```
|
||||
|
||||
3. Verify signature format:
|
||||
```bash
|
||||
cat bundle/dsse-envelope.json | jq '.signatures[0].sig' | base64 -d | xxd | head
|
||||
```
|
||||
|
||||
**Common Causes**:
|
||||
|
||||
| Cause | Resolution |
|
||||
|-------|------------|
|
||||
| Corrupted bundle | Re-download from API |
|
||||
| Wrong public key | Check trust roots configuration |
|
||||
| Signature algorithm mismatch | Verify ECDSA-P256 or RSA support |
|
||||
| Encoding issue | Check Base64 encoding |
|
||||
|
||||
### 5.2 Merkle Root Mismatch
|
||||
|
||||
**Symptoms**: `Merkle root does not match expected value`
|
||||
|
||||
**Diagnostic Steps**:
|
||||
|
||||
1. Recompute Merkle root locally:
|
||||
```bash
|
||||
stella proof compute-root --bundle bundle.tar.gz
|
||||
```
|
||||
|
||||
2. Compare manifest hashes:
|
||||
```bash
|
||||
cat bundle/manifest.json | jq '.hashes'
|
||||
```
|
||||
|
||||
3. Check for trailing whitespace or encoding:
|
||||
```bash
|
||||
sha256sum bundle/manifest.json
|
||||
```
|
||||
|
||||
**Resolution**:
|
||||
- Bundle may have been modified after signing
|
||||
- Re-export bundle from source system
|
||||
- If legitimate change, re-sign bundle
|
||||
|
||||
### 5.3 Certificate Chain Validation Failed
|
||||
|
||||
**Symptoms**: `Certificate chain verification failed`
|
||||
|
||||
**Diagnostic Steps**:
|
||||
|
||||
1. Check certificate expiry:
|
||||
```bash
|
||||
openssl x509 -in bundle/certificate.pem -noout -dates
|
||||
```
|
||||
|
||||
2. Verify chain:
|
||||
```bash
|
||||
openssl verify -verbose -CAfile /etc/stellaops/ca-bundle.pem bundle/certificate.pem
|
||||
```
|
||||
|
||||
3. Check for missing intermediates:
|
||||
```bash
|
||||
openssl x509 -in bundle/certificate.pem -noout -issuer
|
||||
```
|
||||
|
||||
**Common Errors**:
|
||||
|
||||
| Error | Cause | Fix |
|
||||
|-------|-------|-----|
|
||||
| `certificate has expired` | Cert past validity | Re-sign with valid cert |
|
||||
| `unable to get issuer certificate` | Missing intermediate | Update CA bundle |
|
||||
| `certificate revoked` | Key compromised | Use new signing key |
|
||||
| `self-signed certificate` | Wrong trust root | Import correct CA |
|
||||
|
||||
### 5.4 Bundle Extraction Fails
|
||||
|
||||
**Symptoms**: `Failed to extract bundle` or `Invalid archive format`
|
||||
|
||||
**Diagnostic Steps**:
|
||||
|
||||
1. Check file type:
|
||||
```bash
|
||||
file bundle.tar.gz
|
||||
```
|
||||
|
||||
2. Test archive integrity:
|
||||
```bash
|
||||
gzip -t bundle.tar.gz
|
||||
tar -tzf bundle.tar.gz
|
||||
```
|
||||
|
||||
3. Check for truncation:
|
||||
```bash
|
||||
ls -la bundle.tar.gz
|
||||
# Compare with expected size from API
|
||||
```
|
||||
|
||||
**Resolution**:
|
||||
- Re-download if corrupted
|
||||
- Check network transfer (use checksums)
|
||||
- Verify sufficient disk space
|
||||
|
||||
---
|
||||
|
||||
## 6. Monitoring & Alerting
|
||||
|
||||
### 6.1 Key Metrics
|
||||
|
||||
| Metric | Description | Alert Threshold |
|
||||
|--------|-------------|-----------------|
|
||||
| `proof_verification_total` | Total verifications | Baseline |
|
||||
| `proof_verification_failures` | Failed verifications | > 5/hour |
|
||||
| `proof_verification_duration_ms` | Verification latency | p99 > 5s |
|
||||
| `certificate_expiry_days` | Days until cert expiry | < 30 days |
|
||||
| `rekor_verification_failures` | Rekor check failures | > 0 (warning) |
|
||||
|
||||
### 6.2 Grafana Queries
|
||||
|
||||
```promql
|
||||
# Verification success rate
|
||||
sum(rate(proof_verification_success_total[1h])) /
|
||||
sum(rate(proof_verification_total[1h])) * 100
|
||||
|
||||
# Verification latency
|
||||
histogram_quantile(0.99, rate(proof_verification_duration_ms_bucket[5m]))
|
||||
|
||||
# Certificate expiry countdown
|
||||
min(certificate_expiry_days) by (certificate_id)
|
||||
|
||||
# Failures by type
|
||||
sum by (failure_reason) (rate(proof_verification_failures_total[1h]))
|
||||
```
|
||||
|
||||
### 6.3 Alert Rules
|
||||
|
||||
```yaml
|
||||
groups:
|
||||
- name: proof-verification
|
||||
rules:
|
||||
- alert: ProofVerificationFailuresHigh
|
||||
expr: rate(proof_verification_failures_total[1h]) > 5
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: High proof verification failure rate
|
||||
|
||||
- alert: SigningCertificateExpiringSoon
|
||||
expr: certificate_expiry_days < 30
|
||||
for: 1h
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: Signing certificate expires in {{ $value }} days
|
||||
|
||||
- alert: SigningCertificateExpired
|
||||
expr: certificate_expiry_days <= 0
|
||||
for: 1m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: Signing certificate has expired
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7. Escalation Procedures
|
||||
|
||||
### 7.1 Escalation Matrix
|
||||
|
||||
| Severity | Condition | Response Time | Escalate To |
|
||||
|----------|-----------|---------------|-------------|
|
||||
| P1 - Critical | Signing certificate expired | Immediate | Security Team + Platform Lead |
|
||||
| P1 - Critical | Mass verification failures | 15 minutes | Platform Team |
|
||||
| P2 - High | Rekor unavailable | 1 hour | Platform Team |
|
||||
| P3 - Medium | Single verification failure | 4 hours | Support Queue |
|
||||
| P4 - Low | Certificate expiring (>7 days) | Next sprint | Security Team |
|
||||
|
||||
### 7.2 P1: Certificate Expired Response
|
||||
|
||||
1. **Immediate Actions** (0-15 min):
|
||||
- Stop accepting new scans (if signing required)
|
||||
- Notify stakeholders
|
||||
- Begin emergency certificate rotation
|
||||
|
||||
2. **Certificate Rotation** (15-60 min):
|
||||
```bash
|
||||
# Generate new certificate
|
||||
stella signer cert rotate --emergency
|
||||
|
||||
# Verify new certificate
|
||||
stella signer cert show --current
|
||||
|
||||
# Resume operations
|
||||
stella signer status
|
||||
```
|
||||
|
||||
3. **Post-Incident**:
|
||||
- Implement certificate expiry monitoring
|
||||
- Schedule proactive rotations
|
||||
- Update runbooks
|
||||
|
||||
### 7.3 Contacts
|
||||
|
||||
| Role | Contact | Availability |
|
||||
|------|---------|--------------|
|
||||
| Security Team | security@stellaops.io | Business hours |
|
||||
| Platform On-Call | platform-oncall@stellaops.io | 24/7 |
|
||||
| Attestor Team | attestor-team@stellaops.io | Business hours |
|
||||
|
||||
---
|
||||
|
||||
## Appendix A: DSSE Envelope Format
|
||||
|
||||
```json
|
||||
{
|
||||
"payloadType": "application/vnd.stellaops.proof+json",
|
||||
"payload": "<base64-encoded-proof>",
|
||||
"signatures": [
|
||||
{
|
||||
"keyid": "sha256:signing-key-fingerprint",
|
||||
"sig": "<base64-encoded-signature>"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Payload Structure
|
||||
|
||||
```json
|
||||
{
|
||||
"_type": "https://stellaops.io/proof/v1",
|
||||
"subject": [
|
||||
{
|
||||
"name": "scan-123",
|
||||
"digest": {
|
||||
"sha256": "abc123..."
|
||||
}
|
||||
}
|
||||
],
|
||||
"predicateType": "https://stellaops.io/attestation/score/v1",
|
||||
"predicate": {
|
||||
"manifest": {
|
||||
"sbomHash": "sha256:...",
|
||||
"rulesHash": "sha256:...",
|
||||
"policyHash": "sha256:...",
|
||||
"feedHash": "sha256:..."
|
||||
},
|
||||
"score": 7.5,
|
||||
"rootHash": "sha256:...",
|
||||
"timestamp": "2025-01-15T10:30:00Z"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Appendix B: CLI Quick Reference
|
||||
|
||||
```bash
|
||||
# Verification Commands
|
||||
stella proof verify --bundle <path> # Verify bundle
|
||||
stella proof verify --bundle <path> --offline # Offline verification
|
||||
stella proof verify --bundle <path> --verbose # Detailed output
|
||||
stella proof verify --bundle <path> --check-rekor # Include Rekor check
|
||||
|
||||
# Inspection Commands
|
||||
stella proof spine --bundle <path> # Show Merkle tree
|
||||
stella proof show --bundle <path> # Show bundle contents
|
||||
stella proof rekor-entry --bundle <path> # Show Rekor entry
|
||||
|
||||
# Offline Kit
|
||||
stella proof offline-kit create --output <dir> # Create offline kit
|
||||
stella proof offline-kit verify --kit <dir> --bundle <path> # Use kit
|
||||
|
||||
# Certificate Commands
|
||||
stella signer cert show # Show current cert
|
||||
stella signer cert rotate # Rotate certificate
|
||||
stella signer cert export --output <path> # Export public cert
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Revision History
|
||||
|
||||
| Version | Date | Author | Changes |
|
||||
|---------|------|--------|---------|
|
||||
| 1.0.0 | 2025-12-20 | Agent | Initial release |
|
||||
Reference in New Issue
Block a user