git.stella-ops.org/docs/operations/proof-verification-runbook.md

Proof Verification Operations Runbook

Version: 1.0.0
Sprint: 3500.0004.0004
Last Updated: 2025-12-20

This runbook covers operational procedures for Proof Verification, including DSSE signature validation, Merkle tree verification, transparency log checks, and offline verification workflows.

---

Table of Contents

1. Overview
2. Verification Operations
3. Offline Verification
4. Transparency Log Integration
5. Troubleshooting
6. Monitoring & Alerting
7. Escalation Procedures

---

1. Overview

What is Proof Verification?

Proof Verification is the process of cryptographically validating that a scan result has not been tampered with and was produced by an authorized StellaOps instance. It involves:

• DSSE Signature Verification: Validate the signing envelope
• Merkle Tree Verification: Confirm the root hash matches the proof
• Certificate Chain Validation: Verify the signing certificate
• Transparency Log Check: Optional Rekor/Sigstore verification

Verification Components

Component	Purpose	Verification Type
DSSE Envelope	Contains signed payload	Signature validation
Merkle Proof	Cryptographic proof of inclusion	Hash verification
Certificate	Signing identity	Chain validation
Rekor Entry	Transparency log record	Log inclusion proof

Trust Model

┌─────────────────────────────────────────────────────────────┐
│                    Trust Hierarchy                          │
├─────────────────────────────────────────────────────────────┤
│  Root CA (Offline)                                          │
│    └── Intermediate CA                                      │
│          └── Signing Certificate (Scanner Instance)         │
│                └── DSSE Envelope                            │
│                      └── Proof Bundle                       │
│                            └── Manifest + Score             │
└─────────────────────────────────────────────────────────────┘

---

2. Verification Operations

2.1 Basic Proof Verification

Via CLI

# Verify a proof bundle file
stella proof verify --bundle bundle.tar.gz

# Verify with verbose output
stella proof verify --bundle bundle.tar.gz --verbose

# Verify and output as JSON
stella proof verify --bundle bundle.tar.gz --output json

Expected Output (Success)

Proof Verification Result
══════════════════════════════════════════
✓ DSSE Signature      VALID
✓ Merkle Root         VALID
✓ Certificate Chain   VALID
✓ Not Expired         VALID
──────────────────────────────────────────
Overall: VERIFIED

Root Hash:   sha256:abc123...
Signed By:   scanner-prod-01.stellaops.local
Signed At:   2025-01-15T10:30:00Z
Valid Until: 2026-01-15T10:30:00Z

Expected Output (Failure)

Proof Verification Result
══════════════════════════════════════════
✓ DSSE Signature      VALID
✗ Merkle Root         INVALID
✓ Certificate Chain   VALID
✓ Not Expired         VALID
──────────────────────────────────────────
Overall: FAILED

Error: Merkle root mismatch
  Expected: sha256:abc123...
  Actual:   sha256:def456...

2.2 Verification via API

# Verify by scan ID
curl -X POST "https://scanner.stellaops.local/api/v1/scanner/scans/$SCAN_ID/proofs/$ROOT_HASH/verify" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json"

# Response
{
  "valid": true,
  "rootHash": "sha256:abc123...",
  "checks": [
    {"name": "dsse_signature", "passed": true, "message": "Signature valid"},
    {"name": "merkle_root", "passed": true, "message": "Root hash matches"},
    {"name": "certificate_chain", "passed": true, "message": "Chain valid"},
    {"name": "not_expired", "passed": true, "message": "Certificate not expired"}
  ],
  "verifiedAt": "2025-01-16T10:30:00Z"
}

2.3 Viewing Merkle Spine

The Merkle spine shows the path from leaf nodes to the root:

stella proof spine --bundle bundle.tar.gz

Output:

Merkle Tree Spine
══════════════════════════════════════════
Root:   sha256:abc123...
        ├── sha256:node1... (sbom_hash)
        ├── sha256:node2... (rules_hash)
        ├── sha256:node3... (policy_hash)
        └── sha256:node4... (feed_hash)

Depth: 3
Leaves: 4
Algorithm: SHA-256

2.4 Certificate Inspection

# Extract and inspect certificate
tar -xzf bundle.tar.gz
openssl x509 -in bundle/certificate.pem -noout -text

# Check validity period
openssl x509 -in bundle/certificate.pem -noout -dates

# Verify against CA bundle
openssl verify -CAfile /etc/stellaops/ca-bundle.pem bundle/certificate.pem

---

3. Offline Verification

3.1 When to Use Offline Verification

• Air-gapped environments
• Network-restricted systems
• Compliance audits without API access
• Disaster recovery scenarios

3.2 Prerequisites for Offline Verification

Required files:

• Proof bundle (.tar.gz)
• CA certificate bundle (ca-bundle.pem)
• Trust root configuration (trust-roots.json)

# Prepare offline verification kit
stella proof offline-kit create \
  --output /path/to/offline-kit/ \
  --include-ca \
  --include-trust-roots

Kit contents:

offline-kit/
├── ca-bundle.pem           # Certificate authority chain
├── trust-roots.json        # Trusted signing keys
├── verify.sh               # Standalone verification script
└── README.md               # Instructions

3.3 Running Offline Verification

# Using CLI with offline flag
stella proof verify --bundle bundle.tar.gz --offline

# Using standalone script
./verify.sh bundle.tar.gz

# Manual verification with OpenSSL
./verify.sh bundle.tar.gz --ca-bundle ./ca-bundle.pem

3.4 Offline Verification Checks

Check	Online	Offline	Notes
DSSE Signature	✓	✓	Local crypto
Merkle Root	✓	✓	Local hash computation
Certificate Chain	✓	✓	Requires CA bundle
Certificate Revocation	✓	✗	Needs CRL/OCSP
Rekor Transparency	✓	✗	Needs network

3.5 Air-Gap Considerations

For fully air-gapped environments:

1. Pre-stage CA bundle:

   # On connected system
   stella ca export --output ca-bundle.pem
   
   # Transfer to air-gapped system
   scp ca-bundle.pem airgap:/etc/stellaops/
   

2. Pre-stage CRL (optional):

   # Download latest CRL
   curl -o crl.pem https://ca.stellaops.io/crl/latest.pem
   
   # Transfer and use
   stella proof verify --bundle bundle.tar.gz --offline --crl crl.pem
   

---

4. Transparency Log Integration

4.1 Rekor Overview

StellaOps optionally publishes proof attestations to Sigstore Rekor for immutable transparency logging.

┌─────────────────────────────────────────────────────────────┐
│                  Transparency Flow                          │
├─────────────────────────────────────────────────────────────┤
│  Proof Bundle                                               │
│       │                                                     │
│       ▼                                                     │
│  DSSE Envelope ──────► Rekor ──────► Inclusion Proof        │
│       │                  │                                  │
│       ▼                  ▼                                  │
│  Local Verify        Log Entry ID                           │
└─────────────────────────────────────────────────────────────┘

4.2 Checking Rekor Entry

# Verify with Rekor check
stella proof verify --bundle bundle.tar.gz --check-rekor

# Get Rekor entry details
stella proof rekor-entry --bundle bundle.tar.gz

Output:

Rekor Entry
══════════════════════════════════════════
Log Index:    12345678
Entry UUID:   24296fb24b8ad77a...
Log ID:       c0d23d6ad406973f...
Integrated:   2025-01-15T10:30:05Z

Inclusion Proof:
  Root Hash:  sha256:rekor-root...
  Tree Size:  98765432
  Hashes:     [sha256:a1b2..., sha256:c3d4...]

Verification: ✓ INCLUDED

4.3 Manual Rekor Verification

# Using rekor-cli
rekor-cli verify --artifact bundle.tar.gz \
  --signature bundle/dsse-envelope.json \
  --public-key bundle/certificate.pem

# Search for entries
rekor-cli search --sha sha256:abc123...

4.4 When Rekor is Unavailable

If Rekor is temporarily unavailable:

1. Verification still succeeds for DSSE and Merkle checks
2. Rekor check is marked as "SKIPPED"
3. Re-verify later when Rekor is available

# Skip Rekor check
stella proof verify --bundle bundle.tar.gz --skip-rekor

---

5. Troubleshooting

5.1 DSSE Signature Invalid

Symptoms: DSSE signature verification failed

Diagnostic Steps:

1. Extract and inspect envelope:

   tar -xzf bundle.tar.gz
   cat bundle/dsse-envelope.json | jq .
   

2. Check payload type:

   cat bundle/dsse-envelope.json | jq -r '.payloadType'
   # Expected: application/vnd.stellaops.proof+json
   

3. Verify signature format:

   cat bundle/dsse-envelope.json | jq '.signatures[0].sig' | base64 -d | xxd | head
   

Common Causes:

Cause	Resolution
Corrupted bundle	Re-download from API
Wrong public key	Check trust roots configuration
Signature algorithm mismatch	Verify ECDSA-P256 or RSA support
Encoding issue	Check Base64 encoding

5.2 Merkle Root Mismatch

Symptoms: Merkle root does not match expected value

Diagnostic Steps:

1. Recompute Merkle root locally:

   stella proof compute-root --bundle bundle.tar.gz
   

2. Compare manifest hashes:

   cat bundle/manifest.json | jq '.hashes'
   

3. Check for trailing whitespace or encoding:

   sha256sum bundle/manifest.json
   

Resolution:

• Bundle may have been modified after signing
• Re-export bundle from source system
• If legitimate change, re-sign bundle

5.3 Certificate Chain Validation Failed

Symptoms: Certificate chain verification failed

Diagnostic Steps:

1. Check certificate expiry:

   openssl x509 -in bundle/certificate.pem -noout -dates
   

2. Verify chain:

   openssl verify -verbose -CAfile /etc/stellaops/ca-bundle.pem bundle/certificate.pem
   

3. Check for missing intermediates:

   openssl x509 -in bundle/certificate.pem -noout -issuer
   

Common Errors:

Error	Cause	Fix
certificate has expired	Cert past validity	Re-sign with valid cert
unable to get issuer certificate	Missing intermediate	Update CA bundle
certificate revoked	Key compromised	Use new signing key
self-signed certificate	Wrong trust root	Import correct CA

5.4 Bundle Extraction Fails

Symptoms: Failed to extract bundle or Invalid archive format

Diagnostic Steps:

1. Check file type:

   file bundle.tar.gz
   

2. Test archive integrity:

   gzip -t bundle.tar.gz
   tar -tzf bundle.tar.gz
   

3. Check for truncation:

   ls -la bundle.tar.gz
   # Compare with expected size from API
   

Resolution:

• Re-download if corrupted
• Check network transfer (use checksums)
• Verify sufficient disk space

---

6. Monitoring & Alerting

6.1 Key Metrics

Metric	Description	Alert Threshold
proof_verification_total	Total verifications	Baseline
proof_verification_failures	Failed verifications	> 5/hour
proof_verification_duration_ms	Verification latency	p99 > 5s
certificate_expiry_days	Days until cert expiry	< 30 days
rekor_verification_failures	Rekor check failures	> 0 (warning)

6.2 Grafana Queries

# Verification success rate
sum(rate(proof_verification_success_total[1h])) / 
sum(rate(proof_verification_total[1h])) * 100

# Verification latency
histogram_quantile(0.99, rate(proof_verification_duration_ms_bucket[5m]))

# Certificate expiry countdown
min(certificate_expiry_days) by (certificate_id)

# Failures by type
sum by (failure_reason) (rate(proof_verification_failures_total[1h]))

6.3 Alert Rules

groups:
  - name: proof-verification
    rules:
      - alert: ProofVerificationFailuresHigh
        expr: rate(proof_verification_failures_total[1h]) > 5
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: High proof verification failure rate

      - alert: SigningCertificateExpiringSoon
        expr: certificate_expiry_days < 30
        for: 1h
        labels:
          severity: warning
        annotations:
          summary: Signing certificate expires in {{ $value }} days

      - alert: SigningCertificateExpired
        expr: certificate_expiry_days <= 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: Signing certificate has expired

---

7. Escalation Procedures

7.1 Escalation Matrix

Severity	Condition	Response Time	Escalate To
P1 - Critical	Signing certificate expired	Immediate	Security Team + Platform Lead
P1 - Critical	Mass verification failures	15 minutes	Platform Team
P2 - High	Rekor unavailable	1 hour	Platform Team
P3 - Medium	Single verification failure	4 hours	Support Queue
P4 - Low	Certificate expiring (>7 days)	Next sprint	Security Team

7.2 P1: Certificate Expired Response

1. Immediate Actions (0-15 min):

   • Stop accepting new scans (if signing required)
   • Notify stakeholders
   • Begin emergency certificate rotation

2. Certificate Rotation (15-60 min):

   # Generate new certificate
   stella signer cert rotate --emergency
   
   # Verify new certificate
   stella signer cert show --current
   
   # Resume operations
   stella signer status
   

3. Post-Incident:

   • Implement certificate expiry monitoring
   • Schedule proactive rotations
   • Update runbooks

7.3 Contacts

Role	Contact	Availability
Security Team	security@stellaops.io	Business hours
Platform On-Call	platform-oncall@stellaops.io	24/7
Attestor Team	attestor-team@stellaops.io	Business hours

---

Appendix A: DSSE Envelope Format

{
  "payloadType": "application/vnd.stellaops.proof+json",
  "payload": "<base64-encoded-proof>",
  "signatures": [
    {
      "keyid": "sha256:signing-key-fingerprint",
      "sig": "<base64-encoded-signature>"
    }
  ]
}

Payload Structure

{
  "_type": "https://stellaops.io/proof/v1",
  "subject": [
    {
      "name": "scan-123",
      "digest": {
        "sha256": "abc123..."
      }
    }
  ],
  "predicateType": "https://stellaops.io/attestation/score/v1",
  "predicate": {
    "manifest": {
      "sbomHash": "sha256:...",
      "rulesHash": "sha256:...",
      "policyHash": "sha256:...",
      "feedHash": "sha256:..."
    },
    "score": 7.5,
    "rootHash": "sha256:...",
    "timestamp": "2025-01-15T10:30:00Z"
  }
}

---

Appendix B: CLI Quick Reference

# Verification Commands
stella proof verify --bundle <path>           # Verify bundle
stella proof verify --bundle <path> --offline # Offline verification
stella proof verify --bundle <path> --verbose # Detailed output
stella proof verify --bundle <path> --check-rekor  # Include Rekor check

# Inspection Commands
stella proof spine --bundle <path>            # Show Merkle tree
stella proof show --bundle <path>             # Show bundle contents
stella proof rekor-entry --bundle <path>      # Show Rekor entry

# Offline Kit
stella proof offline-kit create --output <dir>  # Create offline kit
stella proof offline-kit verify --kit <dir> --bundle <path>  # Use kit

# Certificate Commands
stella signer cert show                       # Show current cert
stella signer cert rotate                     # Rotate certificate
stella signer cert export --output <path>     # Export public cert

---

Revision History

Version	Date	Author	Changes
1.0.0	2025-12-20	Agent	Initial release