git.stella-ops.org/docs/operations/proof-verification-runbook.md

# Proof Verification Operations Runbook

> **Version**: 1.0.0  
> **Sprint**: 3500.0004.0004  
> **Last Updated**: 2025-12-20

This runbook covers operational procedures for Proof Verification, including DSSE signature validation, Merkle tree verification, transparency log checks, and offline verification workflows.

---

## Table of Contents

1. [Overview](#1-overview)
2. [Verification Operations](#2-verification-operations)
3. [Offline Verification](#3-offline-verification)
4. [Transparency Log Integration](#4-transparency-log-integration)
5. [Troubleshooting](#5-troubleshooting)
6. [Monitoring & Alerting](#6-monitoring--alerting)
7. [Escalation Procedures](#7-escalation-procedures)

---

## 1. Overview

### What is Proof Verification?

Proof Verification is the process of cryptographically validating that a scan result has not been tampered with and was produced by an authorized StellaOps instance. It involves:

- **DSSE Signature Verification**: Validate the signing envelope
- **Merkle Tree Verification**: Confirm the root hash matches the proof
- **Certificate Chain Validation**: Verify the signing certificate
- **Transparency Log Check**: Optional Rekor/Sigstore verification

### Verification Components

| Component | Purpose | Verification Type |
|-----------|---------|-------------------|
| DSSE Envelope | Contains signed payload | Signature validation |
| Merkle Proof | Cryptographic proof of inclusion | Hash verification |
| Certificate | Signing identity | Chain validation |
| Rekor Entry | Transparency log record | Log inclusion proof |

### Trust Model

```
┌─────────────────────────────────────────────────────────────┐
│                    Trust Hierarchy                          │
├─────────────────────────────────────────────────────────────┤
│  Root CA (Offline)                                          │
│    └── Intermediate CA                                      │
│          └── Signing Certificate (Scanner Instance)         │
│                └── DSSE Envelope                            │
│                      └── Proof Bundle                       │
│                            └── Manifest + Score             │
└─────────────────────────────────────────────────────────────┘
```

---

## 2. Verification Operations

### 2.1 Basic Proof Verification

#### Via CLI

```bash
# Verify a proof bundle file
stella proof verify --bundle bundle.tar.gz

# Verify with verbose output
stella proof verify --bundle bundle.tar.gz --verbose

# Verify and output as JSON
stella proof verify --bundle bundle.tar.gz --output json
```

#### Expected Output (Success)

```
Proof Verification Result
══════════════════════════════════════════
✓ DSSE Signature      VALID
✓ Merkle Root         VALID
✓ Certificate Chain   VALID
✓ Not Expired         VALID
──────────────────────────────────────────
Overall: VERIFIED

Root Hash:   sha256:abc123...
Signed By:   scanner-prod-01.stellaops.local
Signed At:   2025-01-15T10:30:00Z
Valid Until: 2026-01-15T10:30:00Z
```

#### Expected Output (Failure)

```
Proof Verification Result
══════════════════════════════════════════
✓ DSSE Signature      VALID
✗ Merkle Root         INVALID
✓ Certificate Chain   VALID
✓ Not Expired         VALID
──────────────────────────────────────────
Overall: FAILED

Error: Merkle root mismatch
  Expected: sha256:abc123...
  Actual:   sha256:def456...
```

### 2.2 Verification via API

```bash
# Verify by scan ID
curl -X POST "https://scanner.stellaops.local/api/v1/scanner/scans/$SCAN_ID/proofs/$ROOT_HASH/verify" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json"

# Response
{
  "valid": true,
  "rootHash": "sha256:abc123...",
  "checks": [
    {"name": "dsse_signature", "passed": true, "message": "Signature valid"},
    {"name": "merkle_root", "passed": true, "message": "Root hash matches"},
    {"name": "certificate_chain", "passed": true, "message": "Chain valid"},
    {"name": "not_expired", "passed": true, "message": "Certificate not expired"}
  ],
  "verifiedAt": "2025-01-16T10:30:00Z"
}
```

### 2.3 Viewing Merkle Spine

The Merkle spine shows the path from leaf nodes to the root:

```bash
stella proof spine --bundle bundle.tar.gz
```

Output:
```
Merkle Tree Spine
══════════════════════════════════════════
Root:   sha256:abc123...
        ├── sha256:node1... (sbom_hash)
        ├── sha256:node2... (rules_hash)
        ├── sha256:node3... (policy_hash)
        └── sha256:node4... (feed_hash)

Depth: 3
Leaves: 4
Algorithm: SHA-256
```

### 2.4 Certificate Inspection

```bash
# Extract and inspect certificate
tar -xzf bundle.tar.gz
openssl x509 -in bundle/certificate.pem -noout -text

# Check validity period
openssl x509 -in bundle/certificate.pem -noout -dates

# Verify against CA bundle
openssl verify -CAfile /etc/stellaops/ca-bundle.pem bundle/certificate.pem
```

---

## 3. Offline Verification

### 3.1 When to Use Offline Verification

- Air-gapped environments
- Network-restricted systems
- Compliance audits without API access
- Disaster recovery scenarios

### 3.2 Prerequisites for Offline Verification

Required files:
- Proof bundle (`.tar.gz`)
- CA certificate bundle (`ca-bundle.pem`)
- Trust root configuration (`trust-roots.json`)

```bash
# Prepare offline verification kit
stella proof offline-kit create \
  --output /path/to/offline-kit/ \
  --include-ca \
  --include-trust-roots
```

Kit contents:
```
offline-kit/
├── ca-bundle.pem           # Certificate authority chain
├── trust-roots.json        # Trusted signing keys
├── verify.sh               # Standalone verification script
└── README.md               # Instructions
```

### 3.3 Running Offline Verification

```bash
# Using CLI with offline flag
stella proof verify --bundle bundle.tar.gz --offline

# Using standalone script
./verify.sh bundle.tar.gz

# Manual verification with OpenSSL
./verify.sh bundle.tar.gz --ca-bundle ./ca-bundle.pem
```

### 3.4 Offline Verification Checks

| Check | Online | Offline | Notes |
|-------|--------|---------|-------|
| DSSE Signature | ✓ | ✓ | Local crypto |
| Merkle Root | ✓ | ✓ | Local hash computation |
| Certificate Chain | ✓ | ✓ | Requires CA bundle |
| Certificate Revocation | ✓ | ✗ | Needs CRL/OCSP |
| Rekor Transparency | ✓ | ✗ | Needs network |

### 3.5 Air-Gap Considerations

For fully air-gapped environments:

1. **Pre-stage CA bundle**:
   ```bash
   # On connected system
   stella ca export --output ca-bundle.pem
   
   # Transfer to air-gapped system
   scp ca-bundle.pem airgap:/etc/stellaops/
   ```

2. **Pre-stage CRL (optional)**:
   ```bash
   # Download latest CRL
   curl -o crl.pem https://ca.stellaops.io/crl/latest.pem
   
   # Transfer and use
   stella proof verify --bundle bundle.tar.gz --offline --crl crl.pem
   ```

---

## 4. Transparency Log Integration

### 4.1 Rekor Overview

StellaOps optionally publishes proof attestations to Sigstore Rekor for immutable transparency logging.

```
┌─────────────────────────────────────────────────────────────┐
│                  Transparency Flow                          │
├─────────────────────────────────────────────────────────────┤
│  Proof Bundle                                               │
│       │                                                     │
│       ▼                                                     │
│  DSSE Envelope ──────► Rekor ──────► Inclusion Proof        │
│       │                  │                                  │
│       ▼                  ▼                                  │
│  Local Verify        Log Entry ID                           │
└─────────────────────────────────────────────────────────────┘
```

### 4.2 Checking Rekor Entry

```bash
# Verify with Rekor check
stella proof verify --bundle bundle.tar.gz --check-rekor

# Get Rekor entry details
stella proof rekor-entry --bundle bundle.tar.gz
```

Output:
```
Rekor Entry
══════════════════════════════════════════
Log Index:    12345678
Entry UUID:   24296fb24b8ad77a...
Log ID:       c0d23d6ad406973f...
Integrated:   2025-01-15T10:30:05Z

Inclusion Proof:
  Root Hash:  sha256:rekor-root...
  Tree Size:  98765432
  Hashes:     [sha256:a1b2..., sha256:c3d4...]

Verification: ✓ INCLUDED
```

### 4.3 Manual Rekor Verification

```bash
# Using rekor-cli
rekor-cli verify --artifact bundle.tar.gz \
  --signature bundle/dsse-envelope.json \
  --public-key bundle/certificate.pem

# Search for entries
rekor-cli search --sha sha256:abc123...
```

### 4.4 When Rekor is Unavailable

If Rekor is temporarily unavailable:

1. Verification still succeeds for DSSE and Merkle checks
2. Rekor check is marked as "SKIPPED"
3. Re-verify later when Rekor is available

```bash
# Skip Rekor check
stella proof verify --bundle bundle.tar.gz --skip-rekor
```

---

## 5. Troubleshooting

### 5.1 DSSE Signature Invalid

**Symptoms**: `DSSE signature verification failed`

**Diagnostic Steps**:

1. Extract and inspect envelope:
   ```bash
   tar -xzf bundle.tar.gz
   cat bundle/dsse-envelope.json | jq .
   ```

2. Check payload type:
   ```bash
   cat bundle/dsse-envelope.json | jq -r '.payloadType'
   # Expected: application/vnd.stellaops.proof+json
   ```

3. Verify signature format:
   ```bash
   cat bundle/dsse-envelope.json | jq '.signatures[0].sig' | base64 -d | xxd | head
   ```

**Common Causes**:

| Cause | Resolution |
|-------|------------|
| Corrupted bundle | Re-download from API |
| Wrong public key | Check trust roots configuration |
| Signature algorithm mismatch | Verify ECDSA-P256 or RSA support |
| Encoding issue | Check Base64 encoding |

### 5.2 Merkle Root Mismatch

**Symptoms**: `Merkle root does not match expected value`

**Diagnostic Steps**:

1. Recompute Merkle root locally:
   ```bash
   stella proof compute-root --bundle bundle.tar.gz
   ```

2. Compare manifest hashes:
   ```bash
   cat bundle/manifest.json | jq '.hashes'
   ```

3. Check for trailing whitespace or encoding:
   ```bash
   sha256sum bundle/manifest.json
   ```

**Resolution**:
- Bundle may have been modified after signing
- Re-export bundle from source system
- If legitimate change, re-sign bundle

### 5.3 Certificate Chain Validation Failed

**Symptoms**: `Certificate chain verification failed`

**Diagnostic Steps**:

1. Check certificate expiry:
   ```bash
   openssl x509 -in bundle/certificate.pem -noout -dates
   ```

2. Verify chain:
   ```bash
   openssl verify -verbose -CAfile /etc/stellaops/ca-bundle.pem bundle/certificate.pem
   ```

3. Check for missing intermediates:
   ```bash
   openssl x509 -in bundle/certificate.pem -noout -issuer
   ```

**Common Errors**:

| Error | Cause | Fix |
|-------|-------|-----|
| `certificate has expired` | Cert past validity | Re-sign with valid cert |
| `unable to get issuer certificate` | Missing intermediate | Update CA bundle |
| `certificate revoked` | Key compromised | Use new signing key |
| `self-signed certificate` | Wrong trust root | Import correct CA |

### 5.4 Bundle Extraction Fails

**Symptoms**: `Failed to extract bundle` or `Invalid archive format`

**Diagnostic Steps**:

1. Check file type:
   ```bash
   file bundle.tar.gz
   ```

2. Test archive integrity:
   ```bash
   gzip -t bundle.tar.gz
   tar -tzf bundle.tar.gz
   ```

3. Check for truncation:
   ```bash
   ls -la bundle.tar.gz
   # Compare with expected size from API
   ```

**Resolution**:
- Re-download if corrupted
- Check network transfer (use checksums)
- Verify sufficient disk space

---

## 6. Monitoring & Alerting

### 6.1 Key Metrics

| Metric | Description | Alert Threshold |
|--------|-------------|-----------------|
| `proof_verification_total` | Total verifications | Baseline |
| `proof_verification_failures` | Failed verifications | > 5/hour |
| `proof_verification_duration_ms` | Verification latency | p99 > 5s |
| `certificate_expiry_days` | Days until cert expiry | < 30 days |
| `rekor_verification_failures` | Rekor check failures | > 0 (warning) |

### 6.2 Grafana Queries

```promql
# Verification success rate
sum(rate(proof_verification_success_total[1h])) / 
sum(rate(proof_verification_total[1h])) * 100

# Verification latency
histogram_quantile(0.99, rate(proof_verification_duration_ms_bucket[5m]))

# Certificate expiry countdown
min(certificate_expiry_days) by (certificate_id)

# Failures by type
sum by (failure_reason) (rate(proof_verification_failures_total[1h]))
```

### 6.3 Alert Rules

```yaml
groups:
  - name: proof-verification
    rules:
      - alert: ProofVerificationFailuresHigh
        expr: rate(proof_verification_failures_total[1h]) > 5
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: High proof verification failure rate

      - alert: SigningCertificateExpiringSoon
        expr: certificate_expiry_days < 30
        for: 1h
        labels:
          severity: warning
        annotations:
          summary: Signing certificate expires in {{ $value }} days

      - alert: SigningCertificateExpired
        expr: certificate_expiry_days <= 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: Signing certificate has expired
```

---

## 7. Escalation Procedures

### 7.1 Escalation Matrix

| Severity | Condition | Response Time | Escalate To |
|----------|-----------|---------------|-------------|
| P1 - Critical | Signing certificate expired | Immediate | Security Team + Platform Lead |
| P1 - Critical | Mass verification failures | 15 minutes | Platform Team |
| P2 - High | Rekor unavailable | 1 hour | Platform Team |
| P3 - Medium | Single verification failure | 4 hours | Support Queue |
| P4 - Low | Certificate expiring (>7 days) | Next sprint | Security Team |

### 7.2 P1: Certificate Expired Response

1. **Immediate Actions** (0-15 min):
   - Stop accepting new scans (if signing required)
   - Notify stakeholders
   - Begin emergency certificate rotation

2. **Certificate Rotation** (15-60 min):
   ```bash
   # Generate new certificate
   stella signer cert rotate --emergency
   
   # Verify new certificate
   stella signer cert show --current
   
   # Resume operations
   stella signer status
   ```

3. **Post-Incident**:
   - Implement certificate expiry monitoring
   - Schedule proactive rotations
   - Update runbooks

### 7.3 Contacts

| Role | Contact | Availability |
|------|---------|--------------|
| Security Team | security@stellaops.io | Business hours |
| Platform On-Call | platform-oncall@stellaops.io | 24/7 |
| Attestor Team | attestor-team@stellaops.io | Business hours |

---

## Appendix A: DSSE Envelope Format

```json
{
  "payloadType": "application/vnd.stellaops.proof+json",
  "payload": "<base64-encoded-proof>",
  "signatures": [
    {
      "keyid": "sha256:signing-key-fingerprint",
      "sig": "<base64-encoded-signature>"
    }
  ]
}
```

### Payload Structure

```json
{
  "_type": "https://stellaops.io/proof/v1",
  "subject": [
    {
      "name": "scan-123",
      "digest": {
        "sha256": "abc123..."
      }
    }
  ],
  "predicateType": "https://stellaops.io/attestation/score/v1",
  "predicate": {
    "manifest": {
      "sbomHash": "sha256:...",
      "rulesHash": "sha256:...",
      "policyHash": "sha256:...",
      "feedHash": "sha256:..."
    },
    "score": 7.5,
    "rootHash": "sha256:...",
    "timestamp": "2025-01-15T10:30:00Z"
  }
}
```

---

## Appendix B: CLI Quick Reference

```bash
# Verification Commands
stella proof verify --bundle <path>           # Verify bundle
stella proof verify --bundle <path> --offline # Offline verification
stella proof verify --bundle <path> --verbose # Detailed output
stella proof verify --bundle <path> --check-rekor  # Include Rekor check

# Inspection Commands
stella proof spine --bundle <path>            # Show Merkle tree
stella proof show --bundle <path>             # Show bundle contents
stella proof rekor-entry --bundle <path>      # Show Rekor entry

# Offline Kit
stella proof offline-kit create --output <dir>  # Create offline kit
stella proof offline-kit verify --kit <dir> --bundle <path>  # Use kit

# Certificate Commands
stella signer cert show                       # Show current cert
stella signer cert rotate                     # Rotate certificate
stella signer cert export --output <path>     # Export public cert
```

---

## Revision History

| Version | Date | Author | Changes |
|---------|------|--------|---------|
| 1.0.0 | 2025-12-20 | Agent | Initial release |