# Air-Gap Operations Runbook for Score Proofs & Reachability

> **Version**: 1.0.0  
> **Sprint**: 3500.0004.0004  
> **Last Updated**: 2025-12-20

This runbook covers air-gapped operations for Score Proofs and Reachability features, including offline kit deployment, proof verification, and bundle management.

---

## Table of Contents

1. [Overview](#1-overview)
2. [Offline Kit Deployment](#2-offline-kit-deployment)
3. [Score Proofs in Air-Gap Mode](#3-score-proofs-in-air-gap-mode)
4. [Reachability in Air-Gap Mode](#4-reachability-in-air-gap-mode)
5. [Bundle Import Operations](#5-bundle-import-operations)
6. [Proof Verification Offline](#6-proof-verification-offline)
7. [Troubleshooting](#7-troubleshooting)
8. [Monitoring & Alerting](#8-monitoring--alerting)

---

## 1. Overview

### Air-Gap Modes

| Mode | Network | Use Case |
|------|---------|----------|
| **Sealed** | No external connectivity | Classified environments |
| **Constrained** | Limited egress (allowlist) | Regulated networks |
| **Hybrid** | Selective connectivity | Standard enterprise |

### Score Proofs Air-Gap Capabilities

| Feature | Sealed | Constrained | Hybrid |
|---------|--------|-------------|--------|
| Score computation | ✅ | ✅ | ✅ |
| Score replay | ✅ | ✅ | ✅ |
| Proof generation | ✅ | ✅ | ✅ |
| Proof verification | ✅ | ✅ | ✅ |
| Rekor logging | ❌ | 🔶 (optional) | ✅ |
| Feed updates | Bundle | Bundle | Online |

### Reachability Air-Gap Capabilities

| Feature | Sealed | Constrained | Hybrid |
|---------|--------|-------------|--------|
| Call graph upload | ✅ | ✅ | ✅ |
| Reachability compute | ✅ | ✅ | ✅ |
| Explain queries | ✅ | ✅ | ✅ |
| Symbol resolution | Bundle | Bundle | Online |

---

## 2. Offline Kit Deployment

### 2.1 Offline Kit Contents

The offline kit contains everything needed for air-gapped Score Proofs and Reachability:

```
offline-kit/
├── manifests/
│   ├── kit-manifest.json          # Kit metadata and versions
│   ├── feed-manifest.json         # Advisory feed snapshot
│   └── vex-manifest.json          # VEX data snapshot
├── feeds/
│   ├── concelier/                 # Advisory feed data
│   │   ├── advisories.ndjson
│   │   └── snapshot.dsse.json
│   └── excititor/                 # VEX data
│       ├── vex-statements.ndjson
│       └── snapshot.dsse.json
├── policies/
│   ├── scoring-policy.yaml
│   └── policy.dsse.json
├── trust/
│   ├── trust-anchors.json         # Public keys for verification
│   └── time-anchor.json           # Time attestation
├── symbols/
│   └── symbol-index.db            # Symbol resolution database
└── tools/
    ├── stella-cli                 # CLI binary
    └── verify-kit.sh              # Kit verification script
```

### 2.2 Verify Kit Integrity

Before deployment, always verify the offline kit:

```bash
# Verify kit signature
stella airgap verify-kit --kit /path/to/offline-kit

# Output:
# Kit manifest: VALID
# Feed snapshot: VALID (sha256:feed123...)
# VEX snapshot: VALID (sha256:vex456...)
# Policy: VALID (sha256:policy789...)
# Trust anchors: VALID (3 anchors)
# Time anchor: VALID (expires: 2025-12-31T00:00:00Z)
# 
# Kit verification: PASSED

# Verify individual components
stella airgap verify --file feeds/concelier/snapshot.dsse.json
stella airgap verify --file feeds/excititor/snapshot.dsse.json
stella airgap verify --file policies/policy.dsse.json
```

### 2.3 Deploy Offline Kit

```bash
# Deploy kit (sealed mode)
stella airgap deploy --kit /path/to/offline-kit --mode sealed

# Deploy kit (constrained mode with limited egress)
stella airgap deploy --kit /path/to/offline-kit \
  --mode constrained \
  --egress-allowlist https://rekor.sigstore.dev

# Verify deployment
stella airgap status

# Output:
# Mode: sealed
# Kit version: 2025.12.20
# Feed snapshot: sha256:feed123... (2025-12-20)
# VEX snapshot: sha256:vex456... (2025-12-20)
# Policy: sha256:policy789... (v1.2.3)
# Trust anchors: 3 active
# Time anchor: Valid until 2025-12-31
# Staleness: OK (0 days)
```

### 2.4 Kit Updates

```bash
# Check for kit updates (requires external access or new media)
stella airgap check-update --current-kit /path/to/current-kit

# Import new kit
stella airgap import-kit --kit /path/to/new-kit --validate

# Rollback to previous kit
stella airgap rollback --generation previous
```

---

## 3. Score Proofs in Air-Gap Mode

### 3.1 Create Scan (Air-Gap)

```bash
# Create scan referencing offline kit snapshots
stella scan create --artifact sha256:abc123... \
  --airgap \
  --feed-snapshot sha256:feed123... \
  --vex-snapshot sha256:vex456... \
  --policy-snapshot sha256:policy789...

# Or auto-detect from deployed kit
stella scan create --artifact sha256:abc123... --use-offline-kit
```

### 3.2 Score Replay (Air-Gap)

```bash
# Replay with offline kit snapshots
stella score replay --scan-id $SCAN_ID --offline

# Replay with specific bundle
stella score replay --scan-id $SCAN_ID \
  --offline \
  --bundle /path/to/proof-bundle.zip

# Compare with different kit versions
stella score replay --scan-id $SCAN_ID \
  --offline \
  --feed-snapshot sha256:newfeed... \
  --diff
```

### 3.3 Generate Proof Bundle (Air-Gap)

```bash
# Generate proof bundle for export
stella proof export --scan-id $SCAN_ID \
  --include-manifest \
  --include-chain \
  --output proof-bundle.zip

# Proof bundle contents (air-gap safe):
# - manifest.json (canonical)
# - manifest.dsse.json
# - score_proof.json
# - proof_root.dsse.json
# - meta.json
# - NO external references

# Generate portable bundle
stella proof export --scan-id $SCAN_ID \
  --portable \
  --include-trust-anchors \
  --output portable-proof.zip
```

---

## 4. Reachability in Air-Gap Mode

### 4.1 Call Graph Operations (Air-Gap)

```bash
# Upload call graph (works identically)
stella scan graph upload --scan-id $SCAN_ID --file callgraph.json

# Call graph processing is fully local
# No external network required
```

### 4.2 Compute Reachability (Air-Gap)

```bash
# Compute reachability (fully offline)
stella reachability compute --scan-id $SCAN_ID --offline

# Symbol resolution uses offline database
stella reachability compute --scan-id $SCAN_ID \
  --offline \
  --symbol-db /path/to/offline-kit/symbols/symbol-index.db
```

### 4.3 Explain Queries (Air-Gap)

```bash
# Explain queries work offline
stella reachability explain --scan-id $SCAN_ID \
  --cve CVE-2024-1234 \
  --purl "pkg:npm/lodash@4.17.20" \
  --offline

# Export explanations for external review
stella reachability explain-all --scan-id $SCAN_ID \
  --output explanations.json \
  --offline
```

---

## 5. Bundle Import Operations

### 5.1 Import Feed Updates

```bash
# Verify feed bundle before import
stella airgap verify --bundle feed-update.zip

# Dry-run import
stella airgap import --bundle feed-update.zip \
  --type feed \
  --dry-run

# Import feed bundle
stella airgap import --bundle feed-update.zip \
  --type feed \
  --generation 2025.12.21

# Verify import
stella airgap verify-import --generation 2025.12.21
```

### 5.2 Import VEX Updates

```bash
# Import VEX bundle
stella airgap import --bundle vex-update.zip \
  --type vex \
  --generation 2025.12.21

# Verify VEX statements
stella airgap vex-status

# Output:
# VEX statements: 15,432
# Last update: 2025-12-21
# Generation: 2025.12.21
# Signature: VALID
```

### 5.3 Import Trust Anchors

```bash
# Import new trust anchor (requires approval)
stella airgap import-anchor --file new-anchor.json \
  --reason "Key rotation Q4 2025" \
  --approver admin@example.com

# Verify anchor chain
stella airgap verify-anchors

# List active anchors
stella airgap anchors list
```

### 5.4 Import Checklist

**Pre-Import**:
- [ ] Verify bundle signature (DSSE)
- [ ] Verify bundle hash matches manifest
- [ ] Confirm sealed/constrained mode is set
- [ ] Backup current generation

**Import**:
- [ ] Run dry-run import
- [ ] Apply import
- [ ] Verify import succeeded

**Post-Import**:
- [ ] Verify timeline event emitted
- [ ] Update staleness dashboard
- [ ] Archive import manifest
- [ ] Update audit log

---

## 6. Proof Verification Offline

### 6.1 Verify Proof Bundle (Full Offline)

```bash
# Verify proof bundle without any network access
stella proof verify --bundle proof-bundle.zip \
  --offline \
  --trust-anchor /path/to/trust-anchors.json

# Verification checks (offline):
# ✅ Signature valid (DSSE)
# ✅ Content-addressed ID matches
# ✅ Merkle path valid
# ⏭️ Rekor inclusion (SKIPPED - offline mode)
# ✅ Time anchor valid
```

### 6.2 Verify with Portable Bundle

```bash
# Portable bundles include trust anchors
stella proof verify --bundle portable-proof.zip \
  --offline \
  --self-contained

# Output:
# Using embedded trust anchors
# Signature verification: PASS
# ID recomputation: PASS
# Merkle path: PASS
# Time anchor: VALID
# Overall: VERIFIED
```

### 6.3 Batch Verification

```bash
# Verify multiple bundles
stella proof verify-batch --dir /path/to/bundles/ \
  --offline \
  --trust-anchor /path/to/trust-anchors.json \
  --output verification-report.json

# Generate verification report
cat verification-report.json | jq '.summary'
# Output:
# {
#   "total": 100,
#   "verified": 98,
#   "failed": 2,
#   "skipped": 0
# }
```

### 6.4 Verification Without CLI

For environments without the CLI, manual verification is possible:

```bash
# 1. Extract bundle
unzip proof-bundle.zip -d ./verify/

# 2. Verify DSSE signature (using openssl)
# Extract payload from DSSE envelope
cat ./verify/manifest.dsse.json | jq -r '.payload' | base64 -d > payload.json

# Verify signature
cat ./verify/manifest.dsse.json | jq -r '.signatures[0].sig' | base64 -d > signature.bin
openssl dgst -sha256 -verify trust-anchor-pubkey.pem -signature signature.bin payload.json

# 3. Verify content-addressed ID
# Compute canonical hash
cat ./verify/manifest.json | jq -cS . | sha256sum
# Compare with manifestHash in bundle

# 4. Verify merkle path
# (See docs/airgap/proof-chain-verification.md for algorithm)
```

---

## 7. Troubleshooting

### 7.1 Kit Verification Failed

**Symptom**: `stella airgap verify-kit` fails.

**Diagnosis**:

```bash
# Check specific component
stella airgap verify-kit --verbose --component feeds

# Common errors:
# - "Signature verification failed": Key mismatch
# - "Hash mismatch": Bundle corrupted during transfer
# - "Time anchor expired": Anchor needs refresh
```

**Resolution**:

| Error | Cause | Resolution |
|-------|-------|------------|
| Signature failed | Wrong trust anchor | Import correct anchor |
| Hash mismatch | Corruption | Re-transfer bundle |
| Time anchor expired | Clock drift or expired | Import new time anchor |

### 7.2 Staleness Alert

**Symptom**: Staleness warning/alert.

**Diagnosis**:

```bash
# Check staleness status
stella airgap staleness-status

# Output:
# Feed age: 5 days (threshold: 7 days)
# VEX age: 3 days (threshold: 7 days)
# Policy age: 30 days (threshold: 90 days)
# Status: AMBER (approaching threshold)
```

**Resolution**:

```bash
# Import updated bundles
stella airgap import --bundle /path/to/latest-feed.zip --type feed

# If bundles unavailable and breach imminent:
# - Raise amber alert (5-7 days)
# - If >7 days, raise red alert and halt new ingests
# - Request emergency bundle via secure channel
```

### 7.3 Proof Verification Fails Offline

**Symptom**: Proof verification fails in sealed mode.

**Diagnosis**:

```bash
# Check verification error
stella proof verify --bundle proof.zip --offline --verbose

# Common errors:
# - "Trust anchor not found": Missing anchor in offline kit
# - "Time anchor expired": Time validation failed
# - "Unsupported algorithm": Key algorithm not supported
```

**Resolution**:

```bash
# For missing trust anchor:
# Import the required anchor
stella airgap import-anchor --file required-anchor.json

# For expired time anchor:
# Import new time anchor
stella airgap import-time-anchor --file new-time-anchor.json

# For algorithm issues:
# Regenerate proof with supported algorithm
stella proof regenerate --scan-id $SCAN_ID --algorithm ECDSA-P256
```

### 7.4 Symbol Resolution Fails

**Symptom**: Reachability shows "symbol not found" errors.

**Diagnosis**:

```bash
# Check symbol database status
stella airgap symbols-status

# Output:
# Symbol DB: /path/to/symbols/symbol-index.db
# Version: 2025.12.15
# Entries: 5,234,567
# Coverage: Java, .NET, Python
```

**Resolution**:

```bash
# Import updated symbol database
stella airgap import --bundle symbol-update.zip --type symbols

# Recompute reachability with new symbols
stella reachability compute --scan-id $SCAN_ID --offline --force
```

---

## 8. Monitoring & Alerting

### 8.1 Key Metrics (Air-Gap)

| Metric | Description | Alert Threshold |
|--------|-------------|-----------------|
| `airgap_staleness_days` | Days since last bundle import | > 5 (amber), > 7 (red) |
| `airgap_time_anchor_validity_days` | Days until time anchor expires | < 7 |
| `airgap_verification_failures` | Offline verification failures | > 0 |
| `airgap_import_failures` | Bundle import failures | > 0 |

### 8.2 Alerting Rules

```yaml
groups:
  - name: airgap-operations
    rules:
      - alert: AirgapStalenessAmber
        expr: airgap_staleness_days > 5
        for: 1h
        labels:
          severity: warning
        annotations:
          summary: "Air-gap feed staleness approaching threshold"
          
      - alert: AirgapStalenessRed
        expr: airgap_staleness_days > 7
        for: 1h
        labels:
          severity: critical
        annotations:
          summary: "Air-gap feed staleness breach - halt new ingests"
          
      - alert: AirgapTimeAnchorExpiring
        expr: airgap_time_anchor_validity_days < 7
        for: 1h
        labels:
          severity: warning
        annotations:
          summary: "Time anchor expiring in {{ $value }} days"
          
      - alert: AirgapVerificationFailure
        expr: increase(airgap_verification_failures_total[1h]) > 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Air-gap verification failures detected"
```

### 8.3 Audit Requirements

For air-gapped environments, maintain strict audit trails:

```bash
# Record every import
{
  "timestamp": "2025-12-20T10:00:00Z",
  "action": "import",
  "bundleType": "feed",
  "bundleHash": "sha256:feed123...",
  "generation": "2025.12.20",
  "actor": "operator@example.com",
  "mode": "sealed",
  "verification": "PASS"
}

# Daily audit log export
stella airgap audit-export --date today --output audit-$(date +%Y%m%d).json

# Verify audit log integrity
stella airgap audit-verify --log audit-20251220.json
```

---

## Related Documentation

- [Air-Gap Overview](./overview.md)
- [Offline Bundle Format](./offline-bundle-format.md)
- [Proof Chain Verification](./proof-chain-verification.md)
- [Time Anchor Schema](./time-anchor-schema.md)
- [Score Proofs Runbook](../operations/score-proofs-runbook.md)
- [Reachability Runbook](../operations/reachability-runbook.md)

---

**Last Updated**: 2025-12-20  
**Version**: 1.0.0  
**Sprint**: 3500.0004.0004