stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 3 Releases Wiki Activity

docs(ops): Complete operations runbooks for Epic 3500

Browse Source 
Sprint 3500.0004.0004 (Documentation & Handoff) - T2 DONE

Operations Runbooks Added:
- score-replay-runbook.md: Deterministic replay procedures
- proof-verification-runbook.md: DSSE/Merkle verification ops
- airgap-operations-runbook.md: Offline kit management

CLI Reference Docs:
- reachability-cli-reference.md
- score-proofs-cli-reference.md
- unknowns-cli-reference.md

Air-Gap Guides:
- score-proofs-reachability-airgap-runbook.md

Training Materials:
- score-proofs-concept-guide.md

UI API Clients:
- proof.client.ts
- reachability.client.ts
- unknowns.client.ts

All 5 operations runbooks now complete (reachability, unknowns-queue,
score-replay, proof-verification, airgap-operations).
This commit is contained in:
StellaOps Bot
2025-12-20 22:30:02 +02:00
parent 09c7155f1b
commit 4b3db9ca85
13 changed files with 5630 additions and 12 deletions
Download Patch File Download Diff File Expand all files Collapse all files

688
docs/operations/airgap-operations-runbook.md Normal file
View File

@@ -0,0 +1,688 @@
# Air-Gap Operations Runbook
> **Version**: 1.0.0
> **Sprint**: 3500.0004.0004
> **Last Updated**: 2025-12-20
This runbook covers operational procedures for running StellaOps in air-gapped (offline) environments, including offline kit management, feed updates, and isolated verification workflows.
---
## Table of Contents
1. [Overview](#1-overview)
2. [Offline Kit Management](#2-offline-kit-management)
3. [Feed Updates](#3-feed-updates)
4. [Scanning in Air-Gap Mode](#4-scanning-in-air-gap-mode)
5. [Verification in Air-Gap Mode](#5-verification-in-air-gap-mode)
6. [Troubleshooting](#6-troubleshooting)
7. [Monitoring & Health Checks](#7-monitoring--health-checks)
8. [Escalation Procedures](#8-escalation-procedures)
---
## 1. Overview
### What is Air-Gap Mode?
Air-gap mode allows StellaOps to operate in environments with no external network connectivity. This is required for:
- Classified or sensitive environments
- High-security facilities
- Regulatory compliance (certain industries)
- Disaster recovery scenarios
### Air-Gap Architecture
```
┌─────────────────────────────────────────────────────────────┐
│ Connected Environment │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ Feed Sync │───►│ Bundle │───►│ Offline Kit │ │
│ │ Service │ │ Generator │ │ (.tar.gz) │ │
│ └─────────────┘ └─────────────┘ └──────┬──────┘ │
└─────────────────────────────────────────────────┼───────────┘
│ Physical
│ Transfer
┌─────────────────────────────────────────────────┼───────────┐
│ Air-Gapped Environment │ │
│ ┌──────────────┐ ┌─────────────┐ ┌──────▼──────┐ │
│ │ StellaOps │◄───│ Offline │◄───│ Import │ │
│ │ Scanner │ │ Data Store │ │ Service │ │
│ └──────────────┘ └─────────────┘ └─────────────┘ │
└─────────────────────────────────────────────────────────────┘
```
### Offline Kit Contents
| Component | Description | Update Frequency |
|-----------|-------------|------------------|
| Vulnerability Database | NVD, OSV, vendor advisories | Daily/Weekly |
| Advisory Feeds | CVE details, EPSS scores | Daily |
| Trust Bundles | CA certificates, signing keys | Quarterly |
| Rules Engine | Scoring rules and policies | Monthly |
| Offline Binaries | CLI tools, extractors | Per release |
---
## 2. Offline Kit Management
### 2.1 Generating an Offline Kit
On the connected system:
```bash
# Generate full offline kit
stella offline-kit create \
--output /path/to/offline-kit.tar.gz \
--include-all
# Generate minimal kit (feeds only)
stella offline-kit create \
--output /path/to/offline-kit-feeds.tar.gz \
--feeds-only
# Generate with specific components
stella offline-kit create \
--output /path/to/offline-kit.tar.gz \
--include vuln-db \
--include advisories \
--include trust-bundles \
--include rules
```
### 2.2 Kit Manifest
Each kit includes a manifest for verification:
```json
{
"version": "1.0.0",
"createdAt": "2025-01-15T00:00:00Z",
"expiresAt": "2025-02-15T00:00:00Z",
"components": {
"vulnerability-database": {
"hash": "sha256:abc123...",
"size": 1073741824,
"records": 245000
},
"advisory-feeds": {
"hash": "sha256:def456...",
"size": 536870912,
"lastUpdate": "2025-01-15T00:00:00Z"
},
"trust-bundles": {
"hash": "sha256:ghi789...",
"size": 65536,
"certificates": 12
},
"signing-keys": {
"hash": "sha256:jkl012...",
"keyIds": ["key-001", "key-002"]
}
},
"signature": "base64-signature..."
}
```
### 2.3 Transferring to Air-Gapped Environment
#### Physical Media Transfer
```bash
# On connected system - write to media
cp offline-kit.tar.gz /media/secure-usb/
sha256sum offline-kit.tar.gz > /media/secure-usb/offline-kit.tar.gz.sha256
# On air-gapped system - verify and import
cd /media/secure-usb/
sha256sum -c offline-kit.tar.gz.sha256
stella offline-kit import --kit offline-kit.tar.gz
```
#### Secure File Transfer (if available)
```bash
# Using data diode or one-way transfer
scp offline-kit.tar.gz airgap-gateway:/incoming/
```
### 2.4 Installing Offline Kit
```bash
# Import and install kit
stella offline-kit import \
--kit /path/to/offline-kit.tar.gz \
--verify \
--install
# Verify installation
stella offline-kit status
# List installed components
stella offline-kit list
```
Expected output:
```
Offline Kit Status
══════════════════════════════════════════
Mode: AIR-GAP
Kit Version: 1.0.0
Installed At: 2025-01-15T10:30:00Z
Expires At: 2025-02-15T00:00:00Z
Components:
✓ vulnerability-database 2025-01-15 245,000 records
✓ advisory-feeds 2025-01-15 Active
✓ trust-bundles 2025-01-15 12 certificates
✓ signing-keys 2025-01-15 2 keys
✓ rules-engine 2025-01-15 v2.3.0
Health: HEALTHY
Days Until Expiry: 31
```
---
## 3. Feed Updates
### 3.1 Update Workflow
```
┌────────────────────────────────────────────────────────────┐
│ Weekly Feed Update Process │
├────────────────────────────────────────────────────────────┤
│ Day 1: Generate kit on connected system │
│ Day 2: Security review and approval │
│ Day 3: Transfer to air-gapped environment │
│ Day 4: Import and verify │
│ Day 5: Activate new feeds │
└────────────────────────────────────────────────────────────┘
```
### 3.2 Generating Delta Updates
For faster updates, generate delta kits:
```bash
# On connected system
stella offline-kit create \
--output delta-kit.tar.gz \
--delta-from 2025-01-08 \
--delta-to 2025-01-15
# Delta kit is smaller, contains only changes
```
### 3.3 Applying Updates
```bash
# Import delta update
stella offline-kit import \
--kit delta-kit.tar.gz \
--delta \
--verify
# Verify feed freshness
stella feeds status
```
### 3.4 Rollback Procedure
If an update causes issues:
```bash
# List available snapshots
stella offline-kit snapshots
# Rollback to previous version
stella offline-kit rollback --to 2025-01-08
# Verify rollback
stella feeds status
```
---
## 4. Scanning in Air-Gap Mode
### 4.1 Enabling Air-Gap Mode
```bash
# Enable air-gap mode
stella config set mode air-gap
# Verify mode
stella config get mode
# Output: air-gap
```
### 4.2 Running Scans
```bash
# Scan a local image (no registry pull)
stella scan image --local /path/to/image.tar
# Scan from local registry
stella scan image localhost:5000/myapp:v1.0
# Scan with offline feeds explicitly
stella scan image myapp:v1.0 --offline-feeds
```
### 4.3 Local Image Preparation
For images that need to be scanned:
```bash
# On connected system - save image
docker save myapp:v1.0 -o myapp-v1.0.tar
sha256sum myapp-v1.0.tar > myapp-v1.0.tar.sha256
# Transfer to air-gapped system
# ... physical transfer ...
# On air-gapped system - verify and load
sha256sum -c myapp-v1.0.tar.sha256
docker load -i myapp-v1.0.tar
stella scan image myapp:v1.0 --local
```
### 4.4 SBOM Generation
```bash
# Generate SBOM for local image
stella sbom generate --image myapp:v1.0 --output sbom.json
# Scan existing SBOM
stella scan sbom --file sbom.json
```
---
## 5. Verification in Air-Gap Mode
### 5.1 Offline Proof Verification
```bash
# Verify proof bundle offline
stella proof verify --bundle bundle.tar.gz --offline
# Verify with explicit trust store
stella proof verify --bundle bundle.tar.gz \
--offline \
--trust-store /etc/stellaops/offline/trust-roots.json
```
### 5.2 Preparing Trust Store
Before air-gapped deployment:
```bash
# On connected system - export trust configuration
stella trust export \
--output trust-roots.json \
--include-ca \
--include-signing-keys
# Transfer to air-gapped system
# ... physical transfer ...
# On air-gapped system - import trust
stella trust import --file trust-roots.json
```
### 5.3 Score Replay Offline
```bash
# Replay score using offline data
stella score replay --scan $SCAN_ID --offline
# Replay with frozen time
stella score replay --scan $SCAN_ID \
--offline \
--freeze 2025-01-15T00:00:00Z
```
---
## 6. Troubleshooting
### 6.1 Kit Import Fails
**Symptoms**: `Failed to import offline kit`
**Diagnostic Steps**:
1. Verify kit integrity:
```bash
sha256sum offline-kit.tar.gz
# Compare with manifest
```
2. Check kit signature:
```bash
stella offline-kit verify --kit offline-kit.tar.gz
```
3. Check disk space:
```bash
df -h /var/lib/stellaops/
```
**Common Causes**:
| Cause | Resolution |
|-------|------------|
| Corrupted transfer | Re-transfer, verify checksum |
| Invalid signature | Regenerate kit with valid signing key |
| Insufficient space | Free disk space or expand volume |
| Expired kit | Generate fresh kit |
### 6.2 Stale Feed Data
**Symptoms**: Scans report old vulnerabilities, miss new CVEs
**Diagnostic Steps**:
1. Check feed age:
```bash
stella feeds status
```
2. Verify last update:
```bash
stella offline-kit status | grep "Installed At"
```
**Resolution**:
- Generate and import fresh offline kit
- Establish regular update schedule
- Set up expiry alerts
### 6.3 Trust Verification Fails
**Symptoms**: `Certificate chain verification failed` in offline mode
**Diagnostic Steps**:
1. Check trust store:
```bash
stella trust list
```
2. Verify CA bundle:
```bash
openssl verify -CAfile /etc/stellaops/offline/ca-bundle.pem \
/path/to/certificate.pem
```
3. Check for expired roots:
```bash
stella trust check-expiry
```
**Resolution**:
- Update trust bundle in offline kit
- Import new CA certificates
- Rotate expired signing keys
### 6.4 Network Access Attempted
**Symptoms**: Air-gapped system attempts network connection
**Diagnostic Steps**:
1. Check mode configuration:
```bash
stella config get mode
```
2. Audit network attempts:
```bash
journalctl -u stellaops | grep -i "network\|connect\|http"
```
3. Verify no external URLs in config:
```bash
grep -r "http" /etc/stellaops/
```
**Resolution**:
- Ensure `mode: air-gap` in configuration
- Remove any hardcoded URLs
- Block outbound traffic at firewall level
---
## 7. Monitoring & Health Checks
### 7.1 Air-Gap Health Checks
```bash
# Run comprehensive health check
stella health check --air-gap
# Output
Air-Gap Health Check
══════════════════════════════════════════
✓ Mode: air-gap
✓ Feed Freshness: 7 days old (OK)
✓ Trust Store: Valid
✓ Signing Keys: 2 active
✓ Disk Space: 45% used
✓ Database: Healthy
⚠ Kit Expiry: 24 days remaining
Overall: HEALTHY (1 warning)
```
### 7.2 Automated Monitoring Script
```bash
#!/bin/bash
# /etc/stellaops/scripts/airgap-health.sh
set -e
# Check feed age
FEED_AGE=$(stella feeds age --days)
if [ "$FEED_AGE" -gt 14 ]; then
echo "CRITICAL: Feeds are $FEED_AGE days old"
exit 2
fi
# Check kit expiry
DAYS_LEFT=$(stella offline-kit days-until-expiry)
if [ "$DAYS_LEFT" -lt 7 ]; then
echo "WARNING: Kit expires in $DAYS_LEFT days"
exit 1
fi
# Check trust store
if ! stella trust verify --quiet; then
echo "CRITICAL: Trust store verification failed"
exit 2
fi
echo "OK: Air-gap health check passed"
exit 0
```
### 7.3 Metrics for Air-Gap
| Metric | Description | Alert Threshold |
|--------|-------------|-----------------|
| `offline_kit_age_days` | Days since kit import | > 14 days |
| `offline_kit_expiry_days` | Days until kit expires | < 7 days |
| `feed_freshness_days` | Age of vulnerability feeds | > 7 days |
| `trust_store_valid` | Trust store validity (0/1) | = 0 |
| `disk_usage_percent` | Data store disk usage | > 80% |
### 7.4 Alert Configuration
```yaml
# /etc/stellaops/alerts/airgap.yaml
alerts:
- name: offline_kit_expiring
condition: offline_kit_expiry_days < 7
severity: warning
message: "Offline kit expires in {{ .Value }} days"
- name: offline_kit_expired
condition: offline_kit_expiry_days <= 0
severity: critical
message: "Offline kit has expired"
- name: feeds_stale
condition: feed_freshness_days > 14
severity: warning
message: "Vulnerability feeds are {{ .Value }} days old"
```
---
## 8. Escalation Procedures
### 8.1 Escalation Matrix
| Severity | Condition | Response Time | Action |
|----------|-----------|---------------|--------|
| P1 - Critical | Offline kit expired | 4 hours | Emergency kit transfer |
| P1 - Critical | Trust store invalid | 4 hours | Restore from backup |
| P2 - High | Feeds > 14 days old | 24 hours | Schedule kit update |
| P3 - Medium | Kit expiring in < 7 days | 48 hours | Plan kit update |
| P4 - Low | Minor health check warnings | Next maintenance | Review and address |
### 8.2 Emergency Kit Update Process
When kit expires before scheduled update:
1. **On Connected System** (0-2 hours):
```bash
stella offline-kit create --output emergency-kit.tar.gz --include-all
```
2. **Security Review** (2-4 hours):
- Verify kit signature
- Check for known vulnerabilities in kit
- Get approval for transfer
3. **Transfer** (4-6 hours):
- Physical media preparation
- Chain of custody documentation
- Transfer to air-gapped environment
4. **Import** (6-8 hours):
```bash
stella offline-kit import --kit emergency-kit.tar.gz --verify --install
```
### 8.3 Contacts
| Role | Contact | Availability |
|------|---------|--------------|
| Air-Gap Operations | airgap-ops@stellaops.io | Business hours |
| Security Team | security@stellaops.io | Business hours |
| Platform On-Call | platform-oncall@stellaops.io | 24/7 |
---
## Appendix A: Configuration Reference
### Air-Gap Mode Configuration
```yaml
# /etc/stellaops/config.yaml
mode: air-gap
offline:
dataDir: /var/lib/stellaops/offline
feedsDir: /var/lib/stellaops/offline/feeds
trustStore: /etc/stellaops/offline/trust-roots.json
caBundle: /etc/stellaops/offline/ca-bundle.pem
# Disable all external network calls
disableNetworking: true
# Kit expiry settings
kit:
expiryWarningDays: 7
maxAgeDays: 30
# Feed freshness settings
feeds:
maxAgeDays: 14
warnAgeDays: 7
```
### Environment Variables
```bash
# Air-gap specific
export STELLAOPS_MODE=air-gap
export STELLAOPS_OFFLINE_DATA_DIR=/var/lib/stellaops/offline
export STELLAOPS_DISABLE_NETWORKING=true
# Trust configuration
export STELLAOPS_TRUST_STORE=/etc/stellaops/offline/trust-roots.json
export STELLAOPS_CA_BUNDLE=/etc/stellaops/offline/ca-bundle.pem
```
---
## Appendix B: CLI Quick Reference
```bash
# Offline Kit Commands
stella offline-kit create --output <path> # Generate kit
stella offline-kit import --kit <path> # Import kit
stella offline-kit status # Show status
stella offline-kit verify --kit <path> # Verify kit
stella offline-kit rollback --to <date> # Rollback feeds
# Feed Commands (Air-Gap)
stella feeds status # Show feed status
stella feeds age --days # Get feed age
# Trust Commands
stella trust list # List trust roots
stella trust import --file <path> # Import trust config
stella trust verify # Verify trust store
stella trust export --output <path> # Export trust config
# Scanning (Air-Gap)
stella scan image --local <path> # Scan local image
stella scan sbom --file <path> # Scan SBOM file
# Verification (Air-Gap)
stella proof verify --bundle <path> --offline # Offline verification
stella score replay --scan <id> --offline # Offline replay
# Health
stella health check --air-gap # Air-gap health check
stella config get mode # Check current mode
```
---
## Appendix C: Update Schedule Template
| Week | Activity | Owner | Notes |
|------|----------|-------|-------|
| 1 | Generate kit | Connected Ops | Monday |
| 1 | Security review | Security Team | Tuesday |
| 1 | Approval | Security Lead | Wednesday |
| 2 | Transfer | Air-Gap Ops | Monday |
| 2 | Import & verify | Air-Gap Ops | Tuesday |
| 2 | Activate | Air-Gap Ops | Wednesday |
| 2 | Validate scans | QA Team | Thursday |
---
## Revision History
| Version | Date | Author | Changes |
|---------|------|--------|---------|
| 1.0.0 | 2025-12-20 | Agent | Initial release |