stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 2 Releases Wiki Activity
Files
702a27ac839f1de50f9c9f460000b8f7b2242efd
git.stella-ops.org/docs/operations/runbooks/crypto-ops.md

8.4 KiB
Raw Blame History

Sprint: SPRINT_20260117_029_Runbook_coverage_expansion

Task: RUN-002 - Crypto Subsystem Runbook

Regional Crypto Operations Runbook

Status: PRODUCTION-READY (2026-01-17 UTC)

Scope

Cryptographic subsystem operations including HSM management, regional crypto profile configuration, key rotation, and certificate management for all supported crypto profiles (International, FIPS, eIDAS, GOST, SM).

Pre-flight Checklist

Environment Verification

# Check crypto subsystem health
stella doctor --category crypto

# Verify active crypto profile
stella crypto profile show

# List loaded crypto providers
stella crypto providers list

# Check key status
stella crypto keys status

Metrics to Watch

  • stella_crypto_operations_total - Crypto operation count by type
  • stella_crypto_operation_duration_seconds - Signing/verification latency
  • stella_hsm_availability - HSM availability (if configured)
  • stella_cert_expiry_days - Certificate expiration countdown

Regional Crypto Profiles

Profile Overview

Profile Use Case Key Algorithms Compliance
international Default, most deployments RSA-2048+, ECDSA P-256/P-384, Ed25519 General
fips US Government / FedRAMP FIPS 140-2 approved algorithms only FIPS 140-2
eidas European Union RSA-PSS, ECDSA, Ed25519 per ETSI TS 119 312 eIDAS
gost Russian Federation GOST R 34.10-2012, GOST R 34.11-2012 Russian standards
sm China SM2, SM3, SM4 GM/T 0003-2012

Switching Profiles

  1. Pre-switch verification:

    # Verify target profile is available
stella crypto profile verify --profile <target-profile>

# Check for incompatible existing signatures
stella crypto audit --check-compatibility --target-profile <target-profile>

  2. Profile switch:

    # Switch profile (requires service restart)
stella crypto profile set --profile <target-profile>

# Restart services to apply
stella service restart --graceful

  3. Post-switch verification:

    stella doctor --check check.crypto.fips,check.crypto.eidas,check.crypto.gost,check.crypto.sm

Standard Procedures

SP-001: Key Rotation

Frequency: Quarterly or per policy Duration: ~15 minutes (no downtime)

  1. Generate new key:

    # For software keys
stella crypto keys generate --type signing --algorithm ecdsa-p256 --name signing-$(date +%Y%m)

# For HSM-backed keys
stella crypto keys generate --type signing --algorithm ecdsa-p256 --provider hsm --name signing-$(date +%Y%m)

  2. Activate new key:

    stella crypto keys activate --name signing-$(date +%Y%m)

  3. Verify signing with new key:

    echo "test" | stella crypto sign --output /dev/null

  4. Schedule old key deactivation:

    stella crypto keys schedule-deactivation --name <old-key-name> --in 30d

SP-002: Certificate Renewal

When: Certificate expiring within 30 days

  1. Check expiration:

    stella crypto certs check-expiry

  2. Generate CSR:

    stella crypto certs csr --subject "CN=stellaops.example.com,O=Example Corp" --output cert.csr

  3. Install renewed certificate:

    stella crypto certs install --cert renewed-cert.pem --chain ca-chain.pem

  4. Verify certificate chain:

    stella doctor --check check.crypto.certchain

  5. Restart services:

    stella service restart --graceful

SP-003: HSM Health Check

Frequency: Daily (automated) or on-demand

  1. Check HSM connectivity:

    stella crypto hsm status

  2. Verify slot access:

    stella crypto hsm slots list

  3. Test signing operation:

    stella crypto hsm test-sign

  4. Check HSM metrics:

    • Free objects/sessions
    • Temperature/health (vendor-specific)

Incident Procedures

INC-001: HSM Unavailable

Symptoms:

  • Alert: StellaHsmUnavailable
  • Signing operations failing with "HSM connection error"

Investigation:

# Check HSM status
stella crypto hsm status

# Test PKCS#11 module
stella crypto hsm test-module

# Check network to HSM
stella network test --host <hsm-host> --port <hsm-port>

Resolution:

  1. Network issue:

    • Verify network path to HSM
    • Check firewall rules
    • Verify HSM appliance is powered on

  2. Session exhaustion:

    # Release stale sessions
stella crypto hsm sessions release --stale

# Restart crypto service
stella service restart --service crypto-signer

  3. HSM failure:

    • Fail over to secondary HSM (if configured)
    • Contact HSM vendor support
    • Consider temporary fallback to software keys (with approval)

INC-002: Signing Key Compromised

CRITICAL - Follow incident response procedure

  1. Immediate containment:

    # Revoke compromised key
stella crypto keys revoke --name <compromised-key> --reason compromise

# Block signing with compromised key
stella crypto keys block --name <compromised-key>

  2. Generate replacement key:

    stella crypto keys generate --type signing --algorithm ecdsa-p256 --name emergency-signing
stella crypto keys activate --name emergency-signing

  3. Notify downstream:

    • Update trust registries with new key
    • Notify relying parties
    • Publish key revocation notice

  4. Forensics:

    # Export key usage audit log
stella crypto audit export --key <compromised-key> --output /secure/key-audit.json

INC-003: Certificate Expired

Symptoms:

  • TLS connection failures
  • Alert: StellaCertExpired

Immediate Resolution:

  1. If renewed certificate is available:

    stella crypto certs install --cert renewed-cert.pem --chain ca-chain.pem
stella service restart --graceful

  2. If renewal not ready - emergency self-signed (temporary):

    # Generate emergency certificate (NOT for production use)
stella crypto certs generate-self-signed --days 7 --name emergency
stella crypto certs install --cert emergency.pem
stella service restart --graceful

  3. Expedite certificate renewal process

INC-004: FIPS Mode Not Enabled

Symptoms:

  • Alert: StellaFipsNotEnabled
  • Compliance audit failure

Resolution:

  1. Linux:

    # Enable FIPS mode
sudo fips-mode-setup --enable

# Reboot required
sudo reboot

# Verify after reboot
fips-mode-setup --check

  2. Windows:

    • Enable via Group Policy
    • Or via registry: 
      Set-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Control\Lsa\FipsAlgorithmPolicy" -Name "Enabled" -Value 1
Restart-Computer

  3. Restart Stella services:

    stella service restart
stella doctor --check check.crypto.fips

Regional-Specific Procedures

GOST Configuration (Russian Federation)

  1. Install GOST engine:

    sudo apt install libengine-gost-openssl1.1

  2. Configure Stella:

    stella crypto profile set --profile gost
stella crypto config set --gost-engine-path /usr/lib/x86_64-linux-gnu/engines-3/gost.so

  3. Verify:

    stella doctor --check check.crypto.gost

SM Configuration (China)

  1. Ensure OpenSSL 1.1.1+ with SM support:

    openssl version
openssl list -cipher-algorithms | grep -i sm

  2. Configure Stella:

    stella crypto profile set --profile sm

  3. Verify:

    stella doctor --check check.crypto.sm

Monitoring Dashboard

Access: Grafana → Dashboards → Stella Ops → Crypto Subsystem

Key panels:

  • Signing operation latency
  • Key usage by key ID
  • HSM availability
  • Certificate expiration countdown
  • Crypto profile in use

Evidence Capture

# Comprehensive crypto diagnostics
stella crypto diagnostics --output /tmp/crypto-diag-$(date +%Y%m%dT%H%M%S).tar.gz

Bundle includes:

  • Active crypto profile
  • Key inventory (public keys only)
  • Certificate chain
  • HSM status
  • Operation audit log (last 24h)

Escalation Path

  1. L1 (On-call): Certificate installs, key activation
  2. L2 (Security team): Key rotation, HSM issues
  3. L3 (Crypto SME): Algorithm issues, compliance questions
  4. HSM Vendor: Hardware failures

Last updated: 2026-01-17 (UTC)