git.stella-ops.org/docs/operations/hsm-setup-runbook.md

# HSM Setup and Configuration Runbook

This runbook provides step-by-step procedures for configuring Hardware Security Module (HSM) integration with Stella Ops.

## Overview

Stella Ops supports PKCS#11-compatible HSMs for cryptographic key storage and signing operations. This includes:
- YubiHSM 2
- Thales Luna Network HSM
- AWS CloudHSM
- SoftHSM2 (development/testing)

## Prerequisites

### Hardware Requirements

| Component | Requirement |
|-----------|-------------|
| HSM Device | PKCS#11 compatible |
| Network | HSM accessible from Stella Ops services |
| Backup | Secondary HSM for key backup |

### Software Requirements

```bash
# PKCS#11 library for your HSM
# Example for SoftHSM2 (development)
apt-get install softhsm2 opensc

# Verify installation
softhsm2-util --version
pkcs11-tool --version
```

## SoftHSM2 Setup (Development)

### Step 1: Initialize SoftHSM

```bash
# Create token directory
mkdir -p /var/lib/softhsm/tokens
chmod 700 /var/lib/softhsm/tokens

# Initialize token
softhsm2-util --init-token \
  --slot 0 \
  --label "StellaOps-Dev" \
  --so-pin 12345678 \
  --pin 87654321

# Verify token
softhsm2-util --show-slots
```

### Step 2: Generate Signing Key

```bash
# Generate ECDSA P-256 key
pkcs11-tool --module /usr/lib/softhsm/libsofthsm2.so \
  --login --pin 87654321 \
  --keypairgen \
  --key-type EC:prime256v1 \
  --id 01 \
  --label "stellaops-signing-2026"

# List keys
pkcs11-tool --module /usr/lib/softhsm/libsofthsm2.so \
  --login --pin 87654321 \
  --list-objects
```

### Step 3: Export Public Key

```bash
# Export public key for distribution
pkcs11-tool --module /usr/lib/softhsm/libsofthsm2.so \
  --login --pin 87654321 \
  --read-object \
  --type pubkey \
  --id 01 \
  --output-file stellaops-signing-2026.pub.der

# Convert to PEM
openssl ec -pubin -inform DER \
  -in stellaops-signing-2026.pub.der \
  -outform PEM \
  -out stellaops-signing-2026.pub.pem
```

## YubiHSM 2 Setup

### Step 1: Install YubiHSM SDK

```bash
# Download YubiHSM SDK
wget https://developers.yubico.com/YubiHSM2/Releases/yubihsm2-sdk-2023.01-ubuntu2204-amd64.tar.gz
tar xzf yubihsm2-sdk-*.tar.gz
cd yubihsm2-sdk
sudo ./install.sh

# Start connector
sudo systemctl enable yubihsm-connector
sudo systemctl start yubihsm-connector
```

### Step 2: Initialize YubiHSM

```bash
# Connect to YubiHSM shell
yubihsm-shell

# Authenticate with default auth key
connect
session open 1 password

# Create authentication key for Stella Ops
generate authkey 0 100 "StellaOps-Auth" 1 generate-asymmetric-key:sign-ecdsa:delete-asymmetric-key

# Generate signing key
generate asymmetric 0 200 "StellaOps-Signing" 1 sign-ecdsa ecp256

# Export public key
get public key 0 200 stellaops-yubihsm.pub

session close 0
quit
```

### Step 3: Configure PKCS#11

```bash
# Create PKCS#11 configuration
cat > /etc/yubihsm_pkcs11.conf <<EOF
connector = http://127.0.0.1:12345
EOF

# Test PKCS#11 access
pkcs11-tool --module /usr/lib/libyubihsm_pkcs11.so \
  --list-slots
```

## Stella Ops Configuration

### Basic HSM Configuration

```yaml
# etc/stellaops.yaml
signing:
  provider: "hsm"
  hsm:
    type: "pkcs11"
    libraryPath: "/usr/lib/softhsm/libsofthsm2.so"  # or /usr/lib/libyubihsm_pkcs11.so
    slotId: 0
    tokenLabel: "StellaOps-Dev"
    pin: "${HSM_PIN}"  # Use environment variable
    keyId: "01"
    keyLabel: "stellaops-signing-2026"

    # Connection settings
    connectionTimeoutSeconds: 30
    maxSessions: 10
    sessionIdleTimeoutSeconds: 300

    # Retry settings
    maxRetries: 3
    retryDelayMs: 100
```

### Environment Variables

```bash
# Set HSM PIN securely
export HSM_PIN="87654321"

# Or use secrets manager
export HSM_PIN=$(aws secretsmanager get-secret-value \
  --secret-id stellaops/hsm-pin \
  --query SecretString --output text)
```

### Kubernetes Secret

```yaml
apiVersion: v1
kind: Secret
metadata:
  name: stellaops-hsm
  namespace: stellaops
type: Opaque
stringData:
  HSM_PIN: "87654321"
```

## Verification

### Step 1: Connectivity Check

```bash
# Run HSM connectivity doctor check
stella doctor --check hsm

# Expected output:
# [PASS] HSM Connectivity
#   - Library loaded: /usr/lib/softhsm/libsofthsm2.so
#   - Slot available: 0 (StellaOps-Dev)
#   - Key found: stellaops-signing-2026
#   - Sign/verify test: PASSED
```

### Step 2: Signing Test

```bash
# Test signing operation
stella sign test \
  --message "test message" \
  --key-label "stellaops-signing-2026"

# Expected output:
# Signature: base64...
# Algorithm: ECDSA-P256
# Key ID: 01
```

### Step 3: Integration Test

```bash
# Run HSM integration tests
stella test integration --filter "HSM*"
```

## Key Rotation

### Step 1: Generate New Key

```bash
# Generate new key in HSM
pkcs11-tool --module /usr/lib/softhsm/libsofthsm2.so \
  --login --pin ${HSM_PIN} \
  --keypairgen \
  --key-type EC:prime256v1 \
  --id 02 \
  --label "stellaops-signing-2027"
```

### Step 2: Add to Trust Anchor

```bash
# Add new key to Stella Ops
stella key add \
  --key-id "stellaops-signing-2027" \
  --algorithm EC-P256 \
  --public-key stellaops-signing-2027.pub.pem
```

### Step 3: Transition Period

```yaml
# Update configuration for dual-key
signing:
  activeKeyId: "stellaops-signing-2027"
  additionalKeys:
    - keyId: "stellaops-signing-2026"
      keyLabel: "stellaops-signing-2026"
```

### Step 4: Revoke Old Key

```bash
# After transition period (2-4 weeks)
stella key revoke \
  --key-id "stellaops-signing-2026" \
  --reason "scheduled-rotation"
```

## Troubleshooting

### Common Issues

| Issue | Symptom | Resolution |
|-------|---------|------------|
| Library not found | `PKCS11 library not found` | Verify `libraryPath` in config |
| Slot not available | `Slot 0 not found` | Run `pkcs11-tool --list-slots` |
| Key not found | `Key stellaops-signing not found` | Verify key label with `--list-objects` |
| Pin incorrect | `CKR_PIN_INCORRECT` | Check HSM_PIN environment variable |
| Session limit | `CKR_SESSION_COUNT` | Increase `maxSessions` or restart |

### Debug Logging

```yaml
# Enable HSM debug logging
logging:
  levels:
    StellaOps.Cryptography.Hsm: Debug
```

### Session Recovery

```bash
# If sessions exhausted, restart service
kubectl rollout restart deployment stellaops-signer -n stellaops
```

## Security Best Practices

1. **PIN Management**
   - Never hardcode PINs in configuration files
   - Use secrets management (Vault, AWS Secrets Manager)
   - Rotate PINs periodically

2. **Key Backup**
   - Configure HSM key backup/replication
   - Test key recovery procedures regularly
   - Document recovery process

3. **Access Control**
   - Limit HSM access to required services only
   - Use separate authentication keys per service
   - Audit HSM access logs

4. **Network Security**
   - Use TLS for network HSM connections
   - Firewall HSM to authorized hosts only
   - Monitor for unauthorized access attempts

## Related Documentation

- [Key Rotation Runbook](key-rotation-runbook.md)
- [Dual-Control Ceremonies](dual-control-ceremonies.md)
- [Signer Architecture](../modules/signer/architecture.md)