194 lines
4.1 KiB
Markdown
194 lines
4.1 KiB
Markdown
# Runbook: Attestor - HSM Connection Issues
|
|
|
|
> **Sprint:** SPRINT_20260117_029_DOCS_runbook_coverage
|
|
> **Task:** RUN-005 - Attestor Runbooks
|
|
|
|
## Metadata
|
|
|
|
| Field | Value |
|
|
|-------|-------|
|
|
| **Component** | Attestor / Cryptography |
|
|
| **Severity** | Critical |
|
|
| **On-call scope** | Platform team, Security team |
|
|
| **Last updated** | 2026-01-17 |
|
|
| **Doctor check** | `check.crypto.hsm-availability` |
|
|
|
|
---
|
|
|
|
## Symptoms
|
|
|
|
- [ ] Signing operations failing with "HSM unavailable"
|
|
- [ ] Alert `AttestorHsmConnectionFailed` firing
|
|
- [ ] Error: "PKCS#11 operation failed" or "HSM session timeout"
|
|
- [ ] Attestations cannot be created
|
|
- [ ] Key operations (sign, verify) failing
|
|
|
|
---
|
|
|
|
## Impact
|
|
|
|
| Impact Type | Description |
|
|
|-------------|-------------|
|
|
| **User-facing** | No attestations can be signed; releases blocked |
|
|
| **Data integrity** | Keys are safe in HSM; operations resume when connection restored |
|
|
| **SLA impact** | All signing operations blocked; compliance posture at risk |
|
|
|
|
---
|
|
|
|
## Diagnosis
|
|
|
|
### Quick checks
|
|
|
|
1. **Check Doctor diagnostics:**
|
|
```bash
|
|
stella doctor --check check.crypto.hsm-availability
|
|
```
|
|
|
|
2. **Check HSM connection status:**
|
|
```bash
|
|
stella crypto hsm status
|
|
```
|
|
|
|
3. **Test HSM connectivity:**
|
|
```bash
|
|
stella crypto hsm test
|
|
```
|
|
|
|
### Deep diagnosis
|
|
|
|
1. **Check PKCS#11 library status:**
|
|
```bash
|
|
stella crypto hsm pkcs11-status
|
|
```
|
|
Look for: Library loaded, slot available, session active
|
|
|
|
2. **Check HSM network connectivity:**
|
|
```bash
|
|
stella crypto hsm ping
|
|
```
|
|
|
|
3. **Check HSM session logs:**
|
|
```bash
|
|
stella crypto hsm logs --last 30m
|
|
```
|
|
Look for: Session errors, timeout, authentication failures
|
|
|
|
4. **Check HSM slot status:**
|
|
```bash
|
|
stella crypto hsm slots list
|
|
```
|
|
Problem if: Slot not found, slot busy, token not present
|
|
|
|
---
|
|
|
|
## Resolution
|
|
|
|
### Immediate mitigation
|
|
|
|
1. **Attempt HSM reconnection:**
|
|
```bash
|
|
stella crypto hsm reconnect
|
|
```
|
|
|
|
2. **If HSM unreachable, switch to software signing (if permitted):**
|
|
```bash
|
|
stella attest config set signing.mode software
|
|
stella attest reload
|
|
```
|
|
**Warning:** Software signing may not meet compliance requirements
|
|
|
|
3. **Use backup HSM if configured:**
|
|
```bash
|
|
stella crypto hsm failover --to backup
|
|
```
|
|
|
|
### Root cause fix
|
|
|
|
**If network connectivity issue:**
|
|
|
|
1. Check HSM network path:
|
|
```bash
|
|
stella crypto hsm connectivity --verbose
|
|
```
|
|
|
|
2. Verify firewall rules allow HSM port (typically 1792 for Luna, 2225 for SafeNet)
|
|
|
|
3. Check HSM server status with vendor tools
|
|
|
|
**If session timeout:**
|
|
|
|
1. Increase session timeout:
|
|
```bash
|
|
stella crypto hsm config set session.timeout 300s
|
|
stella crypto hsm reconnect
|
|
```
|
|
|
|
2. Enable session keep-alive:
|
|
```bash
|
|
stella crypto hsm config set session.keepalive true
|
|
stella crypto hsm config set session.keepalive_interval 60s
|
|
```
|
|
|
|
**If authentication failed:**
|
|
|
|
1. Verify HSM credentials:
|
|
```bash
|
|
stella crypto hsm auth verify
|
|
```
|
|
|
|
2. Update HSM PIN if changed:
|
|
```bash
|
|
stella crypto hsm auth update --slot <slot-id>
|
|
```
|
|
|
|
**If PKCS#11 library issue:**
|
|
|
|
1. Verify library path:
|
|
```bash
|
|
stella crypto hsm config get pkcs11.library_path
|
|
```
|
|
|
|
2. Reload PKCS#11 library:
|
|
```bash
|
|
stella crypto hsm pkcs11-reload
|
|
```
|
|
|
|
3. Check library compatibility:
|
|
```bash
|
|
stella crypto hsm pkcs11-info
|
|
```
|
|
|
|
### Verification
|
|
|
|
```bash
|
|
# Test HSM connectivity
|
|
stella crypto hsm test
|
|
|
|
# Test signing operation
|
|
stella attest test-sign
|
|
|
|
# Verify key access
|
|
stella keys verify <key-id> --operation sign
|
|
|
|
# Check no errors in logs
|
|
stella crypto hsm logs --level error --last 30m
|
|
```
|
|
|
|
---
|
|
|
|
## Prevention
|
|
|
|
- [ ] **Redundancy:** Configure backup HSM for failover
|
|
- [ ] **Monitoring:** Alert on HSM connection failures immediately
|
|
- [ ] **Keep-alive:** Enable session keep-alive to prevent timeouts
|
|
- [ ] **Testing:** Include HSM health in regular health checks
|
|
|
|
---
|
|
|
|
## Related Resources
|
|
|
|
- **Architecture:** `docs/modules/cryptography/hsm-integration.md`
|
|
- **Related runbooks:** `attestor-signing-failed.md`, `crypto-ops.md`
|
|
- **Doctor check:** `src/Doctor/__Plugins/StellaOps.Doctor.Plugin.Crypto/`
|
|
- **HSM setup:** `docs/operations/hsm-configuration.md`
|