177 lines
3.9 KiB
Markdown
177 lines
3.9 KiB
Markdown
# Runbook: Attestor - Signature Generation Failures
|
|
|
|
> **Sprint:** SPRINT_20260117_029_DOCS_runbook_coverage
|
|
> **Task:** RUN-005 - Attestor Runbooks
|
|
|
|
## Metadata
|
|
|
|
| Field | Value |
|
|
|-------|-------|
|
|
| **Component** | Attestor |
|
|
| **Severity** | Critical |
|
|
| **On-call scope** | Platform team, Security team |
|
|
| **Last updated** | 2026-01-17 |
|
|
| **Doctor check** | `check.attestor.signing-health` |
|
|
|
|
---
|
|
|
|
## Symptoms
|
|
|
|
- [ ] Attestation requests failing with "signing failed" error
|
|
- [ ] Alert `AttestorSigningFailed` firing
|
|
- [ ] Evidence bundles missing signatures
|
|
- [ ] Metric `attestor_signing_failures_total` increasing
|
|
- [ ] Release pipeline blocked due to unsigned attestations
|
|
|
|
---
|
|
|
|
## Impact
|
|
|
|
| Impact Type | Description |
|
|
|-------------|-------------|
|
|
| **User-facing** | Releases blocked; attestations cannot be created |
|
|
| **Data integrity** | Evidence is recorded but unsigned; can be signed later |
|
|
| **SLA impact** | Release SLO violated; evidence integrity compromised |
|
|
|
|
---
|
|
|
|
## Diagnosis
|
|
|
|
### Quick checks
|
|
|
|
1. **Check Doctor diagnostics:**
|
|
```bash
|
|
stella doctor --check check.attestor.signing-health
|
|
```
|
|
|
|
2. **Check attestor service status:**
|
|
```bash
|
|
stella attest status
|
|
```
|
|
|
|
3. **Check signing key availability:**
|
|
```bash
|
|
stella keys list --type signing --status active
|
|
```
|
|
Problem if: No active signing keys
|
|
|
|
### Deep diagnosis
|
|
|
|
1. **Test signing operation:**
|
|
```bash
|
|
stella attest test-sign --verbose
|
|
```
|
|
Look for: Specific error message
|
|
|
|
2. **Check key material access:**
|
|
```bash
|
|
stella keys verify <key-id> --operation sign
|
|
```
|
|
|
|
3. **If using HSM, check HSM connectivity:**
|
|
```bash
|
|
stella doctor --check check.crypto.hsm-availability
|
|
```
|
|
|
|
4. **Check for key expiration:**
|
|
```bash
|
|
stella keys list --expiring-within 7d
|
|
```
|
|
|
|
---
|
|
|
|
## Resolution
|
|
|
|
### Immediate mitigation
|
|
|
|
1. **If key expired, rotate to backup key:**
|
|
```bash
|
|
stella keys activate <backup-key-id>
|
|
stella attest config set signing.key_id <backup-key-id>
|
|
```
|
|
|
|
2. **If HSM unavailable, switch to software signing (temporary):**
|
|
```bash
|
|
stella attest config set signing.mode software
|
|
stella attest reload
|
|
```
|
|
⚠️ **Warning:** Software signing may not meet compliance requirements
|
|
|
|
3. **Retry failed attestations:**
|
|
```bash
|
|
stella attest retry --failed --last 1h
|
|
```
|
|
|
|
### Root cause fix
|
|
|
|
**If key expired:**
|
|
|
|
1. Generate new signing key:
|
|
```bash
|
|
stella keys generate --type signing --algorithm ecdsa-p256
|
|
```
|
|
|
|
2. Configure key rotation schedule:
|
|
```bash
|
|
stella keys config set rotation.auto true
|
|
stella keys config set rotation.overlap_days 14
|
|
```
|
|
|
|
**If HSM connection failed:**
|
|
|
|
1. Verify HSM configuration:
|
|
```bash
|
|
stella crypto hsm verify
|
|
```
|
|
|
|
2. Restart HSM connection:
|
|
```bash
|
|
stella crypto hsm reconnect
|
|
```
|
|
|
|
**If certificate chain issue:**
|
|
|
|
1. Verify certificate chain:
|
|
```bash
|
|
stella crypto cert verify-chain --key <key-id>
|
|
```
|
|
|
|
2. Update intermediate certificates:
|
|
```bash
|
|
stella crypto cert update-chain --key <key-id>
|
|
```
|
|
|
|
### Verification
|
|
|
|
```bash
|
|
# Test signing
|
|
stella attest test-sign
|
|
|
|
# Create test attestation
|
|
stella attest create --type test --subject "test:verification"
|
|
|
|
# Verify the attestation
|
|
stella verify attestation --last
|
|
|
|
# Check no failures in recent operations
|
|
stella attest logs --level error --last 30m
|
|
```
|
|
|
|
---
|
|
|
|
## Prevention
|
|
|
|
- [ ] **Key rotation:** Enable automatic key rotation with 14-day overlap
|
|
- [ ] **Monitoring:** Alert on keys expiring within 30 days
|
|
- [ ] **Backup:** Maintain backup signing key in different HSM slot
|
|
- [ ] **Testing:** Include signing test in health check schedule
|
|
|
|
---
|
|
|
|
## Related Resources
|
|
|
|
- **Architecture:** `docs/modules/attestor/architecture.md`
|
|
- **Related runbooks:** `attestor-key-expired.md`, `attestor-hsm-connection.md`
|
|
- **Doctor check:** `src/Doctor/__Plugins/StellaOps.Doctor.Plugin.Attestor/`
|
|
- **Dashboard:** Grafana > Stella Ops > Attestor
|