191 lines
4.3 KiB
Markdown
191 lines
4.3 KiB
Markdown
# Runbook: Attestor - Signing Key Expired
|
|
|
|
> **Sprint:** SPRINT_20260117_029_DOCS_runbook_coverage
|
|
> **Task:** RUN-005 - Attestor Runbooks
|
|
|
|
## Metadata
|
|
|
|
| Field | Value |
|
|
|-------|-------|
|
|
| **Component** | Attestor |
|
|
| **Severity** | Critical |
|
|
| **On-call scope** | Platform team, Security team |
|
|
| **Last updated** | 2026-01-17 |
|
|
| **Doctor check** | `check.attestor.key-expiration` |
|
|
|
|
---
|
|
|
|
## Symptoms
|
|
|
|
- [ ] Attestation creation failing with "key expired" error
|
|
- [ ] Alert `AttestorKeyExpired` firing
|
|
- [ ] Error: "signing key certificate has expired"
|
|
- [ ] New attestations cannot be created
|
|
- [ ] Verification of new attestations failing
|
|
|
|
---
|
|
|
|
## Impact
|
|
|
|
| Impact Type | Description |
|
|
|-------------|-------------|
|
|
| **User-facing** | No new attestations can be signed; releases blocked |
|
|
| **Data integrity** | Existing attestations remain valid; new ones cannot be created |
|
|
| **SLA impact** | Release SLO violated; compliance posture compromised |
|
|
|
|
---
|
|
|
|
## Diagnosis
|
|
|
|
### Quick checks
|
|
|
|
1. **Check Doctor diagnostics:**
|
|
```bash
|
|
stella doctor --check check.attestor.key-expiration
|
|
```
|
|
|
|
2. **List signing keys and expiration:**
|
|
```bash
|
|
stella keys list --type signing --show-expiration
|
|
```
|
|
Look for: Keys with status "expired" or expiring soon
|
|
|
|
3. **Check active signing key:**
|
|
```bash
|
|
stella attest config get signing.key_id
|
|
stella keys show <key-id> --details
|
|
```
|
|
|
|
### Deep diagnosis
|
|
|
|
1. **Check certificate chain validity:**
|
|
```bash
|
|
stella crypto cert verify-chain --key <key-id>
|
|
```
|
|
Problem if: Any certificate in chain expired
|
|
|
|
2. **Check for backup keys:**
|
|
```bash
|
|
stella keys list --type signing --status inactive
|
|
```
|
|
Look for: Unexpired backup keys that can be activated
|
|
|
|
3. **Check key rotation history:**
|
|
```bash
|
|
stella keys rotation-history --key <key-id>
|
|
```
|
|
|
|
---
|
|
|
|
## Resolution
|
|
|
|
### Immediate mitigation
|
|
|
|
1. **If backup key available, activate it:**
|
|
```bash
|
|
stella keys activate <backup-key-id>
|
|
stella attest config set signing.key_id <backup-key-id>
|
|
stella attest reload
|
|
```
|
|
|
|
2. **Verify signing works:**
|
|
```bash
|
|
stella attest test-sign
|
|
```
|
|
|
|
3. **Retry failed attestations:**
|
|
```bash
|
|
stella attest retry --failed --last 1h
|
|
```
|
|
|
|
### Root cause fix
|
|
|
|
**Generate new signing key:**
|
|
|
|
1. Generate new key pair:
|
|
```bash
|
|
stella keys generate \
|
|
--type signing \
|
|
--algorithm ecdsa-p256 \
|
|
--validity 365d \
|
|
--name "signing-key-$(date +%Y%m%d)"
|
|
```
|
|
|
|
2. If using HSM:
|
|
```bash
|
|
stella keys generate \
|
|
--type signing \
|
|
--algorithm ecdsa-p256 \
|
|
--validity 365d \
|
|
--hsm-slot <slot> \
|
|
--name "signing-key-$(date +%Y%m%d)"
|
|
```
|
|
|
|
3. Register the new key:
|
|
```bash
|
|
stella keys register <new-key-id> --purpose attestation-signing
|
|
```
|
|
|
|
4. Update signing configuration:
|
|
```bash
|
|
stella attest config set signing.key_id <new-key-id>
|
|
stella attest reload
|
|
```
|
|
|
|
5. Publish new public key to trust anchors:
|
|
```bash
|
|
stella issuer keys publish <new-key-id>
|
|
```
|
|
|
|
**Configure automatic rotation:**
|
|
|
|
1. Enable auto-rotation:
|
|
```bash
|
|
stella keys config set rotation.auto true
|
|
stella keys config set rotation.before_expiry 30d
|
|
stella keys config set rotation.overlap_days 14
|
|
```
|
|
|
|
2. Set up rotation alerts:
|
|
```bash
|
|
stella keys config set alerts.expiring_days 30
|
|
stella keys config set alerts.expiring_days_critical 7
|
|
```
|
|
|
|
### Verification
|
|
|
|
```bash
|
|
# Verify new key is active
|
|
stella keys list --type signing --status active
|
|
|
|
# Test signing
|
|
stella attest test-sign
|
|
|
|
# Create test attestation
|
|
stella attest create --type test --subject "test:key-rotation"
|
|
|
|
# Verify the attestation
|
|
stella verify attestation --last
|
|
|
|
# Check key expiration
|
|
stella keys show <new-key-id> --details | grep -i expir
|
|
```
|
|
|
|
---
|
|
|
|
## Prevention
|
|
|
|
- [ ] **Rotation:** Enable automatic key rotation 30 days before expiry
|
|
- [ ] **Monitoring:** Alert on keys expiring within 30 days (warning) and 7 days (critical)
|
|
- [ ] **Backup:** Maintain at least one backup signing key
|
|
- [ ] **Documentation:** Document key rotation procedures and approval process
|
|
|
|
---
|
|
|
|
## Related Resources
|
|
|
|
- **Architecture:** `docs/modules/attestor/architecture.md`
|
|
- **Related runbooks:** `attestor-signing-failed.md`, `attestor-hsm-connection.md`
|
|
- **Doctor check:** `src/Doctor/__Plugins/StellaOps.Doctor.Plugin.Attestor/`
|
|
- **Key management:** `docs/operations/key-management.md`
|