synergy moats product advisory implementations

docs/operations/runbooks/attestor-signing-failed.md

@@ -0,0 +1,176 @@
+# Runbook: Attestor - Signature Generation Failures
+
+> **Sprint:** SPRINT_20260117_029_DOCS_runbook_coverage
+> **Task:** RUN-005 - Attestor Runbooks
+
+## Metadata
+
+| Field | Value |
+|-------|-------|
+| **Component** | Attestor |
+| **Severity** | Critical |
+| **On-call scope** | Platform team, Security team |
+| **Last updated** | 2026-01-17 |
+| **Doctor check** | `check.attestor.signing-health` |
+
+---
+
+## Symptoms
+
+- [ ] Attestation requests failing with "signing failed" error
+- [ ] Alert `AttestorSigningFailed` firing
+- [ ] Evidence bundles missing signatures
+- [ ] Metric `attestor_signing_failures_total` increasing
+- [ ] Release pipeline blocked due to unsigned attestations
+
+---
+
+## Impact
+
+| Impact Type | Description |
+|-------------|-------------|
+| **User-facing** | Releases blocked; attestations cannot be created |
+| **Data integrity** | Evidence is recorded but unsigned; can be signed later |
+| **SLA impact** | Release SLO violated; evidence integrity compromised |
+
+---
+
+## Diagnosis
+
+### Quick checks
+
+1. **Check Doctor diagnostics:**
+   ```bash
+   stella doctor --check check.attestor.signing-health
+   ```
+
+2. **Check attestor service status:**
+   ```bash
+   stella attest status
+   ```
+
+3. **Check signing key availability:**
+   ```bash
+   stella keys list --type signing --status active
+   ```
+   Problem if: No active signing keys
+
+### Deep diagnosis
+
+1. **Test signing operation:**
+   ```bash
+   stella attest test-sign --verbose
+   ```
+   Look for: Specific error message
+
+2. **Check key material access:**
+   ```bash
+   stella keys verify <key-id> --operation sign
+   ```
+
+3. **If using HSM, check HSM connectivity:**
+   ```bash
+   stella doctor --check check.crypto.hsm-availability
+   ```
+
+4. **Check for key expiration:**
+   ```bash
+   stella keys list --expiring-within 7d
+   ```
+
+---
+
+## Resolution
+
+### Immediate mitigation
+
+1. **If key expired, rotate to backup key:**
+   ```bash
+   stella keys activate <backup-key-id>
+   stella attest config set signing.key_id <backup-key-id>
+   ```
+
+2. **If HSM unavailable, switch to software signing (temporary):**
+   ```bash
+   stella attest config set signing.mode software
+   stella attest reload
+   ```
+   ⚠️ **Warning:** Software signing may not meet compliance requirements
+
+3. **Retry failed attestations:**
+   ```bash
+   stella attest retry --failed --last 1h
+   ```
+
+### Root cause fix
+
+**If key expired:**
+
+1. Generate new signing key:
+   ```bash
+   stella keys generate --type signing --algorithm ecdsa-p256
+   ```
+
+2. Configure key rotation schedule:
+   ```bash
+   stella keys config set rotation.auto true
+   stella keys config set rotation.overlap_days 14
+   ```
+
+**If HSM connection failed:**
+
+1. Verify HSM configuration:
+   ```bash
+   stella crypto hsm verify
+   ```
+
+2. Restart HSM connection:
+   ```bash
+   stella crypto hsm reconnect
+   ```
+
+**If certificate chain issue:**
+
+1. Verify certificate chain:
+   ```bash
+   stella crypto cert verify-chain --key <key-id>
+   ```
+
+2. Update intermediate certificates:
+   ```bash
+   stella crypto cert update-chain --key <key-id>
+   ```
+
+### Verification
+
+```bash
+# Test signing
+stella attest test-sign
+
+# Create test attestation
+stella attest create --type test --subject "test:verification"
+
+# Verify the attestation
+stella verify attestation --last
+
+# Check no failures in recent operations
+stella attest logs --level error --last 30m
+```
+
+---
+
+## Prevention
+
+- [ ] **Key rotation:** Enable automatic key rotation with 14-day overlap
+- [ ] **Monitoring:** Alert on keys expiring within 30 days
+- [ ] **Backup:** Maintain backup signing key in different HSM slot
+- [ ] **Testing:** Include signing test in health check schedule
+
+---
+
+## Related Resources
+
+- **Architecture:** `docs/modules/attestor/architecture.md`
+- **Related runbooks:** `attestor-key-expired.md`, `attestor-hsm-connection.md`
+- **Doctor check:** `src/Doctor/__Plugins/StellaOps.Doctor.Plugin.Attestor/`
+- **Dashboard:** Grafana > Stella Ops > Attestor