stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 2 Releases Wiki Activity
Files
702a27ac839f1de50f9c9f460000b8f7b2242efd
git.stella-ops.org/docs/operations/runbooks/attestor-signing-failed.md

3.9 KiB
Raw Blame History

Runbook: Attestor - Signature Generation Failures

Sprint: SPRINT_20260117_029_DOCS_runbook_coverage Task: RUN-005 - Attestor Runbooks

Metadata

Field Value
Component Attestor
Severity Critical
On-call scope Platform team, Security team
Last updated 2026-01-17
Doctor check check.attestor.signing-health

Symptoms

  • Attestation requests failing with "signing failed" error
  • Alert AttestorSigningFailed firing
  • Evidence bundles missing signatures
  • Metric attestor_signing_failures_total increasing
  • Release pipeline blocked due to unsigned attestations

Impact

Impact Type Description
User-facing Releases blocked; attestations cannot be created
Data integrity Evidence is recorded but unsigned; can be signed later
SLA impact Release SLO violated; evidence integrity compromised

Diagnosis

Quick checks

  1. Check Doctor diagnostics:

    stella doctor --check check.attestor.signing-health

  2. Check attestor service status:

    stella attest status

  3. Check signing key availability:

    stella keys list --type signing --status active

    Problem if: No active signing keys

Deep diagnosis

  1. Test signing operation:

    stella attest test-sign --verbose

    Look for: Specific error message

  2. Check key material access:

    stella keys verify <key-id> --operation sign

  3. If using HSM, check HSM connectivity:

    stella doctor --check check.crypto.hsm-availability

  4. Check for key expiration:

    stella keys list --expiring-within 7d

Resolution

Immediate mitigation

  1. If key expired, rotate to backup key:

    stella keys activate <backup-key-id>
stella attest config set signing.key_id <backup-key-id>

  2. If HSM unavailable, switch to software signing (temporary):

    stella attest config set signing.mode software
stella attest reload

    ⚠️ Warning: Software signing may not meet compliance requirements

  3. Retry failed attestations:

    stella attest retry --failed --last 1h

Root cause fix

If key expired:

  1. Generate new signing key:

    stella keys generate --type signing --algorithm ecdsa-p256

  2. Configure key rotation schedule:

    stella keys config set rotation.auto true
stella keys config set rotation.overlap_days 14

If HSM connection failed:

  1. Verify HSM configuration:

    stella crypto hsm verify

  2. Restart HSM connection:

    stella crypto hsm reconnect

If certificate chain issue:

  1. Verify certificate chain:

    stella crypto cert verify-chain --key <key-id>

  2. Update intermediate certificates:

    stella crypto cert update-chain --key <key-id>

Verification

# Test signing
stella attest test-sign

# Create test attestation
stella attest create --type test --subject "test:verification"

# Verify the attestation
stella verify attestation --last

# Check no failures in recent operations
stella attest logs --level error --last 30m

Prevention

  • Key rotation: Enable automatic key rotation with 14-day overlap
  • Monitoring: Alert on keys expiring within 30 days
  • Backup: Maintain backup signing key in different HSM slot
  • Testing: Include signing test in health check schedule

Related Resources

  • Architecture: docs/modules/attestor/architecture.md
  • Related runbooks: attestor-key-expired.md, attestor-hsm-connection.md
  • Doctor check: src/Doctor/__Plugins/StellaOps.Doctor.Plugin.Attestor/
  • Dashboard: Grafana > Stella Ops > Attestor