git.stella-ops.org/docs/operations/key-rotation-runbook.md

# Key Rotation Runbook

> **Module**: Signer / Key Management  
> **Version**: 1.0.0  
> **Last Updated**: 2025-12-17  

This runbook describes procedures for managing signing key lifecycle in StellaOps, including key rotation, revocation, and trust anchor management.

---

## Overview

StellaOps uses signing keys to create DSSE envelopes for proof chain attestations. Key rotation is critical for:
- Limiting exposure from compromised keys
- Compliance with key age policies (e.g., NIST SP 800-57)
- Transitioning between cryptographic algorithms

### Key Principles

1. **Never mutate old DSSE envelopes** - Signed content is immutable
2. **Never remove keys from history** - Move to `revokedKeys`, don't delete
3. **Publish key material** - Via attestation feed or Rekor-mirror
4. **Audit all changes** - Full log of key lifecycle events
5. **Maintain key version history** - For forensic verification

---

## Signing Key Profiles

StellaOps supports multiple signing key profiles for different security requirements:

| Profile | Algorithm | Key Store | Use Case |
|---------|-----------|-----------|----------|
| `default` | SHA256-ED25519 | AWS KMS | Standard production |
| `fips` | SHA256-ECDSA-P256 | HSM (PKCS#11) | FIPS 140-2 environments |
| `gost` | GOST-R-34.10-2012 | Local HSM | Russian regulatory |
| `sm2` | SM2-P256 | Local HSM | Chinese regulatory |
| `pqc` | ML-DSA-65 | Software | Post-quantum ready |

### Profile Configuration

```yaml
# /etc/stellaops/signer.yaml
signer:
  profiles:
    default:
      algorithm: "SHA256-ED25519"
      keyStore: "kms://aws/key/stellaops-default"
      rotation:
        enabled: true
        maxAgeMonths: 12
        warningMonths: 2

    fips:
      algorithm: "SHA256-ECDSA-P256"
      keyStore: "hsm://pkcs11/slot/0"
      rotation:
        enabled: true
        maxAgeMonths: 12
        warningMonths: 2
```

---

## Key Rotation Workflow

### Step 1: Generate New Key

Generate a new signing key in the configured key store:

```bash
# Using CLI
stellaops key generate \
  --profile default \
  --key-id key-2025-prod \
  --algorithm SHA256-ED25519

# Via API
curl -X POST https://api.stellaops.local/v1/signer/keys \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"profile": "default", "keyId": "key-2025-prod", "algorithm": "SHA256-ED25519"}'
```

### Step 2: Add Key to Trust Anchor

Add the new key to the trust anchor without removing the old key:

```bash
# Using CLI
stellaops anchor add-key \
  --anchor-id 550e8400-e29b-41d4-a716-446655440000 \
  --key-id key-2025-prod

# Via API
curl -X POST https://api.stellaops.local/v1/anchors/550e8400.../keys \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"keyid": "key-2025-prod", "publicKey": "<pem-encoded>"}'
```

**Result:** Trust anchor now accepts signatures from both old and new keys.

### Step 3: Transition Period

During transition:
- New signatures are created with the new key
- Old proofs are verified with either key
- Monitor for verification failures

**Recommended transition period:** 2-4 weeks

```bash
# Check verification status
stellaops anchor status --anchor-id 550e8400...

# Expected output:
# Anchor: 550e8400-e29b-41d4-a716-446655440000
# Active Keys: key-2024-prod, key-2025-prod
# Verification Success Rate: 100%
# Pending Rescans: 0
```

### Step 4: Revoke Old Key (Optional)

After transition is complete, revoke the old key:

```bash
# Using CLI
stellaops anchor revoke-key \
  --anchor-id 550e8400... \
  --key-id key-2024-prod \
  --reason "annual-rotation" \
  --effective-at "2025-02-01T00:00:00Z"

# Via API
curl -X POST https://api.stellaops.local/v1/anchors/550e8400.../keys/key-2024-prod/revoke \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"reason": "annual-rotation", "effectiveAt": "2025-02-01T00:00:00Z"}'
```

**Important:** The old key remains valid for verifying proofs signed before the revocation date.

### Step 5: Publish Key Material

Publish updated key material:

```bash
# Update attestation feed
stellaops feed publish --include-keys

# Sync to Rekor mirror (if applicable)
stellaops rekor sync --keys-only
```

---

## Trust Anchor Management

### Trust Anchor Structure

```json
{
  "trustAnchorId": "550e8400-e29b-41d4-a716-446655440000",
  "purlPattern": "pkg:npm/*",
  "allowedKeyids": ["key-2024-prod", "key-2025-prod"],
  "allowedPredicateTypes": [
    "evidence.stella/v1",
    "reasoning.stella/v1",
    "cdx-vex.stella/v1",
    "proofspine.stella/v1"
  ],
  "policyVersion": "v2.3.1",
  "revokedKeys": ["key-2023-prod"],
  "keyHistory": [
    {
      "keyid": "key-2023-prod",
      "addedAt": "2023-01-15T00:00:00Z",
      "revokedAt": "2024-01-15T00:00:00Z",
      "revokeReason": "annual-rotation"
    }
  ]
}
```

### Create Trust Anchor

```bash
stellaops anchor create \
  --purl-pattern "pkg:npm/*" \
  --key-ids key-2025-prod \
  --predicate-types evidence.stella/v1,reasoning.stella/v1
```

### List Trust Anchors

```bash
stellaops anchor list

# Output:
# ID                                    Pattern        Keys              Status
# 550e8400-e29b-41d4-a716-446655440000  pkg:npm/*      key-2025-prod     active
# 660f9500-f39c-51e5-b827-557766551111  pkg:maven/*    key-2025-java     active
```

### PURL Pattern Matching

Trust anchors use PURL patterns for scope:

| Pattern | Matches |
|---------|---------|
| `pkg:npm/*` | All npm packages |
| `pkg:maven/org.apache.*` | Apache Maven packages |
| `pkg:docker/myregistry/*` | All images from myregistry |
| `*` | Universal (all packages) |

---

## Verification with Key History

When verifying a proof signed at time T:

1. Lookup trust anchor for the artifact PURL
2. Find keys that were valid at time T:
   - Key was added before T
   - Key was not revoked, OR revoked after T
3. Verify signature against valid keys
4. Return success if any valid key verifies

### Temporal Verification

```bash
# Verify proof at specific point in time
stellaops verify \
  --proof-bundle sha256:abc123... \
  --at-time "2024-06-15T12:00:00Z"

# Check key validity at time
stellaops key check-validity \
  --key-id key-2024-prod \
  --at-time "2024-06-15T12:00:00Z"
```

---

## Emergency Key Revocation

In case of key compromise:

### Immediate Actions

1. **Revoke the compromised key immediately**
   ```bash
   stellaops anchor revoke-key \
     --anchor-id ALL \
     --key-id compromised-key-id \
     --reason "compromise" \
     --effective-at "NOW"
   ```

2. **Generate new key**
   ```bash
   stellaops key generate \
     --profile default \
     --key-id emergency-key-$(date +%Y%m%d)
   ```

3. **Add new key to all affected anchors**
   ```bash
   stellaops anchor add-key \
     --anchor-id ALL \
     --key-id emergency-key-$(date +%Y%m%d)
   ```

4. **Publish updated key material**
   ```bash
   stellaops feed publish --include-keys --urgent
   ```

### Post-Incident Actions

1. Review all proofs signed with compromised key
2. Determine if any tampering occurred
3. Re-sign critical proofs with new key if needed
4. File incident report

---

## Rotation Warnings

Configure rotation warnings to proactively manage key lifecycle:

```yaml
signer:
  rotation:
    warningMonths: 2
    alerts:
      - type: slack
        channel: "#security-ops"
      - type: email
        recipients: ["security@example.com"]
```

### Check Rotation Warnings

```bash
stellaops key rotation-warnings

# Output:
# Key ID           Profile   Age      Max Age  Warning
# key-2024-prod    default   10mo     12mo     ⚠️ Rotation due in 2 months
# key-2024-java    fips      6mo      12mo     ✓ OK
```

---

## Audit Trail

All key operations are logged to `key_audit_log`:

| Field | Description |
|-------|-------------|
| `event_id` | Unique event identifier |
| `event_type` | `KEY_GENERATED`, `KEY_ADDED`, `KEY_REVOKED`, etc. |
| `key_id` | Affected key identifier |
| `anchor_id` | Affected trust anchor (if applicable) |
| `actor` | User/service that performed action |
| `timestamp` | UTC timestamp |
| `details` | JSON with additional context |

### Query Audit Log

```bash
stellaops audit query \
  --type KEY_* \
  --from "2025-01-01" \
  --to "2025-12-31"

# Via SQL
SELECT * FROM signer.key_audit_log
WHERE event_type LIKE 'KEY_%'
  AND timestamp >= '2025-01-01'
ORDER BY timestamp DESC;
```

---

## Database Schema

### key_history Table

```sql
CREATE TABLE signer.key_history (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    anchor_id UUID NOT NULL REFERENCES signer.trust_anchors(id),
    key_id TEXT NOT NULL,
    public_key TEXT NOT NULL,
    algorithm TEXT NOT NULL,
    added_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    revoked_at TIMESTAMPTZ,
    revoke_reason TEXT,
    metadata JSONB,
    UNIQUE(anchor_id, key_id)
);

CREATE INDEX idx_key_history_validity 
ON signer.key_history (anchor_id, added_at, revoked_at);
```

### key_audit_log Table

```sql
CREATE TABLE signer.key_audit_log (
    event_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    event_type TEXT NOT NULL,
    key_id TEXT,
    anchor_id UUID,
    actor TEXT NOT NULL,
    timestamp TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    details JSONB
);

CREATE INDEX idx_audit_log_time ON signer.key_audit_log (timestamp DESC);
CREATE INDEX idx_audit_log_key ON signer.key_audit_log (key_id);
```

---

## Metrics

Key rotation metrics exposed via Prometheus:

| Metric | Type | Description |
|--------|------|-------------|
| `signer_key_age_days` | Gauge | Age of each active key in days |
| `signer_keys_active_total` | Gauge | Number of active keys per profile |
| `signer_keys_revoked_total` | Counter | Total revoked keys |
| `signer_rotation_events_total` | Counter | Key rotation events |
| `signer_verification_key_lookups_total` | Counter | Temporal key lookups |

### Alerting Rules

```yaml
groups:
  - name: key-rotation
    rules:
      - alert: SigningKeyNearExpiry
        expr: signer_key_age_days > (365 - 60)
        for: 1d
        labels:
          severity: warning
        annotations:
          summary: "Signing key approaching rotation deadline"
          
      - alert: SigningKeyExpired
        expr: signer_key_age_days > 365
        for: 1h
        labels:
          severity: critical
        annotations:
          summary: "Signing key exceeded maximum age"
```

---

## Related Documentation

- [Proof Chain API](../api/proofs.md)
- [Attestor Architecture](../modules/attestor/architecture.md)
- [Signer Architecture](../modules/signer/architecture.md)
- [NIST SP 800-57](https://csrc.nist.gov/publications/detail/sp/800-57-part-1/rev-5/final) - Key Management Guidelines