git.stella-ops.org/docs/operations/dual-control-ceremony-runbook.md

# Dual-Control Ceremony Runbook

This runbook documents M-of-N threshold signing ceremonies for high-assurance key operations in Stella Ops.

> **Sprint:** SPRINT_20260112_018_SIGNER_dual_control_ceremonies

## Overview

Dual-control ceremonies ensure critical cryptographic operations require approval from multiple authorized individuals before execution. This prevents single points of compromise for sensitive operations like:

- Root key rotation
- Trust anchor updates
- Emergency key revocation
- HSM key generation
- Recovery key activation

## When Ceremonies Are Required

| Operation | Default Threshold | Configurable |
|-----------|------------------|--------------|
| Root signing key rotation | 2-of-3 | Yes |
| Trust anchor update | 2-of-3 | Yes |
| Key revocation | 2-of-3 | Yes |
| HSM key generation | 2-of-4 | Yes |
| Recovery key activation | 3-of-5 | Yes |

## Ceremony Lifecycle

### State Machine

```
         +------------------+
         |     Pending      |
         +--------+---------+
                  |
                  | Approvals collected
                  v
    +-------------+-------------+
    |   PartiallyApproved      |
    +-------------+-------------+
                  |
                  | Threshold reached
                  v
         +--------+---------+
         |     Approved     |
         +--------+---------+
                  |
                  | Execute
                  v
         +--------+---------+
         |     Executed     |
         +------------------+

   Alternative paths:
   - Pending -> Expired (timeout)
   - Pending -> Cancelled (initiator cancel)
   - PartiallyApproved -> Expired (timeout)
   - PartiallyApproved -> Cancelled
```

### State Descriptions

| State | Description |
|-------|-------------|
| `Pending` | Ceremony created, awaiting first approval |
| `PartiallyApproved` | At least one approval, threshold not reached |
| `Approved` | Threshold reached, ready for execution |
| `Executed` | Operation completed successfully |
| `Expired` | Timeout reached without execution |
| `Cancelled` | Explicitly cancelled before execution |

## Creating a Ceremony

### Via CLI

```bash
stella ceremony create \
  --type key-rotation \
  --subject "Root signing key Q1-2026" \
  --threshold 2 \
  --required-approvers 3 \
  --expires-in 24h \
  --payload '{"keyId": "root-2026-q1", "algorithm": "ecdsa-p384"}'
```

### Via API

```bash
curl -X POST https://signer.example.com/api/v1/ceremonies \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "type": "key-rotation",
    "subject": "Root signing key Q1-2026",
    "threshold": 2,
    "requiredApprovers": 3,
    "expiresAt": "2026-01-17T10:00:00Z",
    "payload": {
      "keyId": "root-2026-q1",
      "algorithm": "ecdsa-p384"
    }
  }'
```

### Response

```json
{
  "ceremonyId": "cer-abc123",
  "type": "key-rotation",
  "state": "Pending",
  "threshold": 2,
  "requiredApprovers": 3,
  "currentApprovals": 0,
  "createdAt": "2026-01-16T10:00:00Z",
  "expiresAt": "2026-01-17T10:00:00Z",
  "initiator": "admin@company.com"
}
```

## Approving a Ceremony

### Prerequisites

Approvers must:
1. Be in the ceremony's allowed approvers list
2. Have the `ceremony:approve` scope
3. Have valid authentication (OIDC or break-glass)
4. Not have already approved this ceremony

### Via CLI

```bash
stella ceremony approve \
  --ceremony-id cer-abc123 \
  --reason "Reviewed rotation plan, verified key parameters" \
  --sign
```

The `--sign` flag creates a DSSE signature over the approval using the approver's signing key.

### Via API

```bash
curl -X POST https://signer.example.com/api/v1/ceremonies/cer-abc123/approve \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "reason": "Reviewed rotation plan, verified key parameters",
    "signature": "base64-encoded-dsse-signature"
  }'
```

### Approval Response

```json
{
  "ceremonyId": "cer-abc123",
  "state": "PartiallyApproved",
  "currentApprovals": 1,
  "threshold": 2,
  "approval": {
    "approvalId": "apr-def456",
    "approver": "security-lead@company.com",
    "approvedAt": "2026-01-16T11:30:00Z",
    "reason": "Reviewed rotation plan, verified key parameters",
    "signatureValid": true
  }
}
```

## Executing a Ceremony

Once the approval threshold is reached:

### Via CLI

```bash
stella ceremony execute --ceremony-id cer-abc123
```

### Via API

```bash
curl -X POST https://signer.example.com/api/v1/ceremonies/cer-abc123/execute \
  -H "Authorization: Bearer $TOKEN"
```

### Execution Response

```json
{
  "ceremonyId": "cer-abc123",
  "state": "Executed",
  "executedAt": "2026-01-16T14:00:00Z",
  "result": {
    "keyId": "root-2026-q1",
    "publicKey": "-----BEGIN PUBLIC KEY-----...",
    "fingerprint": "SHA256:abc123...",
    "activatedAt": "2026-01-16T14:00:00Z"
  }
}
```

## Monitoring Ceremonies

### List Active Ceremonies

```bash
# CLI
stella ceremony list --state pending,partially-approved

# API
curl "https://signer.example.com/api/v1/ceremonies?state=pending,partially-approved"
```

### Check Ceremony Status

```bash
# CLI
stella ceremony status --ceremony-id cer-abc123

# API
curl "https://signer.example.com/api/v1/ceremonies/cer-abc123"
```

## Cancelling a Ceremony

Ceremonies can be cancelled before execution:

```bash
# CLI
stella ceremony cancel \
  --ceremony-id cer-abc123 \
  --reason "Postponed due to schedule conflict"

# API
curl -X DELETE https://signer.example.com/api/v1/ceremonies/cer-abc123 \
  -H "Authorization: Bearer $TOKEN"
```

Only the initiator or users with `ceremony:cancel` scope can cancel.

## Audit Events

All ceremony actions are logged:

| Event | Description |
|-------|-------------|
| `signer.ceremony.initiated` | Ceremony created |
| `signer.ceremony.approved` | Approval submitted |
| `signer.ceremony.approval_rejected` | Approval rejected (invalid signature, unauthorized) |
| `signer.ceremony.executed` | Operation executed |
| `signer.ceremony.expired` | Timeout reached |
| `signer.ceremony.cancelled` | Explicitly cancelled |

### Audit Event Structure

```json
{
  "eventType": "signer.ceremony.approved",
  "timestamp": "2026-01-16T11:30:00Z",
  "ceremonyId": "cer-abc123",
  "ceremonyType": "key-rotation",
  "actor": "security-lead@company.com",
  "approvalId": "apr-def456",
  "currentApprovals": 1,
  "threshold": 2,
  "signatureAlgorithm": "ecdsa-p256",
  "signatureKeyId": "user-signing-key-456"
}
```

### Query Audit Logs

```bash
stella audit query \
  --event-type "signer.ceremony.*" \
  --since 7d \
  --ceremony-id cer-abc123
```

## Configuration

### Ceremony Settings

```yaml
# signer-config.yaml
ceremonies:
  enabled: true
  defaultTimeout: 24h
  maxTimeout: 168h  # 7 days
  requireSignedApprovals: true
  
  thresholds:
    key-rotation: 
      minimum: 2
      default: 2
      maximum: 5
    key-revocation:
      minimum: 2
      default: 3
      maximum: 5
    trust-anchor-update:
      minimum: 2
      default: 2
      maximum: 4
```

### Approver Configuration

```yaml
# approvers.yaml
approverGroups:
  - name: key-custodians
    members:
      - security-lead@company.com
      - ciso@company.com
      - key-officer-1@company.com
      - key-officer-2@company.com
    operations:
      - key-rotation
      - key-revocation
      
  - name: trust-admins
    members:
      - trust-admin@company.com
      - security-lead@company.com
    operations:
      - trust-anchor-update
```

## Notifications

Ceremonies trigger notifications to approvers:

| Event | Notification |
|-------|-------------|
| Ceremony created | Email/Slack to all eligible approvers |
| Approval submitted | Email/Slack to remaining approvers |
| Threshold reached | Email/Slack to initiator |
| Approaching expiry | Email/Slack at 75% and 90% of timeout |
| Expired | Email/Slack to initiator and approvers |

Configure notifications in `notifier-config.yaml`:

```yaml
notifications:
  ceremonies:
    enabled: true
    channels:
      - type: email
        recipients: "@approverGroup"
      - type: slack
        webhook: ${SLACK_CEREMONY_WEBHOOK}
        channel: "#key-ceremonies"
```

## Security Best Practices

### Approver Requirements

- Maintain at least N+1 approvers for N-of-M ceremonies
- Distribute approvers across security domains
- Require hardware tokens for signing keys
- Rotate approver list quarterly

### Ceremony Hygiene

- Use descriptive subjects for audit clarity
- Set reasonable timeouts (not too long, not too short)
- Document approval reasons thoroughly
- Review executed ceremonies monthly

### Monitoring

Set up alerts for:

```yaml
alerts:
  - name: CeremonyPendingTooLong
    condition: ceremony.pending_duration > 12h
    severity: warning
    
  - name: CeremonyApprovalRejected
    condition: ceremony.approval_rejected
    severity: critical
    
  - name: UnauthorizedCeremonyAttempt
    condition: ceremony.unauthorized_access
    severity: critical
```

## Troubleshooting

### Common Issues

| Issue | Cause | Resolution |
|-------|-------|------------|
| Approval rejected | Invalid signature | Re-sign with correct key |
| Cannot approve | Already approved | Different approver must approve |
| Cannot execute | Threshold not met | Collect more approvals |
| Ceremony expired | Timeout reached | Create new ceremony |

### Signature Verification Failures

```bash
# Verify signing key is accessible
stella auth keys list

# Test signature
echo "test" | stella sign --key-id my-signing-key | stella verify

# Check key permissions
stella auth keys info --key-id my-signing-key
```

## Emergency Procedures

### Stuck Ceremony

If a ceremony is stuck (approvers unavailable):

1. Cancel the stuck ceremony
2. Create new ceremony with available approvers
3. Document the situation in audit notes

### Compromised Approver

If an approver's credentials are compromised:

1. Revoke approver's signing key immediately
2. Cancel any pending ceremonies they created
3. Review recent approvals for anomalies
4. Remove from approver groups
5. Document in incident report

## Related Documentation

- [Key Rotation Runbook](./key-rotation-runbook.md)
- [HSM Setup Runbook](./hsm-setup-runbook.md)
- [Signer Architecture](../modules/signer/architecture.md)
- [Break-Glass Runbook](./break-glass-runbook.md)