stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 2 Releases Wiki Activity
Files
main
git.stella-ops.org/docs/operations/dual-control-ceremony-runbook.md

10 KiB
Raw Permalink Blame History

Dual-Control Ceremony Runbook

This runbook documents M-of-N threshold signing ceremonies for high-assurance key operations in Stella Ops.

Sprint: SPRINT_20260112_018_SIGNER_dual_control_ceremonies

Overview

Dual-control ceremonies ensure critical cryptographic operations require approval from multiple authorized individuals before execution. This prevents single points of compromise for sensitive operations like:

  • Root key rotation
  • Trust anchor updates
  • Emergency key revocation
  • HSM key generation
  • Recovery key activation

When Ceremonies Are Required

Operation Default Threshold Configurable
Root signing key rotation 2-of-3 Yes
Trust anchor update 2-of-3 Yes
Key revocation 2-of-3 Yes
HSM key generation 2-of-4 Yes
Recovery key activation 3-of-5 Yes

Ceremony Lifecycle

State Machine

         +------------------+
         |     Pending      |
         +--------+---------+
                  |
                  | Approvals collected
                  v
    +-------------+-------------+
    |   PartiallyApproved      |
    +-------------+-------------+
                  |
                  | Threshold reached
                  v
         +--------+---------+
         |     Approved     |
         +--------+---------+
                  |
                  | Execute
                  v
         +--------+---------+
         |     Executed     |
         +------------------+

   Alternative paths:
   - Pending -> Expired (timeout)
   - Pending -> Cancelled (initiator cancel)
   - PartiallyApproved -> Expired (timeout)
   - PartiallyApproved -> Cancelled

State Descriptions

State Description
Pending Ceremony created, awaiting first approval
PartiallyApproved At least one approval, threshold not reached
Approved Threshold reached, ready for execution
Executed Operation completed successfully
Expired Timeout reached without execution
Cancelled Explicitly cancelled before execution

Creating a Ceremony

Via CLI

stella ceremony create \
  --type key-rotation \
  --subject "Root signing key Q1-2026" \
  --threshold 2 \
  --required-approvers 3 \
  --expires-in 24h \
  --payload '{"keyId": "root-2026-q1", "algorithm": "ecdsa-p384"}'

Via API

curl -X POST https://signer.example.com/api/v1/ceremonies \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "type": "key-rotation",
    "subject": "Root signing key Q1-2026",
    "threshold": 2,
    "requiredApprovers": 3,
    "expiresAt": "2026-01-17T10:00:00Z",
    "payload": {
      "keyId": "root-2026-q1",
      "algorithm": "ecdsa-p384"
    }
  }'

Response

{
  "ceremonyId": "cer-abc123",
  "type": "key-rotation",
  "state": "Pending",
  "threshold": 2,
  "requiredApprovers": 3,
  "currentApprovals": 0,
  "createdAt": "2026-01-16T10:00:00Z",
  "expiresAt": "2026-01-17T10:00:00Z",
  "initiator": "admin@company.com"
}

Approving a Ceremony

Prerequisites

Approvers must:

  1. Be in the ceremony's allowed approvers list
  2. Have the ceremony:approve scope
  3. Have valid authentication (OIDC or break-glass)
  4. Not have already approved this ceremony

Via CLI

stella ceremony approve \
  --ceremony-id cer-abc123 \
  --reason "Reviewed rotation plan, verified key parameters" \
  --sign

The --sign flag creates a DSSE signature over the approval using the approver's signing key.

Via API

curl -X POST https://signer.example.com/api/v1/ceremonies/cer-abc123/approve \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "reason": "Reviewed rotation plan, verified key parameters",
    "signature": "base64-encoded-dsse-signature"
  }'

Approval Response

{
  "ceremonyId": "cer-abc123",
  "state": "PartiallyApproved",
  "currentApprovals": 1,
  "threshold": 2,
  "approval": {
    "approvalId": "apr-def456",
    "approver": "security-lead@company.com",
    "approvedAt": "2026-01-16T11:30:00Z",
    "reason": "Reviewed rotation plan, verified key parameters",
    "signatureValid": true
  }
}

Executing a Ceremony

Once the approval threshold is reached:

Via CLI

stella ceremony execute --ceremony-id cer-abc123

Via API

curl -X POST https://signer.example.com/api/v1/ceremonies/cer-abc123/execute \
  -H "Authorization: Bearer $TOKEN"

Execution Response

{
  "ceremonyId": "cer-abc123",
  "state": "Executed",
  "executedAt": "2026-01-16T14:00:00Z",
  "result": {
    "keyId": "root-2026-q1",
    "publicKey": "-----BEGIN PUBLIC KEY-----...",
    "fingerprint": "SHA256:abc123...",
    "activatedAt": "2026-01-16T14:00:00Z"
  }
}

Monitoring Ceremonies

List Active Ceremonies

# CLI
stella ceremony list --state pending,partially-approved

# API
curl "https://signer.example.com/api/v1/ceremonies?state=pending,partially-approved"

Check Ceremony Status

# CLI
stella ceremony status --ceremony-id cer-abc123

# API
curl "https://signer.example.com/api/v1/ceremonies/cer-abc123"

Cancelling a Ceremony

Ceremonies can be cancelled before execution:

# CLI
stella ceremony cancel \
  --ceremony-id cer-abc123 \
  --reason "Postponed due to schedule conflict"

# API
curl -X DELETE https://signer.example.com/api/v1/ceremonies/cer-abc123 \
  -H "Authorization: Bearer $TOKEN"

Only the initiator or users with ceremony:cancel scope can cancel.

Audit Events

All ceremony actions are logged:

Event Description
signer.ceremony.initiated Ceremony created
signer.ceremony.approved Approval submitted
signer.ceremony.approval_rejected Approval rejected (invalid signature, unauthorized)
signer.ceremony.executed Operation executed
signer.ceremony.expired Timeout reached
signer.ceremony.cancelled Explicitly cancelled

Audit Event Structure

{
  "eventType": "signer.ceremony.approved",
  "timestamp": "2026-01-16T11:30:00Z",
  "ceremonyId": "cer-abc123",
  "ceremonyType": "key-rotation",
  "actor": "security-lead@company.com",
  "approvalId": "apr-def456",
  "currentApprovals": 1,
  "threshold": 2,
  "signatureAlgorithm": "ecdsa-p256",
  "signatureKeyId": "user-signing-key-456"
}

Query Audit Logs

stella audit query \
  --event-type "signer.ceremony.*" \
  --since 7d \
  --ceremony-id cer-abc123

Configuration

Ceremony Settings

# signer-config.yaml
ceremonies:
  enabled: true
  defaultTimeout: 24h
  maxTimeout: 168h  # 7 days
  requireSignedApprovals: true
  
  thresholds:
    key-rotation: 
      minimum: 2
      default: 2
      maximum: 5
    key-revocation:
      minimum: 2
      default: 3
      maximum: 5
    trust-anchor-update:
      minimum: 2
      default: 2
      maximum: 4

Approver Configuration

# approvers.yaml
approverGroups:
  - name: key-custodians
    members:
      - security-lead@company.com
      - ciso@company.com
      - key-officer-1@company.com
      - key-officer-2@company.com
    operations:
      - key-rotation
      - key-revocation
      
  - name: trust-admins
    members:
      - trust-admin@company.com
      - security-lead@company.com
    operations:
      - trust-anchor-update

Notifications

Ceremonies trigger notifications to approvers:

Event Notification
Ceremony created Email/Slack to all eligible approvers
Approval submitted Email/Slack to remaining approvers
Threshold reached Email/Slack to initiator
Approaching expiry Email/Slack at 75% and 90% of timeout
Expired Email/Slack to initiator and approvers

Configure notifications in notifier-config.yaml:

notifications:
  ceremonies:
    enabled: true
    channels:
      - type: email
        recipients: "@approverGroup"
      - type: slack
        webhook: ${SLACK_CEREMONY_WEBHOOK}
        channel: "#key-ceremonies"

Security Best Practices

Approver Requirements

  • Maintain at least N+1 approvers for N-of-M ceremonies
  • Distribute approvers across security domains
  • Require hardware tokens for signing keys
  • Rotate approver list quarterly

Ceremony Hygiene

  • Use descriptive subjects for audit clarity
  • Set reasonable timeouts (not too long, not too short)
  • Document approval reasons thoroughly
  • Review executed ceremonies monthly

Monitoring

Set up alerts for:

alerts:
  - name: CeremonyPendingTooLong
    condition: ceremony.pending_duration > 12h
    severity: warning
    
  - name: CeremonyApprovalRejected
    condition: ceremony.approval_rejected
    severity: critical
    
  - name: UnauthorizedCeremonyAttempt
    condition: ceremony.unauthorized_access
    severity: critical

Troubleshooting

Common Issues

Issue Cause Resolution
Approval rejected Invalid signature Re-sign with correct key
Cannot approve Already approved Different approver must approve
Cannot execute Threshold not met Collect more approvals
Ceremony expired Timeout reached Create new ceremony

Signature Verification Failures

# Verify signing key is accessible
stella auth keys list

# Test signature
echo "test" | stella sign --key-id my-signing-key | stella verify

# Check key permissions
stella auth keys info --key-id my-signing-key

Emergency Procedures

Stuck Ceremony

If a ceremony is stuck (approvers unavailable):

  1. Cancel the stuck ceremony
  2. Create new ceremony with available approvers
  3. Document the situation in audit notes

Compromised Approver

If an approver's credentials are compromised:

  1. Revoke approver's signing key immediately
  2. Cancel any pending ceremonies they created
  3. Review recent approvals for anomalies
  4. Remove from approver groups
  5. Document in incident report

Related Documentation