# Dual-Control Ceremony Runbook This runbook documents M-of-N threshold signing ceremonies for high-assurance key operations in Stella Ops. > **Sprint:** SPRINT_20260112_018_SIGNER_dual_control_ceremonies ## Overview Dual-control ceremonies ensure critical cryptographic operations require approval from multiple authorized individuals before execution. This prevents single points of compromise for sensitive operations like: - Root key rotation - Trust anchor updates - Emergency key revocation - HSM key generation - Recovery key activation ## When Ceremonies Are Required | Operation | Default Threshold | Configurable | |-----------|------------------|--------------| | Root signing key rotation | 2-of-3 | Yes | | Trust anchor update | 2-of-3 | Yes | | Key revocation | 2-of-3 | Yes | | HSM key generation | 2-of-4 | Yes | | Recovery key activation | 3-of-5 | Yes | ## Ceremony Lifecycle ### State Machine ``` +------------------+ | Pending | +--------+---------+ | | Approvals collected v +-------------+-------------+ | PartiallyApproved | +-------------+-------------+ | | Threshold reached v +--------+---------+ | Approved | +--------+---------+ | | Execute v +--------+---------+ | Executed | +------------------+ Alternative paths: - Pending -> Expired (timeout) - Pending -> Cancelled (initiator cancel) - PartiallyApproved -> Expired (timeout) - PartiallyApproved -> Cancelled ``` ### State Descriptions | State | Description | |-------|-------------| | `Pending` | Ceremony created, awaiting first approval | | `PartiallyApproved` | At least one approval, threshold not reached | | `Approved` | Threshold reached, ready for execution | | `Executed` | Operation completed successfully | | `Expired` | Timeout reached without execution | | `Cancelled` | Explicitly cancelled before execution | ## Creating a Ceremony ### Via CLI ```bash stella ceremony create \ --type key-rotation \ --subject "Root signing key Q1-2026" \ --threshold 2 \ --required-approvers 3 \ --expires-in 24h \ --payload '{"keyId": "root-2026-q1", "algorithm": "ecdsa-p384"}' ``` ### Via API ```bash curl -X POST https://signer.example.com/api/v1/ceremonies \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{ "type": "key-rotation", "subject": "Root signing key Q1-2026", "threshold": 2, "requiredApprovers": 3, "expiresAt": "2026-01-17T10:00:00Z", "payload": { "keyId": "root-2026-q1", "algorithm": "ecdsa-p384" } }' ``` ### Response ```json { "ceremonyId": "cer-abc123", "type": "key-rotation", "state": "Pending", "threshold": 2, "requiredApprovers": 3, "currentApprovals": 0, "createdAt": "2026-01-16T10:00:00Z", "expiresAt": "2026-01-17T10:00:00Z", "initiator": "admin@company.com" } ``` ## Approving a Ceremony ### Prerequisites Approvers must: 1. Be in the ceremony's allowed approvers list 2. Have the `ceremony:approve` scope 3. Have valid authentication (OIDC or break-glass) 4. Not have already approved this ceremony ### Via CLI ```bash stella ceremony approve \ --ceremony-id cer-abc123 \ --reason "Reviewed rotation plan, verified key parameters" \ --sign ``` The `--sign` flag creates a DSSE signature over the approval using the approver's signing key. ### Via API ```bash curl -X POST https://signer.example.com/api/v1/ceremonies/cer-abc123/approve \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{ "reason": "Reviewed rotation plan, verified key parameters", "signature": "base64-encoded-dsse-signature" }' ``` ### Approval Response ```json { "ceremonyId": "cer-abc123", "state": "PartiallyApproved", "currentApprovals": 1, "threshold": 2, "approval": { "approvalId": "apr-def456", "approver": "security-lead@company.com", "approvedAt": "2026-01-16T11:30:00Z", "reason": "Reviewed rotation plan, verified key parameters", "signatureValid": true } } ``` ## Executing a Ceremony Once the approval threshold is reached: ### Via CLI ```bash stella ceremony execute --ceremony-id cer-abc123 ``` ### Via API ```bash curl -X POST https://signer.example.com/api/v1/ceremonies/cer-abc123/execute \ -H "Authorization: Bearer $TOKEN" ``` ### Execution Response ```json { "ceremonyId": "cer-abc123", "state": "Executed", "executedAt": "2026-01-16T14:00:00Z", "result": { "keyId": "root-2026-q1", "publicKey": "-----BEGIN PUBLIC KEY-----...", "fingerprint": "SHA256:abc123...", "activatedAt": "2026-01-16T14:00:00Z" } } ``` ## Monitoring Ceremonies ### List Active Ceremonies ```bash # CLI stella ceremony list --state pending,partially-approved # API curl "https://signer.example.com/api/v1/ceremonies?state=pending,partially-approved" ``` ### Check Ceremony Status ```bash # CLI stella ceremony status --ceremony-id cer-abc123 # API curl "https://signer.example.com/api/v1/ceremonies/cer-abc123" ``` ## Cancelling a Ceremony Ceremonies can be cancelled before execution: ```bash # CLI stella ceremony cancel \ --ceremony-id cer-abc123 \ --reason "Postponed due to schedule conflict" # API curl -X DELETE https://signer.example.com/api/v1/ceremonies/cer-abc123 \ -H "Authorization: Bearer $TOKEN" ``` Only the initiator or users with `ceremony:cancel` scope can cancel. ## Audit Events All ceremony actions are logged: | Event | Description | |-------|-------------| | `signer.ceremony.initiated` | Ceremony created | | `signer.ceremony.approved` | Approval submitted | | `signer.ceremony.approval_rejected` | Approval rejected (invalid signature, unauthorized) | | `signer.ceremony.executed` | Operation executed | | `signer.ceremony.expired` | Timeout reached | | `signer.ceremony.cancelled` | Explicitly cancelled | ### Audit Event Structure ```json { "eventType": "signer.ceremony.approved", "timestamp": "2026-01-16T11:30:00Z", "ceremonyId": "cer-abc123", "ceremonyType": "key-rotation", "actor": "security-lead@company.com", "approvalId": "apr-def456", "currentApprovals": 1, "threshold": 2, "signatureAlgorithm": "ecdsa-p256", "signatureKeyId": "user-signing-key-456" } ``` ### Query Audit Logs ```bash stella audit query \ --event-type "signer.ceremony.*" \ --since 7d \ --ceremony-id cer-abc123 ``` ## Configuration ### Ceremony Settings ```yaml # signer-config.yaml ceremonies: enabled: true defaultTimeout: 24h maxTimeout: 168h # 7 days requireSignedApprovals: true thresholds: key-rotation: minimum: 2 default: 2 maximum: 5 key-revocation: minimum: 2 default: 3 maximum: 5 trust-anchor-update: minimum: 2 default: 2 maximum: 4 ``` ### Approver Configuration ```yaml # approvers.yaml approverGroups: - name: key-custodians members: - security-lead@company.com - ciso@company.com - key-officer-1@company.com - key-officer-2@company.com operations: - key-rotation - key-revocation - name: trust-admins members: - trust-admin@company.com - security-lead@company.com operations: - trust-anchor-update ``` ## Notifications Ceremonies trigger notifications to approvers: | Event | Notification | |-------|-------------| | Ceremony created | Email/Slack to all eligible approvers | | Approval submitted | Email/Slack to remaining approvers | | Threshold reached | Email/Slack to initiator | | Approaching expiry | Email/Slack at 75% and 90% of timeout | | Expired | Email/Slack to initiator and approvers | Configure notifications in `notifier-config.yaml`: ```yaml notifications: ceremonies: enabled: true channels: - type: email recipients: "@approverGroup" - type: slack webhook: ${SLACK_CEREMONY_WEBHOOK} channel: "#key-ceremonies" ``` ## Security Best Practices ### Approver Requirements - Maintain at least N+1 approvers for N-of-M ceremonies - Distribute approvers across security domains - Require hardware tokens for signing keys - Rotate approver list quarterly ### Ceremony Hygiene - Use descriptive subjects for audit clarity - Set reasonable timeouts (not too long, not too short) - Document approval reasons thoroughly - Review executed ceremonies monthly ### Monitoring Set up alerts for: ```yaml alerts: - name: CeremonyPendingTooLong condition: ceremony.pending_duration > 12h severity: warning - name: CeremonyApprovalRejected condition: ceremony.approval_rejected severity: critical - name: UnauthorizedCeremonyAttempt condition: ceremony.unauthorized_access severity: critical ``` ## Troubleshooting ### Common Issues | Issue | Cause | Resolution | |-------|-------|------------| | Approval rejected | Invalid signature | Re-sign with correct key | | Cannot approve | Already approved | Different approver must approve | | Cannot execute | Threshold not met | Collect more approvals | | Ceremony expired | Timeout reached | Create new ceremony | ### Signature Verification Failures ```bash # Verify signing key is accessible stella auth keys list # Test signature echo "test" | stella sign --key-id my-signing-key | stella verify # Check key permissions stella auth keys info --key-id my-signing-key ``` ## Emergency Procedures ### Stuck Ceremony If a ceremony is stuck (approvers unavailable): 1. Cancel the stuck ceremony 2. Create new ceremony with available approvers 3. Document the situation in audit notes ### Compromised Approver If an approver's credentials are compromised: 1. Revoke approver's signing key immediately 2. Cancel any pending ceremonies they created 3. Review recent approvals for anomalies 4. Remove from approver groups 5. Document in incident report ## Related Documentation - [Key Rotation Runbook](./key-rotation-runbook.md) - [HSM Setup Runbook](./hsm-setup-runbook.md) - [Signer Architecture](../modules/signer/architecture.md) - [Break-Glass Runbook](./break-glass-runbook.md)