# Launch Cutover Runbook - Stella Ops

_Document owner: DevOps Guild (2025-10-26)_  
_Scope:_ Full-platform launch from staging to production for release `2025.09.2`.

## 1. Roles and Communication

| Role | Primary | Backup | Contact |
| --- | --- | --- | --- |
| Cutover lead | DevOps Guild (on-call engineer) | Platform Ops lead | `#launch-bridge` (Mattermost) |
| Authority stack | Authority Core guild rep | Security guild rep | `#authority` |
| Scanner / Queue | Scanner WebService guild rep | Runtime guild rep | `#scanner` |
| Storage | Mongo/MinIO operators | Backup DB admin | Pager escalation |
| Observability | Telemetry guild rep | SRE on-call | `#telemetry` |
| Approvals | Product owner + CTO | DevOps lead | Approval recorded in change ticket |

Set up a bridge call 30 minutes before start and keep `#launch-bridge` updated every 10 minutes.

## 2. Timeline Overview (UTC)

| Time | Activity | Owner |
| --- | --- | --- |
| T-24h | Change ticket approved, prod secrets verified, offline kit build status checked (`DEVOPS-OFFLINE-18-005`). | DevOps lead |
| T-12h | Run `deploy/tools/validate-profiles.sh`; capture logs in ticket. | DevOps engineer |
| T-6h | Freeze non-launch deployments; notify guild leads. | Product owner |
| T-2h | Execute rehearsal in staging (Section 3) using `values-stage.yaml` to verify scripts. | DevOps + module reps |
| T-30m | Final go/no-go with guild leads; confirm monitoring dashboards green. | Cutover lead |
| T0 | Execute production cutover steps (Section 4). | Cutover team |
| T+45m | Smoke tests complete (Section 5); announce success or trigger rollback. | Cutover lead |
| T+4h | Post-cutover metrics review, notify stakeholders, close ticket. | DevOps + product owner |

## 3. Rehearsal (Staging) Checklist

1. `docker network create stellaops_frontdoor || true` (if not present on staging jump host).
2. Run `deploy/tools/validate-profiles.sh` and archive output.
3. Apply staging secrets (`kubectl apply -f secrets/stage/*.yaml` or `helm secrets upgrade`) ensuring `stellaops-stage` credentials align with `values-stage.yaml`.
4. Perform `helm upgrade stellaops deploy/helm/stellaops -f deploy/helm/stellaops/values-stage.yaml` in staging cluster.
5. Verify health endpoints: `curl https://authority.stage.../healthz`, `curl https://scanner.stage.../healthz`.
6. Execute smoke CLI: `stellaops-cli scan submit --profile staging --sbom samples/sbom/demo.json` and confirm report status in UI.
7. Document total wall time and any deviations in the rehearsal log.

Rehearsal must complete without manual interventions before proceeding to production.

## 4. Production Cutover Steps

### 4.1 Pre-flight
- Confirm production secrets in the appropriate secret store (`stellaops-prod-core`, `stellaops-prod-mongo`, `stellaops-prod-minio`, `stellaops-prod-notify`) contain the keys referenced in `values-prod.yaml`.
- Ensure the external reverse proxy network exists: `docker network create stellaops_frontdoor || true` on each compose host.
- Back up current configuration and data:
  - Mongo snapshot: `mongodump --uri "$MONGO_BACKUP_URI" --out /backups/launch-$(date -Iseconds)`.
  - MinIO policy export: `mc mirror --overwrite minio/stellaops minio-backup/stellaops-$(date +%Y%m%d%H%M)`.

### 4.2 Apply Updates (Compose)
1. On each compose node, pull updated images for release `2025.09.2`:
   ```bash
   docker compose --env-file prod.env -f deploy/compose/docker-compose.prod.yaml pull
   ```
2. Deploy changes:
   ```bash
   docker compose --env-file prod.env -f deploy/compose/docker-compose.prod.yaml up -d
   ```
3. Confirm containers healthy via `docker compose ps` and `docker logs <service> --tail 50`.

### 4.3 Apply Updates (Helm/Kubernetes)
If using Kubernetes, perform:
```bash
helm upgrade stellaops deploy/helm/stellaops -f deploy/helm/stellaops/values-prod.yaml --atomic --timeout 15m
```
Monitor rollout with `kubectl get pods -n stellaops --watch` and `kubectl rollout status deployment/<service>`.

### 4.4 Configuration Validation
- Verify Authority issuer metadata: `curl https://authority.prod.../.well-known/openid-configuration`.
- Validate Signer DSSE endpoint: `stellaops-cli signer verify --base-url https://signer.prod... --bundle samples/dsse/demo.json`.
- Check Scanner queue connectivity: `docker exec stellaops-scanner-web dotnet StellaOps.Scanner.WebService.dll health queue` (returns success).
- Ensure Notify (legacy) still accessible while Notifier migration pending.

## 5. Smoke Tests

| Test | Command / Action | Expected Result |
| --- | --- | --- |
| API health | `curl https://scanner.prod.../healthz` | HTTP 200 with `status":"Healthy"` |
| Scan submit | `stellaops-cli scan submit --profile prod --sbom samples/sbom/demo.json` | Scan completes < 5 minutes; report accessible with signed DSSE |
| Runtime event ingest | Post sample event from Zastava observer fixture | `/runtime/events` responds 202 Accepted; record visible in Mongo `runtime_events` |
| Signing | `stellaops-cli signer sign --bundle demo.json` | Returns DSSE with matching SHA256 and signer metadata |
| Attestor verify | `stellaops-cli attestor verify --uuid <uuid>` | Verification result `ok=true` |
| Web UI | Manual login, verify dashboards render and latency within budget | UI loads under 2 seconds; policy views consistent |

Log results in the change ticket with timestamps and screenshots where applicable.

## 6. Rollback Procedure

1. Assess failure scope; if systemic, initiate rollback immediately while preserving logs/artifacts.
2. For Compose:
   ```bash
   docker compose --env-file prod.env -f deploy/compose/docker-compose.prod.yaml down
   docker compose --env-file stage.env -f deploy/compose/docker-compose.stage.yaml up -d
   ```
3. For Helm:
   ```bash
   helm rollback stellaops <previous-release-number> --namespace stellaops
   ```
4. Restore Mongo snapshot if data inconsistency detected: `mongorestore --uri "$MONGO_BACKUP_URI" --drop /backups/launch-<timestamp>`.
5. Restore MinIO mirror if required: `mc mirror minio-backup/stellaops-<timestamp> minio/stellaops`.
6. Notify stakeholders of rollback and capture root cause notes in incident ticket.

## 7. Post-cutover Actions

- Keep heightened monitoring for 4 hours post cutover; track latency, error rates, and queue depth.
- Confirm audit trails: Authority tokens issued, Scanner events recorded, Attestor submissions stored.
- Update `docs/ops/launch-readiness.md` if any new gaps or follow-ups discovered.
- Schedule retrospective within 48 hours; include DevOps, module guilds, and product owner.

## 8. Approval Matrix

| Step | Required Approvers | Record Location |
| --- | --- | --- |
| Production deployment plan | CTO + DevOps lead | Change ticket comment |
| Cutover start (T0) | DevOps lead + module reps | `#launch-bridge` summary |
| Post-smoke success | DevOps lead + product owner | Change ticket closure |
| Rollback (if invoked) | DevOps lead + CTO | Incident ticket |

Retain all approvals and logs for audit. Update this runbook after each execution to record actual timings and lessons learned.

## 9. Rehearsal Log

| Date (UTC) | What We Exercised | Outcome | Follow-up |
| --- | --- | --- | --- |
| 2025-10-26 | Dry-run of compose/Helm validation via `deploy/tools/validate-profiles.sh` (dev/stage/prod/airgap/mirror). Network creation simulated (`docker network create stellaops_frontdoor` planned) and stage CLI submission reviewed. | Validation script succeeded; all profiles templated cleanly. Stage deployment apply deferred because no staging cluster is accessible from the current environment. | Schedule full stage rehearsal once staging cluster credentials are available; reuse this log section to capture timings. |