stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 2 Releases Wiki Activity
Files
702a27ac839f1de50f9c9f460000b8f7b2242efd
git.stella-ops.org/docs/operations/runbooks/orchestrator-evidence-missing.md

4.1 KiB
Raw Blame History

Runbook: Release Orchestrator - Required Evidence Not Found

Sprint: SPRINT_20260117_029_DOCS_runbook_coverage Task: RUN-004 - Release Orchestrator Runbooks

Metadata

Field Value
Component Release Orchestrator
Severity High
On-call scope Platform team, Security team
Last updated 2026-01-17
Doctor check check.orchestrator.evidence-availability

Symptoms

  • Promotion failing with "required evidence not found"
  • Alert OrchestratorEvidenceMissing firing
  • Gate evaluation blocked waiting for evidence
  • Error: "SBOM not found" or "attestation missing"
  • Evidence chain incomplete for artifact

Impact

Impact Type Description
User-facing Promotion blocked until evidence is generated
Data integrity Indicates missing security artifact - must be resolved
SLA impact Release blocked; compliance requirements not met

Diagnosis

Quick checks

  1. Check Doctor diagnostics:

    stella doctor --check check.orchestrator.evidence-availability

  2. List missing evidence for promotion:

    stella promotion evidence <promotion-id> --missing

  3. Check what evidence exists for artifact:

    stella evidence list --artifact <digest>

Deep diagnosis

  1. Check evidence chain completeness:

    stella evidence chain --artifact <digest> --verbose

    Look for: Missing nodes in the chain

  2. Check if scan completed:

    stella scanner jobs list --artifact <digest>

    Problem if: No completed scan or scan failed

  3. Check if attestation was created:

    stella attest list --subject <digest>

    Problem if: No attestation or attestation failed

  4. Check evidence store health:

    stella evidence store health

Resolution

Immediate mitigation

  1. Generate missing SBOM:

    stella scan image --image <image-ref> --sbom-only

  2. Generate missing attestation:

    stella attest create --subject <digest> --type slsa-provenance

  3. Re-scan artifact to regenerate all evidence:

    stella scan image --image <image-ref> --force

Root cause fix

If scan never ran:

  1. Check why artifact wasn't scanned:

    stella scanner queue list --artifact <digest>

  2. Configure automatic scanning on push:

    stella scanner config set auto_scan.enabled true
stella scanner config set auto_scan.triggers "push,promote"

If evidence was generated but not stored:

  1. Check evidence store connectivity:

    stella evidence store health

  2. Retry evidence storage:

    stella evidence retry-store --artifact <digest>

If attestation signing failed:

  1. Check attestor status:

    stella attest status

  2. See attestor-signing-failed.md runbook

If evidence expired or was deleted:

  1. Check evidence retention policy:

    stella evidence policy show

  2. Regenerate evidence:

    stella scan image --image <image-ref> --force
stella attest create --subject <digest> --type slsa-provenance

Verification

# Check all evidence now exists
stella evidence list --artifact <digest>

# Verify evidence chain is complete
stella evidence chain --artifact <digest>

# Retry promotion
stella promotion retry <promotion-id>

# Verify promotion proceeds
stella promotion status <promotion-id>

Prevention

  • Auto-scan: Enable automatic scanning for all pushed images
  • Gates: Configure evidence requirements clearly in promotion policy
  • Monitoring: Alert on evidence generation failures
  • Retention: Set appropriate evidence retention periods

Related Resources

  • Architecture: docs/modules/evidence-locker/architecture.md
  • Related runbooks: orchestrator-promotion-stuck.md, attestor-signing-failed.md
  • Evidence requirements: docs/operations/evidence-requirements.md