git.stella-ops.org/docs/testing/ci-quality-gates.md

CI Quality Gates

Sprint: SPRINT_0350_0001_0001_ci_quality_gates_foundation
Task: QGATE-0350-009

Overview

StellaOps implements automated quality gates in CI to enforce:

• Reachability Quality - Recall/precision thresholds for vulnerability detection
• TTFS Regression - Time-to-First-Signal performance tracking
• Performance SLOs - Scan time and compute budget enforcement

These gates run as part of the build-test-deploy.yml workflow after the main test suite completes.

Quality Gate Jobs

Reachability Quality Gate

Script: scripts/ci/compute-reachability-metrics.sh
Config: scripts/ci/reachability-thresholds.yaml

Validates that the scanner meets recall/precision thresholds against the ground-truth corpus.

Metrics Computed

Metric	Description	Threshold
runtime_dependency_recall	% of runtime dep vulns detected	≥ 95%
unreachable_false_positives	FP rate for unreachable findings	≤ 5%
reachability_underreport	Underreporting rate	≤ 10%
os_package_recall	% of OS package vulns detected	≥ 92%
code_vuln_recall	% of code vulns detected	≥ 88%
config_vuln_recall	% of config vulns detected	≥ 85%

Running Locally

# Dry run (no enforcement)
./scripts/ci/compute-reachability-metrics.sh --dry-run

# Full run against corpus
./scripts/ci/compute-reachability-metrics.sh

TTFS Regression Gate

Script: scripts/ci/compute-ttfs-metrics.sh
Baseline: bench/baselines/ttfs-baseline.json

Detects performance regressions in Time-to-First-Signal.

Metrics Computed

Metric	Description	Threshold
ttfs_p50_ms	P50 time to first signal	≤ baseline + 10%
ttfs_p95_ms	P95 time to first signal	≤ baseline + 15%
ttfs_max_ms	Maximum TTFS	≤ baseline + 25%

Baseline Format

{
  "ttfs_p50_ms": 450,
  "ttfs_p95_ms": 1200,
  "ttfs_max_ms": 3000,
  "measured_at": "2025-12-16T00:00:00Z",
  "sample_count": 1000
}

Performance SLO Gate

Script: scripts/ci/enforce-performance-slos.sh
Config: scripts/ci/performance-slos.yaml

Enforces scan time and compute budget SLOs.

SLOs Enforced

SLO	Description	Target
scan_time_p50_ms	P50 scan time	≤ 120,000ms (2 min)
scan_time_p95_ms	P95 scan time	≤ 300,000ms (5 min)
memory_peak_mb	Peak memory usage	≤ 2048 MB
cpu_seconds	Total CPU time	≤ 120 seconds

Workflow Integration

Quality gates are integrated into the main CI workflow:

# .gitea/workflows/build-test-deploy.yml

quality-gates:
  runs-on: ubuntu-22.04
  needs: build-test
  steps:
    - name: Reachability quality gate
      run: ./scripts/ci/compute-reachability-metrics.sh

    - name: TTFS regression gate
      run: ./scripts/ci/compute-ttfs-metrics.sh

    - name: Performance SLO gate
      run: ./scripts/ci/enforce-performance-slos.sh --warn-only

Failure Modes

Hard Failure (Blocks Merge)

• Reachability recall below threshold
• TTFS regression exceeds 25%
• Memory budget exceeded by 50%

Soft Failure (Warning Only)

• Minor TTFS regression (< 15%)
• Memory near budget limit
• Missing baseline data (new fixtures)

Adding New Quality Gates

1. Create computation script in scripts/ci/
2. Add threshold configuration (YAML or JSON)
3. Integrate into workflow as a new step
4. Update this documentation
5. Add to sprint tracking

Troubleshooting

Gate Fails on PR but Passes on Main

Check for:

• Non-deterministic test execution
• Timing-sensitive assertions
• Missing test fixtures in PR branch

Baseline Drift

If baselines become stale:

# Regenerate baselines
./scripts/ci/compute-ttfs-metrics.sh --update-baseline
./scripts/ci/compute-reachability-metrics.sh --update-baseline

Related Documentation

• Test Suite Overview
• Testing Strategy Models
• Test Catalog
• Reachability Corpus Plan
• Performance Workbook
• Testing Quality Guardrails