stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 2 Releases Wiki Activity
Files
b9f71fc7e9e1fb9b1c366ebd75f92483b2f94b67
git.stella-ops.org/docs/testing/competitor-parity-testing.md
StellaOps Bot 9a08d10b89 docs consolidation
2025-12-24 12:38:14 +02:00

8.0 KiB
Raw Blame History

Competitor Parity Testing Guide

This document describes StellaOps' competitor parity testing methodology, which ensures we maintain feature parity and performance competitiveness with industry-standard scanners.

Overview

Parity testing compares StellaOps against three primary competitors:

Tool Type Primary Function
Syft (Anchore) SBOM Generator Package extraction and SBOM creation
Grype (Anchore) Vulnerability Scanner CVE detection using Syft SBOMs
Trivy (Aqua) Multi-scanner SBOM + vulnerability + secrets + misconfig

Goals

  1. Prevent regression: Detect when StellaOps falls behind competitors on key metrics
  2. Track trends: Monitor parity metrics over time to identify drift patterns
  3. Guide development: Use competitor gaps to prioritize feature work
  4. Validate claims: Ensure marketing claims are backed by measurable data

Fixture Set

Tests run against a standardized set of container images that represent diverse workloads:

Quick Set (Nightly)

  • Alpine 3.19
  • Debian 12
  • Ubuntu 24.04
  • Python 3.12-slim
  • Node.js 22-alpine
  • nginx:1.25 (known vulnerabilities)

Full Set (Weekly)

All quick set images plus:

  • RHEL 9 UBI
  • Go 1.22-alpine
  • Rust 1.79-slim
  • OpenJDK 21-slim
  • .NET 8.0-alpine
  • postgres:16 (known vulnerabilities)
  • wordpress:6.5 (complex application)
  • redis:7-alpine
  • Multi-layer image (test depth)

Metrics Collected

SBOM Metrics

Metric Description Target
Package Count Total packages detected ≥95% of Syft
PURL Completeness Packages with valid PURLs ≥98%
License Detection Packages with license info ≥90%
CPE Mapping Packages with CPE identifiers ≥85%

Vulnerability Metrics

Metric Description Target
Recall CVEs found vs. union of all scanners ≥95%
Precision True positives vs. total findings ≥90%
F1 Score Harmonic mean of precision/recall ≥92%
Severity Distribution Breakdown by Critical/High/Medium/Low Match competitors ±10%

Latency Metrics

Metric Description Target
P50 Median scan time ≤1.5x Grype
P95 95th percentile scan time ≤2x Grype
P99 99th percentile scan time ≤3x Grype
Time-to-First-Signal Time to first vulnerability found ≤Grype

Error Handling Metrics

Scenario Expected Behavior
Malformed manifest Graceful degradation with partial results
Network timeout Clear error message; cached feed fallback
Large image (>5GB) Streaming extraction; no OOM
Corrupt layer Skip layer; report warning
Missing base layer Report incomplete scan
Registry auth failure Clear auth error; suggest remediation
Rate limiting Backoff + retry; clear message

Running Parity Tests

Locally

# Build the parity test project
dotnet build tests/parity/StellaOps.Parity.Tests

# Run quick fixture set
dotnet test tests/parity/StellaOps.Parity.Tests \
  -e PARITY_FIXTURE_SET=quick \
  -e PARITY_OUTPUT_PATH=./parity-results

# Run full fixture set
dotnet test tests/parity/StellaOps.Parity.Tests \
  -e PARITY_FIXTURE_SET=full \
  -e PARITY_OUTPUT_PATH=./parity-results

Prerequisites

Ensure competitor tools are installed and in PATH:

# Install Syft
curl -sSfL https://raw.githubusercontent.com/anchore/syft/main/install.sh | sh -s -- -b /usr/local/bin v1.9.0

# Install Grype
curl -sSfL https://raw.githubusercontent.com/anchore/grype/main/install.sh | sh -s -- -b /usr/local/bin v0.79.3

# Install Trivy
curl -sfL https://raw.githubusercontent.com/aquasecurity/trivy/main/contrib/install.sh | sh -s -- -b /usr/local/bin v0.54.1

CI/CD

Parity tests run automatically:

  • Nightly (02:00 UTC): Quick fixture set
  • Weekly (Sunday 00:00 UTC): Full fixture set

Results are stored as workflow artifacts and optionally pushed to Prometheus.

Drift Detection

The parity system includes automated drift detection that alerts when StellaOps falls behind competitors:

Thresholds

Metric Threshold Trend Period
SBOM Completeness >5% decline 3 days
Vulnerability Recall >5% decline 3 days
Latency vs Grype >10% increase 3 days
PURL Completeness >5% decline 3 days
F1 Score >5% decline 3 days

Alert Severity

Severity Condition Action
Low 1-1.5x threshold Monitor
Medium 1.5-2x threshold Investigate within sprint
High 2-3x threshold Prioritize fix
Critical >3x threshold Immediate action required

Analyzing Drift

# Run drift analysis on stored results
dotnet run --project tests/parity/StellaOps.Parity.Tests \
  -- analyze-drift \
  --results-path ./parity-results \
  --threshold 0.05 \
  --trend-days 3

Result Storage

JSON Format

Results are stored as timestamped JSON files:

parity-results/
├── parity-20250115T020000Z-abc123.json
├── parity-20250116T020000Z-def456.json
└── parity-20250122T000000Z-ghi789.json  # Weekly

Retention Policy

  • Last 90 days: Full detail retained
  • Older than 90 days: Aggregated to weekly summaries
  • Artifacts: Workflow artifacts retained for 90 days

Prometheus Export

Results can be exported to Prometheus for dashboarding:

stellaops_parity_sbom_completeness_ratio{run_id="..."} 0.97
stellaops_parity_vuln_recall{run_id="..."} 0.95
stellaops_parity_latency_p95_ms{scanner="stellaops",run_id="..."} 1250

InfluxDB Export

For InfluxDB time-series storage:

parity_sbom,run_id=abc123 completeness_ratio=0.97,matched_count=142i 1705280400000000000
parity_vuln,run_id=abc123 recall=0.95,precision=0.92,f1=0.935 1705280400000000000

Competitor Version Tracking

Competitor tools are pinned to specific versions to ensure reproducibility:

Tool Current Version Last Updated
Syft 1.9.0 2025-01-15
Grype 0.79.3 2025-01-15
Trivy 0.54.1 2025-01-15

Updating Versions

  1. Update version in .gitea/workflows/parity-tests.yml
  2. Update version in tests/parity/StellaOps.Parity.Tests/ParityHarness.cs
  3. Run full parity test to establish new baseline
  4. Document version change in sprint execution log

Troubleshooting

Common Issues

Syft not found

Error: Syft executable not found in PATH

Solution: Install Syft or set SYFT_PATH environment variable.

Grype DB outdated

Warning: Grype vulnerability database is 7+ days old

Solution: Run grype db update to refresh the database.

Image pull rate limit

Error: docker pull rate limit exceeded

Solution: Use authenticated Docker Hub credentials or local registry mirror.

Test timeout

Error: Test exceeded 30 minute timeout

Solution: Increase timeout or use quick fixture set.

Debug Mode

Enable verbose logging:

dotnet test tests/parity/StellaOps.Parity.Tests \
  -e PARITY_DEBUG=true \
  -e PARITY_KEEP_OUTPUTS=true

Architecture

tests/parity/StellaOps.Parity.Tests/
├── StellaOps.Parity.Tests.csproj    # Project file
├── ParityTestFixtureSet.cs           # Container image fixtures
├── ParityHarness.cs                  # Scanner execution harness
├── SbomComparisonLogic.cs            # SBOM comparison
├── VulnerabilityComparisonLogic.cs   # Vulnerability comparison
├── LatencyComparisonLogic.cs         # Latency comparison
├── ErrorModeComparisonLogic.cs       # Error handling comparison
└── Storage/
    ├── ParityResultStore.cs          # Time-series storage
    └── ParityDriftDetector.cs        # Drift detection

See Also