stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 2 Releases Wiki Activity
Files
9b58589ba01342d5df7ab3bbfd7370cc970cdd21
git.stella-ops.org/docs/features/dropped/multi-scanner-comparative-benchmarking.md

1.3 KiB
Raw Blame History

Multi-scanner Comparative Benchmarking

Status

NOT_FOUND

Description

Advisory describes a benchmarking protocol comparing StellaOps scan results against Trivy/Grype/Snyk with precision/recall metrics. No CLI comparison tool or benchmark harness found.

Why Not Implemented

  • No dedicated CLI comparison tool or multi-scanner benchmark harness found
  • A compare.py script exists at src/__Tests/__Benchmarks/tools/compare.py but it appears to be a general comparison utility, not a full multi-scanner benchmarking protocol
  • The Bench module (src/Bench/) has benchmarking infrastructure (LinkNotMerge scenarios, Prometheus reporting) but not scanner comparison harnesses
  • Golden corpus exists at src/__Tests/__Benchmarks/golden-corpus/ with VEX scenarios and severity levels, which could serve as ground truth for scanner comparison
  • The Scanner module has its own benchmark and test infrastructure but does not compare against external scanners (Trivy/Grype/Snyk)
  • This would require external scanner integration which conflicts with the offline-first posture

Source

  • Feature matrix scan

Notes

  • Module: Bench
  • Modules referenced: N/A
  • Related: src/__Tests/__Benchmarks/tools/compare.py (comparison utility), src/__Tests/__Benchmarks/golden-corpus/ (ground truth data)