git.stella-ops.org/docs/modules/scanner/design/competitor-benchmark-parity.md

Competitor Benchmark Parity Plan (CM7, CM8, CM9)

Status: Draft · Date: 2025-12-04 Scope: Define source transparency fields (CM7), benchmark parity requirements (CM8), and ecosystem coverage tracking (CM9).

CM7: Source Transparency Fields

Required Metadata Fields

Field	Source	Storage Location	API Exposure
source.tool	Ingest input	ingest_metadata.tool	Yes
source.version	Ingest input	ingest_metadata.tool_version	Yes
source.hash	Computed	ingest_metadata.tool_hash	Yes
adapter.version	Adapter manifest	ingest_metadata.adapter_version	Yes
normalized_hash	Computed	ingest_metadata.normalized_hash	Yes
import_timestamp	System	ingest_metadata.imported_at	Yes

Metadata Schema

{
  "ingest_metadata": {
    "source": {
      "tool": "syft",
      "version": "1.0.0",
      "hash": "sha256:...",
      "uri": "https://github.com/anchore/syft/releases/v1.0.0"
    },
    "adapter": {
      "version": "1.0.0",
      "mappingHash": "b3:..."
    },
    "normalized": {
      "hash": "b3:aa42c167...",
      "recordCount": 42,
      "format": "stellaops-v1"
    },
    "import": {
      "timestamp": "2025-12-04T12:00:00Z",
      "user": "system",
      "snapshotId": "syft-20251204T120000Z-001"
    }
  }
}

API Exposure

GET /api/v1/ingest/metadata/{snapshotId}

Response:

{
  "metadata": {
    "snapshotId": "syft-20251204T120000Z-001",
    "source": {
      "tool": "syft",
      "version": "1.0.0",
      "hash": "sha256:..."
    },
    "adapter": {
      "version": "1.0.0"
    },
    "normalized": {
      "hash": "b3:...",
      "recordCount": 42
    }
  }
}

CM8: Benchmark Parity

Pinned Tool Versions

Tool	Pinned Version	Test Frequency	Baseline Date
Syft	1.0.0	Weekly	2025-12-01
Trivy	0.50.0	Weekly	2025-12-01
Clair	6.0.0	Weekly	2025-12-01

Benchmark Test Suite

tests/benchmark/
├── syft/
│   ├── inputs/
│   │   ├── alpine-3.19.json       # Container image
│   │   ├── node-app.json          # Node.js project
│   │   └── java-app.json          # Java project
│   ├── expected/
│   │   ├── alpine-3.19-expected.json
│   │   ├── node-app-expected.json
│   │   └── java-app-expected.json
│   └── hashes.txt
├── trivy/
│   └── ... (same structure)
└── clair/
    └── ... (same structure)

Benchmark Workflow

# .gitea/workflows/benchmark-parity.yml
name: Benchmark Parity Check

on:
  schedule:
    - cron: '0 0 * * 0'  # Weekly
  workflow_dispatch:

jobs:
  benchmark:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        tool: [syft, trivy, clair]
    steps:
      - uses: actions/checkout@v4

      - name: Run ${{ matrix.tool }} benchmark
        run: |
          scripts/benchmark/run-benchmark.sh ${{ matrix.tool }}

      - name: Compare results
        run: |
          scripts/benchmark/compare-results.sh ${{ matrix.tool }}

      - name: Upload logs
        uses: actions/upload-artifact@v4
        with:
          name: benchmark-${{ matrix.tool }}
          path: benchmark-results/

Benchmark Comparison

#!/bin/bash
# scripts/benchmark/compare-results.sh

TOOL=$1
BENCHMARK_DIR="tests/benchmark/${TOOL}"

for input in "${BENCHMARK_DIR}/inputs/"*.json; do
    name=$(basename "${input}" .json)
    expected="${BENCHMARK_DIR}/expected/${name}-expected.json"
    actual="benchmark-results/${name}-actual.json"

    # Run tool
    stellaops ingest normalize \
        --tool "${TOOL}" \
        --input "${input}" \
        --output "${actual}"

    # Compare
    diff_result=$(diff <(jq -S . "${expected}") <(jq -S . "${actual}"))

    if [[ -n "${diff_result}" ]]; then
        echo "DRIFT: ${name}"
        echo "${diff_result}"
        exit 1
    fi

    echo "PASS: ${name}"
done

Drift Detection

When benchmark drift detected:

1. Log drift details with hash comparison
2. Create issue in tracking system
3. Notify Scanner Guild
4. Block release if critical drift

{
  "drift": {
    "tool": "syft",
    "version": "1.0.0",
    "testCase": "alpine-3.19",
    "detected": "2025-12-04T00:00:00Z",
    "details": {
      "expectedHash": "b3:expected...",
      "actualHash": "b3:actual...",
      "diffCount": 3,
      "fields": [
        "/components/5/version",
        "/components/12/licenses",
        "/vulnerabilities/2/ratings/0/score"
      ]
    }
  }
}

CM9: Coverage Tracking

Coverage Matrix

Location: docs/modules/scanner/fixtures/competitor-adapters/coverage.csv

ecosystem,syft,trivy,clair,notes
container,yes,yes,yes,All tools support OCI images
java,yes,yes,no,Clair Java support pending
python,yes,yes,no,Trivy has best pip/poetry coverage
dotnet,no,yes,no,Trivy only; Syft support pending
go,yes,yes,no,Both tools have good go.mod support
rust,yes,yes,no,Cargo.lock parsing
ruby,yes,yes,no,Gemfile.lock parsing
php,yes,yes,no,composer.lock parsing
os-pkgs,yes,yes,yes,APK/DEB/RPM supported
node,yes,yes,no,package-lock.json/yarn.lock

Coverage API

GET /api/v1/ingest/coverage

Response:

{
  "coverage": {
    "lastUpdated": "2025-12-04T00:00:00Z",
    "ecosystems": {
      "container": {
        "syft": {"supported": true, "tested": true},
        "trivy": {"supported": true, "tested": true},
        "clair": {"supported": true, "tested": true}
      },
      "java": {
        "syft": {"supported": true, "tested": true},
        "trivy": {"supported": true, "tested": true},
        "clair": {"supported": false, "tested": false, "planned": "2026-Q1"}
      }
    },
    "gaps": [
      {"ecosystem": "dotnet", "tool": "syft", "priority": "high"},
      {"ecosystem": "java", "tool": "clair", "priority": "medium"}
    ]
  }
}

Gap Tracking

{
  "gaps": [
    {
      "id": "gap-001",
      "ecosystem": "dotnet",
      "tool": "syft",
      "priority": "high",
      "reason": "Customer demand for .NET scanning",
      "status": "planned",
      "targetDate": "2025-Q2",
      "blockers": ["Upstream syft issue #1234"]
    }
  ]
}

Coverage CI Check

# Check coverage doesn't regress
- name: Verify coverage matrix
  run: |
    # Ensure no "yes" changed to "no" without documentation
    git diff HEAD~1 docs/modules/scanner/fixtures/competitor-adapters/coverage.csv \
      | grep -E '^\-.*yes.*$' && {
        echo "Coverage regression detected"
        exit 1
      }

Reporting

Weekly Coverage Report

{
  "report": {
    "period": "2025-W49",
    "coverage": {
      "total_ecosystems": 10,
      "full_coverage": 3,
      "partial_coverage": 5,
      "no_coverage": 2
    },
    "benchmark": {
      "tests_run": 45,
      "tests_passed": 44,
      "tests_failed": 1,
      "drift_detected": ["trivy/alpine-3.19"]
    },
    "metadata": {
      "snapshots_imported": 156,
      "tools_seen": ["syft", "trivy", "clair"],
      "versions_seen": {
        "syft": ["1.0.0", "1.0.1"],
        "trivy": ["0.50.0"],
        "clair": ["6.0.0"]
      }
    }
  }
}

Links

• Sprint: docs/implplan/SPRINT_0186_0001_0001_record_deterministic_execution.md (CM7, CM8, CM9)
• Normalization: docs/modules/scanner/design/competitor-ingest-normalization.md (CM1)
• Coverage CSV: docs/modules/scanner/fixtures/competitor-adapters/coverage.csv