Add signal contracts for reachability, exploitability, trust, and unknown symbols
- Introduced `ReachabilityState`, `RuntimeHit`, `ExploitabilitySignal`, `ReachabilitySignal`, `SignalEnvelope`, `SignalType`, `TrustSignal`, and `UnknownSymbolSignal` records to define various signal types and their properties. - Implemented JSON serialization attributes for proper data interchange. - Created project files for the new signal contracts library and corresponding test projects. - Added deterministic test fixtures for micro-interaction testing. - Included cryptographic keys for secure operations with cosign.
This commit is contained in:
StellaOps Bot
324
docs/modules/scanner/design/competitor-benchmark-parity.md
Normal file
324
docs/modules/scanner/design/competitor-benchmark-parity.md
Normal file
@@ -0,0 +1,324 @@
|
||||
# Competitor Benchmark Parity Plan (CM7, CM8, CM9)
|
||||
|
||||
Status: Draft · Date: 2025-12-04
|
||||
Scope: Define source transparency fields (CM7), benchmark parity requirements (CM8), and ecosystem coverage tracking (CM9).
|
||||
|
||||
## CM7: Source Transparency Fields
|
||||
|
||||
### Required Metadata Fields
|
||||
|
||||
| Field | Source | Storage Location | API Exposure |
|
||||
|-------|--------|------------------|--------------|
|
||||
| `source.tool` | Ingest input | `ingest_metadata.tool` | Yes |
|
||||
| `source.version` | Ingest input | `ingest_metadata.tool_version` | Yes |
|
||||
| `source.hash` | Computed | `ingest_metadata.tool_hash` | Yes |
|
||||
| `adapter.version` | Adapter manifest | `ingest_metadata.adapter_version` | Yes |
|
||||
| `normalized_hash` | Computed | `ingest_metadata.normalized_hash` | Yes |
|
||||
| `import_timestamp` | System | `ingest_metadata.imported_at` | Yes |
|
||||
|
||||
### Metadata Schema
|
||||
|
||||
```json
|
||||
{
|
||||
"ingest_metadata": {
|
||||
"source": {
|
||||
"tool": "syft",
|
||||
"version": "1.0.0",
|
||||
"hash": "sha256:...",
|
||||
"uri": "https://github.com/anchore/syft/releases/v1.0.0"
|
||||
},
|
||||
"adapter": {
|
||||
"version": "1.0.0",
|
||||
"mappingHash": "b3:..."
|
||||
},
|
||||
"normalized": {
|
||||
"hash": "b3:aa42c167...",
|
||||
"recordCount": 42,
|
||||
"format": "stellaops-v1"
|
||||
},
|
||||
"import": {
|
||||
"timestamp": "2025-12-04T12:00:00Z",
|
||||
"user": "system",
|
||||
"snapshotId": "syft-20251204T120000Z-001"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### API Exposure
|
||||
|
||||
```http
|
||||
GET /api/v1/ingest/metadata/{snapshotId}
|
||||
```
|
||||
|
||||
Response:
|
||||
```json
|
||||
{
|
||||
"metadata": {
|
||||
"snapshotId": "syft-20251204T120000Z-001",
|
||||
"source": {
|
||||
"tool": "syft",
|
||||
"version": "1.0.0",
|
||||
"hash": "sha256:..."
|
||||
},
|
||||
"adapter": {
|
||||
"version": "1.0.0"
|
||||
},
|
||||
"normalized": {
|
||||
"hash": "b3:...",
|
||||
"recordCount": 42
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## CM8: Benchmark Parity
|
||||
|
||||
### Pinned Tool Versions
|
||||
|
||||
| Tool | Pinned Version | Test Frequency | Baseline Date |
|
||||
|------|----------------|----------------|---------------|
|
||||
| Syft | 1.0.0 | Weekly | 2025-12-01 |
|
||||
| Trivy | 0.50.0 | Weekly | 2025-12-01 |
|
||||
| Clair | 6.0.0 | Weekly | 2025-12-01 |
|
||||
|
||||
### Benchmark Test Suite
|
||||
|
||||
```
|
||||
tests/benchmark/
|
||||
├── syft/
|
||||
│ ├── inputs/
|
||||
│ │ ├── alpine-3.19.json # Container image
|
||||
│ │ ├── node-app.json # Node.js project
|
||||
│ │ └── java-app.json # Java project
|
||||
│ ├── expected/
|
||||
│ │ ├── alpine-3.19-expected.json
|
||||
│ │ ├── node-app-expected.json
|
||||
│ │ └── java-app-expected.json
|
||||
│ └── hashes.txt
|
||||
├── trivy/
|
||||
│ └── ... (same structure)
|
||||
└── clair/
|
||||
└── ... (same structure)
|
||||
```
|
||||
|
||||
### Benchmark Workflow
|
||||
|
||||
```yaml
|
||||
# .gitea/workflows/benchmark-parity.yml
|
||||
name: Benchmark Parity Check
|
||||
|
||||
on:
|
||||
schedule:
|
||||
- cron: '0 0 * * 0' # Weekly
|
||||
workflow_dispatch:
|
||||
|
||||
jobs:
|
||||
benchmark:
|
||||
runs-on: ubuntu-latest
|
||||
strategy:
|
||||
matrix:
|
||||
tool: [syft, trivy, clair]
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Run ${{ matrix.tool }} benchmark
|
||||
run: |
|
||||
scripts/benchmark/run-benchmark.sh ${{ matrix.tool }}
|
||||
|
||||
- name: Compare results
|
||||
run: |
|
||||
scripts/benchmark/compare-results.sh ${{ matrix.tool }}
|
||||
|
||||
- name: Upload logs
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: benchmark-${{ matrix.tool }}
|
||||
path: benchmark-results/
|
||||
```
|
||||
|
||||
### Benchmark Comparison
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# scripts/benchmark/compare-results.sh
|
||||
|
||||
TOOL=$1
|
||||
BENCHMARK_DIR="tests/benchmark/${TOOL}"
|
||||
|
||||
for input in "${BENCHMARK_DIR}/inputs/"*.json; do
|
||||
name=$(basename "${input}" .json)
|
||||
expected="${BENCHMARK_DIR}/expected/${name}-expected.json"
|
||||
actual="benchmark-results/${name}-actual.json"
|
||||
|
||||
# Run tool
|
||||
stellaops ingest normalize \
|
||||
--tool "${TOOL}" \
|
||||
--input "${input}" \
|
||||
--output "${actual}"
|
||||
|
||||
# Compare
|
||||
diff_result=$(diff <(jq -S . "${expected}") <(jq -S . "${actual}"))
|
||||
|
||||
if [[ -n "${diff_result}" ]]; then
|
||||
echo "DRIFT: ${name}"
|
||||
echo "${diff_result}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "PASS: ${name}"
|
||||
done
|
||||
```
|
||||
|
||||
### Drift Detection
|
||||
|
||||
When benchmark drift detected:
|
||||
|
||||
1. Log drift details with hash comparison
|
||||
2. Create issue in tracking system
|
||||
3. Notify Scanner Guild
|
||||
4. Block release if critical drift
|
||||
|
||||
```json
|
||||
{
|
||||
"drift": {
|
||||
"tool": "syft",
|
||||
"version": "1.0.0",
|
||||
"testCase": "alpine-3.19",
|
||||
"detected": "2025-12-04T00:00:00Z",
|
||||
"details": {
|
||||
"expectedHash": "b3:expected...",
|
||||
"actualHash": "b3:actual...",
|
||||
"diffCount": 3,
|
||||
"fields": [
|
||||
"/components/5/version",
|
||||
"/components/12/licenses",
|
||||
"/vulnerabilities/2/ratings/0/score"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## CM9: Coverage Tracking
|
||||
|
||||
### Coverage Matrix
|
||||
|
||||
Location: `docs/modules/scanner/fixtures/competitor-adapters/coverage.csv`
|
||||
|
||||
```csv
|
||||
ecosystem,syft,trivy,clair,notes
|
||||
container,yes,yes,yes,All tools support OCI images
|
||||
java,yes,yes,no,Clair Java support pending
|
||||
python,yes,yes,no,Trivy has best pip/poetry coverage
|
||||
dotnet,no,yes,no,Trivy only; Syft support pending
|
||||
go,yes,yes,no,Both tools have good go.mod support
|
||||
rust,yes,yes,no,Cargo.lock parsing
|
||||
ruby,yes,yes,no,Gemfile.lock parsing
|
||||
php,yes,yes,no,composer.lock parsing
|
||||
os-pkgs,yes,yes,yes,APK/DEB/RPM supported
|
||||
node,yes,yes,no,package-lock.json/yarn.lock
|
||||
```
|
||||
|
||||
### Coverage API
|
||||
|
||||
```http
|
||||
GET /api/v1/ingest/coverage
|
||||
```
|
||||
|
||||
Response:
|
||||
```json
|
||||
{
|
||||
"coverage": {
|
||||
"lastUpdated": "2025-12-04T00:00:00Z",
|
||||
"ecosystems": {
|
||||
"container": {
|
||||
"syft": {"supported": true, "tested": true},
|
||||
"trivy": {"supported": true, "tested": true},
|
||||
"clair": {"supported": true, "tested": true}
|
||||
},
|
||||
"java": {
|
||||
"syft": {"supported": true, "tested": true},
|
||||
"trivy": {"supported": true, "tested": true},
|
||||
"clair": {"supported": false, "tested": false, "planned": "2026-Q1"}
|
||||
}
|
||||
},
|
||||
"gaps": [
|
||||
{"ecosystem": "dotnet", "tool": "syft", "priority": "high"},
|
||||
{"ecosystem": "java", "tool": "clair", "priority": "medium"}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Gap Tracking
|
||||
|
||||
```json
|
||||
{
|
||||
"gaps": [
|
||||
{
|
||||
"id": "gap-001",
|
||||
"ecosystem": "dotnet",
|
||||
"tool": "syft",
|
||||
"priority": "high",
|
||||
"reason": "Customer demand for .NET scanning",
|
||||
"status": "planned",
|
||||
"targetDate": "2025-Q2",
|
||||
"blockers": ["Upstream syft issue #1234"]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Coverage CI Check
|
||||
|
||||
```yaml
|
||||
# Check coverage doesn't regress
|
||||
- name: Verify coverage matrix
|
||||
run: |
|
||||
# Ensure no "yes" changed to "no" without documentation
|
||||
git diff HEAD~1 docs/modules/scanner/fixtures/competitor-adapters/coverage.csv \
|
||||
| grep -E '^\-.*yes.*$' && {
|
||||
echo "Coverage regression detected"
|
||||
exit 1
|
||||
}
|
||||
```
|
||||
|
||||
## Reporting
|
||||
|
||||
### Weekly Coverage Report
|
||||
|
||||
```json
|
||||
{
|
||||
"report": {
|
||||
"period": "2025-W49",
|
||||
"coverage": {
|
||||
"total_ecosystems": 10,
|
||||
"full_coverage": 3,
|
||||
"partial_coverage": 5,
|
||||
"no_coverage": 2
|
||||
},
|
||||
"benchmark": {
|
||||
"tests_run": 45,
|
||||
"tests_passed": 44,
|
||||
"tests_failed": 1,
|
||||
"drift_detected": ["trivy/alpine-3.19"]
|
||||
},
|
||||
"metadata": {
|
||||
"snapshots_imported": 156,
|
||||
"tools_seen": ["syft", "trivy", "clair"],
|
||||
"versions_seen": {
|
||||
"syft": ["1.0.0", "1.0.1"],
|
||||
"trivy": ["0.50.0"],
|
||||
"clair": ["6.0.0"]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Links
|
||||
|
||||
- Sprint: `docs/implplan/SPRINT_0186_0001_0001_record_deterministic_execution.md` (CM7, CM8, CM9)
|
||||
- Normalization: `docs/modules/scanner/design/competitor-ingest-normalization.md` (CM1)
|
||||
- Coverage CSV: `docs/modules/scanner/fixtures/competitor-adapters/coverage.csv`
|
||||
Reference in New Issue
Block a user