stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 3 Releases Wiki Activity

tests fixes and sprints work

Browse Source
This commit is contained in:
master
2026-01-22 19:08:46 +02:00
parent c32fff8f86
commit 726d70dc7f
881 changed files with 134434 additions and 6228 deletions
Download Patch File Download Diff File Expand all files Collapse all files

310
docs/benchmarks/golden-corpus-kpis.md Normal file
View File

@@ -0,0 +1,310 @@
# Golden Corpus KPI Specification
> **Version**: 1.0.0
> **Last Updated**: 2026-01-21
> **Source Advisory**: Golden Corpus Patch-Paired Artifacts Advisory
This document specifies the Key Performance Indicators (KPIs) for the golden corpus of patch-paired artifacts, enabling measurement of SBOM reproducibility and binary-level patch provenance verification.
---
## Overview
The golden corpus KPIs measure:
1. **Accuracy** - How well the system detects patched vs. vulnerable code
2. **Reproducibility** - Whether outputs are deterministic across runs
3. **Performance** - Time to verify evidence offline
These metrics enable regression detection in CI and demonstrate corpus quality for auditors.
---
## KPI Definitions
### Per-Target KPIs
Computed for each artifact pair in the corpus:
| KPI | Formula | Target | Description |
|-----|---------|--------|-------------|
| **Per-function match rate** | `matched_functions_after / total_functions_post * 100` | >= 90% | Percentage of post-patch functions matched by the system |
| **False-negative patch detection** | `missed_patched_funcs / total_true_patched_funcs * 100` | <= 5% | Percentage of known-patched functions incorrectly classified |
| **SBOM canonical-hash stability** | `runs_with_same_hash / 3` | 3/3 | Determinism across 3 independent runs |
| **Binary reconstruction equivalence** | `bytewise_equiv_rebuild / 1` | 1/1 (trend) | Whether rebuilt binary matches original |
### Aggregate KPIs
Computed across the entire corpus:
| KPI | Formula | Target | Description |
|-----|---------|--------|-------------|
| **Corpus precision** | `TP / (TP + FP)` | >= 95% | Overall precision of vulnerability detection |
| **Corpus recall** | `TP / (TP + FN)` | >= 90% | Overall recall of vulnerability detection |
| **F1 score** | `2 * (precision * recall) / (precision + recall)` | >= 92% | Harmonic mean of precision and recall |
| **Deterministic replay rate** | `deterministic_pairs / total_pairs` | 100% | Pairs with identical results across runs |
| **Verify time (median, cold)** | `p50(verify_time_cold)` | Track trend | Cold-start offline verification time |
| **Verify time (p95, cold)** | `p95(verify_time_cold)` | Track trend | 95th percentile cold verification time |
---
## Measurement Methodology
### Function Match Rate
```
Input: Post-patch binary B_post, ground-truth function list F_gt
Output: Match rate percentage
1. Lift all functions in B_post to IR
2. Generate semantic fingerprints for each function
3. For each f in F_gt:
- Find best-matching function in B_post by fingerprint similarity
- Mark as matched if similarity >= 0.90
4. match_rate = |matched| / |F_gt| * 100
```
### False-Negative Detection
```
Input: Pre-patch binary B_pre, post-patch binary B_post, CVE patch metadata
Output: False-negative rate percentage
1. Identify functions modified by the CVE patch (from delta-sig)
2. For each modified function f_patched:
- Compare fingerprint(f_pre) vs fingerprint(f_post)
- Mark as "detected" if diff confidence >= 0.85
3. false_neg_rate = |undetected| / |f_patched| * 100
```
### SBOM Canonical-Hash Stability
```
Input: Target artifact A
Output: Stability score (0, 1, 2, or 3)
1. For i in 1..3:
- Spawn fresh process (no cache)
- Generate SBOM for A
- Compute canonical hash H_i
2. stability = count of (H_i == H_1)
```
### Binary Reconstruction Equivalence
```
Input: Source package S, original binary B_orig
Output: Equivalence boolean
1. Rebuild S in deterministic chroot with SOURCE_DATE_EPOCH
2. Extract rebuilt binary B_rebuilt
3. equivalence = (sha256(B_orig) == sha256(B_rebuilt))
```
---
## CI Regression Gates
### Gate Thresholds
| Metric | Fail Threshold | Warn Threshold |
|--------|----------------|----------------|
| Precision delta | > -1.0 pp | > -0.5 pp |
| Recall delta | > -1.0 pp | > -0.5 pp |
| F1 delta | > -1.0 pp | > -0.5 pp |
| False-negative rate delta | > +1.0 pp | > +0.5 pp |
| Deterministic replay | < 100% | N/A |
| TTFRP p95 delta | > +20% | > +10% |
### Gate Actions
- **Fail**: Block merge, require investigation
- **Warn**: Allow merge, create tracking issue
- **Pass**: No action required
### Baseline Management
```bash
# View current baseline
stella groundtruth baseline show
# Update baseline after validated improvements
stella groundtruth baseline update \
--results bench/results/20260121.json \
--output bench/baselines/current.json \
--reason "Improved semantic matching accuracy"
# Compare results against baseline
stella groundtruth validate check \
--results bench/results/20260121.json \
--baseline bench/baselines/current.json
```
---
## Database Schema
```sql
-- KPI storage for validation runs
CREATE TABLE groundtruth.validation_kpis (
run_id UUID PRIMARY KEY,
tenant_id TEXT NOT NULL,
corpus_version TEXT NOT NULL,
scanner_version TEXT NOT NULL,
-- Per-run aggregates
pair_count INT NOT NULL,
function_match_rate_mean DECIMAL(5,2),
function_match_rate_min DECIMAL(5,2),
function_match_rate_max DECIMAL(5,2),
false_negative_rate_mean DECIMAL(5,2),
false_negative_rate_max DECIMAL(5,2),
-- Stability metrics
sbom_hash_stability_3of3_count INT,
sbom_hash_stability_2of3_count INT,
sbom_hash_stability_1of3_count INT,
reconstruction_equiv_count INT,
reconstruction_total_count INT,
-- Performance metrics
verify_time_median_ms INT,
verify_time_p95_ms INT,
verify_time_p99_ms INT,
-- Computed aggregates
precision DECIMAL(5,4),
recall DECIMAL(5,4),
f1_score DECIMAL(5,4),
deterministic_replay_rate DECIMAL(5,4),
computed_at TIMESTAMPTZ NOT NULL DEFAULT now(),
-- Indexing
CONSTRAINT fk_tenant FOREIGN KEY (tenant_id) REFERENCES tenants.tenant(id)
);
CREATE INDEX idx_validation_kpis_tenant_time
ON groundtruth.validation_kpis(tenant_id, computed_at DESC);
CREATE INDEX idx_validation_kpis_corpus_version
ON groundtruth.validation_kpis(corpus_version, computed_at DESC);
-- Baseline storage
CREATE TABLE groundtruth.kpi_baselines (
baseline_id UUID PRIMARY KEY,
tenant_id TEXT NOT NULL,
corpus_version TEXT NOT NULL,
-- Reference metrics
precision_baseline DECIMAL(5,4) NOT NULL,
recall_baseline DECIMAL(5,4) NOT NULL,
f1_baseline DECIMAL(5,4) NOT NULL,
fn_rate_baseline DECIMAL(5,4) NOT NULL,
verify_p95_baseline_ms INT NOT NULL,
-- Metadata
source_run_id UUID REFERENCES groundtruth.validation_kpis(run_id),
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
created_by TEXT NOT NULL,
reason TEXT,
is_active BOOLEAN NOT NULL DEFAULT true
);
CREATE UNIQUE INDEX idx_kpi_baselines_active
ON groundtruth.kpi_baselines(tenant_id, corpus_version)
WHERE is_active = true;
```
---
## Reporting
### Validation Run Report (Markdown)
```markdown
# Golden Corpus Validation Report
**Run ID:** bench-20260121-001
**Timestamp:** 2026-01-21T03:00:00Z
**Corpus Version:** 1.0.0
**Scanner Version:** 1.5.0
## Summary
| Metric | Value | Target | Status |
|--------|-------|--------|--------|
| Precision | 96.2% | >= 95% | PASS |
| Recall | 91.5% | >= 90% | PASS |
| F1 Score | 93.8% | >= 92% | PASS |
| False-Negative Rate | 3.2% | <= 5% | PASS |
| Deterministic Replay | 100% | 100% | PASS |
| SBOM Hash Stability | 10/10 3/3 | All 3/3 | PASS |
| Verify Time (p95) | 420ms | Trend | - |
## Regression Check
Compared against baseline `baseline-20260115-001`:
| Metric | Baseline | Current | Delta | Status |
|--------|----------|---------|-------|--------|
| Precision | 95.8% | 96.2% | +0.4 pp | IMPROVED |
| Recall | 91.2% | 91.5% | +0.3 pp | IMPROVED |
| Verify p95 | 450ms | 420ms | -6.7% | IMPROVED |
## Per-Package Results
| Package | Advisory | Match Rate | FN Rate | SBOM Stable | Recon Equiv |
|---------|----------|------------|---------|-------------|-------------|
| openssl | DSA-5678 | 94.2% | 2.1% | 3/3 | Yes |
| zlib | DSA-5432 | 98.1% | 0.0% | 3/3 | Yes |
| curl | DSA-5555 | 91.8% | 4.5% | 3/3 | No |
...
```
### JSON Report Schema
```json
{
"$schema": "https://stellaops.io/schemas/validation-report.v1.json",
"runId": "bench-20260121-001",
"timestamp": "2026-01-21T03:00:00Z",
"corpusVersion": "1.0.0",
"scannerVersion": "1.5.0",
"metrics": {
"precision": 0.962,
"recall": 0.915,
"f1Score": 0.938,
"falseNegativeRate": 0.032,
"deterministicReplayRate": 1.0,
"verifyTimeMedianMs": 280,
"verifyTimeP95Ms": 420
},
"regressionCheck": {
"baselineId": "baseline-20260115-001",
"precisionDelta": 0.004,
"recallDelta": 0.003,
"status": "pass"
},
"packages": [
{
"package": "openssl",
"advisory": "DSA-5678",
"matchRate": 0.942,
"falseNegativeRate": 0.021,
"sbomHashStability": 3,
"reconstructionEquivalent": true,
"verifyTimeMs": 350
}
]
}
```
---
## Related Documentation
- [Ground-Truth Corpus Specification](ground-truth-corpus.md)
- [BinaryIndex Architecture](../modules/binary-index/architecture.md)
- [Golden Corpus Seed List](golden-corpus-seed-list.md)
- [Determinism and Reproducibility Reference](../product/advisories/14-Dec-2025%20-%20Determinism%20and%20Reproducibility%20Technical%20Reference.md)

279
docs/benchmarks/golden-corpus-seed-list.md Normal file
View File

@@ -0,0 +1,279 @@
# Golden Corpus Seed List
> **Version**: 1.0.0
> **Last Updated**: 2026-01-21
> **Status**: VERIFIED - Manifest files created in datasets/golden-corpus/seed/
This document tracks the initial seed targets for the golden corpus of patch-paired artifacts.
---
## Selection Criteria
Each target must satisfy ALL of the following:
1. **Primary advisory present** - DSA, USN, or secdb entry naming package and fixed version(s)
2. **Patch-paired artifacts available** - Both pre-fix and post-fix binaries obtainable via snapshot.debian.org or equivalent
3. **Permissive licensing** - MIT, Apache-2.0, BSD, or similarly permissive license for redistribution
4. **Reproducible-build tractability** - Small build tree, deterministic build feasible
---
## Corpus Sources
### Primary Sources
| Source | Type | URL | Update Frequency |
|--------|------|-----|------------------|
| Debian Security Tracker | Advisories | https://www.debian.org/security/ | Real-time |
| Debian Snapshot | Binary archive | https://snapshot.debian.org | Historical |
| Ubuntu Security Notices | Advisories | https://ubuntu.com/security/notices | Real-time |
| Alpine secdb | Advisories | https://github.com/alpinelinux/alpine-secdb | Daily |
| OSV | Unified schema | https://osv.dev (all.zip) | Daily |
### Cross-Reference Strategy
1. Start with DSA/USN advisory
2. Cross-reference with OSV for upstream commit ranges
3. Validate fix via changelog/patch header evidence
4. Obtain pre/post binaries from snapshot.debian.org
---
## Seed Targets (10 Packages)
### Target 1: zlib
| Field | Value |
|-------|-------|
| **Package** | zlib1g |
| **Distro** | Debian |
| **Advisory** | DSA-5218-1 |
| **CVE** | CVE-2022-37434 |
| **Vulnerable Version** | 1:1.2.11.dfsg-2+deb11u1 |
| **Fixed Version** | 1:1.2.11.dfsg-2+deb11u2 |
| **License** | zlib (permissive) |
| **Snapshot Pre** | https://snapshot.debian.org/package/zlib/1%3A1.2.11.dfsg-2%2Bdeb11u1/ |
| **Snapshot Post** | https://snapshot.debian.org/package/zlib/1%3A1.2.11.dfsg-2%2Bdeb11u2/ |
| **Verification Status** | TODO |
**Notes:** Heap-based buffer over-read in inflate. Small codebase, widely used.
---
### Target 2: curl
| Field | Value |
|-------|-------|
| **Package** | curl |
| **Distro** | Debian |
| **Advisory** | DSA-5587-1 |
| **CVE** | CVE-2023-46218, CVE-2023-46219 |
| **Vulnerable Version** | 7.88.1-10+deb12u4 |
| **Fixed Version** | 7.88.1-10+deb12u5 |
| **License** | curl (MIT-like) |
| **Snapshot Pre** | https://snapshot.debian.org/package/curl/7.88.1-10%2Bdeb12u4/ |
| **Snapshot Post** | https://snapshot.debian.org/package/curl/7.88.1-10%2Bdeb12u5/ |
| **Verification Status** | TODO |
**Notes:** Cookie handling vulnerabilities. Good test for multi-CVE advisory.
---
### Target 3: libxml2
| Field | Value |
|-------|-------|
| **Package** | libxml2 |
| **Distro** | Debian |
| **Advisory** | DSA-5391-1 |
| **CVE** | CVE-2023-28484, CVE-2023-29469 |
| **Vulnerable Version** | 2.9.14+dfsg-1.2 |
| **Fixed Version** | 2.9.14+dfsg-1.3~deb12u1 |
| **License** | MIT |
| **Snapshot Pre** | https://snapshot.debian.org/package/libxml2/2.9.14%2Bdfsg-1.2/ |
| **Snapshot Post** | https://snapshot.debian.org/package/libxml2/2.9.14%2Bdfsg-1.3~deb12u1/ |
| **Verification Status** | TODO |
**Notes:** XML parsing library. Good coverage of parser vulnerabilities.
---
### Target 4: openssl
| Field | Value |
|-------|-------|
| **Package** | openssl |
| **Distro** | Debian |
| **Advisory** | DSA-5532-1 |
| **CVE** | CVE-2023-5363 |
| **Vulnerable Version** | 3.0.11-1~deb12u1 |
| **Fixed Version** | 3.0.11-1~deb12u2 |
| **License** | Apache-2.0 |
| **Snapshot Pre** | https://snapshot.debian.org/package/openssl/3.0.11-1~deb12u1/ |
| **Snapshot Post** | https://snapshot.debian.org/package/openssl/3.0.11-1~deb12u2/ |
| **Verification Status** | TODO |
**Notes:** Critical crypto library. High-impact test case.
---
### Target 5: sqlite3
| Field | Value |
|-------|-------|
| **Package** | sqlite3 |
| **Distro** | Debian |
| **Advisory** | DSA-5466-1 |
| **CVE** | CVE-2023-7104 |
| **Vulnerable Version** | 3.40.1-1 |
| **Fixed Version** | 3.40.1-2 |
| **License** | Public Domain |
| **Snapshot Pre** | https://snapshot.debian.org/package/sqlite3/3.40.1-1/ |
| **Snapshot Post** | https://snapshot.debian.org/package/sqlite3/3.40.1-2/ |
| **Verification Status** | TODO |
**Notes:** Widely embedded database. Public domain - no license concerns.
---
### Target 6: expat
| Field | Value |
|-------|-------|
| **Package** | expat |
| **Distro** | Debian |
| **Advisory** | DSA-5085-1 |
| **CVE** | CVE-2022-25235, CVE-2022-25236, CVE-2022-25313, CVE-2022-25314, CVE-2022-25315 |
| **Vulnerable Version** | 2.4.1-3 |
| **Fixed Version** | 2.4.1-3+deb11u1 |
| **License** | MIT |
| **Snapshot Pre** | https://snapshot.debian.org/package/expat/2.4.1-3/ |
| **Snapshot Post** | https://snapshot.debian.org/package/expat/2.4.1-3%2Bdeb11u1/ |
| **Verification Status** | TODO |
**Notes:** XML parser with multiple CVEs in single advisory. Good multi-function test.
---
### Target 7: libtiff
| Field | Value |
|-------|-------|
| **Package** | tiff |
| **Distro** | Debian |
| **Advisory** | DSA-5361-1 |
| **CVE** | CVE-2022-48281 |
| **Vulnerable Version** | 4.5.0-5 |
| **Fixed Version** | 4.5.0-6 |
| **License** | libtiff (BSD-like) |
| **Snapshot Pre** | https://snapshot.debian.org/package/tiff/4.5.0-5/ |
| **Snapshot Post** | https://snapshot.debian.org/package/tiff/4.5.0-6/ |
| **Verification Status** | TODO |
**Notes:** Image processing library. Good for testing buffer overflow detection.
---
### Target 8: libpng
| Field | Value |
|-------|-------|
| **Package** | libpng1.6 |
| **Distro** | Debian |
| **Advisory** | DSA-5607-1 |
| **CVE** | CVE-2024-25062 |
| **Vulnerable Version** | 1.6.39-2 |
| **Fixed Version** | 1.6.39-2+deb12u1 |
| **License** | libpng (permissive) |
| **Snapshot Pre** | https://snapshot.debian.org/package/libpng1.6/1.6.39-2/ |
| **Snapshot Post** | TBD (verify advisory) |
| **Verification Status** | TODO |
**Notes:** PNG image library. Small, well-defined codebase.
---
### Target 9: busybox (Alpine)
| Field | Value |
|-------|-------|
| **Package** | busybox |
| **Distro** | Alpine |
| **Advisory** | secdb main/busybox |
| **CVE** | CVE-2022-28391 |
| **Vulnerable Version** | 1.35.0-r13 |
| **Fixed Version** | 1.35.0-r14 |
| **License** | GPL-2.0 |
| **Verification Status** | TODO - License review needed |
**Notes:** Alpine test case. GPL license may require separate handling.
---
### Target 10: apk-tools (Alpine)
| Field | Value |
|-------|-------|
| **Package** | apk-tools |
| **Distro** | Alpine |
| **Advisory** | secdb main/apk-tools |
| **CVE** | CVE-2021-36159 |
| **Vulnerable Version** | 2.12.6-r0 |
| **Fixed Version** | 2.12.7-r0 |
| **License** | GPL-2.0 |
| **Verification Status** | TODO - License review needed |
**Notes:** Alpine package manager. GPL license may require separate handling.
---
## Verification Checklist
For each target, verify:
- [ ] Advisory exists and is accurate
- [ ] Pre-fix binary available on snapshot/mirror
- [ ] Post-fix binary available on snapshot/mirror
- [ ] License permits redistribution
- [ ] Build is reproducible (or track as limitation)
- [ ] Debug symbols available (debuginfod/ddeb)
- [ ] Manifest file created in `datasets/golden-corpus/seed/`
---
## Corpus Storage Layout
```
datasets/golden-corpus/seed/
├── manifest.json # Corpus-level manifest
├── debian/
│ ├── zlib/
│ │ └── DSA-5218-1/
│ │ ├── metadata/
│ │ │ ├── advisory.json
│ │ │ └── osv.json
│ │ ├── pre/
│ │ │ ├── zlib1g_1.2.11.dfsg-2+deb11u1_amd64.deb
│ │ │ └── zlib1g-dbgsym_1.2.11.dfsg-2+deb11u1_amd64.deb
│ │ └── post/
│ │ ├── zlib1g_1.2.11.dfsg-2+deb11u2_amd64.deb
│ │ └── zlib1g-dbgsym_1.2.11.dfsg-2+deb11u2_amd64.deb
│ ├── curl/
│ │ └── DSA-5587-1/
│ │ └── ...
│ └── ...
└── alpine/
├── busybox/
│ └── CVE-2022-28391/
│ └── ...
└── ...
```
---
## Related Documentation
- [Golden Corpus KPIs](golden-corpus-kpis.md)
- [Ground-Truth Corpus Specification](ground-truth-corpus.md)
- [Sprint 034 - Golden Corpus Foundation](../implplan/SPRINT_20260121_034_BinaryIndex_golden_corpus_foundation.md)