git.stella-ops.org/bench/unknowns

Unknowns Tracking Benchmark Suite

Purpose: Verify epistemic uncertainty tracking and unknown state management. Status: Active Sprint: SPRINT_3850_0001_0001 (Competitive Gap Closure)

Overview

StellaOps tracks "unknowns" - gaps in knowledge that affect confidence:

• Missing SBOM components
• Unmatched CVEs
• Stale feed data
• Zero-day windows
• Analysis limitations

What Gets Tested

Unknown State Lifecycle

1. Detection of unknown conditions
2. Propagation to affected findings
3. Score penalty application
4. Resolution tracking

Unknown Categories

• SBOM_GAP: Component not in SBOM
• CVE_UNMATCHED: CVE without component mapping
• FEED_STALE: Feed data older than threshold
• ZERO_DAY_WINDOW: Time between disclosure and feed update
• ANALYSIS_LIMIT: Depth/timeout constraints

Score Impact

• Each unknown type has a penalty weight
• Penalties reduce overall confidence
• Resolved unknowns restore confidence

Test Cases

TC-001: SBOM Gap Detection

{
  "scenario": "Package in image not in SBOM",
  "input": {
    "image_packages": ["openssl@3.0.1", "curl@7.86"],
    "sbom_packages": ["openssl@3.0.1"]
  },
  "expected": {
    "unknowns": [{ "type": "SBOM_GAP", "package": "curl@7.86" }],
    "confidence_penalty": 0.15
  }
}

TC-002: Zero-Day Window Tracking

{
  "scenario": "CVE disclosed before feed update",
  "input": {
    "cve_disclosure": "2025-01-01T00:00:00Z",
    "feed_update": "2025-01-03T00:00:00Z",
    "scan_time": "2025-01-02T12:00:00Z"
  },
  "expected": {
    "unknowns": [{
      "type": "ZERO_DAY_WINDOW",
      "cve": "CVE-2025-0001",
      "window_hours": 36
    }],
    "risk_note": "Scan occurred during zero-day window"
  }
}

TC-003: Feed Staleness

{
  "scenario": "NVD feed older than 24 hours",
  "input": {
    "feed_last_update": "2025-01-01T00:00:00Z",
    "scan_time": "2025-01-02T12:00:00Z",
    "staleness_threshold_hours": 24
  },
  "expected": {
    "unknowns": [{
      "type": "FEED_STALE",
      "feed": "nvd",
      "age_hours": 36
    }]
  }
}

TC-004: Score Penalty Application

{
  "scenario": "Multiple unknowns compound penalty",
  "input": {
    "base_confidence": 0.95,
    "unknowns": [
      { "type": "SBOM_GAP", "penalty": 0.15 },
      { "type": "FEED_STALE", "penalty": 0.10 }
    ]
  },
  "expected": {
    "final_confidence": 0.70,
    "penalty_formula": "0.95 * (1 - 0.15) * (1 - 0.10)"
  }
}

Fixtures

fixtures/
├── sbom-gaps/
│   ├── single-missing.json
│   ├── multiple-missing.json
│   └── layer-specific.json
├── zero-day/
│   ├── within-window.json
│   ├── after-window.json
│   └── ongoing.json
├── feed-staleness/
│   ├── nvd-stale.json
│   ├── osv-stale.json
│   └── multiple-stale.json
└── expected/
    └── all-tests.results.json

Running the Suite

# Run unknowns tests
dotnet test tests/StellaOps.Unknowns.Tests

# Run penalty calculation tests
./run-penalty-tests.sh

# Run full benchmark
./run-benchmark.sh --all

Metrics

Metric	Target	Description
Detection rate	100%	All unknown conditions detected
Penalty accuracy	±1%	Penalties match expected values
Resolution tracking	100%	All resolutions properly logged

UI Integration

Unknowns appear as:

• Chips in findings table
• Warning banners on scan results
• Confidence reduction indicators
• Triage action suggestions

Integration with CI

# .gitea/workflows/bench-unknowns.yaml
name: Unknowns Benchmark
on:
  push:
    paths:
      - 'src/Unknowns/**'
      - 'bench/unknowns/**'

jobs:
  unknowns:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run Unknowns Tests
        run: dotnet test tests/StellaOps.Unknowns.Tests
      - name: Run Benchmark
        run: ./bench/unknowns/run-benchmark.sh