Add Canonical JSON serialization library with tests and documentation
- Implemented CanonJson class for deterministic JSON serialization and hashing. - Added unit tests for CanonJson functionality, covering various scenarios including key sorting, handling of nested objects, arrays, and special characters. - Created project files for the Canonical JSON library and its tests, including necessary package references. - Added README.md for library usage and API reference. - Introduced RabbitMqIntegrationFactAttribute for conditional RabbitMQ integration tests.
This commit is contained in:
master
183
bench/unknowns/README.md
Normal file
183
bench/unknowns/README.md
Normal file
@@ -0,0 +1,183 @@
|
||||
# Unknowns Tracking Benchmark Suite
|
||||
|
||||
> **Purpose:** Verify epistemic uncertainty tracking and unknown state management.
|
||||
> **Status:** Active
|
||||
> **Sprint:** SPRINT_3850_0001_0001 (Competitive Gap Closure)
|
||||
|
||||
## Overview
|
||||
|
||||
StellaOps tracks "unknowns" - gaps in knowledge that affect confidence:
|
||||
- Missing SBOM components
|
||||
- Unmatched CVEs
|
||||
- Stale feed data
|
||||
- Zero-day windows
|
||||
- Analysis limitations
|
||||
|
||||
## What Gets Tested
|
||||
|
||||
### Unknown State Lifecycle
|
||||
1. Detection of unknown conditions
|
||||
2. Propagation to affected findings
|
||||
3. Score penalty application
|
||||
4. Resolution tracking
|
||||
|
||||
### Unknown Categories
|
||||
- `SBOM_GAP`: Component not in SBOM
|
||||
- `CVE_UNMATCHED`: CVE without component mapping
|
||||
- `FEED_STALE`: Feed data older than threshold
|
||||
- `ZERO_DAY_WINDOW`: Time between disclosure and feed update
|
||||
- `ANALYSIS_LIMIT`: Depth/timeout constraints
|
||||
|
||||
### Score Impact
|
||||
- Each unknown type has a penalty weight
|
||||
- Penalties reduce overall confidence
|
||||
- Resolved unknowns restore confidence
|
||||
|
||||
## Test Cases
|
||||
|
||||
### TC-001: SBOM Gap Detection
|
||||
|
||||
```json
|
||||
{
|
||||
"scenario": "Package in image not in SBOM",
|
||||
"input": {
|
||||
"image_packages": ["openssl@3.0.1", "curl@7.86"],
|
||||
"sbom_packages": ["openssl@3.0.1"]
|
||||
},
|
||||
"expected": {
|
||||
"unknowns": [{ "type": "SBOM_GAP", "package": "curl@7.86" }],
|
||||
"confidence_penalty": 0.15
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### TC-002: Zero-Day Window Tracking
|
||||
|
||||
```json
|
||||
{
|
||||
"scenario": "CVE disclosed before feed update",
|
||||
"input": {
|
||||
"cve_disclosure": "2025-01-01T00:00:00Z",
|
||||
"feed_update": "2025-01-03T00:00:00Z",
|
||||
"scan_time": "2025-01-02T12:00:00Z"
|
||||
},
|
||||
"expected": {
|
||||
"unknowns": [{
|
||||
"type": "ZERO_DAY_WINDOW",
|
||||
"cve": "CVE-2025-0001",
|
||||
"window_hours": 36
|
||||
}],
|
||||
"risk_note": "Scan occurred during zero-day window"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### TC-003: Feed Staleness
|
||||
|
||||
```json
|
||||
{
|
||||
"scenario": "NVD feed older than 24 hours",
|
||||
"input": {
|
||||
"feed_last_update": "2025-01-01T00:00:00Z",
|
||||
"scan_time": "2025-01-02T12:00:00Z",
|
||||
"staleness_threshold_hours": 24
|
||||
},
|
||||
"expected": {
|
||||
"unknowns": [{
|
||||
"type": "FEED_STALE",
|
||||
"feed": "nvd",
|
||||
"age_hours": 36
|
||||
}]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### TC-004: Score Penalty Application
|
||||
|
||||
```json
|
||||
{
|
||||
"scenario": "Multiple unknowns compound penalty",
|
||||
"input": {
|
||||
"base_confidence": 0.95,
|
||||
"unknowns": [
|
||||
{ "type": "SBOM_GAP", "penalty": 0.15 },
|
||||
{ "type": "FEED_STALE", "penalty": 0.10 }
|
||||
]
|
||||
},
|
||||
"expected": {
|
||||
"final_confidence": 0.70,
|
||||
"penalty_formula": "0.95 * (1 - 0.15) * (1 - 0.10)"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Fixtures
|
||||
|
||||
```
|
||||
fixtures/
|
||||
├── sbom-gaps/
|
||||
│ ├── single-missing.json
|
||||
│ ├── multiple-missing.json
|
||||
│ └── layer-specific.json
|
||||
├── zero-day/
|
||||
│ ├── within-window.json
|
||||
│ ├── after-window.json
|
||||
│ └── ongoing.json
|
||||
├── feed-staleness/
|
||||
│ ├── nvd-stale.json
|
||||
│ ├── osv-stale.json
|
||||
│ └── multiple-stale.json
|
||||
└── expected/
|
||||
└── all-tests.results.json
|
||||
```
|
||||
|
||||
## Running the Suite
|
||||
|
||||
```bash
|
||||
# Run unknowns tests
|
||||
dotnet test tests/StellaOps.Unknowns.Tests
|
||||
|
||||
# Run penalty calculation tests
|
||||
./run-penalty-tests.sh
|
||||
|
||||
# Run full benchmark
|
||||
./run-benchmark.sh --all
|
||||
```
|
||||
|
||||
## Metrics
|
||||
|
||||
| Metric | Target | Description |
|
||||
|--------|--------|-------------|
|
||||
| Detection rate | 100% | All unknown conditions detected |
|
||||
| Penalty accuracy | ±1% | Penalties match expected values |
|
||||
| Resolution tracking | 100% | All resolutions properly logged |
|
||||
|
||||
## UI Integration
|
||||
|
||||
Unknowns appear as:
|
||||
- Chips in findings table
|
||||
- Warning banners on scan results
|
||||
- Confidence reduction indicators
|
||||
- Triage action suggestions
|
||||
|
||||
## Integration with CI
|
||||
|
||||
```yaml
|
||||
# .gitea/workflows/bench-unknowns.yaml
|
||||
name: Unknowns Benchmark
|
||||
on:
|
||||
push:
|
||||
paths:
|
||||
- 'src/Unknowns/**'
|
||||
- 'bench/unknowns/**'
|
||||
|
||||
jobs:
|
||||
unknowns:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- name: Run Unknowns Tests
|
||||
run: dotnet test tests/StellaOps.Unknowns.Tests
|
||||
- name: Run Benchmark
|
||||
run: ./bench/unknowns/run-benchmark.sh
|
||||
```
|
||||
Reference in New Issue
Block a user