stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 1 Releases Wiki Activity

feat: Add new projects to solution and implement contract testing documentation

Browse Source 
- Added "StellaOps.Policy.Engine", "StellaOps.Cartographer", and "StellaOps.SbomService" projects to the StellaOps solution.
- Created AGENTS.md to outline the Contract Testing Guild Charter, detailing mission, scope, and definition of done.
- Established TASKS.md for the Contract Testing Task Board, outlining tasks for Sprint 62 and Sprint 63 related to mock servers and replay testing.
This commit is contained in:
master
2025-10-27 07:57:55 +02:00
parent 1e41ba7ffa
commit 651b8e0fa3
355 changed files with 17276 additions and 1160 deletions
Download Patch File Download Diff File Expand all files Collapse all files

326
docs/12_PERFORMANCE_WORKBOOK.md
View File

@@ -1,170 +1,170 @@
# 12 - Performance Workbook
*Purpose* – define **repeatable, data‑driven** benchmarks that guard Stella Ops’ core pledge:
> *“P95 vulnerability feedback in ≤ 5 seconds.”*
---
## 0 Benchmark Scope
| Area | Included | Excluded |
|------------------|----------------------------------|---------------------------|
| SBOM‑first scan | Trivy engine w/ warmed DB | Full image unpack ≥ 300 MB |
| Delta SBOM ⭑ | Missing‑layer lookup & merge | Multi‑arch images |
| Policy eval ⭑ | YAML → JSON → rule match | Rego (until GA) |
| Feed merge | NVD JSON 2023–2025 | GHSA GraphQL (plugin) |
| Quota wait‑path | 5 s soft‑wait, 60 s hard‑wait behaviour | Paid tiers (unlimited) |
| API latency | REST `/scan`, `/layers/missing` | UI SPA calls |
⭑ = new in July 2025.
---
## 1 Hardware Baseline (Reference Rig)
| Element | Spec |
|-------------|------------------------------------|
| CPU | 8 vCPU (Intel Ice‑Lake equiv.) |
| Memory | 16 GiB |
| Disk | NVMe SSD, 3 GB/s R/W |
| Network | 1 Gbit virt. switch |
| Container | Docker 25.0 + overlay2 |
| OS | Ubuntu 22.04 LTS (kernel 6.8) |
*All P95 targets assume a **single‑node** deployment on this rig unless stated.*
---
## 2 Phase Targets & Gates
| Phase (ID) | Target P95 | Gate (CI) | Rationale |
|-----------------------|-----------:|-----------|----------------------------------------|
| **SBOM_FIRST** | ≤ 5 s | `hard` | Core UX promise. |
| **IMAGE_UNPACK** | ≤ 10 s | `soft` | Fallback path for legacy flows. |
| **DELTA_SBOM** ⭑ | ≤ 1 s | `hard` | Needed to stay sub‑5 s for big bases. |
| **POLICY_EVAL** ⭑ | ≤ 50 ms | `hard` | Keeps gate latency invisible to users. |
| **QUOTA_WAIT** ⭑ | *soft* ≤ 5 s<br>*hard* ≤ 60 s | `hard` | Ensures graceful Free‑tier throttling. |
| **SCHED_RESCAN** | ≤ 30 s | `soft` | Nightly batch – not user‑facing. |
| **FEED_MERGE** | ≤ 60 s | `soft` | Off‑peak cron @ 01:00. |
| **API_P95** | ≤ 200 ms | `hard` | UI snappiness. |
*Gate* legend — `hard`: break CI if regression > 3 × target,
`soft`: raise warning & issue ticket.
---
## 3 Test Harness
# 12 - Performance Workbook
*Purpose* – define **repeatable, data‑driven** benchmarks that guard Stella Ops’ core pledge:
> *“P95 vulnerability feedback in ≤ 5 seconds.”*
---
## 0 Benchmark Scope
| Area | Included | Excluded |
|------------------|----------------------------------|---------------------------|
| SBOM‑first scan | Trivy engine w/ warmed DB | Full image unpack ≥ 300 MB |
| Delta SBOM ⭑ | Missing‑layer lookup & merge | Multi‑arch images |
| Policy eval ⭑ | YAML → JSON → rule match | Rego (until GA) |
| Feed merge | NVD JSON 2023–2025 | GHSA GraphQL (plugin) |
| Quota wait‑path | 5 s soft‑wait, 60 s hard‑wait behaviour | Paid tiers (unlimited) |
| API latency | REST `/scan`, `/layers/missing` | UI SPA calls |
⭑ = new in July 2025.
---
## 1 Hardware Baseline (Reference Rig)
| Element | Spec |
|-------------|------------------------------------|
| CPU | 8 vCPU (Intel Ice‑Lake equiv.) |
| Memory | 16 GiB |
| Disk | NVMe SSD, 3 GB/s R/W |
| Network | 1 Gbit virt. switch |
| Container | Docker 25.0 + overlay2 |
| OS | Ubuntu 22.04 LTS (kernel 6.8) |
*All P95 targets assume a **single‑node** deployment on this rig unless stated.*
---
## 2 Phase Targets & Gates
| Phase (ID) | Target P95 | Gate (CI) | Rationale |
|-----------------------|-----------:|-----------|----------------------------------------|
| **SBOM_FIRST** | ≤ 5 s | `hard` | Core UX promise. |
| **IMAGE_UNPACK** | ≤ 10 s | `soft` | Fallback path for legacy flows. |
| **DELTA_SBOM** ⭑ | ≤ 1 s | `hard` | Needed to stay sub‑5 s for big bases. |
| **POLICY_EVAL** ⭑ | ≤ 50 ms | `hard` | Keeps gate latency invisible to users. |
| **QUOTA_WAIT** ⭑ | *soft* ≤ 5 s<br>*hard* ≤ 60 s | `hard` | Ensures graceful Free‑tier throttling. |
| **SCHED_RESCAN** | ≤ 30 s | `soft` | Nightly batch – not user‑facing. |
| **FEED_MERGE** | ≤ 60 s | `soft` | Off‑peak cron @ 01:00. |
| **API_P95** | ≤ 200 ms | `hard` | UI snappiness. |
*Gate* legend — `hard`: break CI if regression > 3 × target,
`soft`: raise warning & issue ticket.
---
## 3 Test Harness
* **Runner** – `perf/run.sh`, accepts `--phase` and `--samples`.
* **Language analyzers microbench** – `dotnet run --project bench/Scanner.Analyzers/StellaOps.Bench.ScannerAnalyzers/StellaOps.Bench.ScannerAnalyzers.csproj -- --repo-root . --out bench/Scanner.Analyzers/baseline.csv --json out/bench/scanner-analyzers/latest.json --prom out/bench/scanner-analyzers/latest.prom --commit $(git rev-parse HEAD)` produces CSV + JSON + Prometheus gauges for analyzer scenarios. Runs fail if `max_ms` regresses ≥ 20 % against `baseline.csv` or if thresholds are exceeded.
* **Language analyzers microbench** – `dotnet run --project src/StellaOps.Bench/Scanner.Analyzers/StellaOps.Bench.ScannerAnalyzers/StellaOps.Bench.ScannerAnalyzers.csproj -- --repo-root . --out src/StellaOps.Bench/Scanner.Analyzers/baseline.csv --json out/bench/scanner-analyzers/latest.json --prom out/bench/scanner-analyzers/latest.prom --commit $(git rev-parse HEAD)` produces CSV + JSON + Prometheus gauges for analyzer scenarios. Runs fail if `max_ms` regresses ≥ 20 % against `baseline.csv` or if thresholds are exceeded.
* **Metrics** – Prometheus + `jq` extracts; aggregated via `scripts/aggregate.ts`.
* **CI** – GitLab CI job *benchmark* publishes JSON to `bench‑artifacts/`.
* **Visualisation** – Grafana dashboard *Stella‑Perf* (provisioned JSON).
> **Note** – harness mounts `/var/cache/trivy` tmpfs to avoid disk noise.
---
## 4 Current Results (July 2025)
| Phase | Samples | Mean (s) | P95 (s) | Target OK? |
|---------------|--------:|---------:|--------:|-----------:|
| SBOM_FIRST | 100 | 3.7 | 4.9 | ✅ |
| IMAGE_UNPACK | 50 | 6.4 | 9.2 | ✅ |
| **DELTA_SBOM**| 100 | 0.46 | 0.83 | ✅ |
| **POLICY_EVAL** | 1 000 | 0.021 | 0.041 | ✅ |
| **QUOTA_WAIT** | 80 | 4.0* | 4.9* | ✅ |
| SCHED_RESCAN | 10 | 18.3 | 24.9 | ✅ |
| FEED_MERGE | 3 | 38.1 | 41.0 | ✅ |
| API_P95 | 20 000 | 0.087 | 0.143 | ✅ |
*Data files:* `bench-artifacts/2025‑07‑14/phase‑stats.json`.
---
## 5 Δ‑SBOM Micro‑Benchmark Detail
### 5.1 Scenario
1. Base image `python:3.12-slim` already scanned (all layers cached).
2. Application layer (`COPY . /app`) triggers new digest.
3. `Stella CLI` lists **7** layers, backend replies *6 hit*, *1 miss*.
4. Builder scans **only 1 layer** (~9 MiB, 217 files) & uploads delta.
### 5.2 Key Timings
| Step | Time (ms) |
|---------------------|----------:|
| `/layers/missing` | 13 |
| Trivy single layer | 655 |
| Upload delta blob | 88 |
| Backend merge + CVE | 74 |
| **Total wall‑time** | **830 ms** |
---
## 6 Quota Wait‑Path Benchmark Detail
### 6.1 Scenario
1. Free‑tier token reaches **scan #200** – dashboard shows yellow banner.
### 6.2 Key Timings
| Step | Time (ms) |
|------------------------------------|----------:|
| `/quota/check` Redis LUA INCR | 0.8 |
| Soft wait sleep (server) | 5 000 |
| Hard wait sleep (server) | 60 000 |
| End‑to‑end wall‑time (soft‑hit) | 5 003 |
| End‑to‑end wall‑time (hard‑hit) | 60 004 |
---
## 7 Policy Eval Bench
### 7.1 Setup
* Policy YAML: **28** rules, mix severity & package conditions.
* Input: scan result JSON with **1 026** findings.
* Evaluator: custom rules engine (Go structs → map look‑ups).
### 7.2 Latency Histogram
```
0‑10 ms ▇▇▇▇▇▇▇▇▇▇ 38 %
10‑20 ms ▇▇▇▇▇▇▇▇▇▇ 42 %
20‑40 ms ▇▇▇▇▇▇ 17 %
40‑50 ms ▇ 3 %
```
P99 = 48 ms. Meets 50 ms gate.
---
## 8 Trend Snapshot
> **Note** – harness mounts `/var/cache/trivy` tmpfs to avoid disk noise.
---
## 4 Current Results (July 2025)
| Phase | Samples | Mean (s) | P95 (s) | Target OK? |
|---------------|--------:|---------:|--------:|-----------:|
| SBOM_FIRST | 100 | 3.7 | 4.9 | ✅ |
| IMAGE_UNPACK | 50 | 6.4 | 9.2 | ✅ |
| **DELTA_SBOM**| 100 | 0.46 | 0.83 | ✅ |
| **POLICY_EVAL** | 1 000 | 0.021 | 0.041 | ✅ |
| **QUOTA_WAIT** | 80 | 4.0* | 4.9* | ✅ |
| SCHED_RESCAN | 10 | 18.3 | 24.9 | ✅ |
| FEED_MERGE | 3 | 38.1 | 41.0 | ✅ |
| API_P95 | 20 000 | 0.087 | 0.143 | ✅ |
*Data files:* `bench-artifacts/2025‑07‑14/phase‑stats.json`.
---
## 5 Δ‑SBOM Micro‑Benchmark Detail
### 5.1 Scenario
1. Base image `python:3.12-slim` already scanned (all layers cached).
2. Application layer (`COPY . /app`) triggers new digest.
3. `Stella CLI` lists **7** layers, backend replies *6 hit*, *1 miss*.
4. Builder scans **only 1 layer** (~9 MiB, 217 files) & uploads delta.
### 5.2 Key Timings
| Step | Time (ms) |
|---------------------|----------:|
| `/layers/missing` | 13 |
| Trivy single layer | 655 |
| Upload delta blob | 88 |
| Backend merge + CVE | 74 |
| **Total wall‑time** | **830 ms** |
---
## 6 Quota Wait‑Path Benchmark Detail
### 6.1 Scenario
1. Free‑tier token reaches **scan #200** – dashboard shows yellow banner.
### 6.2 Key Timings
| Step | Time (ms) |
|------------------------------------|----------:|
| `/quota/check` Redis LUA INCR | 0.8 |
| Soft wait sleep (server) | 5 000 |
| Hard wait sleep (server) | 60 000 |
| End‑to‑end wall‑time (soft‑hit) | 5 003 |
| End‑to‑end wall‑time (hard‑hit) | 60 004 |
---
## 7 Policy Eval Bench
### 7.1 Setup
* Policy YAML: **28** rules, mix severity & package conditions.
* Input: scan result JSON with **1 026** findings.
* Evaluator: custom rules engine (Go structs → map look‑ups).
### 7.2 Latency Histogram
```
0‑10 ms ▇▇▇▇▇▇▇▇▇▇ 38 %
10‑20 ms ▇▇▇▇▇▇▇▇▇▇ 42 %
20‑40 ms ▇▇▇▇▇▇ 17 %
40‑50 ms ▇ 3 %
```
P99 = 48 ms. Meets 50 ms gate.
---
## 8 Trend Snapshot
![Perf trend spark‑line placeholder](perf‑trend.svg)
> **Grafana/Alerting** – Import `docs/ops/scanner-analyzers-grafana-dashboard.json` and point it at the Prometheus datasource storing `scanner_analyzer_bench_*` metrics. Configure an alert on `scanner_analyzer_bench_regression_ratio` ≥ 1.20 (default limit); the bundled Stat panel surfaces breached scenarios (non-zero values). On-call runbook: `docs/ops/scanner-analyzers-operations.md`.
_Plot generated weekly by `scripts/update‑trend.py`; shows last 12 weeks P95 per phase._
---
## 9 Action Items
1. **Image Unpack** – Evaluate zstd for layer decompress; aim to shave 1 s.
2. **Feed Merge** – Parallelise regional XML feed parse (plugin) once stable.
3. **Rego Support** – Prototype OPA side‑car; target ≤ 100 ms eval.
4. **Concurrency** – Stress‑test 100 rps on 4‑node Redis cluster (Q4‑2025).
---
## 10 Change Log
| Date | Note |
|------------|-------------------------------------------------------------------------|
| 2025‑07‑14 | Added Δ‑SBOM & Policy Eval phases; updated targets & current results. |
| 2025‑07‑12 | First public workbook (SBOM‑first, image‑unpack, feed merge). |
---
_Plot generated weekly by `scripts/update‑trend.py`; shows last 12 weeks P95 per phase._
---
## 9 Action Items
1. **Image Unpack** – Evaluate zstd for layer decompress; aim to shave 1 s.
2. **Feed Merge** – Parallelise regional XML feed parse (plugin) once stable.
3. **Rego Support** – Prototype OPA side‑car; target ≤ 100 ms eval.
4. **Concurrency** – Stress‑test 100 rps on 4‑node Redis cluster (Q4‑2025).
---
## 10 Change Log
| Date | Note |
|------------|-------------------------------------------------------------------------|
| 2025‑07‑14 | Added Δ‑SBOM & Policy Eval phases; updated targets & current results. |
| 2025‑07‑12 | First public workbook (SBOM‑first, image‑unpack, feed merge). |
---