feat(metrics): Implement scan metrics repository and PostgreSQL integration
Some checks failed
Docs CI / lint-and-preview (push) Has been cancelled

- Added IScanMetricsRepository interface for scan metrics persistence and retrieval.
- Implemented PostgresScanMetricsRepository for PostgreSQL database interactions, including methods for saving and retrieving scan metrics and execution phases.
- Introduced methods for obtaining TTE statistics and recent scans for tenants.
- Implemented deletion of old metrics for retention purposes.

test(tests): Add SCA Failure Catalogue tests for FC6-FC10

- Created ScaCatalogueDeterminismTests to validate determinism properties of SCA Failure Catalogue fixtures.
- Developed ScaFailureCatalogueTests to ensure correct handling of specific failure modes in the scanner.
- Included tests for manifest validation, file existence, and expected findings across multiple failure cases.

feat(telemetry): Integrate scan completion metrics into the pipeline

- Introduced IScanCompletionMetricsIntegration interface and ScanCompletionMetricsIntegration class to record metrics upon scan completion.
- Implemented proof coverage and TTE metrics recording with logging for scan completion summaries.
This commit is contained in:
master
2025-12-16 14:00:35 +02:00
parent b55d9fa68d
commit 415eff1207
27 changed files with 3620 additions and 35 deletions

View File

@@ -334,6 +334,50 @@ cmd.Parameters.AddWithValue("config", json);
var json = Newtonsoft.Json.JsonConvert.SerializeObject(obj);
```
### 5.3.1 Generated Columns for JSONB Hot Keys
**RULE:** Frequently-queried JSONB fields (>10% of queries) SHOULD be extracted as generated columns.
**When to use generated columns:**
- Field is used in WHERE clauses frequently
- Field is used in JOIN conditions
- Field is used in GROUP BY or ORDER BY
- Query planner needs cardinality statistics
```sql
-- ✓ CORRECT: Generated column for hot JSONB field
ALTER TABLE scheduler.runs
ADD COLUMN finding_count INT GENERATED ALWAYS AS ((stats->>'findingCount')::int) STORED;
CREATE INDEX idx_runs_finding_count ON scheduler.runs(tenant_id, finding_count);
```
**RULE:** Generated column names MUST follow snake_case convention matching the JSON path.
```sql
-- ✓ CORRECT naming
doc->>'bomFormat' bom_format
stats->>'findingCount' finding_count
raw->>'schemaVersion' schema_version
-- ✗ INCORRECT naming
doc->>'bomFormat' bomFormat, format, bf
```
**RULE:** Generated columns MUST be added with concurrent index creation in production.
```sql
-- ✓ CORRECT: Non-blocking migration
ALTER TABLE scheduler.runs ADD COLUMN finding_count INT GENERATED ALWAYS AS (...) STORED;
CREATE INDEX CONCURRENTLY idx_runs_finding_count ON scheduler.runs(finding_count);
ANALYZE scheduler.runs;
-- ✗ INCORRECT: Blocking migration
CREATE INDEX idx_runs_finding_count ON scheduler.runs(finding_count); -- Blocks table
```
**Reference:** See `SPECIFICATION.md` Section 6.4 for detailed guidelines.
### 5.4 Null Handling
**RULE:** Nullable values MUST use `DBNull.Value` when null.

View File

@@ -1173,6 +1173,67 @@ CREATE INDEX idx_metadata_active ON scheduler.runs USING GIN (stats)
WHERE state = 'completed';
```
### 6.4 Generated Columns for JSONB Hot Keys
For frequently-queried JSONB fields, use PostgreSQL generated columns to enable efficient B-tree indexing and query planning statistics.
**Problem with expression indexes:**
```sql
-- Expression indexes don't collect statistics
CREATE INDEX idx_format ON sbom_docs ((doc->>'bomFormat'));
-- Query planner can't estimate cardinality, may choose suboptimal plans
```
**Solution: Generated columns (PostgreSQL 12+):**
```sql
-- Add generated column that extracts JSONB field
ALTER TABLE scanner.sbom_documents
ADD COLUMN bom_format TEXT GENERATED ALWAYS AS ((doc->>'bomFormat')) STORED;
-- Standard B-tree index with full statistics
CREATE INDEX idx_sbom_bom_format ON scanner.sbom_documents(bom_format);
```
**Benefits:**
- **B-tree indexable**: Standard index on generated column
- **Statistics**: `ANALYZE` collects cardinality, MCV, histogram
- **Index-only scans**: Visible to covering indexes
- **Zero application changes**: Transparent to ORM/queries
**When to use generated columns:**
- Field queried in >10% of queries against the table
- Cardinality >100 distinct values (worth collecting stats)
- Field used in JOIN conditions or GROUP BY
- Index-only scans are beneficial
**Naming convention:**
```
<json_path_snake_case>
Examples:
doc->>'bomFormat' → bom_format
raw->>'schemaVersion' → schema_version
stats->>'findingCount'→ finding_count
```
**Migration pattern:**
```sql
-- Step 1: Add generated column (no lock on existing rows)
ALTER TABLE scheduler.runs
ADD COLUMN finding_count INT GENERATED ALWAYS AS ((stats->>'findingCount')::int) STORED;
-- Step 2: Create index concurrently
CREATE INDEX CONCURRENTLY idx_runs_finding_count
ON scheduler.runs(tenant_id, finding_count);
-- Step 3: Analyze for statistics
ANALYZE scheduler.runs;
```
**Reference implementations:**
- `src/Scheduler/...Storage.Postgres/Migrations/010_generated_columns_runs.sql`
- `src/Excititor/...Storage.Postgres/Migrations/004_generated_columns_vex.sql`
- `src/Concelier/...Storage.Postgres/Migrations/007_generated_columns_advisories.sql`
---
## 7. Partitioning Strategy

View File

@@ -0,0 +1,195 @@
# Scan Metrics Schema
Sprint: `SPRINT_3406_0001_0001_metrics_tables`
Task: `METRICS-3406-013`
Working Directory: `src/Scanner/__Libraries/StellaOps.Scanner.Storage/Postgres/Migrations/`
## Overview
The scan metrics schema provides relational PostgreSQL tables for tracking Time-to-Evidence (TTE) and scan performance metrics. This is a hybrid approach where metrics are stored in PostgreSQL while replay manifests remain in the document store.
## Tables
### `scanner.scan_metrics`
Primary table for per-scan metrics.
| Column | Type | Description |
|--------|------|-------------|
| `metrics_id` | UUID | Primary key |
| `scan_id` | UUID | Unique scan identifier |
| `tenant_id` | UUID | Tenant identifier |
| `surface_id` | UUID | Optional attack surface identifier |
| `artifact_digest` | TEXT | Artifact content hash |
| `artifact_type` | TEXT | Type: `oci_image`, `tarball`, `directory`, `other` |
| `replay_manifest_hash` | TEXT | Reference to replay manifest in document store |
| `findings_sha256` | TEXT | Findings content hash |
| `vex_bundle_sha256` | TEXT | VEX bundle content hash |
| `proof_bundle_sha256` | TEXT | Proof bundle content hash |
| `sbom_sha256` | TEXT | SBOM content hash |
| `policy_digest` | TEXT | Policy version hash |
| `feed_snapshot_id` | TEXT | Feed snapshot identifier |
| `started_at` | TIMESTAMPTZ | Scan start time |
| `finished_at` | TIMESTAMPTZ | Scan completion time |
| `total_duration_ms` | INT | TTE in milliseconds (generated) |
| `t_ingest_ms` | INT | Ingest phase duration |
| `t_analyze_ms` | INT | Analyze phase duration |
| `t_reachability_ms` | INT | Reachability phase duration |
| `t_vex_ms` | INT | VEX phase duration |
| `t_sign_ms` | INT | Sign phase duration |
| `t_publish_ms` | INT | Publish phase duration |
| `package_count` | INT | Number of packages analyzed |
| `finding_count` | INT | Number of findings |
| `vex_decision_count` | INT | Number of VEX decisions |
| `scanner_version` | TEXT | Scanner version |
| `scanner_image_digest` | TEXT | Scanner container digest |
| `is_replay` | BOOLEAN | Replay mode flag |
| `created_at` | TIMESTAMPTZ | Record creation time |
### `scanner.execution_phases`
Detailed phase execution tracking.
| Column | Type | Description |
|--------|------|-------------|
| `id` | BIGSERIAL | Primary key |
| `metrics_id` | UUID | Foreign key to `scan_metrics` |
| `phase_name` | TEXT | Phase: `ingest`, `analyze`, `reachability`, `vex`, `sign`, `publish`, `other` |
| `phase_order` | INT | Execution order |
| `started_at` | TIMESTAMPTZ | Phase start time |
| `finished_at` | TIMESTAMPTZ | Phase completion time |
| `duration_ms` | INT | Duration in milliseconds (generated) |
| `success` | BOOLEAN | Phase success status |
| `error_code` | TEXT | Error code if failed |
| `error_message` | TEXT | Error message if failed |
| `phase_metrics` | JSONB | Phase-specific metrics |
## Views
### `scanner.scan_tte`
Time-to-Evidence view with phase breakdowns.
```sql
SELECT
metrics_id,
scan_id,
tte_ms,
tte_seconds,
ingest_percent,
analyze_percent,
reachability_percent,
vex_percent,
sign_percent,
publish_percent
FROM scanner.scan_tte
WHERE tenant_id = :tenant_id;
```
### `scanner.tte_stats`
Hourly TTE statistics with SLO compliance.
```sql
SELECT
hour_bucket,
scan_count,
tte_avg_ms,
tte_p50_ms,
tte_p95_ms,
slo_p50_compliance_percent,
slo_p95_compliance_percent
FROM scanner.tte_stats
WHERE tenant_id = :tenant_id;
```
## Functions
### `scanner.tte_percentile`
Calculate TTE percentile for a tenant.
```sql
SELECT scanner.tte_percentile(
p_tenant_id := :tenant_id,
p_percentile := 0.95,
p_since := NOW() - INTERVAL '7 days'
);
```
## Indexes
| Index | Columns | Purpose |
|-------|---------|---------|
| `idx_scan_metrics_tenant` | `tenant_id` | Tenant queries |
| `idx_scan_metrics_artifact` | `artifact_digest` | Artifact lookups |
| `idx_scan_metrics_started` | `started_at` | Time-range queries |
| `idx_scan_metrics_surface` | `surface_id` | Surface queries |
| `idx_scan_metrics_replay` | `is_replay` | Filter replays |
| `idx_scan_metrics_tenant_started` | `tenant_id, started_at` | Compound tenant+time |
| `idx_execution_phases_metrics` | `metrics_id` | Phase lookups |
| `idx_execution_phases_name` | `phase_name` | Phase filtering |
## SLO Thresholds
Per the advisory section 13.1:
| Metric | Target |
|--------|--------|
| TTE P50 | < 120 seconds |
| TTE P95 | < 300 seconds |
## Usage Examples
### Get TTE for recent scans
```sql
SELECT scan_id, tte_ms, tte_seconds
FROM scanner.scan_tte
WHERE tenant_id = :tenant_id
AND NOT is_replay
ORDER BY started_at DESC
LIMIT 100;
```
### Check SLO compliance
```sql
SELECT
hour_bucket,
slo_p50_compliance_percent,
slo_p95_compliance_percent
FROM scanner.tte_stats
WHERE tenant_id = :tenant_id
AND hour_bucket >= NOW() - INTERVAL '24 hours';
```
### Phase breakdown analysis
```sql
SELECT
phase_name,
AVG(duration_ms) as avg_ms,
PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY duration_ms) as p95_ms
FROM scanner.execution_phases ep
JOIN scanner.scan_metrics sm ON ep.metrics_id = sm.metrics_id
WHERE sm.tenant_id = :tenant_id
AND sm.started_at >= NOW() - INTERVAL '7 days'
GROUP BY phase_name
ORDER BY phase_order;
```
## Migration
Migration file: `src/Scanner/__Libraries/StellaOps.Scanner.Storage/Postgres/Migrations/004_scan_metrics.sql`
Apply with:
```bash
psql -d stellaops -f 004_scan_metrics.sql
```
## Related
- [Database Specification](./SPECIFICATION.md)
- [Determinism Advisory §13.1](../product-advisories/14-Dec-2025%20-%20Determinism%20and%20Reproducibility%20Technical%20Reference.md)
- [Scheduler Schema](./schemas/scheduler.sql)

View File

@@ -42,14 +42,14 @@ Read before implementation:
|---|---------|--------|---------------------------|--------|-----------------|
| 1 | QGATE-0350-001 | DONE | None | Platform | Create `scripts/ci/compute-reachability-metrics.sh` to compute recall/precision from corpus |
| 2 | QGATE-0350-002 | DONE | After #1 | Platform | Create `scripts/ci/reachability-thresholds.yaml` with enforcement thresholds |
| 3 | QGATE-0350-003 | TODO | After #2 | Platform | Add reachability gate job to `build-test-deploy.yml` |
| 3 | QGATE-0350-003 | DONE | After #2 | Platform | Add reachability gate job to `build-test-deploy.yml` |
| 4 | QGATE-0350-004 | DONE | None | Platform | Create `scripts/ci/compute-ttfs-metrics.sh` to extract TTFS from test runs |
| 5 | QGATE-0350-005 | DONE | After #4 | Platform | Create `bench/baselines/ttfs-baseline.json` with p50/p95 targets |
| 6 | QGATE-0350-006 | TODO | After #5 | Platform | Add TTFS regression gate to `build-test-deploy.yml` |
| 6 | QGATE-0350-006 | DONE | After #5 | Platform | Add TTFS regression gate to `build-test-deploy.yml` |
| 7 | QGATE-0350-007 | DONE | None | Platform | Create `scripts/ci/enforce-performance-slos.sh` for scan/compute SLOs |
| 8 | QGATE-0350-008 | TODO | After #7 | Platform | Add performance SLO gate to `build-test-deploy.yml` |
| 9 | QGATE-0350-009 | TODO | After #3, #6, #8 | Platform | Create `docs/testing/ci-quality-gates.md` documentation |
| 10 | QGATE-0350-010 | TODO | After #9 | Platform | Add quality gate status badges to repository README |
| 8 | QGATE-0350-008 | DONE | After #7 | Platform | Add performance SLO gate to `build-test-deploy.yml` |
| 9 | QGATE-0350-009 | DONE | After #3, #6, #8 | Platform | Create `docs/testing/ci-quality-gates.md` documentation |
| 10 | QGATE-0350-010 | DONE | After #9 | Platform | Add quality gate status badges to repository README |
## Wave Coordination

View File

@@ -68,9 +68,9 @@ The SCA Failure Catalogue covers real-world scanner failure modes that have occu
| 5 | SCA-0351-005 | DONE | None | Scanner | Create FC10 fixture: CVE Split/Merge failure case |
| 6 | SCA-0351-006 | DONE | After #1-5 | Scanner | Create DSSE manifests for all new fixtures |
| 7 | SCA-0351-007 | DONE | After #6 | Scanner | Update `tests/fixtures/sca/catalogue/inputs.lock` |
| 8 | SCA-0351-008 | TODO | After #7 | Scanner | Add xUnit tests for FC6-FC10 in Scanner test project |
| 8 | SCA-0351-008 | DONE | After #7 | Scanner | Add xUnit tests for FC6-FC10 in Scanner test project |
| 9 | SCA-0351-009 | DONE | After #8 | Scanner | Update `tests/fixtures/sca/catalogue/README.md` documentation |
| 10 | SCA-0351-010 | TODO | After #9 | Scanner | Validate all fixtures pass determinism checks |
| 10 | SCA-0351-010 | DONE | After #9 | Scanner | Validate all fixtures pass determinism checks |
## Wave Coordination

View File

@@ -379,20 +379,20 @@ public interface ISubjectExtractor
| # | Task ID | Status | Key Dependency / Next Step | Owners | Task Definition |
|---|---------|--------|---------------------------|--------|-----------------|
| 1 | PROOF-ID-0001 | DOING | None | Attestor Guild | Create `StellaOps.Attestor.ProofChain` library project structure |
| 2 | PROOF-ID-0002 | DOING | Task 1 | Attestor Guild | Implement `ContentAddressedId` base record and derived types |
| 3 | PROOF-ID-0003 | DOING | Task 1 | Attestor Guild | Implement `IJsonCanonicalizer` per RFC 8785 |
| 4 | PROOF-ID-0004 | DOING | Task 3 | Attestor Guild | Implement `IContentAddressedIdGenerator` for EvidenceID |
| 5 | PROOF-ID-0005 | DOING | Task 3 | Attestor Guild | Implement `IContentAddressedIdGenerator` for ReasoningID |
| 6 | PROOF-ID-0006 | DOING | Task 3 | Attestor Guild | Implement `IContentAddressedIdGenerator` for VEXVerdictID |
| 7 | PROOF-ID-0007 | DOING | Task 1 | Attestor Guild | Implement `IMerkleTreeBuilder` for deterministic merkle construction |
| 8 | PROOF-ID-0008 | DOING | Task 4-7 | Attestor Guild | Implement `IContentAddressedIdGenerator` for ProofBundleID |
| 9 | PROOF-ID-0009 | DOING | Task 7 | Attestor Guild | Implement `IContentAddressedIdGenerator` for GraphRevisionID |
| 10 | PROOF-ID-0010 | DOING | Task 3 | Attestor Guild | Implement `SbomEntryId` computation from SBOM + PURL |
| 11 | PROOF-ID-0011 | DOING | Task 1 | Attestor Guild | Implement `ISubjectExtractor` for CycloneDX SBOMs |
| 12 | PROOF-ID-0012 | DOING | Task 1 | Attestor Guild | Create all predicate record types (Evidence, Reasoning, VEX, ProofSpine) |
| 13 | PROOF-ID-0013 | TODO | Task 2-12 | QA Guild | Unit tests for all ID generation (determinism verification) |
| 14 | PROOF-ID-0014 | TODO | Task 13 | QA Guild | Property-based tests for canonicalization stability |
| 1 | PROOF-ID-0001 | DONE | None | Attestor Guild | Create `StellaOps.Attestor.ProofChain` library project structure |
| 2 | PROOF-ID-0002 | DONE | Task 1 | Attestor Guild | Implement `ContentAddressedId` base record and derived types |
| 3 | PROOF-ID-0003 | DONE | Task 1 | Attestor Guild | Implement `IJsonCanonicalizer` per RFC 8785 |
| 4 | PROOF-ID-0004 | DONE | Task 3 | Attestor Guild | Implement `IContentAddressedIdGenerator` for EvidenceID |
| 5 | PROOF-ID-0005 | DONE | Task 3 | Attestor Guild | Implement `IContentAddressedIdGenerator` for ReasoningID |
| 6 | PROOF-ID-0006 | DONE | Task 3 | Attestor Guild | Implement `IContentAddressedIdGenerator` for VEXVerdictID |
| 7 | PROOF-ID-0007 | DONE | Task 1 | Attestor Guild | Implement `IMerkleTreeBuilder` for deterministic merkle construction |
| 8 | PROOF-ID-0008 | DONE | Task 4-7 | Attestor Guild | Implement `IContentAddressedIdGenerator` for ProofBundleID |
| 9 | PROOF-ID-0009 | DONE | Task 7 | Attestor Guild | Implement `IContentAddressedIdGenerator` for GraphRevisionID |
| 10 | PROOF-ID-0010 | DONE | Task 3 | Attestor Guild | Implement `SbomEntryId` computation from SBOM + PURL |
| 11 | PROOF-ID-0011 | DONE | Task 1 | Attestor Guild | Implement `ISubjectExtractor` for CycloneDX SBOMs |
| 12 | PROOF-ID-0012 | DONE | Task 1 | Attestor Guild | Create all predicate record types (Evidence, Reasoning, VEX, ProofSpine) |
| 13 | PROOF-ID-0013 | DONE | Task 2-12 | QA Guild | Unit tests for all ID generation (determinism verification) |
| 14 | PROOF-ID-0014 | DONE | Task 13 | QA Guild | Property-based tests for canonicalization stability |
| 15 | PROOF-ID-0015 | TODO | Task 13 | Docs Guild | Document ID format specifications in module architecture |
## Test Specifications

View File

@@ -33,17 +33,17 @@ Implement high-value, low-effort scoring enhancements from the Determinism and R
|---|---------|--------|---------------------------|--------|-----------------|
| 1 | DET-3401-001 | DONE | None | Scoring Team | Define `FreshnessBucket` record and `FreshnessMultiplierConfig` in Policy.Scoring |
| 2 | DET-3401-002 | DONE | After #1 | Scoring Team | Implement `EvidenceFreshnessCalculator` service with basis-points multipliers |
| 3 | DET-3401-003 | TODO | After #2 | Scoring Team | Integrate freshness multiplier into existing evidence scoring pipeline |
| 3 | DET-3401-003 | DONE | After #2 | Scoring Team | Integrate freshness multiplier into existing evidence scoring pipeline |
| 4 | DET-3401-004 | DONE | After #3 | Scoring Team | Add unit tests for freshness buckets (7d, 30d, 90d, 180d, 365d, >365d) |
| 5 | DET-3401-005 | DONE | None | Telemetry Team | Define `ProofCoverageMetrics` class with Prometheus counters/gauges |
| 6 | DET-3401-006 | DONE | After #5 | Telemetry Team | Implement `proof_coverage_all`, `proof_coverage_vex`, `proof_coverage_reachable` gauges |
| 7 | DET-3401-007 | TODO | After #6 | Telemetry Team | Add proof coverage calculation to scan completion pipeline |
| 7 | DET-3401-007 | DONE | After #6 | Telemetry Team | Add proof coverage calculation to scan completion pipeline |
| 8 | DET-3401-008 | DONE | After #7 | Telemetry Team | Add unit tests for proof coverage ratio calculations |
| 9 | DET-3401-009 | DONE | None | Scoring Team | Define `ScoreExplanation` record with factor/value/reason structure |
| 10 | DET-3401-010 | DONE | After #9 | Scoring Team | Implement `ScoreExplainBuilder` to accumulate explanations during scoring |
| 11 | DET-3401-011 | DONE | After #10 | Scoring Team | Refactor `RiskScoringResult` to include `Explain` array |
| 12 | DET-3401-012 | DONE | After #11 | Scoring Team | Add unit tests for explanation generation |
| 13 | DET-3401-013 | TODO | After #4, #8, #12 | QA | Integration tests: freshness + proof coverage + explain in full scan |
| 13 | DET-3401-013 | DONE | After #4, #8, #12 | QA | Integration tests: freshness + proof coverage + explain in full scan |
## Wave Coordination

View File

@@ -30,22 +30,23 @@ Implement relational PostgreSQL tables for scan metrics tracking (hybrid approac
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
|---|---------|--------|---------------------------|--------|-----------------|
| 1 | METRICS-3406-001 | TODO | None | DB Team | Create `scan_metrics` table migration |
| 2 | METRICS-3406-002 | TODO | After #1 | DB Team | Create `execution_phases` table for timing breakdown |
| 3 | METRICS-3406-003 | TODO | After #1 | DB Team | Create `scan_tte` view for TTE calculation |
| 4 | METRICS-3406-004 | TODO | After #1 | DB Team | Create indexes for metrics queries |
| 5 | METRICS-3406-005 | TODO | None | Scanner Team | Define `ScanMetrics` entity and `ExecutionPhase` record |
| 6 | METRICS-3406-006 | TODO | After #1, #5 | Scanner Team | Implement `IScanMetricsRepository` interface |
| 7 | METRICS-3406-007 | TODO | After #6 | Scanner Team | Implement `PostgresScanMetricsRepository` |
| 8 | METRICS-3406-008 | TODO | After #7 | Scanner Team | Implement `ScanMetricsCollector` service |
| 1 | METRICS-3406-001 | DONE | None | DB Team | Create `scan_metrics` table migration |
| 2 | METRICS-3406-002 | DONE | After #1 | DB Team | Create `execution_phases` table for timing breakdown |
| 3 | METRICS-3406-003 | DONE | After #1 | DB Team | Create `scan_tte` view for TTE calculation |
| 4 | METRICS-3406-004 | DONE | After #1 | DB Team | Create indexes for metrics queries |
| 5 | METRICS-3406-005 | DONE | None | Scanner Team | Define `ScanMetrics` entity and `ExecutionPhase` record |
| 6 | METRICS-3406-006 | DONE | After #1, #5 | Scanner Team | Implement `IScanMetricsRepository` interface |
| 7 | METRICS-3406-007 | DONE | After #6 | Scanner Team | Implement `PostgresScanMetricsRepository` |
| 8 | METRICS-3406-008 | DONE | After #7 | Scanner Team | Implement `ScanMetricsCollector` service |
| 9 | METRICS-3406-009 | TODO | After #8 | Scanner Team | Integrate collector into scan completion pipeline |
| 10 | METRICS-3406-010 | TODO | After #3 | Telemetry Team | Export TTE percentiles to Prometheus |
| 11 | METRICS-3406-011 | TODO | After #7 | Scanner Team | Unit tests for repository operations |
| 12 | METRICS-3406-012 | TODO | After #9 | QA | Integration test: metrics captured on scan completion |
| 13 | METRICS-3406-013 | TODO | After #3 | Docs Guild | Document metrics schema in `docs/db/schemas/scan-metrics.md` |
| 13 | METRICS-3406-013 | DONE | After #3 | Docs Guild | Document metrics schema in `docs/db/schemas/scan-metrics.md` |
## Wave Coordination
- **Wave 1** (Parallel): Tasks #1-5 (Schema + Models)
- **Wave 2** (Sequential): Tasks #6-9 (Repository + Collector + Integration)
- **Wave 3** (Parallel): Tasks #10-13 (Telemetry + Tests + Docs)

View File

@@ -75,8 +75,8 @@ Benefits:
| 4.6 | Verify query plans | DONE | | |
| 4.7 | Integration tests | DONE | | Via runbook validation |
| **Phase 5: Documentation** |||||
| 5.1 | Update SPECIFICATION.md with generated column pattern | TODO | | |
| 5.2 | Add generated column guidelines to RULES.md | TODO | | |
| 5.1 | Update SPECIFICATION.md with generated column pattern | DONE | | Added Section 6.4 |
| 5.2 | Add generated column guidelines to RULES.md | DONE | | Added Section 5.3.1 |
| 5.3 | Document query optimization gains | DONE | | postgresql-patterns-runbook.md |
---

View File

@@ -0,0 +1,155 @@
# CI Quality Gates
Sprint: `SPRINT_0350_0001_0001_ci_quality_gates_foundation`
Task: `QGATE-0350-009`
## Overview
StellaOps implements automated quality gates in CI to enforce:
- **Reachability Quality** - Recall/precision thresholds for vulnerability detection
- **TTFS Regression** - Time-to-First-Signal performance tracking
- **Performance SLOs** - Scan time and compute budget enforcement
These gates run as part of the `build-test-deploy.yml` workflow after the main test suite completes.
## Quality Gate Jobs
### Reachability Quality Gate
**Script:** `scripts/ci/compute-reachability-metrics.sh`
**Config:** `scripts/ci/reachability-thresholds.yaml`
Validates that the scanner meets recall/precision thresholds against the ground-truth corpus.
#### Metrics Computed
| Metric | Description | Threshold |
|--------|-------------|-----------|
| `runtime_dependency_recall` | % of runtime dep vulns detected | ≥ 95% |
| `unreachable_false_positives` | FP rate for unreachable findings | ≤ 5% |
| `reachability_underreport` | Underreporting rate | ≤ 10% |
| `os_package_recall` | % of OS package vulns detected | ≥ 92% |
| `code_vuln_recall` | % of code vulns detected | ≥ 88% |
| `config_vuln_recall` | % of config vulns detected | ≥ 85% |
#### Running Locally
```bash
# Dry run (no enforcement)
./scripts/ci/compute-reachability-metrics.sh --dry-run
# Full run against corpus
./scripts/ci/compute-reachability-metrics.sh
```
### TTFS Regression Gate
**Script:** `scripts/ci/compute-ttfs-metrics.sh`
**Baseline:** `bench/baselines/ttfs-baseline.json`
Detects performance regressions in Time-to-First-Signal.
#### Metrics Computed
| Metric | Description | Threshold |
|--------|-------------|-----------|
| `ttfs_p50_ms` | P50 time to first signal | ≤ baseline + 10% |
| `ttfs_p95_ms` | P95 time to first signal | ≤ baseline + 15% |
| `ttfs_max_ms` | Maximum TTFS | ≤ baseline + 25% |
#### Baseline Format
```json
{
"ttfs_p50_ms": 450,
"ttfs_p95_ms": 1200,
"ttfs_max_ms": 3000,
"measured_at": "2025-12-16T00:00:00Z",
"sample_count": 1000
}
```
### Performance SLO Gate
**Script:** `scripts/ci/enforce-performance-slos.sh`
**Config:** `scripts/ci/performance-slos.yaml`
Enforces scan time and compute budget SLOs.
#### SLOs Enforced
| SLO | Description | Target |
|-----|-------------|--------|
| `scan_time_p50_ms` | P50 scan time | ≤ 120,000ms (2 min) |
| `scan_time_p95_ms` | P95 scan time | ≤ 300,000ms (5 min) |
| `memory_peak_mb` | Peak memory usage | ≤ 2048 MB |
| `cpu_seconds` | Total CPU time | ≤ 120 seconds |
## Workflow Integration
Quality gates are integrated into the main CI workflow:
```yaml
# .gitea/workflows/build-test-deploy.yml
quality-gates:
runs-on: ubuntu-22.04
needs: build-test
steps:
- name: Reachability quality gate
run: ./scripts/ci/compute-reachability-metrics.sh
- name: TTFS regression gate
run: ./scripts/ci/compute-ttfs-metrics.sh
- name: Performance SLO gate
run: ./scripts/ci/enforce-performance-slos.sh --warn-only
```
## Failure Modes
### Hard Failure (Blocks Merge)
- Reachability recall below threshold
- TTFS regression exceeds 25%
- Memory budget exceeded by 50%
### Soft Failure (Warning Only)
- Minor TTFS regression (< 15%)
- Memory near budget limit
- Missing baseline data (new fixtures)
## Adding New Quality Gates
1. Create computation script in `scripts/ci/`
2. Add threshold configuration (YAML or JSON)
3. Integrate into workflow as a new step
4. Update this documentation
5. Add to sprint tracking
## Troubleshooting
### Gate Fails on PR but Passes on Main
Check for:
- Non-deterministic test execution
- Timing-sensitive assertions
- Missing test fixtures in PR branch
### Baseline Drift
If baselines become stale:
```bash
# Regenerate baselines
./scripts/ci/compute-ttfs-metrics.sh --update-baseline
./scripts/ci/compute-reachability-metrics.sh --update-baseline
```
## Related Documentation
- [Test Suite Overview](../19_TEST_SUITE_OVERVIEW.md)
- [Reachability Corpus Plan](../reachability/corpus-plan.md)
- [Performance Workbook](../12_PERFORMANCE_WORKBOOK.md)
- [Testing Quality Guardrails](./testing-quality-guardrails-implementation.md)