feat(metrics): Implement scan metrics repository and PostgreSQL integration
Some checks failed
Docs CI / lint-and-preview (push) Has been cancelled
Some checks failed
Docs CI / lint-and-preview (push) Has been cancelled
- Added IScanMetricsRepository interface for scan metrics persistence and retrieval. - Implemented PostgresScanMetricsRepository for PostgreSQL database interactions, including methods for saving and retrieving scan metrics and execution phases. - Introduced methods for obtaining TTE statistics and recent scans for tenants. - Implemented deletion of old metrics for retention purposes. test(tests): Add SCA Failure Catalogue tests for FC6-FC10 - Created ScaCatalogueDeterminismTests to validate determinism properties of SCA Failure Catalogue fixtures. - Developed ScaFailureCatalogueTests to ensure correct handling of specific failure modes in the scanner. - Included tests for manifest validation, file existence, and expected findings across multiple failure cases. feat(telemetry): Integrate scan completion metrics into the pipeline - Introduced IScanCompletionMetricsIntegration interface and ScanCompletionMetricsIntegration class to record metrics upon scan completion. - Implemented proof coverage and TTE metrics recording with logging for scan completion summaries.
This commit is contained in:
@@ -334,6 +334,50 @@ cmd.Parameters.AddWithValue("config", json);
|
||||
var json = Newtonsoft.Json.JsonConvert.SerializeObject(obj);
|
||||
```
|
||||
|
||||
### 5.3.1 Generated Columns for JSONB Hot Keys
|
||||
|
||||
**RULE:** Frequently-queried JSONB fields (>10% of queries) SHOULD be extracted as generated columns.
|
||||
|
||||
**When to use generated columns:**
|
||||
- Field is used in WHERE clauses frequently
|
||||
- Field is used in JOIN conditions
|
||||
- Field is used in GROUP BY or ORDER BY
|
||||
- Query planner needs cardinality statistics
|
||||
|
||||
```sql
|
||||
-- ✓ CORRECT: Generated column for hot JSONB field
|
||||
ALTER TABLE scheduler.runs
|
||||
ADD COLUMN finding_count INT GENERATED ALWAYS AS ((stats->>'findingCount')::int) STORED;
|
||||
|
||||
CREATE INDEX idx_runs_finding_count ON scheduler.runs(tenant_id, finding_count);
|
||||
```
|
||||
|
||||
**RULE:** Generated column names MUST follow snake_case convention matching the JSON path.
|
||||
|
||||
```sql
|
||||
-- ✓ CORRECT naming
|
||||
doc->>'bomFormat' → bom_format
|
||||
stats->>'findingCount' → finding_count
|
||||
raw->>'schemaVersion' → schema_version
|
||||
|
||||
-- ✗ INCORRECT naming
|
||||
doc->>'bomFormat' → bomFormat, format, bf
|
||||
```
|
||||
|
||||
**RULE:** Generated columns MUST be added with concurrent index creation in production.
|
||||
|
||||
```sql
|
||||
-- ✓ CORRECT: Non-blocking migration
|
||||
ALTER TABLE scheduler.runs ADD COLUMN finding_count INT GENERATED ALWAYS AS (...) STORED;
|
||||
CREATE INDEX CONCURRENTLY idx_runs_finding_count ON scheduler.runs(finding_count);
|
||||
ANALYZE scheduler.runs;
|
||||
|
||||
-- ✗ INCORRECT: Blocking migration
|
||||
CREATE INDEX idx_runs_finding_count ON scheduler.runs(finding_count); -- Blocks table
|
||||
```
|
||||
|
||||
**Reference:** See `SPECIFICATION.md` Section 6.4 for detailed guidelines.
|
||||
|
||||
### 5.4 Null Handling
|
||||
|
||||
**RULE:** Nullable values MUST use `DBNull.Value` when null.
|
||||
|
||||
@@ -1173,6 +1173,67 @@ CREATE INDEX idx_metadata_active ON scheduler.runs USING GIN (stats)
|
||||
WHERE state = 'completed';
|
||||
```
|
||||
|
||||
### 6.4 Generated Columns for JSONB Hot Keys
|
||||
|
||||
For frequently-queried JSONB fields, use PostgreSQL generated columns to enable efficient B-tree indexing and query planning statistics.
|
||||
|
||||
**Problem with expression indexes:**
|
||||
```sql
|
||||
-- Expression indexes don't collect statistics
|
||||
CREATE INDEX idx_format ON sbom_docs ((doc->>'bomFormat'));
|
||||
-- Query planner can't estimate cardinality, may choose suboptimal plans
|
||||
```
|
||||
|
||||
**Solution: Generated columns (PostgreSQL 12+):**
|
||||
```sql
|
||||
-- Add generated column that extracts JSONB field
|
||||
ALTER TABLE scanner.sbom_documents
|
||||
ADD COLUMN bom_format TEXT GENERATED ALWAYS AS ((doc->>'bomFormat')) STORED;
|
||||
|
||||
-- Standard B-tree index with full statistics
|
||||
CREATE INDEX idx_sbom_bom_format ON scanner.sbom_documents(bom_format);
|
||||
```
|
||||
|
||||
**Benefits:**
|
||||
- **B-tree indexable**: Standard index on generated column
|
||||
- **Statistics**: `ANALYZE` collects cardinality, MCV, histogram
|
||||
- **Index-only scans**: Visible to covering indexes
|
||||
- **Zero application changes**: Transparent to ORM/queries
|
||||
|
||||
**When to use generated columns:**
|
||||
- Field queried in >10% of queries against the table
|
||||
- Cardinality >100 distinct values (worth collecting stats)
|
||||
- Field used in JOIN conditions or GROUP BY
|
||||
- Index-only scans are beneficial
|
||||
|
||||
**Naming convention:**
|
||||
```
|
||||
<json_path_snake_case>
|
||||
Examples:
|
||||
doc->>'bomFormat' → bom_format
|
||||
raw->>'schemaVersion' → schema_version
|
||||
stats->>'findingCount'→ finding_count
|
||||
```
|
||||
|
||||
**Migration pattern:**
|
||||
```sql
|
||||
-- Step 1: Add generated column (no lock on existing rows)
|
||||
ALTER TABLE scheduler.runs
|
||||
ADD COLUMN finding_count INT GENERATED ALWAYS AS ((stats->>'findingCount')::int) STORED;
|
||||
|
||||
-- Step 2: Create index concurrently
|
||||
CREATE INDEX CONCURRENTLY idx_runs_finding_count
|
||||
ON scheduler.runs(tenant_id, finding_count);
|
||||
|
||||
-- Step 3: Analyze for statistics
|
||||
ANALYZE scheduler.runs;
|
||||
```
|
||||
|
||||
**Reference implementations:**
|
||||
- `src/Scheduler/...Storage.Postgres/Migrations/010_generated_columns_runs.sql`
|
||||
- `src/Excititor/...Storage.Postgres/Migrations/004_generated_columns_vex.sql`
|
||||
- `src/Concelier/...Storage.Postgres/Migrations/007_generated_columns_advisories.sql`
|
||||
|
||||
---
|
||||
|
||||
## 7. Partitioning Strategy
|
||||
|
||||
195
docs/db/schemas/scan-metrics.md
Normal file
195
docs/db/schemas/scan-metrics.md
Normal file
@@ -0,0 +1,195 @@
|
||||
# Scan Metrics Schema
|
||||
|
||||
Sprint: `SPRINT_3406_0001_0001_metrics_tables`
|
||||
Task: `METRICS-3406-013`
|
||||
Working Directory: `src/Scanner/__Libraries/StellaOps.Scanner.Storage/Postgres/Migrations/`
|
||||
|
||||
## Overview
|
||||
|
||||
The scan metrics schema provides relational PostgreSQL tables for tracking Time-to-Evidence (TTE) and scan performance metrics. This is a hybrid approach where metrics are stored in PostgreSQL while replay manifests remain in the document store.
|
||||
|
||||
## Tables
|
||||
|
||||
### `scanner.scan_metrics`
|
||||
|
||||
Primary table for per-scan metrics.
|
||||
|
||||
| Column | Type | Description |
|
||||
|--------|------|-------------|
|
||||
| `metrics_id` | UUID | Primary key |
|
||||
| `scan_id` | UUID | Unique scan identifier |
|
||||
| `tenant_id` | UUID | Tenant identifier |
|
||||
| `surface_id` | UUID | Optional attack surface identifier |
|
||||
| `artifact_digest` | TEXT | Artifact content hash |
|
||||
| `artifact_type` | TEXT | Type: `oci_image`, `tarball`, `directory`, `other` |
|
||||
| `replay_manifest_hash` | TEXT | Reference to replay manifest in document store |
|
||||
| `findings_sha256` | TEXT | Findings content hash |
|
||||
| `vex_bundle_sha256` | TEXT | VEX bundle content hash |
|
||||
| `proof_bundle_sha256` | TEXT | Proof bundle content hash |
|
||||
| `sbom_sha256` | TEXT | SBOM content hash |
|
||||
| `policy_digest` | TEXT | Policy version hash |
|
||||
| `feed_snapshot_id` | TEXT | Feed snapshot identifier |
|
||||
| `started_at` | TIMESTAMPTZ | Scan start time |
|
||||
| `finished_at` | TIMESTAMPTZ | Scan completion time |
|
||||
| `total_duration_ms` | INT | TTE in milliseconds (generated) |
|
||||
| `t_ingest_ms` | INT | Ingest phase duration |
|
||||
| `t_analyze_ms` | INT | Analyze phase duration |
|
||||
| `t_reachability_ms` | INT | Reachability phase duration |
|
||||
| `t_vex_ms` | INT | VEX phase duration |
|
||||
| `t_sign_ms` | INT | Sign phase duration |
|
||||
| `t_publish_ms` | INT | Publish phase duration |
|
||||
| `package_count` | INT | Number of packages analyzed |
|
||||
| `finding_count` | INT | Number of findings |
|
||||
| `vex_decision_count` | INT | Number of VEX decisions |
|
||||
| `scanner_version` | TEXT | Scanner version |
|
||||
| `scanner_image_digest` | TEXT | Scanner container digest |
|
||||
| `is_replay` | BOOLEAN | Replay mode flag |
|
||||
| `created_at` | TIMESTAMPTZ | Record creation time |
|
||||
|
||||
### `scanner.execution_phases`
|
||||
|
||||
Detailed phase execution tracking.
|
||||
|
||||
| Column | Type | Description |
|
||||
|--------|------|-------------|
|
||||
| `id` | BIGSERIAL | Primary key |
|
||||
| `metrics_id` | UUID | Foreign key to `scan_metrics` |
|
||||
| `phase_name` | TEXT | Phase: `ingest`, `analyze`, `reachability`, `vex`, `sign`, `publish`, `other` |
|
||||
| `phase_order` | INT | Execution order |
|
||||
| `started_at` | TIMESTAMPTZ | Phase start time |
|
||||
| `finished_at` | TIMESTAMPTZ | Phase completion time |
|
||||
| `duration_ms` | INT | Duration in milliseconds (generated) |
|
||||
| `success` | BOOLEAN | Phase success status |
|
||||
| `error_code` | TEXT | Error code if failed |
|
||||
| `error_message` | TEXT | Error message if failed |
|
||||
| `phase_metrics` | JSONB | Phase-specific metrics |
|
||||
|
||||
## Views
|
||||
|
||||
### `scanner.scan_tte`
|
||||
|
||||
Time-to-Evidence view with phase breakdowns.
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
metrics_id,
|
||||
scan_id,
|
||||
tte_ms,
|
||||
tte_seconds,
|
||||
ingest_percent,
|
||||
analyze_percent,
|
||||
reachability_percent,
|
||||
vex_percent,
|
||||
sign_percent,
|
||||
publish_percent
|
||||
FROM scanner.scan_tte
|
||||
WHERE tenant_id = :tenant_id;
|
||||
```
|
||||
|
||||
### `scanner.tte_stats`
|
||||
|
||||
Hourly TTE statistics with SLO compliance.
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
hour_bucket,
|
||||
scan_count,
|
||||
tte_avg_ms,
|
||||
tte_p50_ms,
|
||||
tte_p95_ms,
|
||||
slo_p50_compliance_percent,
|
||||
slo_p95_compliance_percent
|
||||
FROM scanner.tte_stats
|
||||
WHERE tenant_id = :tenant_id;
|
||||
```
|
||||
|
||||
## Functions
|
||||
|
||||
### `scanner.tte_percentile`
|
||||
|
||||
Calculate TTE percentile for a tenant.
|
||||
|
||||
```sql
|
||||
SELECT scanner.tte_percentile(
|
||||
p_tenant_id := :tenant_id,
|
||||
p_percentile := 0.95,
|
||||
p_since := NOW() - INTERVAL '7 days'
|
||||
);
|
||||
```
|
||||
|
||||
## Indexes
|
||||
|
||||
| Index | Columns | Purpose |
|
||||
|-------|---------|---------|
|
||||
| `idx_scan_metrics_tenant` | `tenant_id` | Tenant queries |
|
||||
| `idx_scan_metrics_artifact` | `artifact_digest` | Artifact lookups |
|
||||
| `idx_scan_metrics_started` | `started_at` | Time-range queries |
|
||||
| `idx_scan_metrics_surface` | `surface_id` | Surface queries |
|
||||
| `idx_scan_metrics_replay` | `is_replay` | Filter replays |
|
||||
| `idx_scan_metrics_tenant_started` | `tenant_id, started_at` | Compound tenant+time |
|
||||
| `idx_execution_phases_metrics` | `metrics_id` | Phase lookups |
|
||||
| `idx_execution_phases_name` | `phase_name` | Phase filtering |
|
||||
|
||||
## SLO Thresholds
|
||||
|
||||
Per the advisory section 13.1:
|
||||
|
||||
| Metric | Target |
|
||||
|--------|--------|
|
||||
| TTE P50 | < 120 seconds |
|
||||
| TTE P95 | < 300 seconds |
|
||||
|
||||
## Usage Examples
|
||||
|
||||
### Get TTE for recent scans
|
||||
|
||||
```sql
|
||||
SELECT scan_id, tte_ms, tte_seconds
|
||||
FROM scanner.scan_tte
|
||||
WHERE tenant_id = :tenant_id
|
||||
AND NOT is_replay
|
||||
ORDER BY started_at DESC
|
||||
LIMIT 100;
|
||||
```
|
||||
|
||||
### Check SLO compliance
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
hour_bucket,
|
||||
slo_p50_compliance_percent,
|
||||
slo_p95_compliance_percent
|
||||
FROM scanner.tte_stats
|
||||
WHERE tenant_id = :tenant_id
|
||||
AND hour_bucket >= NOW() - INTERVAL '24 hours';
|
||||
```
|
||||
|
||||
### Phase breakdown analysis
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
phase_name,
|
||||
AVG(duration_ms) as avg_ms,
|
||||
PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY duration_ms) as p95_ms
|
||||
FROM scanner.execution_phases ep
|
||||
JOIN scanner.scan_metrics sm ON ep.metrics_id = sm.metrics_id
|
||||
WHERE sm.tenant_id = :tenant_id
|
||||
AND sm.started_at >= NOW() - INTERVAL '7 days'
|
||||
GROUP BY phase_name
|
||||
ORDER BY phase_order;
|
||||
```
|
||||
|
||||
## Migration
|
||||
|
||||
Migration file: `src/Scanner/__Libraries/StellaOps.Scanner.Storage/Postgres/Migrations/004_scan_metrics.sql`
|
||||
|
||||
Apply with:
|
||||
```bash
|
||||
psql -d stellaops -f 004_scan_metrics.sql
|
||||
```
|
||||
|
||||
## Related
|
||||
|
||||
- [Database Specification](./SPECIFICATION.md)
|
||||
- [Determinism Advisory §13.1](../product-advisories/14-Dec-2025%20-%20Determinism%20and%20Reproducibility%20Technical%20Reference.md)
|
||||
- [Scheduler Schema](./schemas/scheduler.sql)
|
||||
@@ -42,14 +42,14 @@ Read before implementation:
|
||||
|---|---------|--------|---------------------------|--------|-----------------|
|
||||
| 1 | QGATE-0350-001 | DONE | None | Platform | Create `scripts/ci/compute-reachability-metrics.sh` to compute recall/precision from corpus |
|
||||
| 2 | QGATE-0350-002 | DONE | After #1 | Platform | Create `scripts/ci/reachability-thresholds.yaml` with enforcement thresholds |
|
||||
| 3 | QGATE-0350-003 | TODO | After #2 | Platform | Add reachability gate job to `build-test-deploy.yml` |
|
||||
| 3 | QGATE-0350-003 | DONE | After #2 | Platform | Add reachability gate job to `build-test-deploy.yml` |
|
||||
| 4 | QGATE-0350-004 | DONE | None | Platform | Create `scripts/ci/compute-ttfs-metrics.sh` to extract TTFS from test runs |
|
||||
| 5 | QGATE-0350-005 | DONE | After #4 | Platform | Create `bench/baselines/ttfs-baseline.json` with p50/p95 targets |
|
||||
| 6 | QGATE-0350-006 | TODO | After #5 | Platform | Add TTFS regression gate to `build-test-deploy.yml` |
|
||||
| 6 | QGATE-0350-006 | DONE | After #5 | Platform | Add TTFS regression gate to `build-test-deploy.yml` |
|
||||
| 7 | QGATE-0350-007 | DONE | None | Platform | Create `scripts/ci/enforce-performance-slos.sh` for scan/compute SLOs |
|
||||
| 8 | QGATE-0350-008 | TODO | After #7 | Platform | Add performance SLO gate to `build-test-deploy.yml` |
|
||||
| 9 | QGATE-0350-009 | TODO | After #3, #6, #8 | Platform | Create `docs/testing/ci-quality-gates.md` documentation |
|
||||
| 10 | QGATE-0350-010 | TODO | After #9 | Platform | Add quality gate status badges to repository README |
|
||||
| 8 | QGATE-0350-008 | DONE | After #7 | Platform | Add performance SLO gate to `build-test-deploy.yml` |
|
||||
| 9 | QGATE-0350-009 | DONE | After #3, #6, #8 | Platform | Create `docs/testing/ci-quality-gates.md` documentation |
|
||||
| 10 | QGATE-0350-010 | DONE | After #9 | Platform | Add quality gate status badges to repository README |
|
||||
|
||||
## Wave Coordination
|
||||
|
||||
|
||||
@@ -68,9 +68,9 @@ The SCA Failure Catalogue covers real-world scanner failure modes that have occu
|
||||
| 5 | SCA-0351-005 | DONE | None | Scanner | Create FC10 fixture: CVE Split/Merge failure case |
|
||||
| 6 | SCA-0351-006 | DONE | After #1-5 | Scanner | Create DSSE manifests for all new fixtures |
|
||||
| 7 | SCA-0351-007 | DONE | After #6 | Scanner | Update `tests/fixtures/sca/catalogue/inputs.lock` |
|
||||
| 8 | SCA-0351-008 | TODO | After #7 | Scanner | Add xUnit tests for FC6-FC10 in Scanner test project |
|
||||
| 8 | SCA-0351-008 | DONE | After #7 | Scanner | Add xUnit tests for FC6-FC10 in Scanner test project |
|
||||
| 9 | SCA-0351-009 | DONE | After #8 | Scanner | Update `tests/fixtures/sca/catalogue/README.md` documentation |
|
||||
| 10 | SCA-0351-010 | TODO | After #9 | Scanner | Validate all fixtures pass determinism checks |
|
||||
| 10 | SCA-0351-010 | DONE | After #9 | Scanner | Validate all fixtures pass determinism checks |
|
||||
|
||||
## Wave Coordination
|
||||
|
||||
|
||||
@@ -379,20 +379,20 @@ public interface ISubjectExtractor
|
||||
|
||||
| # | Task ID | Status | Key Dependency / Next Step | Owners | Task Definition |
|
||||
|---|---------|--------|---------------------------|--------|-----------------|
|
||||
| 1 | PROOF-ID-0001 | DOING | None | Attestor Guild | Create `StellaOps.Attestor.ProofChain` library project structure |
|
||||
| 2 | PROOF-ID-0002 | DOING | Task 1 | Attestor Guild | Implement `ContentAddressedId` base record and derived types |
|
||||
| 3 | PROOF-ID-0003 | DOING | Task 1 | Attestor Guild | Implement `IJsonCanonicalizer` per RFC 8785 |
|
||||
| 4 | PROOF-ID-0004 | DOING | Task 3 | Attestor Guild | Implement `IContentAddressedIdGenerator` for EvidenceID |
|
||||
| 5 | PROOF-ID-0005 | DOING | Task 3 | Attestor Guild | Implement `IContentAddressedIdGenerator` for ReasoningID |
|
||||
| 6 | PROOF-ID-0006 | DOING | Task 3 | Attestor Guild | Implement `IContentAddressedIdGenerator` for VEXVerdictID |
|
||||
| 7 | PROOF-ID-0007 | DOING | Task 1 | Attestor Guild | Implement `IMerkleTreeBuilder` for deterministic merkle construction |
|
||||
| 8 | PROOF-ID-0008 | DOING | Task 4-7 | Attestor Guild | Implement `IContentAddressedIdGenerator` for ProofBundleID |
|
||||
| 9 | PROOF-ID-0009 | DOING | Task 7 | Attestor Guild | Implement `IContentAddressedIdGenerator` for GraphRevisionID |
|
||||
| 10 | PROOF-ID-0010 | DOING | Task 3 | Attestor Guild | Implement `SbomEntryId` computation from SBOM + PURL |
|
||||
| 11 | PROOF-ID-0011 | DOING | Task 1 | Attestor Guild | Implement `ISubjectExtractor` for CycloneDX SBOMs |
|
||||
| 12 | PROOF-ID-0012 | DOING | Task 1 | Attestor Guild | Create all predicate record types (Evidence, Reasoning, VEX, ProofSpine) |
|
||||
| 13 | PROOF-ID-0013 | TODO | Task 2-12 | QA Guild | Unit tests for all ID generation (determinism verification) |
|
||||
| 14 | PROOF-ID-0014 | TODO | Task 13 | QA Guild | Property-based tests for canonicalization stability |
|
||||
| 1 | PROOF-ID-0001 | DONE | None | Attestor Guild | Create `StellaOps.Attestor.ProofChain` library project structure |
|
||||
| 2 | PROOF-ID-0002 | DONE | Task 1 | Attestor Guild | Implement `ContentAddressedId` base record and derived types |
|
||||
| 3 | PROOF-ID-0003 | DONE | Task 1 | Attestor Guild | Implement `IJsonCanonicalizer` per RFC 8785 |
|
||||
| 4 | PROOF-ID-0004 | DONE | Task 3 | Attestor Guild | Implement `IContentAddressedIdGenerator` for EvidenceID |
|
||||
| 5 | PROOF-ID-0005 | DONE | Task 3 | Attestor Guild | Implement `IContentAddressedIdGenerator` for ReasoningID |
|
||||
| 6 | PROOF-ID-0006 | DONE | Task 3 | Attestor Guild | Implement `IContentAddressedIdGenerator` for VEXVerdictID |
|
||||
| 7 | PROOF-ID-0007 | DONE | Task 1 | Attestor Guild | Implement `IMerkleTreeBuilder` for deterministic merkle construction |
|
||||
| 8 | PROOF-ID-0008 | DONE | Task 4-7 | Attestor Guild | Implement `IContentAddressedIdGenerator` for ProofBundleID |
|
||||
| 9 | PROOF-ID-0009 | DONE | Task 7 | Attestor Guild | Implement `IContentAddressedIdGenerator` for GraphRevisionID |
|
||||
| 10 | PROOF-ID-0010 | DONE | Task 3 | Attestor Guild | Implement `SbomEntryId` computation from SBOM + PURL |
|
||||
| 11 | PROOF-ID-0011 | DONE | Task 1 | Attestor Guild | Implement `ISubjectExtractor` for CycloneDX SBOMs |
|
||||
| 12 | PROOF-ID-0012 | DONE | Task 1 | Attestor Guild | Create all predicate record types (Evidence, Reasoning, VEX, ProofSpine) |
|
||||
| 13 | PROOF-ID-0013 | DONE | Task 2-12 | QA Guild | Unit tests for all ID generation (determinism verification) |
|
||||
| 14 | PROOF-ID-0014 | DONE | Task 13 | QA Guild | Property-based tests for canonicalization stability |
|
||||
| 15 | PROOF-ID-0015 | TODO | Task 13 | Docs Guild | Document ID format specifications in module architecture |
|
||||
|
||||
## Test Specifications
|
||||
|
||||
@@ -33,17 +33,17 @@ Implement high-value, low-effort scoring enhancements from the Determinism and R
|
||||
|---|---------|--------|---------------------------|--------|-----------------|
|
||||
| 1 | DET-3401-001 | DONE | None | Scoring Team | Define `FreshnessBucket` record and `FreshnessMultiplierConfig` in Policy.Scoring |
|
||||
| 2 | DET-3401-002 | DONE | After #1 | Scoring Team | Implement `EvidenceFreshnessCalculator` service with basis-points multipliers |
|
||||
| 3 | DET-3401-003 | TODO | After #2 | Scoring Team | Integrate freshness multiplier into existing evidence scoring pipeline |
|
||||
| 3 | DET-3401-003 | DONE | After #2 | Scoring Team | Integrate freshness multiplier into existing evidence scoring pipeline |
|
||||
| 4 | DET-3401-004 | DONE | After #3 | Scoring Team | Add unit tests for freshness buckets (7d, 30d, 90d, 180d, 365d, >365d) |
|
||||
| 5 | DET-3401-005 | DONE | None | Telemetry Team | Define `ProofCoverageMetrics` class with Prometheus counters/gauges |
|
||||
| 6 | DET-3401-006 | DONE | After #5 | Telemetry Team | Implement `proof_coverage_all`, `proof_coverage_vex`, `proof_coverage_reachable` gauges |
|
||||
| 7 | DET-3401-007 | TODO | After #6 | Telemetry Team | Add proof coverage calculation to scan completion pipeline |
|
||||
| 7 | DET-3401-007 | DONE | After #6 | Telemetry Team | Add proof coverage calculation to scan completion pipeline |
|
||||
| 8 | DET-3401-008 | DONE | After #7 | Telemetry Team | Add unit tests for proof coverage ratio calculations |
|
||||
| 9 | DET-3401-009 | DONE | None | Scoring Team | Define `ScoreExplanation` record with factor/value/reason structure |
|
||||
| 10 | DET-3401-010 | DONE | After #9 | Scoring Team | Implement `ScoreExplainBuilder` to accumulate explanations during scoring |
|
||||
| 11 | DET-3401-011 | DONE | After #10 | Scoring Team | Refactor `RiskScoringResult` to include `Explain` array |
|
||||
| 12 | DET-3401-012 | DONE | After #11 | Scoring Team | Add unit tests for explanation generation |
|
||||
| 13 | DET-3401-013 | TODO | After #4, #8, #12 | QA | Integration tests: freshness + proof coverage + explain in full scan |
|
||||
| 13 | DET-3401-013 | DONE | After #4, #8, #12 | QA | Integration tests: freshness + proof coverage + explain in full scan |
|
||||
|
||||
## Wave Coordination
|
||||
|
||||
|
||||
@@ -30,22 +30,23 @@ Implement relational PostgreSQL tables for scan metrics tracking (hybrid approac
|
||||
|
||||
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
|
||||
|---|---------|--------|---------------------------|--------|-----------------|
|
||||
| 1 | METRICS-3406-001 | TODO | None | DB Team | Create `scan_metrics` table migration |
|
||||
| 2 | METRICS-3406-002 | TODO | After #1 | DB Team | Create `execution_phases` table for timing breakdown |
|
||||
| 3 | METRICS-3406-003 | TODO | After #1 | DB Team | Create `scan_tte` view for TTE calculation |
|
||||
| 4 | METRICS-3406-004 | TODO | After #1 | DB Team | Create indexes for metrics queries |
|
||||
| 5 | METRICS-3406-005 | TODO | None | Scanner Team | Define `ScanMetrics` entity and `ExecutionPhase` record |
|
||||
| 6 | METRICS-3406-006 | TODO | After #1, #5 | Scanner Team | Implement `IScanMetricsRepository` interface |
|
||||
| 7 | METRICS-3406-007 | TODO | After #6 | Scanner Team | Implement `PostgresScanMetricsRepository` |
|
||||
| 8 | METRICS-3406-008 | TODO | After #7 | Scanner Team | Implement `ScanMetricsCollector` service |
|
||||
| 1 | METRICS-3406-001 | DONE | None | DB Team | Create `scan_metrics` table migration |
|
||||
| 2 | METRICS-3406-002 | DONE | After #1 | DB Team | Create `execution_phases` table for timing breakdown |
|
||||
| 3 | METRICS-3406-003 | DONE | After #1 | DB Team | Create `scan_tte` view for TTE calculation |
|
||||
| 4 | METRICS-3406-004 | DONE | After #1 | DB Team | Create indexes for metrics queries |
|
||||
| 5 | METRICS-3406-005 | DONE | None | Scanner Team | Define `ScanMetrics` entity and `ExecutionPhase` record |
|
||||
| 6 | METRICS-3406-006 | DONE | After #1, #5 | Scanner Team | Implement `IScanMetricsRepository` interface |
|
||||
| 7 | METRICS-3406-007 | DONE | After #6 | Scanner Team | Implement `PostgresScanMetricsRepository` |
|
||||
| 8 | METRICS-3406-008 | DONE | After #7 | Scanner Team | Implement `ScanMetricsCollector` service |
|
||||
| 9 | METRICS-3406-009 | TODO | After #8 | Scanner Team | Integrate collector into scan completion pipeline |
|
||||
| 10 | METRICS-3406-010 | TODO | After #3 | Telemetry Team | Export TTE percentiles to Prometheus |
|
||||
| 11 | METRICS-3406-011 | TODO | After #7 | Scanner Team | Unit tests for repository operations |
|
||||
| 12 | METRICS-3406-012 | TODO | After #9 | QA | Integration test: metrics captured on scan completion |
|
||||
| 13 | METRICS-3406-013 | TODO | After #3 | Docs Guild | Document metrics schema in `docs/db/schemas/scan-metrics.md` |
|
||||
| 13 | METRICS-3406-013 | DONE | After #3 | Docs Guild | Document metrics schema in `docs/db/schemas/scan-metrics.md` |
|
||||
|
||||
## Wave Coordination
|
||||
|
||||
|
||||
- **Wave 1** (Parallel): Tasks #1-5 (Schema + Models)
|
||||
- **Wave 2** (Sequential): Tasks #6-9 (Repository + Collector + Integration)
|
||||
- **Wave 3** (Parallel): Tasks #10-13 (Telemetry + Tests + Docs)
|
||||
|
||||
@@ -75,8 +75,8 @@ Benefits:
|
||||
| 4.6 | Verify query plans | DONE | | |
|
||||
| 4.7 | Integration tests | DONE | | Via runbook validation |
|
||||
| **Phase 5: Documentation** |||||
|
||||
| 5.1 | Update SPECIFICATION.md with generated column pattern | TODO | | |
|
||||
| 5.2 | Add generated column guidelines to RULES.md | TODO | | |
|
||||
| 5.1 | Update SPECIFICATION.md with generated column pattern | DONE | | Added Section 6.4 |
|
||||
| 5.2 | Add generated column guidelines to RULES.md | DONE | | Added Section 5.3.1 |
|
||||
| 5.3 | Document query optimization gains | DONE | | postgresql-patterns-runbook.md |
|
||||
|
||||
---
|
||||
|
||||
155
docs/testing/ci-quality-gates.md
Normal file
155
docs/testing/ci-quality-gates.md
Normal file
@@ -0,0 +1,155 @@
|
||||
# CI Quality Gates
|
||||
|
||||
Sprint: `SPRINT_0350_0001_0001_ci_quality_gates_foundation`
|
||||
Task: `QGATE-0350-009`
|
||||
|
||||
## Overview
|
||||
|
||||
StellaOps implements automated quality gates in CI to enforce:
|
||||
- **Reachability Quality** - Recall/precision thresholds for vulnerability detection
|
||||
- **TTFS Regression** - Time-to-First-Signal performance tracking
|
||||
- **Performance SLOs** - Scan time and compute budget enforcement
|
||||
|
||||
These gates run as part of the `build-test-deploy.yml` workflow after the main test suite completes.
|
||||
|
||||
## Quality Gate Jobs
|
||||
|
||||
### Reachability Quality Gate
|
||||
|
||||
**Script:** `scripts/ci/compute-reachability-metrics.sh`
|
||||
**Config:** `scripts/ci/reachability-thresholds.yaml`
|
||||
|
||||
Validates that the scanner meets recall/precision thresholds against the ground-truth corpus.
|
||||
|
||||
#### Metrics Computed
|
||||
|
||||
| Metric | Description | Threshold |
|
||||
|--------|-------------|-----------|
|
||||
| `runtime_dependency_recall` | % of runtime dep vulns detected | ≥ 95% |
|
||||
| `unreachable_false_positives` | FP rate for unreachable findings | ≤ 5% |
|
||||
| `reachability_underreport` | Underreporting rate | ≤ 10% |
|
||||
| `os_package_recall` | % of OS package vulns detected | ≥ 92% |
|
||||
| `code_vuln_recall` | % of code vulns detected | ≥ 88% |
|
||||
| `config_vuln_recall` | % of config vulns detected | ≥ 85% |
|
||||
|
||||
#### Running Locally
|
||||
|
||||
```bash
|
||||
# Dry run (no enforcement)
|
||||
./scripts/ci/compute-reachability-metrics.sh --dry-run
|
||||
|
||||
# Full run against corpus
|
||||
./scripts/ci/compute-reachability-metrics.sh
|
||||
```
|
||||
|
||||
### TTFS Regression Gate
|
||||
|
||||
**Script:** `scripts/ci/compute-ttfs-metrics.sh`
|
||||
**Baseline:** `bench/baselines/ttfs-baseline.json`
|
||||
|
||||
Detects performance regressions in Time-to-First-Signal.
|
||||
|
||||
#### Metrics Computed
|
||||
|
||||
| Metric | Description | Threshold |
|
||||
|--------|-------------|-----------|
|
||||
| `ttfs_p50_ms` | P50 time to first signal | ≤ baseline + 10% |
|
||||
| `ttfs_p95_ms` | P95 time to first signal | ≤ baseline + 15% |
|
||||
| `ttfs_max_ms` | Maximum TTFS | ≤ baseline + 25% |
|
||||
|
||||
#### Baseline Format
|
||||
|
||||
```json
|
||||
{
|
||||
"ttfs_p50_ms": 450,
|
||||
"ttfs_p95_ms": 1200,
|
||||
"ttfs_max_ms": 3000,
|
||||
"measured_at": "2025-12-16T00:00:00Z",
|
||||
"sample_count": 1000
|
||||
}
|
||||
```
|
||||
|
||||
### Performance SLO Gate
|
||||
|
||||
**Script:** `scripts/ci/enforce-performance-slos.sh`
|
||||
**Config:** `scripts/ci/performance-slos.yaml`
|
||||
|
||||
Enforces scan time and compute budget SLOs.
|
||||
|
||||
#### SLOs Enforced
|
||||
|
||||
| SLO | Description | Target |
|
||||
|-----|-------------|--------|
|
||||
| `scan_time_p50_ms` | P50 scan time | ≤ 120,000ms (2 min) |
|
||||
| `scan_time_p95_ms` | P95 scan time | ≤ 300,000ms (5 min) |
|
||||
| `memory_peak_mb` | Peak memory usage | ≤ 2048 MB |
|
||||
| `cpu_seconds` | Total CPU time | ≤ 120 seconds |
|
||||
|
||||
## Workflow Integration
|
||||
|
||||
Quality gates are integrated into the main CI workflow:
|
||||
|
||||
```yaml
|
||||
# .gitea/workflows/build-test-deploy.yml
|
||||
|
||||
quality-gates:
|
||||
runs-on: ubuntu-22.04
|
||||
needs: build-test
|
||||
steps:
|
||||
- name: Reachability quality gate
|
||||
run: ./scripts/ci/compute-reachability-metrics.sh
|
||||
|
||||
- name: TTFS regression gate
|
||||
run: ./scripts/ci/compute-ttfs-metrics.sh
|
||||
|
||||
- name: Performance SLO gate
|
||||
run: ./scripts/ci/enforce-performance-slos.sh --warn-only
|
||||
```
|
||||
|
||||
## Failure Modes
|
||||
|
||||
### Hard Failure (Blocks Merge)
|
||||
|
||||
- Reachability recall below threshold
|
||||
- TTFS regression exceeds 25%
|
||||
- Memory budget exceeded by 50%
|
||||
|
||||
### Soft Failure (Warning Only)
|
||||
|
||||
- Minor TTFS regression (< 15%)
|
||||
- Memory near budget limit
|
||||
- Missing baseline data (new fixtures)
|
||||
|
||||
## Adding New Quality Gates
|
||||
|
||||
1. Create computation script in `scripts/ci/`
|
||||
2. Add threshold configuration (YAML or JSON)
|
||||
3. Integrate into workflow as a new step
|
||||
4. Update this documentation
|
||||
5. Add to sprint tracking
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Gate Fails on PR but Passes on Main
|
||||
|
||||
Check for:
|
||||
- Non-deterministic test execution
|
||||
- Timing-sensitive assertions
|
||||
- Missing test fixtures in PR branch
|
||||
|
||||
### Baseline Drift
|
||||
|
||||
If baselines become stale:
|
||||
|
||||
```bash
|
||||
# Regenerate baselines
|
||||
./scripts/ci/compute-ttfs-metrics.sh --update-baseline
|
||||
./scripts/ci/compute-reachability-metrics.sh --update-baseline
|
||||
```
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Test Suite Overview](../19_TEST_SUITE_OVERVIEW.md)
|
||||
- [Reachability Corpus Plan](../reachability/corpus-plan.md)
|
||||
- [Performance Workbook](../12_PERFORMANCE_WORKBOOK.md)
|
||||
- [Testing Quality Guardrails](./testing-quality-guardrails-implementation.md)
|
||||
Reference in New Issue
Block a user