tests fixes and sprints work

2026-01-22 19:08:46 +02:00
parent c32fff8f86
commit 726d70dc7f
881 changed files with 134434 additions and 6228 deletions
--- a/docs/modules/analytics/README.md
+++ b/docs/modules/analytics/README.md
@@ -17,7 +17,7 @@ Stella Ops generates rich data through SBOM ingestion, vulnerability correlation
 |------------|-------------|
 | Unified component registry | Canonical component table with normalized suppliers and licenses |
 | Vulnerability correlation | Pre-joined component-vulnerability mapping with EPSS/KEV flags |
-| VEX-adjusted exposure | Vulnerability counts that respect VEX overrides |
+| VEX-adjusted exposure | Vulnerability counts that respect active VEX overrides (validity windows applied) |
 | Attestation tracking | Provenance and SLSA level coverage by environment/team |
 | Time-series rollups | Daily snapshots for trend analysis |
 | Materialized views | Pre-computed aggregations for dashboard performance |
@@ -68,6 +68,14 @@ Stella Ops generates rich data through SBOM ingestion, vulnerability correlation
 | `daily_vulnerability_counts` | Rollup | Daily vuln aggregations |
 | `daily_component_counts` | Rollup | Daily component aggregations |

+Rollup retention is 90 days in hot storage. `compute_daily_rollups()` prunes
+older rows after each run; archival follows operations runbooks.
+Platform WebService can automate rollups + materialized view refreshes via
+`PlatformAnalyticsMaintenanceService` (see `architecture.md` for schedule and
+configuration).
+Use `Platform:AnalyticsMaintenance:BackfillDays` to recompute the most recent
+N days of rollups on the first maintenance run after downtime (set to `0` to disable).
+
 ### Materialized Views

 | View | Refresh | Purpose |
@@ -77,33 +85,36 @@ Stella Ops generates rich data through SBOM ingestion, vulnerability correlation
 | `mv_vuln_exposure` | Daily | CVE exposure adjusted by VEX |
 | `mv_attestation_coverage` | Daily | Provenance/SLSA coverage by env/team |

+Array-valued fields (for example `environments` and `ecosystems`) are ordered
+alphabetically to keep analytics outputs deterministic.
+
 ## Quick Start

 ### Day-1 Queries

-**Top supplier concentration (supply chain risk):**
+**Top supplier concentration (supply chain risk, optional environment filter):**
 ```sql
-SELECT * FROM analytics.sp_top_suppliers(20);
+SELECT analytics.sp_top_suppliers(20, 'prod');
 ```

-**License risk heatmap:**
+**License risk heatmap (optional environment filter):**
 ```sql
-SELECT * FROM analytics.sp_license_heatmap();
+SELECT analytics.sp_license_heatmap('prod');
 ```

 **CVE exposure adjusted by VEX:**
 ```sql
-SELECT * FROM analytics.sp_vuln_exposure('prod', 'high');
+SELECT analytics.sp_vuln_exposure('prod', 'high');
 ```

 **Fixable vulnerability backlog:**
 ```sql
-SELECT * FROM analytics.sp_fixable_backlog('prod');
+SELECT analytics.sp_fixable_backlog('prod');
 ```

 **Attestation coverage gaps:**
 ```sql
-SELECT * FROM analytics.sp_attestation_gaps('prod');
+SELECT analytics.sp_attestation_gaps('prod');
 ```

 ### API Endpoints
@@ -118,6 +129,82 @@ SELECT * FROM analytics.sp_attestation_gaps('prod');
 | `/api/analytics/trends/vulnerabilities` | GET | Vulnerability time-series |
 | `/api/analytics/trends/components` | GET | Component time-series |

+All analytics endpoints require the `analytics.read` scope.
+The platform metadata capability `analytics` reports whether analytics storage is configured.
+
+#### Query Parameters
+- `/api/analytics/suppliers`: `limit` (optional, default 20), `environment` (optional)
+- `/api/analytics/licenses`: `environment` (optional)
+- `/api/analytics/vulnerabilities`: `minSeverity` (optional, default `low`), `environment` (optional)
+- `/api/analytics/backlog`: `environment` (optional)
+- `/api/analytics/attestation-coverage`: `environment` (optional)
+- `/api/analytics/trends/vulnerabilities`: `environment` (optional), `days` (optional, default 30)
+- `/api/analytics/trends/components`: `environment` (optional), `days` (optional, default 30)
+
+## Ingestion Configuration
+
+Analytics ingestion runs inside the Platform WebService and subscribes to Scanner, Concelier, and Attestor streams. Configure ingestion via `Platform:AnalyticsIngestion`:
+
+```yaml
+Platform:
+  Storage:
+    PostgresConnectionString: "Host=...;Database=analytics;Username=...;Password=..."
+  AnalyticsIngestion:
+    Enabled: true
+    PostgresConnectionString: "" # optional; defaults to Platform:Storage
+    AllowedTenants: ["tenant-a", "tenant-b"]
+    Streams:
+      ScannerStream: "orchestrator:events"
+      ConcelierObservationStream: "concelier:advisory.observation.updated:v1"
+      ConcelierLinksetStream: "concelier:advisory.linkset.updated:v1"
+      AttestorStream: "attestor:events"
+      StartFromBeginning: false
+    Cas:
+      RootPath: "/var/lib/stellaops/cas"
+      DefaultBucket: "attestations"
+    Attestations:
+      BundleUriTemplate: "bundle:{digest}"
+```
+
+Bundle URI templates support:
+- `{digest}` for the full digest string (for example `sha256:...`).
+- `{hash}` for the raw hex digest (no algorithm prefix).
+- `bundle:{digest}` which resolves to `cas://<DefaultBucket>/{digest}` by default.
+- `file:/path/to/bundles/bundle-{hash}.json` for offline file ingestion.
+
+For offline workflows, verify bundles with `stella bundle verify` before ingesting them.
+
+## Console UI
+
+SBOM Lake analytics are exposed in the Console under `Analytics > SBOM Lake` (`/analytics/sbom-lake`).
+Console access requires `ui.read` plus `analytics.read` scopes.
+
+Key UI features:
+- Filters for environment, minimum severity, and time window.
+- Panels for suppliers, licenses, vulnerability exposure, and attestation coverage.
+- Trend views for vulnerabilities and components.
+- Fixable backlog table with CSV export.
+
+See [console.md](./console.md) for operator guidance and filter behavior.
+
+## CLI Access
+
+SBOM lake analytics are exposed via the CLI under `stella analytics sbom-lake`
+(requires `analytics.read` scope).
+
+```bash
+# Top suppliers
+stella analytics sbom-lake suppliers --limit 20
+
+# Vulnerability exposure in prod (high+), CSV export
+stella analytics sbom-lake vulnerabilities --environment prod --min-severity high --format csv --output vuln.csv
+
+# 30-day trends for both series
+stella analytics sbom-lake trends --days 30 --series all --format json
+```
+
+See `docs/modules/cli/guides/commands/analytics.md` for command-level details.
+
 ## Architecture

 See [architecture.md](./architecture.md) for detailed design decisions, data flow, and normalization rules.
@@ -133,4 +220,6 @@ See [analytics_schema.sql](../../db/analytics_schema.sql) for complete DDL inclu

 ## Sprint Reference

-Implementation tracked in: `docs/implplan/SPRINT_20260120_030_Platform_sbom_analytics_lake.md`
+Implementation tracked in:
+- `docs/implplan/SPRINT_20260120_030_Platform_sbom_analytics_lake.md`
+- `docs/implplan/SPRINT_20260120_032_Cli_sbom_analytics_cli.md`
--- a/docs/modules/analytics/architecture.md
+++ b/docs/modules/analytics/architecture.md
@@ -7,7 +7,7 @@ The Analytics module implements a **star-schema data warehouse** pattern optimiz
 1. **Separation of concerns**: Analytics schema is isolated from operational schemas (scanner, vex, proof_system)
 2. **Pre-computation**: Expensive aggregations computed in advance via materialized views
 3. **Audit trail**: Raw payloads preserved for reprocessing and compliance
-4. **Determinism**: All normalization functions are immutable and reproducible
+4. **Determinism**: Normalization functions are immutable and reproducible; array aggregates are ordered for stable outputs
 5. **Incremental updates**: Supports both full refresh and incremental ingestion

 ## Data Flow
@@ -120,10 +120,9 @@ When a component is upserted, the `VulnerabilityCorrelationService` queries Conc
 2. Filter by version range matching
 3. Upsert to `component_vulns` with severity, EPSS, KEV flags

-**Version range matching** uses Concelier's existing logic to handle:
- Semver ranges: `>=1.0.0 <2.0.0`
- Exact versions: `1.2.3`
- Wildcards: `1.x`
+**Version range matching** currently supports semver ranges and exact matches via
+`VersionRuleEvaluator`. Non-semver schemes fall back to exact string matches; wildcard
+and ecosystem-specific ranges require upstream normalization.

 ## VEX Override Logic

@@ -145,7 +144,21 @@ COUNT(DISTINCT ac.artifact_id) FILTER (
 **Override validity:**
 - `valid_from`: When the override became effective
 - `valid_until`: Expiration (NULL = no expiration)
- Only `status = 'not_affected'` reduces exposure counts
+- Only `status = 'not_affected'` reduces exposure counts, and only when the override is active in its validity window.
+
+## Attestation Ingestion
+
+Attestation ingestion consumes Attestor Rekor entry events and expects Sigstore bundles
+or raw DSSE envelopes. The ingestion service:
+- Resolves bundle URIs using `BundleUriTemplate`; `bundle:{digest}` maps to
+  `cas://<DefaultBucket>/{digest}` by default.
+- Decodes DSSE payloads, computes `dsse_payload_hash`, and records `predicate_uri` plus
+  Rekor log metadata (`rekor_log_id`, `rekor_log_index`).
+- Uses in-toto `subject` digests to link artifacts when reanalysis hints are absent.
+- Maps predicate URIs into `analytics_attestation_type` values
+  (`provenance`, `sbom`, `vex`, `build`, `scan`, `policy`).
+- Expands VEX statements into `vex_overrides` rows, one per product reference, and
+  captures optional validity timestamps when provided.

 ## Time-Series Rollups

@@ -164,14 +177,14 @@ Daily rollups computed by `compute_daily_rollups()`:
 - `total_components`: Distinct components
 - `unique_suppliers`: Distinct normalized suppliers

-**Retention policy:** 90 days in hot storage; older data archived to cold storage.
+**Retention policy:** 90 days in hot storage; `compute_daily_rollups()` prunes older rows and downstream jobs archive to cold storage.

 ## Materialized View Refresh

 All materialized views support `REFRESH ... CONCURRENTLY` for zero-downtime updates:

 ```sql
-- Refresh all views (run daily via pg_cron or Scheduler)
+-- Refresh all views (non-concurrent; run off-peak)
 SELECT analytics.refresh_all_views();
 ```

@@ -182,6 +195,19 @@ SELECT analytics.refresh_all_views();
 - `mv_attestation_coverage`: 02:45 UTC daily
 - `compute_daily_rollups()`: 03:00 UTC daily

+Platform WebService can run the daily rollup + refresh loop via
+`PlatformAnalyticsMaintenanceService`. Configure the schedule with:
+- `Platform:AnalyticsMaintenance:Enabled` (default `true`)
+- `Platform:AnalyticsMaintenance:IntervalMinutes` (default `1440`)
+- `Platform:AnalyticsMaintenance:RunOnStartup` (default `true`)
+- `Platform:AnalyticsMaintenance:ComputeDailyRollups` (default `true`)
+- `Platform:AnalyticsMaintenance:RefreshMaterializedViews` (default `true`)
+- `Platform:AnalyticsMaintenance:BackfillDays` (default `0`, set to `0` to disable; recompute the most recent N days on the first maintenance run)
+
+The hosted service issues concurrent refresh statements directly for each view.
+Use a DB scheduler (pg_cron) or external orchestrator if you need the staggered
+per-view timing above.
+
 ## Performance Considerations

 ### Indexing Strategy
@@ -198,9 +224,9 @@ SELECT analytics.refresh_all_views();

 | Query | Target | Notes |
 |-------|--------|-------|
-| `sp_top_suppliers(20)` | < 100ms | Uses materialized view |
-| `sp_license_heatmap()` | < 100ms | Uses materialized view |
-| `sp_vuln_exposure()` | < 200ms | Uses materialized view |
+| `sp_top_suppliers(20, 'prod')` | < 100ms | Uses materialized view when env is null; env filter reads base tables |
+| `sp_license_heatmap('prod')` | < 100ms | Uses materialized view when env is null; env filter reads base tables |
+| `sp_vuln_exposure()` | < 200ms | Uses materialized view for global queries; environment filters read base tables |
 | `sp_fixable_backlog()` | < 500ms | Live query with indexes |
 | `sp_attestation_gaps()` | < 100ms | Uses materialized view |

@@ -246,12 +272,12 @@ All tables include `created_at` and `updated_at` timestamps. Raw payload tables

 ### Upstream Dependencies

-| Service | Event | Action |
-|---------|-------|--------|
-| Scanner | SBOM ingested | Normalize and upsert components |
-| Concelier | Advisory updated | Re-correlate affected components |
-| Excititor | VEX observation | Create/update vex_overrides |
-| Attestor | Attestation created | Upsert attestation record |
+| Service | Event | Contract | Action |
+|---------|-------|----------|--------|
+| Scanner | SBOM report ready | `scanner.event.report.ready@1` (`docs/modules/signals/events/orchestrator-scanner-events.md`) | Normalize and upsert components |
+| Concelier | Advisory observation/linkset updated | `advisory.observation.updated@1` (`docs/modules/concelier/events/advisory.observation.updated@1.schema.json`), `advisory.linkset.updated@1` (`docs/modules/concelier/events/advisory.linkset.updated@1.md`) | Re-correlate affected components |
+| Excititor | VEX statement changes | `vex.statement.*` (`docs/modules/excititor/architecture.md`) | Create/update vex_overrides |
+| Attestor | Rekor entry logged | `rekor.entry.logged` (`docs/modules/attestor/architecture.md`) | Upsert attestation record |

 ### Downstream Consumers

--- a/docs/modules/analytics/console.md
+++ b/docs/modules/analytics/console.md
@@ -0,0 +1,64 @@
+# Analytics Console (SBOM Lake)
+
+The Console exposes SBOM analytics lake data under `Analytics > SBOM Lake`.
+This view is read-only and uses the analytics API endpoints documented in `docs/modules/analytics/README.md`.
+
+## Access
+
+- Route: `/analytics/sbom-lake`
+- Required scopes: `ui.read` and `analytics.read`
+- Console admin bundles: `role/analytics-viewer`, `role/analytics-operator`, `role/analytics-admin`
+- Data freshness: the page surfaces the latest `dataAsOf` timestamp returned by the API.
+
+## Filters
+
+The SBOM Lake page supports three filters that round-trip via URL query parameters:
+
+- Environment: `env` (optional, example: `Prod`)
+- Minimum severity: `severity` (optional, example: `high`)
+- Time window (days): `days` (optional, example: `90`)
+
+When a filter changes, the Console reloads all panels using the updated parameters.
+Supplier and license panels honor the environment filter alongside the other views.
+
+## Panels
+
+The dashboard presents four summary panels:
+
+1. Supplier concentration (top suppliers by component count)
+2. License distribution (license categories and counts)
+3. Vulnerability exposure (top CVEs after VEX adjustments)
+4. Attestation coverage (provenance and SLSA 2+ coverage)
+
+Each panel shows a loading state, empty state, and summary counts.
+
+## Trends
+
+Two trend panels are included:
+
+- Vulnerability trend: net exposure over the selected time window
+- Component trend: total components and unique suppliers
+
+The Console aggregates trend points by date and renders a simple bar chart plus a compact list.
+
+## Fixable Backlog
+
+The fixable backlog table lists vulnerabilities with fixes available, grouped by component and service.
+The "Top backlog components" table derives a component summary from the same backlog data.
+
+### CSV Export
+
+The "Export backlog CSV" action downloads a deterministic, ordered CSV with:
+
+- Service
+- Component
+- Version
+- Vulnerability
+- Severity
+- Environment
+- Fixed version
+
+## Troubleshooting
+
+- If panels show "No data", verify that the analytics schema and materialized views are populated.
+- If an error banner appears, check the analytics API availability and ensure the tenant has `analytics.read`.
--- a/docs/modules/analytics/queries.md
+++ b/docs/modules/analytics/queries.md
@@ -9,8 +9,8 @@ This document provides ready-to-use SQL queries for common analytics use cases.
 Identifies suppliers with the highest component footprint, indicating supply chain concentration risk.

 ```sql
-- Via stored procedure (recommended)
-SELECT * FROM analytics.sp_top_suppliers(20);
+-- Via stored procedure (recommended, optional environment filter)
+SELECT analytics.sp_top_suppliers(20, 'prod');

 -- Direct query
 SELECT
@@ -33,8 +33,8 @@ LIMIT 20;
 Shows distribution of components by license category for compliance review.

 ```sql
-- Via stored procedure
-SELECT * FROM analytics.sp_license_heatmap();
+-- Via stored procedure (optional environment filter)
+SELECT analytics.sp_license_heatmap('prod');

 -- Direct query with grouping
 SELECT
@@ -62,9 +62,9 @@ Shows true vulnerability exposure after applying VEX mitigations.

 ```sql
 -- Via stored procedure
-SELECT * FROM analytics.sp_vuln_exposure('prod', 'high');
+SELECT analytics.sp_vuln_exposure('prod', 'high');

-- Direct query showing VEX effectiveness
+-- Direct query showing VEX effectiveness (global view; use sp_vuln_exposure for environment filtering)
 SELECT
  vuln_id,
  severity::TEXT,
@@ -97,7 +97,7 @@ Lists vulnerabilities that can be fixed today (fix available, not VEX-mitigated)

 ```sql
 -- Via stored procedure
-SELECT * FROM analytics.sp_fixable_backlog('prod');
+SELECT analytics.sp_fixable_backlog('prod');

 -- Direct query with priority scoring
 SELECT
@@ -130,6 +130,7 @@ JOIN analytics.artifacts a ON a.artifact_id = ac.artifact_id
 LEFT JOIN analytics.vex_overrides vo ON vo.artifact_id = a.artifact_id
  AND vo.vuln_id = cv.vuln_id
  AND vo.status = 'not_affected'
+  AND vo.valid_from <= now()
  AND (vo.valid_until IS NULL OR vo.valid_until > now())
 WHERE cv.affects = TRUE
  AND cv.fix_available = TRUE
@@ -147,7 +148,7 @@ Shows attestation gaps by environment and team.

 ```sql
 -- Via stored procedure
-SELECT * FROM analytics.sp_attestation_gaps('prod');
+SELECT analytics.sp_attestation_gaps('prod');

 -- Direct query with gap analysis
 SELECT
@@ -267,6 +268,7 @@ JOIN analytics.artifact_components ac ON ac.component_id = c.component_id
 JOIN analytics.artifacts a ON a.artifact_id = ac.artifact_id
 LEFT JOIN analytics.vex_overrides vo ON vo.artifact_id = a.artifact_id
  AND vo.vuln_id = cv.vuln_id
+  AND vo.valid_from <= now()
  AND (vo.valid_until IS NULL OR vo.valid_until > now())
 WHERE cv.vuln_id = 'CVE-2021-44228'
 ORDER BY a.environment, a.name;
@@ -312,7 +314,7 @@ SELECT
  c.license_category::TEXT,
  c.supplier_normalized AS supplier,
  COUNT(DISTINCT a.artifact_id) AS artifact_count,
-  ARRAY_AGG(DISTINCT a.name) AS affected_artifacts
+  ARRAY_AGG(DISTINCT a.name ORDER BY a.name) AS affected_artifacts
 FROM analytics.components c
 JOIN analytics.artifact_components ac ON ac.component_id = c.component_id
 JOIN analytics.artifacts a ON a.artifact_id = ac.artifact_id
@@ -340,6 +342,8 @@ SELECT
 FROM analytics.component_vulns cv
 JOIN analytics.vex_overrides vo ON vo.vuln_id = cv.vuln_id
  AND vo.status = 'not_affected'
+  AND vo.valid_from <= now()
+  AND (vo.valid_until IS NULL OR vo.valid_until > now())
 WHERE cv.published_at >= now() - INTERVAL '90 days'
  AND cv.published_at IS NOT NULL
 GROUP BY cv.severity