semi implemented and features implemented save checkpoint

This commit is contained in:
master
2026-02-08 18:00:49 +02:00
parent 04360dff63
commit 1bf6bbf395
20895 changed files with 716795 additions and 64 deletions

View File

@@ -0,0 +1,23 @@
# Advisory Locks / LISTEN-NOTIFY
## Module
Platform
## Status
IMPLEMENTED
## Description
Advisory lock patterns are used in classification history for safe concurrent updates; LISTEN/NOTIFY patterns support real-time event propagation.
## Implementation Details
- **PlatformAnalyticsMaintenanceService**: `src/Platform/StellaOps.Platform.WebService/Services/PlatformAnalyticsMaintenanceService.cs` -- BackgroundService using PeriodicTimer to run analytics maintenance; calls `REFRESH MATERIALIZED VIEW CONCURRENTLY` for `analytics.mv_supplier_concentration`, `analytics.mv_license_distribution`, `analytics.mv_vuln_exposure`, `analytics.mv_attestation_coverage`; uses `SELECT analytics.compute_daily_rollups(@date)` for daily rollup computation
- **AnalyticsIngestionDataSource**: `src/Platform/StellaOps.Platform.Analytics/Services/AnalyticsIngestionDataSource.cs` -- PostgreSQL data source for analytics with connection management
- **VulnerabilityCorrelationService**: `src/Platform/StellaOps.Platform.Analytics/Services/VulnerabilityCorrelationService.cs` -- correlates vulnerabilities with SBOM components using PURL matching
- **ScannerOrchestratorEvents**: `src/Platform/StellaOps.Platform.Analytics/Models/ScannerOrchestratorEvents.cs` -- event models for scanner report ready / scan completed events via event streams
- **Source**: Feature matrix scan
## E2E Test Plan
- [ ] Verify advisory locks prevent concurrent classification history updates
- [ ] Test LISTEN/NOTIFY propagates real-time events to subscribers
- [ ] Verify materialized view concurrent refresh completes without locking reads
- [ ] Test daily rollup computation produces correct aggregates for a given date

View File

@@ -0,0 +1,24 @@
# Materialized Views for Analytics
## Module
Platform
## Status
IMPLEMENTED
## Description
Materialized views with indexes, VEX validity filters, and deterministic arrays are used for analytics with a dedicated maintenance service for refresh.
## Implementation Details
- **PlatformAnalyticsMaintenanceService**: `src/Platform/StellaOps.Platform.WebService/Services/PlatformAnalyticsMaintenanceService.cs` -- BackgroundService that periodically refreshes 4 materialized views (`mv_supplier_concentration`, `mv_license_distribution`, `mv_vuln_exposure`, `mv_attestation_coverage`) using `REFRESH MATERIALIZED VIEW CONCURRENTLY`; supports daily rollup backfill, configurable interval, and run-on-startup mode via `PlatformAnalyticsMaintenanceOptions`
- **PlatformAnalyticsQueryExecutor**: `src/Platform/StellaOps.Platform.WebService/Services/PlatformAnalyticsQueryExecutor.cs` -- executes analytical queries against materialized views
- **PlatformAnalyticsService**: `src/Platform/StellaOps.Platform.WebService/Services/PlatformAnalyticsService.cs` -- service layer for analytics queries (suppliers, licenses, vulnerabilities, backlog, attestation coverage, vulnerability/component trends)
- **AnalyticsEndpoints**: `src/Platform/StellaOps.Platform.WebService/Endpoints/AnalyticsEndpoints.cs` -- REST API at `/api/analytics` with 7 endpoints: suppliers, licenses, vulnerabilities, backlog, attestation-coverage, trends/vulnerabilities, trends/components; all require `AnalyticsRead` authorization
- **Source**: Feature matrix scan
## E2E Test Plan
- [ ] Verify materialized views refresh concurrently without blocking reads
- [ ] Test daily rollup backfill covers configured BackfillDays range
- [ ] Verify analytics endpoints return correct data from materialized views
- [ ] Test analytics service returns 503 when analytics storage is not configured
- [ ] Verify trend endpoints return daily data points for specified time windows

View File

@@ -0,0 +1,28 @@
# Platform Service Aggregation Layer
## Module
Platform
## Status
IMPLEMENTED
## Description
Backend Platform Service acting as aggregation layer for health status, quotas, onboarding progress, user preferences, and global search across all modules.
## Implementation Details
- **PlatformEndpoints**: `src/Platform/StellaOps.Platform.WebService/Endpoints/PlatformEndpoints.cs` -- REST API at `/api/v1/platform` with 6 endpoint groups: health (summary/dependencies/incidents/metrics), quotas (summary/tenants/alerts), onboarding (status/complete/skip), preferences (dashboard/profiles), search, metadata; all with tenant-scoped authorization policies
- **PlatformHealthService**: `src/Platform/StellaOps.Platform.WebService/Services/PlatformHealthService.cs` -- aggregates health status from all platform services
- **PlatformQuotaService**: service for quota tracking with alert management (create alert thresholds per tenant)
- **PlatformOnboardingService**: `src/Platform/StellaOps.Platform.WebService/Services/PlatformOnboardingService.cs` -- tracks onboarding progress with step completion and skip support
- **PlatformPreferencesService**: `src/Platform/StellaOps.Platform.WebService/Services/PlatformPreferencesService.cs` -- user dashboard preferences with profile management (CRUD)
- **PlatformSearchService**: global search across all modules with source filtering, pagination
- **PlatformMetadataService**: `src/Platform/StellaOps.Platform.WebService/Services/PlatformMetadataService.cs` -- platform metadata retrieval
- **PlatformCache**: `src/Platform/StellaOps.Platform.WebService/Services/PlatformCache.cs` -- caching layer with TTL and data-as-of timestamps
- **Source**: SPRINT_20251229_043_PLATFORM_platform_service_foundation
## E2E Test Plan
- [ ] Verify health summary endpoint aggregates all service statuses
- [ ] Test quota alerts are created and returned with proper authorization
- [ ] Verify onboarding step completion transitions correctly
- [ ] Test global search returns results from multiple sources with pagination
- [ ] Verify dashboard preferences persist and load per tenant/user

View File

@@ -0,0 +1,26 @@
# Platform Setup Wizard Backend API
## Module
Platform
## Status
IMPLEMENTED
## Description
Real /api/v1/setup/* endpoints replacing UI mocks with deterministic session state (create, resume, execute, skip, finalize), tenant scoping, and offline-first "data as of" metadata.
## Implementation Details
- **SetupEndpoints**: `src/Platform/StellaOps.Platform.WebService/Endpoints/SetupEndpoints.cs` -- REST API at `/api/v1/setup` with 3 endpoint groups: sessions (GET current, POST create, POST resume, POST finalize), steps (POST execute, POST skip), definitions (GET step definitions); AllowAnonymous during initial setup, requires auth after completion
- **SetupStateDetector**: detects setup completion state from storage/DB settings; routes between bootstrap context and authenticated context
- **PlatformSetupService**: service layer for setup wizard operations (CreateSessionAsync, ResumeOrCreateSessionAsync, ExecuteStepAsync, SkipStepAsync, FinalizeSessionAsync, GetStepDefinitionsAsync)
- **SetupWizardModels**: `src/Platform/StellaOps.Platform.WebService/Contracts/SetupWizardModels.cs` -- request/response models (CreateSetupSessionRequest, SetupSessionResponse, ExecuteSetupStepRequest, SkipSetupStepRequest, FinalizeSetupSessionRequest, FinalizeSetupSessionResponse, SetupStepDefinitionsResponse)
- **Problem+JSON errors**: all endpoints return RFC 7807 ProblemDetails on errors
- **Source**: SPRINT_20260112_004_PLATFORM_setup_wizard_backend.md
## E2E Test Plan
- [ ] Verify setup session creates with bootstrap context when auth is unavailable
- [ ] Test session resume returns existing session or creates new one
- [ ] Verify step execution updates session state correctly
- [ ] Test step skip marks step as skipped and advances session
- [ ] Verify finalize completes setup and subsequent requests require authentication
- [ ] Test step definitions endpoint returns all available setup steps

View File

@@ -0,0 +1,27 @@
# SBOM Analytics Lake (Star-Schema PostgreSQL)
## Module
Platform
## Status
IMPLEMENTED
## Description
Star-schema PostgreSQL analytics layer for SBOM data with component registry, vulnerability correlation tables, attestation tracking, materialized views for trend analysis, and stored procedures for analytics queries. While "Materialized Views for Analytics" is in the known list, this is a much broader star-schema analytics subsystem with dedicated migration, ingestion services, and multi-table analytics design.
## Implementation Details
- **AnalyticsIngestionService**: `src/Platform/StellaOps.Platform.Analytics/Services/AnalyticsIngestionService.cs` -- BackgroundService subscribing to `scanner.report.ready` events via IEventStream; parses SBOM (CycloneDX/SPDX), resolves artifact digests, upserts into star-schema tables (`analytics.artifacts`, `analytics.raw_sboms`, `analytics.components`, `analytics.artifact_components`); uses stored procedures (`analytics.compute_daily_rollups`, `analytics.parse_purl`, `analytics.normalize_supplier`, `analytics.categorize_license`); builds dependency paths via BFS from root component
- **VulnerabilityCorrelationService**: `src/Platform/StellaOps.Platform.Analytics/Services/VulnerabilityCorrelationService.cs` -- correlates PURL-based components with known vulnerabilities; updates artifact vulnerability counts
- **AttestationIngestionService**: `src/Platform/StellaOps.Platform.Analytics/Services/AttestationIngestionService.cs` -- ingests attestation events into analytics
- **Utilities**: PurlParser (PURL normalization), LicenseExpressionRenderer (license aggregation), Sha256Hasher (digest computation), TenantNormalizer (tenant filtering), VersionRuleEvaluator, VulnerabilityCorrelationRules
- **AnalyticsIngestionOptions**: `src/Platform/StellaOps.Platform.Analytics/Options/AnalyticsIngestionOptions.cs` -- configurable stream names, tenant allowlists, ingest/schema versions
- **Tests**: `src/Platform/__Tests/StellaOps.Platform.Analytics.Tests/`
- **Source**: SPRINT_20260120_030_Platform_sbom_analytics_lake.md
## E2E Test Plan
- [ ] Verify SBOM ingestion from scanner.report.ready events populates all star-schema tables
- [ ] Test component deduplication via (purl, hash_sha256) conflict resolution
- [ ] Verify dependency path BFS builds correct depth and introduced_via values
- [ ] Test vulnerability correlation updates component and artifact vulnerability counts
- [ ] Verify daily rollup stored procedure computes correct aggregates
- [ ] Test tenant filtering respects AllowedTenants configuration

View File

@@ -0,0 +1,25 @@
# Scanner Platform Events (Redis Streams)
## Module
Platform
## Status
IMPLEMENTED
## Description
Scanner WebService emits `scanner.report.ready` and `scanner.scan.completed` platform events via Redis Streams with DSSE envelopes embedded verbatim, configurable via `scanner:events:*` settings.
## Implementation Details
- **AnalyticsIngestionService**: `src/Platform/StellaOps.Platform.Analytics/Services/AnalyticsIngestionService.cs` -- subscribes to event stream (configurable via `scanner:events:*` settings); filters for `scanner.report.ready` and `scanner.scan.completed` event kinds from OrchestratorEventEnvelope; tenant-scoped event filtering
- **ScannerOrchestratorEvents**: `src/Platform/StellaOps.Platform.Analytics/Models/ScannerOrchestratorEvents.cs` -- event models for scanner platform events
- **RekorEvents**: `src/Platform/StellaOps.Platform.Analytics/Models/RekorEvents.cs` -- Rekor transparency log event models
- **AdvisoryEvents**: `src/Platform/StellaOps.Platform.Analytics/Models/AdvisoryEvents.cs` -- advisory event models
- **IEventStream/IEventStreamFactory**: from `StellaOps.Messaging` -- event stream abstraction for subscribing to platform events with position tracking (Beginning/End)
- **Source**: 2025-10-19-scanner-policy.md
## E2E Test Plan
- [ ] Verify scanner.report.ready events are consumed and processed by analytics ingestion
- [ ] Test scanner.scan.completed events trigger appropriate analytics updates
- [ ] Verify DSSE envelopes are embedded verbatim in event payloads
- [ ] Test event stream position tracking resumes from correct position after restart
- [ ] Verify tenant filtering skips events from non-allowed tenants