stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 2 Releases Wiki Activity

documentation cleanse, sprints work and planning. remaining non EF DAL migration to EF

Browse Source
This commit is contained in:
master
2026-02-25 01:24:07 +02:00
parent b07d27772e
commit 4db038123b
9090 changed files with 4836 additions and 2909 deletions
Download Patch File Download Diff File Expand all files Collapse all files

225
docs-archived/modules/analytics/README.md Normal file
View File

@@ -0,0 +1,225 @@
# Analytics Module
The Analytics module provides a star-schema data warehouse layer for SBOM and attestation data, enabling executive reporting, risk dashboards, and ad-hoc analysis.
## Overview
Stella Ops generates rich data through SBOM ingestion, vulnerability correlation, VEX assessments, and attestations. The Analytics module normalizes this data into a queryable warehouse schema optimized for:
- **Executive dashboards**: Risk posture, vulnerability trends, compliance status
- **Supply chain analysis**: Supplier concentration, license distribution
- **Security metrics**: CVE exposure, VEX effectiveness, MTTR tracking
- **Attestation coverage**: SLSA compliance, provenance gaps
## Key Capabilities
| Capability | Description |
|------------|-------------|
| Unified component registry | Canonical component table with normalized suppliers and licenses |
| Vulnerability correlation | Pre-joined component-vulnerability mapping with EPSS/KEV flags |
| VEX-adjusted exposure | Vulnerability counts that respect active VEX overrides (validity windows applied) |
| Attestation tracking | Provenance and SLSA level coverage by environment/team |
| Time-series rollups | Daily snapshots for trend analysis |
| Materialized views | Pre-computed aggregations for dashboard performance |
## Data Model
### Star Schema Overview
```
┌─────────────────┐
│ artifacts │ (dimension)
│ container/app │
└────────┬────────┘
│
┌──────────────┼──────────────┐
│ │ │
┌─────────▼──────┐ ┌─────▼─────┐ ┌──────▼──────┐
│ artifact_ │ │attestations│ │vex_overrides│
│ components │ │ (fact) │ │ (fact) │
│ (bridge) │ └───────────┘ └─────────────┘
└─────────┬──────┘
│
┌─────────▼──────┐
│ components │ (dimension)
│ unified │
│ registry │
└─────────┬──────┘
│
┌─────────▼──────┐
│ component_ │
│ vulns │ (fact)
│ (bridge) │
└────────────────┘
```
### Core Tables
| Table | Type | Purpose |
|-------|------|---------|
| `components` | Dimension | Unified component registry with PURL, supplier, license |
| `artifacts` | Dimension | Container images and applications with SBOM metadata |
| `artifact_components` | Bridge | Links artifacts to their SBOM components |
| `component_vulns` | Fact | Component-to-vulnerability mapping |
| `attestations` | Fact | Attestation metadata (provenance, SBOM, VEX) |
| `vex_overrides` | Fact | VEX status overrides with justifications |
| `raw_sboms` | Audit | Raw SBOM payloads for reprocessing |
| `raw_attestations` | Audit | Raw DSSE envelopes for audit |
| `daily_vulnerability_counts` | Rollup | Daily vuln aggregations |
| `daily_component_counts` | Rollup | Daily component aggregations |
Rollup retention is 90 days in hot storage. `compute_daily_rollups()` prunes
older rows after each run; archival follows operations runbooks.
Platform WebService can automate rollups + materialized view refreshes via
`PlatformAnalyticsMaintenanceService` (see `architecture.md` for schedule and
configuration).
Use `Platform:AnalyticsMaintenance:BackfillDays` to recompute the most recent
N days of rollups on the first maintenance run after downtime (set to `0` to disable).
### Materialized Views
| View | Refresh | Purpose |
|------|---------|---------|
| `mv_supplier_concentration` | Daily | Top suppliers by component count |
| `mv_license_distribution` | Daily | License category distribution |
| `mv_vuln_exposure` | Daily | CVE exposure adjusted by VEX |
| `mv_attestation_coverage` | Daily | Provenance/SLSA coverage by env/team |
Array-valued fields (for example `environments` and `ecosystems`) are ordered
alphabetically to keep analytics outputs deterministic.
## Quick Start
### Day-1 Queries
**Top supplier concentration (supply chain risk, optional environment filter):**
```sql
SELECT analytics.sp_top_suppliers(20, 'prod');
```
**License risk heatmap (optional environment filter):**
```sql
SELECT analytics.sp_license_heatmap('prod');
```
**CVE exposure adjusted by VEX:**
```sql
SELECT analytics.sp_vuln_exposure('prod', 'high');
```
**Fixable vulnerability backlog:**
```sql
SELECT analytics.sp_fixable_backlog('prod');
```
**Attestation coverage gaps:**
```sql
SELECT analytics.sp_attestation_gaps('prod');
```
### API Endpoints
| Endpoint | Method | Description |
|----------|--------|-------------|
| `/api/analytics/suppliers` | GET | Supplier concentration data |
| `/api/analytics/licenses` | GET | License distribution |
| `/api/analytics/vulnerabilities` | GET | CVE exposure (VEX-adjusted) |
| `/api/analytics/backlog` | GET | Fixable vulnerability backlog |
| `/api/analytics/attestation-coverage` | GET | Attestation gaps |
| `/api/analytics/trends/vulnerabilities` | GET | Vulnerability time-series |
| `/api/analytics/trends/components` | GET | Component time-series |
All analytics endpoints require the `analytics.read` scope.
The platform metadata capability `analytics` reports whether analytics storage is configured.
#### Query Parameters
- `/api/analytics/suppliers`: `limit` (optional, default 20), `environment` (optional)
- `/api/analytics/licenses`: `environment` (optional)
- `/api/analytics/vulnerabilities`: `minSeverity` (optional, default `low`), `environment` (optional)
- `/api/analytics/backlog`: `environment` (optional)
- `/api/analytics/attestation-coverage`: `environment` (optional)
- `/api/analytics/trends/vulnerabilities`: `environment` (optional), `days` (optional, default 30)
- `/api/analytics/trends/components`: `environment` (optional), `days` (optional, default 30)
## Ingestion Configuration
Analytics ingestion runs inside the Platform WebService and subscribes to Scanner, Concelier, and Attestor streams. Configure ingestion via `Platform:AnalyticsIngestion`:
```yaml
Platform:
Storage:
PostgresConnectionString: "Host=...;Database=analytics;Username=...;Password=..."
AnalyticsIngestion:
Enabled: true
PostgresConnectionString: "" # optional; defaults to Platform:Storage
AllowedTenants: ["tenant-a", "tenant-b"]
Streams:
ScannerStream: "orchestrator:events"
ConcelierObservationStream: "concelier:advisory.observation.updated:v1"
ConcelierLinksetStream: "concelier:advisory.linkset.updated:v1"
AttestorStream: "attestor:events"
StartFromBeginning: false
Cas:
RootPath: "/var/lib/stellaops/cas"
DefaultBucket: "attestations"
Attestations:
BundleUriTemplate: "bundle:{digest}"
```
Bundle URI templates support:
- `{digest}` for the full digest string (for example `sha256:...`).
- `{hash}` for the raw hex digest (no algorithm prefix).
- `bundle:{digest}` which resolves to `cas://<DefaultBucket>/{digest}` by default.
- `file:/path/to/bundles/bundle-{hash}.json` for offline file ingestion.
For offline workflows, verify bundles with `stella bundle verify` before ingesting them.
## Console UI
SBOM Lake analytics are exposed in the Console under `Analytics > SBOM Lake` (`/analytics/sbom-lake`).
Console access requires `ui.read` plus `analytics.read` scopes.
Key UI features:
- Filters for environment, minimum severity, and time window.
- Panels for suppliers, licenses, vulnerability exposure, and attestation coverage.
- Trend views for vulnerabilities and components.
- Fixable backlog table with CSV export.
See [console.md](./console.md) for operator guidance and filter behavior.
## CLI Access
SBOM lake analytics are exposed via the CLI under `stella analytics sbom-lake`
(requires `analytics.read` scope).
```bash
# Top suppliers
stella analytics sbom-lake suppliers --limit 20
# Vulnerability exposure in prod (high+), CSV export
stella analytics sbom-lake vulnerabilities --environment prod --min-severity high --format csv --output vuln.csv
# 30-day trends for both series
stella analytics sbom-lake trends --days 30 --series all --format json
```
See `docs/modules/cli/guides/commands/analytics.md` for command-level details.
## Architecture
See [architecture.md](./architecture.md) for detailed design decisions, data flow, and normalization rules.
## Schema Reference
See [analytics_schema.sql](../../db/analytics_schema.sql) for complete DDL including:
- Table definitions with indexes
- Normalization functions
- Materialized views
- Stored procedures
- Refresh procedures
## Sprint Reference
Implementation tracked in:
- `docs/implplan/SPRINT_20260120_030_Platform_sbom_analytics_lake.md`
- `docs/implplan/SPRINT_20260120_032_Cli_sbom_analytics_cli.md`

298
docs-archived/modules/analytics/architecture.md Normal file
View File

@@ -0,0 +1,298 @@
# Analytics Module Architecture
> **Implementation Note:** Analytics is a cross-cutting feature integrated into the **Platform WebService** (`src/Platform/`). There is no standalone `src/Analytics/` module. Data ingestion pipelines span Scanner, Concelier, and Attestor modules. See [Platform Architecture](../platform/architecture-overview.md) for service-level integration details.
## Design Philosophy
The Analytics module implements a **star-schema data warehouse** pattern optimized for analytical queries rather than transactional workloads. Key design principles:
1. **Separation of concerns**: Analytics schema is isolated from operational schemas (scanner, vex, proof_system)
2. **Pre-computation**: Expensive aggregations computed in advance via materialized views
3. **Audit trail**: Raw payloads preserved for reprocessing and compliance
4. **Determinism**: Normalization functions are immutable and reproducible; array aggregates are ordered for stable outputs
5. **Incremental updates**: Supports both full refresh and incremental ingestion
## Data Flow
```
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ Scanner │ │ Concelier │ │ Attestor │
│ (SBOM) │ │ (Vuln) │ │ (DSSE) │
└──────┬──────┘ └──────┬──────┘ └──────┬──────┘
│ │ │
│ SBOM Ingested │ Vuln Updated │ Attestation Created
▼ ▼ ▼
┌──────────────────────────────────────────────────────┐
│ AnalyticsIngestionService │
│ - Normalize components (PURL, supplier, license) │
│ - Upsert to unified registry │
│ - Correlate with vulnerabilities │
│ - Store raw payloads │
└──────────────────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────┐
│ analytics schema │
│ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌────────────┐ │
│ │components│ │artifacts│ │comp_vuln│ │attestations│ │
│ └─────────┘ └─────────┘ └─────────┘ └────────────┘ │
└──────────────────────────────────────────────────────┘
│
│ Daily refresh
▼
┌──────────────────────────────────────────────────────┐
│ Materialized Views │
│ mv_supplier_concentration | mv_license_distribution │
│ mv_vuln_exposure | mv_attestation_coverage │
└──────────────────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────┐
│ Platform API Endpoints │
│ (with 5-minute caching) │
└──────────────────────────────────────────────────────┘
```
## Normalization Rules
### PURL Parsing
Package URLs (PURLs) are the canonical identifier for components. The `parse_purl()` function extracts:
| Field | Example | Notes |
|-------|---------|-------|
| `purl_type` | `maven`, `npm`, `pypi` | Ecosystem identifier |
| `purl_namespace` | `org.apache.logging` | Group/org/scope (optional) |
| `purl_name` | `log4j-core` | Package name |
| `purl_version` | `2.17.1` | Version string |
### Supplier Normalization
The `normalize_supplier()` function standardizes supplier names for consistent grouping:
1. Convert to lowercase
2. Trim whitespace
3. Remove legal suffixes: Inc., LLC, Ltd., Corp., GmbH, B.V., S.A., PLC, Co.
4. Normalize internal whitespace
**Examples:**
- `"Apache Software Foundation, Inc."` → `"apache software foundation"`
- `"Google LLC"` → `"google"`
- `" Microsoft Corp. "` → `"microsoft"`
### License Categorization
The `categorize_license()` function maps SPDX expressions to risk categories:
| Category | Examples | Risk Level |
|----------|----------|------------|
| `permissive` | MIT, Apache-2.0, BSD-3-Clause, ISC | Low |
| `copyleft-weak` | LGPL-2.1, MPL-2.0, EPL-2.0 | Medium |
| `copyleft-strong` | GPL-3.0, AGPL-3.0, SSPL | High |
| `proprietary` | Proprietary, Commercial | Review Required |
| `unknown` | Unrecognized expressions | Review Required |
**Special handling:**
- GPL with exceptions (e.g., `GPL-2.0 WITH Classpath-exception-2.0`) → `copyleft-weak`
- Dual-licensed (e.g., `MIT OR Apache-2.0`) → uses first match
## Component Deduplication
Components are deduplicated by `(purl, hash_sha256)`:
1. If same PURL and hash: existing record updated (last_seen_at, counts)
2. If same PURL but different hash: new record created (version change)
3. If same hash but different PURL: new record (aliased package)
**Upsert pattern:**
```sql
INSERT INTO analytics.components (...)
VALUES (...)
ON CONFLICT (purl, hash_sha256) DO UPDATE SET
last_seen_at = now(),
sbom_count = components.sbom_count + 1,
updated_at = now();
```
## Vulnerability Correlation
When a component is upserted, the `VulnerabilityCorrelationService` queries Concelier for matching advisories:
1. Query by PURL type + namespace + name
2. Filter by version range matching
3. Upsert to `component_vulns` with severity, EPSS, KEV flags
**Version range matching** currently supports semver ranges and exact matches via
`VersionRuleEvaluator`. Non-semver schemes fall back to exact string matches; wildcard
and ecosystem-specific ranges require upstream normalization.
## VEX Override Logic
The `mv_vuln_exposure` view implements VEX-adjusted counts:
```sql
-- Effective count excludes artifacts with active VEX overrides
COUNT(DISTINCT ac.artifact_id) FILTER (
WHERE NOT EXISTS (
SELECT 1 FROM analytics.vex_overrides vo
WHERE vo.artifact_id = ac.artifact_id
AND vo.vuln_id = cv.vuln_id
AND vo.status = 'not_affected'
AND (vo.valid_until IS NULL OR vo.valid_until > now())
)
) AS effective_artifact_count
```
**Override validity:**
- `valid_from`: When the override became effective
- `valid_until`: Expiration (NULL = no expiration)
- Only `status = 'not_affected'` reduces exposure counts, and only when the override is active in its validity window.
## Attestation Ingestion
Attestation ingestion consumes Attestor Rekor entry events and expects Sigstore bundles
or raw DSSE envelopes. The ingestion service:
- Resolves bundle URIs using `BundleUriTemplate`; `bundle:{digest}` maps to
`cas://<DefaultBucket>/{digest}` by default.
- Decodes DSSE payloads, computes `dsse_payload_hash`, and records `predicate_uri` plus
Rekor log metadata (`rekor_log_id`, `rekor_log_index`).
- Uses in-toto `subject` digests to link artifacts when reanalysis hints are absent.
- Maps predicate URIs into `analytics_attestation_type` values
(`provenance`, `sbom`, `vex`, `build`, `scan`, `policy`).
- Expands VEX statements into `vex_overrides` rows, one per product reference, and
captures optional validity timestamps when provided.
## Time-Series Rollups
Daily rollups computed by `compute_daily_rollups()`:
**Vulnerability counts** (per environment/team/severity):
- `total_vulns`: All affecting vulnerabilities
- `fixable_vulns`: Vulns with `fix_available = TRUE`
- `vex_mitigated`: Vulns with active `not_affected` override
- `kev_vulns`: Vulns in CISA KEV
- `unique_cves`: Distinct CVE IDs
- `affected_artifacts`: Artifacts containing affected components
- `affected_components`: Components with affecting vulns
**Component counts** (per environment/team/license/type):
- `total_components`: Distinct components
- `unique_suppliers`: Distinct normalized suppliers
**Retention policy:** 90 days in hot storage; `compute_daily_rollups()` prunes older rows and downstream jobs archive to cold storage.
## Materialized View Refresh
All materialized views support `REFRESH ... CONCURRENTLY` for zero-downtime updates:
```sql
-- Refresh all views (non-concurrent; run off-peak)
SELECT analytics.refresh_all_views();
```
**Refresh schedule (recommended):**
- `mv_supplier_concentration`: 02:00 UTC daily
- `mv_license_distribution`: 02:15 UTC daily
- `mv_vuln_exposure`: 02:30 UTC daily
- `mv_attestation_coverage`: 02:45 UTC daily
- `compute_daily_rollups()`: 03:00 UTC daily
Platform WebService can run the daily rollup + refresh loop via
`PlatformAnalyticsMaintenanceService`. Configure the schedule with:
- `Platform:AnalyticsMaintenance:Enabled` (default `true`)
- `Platform:AnalyticsMaintenance:IntervalMinutes` (default `1440`)
- `Platform:AnalyticsMaintenance:RunOnStartup` (default `true`)
- `Platform:AnalyticsMaintenance:ComputeDailyRollups` (default `true`)
- `Platform:AnalyticsMaintenance:RefreshMaterializedViews` (default `true`)
- `Platform:AnalyticsMaintenance:BackfillDays` (default `0`, set to `0` to disable; recompute the most recent N days on the first maintenance run)
The hosted service issues concurrent refresh statements directly for each view.
Use a DB scheduler (pg_cron) or external orchestrator if you need the staggered
per-view timing above.
## Performance Considerations
### Indexing Strategy
| Table | Key Indexes | Query Pattern |
|-------|-------------|---------------|
| `components` | `purl`, `supplier_normalized`, `license_category` | Lookup, aggregation |
| `artifacts` | `digest`, `environment`, `team` | Lookup, filtering |
| `component_vulns` | `vuln_id`, `severity`, `fix_available` | Join, filtering |
| `attestations` | `artifact_id`, `predicate_type` | Join, aggregation |
| `vex_overrides` | `(artifact_id, vuln_id)`, `status` | Subquery exists |
### Query Performance Targets
| Query | Target | Notes |
|-------|--------|-------|
| `sp_top_suppliers(20, 'prod')` | < 100ms | Uses materialized view when env is null; env filter reads base tables |
| `sp_license_heatmap('prod')` | < 100ms | Uses materialized view when env is null; env filter reads base tables |
| `sp_vuln_exposure()` | < 200ms | Uses materialized view for global queries; environment filters read base tables |
| `sp_fixable_backlog()` | < 500ms | Live query with indexes |
| `sp_attestation_gaps()` | < 100ms | Uses materialized view |
### Caching Strategy
Platform API endpoints use a 5-minute TTL cache:
- Cache key: endpoint + query parameters
- Invalidation: Time-based only (no event-driven invalidation)
- Storage: Valkey (in-memory)
## Security Considerations
### Schema Permissions
```sql
-- Read-only role for dashboards
GRANT USAGE ON SCHEMA analytics TO dashboard_reader;
GRANT SELECT ON ALL TABLES IN SCHEMA analytics TO dashboard_reader;
GRANT SELECT ON ALL SEQUENCES IN SCHEMA analytics TO dashboard_reader;
-- Write role for ingestion service
GRANT USAGE ON SCHEMA analytics TO analytics_writer;
GRANT SELECT, INSERT, UPDATE ON ALL TABLES IN SCHEMA analytics TO analytics_writer;
GRANT USAGE, SELECT ON ALL SEQUENCES IN SCHEMA analytics TO analytics_writer;
```
### Data Classification
| Table | Classification | Notes |
|-------|----------------|-------|
| `components` | Internal | Contains package names, versions |
| `artifacts` | Internal | Contains image names, team names |
| `component_vulns` | Internal | Vulnerability data (public CVEs) |
| `vex_overrides` | Confidential | Contains justifications, operator IDs |
| `raw_sboms` | Confidential | Full SBOM payloads |
| `raw_attestations` | Confidential | Signed attestation envelopes |
### Audit Trail
All tables include `created_at` and `updated_at` timestamps. Raw payload tables (`raw_sboms`, `raw_attestations`) are append-only with content hashes for integrity verification.
## Integration Points
### Upstream Dependencies
| Service | Event | Contract | Action |
|---------|-------|----------|--------|
| Scanner | SBOM report ready | `scanner.event.report.ready@1` (`docs/modules/signals/events/orchestrator-scanner-events.md`) | Normalize and upsert components |
| Concelier | Advisory observation/linkset updated | `advisory.observation.updated@1` (`docs/modules/concelier/events/advisory.observation.updated@1.schema.json`), `advisory.linkset.updated@1` (`docs/modules/concelier/events/advisory.linkset.updated@1.md`) | Re-correlate affected components |
| Excititor | VEX statement changes | `vex.statement.*` (`docs/modules/excititor/architecture.md`) | Create/update vex_overrides |
| Attestor | Rekor entry logged | `rekor.entry.logged` (`docs/modules/attestor/architecture.md`) | Upsert attestation record |
### Downstream Consumers
| Consumer | Data | Endpoint |
|----------|------|----------|
| Console UI | Dashboard data | `/api/analytics/*` |
| Export Center | Compliance reports | Direct DB query |
| AdvisoryAI | Risk context | `/api/analytics/vulnerabilities` |
## Future Enhancements
1. **Partitioning**: Partition `daily_*` tables by date for faster queries and archival
2. **Incremental refresh**: Implement incremental materialized view refresh for large datasets
3. **Custom dimensions**: Support user-defined component groupings (business units, cost centers)
4. **Predictive analytics**: Add ML-based risk prediction using historical trends
5. **BI tool integration**: Direct connectors for Tableau, Looker, Metabase

64
docs-archived/modules/analytics/console.md Normal file
View File

@@ -0,0 +1,64 @@
# Analytics Console (SBOM Lake)
The Console exposes SBOM analytics lake data under `Analytics > SBOM Lake`.
This view is read-only and uses the analytics API endpoints documented in `docs/modules/analytics/README.md`.
## Access
- Route: `/analytics/sbom-lake`
- Required scopes: `ui.read` and `analytics.read`
- Console admin bundles: `role/analytics-viewer`, `role/analytics-operator`, `role/analytics-admin`
- Data freshness: the page surfaces the latest `dataAsOf` timestamp returned by the API.
## Filters
The SBOM Lake page supports three filters that round-trip via URL query parameters:
- Environment: `env` (optional, example: `Prod`)
- Minimum severity: `severity` (optional, example: `high`)
- Time window (days): `days` (optional, example: `90`)
When a filter changes, the Console reloads all panels using the updated parameters.
Supplier and license panels honor the environment filter alongside the other views.
## Panels
The dashboard presents four summary panels:
1. Supplier concentration (top suppliers by component count)
2. License distribution (license categories and counts)
3. Vulnerability exposure (top CVEs after VEX adjustments)
4. Attestation coverage (provenance and SLSA 2+ coverage)
Each panel shows a loading state, empty state, and summary counts.
## Trends
Two trend panels are included:
- Vulnerability trend: net exposure over the selected time window
- Component trend: total components and unique suppliers
The Console aggregates trend points by date and renders a simple bar chart plus a compact list.
## Fixable Backlog
The fixable backlog table lists vulnerabilities with fixes available, grouped by component and service.
The "Top backlog components" table derives a component summary from the same backlog data.
### CSV Export
The "Export backlog CSV" action downloads a deterministic, ordered CSV with:
- Service
- Component
- Version
- Vulnerability
- Severity
- Environment
- Fixed version
## Troubleshooting
- If panels show "No data", verify that the analytics schema and materialized views are populated.
- If an error banner appears, check the analytics API availability and ensure the tenant has `analytics.read`.

422
docs-archived/modules/analytics/queries.md Normal file
View File

@@ -0,0 +1,422 @@
# Analytics Query Library
This document provides ready-to-use SQL queries for common analytics use cases. All queries are optimized for the analytics star schema.
## Executive Dashboard Queries
### 1. Top Supplier Concentration (Supply Chain Risk)
Identifies suppliers with the highest component footprint, indicating supply chain concentration risk.
```sql
-- Via stored procedure (recommended, optional environment filter)
SELECT analytics.sp_top_suppliers(20, 'prod');
-- Direct query
SELECT
supplier,
component_count,
artifact_count,
team_count,
critical_vuln_count,
high_vuln_count,
environments
FROM analytics.mv_supplier_concentration
ORDER BY component_count DESC
LIMIT 20;
```
**Use case**: Identify vendors that, if compromised, would affect the most artifacts.
### 2. License Risk Heatmap
Shows distribution of components by license category for compliance review.
```sql
-- Via stored procedure (optional environment filter)
SELECT analytics.sp_license_heatmap('prod');
-- Direct query with grouping
SELECT
license_category,
SUM(component_count) AS total_components,
SUM(artifact_count) AS total_artifacts,
COUNT(DISTINCT license_concluded) AS unique_licenses
FROM analytics.mv_license_distribution
GROUP BY license_category
ORDER BY
CASE license_category
WHEN 'copyleft-strong' THEN 1
WHEN 'proprietary' THEN 2
WHEN 'unknown' THEN 3
WHEN 'copyleft-weak' THEN 4
ELSE 5
END;
```
**Use case**: Compliance review, identify components requiring legal review.
### 3. CVE Exposure Adjusted by VEX
Shows true vulnerability exposure after applying VEX mitigations.
```sql
-- Via stored procedure
SELECT analytics.sp_vuln_exposure('prod', 'high');
-- Direct query showing VEX effectiveness (global view; use sp_vuln_exposure for environment filtering)
SELECT
vuln_id,
severity::TEXT,
cvss_score,
epss_score,
kev_listed,
fix_available,
raw_artifact_count AS total_affected,
effective_artifact_count AS actually_affected,
raw_artifact_count - effective_artifact_count AS vex_mitigated,
ROUND(100.0 * (raw_artifact_count - effective_artifact_count) / NULLIF(raw_artifact_count, 0), 1) AS mitigation_rate
FROM analytics.mv_vuln_exposure
WHERE effective_artifact_count > 0
ORDER BY
CASE severity
WHEN 'critical' THEN 1
WHEN 'high' THEN 2
WHEN 'medium' THEN 3
ELSE 4
END,
effective_artifact_count DESC
LIMIT 50;
```
**Use case**: Show executives the "real" risk after VEX assessment.
### 4. Fixable Vulnerability Backlog
Lists vulnerabilities that can be fixed today (fix available, not VEX-mitigated).
```sql
-- Via stored procedure
SELECT analytics.sp_fixable_backlog('prod');
-- Direct query with priority scoring
SELECT
a.name AS service,
a.environment,
a.team,
c.name AS component,
c.version AS current_version,
cv.vuln_id,
cv.severity::TEXT,
cv.cvss_score,
cv.epss_score,
cv.fixed_version,
cv.kev_listed,
-- Priority score: higher = fix first
(
CASE cv.severity
WHEN 'critical' THEN 100
WHEN 'high' THEN 75
WHEN 'medium' THEN 50
ELSE 25
END
+ COALESCE(cv.epss_score * 100, 0)
+ (CASE WHEN cv.kev_listed THEN 50 ELSE 0 END)
)::INT AS priority_score
FROM analytics.component_vulns cv
JOIN analytics.components c ON c.component_id = cv.component_id
JOIN analytics.artifact_components ac ON ac.component_id = c.component_id
JOIN analytics.artifacts a ON a.artifact_id = ac.artifact_id
LEFT JOIN analytics.vex_overrides vo ON vo.artifact_id = a.artifact_id
AND vo.vuln_id = cv.vuln_id
AND vo.status = 'not_affected'
AND vo.valid_from <= now()
AND (vo.valid_until IS NULL OR vo.valid_until > now())
WHERE cv.affects = TRUE
AND cv.fix_available = TRUE
AND vo.override_id IS NULL
AND a.environment = 'prod'
ORDER BY priority_score DESC, a.name
LIMIT 100;
```
**Use case**: Prioritize remediation work based on risk and fixability.
### 5. Build Integrity / Attestation Coverage
Shows attestation gaps by environment and team.
```sql
-- Via stored procedure
SELECT analytics.sp_attestation_gaps('prod');
-- Direct query with gap analysis
SELECT
environment,
team,
total_artifacts,
with_provenance,
total_artifacts - with_provenance AS missing_provenance,
provenance_pct,
slsa_level_2_plus,
slsa2_pct,
with_sbom_attestation,
with_vex_attestation
FROM analytics.mv_attestation_coverage
WHERE environment = 'prod'
ORDER BY provenance_pct ASC;
```
**Use case**: Identify teams/environments not meeting attestation requirements.
## Trend Analysis Queries
### 6. Vulnerability Trend (30 Days)
```sql
SELECT
snapshot_date,
environment,
SUM(total_vulns) AS total_vulns,
SUM(fixable_vulns) AS fixable_vulns,
SUM(vex_mitigated) AS vex_mitigated,
SUM(total_vulns) - SUM(vex_mitigated) AS net_exposure,
SUM(kev_vulns) AS kev_vulns
FROM analytics.daily_vulnerability_counts
WHERE snapshot_date >= CURRENT_DATE - INTERVAL '30 days'
GROUP BY snapshot_date, environment
ORDER BY environment, snapshot_date;
```
### 7. Vulnerability Trend by Severity
```sql
SELECT
snapshot_date,
severity::TEXT,
SUM(total_vulns) AS total_vulns
FROM analytics.daily_vulnerability_counts
WHERE snapshot_date >= CURRENT_DATE - INTERVAL '30 days'
AND environment = 'prod'
GROUP BY snapshot_date, severity
ORDER BY snapshot_date,
CASE severity
WHEN 'critical' THEN 1
WHEN 'high' THEN 2
WHEN 'medium' THEN 3
ELSE 4
END;
```
### 8. Component Growth Trend
```sql
SELECT
snapshot_date,
environment,
SUM(total_components) AS total_components,
SUM(unique_suppliers) AS unique_suppliers
FROM analytics.daily_component_counts
WHERE snapshot_date >= CURRENT_DATE - INTERVAL '30 days'
GROUP BY snapshot_date, environment
ORDER BY environment, snapshot_date;
```
## Deep-Dive Queries
### 9. Component Impact Analysis
Find all artifacts affected by a specific component.
```sql
SELECT
a.name AS artifact,
a.version,
a.environment,
a.team,
ac.depth AS dependency_depth,
ac.introduced_via
FROM analytics.components c
JOIN analytics.artifact_components ac ON ac.component_id = c.component_id
JOIN analytics.artifacts a ON a.artifact_id = ac.artifact_id
WHERE c.purl LIKE 'pkg:maven/org.apache.logging.log4j/log4j-core%'
ORDER BY a.environment, a.name;
```
### 10. CVE Impact Analysis
Find all artifacts affected by a specific CVE.
```sql
SELECT DISTINCT
a.name AS artifact,
a.version,
a.environment,
a.team,
c.name AS component,
c.version AS component_version,
cv.cvss_score,
cv.fixed_version,
CASE
WHEN vo.status = 'not_affected' THEN 'VEX Mitigated'
WHEN cv.fix_available THEN 'Fix Available'
ELSE 'Vulnerable'
END AS status
FROM analytics.component_vulns cv
JOIN analytics.components c ON c.component_id = cv.component_id
JOIN analytics.artifact_components ac ON ac.component_id = c.component_id
JOIN analytics.artifacts a ON a.artifact_id = ac.artifact_id
LEFT JOIN analytics.vex_overrides vo ON vo.artifact_id = a.artifact_id
AND vo.vuln_id = cv.vuln_id
AND vo.valid_from <= now()
AND (vo.valid_until IS NULL OR vo.valid_until > now())
WHERE cv.vuln_id = 'CVE-2021-44228'
ORDER BY a.environment, a.name;
```
### 11. Supplier Vulnerability Profile
Detailed vulnerability breakdown for a specific supplier.
```sql
SELECT
c.supplier_normalized AS supplier,
c.name AS component,
c.version,
cv.vuln_id,
cv.severity::TEXT,
cv.cvss_score,
cv.kev_listed,
cv.fix_available,
cv.fixed_version
FROM analytics.components c
JOIN analytics.component_vulns cv ON cv.component_id = c.component_id
WHERE c.supplier_normalized = 'apache software foundation'
AND cv.affects = TRUE
ORDER BY
CASE cv.severity
WHEN 'critical' THEN 1
WHEN 'high' THEN 2
ELSE 3
END,
cv.cvss_score DESC;
```
### 12. License Compliance Report
Components with concerning licenses in production.
```sql
SELECT
c.name AS component,
c.version,
c.license_concluded,
c.license_category::TEXT,
c.supplier_normalized AS supplier,
COUNT(DISTINCT a.artifact_id) AS artifact_count,
ARRAY_AGG(DISTINCT a.name ORDER BY a.name) AS affected_artifacts
FROM analytics.components c
JOIN analytics.artifact_components ac ON ac.component_id = c.component_id
JOIN analytics.artifacts a ON a.artifact_id = ac.artifact_id
WHERE c.license_category IN ('copyleft-strong', 'proprietary', 'unknown')
AND a.environment = 'prod'
GROUP BY c.component_id, c.name, c.version, c.license_concluded, c.license_category, c.supplier_normalized
ORDER BY c.license_category, artifact_count DESC;
```
### 13. MTTR Analysis
Mean time to remediate by severity.
```sql
SELECT
cv.severity::TEXT,
COUNT(*) AS remediated_vulns,
AVG(EXTRACT(EPOCH FROM (vo.valid_from - cv.published_at)) / 86400)::NUMERIC(10,2) AS avg_days_to_mitigate,
PERCENTILE_CONT(0.5) WITHIN GROUP (
ORDER BY EXTRACT(EPOCH FROM (vo.valid_from - cv.published_at)) / 86400
)::NUMERIC(10,2) AS median_days,
PERCENTILE_CONT(0.9) WITHIN GROUP (
ORDER BY EXTRACT(EPOCH FROM (vo.valid_from - cv.published_at)) / 86400
)::NUMERIC(10,2) AS p90_days
FROM analytics.component_vulns cv
JOIN analytics.vex_overrides vo ON vo.vuln_id = cv.vuln_id
AND vo.status = 'not_affected'
AND vo.valid_from <= now()
AND (vo.valid_until IS NULL OR vo.valid_until > now())
WHERE cv.published_at >= now() - INTERVAL '90 days'
AND cv.published_at IS NOT NULL
GROUP BY cv.severity
ORDER BY
CASE cv.severity
WHEN 'critical' THEN 1
WHEN 'high' THEN 2
WHEN 'medium' THEN 3
ELSE 4
END;
```
### 14. Transitive Dependency Risk
Components introduced through transitive dependencies.
```sql
SELECT
c.name AS transitive_component,
c.version,
ac.introduced_via AS direct_dependency,
ac.depth,
COUNT(DISTINCT cv.vuln_id) AS vuln_count,
SUM(CASE WHEN cv.severity = 'critical' THEN 1 ELSE 0 END) AS critical_count,
COUNT(DISTINCT a.artifact_id) AS affected_artifacts
FROM analytics.components c
JOIN analytics.artifact_components ac ON ac.component_id = c.component_id
JOIN analytics.artifacts a ON a.artifact_id = ac.artifact_id
LEFT JOIN analytics.component_vulns cv ON cv.component_id = c.component_id AND cv.affects = TRUE
WHERE ac.depth > 0 -- Transitive only
AND a.environment = 'prod'
GROUP BY c.component_id, c.name, c.version, ac.introduced_via, ac.depth
HAVING COUNT(cv.vuln_id) > 0
ORDER BY critical_count DESC, vuln_count DESC
LIMIT 50;
```
### 15. VEX Effectiveness Report
How effective is the VEX program at reducing noise?
```sql
SELECT
DATE_TRUNC('week', vo.created_at)::DATE AS week,
COUNT(*) AS total_overrides,
COUNT(*) FILTER (WHERE vo.status = 'not_affected') AS not_affected,
COUNT(*) FILTER (WHERE vo.status = 'affected') AS confirmed_affected,
COUNT(*) FILTER (WHERE vo.status = 'under_investigation') AS under_investigation,
COUNT(*) FILTER (WHERE vo.status = 'fixed') AS marked_fixed,
-- Noise reduction rate
ROUND(100.0 * COUNT(*) FILTER (WHERE vo.status = 'not_affected') / NULLIF(COUNT(*), 0), 1) AS noise_reduction_pct
FROM analytics.vex_overrides vo
WHERE vo.created_at >= now() - INTERVAL '90 days'
GROUP BY DATE_TRUNC('week', vo.created_at)
ORDER BY week;
```
## Performance Tips
1. **Use materialized views**: Queries prefixed with `mv_` are pre-computed and fast
2. **Add environment filter**: Most queries benefit from `WHERE environment = 'prod'`
3. **Use stored procedures**: `sp_*` functions return JSON and handle caching
4. **Limit results**: Always use `LIMIT` for large result sets
5. **Check refresh times**: Views are refreshed daily; data may be up to 24h stale
## Query Parameters
Common filter parameters:
| Parameter | Type | Example | Notes |
|-----------|------|---------|-------|
| `environment` | TEXT | `'prod'`, `'stage'` | Filter by deployment environment |
| `team` | TEXT | `'platform'` | Filter by owning team |
| `severity` | TEXT | `'critical'`, `'high'` | Minimum severity level |
| `days` | INT | `30`, `90` | Lookback period |
| `limit` | INT | `20`, `100` | Max results |

54
docs-archived/modules/benchmark/README.md Normal file
View File

@@ -0,0 +1,54 @@
# Benchmark
> **Dual Purpose:** This documentation covers two aspects:
> - **Performance Benchmarking** (Production) — BenchmarkDotNet harnesses in `src/Bench/` for scanner, policy, and notification performance testing
> - **Competitive Benchmarking** (Planned) — Accuracy comparison framework in `src/Scanner/__Libraries/StellaOps.Scanner.Benchmark/`
**Status:** Implemented
**Source:** `src/Bench/`
**Owner:** Platform Team
## Purpose
Benchmark provides performance testing and regression analysis for StellaOps components. Ensures deterministic scan times, throughput validation, and performance profiling for critical paths (scanning, policy evaluation, SBOM generation).
## Components
**Services:**
- `StellaOps.Bench` - Benchmarking harness with BenchmarkDotNet integration
**Key Features:**
- Scanner performance benchmarks (per-analyzer, full-scan)
- Policy engine evaluation latency tests
- SBOM generation throughput tests
- Database query performance profiling
- Determinism validation (output stability)
## Configuration
Benchmark configuration via BenchmarkDotNet attributes and runtime parameters.
Key settings:
- Benchmark filters and categories
- Iterations and warmup counts
- Memory profiling and allocation tracking
- Export formats (JSON, HTML, Markdown)
## Dependencies
- BenchmarkDotNet framework
- Scanner (benchmark targets)
- Policy Engine (benchmark targets)
- SbomService (benchmark targets)
- Test fixtures and datasets
## Related Documentation
- Architecture: `./architecture.md`
- Scanner: `../scanner/`
- Policy: `../policy/`
- Operations: `./operations/` (if exists)
## Current Status
Implemented with BenchmarkDotNet harness. Provides performance baselines for scanner analyzers, policy evaluation, and SBOM generation. Used for regression detection in CI/CD pipeline.

447
docs-archived/modules/benchmark/architecture.md Normal file
View File

@@ -0,0 +1,447 @@
# Benchmark Module Architecture
## Overview
The Benchmark module provides infrastructure for validating and demonstrating Stella Ops' competitive advantages through automated comparison against other container security scanners (Trivy, Grype, Syft, etc.).
**Module Path**: `src/Scanner/__Libraries/StellaOps.Scanner.Benchmark/`
**Status**: PLANNED (Sprint 7000.0001.0001)
> **Note:** This module focuses on **competitive benchmarking** (accuracy comparison with other scanners). For **performance benchmarks** of StellaOps modules (LinkNotMerge, Notify, PolicyEngine, Scanner.Analyzers), see `src/Bench/`.
---
## Mission
Establish verifiable, reproducible benchmarks that:
1. Validate competitive claims with evidence
2. Detect regressions in accuracy or performance
3. Generate marketing-ready comparison materials
4. Provide ground-truth corpus for testing
---
## Architecture
```
┌─────────────────────────────────────────────────────────────────┐
│ Benchmark Module │
├─────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ Corpus │ │ Harness │ │ Metrics │ │
│ │ Manager │───▶│ Runner │───▶│ Calculator │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ │
│ │ │ │ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │Ground Truth │ │ Competitor │ │ Claims │ │
│ │ Manifest │ │ Adapters │ │ Index │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────┘
```
---
## Components
### 1. Corpus Manager
**Namespace**: `StellaOps.Scanner.Benchmark.Corpus`
Manages the ground-truth corpus of container images with known vulnerabilities.
```csharp
public interface ICorpusManager
{
Task<Corpus> LoadCorpusAsync(string corpusPath, CancellationToken ct);
Task<CorpusImage> GetImageAsync(string digest, CancellationToken ct);
Task<GroundTruth> GetGroundTruthAsync(string digest, CancellationToken ct);
}
public record Corpus(
string Version,
DateTimeOffset CreatedAt,
ImmutableArray<CorpusImage> Images
);
public record CorpusImage(
string Digest,
string Name,
string Tag,
CorpusCategory Category,
GroundTruth GroundTruth
);
public record GroundTruth(
ImmutableArray<string> TruePositives,
ImmutableArray<string> KnownFalsePositives,
ImmutableArray<string> Notes
);
public enum CorpusCategory
{
BaseOS, // Alpine, Debian, Ubuntu, RHEL
ApplicationNode, // Node.js applications
ApplicationPython,// Python applications
ApplicationJava, // Java applications
ApplicationDotNet,// .NET applications
BackportScenario, // Known backported fixes
Unreachable // Known unreachable vulns
}
```
### 2. Harness Runner
**Namespace**: `StellaOps.Scanner.Benchmark.Harness`
Executes scans using Stella Ops and competitor tools.
```csharp
public interface IHarnessRunner
{
Task<BenchmarkRun> RunAsync(
Corpus corpus,
ImmutableArray<ITool> tools,
BenchmarkOptions options,
CancellationToken ct
);
}
public interface ITool
{
string Name { get; }
string Version { get; }
Task<ToolResult> ScanAsync(string imageRef, CancellationToken ct);
}
public record BenchmarkRun(
string RunId,
DateTimeOffset StartedAt,
DateTimeOffset CompletedAt,
ImmutableArray<ToolResult> Results
);
public record ToolResult(
string ToolName,
string ToolVersion,
string ImageDigest,
ImmutableArray<NormalizedFinding> Findings,
TimeSpan Duration
);
```
### 3. Competitor Adapters
**Namespace**: `StellaOps.Scanner.Benchmark.Adapters`
Normalize output from competitor tools.
```csharp
public interface ICompetitorAdapter : ITool
{
Task<ImmutableArray<NormalizedFinding>> ParseOutputAsync(
string output,
CancellationToken ct
);
}
// Implementations
public class TrivyAdapter : ICompetitorAdapter { }
public class GrypeAdapter : ICompetitorAdapter { }
public class SyftAdapter : ICompetitorAdapter { }
public class StellaOpsAdapter : ICompetitorAdapter { }
```
### 4. Metrics Calculator
**Namespace**: `StellaOps.Scanner.Benchmark.Metrics`
Calculate precision, recall, F1, and other metrics.
```csharp
public interface IMetricsCalculator
{
BenchmarkMetrics Calculate(
ToolResult result,
GroundTruth groundTruth
);
ComparativeMetrics Compare(
BenchmarkMetrics baseline,
BenchmarkMetrics comparison
);
}
public record BenchmarkMetrics(
int TruePositives,
int FalsePositives,
int TrueNegatives,
int FalseNegatives,
double Precision,
double Recall,
double F1Score,
ImmutableDictionary<string, BenchmarkMetrics> ByCategory
);
public record ComparativeMetrics(
string BaselineTool,
string ComparisonTool,
double PrecisionDelta,
double RecallDelta,
double F1Delta,
ImmutableArray<string> UniqueFindings,
ImmutableArray<string> MissedFindings
);
```
### 5. Claims Index
**Namespace**: `StellaOps.Scanner.Benchmark.Claims`
Manage verifiable claims with evidence links.
```csharp
public interface IClaimsIndex
{
Task<ImmutableArray<Claim>> GetAllClaimsAsync(CancellationToken ct);
Task<ClaimVerification> VerifyClaimAsync(string claimId, CancellationToken ct);
Task UpdateClaimsAsync(BenchmarkRun run, CancellationToken ct);
}
public record Claim(
string Id,
ClaimCategory Category,
string Statement,
string EvidencePath,
ClaimStatus Status,
DateTimeOffset LastVerified
);
public enum ClaimStatus { Pending, Verified, Published, Disputed, Resolved }
public record ClaimVerification(
string ClaimId,
bool IsValid,
string? Evidence,
string? FailureReason
);
```
---
## Data Flow
```
┌────────────────┐
│ Corpus Images │
│ (50+ images) │
└───────┬────────┘
│
â–¼
┌────────────────┐ ┌────────────────┐
│ Stella Ops Scan│ │ Trivy/Grype │
│ │ │ Scan │
└───────┬────────┘ └───────┬────────┘
│ │
â–¼ â–¼
┌────────────────┐ ┌────────────────┐
│ Normalized │ │ Normalized │
│ Findings │ │ Findings │
└───────┬────────┘ └───────┬────────┘
│ │
└──────────┬───────────┘
│
â–¼
┌──────────────┐
│ Ground Truth │
│ Comparison │
└──────┬───────┘
│
â–¼
┌──────────────┐
│ Metrics │
│ (P/R/F1) │
└──────┬───────┘
│
â–¼
┌──────────────┐
│ Claims Index │
│ Update │
└──────────────┘
```
---
## Corpus Structure
```
bench/competitors/
├── corpus/
│ ├── manifest.json # Corpus metadata
│ ├── ground-truth/
│ │ ├── alpine-3.18.json # Per-image ground truth
│ │ ├── debian-bookworm.json
│ │ └── ...
│ └── images/
│ ├── base-os/
│ ├── applications/
│ └── edge-cases/
├── results/
│ ├── 2025-12-22/
│ │ ├── stellaops.json
│ │ ├── trivy.json
│ │ ├── grype.json
│ │ └── comparison.json
│ └── latest -> 2025-12-22/
└── fixtures/
└── adapters/ # Test fixtures for adapters
```
---
## Ground Truth Format
```json
{
"imageDigest": "sha256:abc123...",
"imageName": "alpine:3.18",
"category": "BaseOS",
"groundTruth": {
"truePositives": [
{
"cveId": "CVE-2024-1234",
"package": "openssl",
"version": "3.0.8",
"notes": "Fixed in 3.0.9"
}
],
"knownFalsePositives": [
{
"cveId": "CVE-2024-9999",
"package": "zlib",
"version": "1.2.13",
"reason": "Backported in alpine:3.18"
}
],
"expectedUnreachable": [
{
"cveId": "CVE-2024-5678",
"package": "curl",
"reason": "Vulnerable function not linked"
}
]
},
"lastVerified": "2025-12-01T00:00:00Z",
"verifiedBy": "security-team"
}
```
---
## CI Integration
### Workflow: `benchmark-vs-competitors.yml`
```yaml
name: Competitive Benchmark
on:
schedule:
- cron: '0 2 * * 0' # Weekly Sunday 2 AM
workflow_dispatch:
push:
paths:
- 'src/Scanner/__Libraries/StellaOps.Scanner.Benchmark/**'
- 'bench/competitors/**'
jobs:
benchmark:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Install competitor tools
run: |
# Install Trivy
curl -sfL https://raw.githubusercontent.com/aquasecurity/trivy/main/contrib/install.sh | sh
# Install Grype
curl -sSfL https://raw.githubusercontent.com/anchore/grype/main/install.sh | sh
- name: Run benchmark
run: stella benchmark run --corpus bench/competitors/corpus --output bench/competitors/results/$(date +%Y-%m-%d)
- name: Update claims index
run: stella benchmark claims --output docs/claims-index.md
- name: Upload results
uses: actions/upload-artifact@v4
with:
name: benchmark-results
path: bench/competitors/results/
```
---
## CLI Commands
```bash
# Run full benchmark
stella benchmark run --corpus <path> --competitors trivy,grype,syft
# Verify a specific claim
stella benchmark verify <CLAIM_ID>
# Generate claims index
stella benchmark claims --output docs/claims-index.md
# Generate marketing battlecard
stella benchmark battlecard --output docs/product/battlecard.md
# Show comparison summary
stella benchmark summary --format table|json|markdown
```
---
## Testing
| Test Type | Location | Purpose |
|-----------|----------|---------|
| Unit | `StellaOps.Scanner.Benchmark.Tests/` | Adapter parsing, metrics calculation |
| Integration | `StellaOps.Scanner.Benchmark.Integration.Tests/` | Full benchmark flow |
| Golden | `bench/competitors/fixtures/` | Deterministic output verification |
---
## Security Considerations
1. **Competitor binaries**: Run in isolated containers, no network access during scan
2. **Corpus images**: Verified digests, no external pulls during benchmark
3. **Results**: Signed with DSSE before publishing
4. **Claims**: Require PR review before status change
---
## Dependencies
- `StellaOps.Scanner.Core` - Normalized finding models
- `StellaOps.Attestor.Dsse` - Result signing
- Docker - Competitor tool execution
- Ground-truth corpus (maintained separately)
---
## Related Documentation
- [Claims Index](../../claims-index.md)
- [Sprint 7000.0001.0001](../../implplan/SPRINT_7000_0001_0001_competitive_benchmarking.md)
- [Testing Strategy](../../implplan/SPRINT_5100_0000_0000_epic_summary.md)
---
*Document Version*: 1.0.0
*Created*: 2025-12-22

19
docs-archived/modules/ci/architecture.md Normal file
View File

@@ -0,0 +1,19 @@
# CI Architecture
## Purpose
Describe CI workflows, triggers, and offline constraints for Stella Ops.
## Scope
- Gitea workflows and templates under `.gitea/`.
- DevOps scripts under `devops/scripts/` and `.gitea/scripts/`.
- Build and test policy docs under `docs/technical/cicd/`.
## Principles
- Deterministic and offline-first execution.
- Pinned tool versions with explicit provenance.
- Evidence logged to sprint Execution Log and audits.
## References
- `docs/technical/cicd/workflow-triggers.md`
- `docs/technical/cicd/release-pipelines.md`
- `docs/operations/devops/README.md`

8
docs-archived/modules/eventing/README.md Normal file
View File

@@ -0,0 +1,8 @@
# Eventing Module
> **Status: Draft/Planned.** The event envelope SDK is currently in design phase. Implementation is planned for the Timeline and TimelineIndexer modules and will be integrated across all services via `src/__Libraries/StellaOps.Eventing/`. No standalone `src/Eventing/` module exists.
## Related Documentation
- [Event Envelope Schema](event-envelope-schema.md)
- [Timeline UI](timeline-ui.md)

496
docs-archived/modules/eventing/event-envelope-schema.md Normal file
View File

@@ -0,0 +1,496 @@
# Event Envelope Schema
> **Version:** 1.0.0
> **Status:** Draft
> **Sprint:** [SPRINT_20260107_003_001_LB](../../implplan/SPRINT_20260107_003_001_LB_event_envelope_sdk.md)
This document specifies the canonical event envelope schema for the StellaOps Unified Event Timeline.
---
## Overview
The event envelope provides a standardized format for all events emitted across StellaOps services. It enables:
- **Unified Timeline:** Cross-service correlation with HLC ordering
- **Deterministic Replay:** Reproducible event streams for forensics
- **Audit Compliance:** DSSE-signed event bundles for export
- **Causal Analysis:** Stage latency measurement and bottleneck identification
---
## Envelope Schema (v1)
### JSON Schema
```json
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://stellaops.org/schemas/timeline-event.v1.json",
"title": "TimelineEvent",
"description": "Canonical event envelope for StellaOps Unified Event Timeline",
"type": "object",
"required": [
"eventId",
"tHlc",
"tsWall",
"service",
"correlationId",
"kind",
"payload",
"payloadDigest",
"engineVersion",
"schemaVersion"
],
"properties": {
"eventId": {
"type": "string",
"description": "Deterministic event ID: SHA-256(correlationId || tHlc || service || kind)[0:32] hex",
"pattern": "^[a-f0-9]{32}$"
},
"tHlc": {
"type": "string",
"description": "HLC timestamp in sortable string format: <physicalTimeMs>:<logicalCounter>:<nodeId>",
"pattern": "^\\d+:\\d+:[a-zA-Z0-9_-]+$"
},
"tsWall": {
"type": "string",
"format": "date-time",
"description": "Wall-clock time in ISO 8601 format (informational only)"
},
"service": {
"type": "string",
"description": "Service name that emitted the event",
"enum": ["Scheduler", "AirGap", "Attestor", "Policy", "VexLens", "Scanner", "Concelier", "Platform"]
},
"traceParent": {
"type": ["string", "null"],
"description": "W3C Trace Context traceparent header",
"pattern": "^[0-9a-f]{2}-[0-9a-f]{32}-[0-9a-f]{16}-[0-9a-f]{2}$"
},
"correlationId": {
"type": "string",
"description": "Correlation ID linking related events (e.g., scanId, jobId, artifactDigest)"
},
"kind": {
"type": "string",
"description": "Event kind/type",
"enum": [
"ENQUEUE", "DEQUEUE", "EXECUTE", "COMPLETE", "FAIL",
"IMPORT", "EXPORT", "MERGE", "CONFLICT",
"ATTEST", "VERIFY",
"EVALUATE", "GATE_PASS", "GATE_FAIL",
"CONSENSUS", "OVERRIDE",
"SCAN_START", "SCAN_COMPLETE",
"EMIT", "ACK", "ERR"
]
},
"payload": {
"type": "string",
"description": "RFC 8785 canonicalized JSON payload"
},
"payloadDigest": {
"type": "string",
"description": "SHA-256 digest of payload as hex string",
"pattern": "^[a-f0-9]{64}$"
},
"engineVersion": {
"type": "object",
"description": "Engine/resolver version for reproducibility",
"required": ["engineName", "version", "sourceDigest"],
"properties": {
"engineName": {
"type": "string",
"description": "Name of the engine/service"
},
"version": {
"type": "string",
"description": "Semantic version string"
},
"sourceDigest": {
"type": "string",
"description": "SHA-256 digest of engine source/binary"
}
}
},
"dsseSig": {
"type": ["string", "null"],
"description": "Optional DSSE signature in format keyId:base64Signature"
},
"schemaVersion": {
"type": "integer",
"description": "Schema version for envelope evolution",
"const": 1
}
}
}
```
### C# Record Definition
```csharp
/// <summary>
/// Canonical event envelope for unified timeline.
/// </summary>
public sealed record TimelineEvent
{
/// <summary>
/// Deterministic event ID: SHA-256(correlationId || tHlc || service || kind)[0:32] hex.
/// NOT a random ULID - ensures replay determinism.
/// </summary>
[Required]
[RegularExpression("^[a-f0-9]{32}$")]
public required string EventId { get; init; }
/// <summary>
/// HLC timestamp from StellaOps.HybridLogicalClock library.
/// </summary>
[Required]
public required HlcTimestamp THlc { get; init; }
/// <summary>
/// Wall-clock time (informational only, not used for ordering).
/// </summary>
[Required]
public required DateTimeOffset TsWall { get; init; }
/// <summary>
/// Service name that emitted the event.
/// </summary>
[Required]
public required string Service { get; init; }
/// <summary>
/// W3C Trace Context traceparent for OpenTelemetry correlation.
/// </summary>
public string? TraceParent { get; init; }
/// <summary>
/// Correlation ID linking related events.
/// </summary>
[Required]
public required string CorrelationId { get; init; }
/// <summary>
/// Event kind (ENQUEUE, EXECUTE, ATTEST, etc.).
/// </summary>
[Required]
public required string Kind { get; init; }
/// <summary>
/// RFC 8785 canonicalized JSON payload.
/// </summary>
[Required]
public required string Payload { get; init; }
/// <summary>
/// SHA-256 digest of Payload.
/// </summary>
[Required]
public required byte[] PayloadDigest { get; init; }
/// <summary>
/// Engine version for reproducibility (per CLAUDE.md Rule 8.2.1).
/// </summary>
[Required]
public required EngineVersionRef EngineVersion { get; init; }
/// <summary>
/// Optional DSSE signature (keyId:base64Signature).
/// </summary>
public string? DsseSig { get; init; }
/// <summary>
/// Schema version (current: 1).
/// </summary>
public int SchemaVersion { get; init; } = 1;
}
public sealed record EngineVersionRef(
string EngineName,
string Version,
string SourceDigest);
```
---
## Field Specifications
### eventId
**Purpose:** Unique, deterministic identifier for each event.
**Computation:**
```csharp
public static string GenerateEventId(
string correlationId,
HlcTimestamp tHlc,
string service,
string kind)
{
using var hasher = IncrementalHash.CreateHash(HashAlgorithmName.SHA256);
hasher.AppendData(Encoding.UTF8.GetBytes(correlationId));
hasher.AppendData(Encoding.UTF8.GetBytes(tHlc.ToSortableString()));
hasher.AppendData(Encoding.UTF8.GetBytes(service));
hasher.AppendData(Encoding.UTF8.GetBytes(kind));
var hash = hasher.GetHashAndReset();
return Convert.ToHexString(hash.AsSpan(0, 16)).ToLowerInvariant();
}
```
**Rationale:** Unlike ULID or UUID, this deterministic approach ensures that:
- The same event produces the same ID across replays
- Duplicate events can be detected and deduplicated
- Event ordering is verifiable
### tHlc
**Purpose:** Primary ordering timestamp using Hybrid Logical Clock.
**Format:** `<physicalTimeMs>:<logicalCounter>:<nodeId>`
**Example:** `1704585600000:42:scheduler-node-1`
**Ordering:** Lexicographic comparison produces correct temporal order:
1. Compare physical time (milliseconds since Unix epoch)
2. If equal, compare logical counter
3. If equal, compare node ID (for uniqueness)
**Implementation:** Uses existing `StellaOps.HybridLogicalClock.HlcTimestamp` type.
### tsWall
**Purpose:** Human-readable wall-clock timestamp for debugging.
**Format:** ISO 8601 with UTC timezone (e.g., `2026-01-07T12:00:00.000Z`)
**Important:** This field is **informational only**. Never use for ordering or comparison. The `tHlc` field is the authoritative timestamp.
### service
**Purpose:** Identifies the StellaOps service that emitted the event.
**Allowed Values:**
| Value | Description |
|-------|-------------|
| `Scheduler` | Job scheduling and queue management |
| `AirGap` | Offline/air-gap sync operations |
| `Attestor` | DSSE attestation and verification |
| `Policy` | Policy engine evaluation |
| `VexLens` | VEX consensus computation |
| `Scanner` | Container scanning |
| `Concelier` | Advisory ingestion |
| `Platform` | Console backend aggregation |
### traceParent
**Purpose:** W3C Trace Context correlation for OpenTelemetry integration.
**Format:** `00-{trace-id}-{span-id}-{trace-flags}`
**Example:** `00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01`
**Population:** Automatically captured from `Activity.Current?.Id` during event emission.
### correlationId
**Purpose:** Links related events across services.
**Common Patterns:**
| Pattern | Example | Usage |
|---------|---------|-------|
| Scan ID | `scan-abc123` | Container scan lifecycle |
| Job ID | `job-xyz789` | Scheduled job lifecycle |
| Artifact Digest | `sha256:abc...` | Artifact processing |
| Bundle ID | `bundle-def456` | Air-gap bundle operations |
### kind
**Purpose:** Categorizes the event type.
**Event Kinds by Service:**
| Service | Kinds |
|---------|-------|
| Scheduler | `ENQUEUE`, `DEQUEUE`, `EXECUTE`, `COMPLETE`, `FAIL` |
| AirGap | `IMPORT`, `EXPORT`, `MERGE`, `CONFLICT` |
| Attestor | `ATTEST`, `VERIFY` |
| Policy | `EVALUATE`, `GATE_PASS`, `GATE_FAIL` |
| VexLens | `CONSENSUS`, `OVERRIDE` |
| Scanner | `SCAN_START`, `SCAN_COMPLETE` |
| Generic | `EMIT`, `ACK`, `ERR` |
### payload
**Purpose:** Domain-specific event data.
**Requirements:**
1. **RFC 8785 Canonicalization:** Must use `CanonJson.Serialize()` from `StellaOps.Canonical.Json`
2. **No Non-Deterministic Fields:** No random IDs, current timestamps, or environment-specific data
3. **Bounded Size:** Payload should be < 1MB; use references for large data
**Example:**
```json
{
"artifactDigest": "sha256:abc123...",
"jobId": "job-xyz789",
"status": "completed",
"findingsCount": 42
}
```
### payloadDigest
**Purpose:** Integrity verification of payload.
**Computation:**
```csharp
var digest = SHA256.HashData(Encoding.UTF8.GetBytes(payload));
```
**Format:** 64-character lowercase hex string.
### engineVersion
**Purpose:** Records the engine/resolver version for reproducibility verification (per CLAUDE.md Rule 8.2.1).
**Fields:**
| Field | Description | Example |
|-------|-------------|---------|
| `engineName` | Service/engine name | `"Scheduler"` |
| `version` | Semantic version | `"2.5.0"` |
| `sourceDigest` | Build artifact hash | `"sha256:abc..."` |
**Population:** Use `EngineVersionRef.FromAssembly(Assembly.GetExecutingAssembly())`.
### dsseSig
**Purpose:** Optional cryptographic signature for audit compliance.
**Format:** `{keyId}:{base64Signature}`
**Example:** `signing-key-001:MEUCIQD...`
**Integration:** Uses existing `StellaOps.Attestation.DsseHelper` for signature generation.
### schemaVersion
**Purpose:** Enables schema evolution without breaking compatibility.
**Current Value:** `1`
**Migration Strategy:** When schema changes:
1. Increment version number
2. Add migration logic for older versions
3. Document breaking changes
---
## Database Schema
```sql
CREATE SCHEMA IF NOT EXISTS timeline;
CREATE TABLE timeline.events (
event_id TEXT PRIMARY KEY,
t_hlc TEXT NOT NULL,
ts_wall TIMESTAMPTZ NOT NULL,
service TEXT NOT NULL,
trace_parent TEXT,
correlation_id TEXT NOT NULL,
kind TEXT NOT NULL,
payload JSONB NOT NULL,
payload_digest BYTEA NOT NULL,
engine_name TEXT NOT NULL,
engine_version TEXT NOT NULL,
engine_digest TEXT NOT NULL,
dsse_sig TEXT,
schema_version INTEGER NOT NULL DEFAULT 1,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
-- Primary query: events by correlation, HLC ordered
CREATE INDEX idx_events_corr_hlc ON timeline.events (correlation_id, t_hlc);
-- Service-specific queries
CREATE INDEX idx_events_svc_hlc ON timeline.events (service, t_hlc);
-- Payload search (JSONB GIN index)
CREATE INDEX idx_events_payload ON timeline.events USING GIN (payload);
-- Kind filtering
CREATE INDEX idx_events_kind ON timeline.events (kind);
```
---
## Usage Examples
### Emitting an Event
```csharp
public class SchedulerService
{
private readonly ITimelineEventEmitter _emitter;
public async Task EnqueueJobAsync(Job job, CancellationToken ct)
{
// Business logic...
await _queue.EnqueueAsync(job, ct);
// Emit timeline event
await _emitter.EmitAsync(
correlationId: job.Id.ToString(),
kind: "ENQUEUE",
payload: new { jobId = job.Id, priority = job.Priority },
ct);
}
}
```
### Querying Timeline
```csharp
public async Task<IReadOnlyList<TimelineEvent>> GetJobTimelineAsync(
string jobId,
CancellationToken ct)
{
return await _timelineService.GetEventsAsync(
correlationId: jobId,
options: new TimelineQueryOptions
{
Services = ["Scheduler", "Attestor"],
Kinds = ["ENQUEUE", "EXECUTE", "COMPLETE", "ATTEST"]
},
ct);
}
```
---
## Compatibility Notes
### Relation to Existing HLC Infrastructure
This schema builds on the existing `StellaOps.HybridLogicalClock` library:
- Uses `HlcTimestamp` type directly
- Integrates with `IHybridLogicalClock.Tick()` for timestamp generation
- Compatible with air-gap merge algorithms
### Relation to Existing Replay Infrastructure
This schema integrates with `StellaOps.Replay.Core`:
- `KnowledgeSnapshot` can include timeline event references
- Replay uses `FakeTimeProvider` with HLC timestamps
- Verification compares payload digests
---
## References
- [SPRINT_20260107_003_000_INDEX](../../implplan/SPRINT_20260107_003_000_INDEX_unified_event_timeline.md) - Parent sprint index
- [SPRINT_20260105_002_000_INDEX](../../implplan/SPRINT_20260105_002_000_INDEX_hlc_audit_safe_ordering.md) - HLC foundation
- [RFC 8785](https://datatracker.ietf.org/doc/html/rfc8785) - JSON Canonicalization Scheme
- [W3C Trace Context](https://www.w3.org/TR/trace-context/) - Distributed tracing
- CLAUDE.md Section 8.2.1 - Engine version tracking
- CLAUDE.md Section 8.7 - RFC 8785 canonicalization

171
docs-archived/modules/eventing/timeline-ui.md Normal file
View File

@@ -0,0 +1,171 @@
# Timeline UI Component
> **Module:** Eventing / Timeline
> **Status:** Implemented
> **Last Updated:** 2026-01-07
## Overview
The Timeline UI provides a visual representation of HLC-ordered events across StellaOps services. It enables operators to trace the causal flow of operations, identify bottlenecks, and investigate specific events with full evidence links.
## Features
### Causal Lanes Visualization
Events are displayed in swimlanes organized by service:
```
┌─────────────────────────────────────────────────────────────────────┐
│ HLC Timeline Axis │
│ |-------|-------|-------|-------|-------|-------|-------|-------> │
├─────────────────────────────────────────────────────────────────────┤
│ Scheduler [E]─────────[X]───────────────[C] │
├─────────────────────────────────────────────────────────────────────┤
│ AirGap [I]──────────[M] │
├─────────────────────────────────────────────────────────────────────┤
│ Attestor [A]──────────[V] │
├─────────────────────────────────────────────────────────────────────┤
│ Policy [G] │
└─────────────────────────────────────────────────────────────────────┘
```
Legend:
- **[E]** Enqueue - Job queued for processing
- **[X]** Execute - Job execution started
- **[C]** Complete - Job completed
- **[I]** Import - Data imported (e.g., SBOM, advisory)
- **[M]** Merge - Data merged
- **[A]** Attest - Attestation created
- **[V]** Verify - Attestation verified
- **[G]** Gate - Policy gate evaluated
### Critical Path Analysis
The critical path view shows the longest sequence of dependent operations:
- Color-coded by severity (green/yellow/red)
- Bottleneck stage highlighted
- Percentage of total duration shown
- Clickable stages for drill-down
### Event Detail Panel
Selected events display:
- Event ID and metadata
- HLC timestamp and wall-clock time
- Service and event kind
- JSON payload viewer
- Engine version information
- Evidence links (SBOM, VEX, Policy, Attestation)
### Filtering
Events can be filtered by:
- **Services**: Scheduler, AirGap, Attestor, Policy, Scanner, etc.
- **Event Kinds**: ENQUEUE, EXECUTE, COMPLETE, IMPORT, ATTEST, etc.
- **HLC Range**: From/To timestamps
Filter state is persisted in URL query parameters.
### Export
Timeline data can be exported as:
- **NDJSON**: Newline-delimited JSON (streaming-friendly)
- **JSON**: Standard JSON array
- **DSSE-signed**: Cryptographically signed bundles for auditing
## Usage
### Accessing the Timeline
Navigate to `/timeline/{correlationId}` where `correlationId` is the unique identifier for a scan, job, or workflow.
Example:
```
/timeline/scan-abc123-def456
```
### Keyboard Navigation
| Key | Action |
|-----|--------|
| Tab | Navigate between events |
| Enter/Space | Select focused event |
| Escape | Clear selection |
| Arrow keys | Scroll within panel |
### URL Parameters
| Parameter | Description | Example |
|-----------|-------------|---------|
| `services` | Comma-separated service filter | `?services=Scheduler,AirGap` |
| `kinds` | Comma-separated kind filter | `?kinds=EXECUTE,COMPLETE` |
| `fromHlc` | Start of HLC range | `?fromHlc=1704067200000:0:node1` |
| `toHlc` | End of HLC range | `?toHlc=1704153600000:0:node1` |
## Component Architecture
```
timeline/
├── components/
│ ├── causal-lanes/ # Swimlane visualization
│ ├── critical-path/ # Bottleneck bar chart
│ ├── event-detail-panel/ # Selected event details
│ ├── evidence-links/ # Links to SBOM/VEX/Policy
│ ├── export-button/ # Export dropdown
│ └── timeline-filter/ # Service/kind filters
├── models/
│ └── timeline.models.ts # TypeScript interfaces
├── pages/
│ └── timeline-page/ # Main page component
├── services/
│ └── timeline.service.ts # API client
└── timeline.routes.ts # Lazy-loaded routes
```
## API Integration
The Timeline UI integrates with the Timeline API:
| Endpoint | Description |
|----------|-------------|
| `GET /api/v1/timeline/{correlationId}` | Fetch events |
| `GET /api/v1/timeline/{correlationId}/critical-path` | Fetch critical path |
| `POST /api/v1/timeline/{correlationId}/export` | Initiate export |
| `GET /api/v1/timeline/export/{exportId}` | Check export status |
| `GET /api/v1/timeline/export/{exportId}/download` | Download bundle |
## Accessibility
The Timeline UI follows WCAG 2.1 AA guidelines:
- **Keyboard Navigation**: All interactive elements are focusable
- **Screen Readers**: ARIA labels on all regions and controls
- **Color Contrast**: 4.5:1 minimum contrast ratio
- **Focus Indicators**: Visible focus rings on all controls
- **Motion**: Respects `prefers-reduced-motion`
## Performance
- **Virtual Scrolling**: Handles 10K+ events efficiently
- **Lazy Loading**: Events loaded on-demand as user scrolls
- **Caching**: Recent queries cached to reduce API calls
- **Debouncing**: Filter changes debounced to avoid excessive requests
## Screenshots
### Timeline View
![Timeline View](./assets/timeline-view.png)
### Critical Path Analysis
![Critical Path](./assets/critical-path.png)
### Event Detail Panel
![Event Details](./assets/event-details.png)
## Related Documentation
- [Timeline API Reference](../../api/timeline-api.md)
- [HLC Clock Specification](../hlc/architecture.md)
- [Eventing SDK](../eventing/architecture.md)
- [Evidence Model](../../schemas/evidence.md)

49
docs-archived/modules/evidence/README.md Normal file
View File

@@ -0,0 +1,49 @@
# Evidence
**Status:** Design/Planning
**Source:** N/A (cross-cutting concept)
**Owner:** Platform Team
## Purpose
Evidence defines the unified evidence model for vulnerability findings across StellaOps. Provides canonical data structures for evidence capture, aggregation, and scoring used by Signals, Policy Engine, and EvidenceLocker modules.
## Components
**Concept Documentation:**
- `unified-model.md` - Unified evidence data model specification
**Evidence Types:**
- Reachability evidence (call graph, data flow)
- Runtime evidence (eBPF traces, dynamic observations)
- Binary evidence (backport detection, fix validation)
- Exploit evidence (EPSS scores, KEV flags, exploit-db entries)
- VEX evidence (source trust, statement provenance)
- Mitigation evidence (active mitigations, compensating controls)
## Implementation Locations
Evidence structures are implemented across multiple modules:
- **Signals** - Evidence aggregation and normalization
- **Policy Engine** - Reachability analysis and evidence generation
- **EvidenceLocker** - Evidence storage and sealing
- **Scanner** - Binary and vulnerability evidence capture
- **Concelier** - Backport and exploit evidence enrichment
## Dependencies
- All evidence-producing modules (Scanner, Policy, Concelier, etc.)
- Signals (evidence aggregation)
- EvidenceLocker (evidence storage)
## Related Documentation
- Unified Model: `./unified-model.md`
- Signals: `../signals/`
- Policy: `../policy/`
- EvidenceLocker: `../evidence-locker/`
- Data Schemas: `../../11_DATA_SCHEMAS.md`
## Current Status
Evidence model documented in `unified-model.md`. Implementation distributed across Signals (aggregation), Policy (reachability), EvidenceLocker (storage), and Scanner (capture) modules.

360
docs-archived/modules/evidence/unified-model.md Normal file
View File

@@ -0,0 +1,360 @@
# Unified Evidence Model
> **Module:** `StellaOps.Evidence.Core`
> **Status:** Production
> **Owner:** Platform Guild
## Overview
The Unified Evidence Model provides a standardized interface (`IEvidence`) and implementation (`EvidenceRecord`) for representing evidence across all StellaOps modules. This enables:
- **Cross-module evidence linking**: Evidence from Scanner, Attestor, Excititor, and Policy modules share a common contract.
- **Content-addressed verification**: Evidence records are immutable and verifiable via deterministic hashing.
- **Unified storage**: A single `IEvidenceStore` interface abstracts persistence across modules.
- **Cryptographic attestation**: Multiple signatures from different signers (internal, vendor, CI, operator) can vouch for evidence.
## Core Types
### IEvidence Interface
```csharp
public interface IEvidence
{
string SubjectNodeId { get; } // Content-addressed subject
EvidenceType EvidenceType { get; } // Type discriminator
string EvidenceId { get; } // Computed hash identifier
ReadOnlyMemory<byte> Payload { get; } // Canonical JSON payload
IReadOnlyList<EvidenceSignature> Signatures { get; }
EvidenceProvenance Provenance { get; }
string? ExternalPayloadCid { get; } // For large payloads
string PayloadSchemaVersion { get; }
}
```
### EvidenceType Enum
The platform supports these evidence types:
| Type | Value | Description | Example Payload |
|------|-------|-------------|-----------------|
| `Reachability` | 1 | Call graph analysis | Paths, confidence, graph digest |
| `Scan` | 2 | Vulnerability finding | CVE, severity, affected package |
| `Policy` | 3 | Policy evaluation | Rule ID, verdict, inputs |
| `Artifact` | 4 | SBOM entry metadata | PURL, digest, build info |
| `Vex` | 5 | VEX statement | Status, justification, impact |
| `Epss` | 6 | EPSS score | Score, percentile, model date |
| `Runtime` | 7 | Runtime observation | eBPF/ETW traces, call frames |
| `Provenance` | 8 | Build provenance | SLSA attestation, builder info |
| `Exception` | 9 | Applied exception | Exception ID, reason, expiry |
| `Guard` | 10 | Guard/gate analysis | Gate type, condition, bypass |
| `Kev` | 11 | KEV status | In-KEV flag, added date |
| `License` | 12 | License analysis | SPDX ID, compliance status |
| `Dependency` | 13 | Dependency metadata | Graph edge, version range |
| `Custom` | 100 | User-defined | Schema-versioned custom payload |
### EvidenceRecord
The concrete implementation with deterministic identity:
```csharp
public sealed record EvidenceRecord : IEvidence
{
public static EvidenceRecord Create(
string subjectNodeId,
EvidenceType evidenceType,
ReadOnlyMemory<byte> payload,
EvidenceProvenance provenance,
string payloadSchemaVersion,
IReadOnlyList<EvidenceSignature>? signatures = null,
string? externalPayloadCid = null);
public bool VerifyIntegrity();
}
```
**EvidenceId Computation:**
The `EvidenceId` is a SHA-256 hash of the canonicalized fields using versioned prefixing:
```
EvidenceId = "evidence:" + CanonJson.HashVersionedPrefixed("IEvidence", "v1", {
SubjectNodeId,
EvidenceType,
PayloadHash,
Provenance.GeneratorId,
Provenance.GeneratorVersion,
Provenance.GeneratedAt (ISO 8601)
})
```
### EvidenceSignature
Cryptographic attestation by a signer:
```csharp
public sealed record EvidenceSignature
{
public required string SignerId { get; init; }
public required string Algorithm { get; init; } // ES256, RS256, EdDSA
public required string SignatureBase64 { get; init; }
public required DateTimeOffset SignedAt { get; init; }
public SignerType SignerType { get; init; }
public IReadOnlyList<string>? CertificateChain { get; init; }
}
```
**SignerType Values:**
- `Internal` (0): StellaOps service
- `Vendor` (1): External vendor/supplier
- `CI` (2): CI/CD pipeline
- `Operator` (3): Human operator
- `TransparencyLog` (4): Rekor/transparency log
- `Scanner` (5): Security scanner
- `PolicyEngine` (6): Policy engine
- `Unknown` (255): Unclassified
### EvidenceProvenance
Generation context:
```csharp
public sealed record EvidenceProvenance
{
public required string GeneratorId { get; init; }
public required string GeneratorVersion { get; init; }
public required DateTimeOffset GeneratedAt { get; init; }
public string? CorrelationId { get; init; }
public Guid? TenantId { get; init; }
// ... additional fields
}
```
## Adapters
Adapters convert module-specific evidence types to the unified `IEvidence` interface:
### Available Adapters
| Adapter | Source Module | Source Type | Target Evidence Types |
|---------|---------------|-------------|----------------------|
| `EvidenceBundleAdapter` | Scanner | `EvidenceBundle` | Reachability, Vex, Provenance, Scan |
| `EvidenceStatementAdapter` | Attestor | `EvidenceStatement` (in-toto) | Scan |
| `ProofSegmentAdapter` | Scanner | `ProofSegment` | Varies by segment type |
| `VexObservationAdapter` | Excititor | `VexObservation` | Vex, Provenance |
| `ExceptionApplicationAdapter` | Policy | `ExceptionApplication` | Exception |
### Adapter Interface
```csharp
public interface IEvidenceAdapter<TSource>
{
IReadOnlyList<IEvidence> Convert(
TSource source,
string subjectNodeId,
EvidenceProvenance provenance);
bool CanConvert(TSource source);
}
```
### Using Adapters
Adapters use **input DTOs** to avoid circular dependencies:
```csharp
// Using VexObservationAdapter
var adapter = new VexObservationAdapter();
var input = new VexObservationInput
{
ObservationId = "obs-001",
ProviderId = "nvd",
StreamId = "cve-feed",
// ... other fields from VexObservation
};
var provenance = new EvidenceProvenance
{
GeneratorId = "excititor-ingestor",
GeneratorVersion = "1.0.0",
GeneratedAt = DateTimeOffset.UtcNow
};
if (adapter.CanConvert(input))
{
IReadOnlyList<IEvidence> records = adapter.Convert(
input,
subjectNodeId: "sha256:abc123",
provenance);
}
```
## Evidence Store
### IEvidenceStore Interface
```csharp
public interface IEvidenceStore
{
Task<EvidenceRecord> StoreAsync(
EvidenceRecord record,
CancellationToken ct = default);
Task<IReadOnlyList<EvidenceRecord>> StoreBatchAsync(
IEnumerable<EvidenceRecord> records,
CancellationToken ct = default);
Task<EvidenceRecord?> GetByIdAsync(
string evidenceId,
CancellationToken ct = default);
Task<IReadOnlyList<EvidenceRecord>> GetBySubjectAsync(
string subjectNodeId,
EvidenceType? evidenceType = null,
CancellationToken ct = default);
Task<IReadOnlyList<EvidenceRecord>> GetByTypeAsync(
EvidenceType evidenceType,
int limit = 100,
CancellationToken ct = default);
Task<bool> ExistsAsync(
string evidenceId,
CancellationToken ct = default);
Task<bool> DeleteAsync(
string evidenceId,
CancellationToken ct = default);
}
```
### Implementations
- **`InMemoryEvidenceStore`**: Thread-safe in-memory store for testing and development.
- **`PostgresEvidenceStore`** (planned): Production store with tenant isolation and indexing.
## Usage Examples
### Creating Evidence
```csharp
var provenance = new EvidenceProvenance
{
GeneratorId = "scanner-service",
GeneratorVersion = "2.1.0",
GeneratedAt = DateTimeOffset.UtcNow,
TenantId = tenantId
};
// Serialize payload to canonical JSON
var payloadBytes = CanonJson.Canonicalize(new
{
cveId = "CVE-2024-1234",
severity = "HIGH",
affectedPackage = "pkg:npm/lodash@4.17.20"
});
var evidence = EvidenceRecord.Create(
subjectNodeId: "sha256:abc123def456...",
evidenceType: EvidenceType.Scan,
payload: payloadBytes,
provenance: provenance,
payloadSchemaVersion: "scan/v1");
```
### Storing and Retrieving
```csharp
var store = new InMemoryEvidenceStore();
// Store
await store.StoreAsync(evidence);
// Retrieve by ID
var retrieved = await store.GetByIdAsync(evidence.EvidenceId);
// Retrieve all evidence for a subject
var allForSubject = await store.GetBySubjectAsync(
"sha256:abc123def456...",
evidenceType: EvidenceType.Scan);
// Verify integrity
bool isValid = retrieved!.VerifyIntegrity();
```
### Cross-Module Evidence Linking
```csharp
// Scanner produces evidence bundle
var bundle = scanner.ProduceEvidenceBundle(target);
// Convert to unified evidence
var adapter = new EvidenceBundleAdapter();
var evidenceRecords = adapter.Convert(bundle, subjectNodeId, provenance);
// Store all records
await store.StoreBatchAsync(evidenceRecords);
// Later, any module can query by subject
var allEvidence = await store.GetBySubjectAsync(subjectNodeId);
// Filter by type
var reachabilityEvidence = allEvidence
.Where(e => e.EvidenceType == EvidenceType.Reachability);
var vexEvidence = allEvidence
.Where(e => e.EvidenceType == EvidenceType.Vex);
```
## Schema Versioning
Each evidence type payload has a schema version (`PayloadSchemaVersion`) for forward compatibility:
- `scan/v1`: Initial scan evidence schema
- `reachability/v1`: Reachability evidence schema
- `vex-statement/v1`: VEX statement evidence schema
- `proof-segment/v1`: Proof segment evidence schema
- `exception-application/v1`: Exception application schema
Consumers should check `PayloadSchemaVersion` before deserializing payloads to handle schema evolution.
## Integration Patterns
### Module Integration
Each module that produces evidence should:
1. Create an adapter if converting from module-specific types
2. Use `EvidenceRecord.Create()` for new evidence
3. Store evidence via `IEvidenceStore`
4. Include provenance with generator identification
### Verification Flow
```
1. Retrieve evidence by SubjectNodeId
2. Call VerifyIntegrity() to check EvidenceId
3. Verify signatures against known trust roots
4. Deserialize and validate payload against schema
```
## Testing
The `StellaOps.Evidence.Core.Tests` project includes:
- **111 unit tests** covering:
- EvidenceRecord creation and hash computation
- InMemoryEvidenceStore CRUD operations
- All adapter conversions (VexObservation, ExceptionApplication, ProofSegment)
- Edge cases and error handling
Run tests:
```bash
dotnet test src/__Libraries/StellaOps.Evidence.Core.Tests/
```
## Related Documentation
- [Proof Chain Architecture](../attestor/proof-chain.md)
- [Evidence Bundle Design](../scanner/evidence-bundle.md)
- [VEX Observation Model](../excititor/vex-observation.md)
- [Policy Exceptions](../policy/exceptions.md)

41
docs-archived/modules/facet/README.md Normal file
View File

@@ -0,0 +1,41 @@
# Facet
> Cryptographically sealed manifests for logical slices of container images.
## Purpose
The Facet Sealing subsystem provides cryptographically sealed manifests for logical slices of container images, enabling fine-grained drift detection, per-facet quota enforcement, and deterministic change tracking.
## Quick Links
- [Architecture](./architecture.md) - Technical design and implementation details
## Status
| Attribute | Value |
|-----------|-------|
| **Maturity** | Production |
| **Last Reviewed** | 2025-12-29 |
| **Maintainer** | Scanner Guild, Policy Guild |
## Key Features
- **Facet Types**: OS packages, language dependencies, binaries, configs, custom patterns
- **Cryptographic Sealing**: Each facet can be individually sealed with a cryptographic snapshot
- **Drift Detection**: Monitor changes between seals for compliance enforcement
- **Merkle Tree Structure**: Content-addressed storage with integrity verification
## Dependencies
### Upstream (this module depends on)
- **Scanner** - Facet extraction during image analysis
- **Attestor** - DSSE signing for sealed facets
### Downstream (modules that depend on this)
- **Policy** - Drift detection and quota enforcement
- **Replay** - Facet verification in replay workflows
## Related Documentation
- [Scanner Architecture](../scanner/architecture.md)
- [Replay Architecture](../replay/architecture.md)

702
docs-archived/modules/facet/architecture.md Normal file
View File

@@ -0,0 +1,702 @@
# Facet Sealing Architecture
> **Status: Production (Cross-Module Library).** Facet Sealing is a fully implemented subsystem with its core library at `src/__Libraries/StellaOps.Facet/` (30 source files) and integration points spanning **Scanner** (extraction via `FacetSealExtractor`, storage via `PostgresFacetSealStore` in `scanner.facet_seals` table), **Policy** (drift and quota enforcement via `FacetQuotaGate`), **Zastava** (admission validation via `FacetAdmissionValidator`), and **CLI** (`seal`, `drift`, `vex-gen` commands). Comprehensive test coverage exists across 17 test files. This documentation covers the cross-cutting architecture.
> **Ownership:** Scanner Guild, Policy Guild
> **Audience:** Service owners, platform engineers, security architects
> **Related:** [Platform Architecture](../platform/architecture-overview.md), [Scanner Architecture](../scanner/architecture.md), [Replay Architecture](../replay/architecture.md), [Policy Engine](../policy/architecture.md)
This dossier describes the Facet Sealing subsystem, which provides cryptographically sealed manifests for logical slices of container images, enabling fine-grained drift detection, per-facet quota enforcement, and deterministic change tracking.
---
## 1. Overview
A **Facet** is a declared logical slice of a container image representing a cohesive set of files with shared characteristics:
| Facet Type | Description | Examples |
|------------|-------------|----------|
| `os` | Operating system packages | `/var/lib/dpkg/**`, `/var/lib/rpm/**` |
| `lang/<ecosystem>` | Language-specific dependencies | `node_modules/**`, `site-packages/**`, `vendor/**` |
| `binary` | Native binaries and shared libraries | `/usr/bin/*`, `/lib/**/*.so*` |
| `config` | Configuration files | `/etc/**`, `*.conf`, `*.yaml` |
| `custom` | User-defined patterns | Project-specific paths |
Each facet can be individually **sealed** (cryptographic snapshot) and monitored for **drift** (changes between seals).
---
## 2. System Landscape
```mermaid
graph TD
subgraph Scanner["Scanner Services"]
FE[FacetExtractor]
FH[FacetHasher]
MB[MerkleBuilder]
end
subgraph Storage["Facet Storage"]
FS[(PostgreSQL<br/>facet_seals)]
FC[(CAS<br/>facet_manifests)]
end
subgraph Policy["Policy & Enforcement"]
DC[DriftCalculator]
QE[QuotaEnforcer]
AV[AdmissionValidator]
end
subgraph Signing["Attestation"]
DS[DSSE Signer]
AT[Attestor]
end
subgraph CLI["CLI & Integration"]
SealCmd[stella seal]
DriftCmd[stella drift]
VexCmd[stella vex gen]
Zastava[Zastava Webhook]
end
FE --> FH
FH --> MB
MB --> DS
DS --> FS
DS --> FC
FS --> DC
DC --> QE
QE --> AV
AV --> Zastava
SealCmd --> FE
DriftCmd --> DC
VexCmd --> DC
```
---
## 3. Core Data Models
### 3.1 FacetDefinition
Declares a facet with its extraction patterns and quota constraints:
```csharp
public sealed record FacetDefinition
{
public required string FacetId { get; init; } // e.g., "os", "lang/node", "binary"
public required FacetType Type { get; init; } // OS, LangNode, LangPython, Binary, Config, Custom
public required ImmutableArray<string> IncludeGlobs { get; init; }
public ImmutableArray<string> ExcludeGlobs { get; init; } = [];
public FacetQuota? Quota { get; init; }
}
public enum FacetType
{
OS,
LangNode,
LangPython,
LangGo,
LangRust,
LangJava,
LangDotNet,
Binary,
Config,
Custom
}
```
### 3.2 FacetManifest
Per-facet file manifest with Merkle root:
```csharp
public sealed record FacetManifest
{
public required string FacetId { get; init; }
public required FacetType Type { get; init; }
public required ImmutableArray<FacetFileEntry> Files { get; init; }
public required string MerkleRoot { get; init; } // SHA-256 hex
public required int FileCount { get; init; }
public required long TotalBytes { get; init; }
public required DateTimeOffset ExtractedAt { get; init; }
public required string ExtractorVersion { get; init; }
}
public sealed record FacetFileEntry
{
public required string Path { get; init; } // Normalized POSIX path
public required string ContentHash { get; init; } // SHA-256 hex
public required long Size { get; init; }
public required string Mode { get; init; } // POSIX mode string "0644"
public required DateTimeOffset ModTime { get; init; } // Normalized to UTC
}
```
### 3.3 FacetSeal
DSSE-signed seal combining manifest with metadata:
```csharp
public sealed record FacetSeal
{
public required Guid SealId { get; init; }
public required string ImageRef { get; init; } // registry/repo:tag@sha256:...
public required string ImageDigest { get; init; } // sha256:...
public required FacetManifest Manifest { get; init; }
public required DateTimeOffset SealedAt { get; init; }
public required string SealedBy { get; init; } // Identity/service
public required FacetQuota? AppliedQuota { get; init; }
public required DsseEnvelope Envelope { get; init; }
}
```
### 3.4 FacetQuota
Per-facet change budget:
```csharp
public sealed record FacetQuota
{
public required string FacetId { get; init; }
public double MaxChurnPercent { get; init; } = 5.0; // 0-100
public int MaxChangedFiles { get; init; } = 50;
public int MaxAddedFiles { get; init; } = 25;
public int MaxRemovedFiles { get; init; } = 10;
public QuotaAction OnExceed { get; init; } = QuotaAction.Warn;
}
public enum QuotaAction
{
Warn, // Log warning, allow admission
Block, // Reject admission
RequireVex // Require VEX justification before admission
}
```
### 3.5 FacetDrift
Drift calculation result between two seals:
```csharp
public sealed record FacetDrift
{
public required string FacetId { get; init; }
public required Guid BaselineSealId { get; init; }
public required Guid CurrentSealId { get; init; }
public required ImmutableArray<DriftEntry> Added { get; init; }
public required ImmutableArray<DriftEntry> Removed { get; init; }
public required ImmutableArray<DriftEntry> Modified { get; init; }
public required DriftScore Score { get; init; }
public required QuotaVerdict QuotaVerdict { get; init; }
}
public sealed record DriftEntry
{
public required string Path { get; init; }
public string? OldHash { get; init; }
public string? NewHash { get; init; }
public long? OldSize { get; init; }
public long? NewSize { get; init; }
public DriftCause Cause { get; init; } = DriftCause.Unknown;
}
public enum DriftCause
{
Unknown,
PackageUpdate,
ConfigChange,
BinaryRebuild,
NewDependency,
RemovedDependency,
SecurityPatch
}
public sealed record DriftScore
{
public required int TotalChanges { get; init; }
public required double ChurnPercent { get; init; }
public required int AddedCount { get; init; }
public required int RemovedCount { get; init; }
public required int ModifiedCount { get; init; }
}
public sealed record QuotaVerdict
{
public required bool Passed { get; init; }
public required ImmutableArray<QuotaViolation> Violations { get; init; }
public required QuotaAction RecommendedAction { get; init; }
}
public sealed record QuotaViolation
{
public required string QuotaField { get; init; } // e.g., "MaxChurnPercent"
public required double Limit { get; init; }
public required double Actual { get; init; }
public required string Message { get; init; }
}
```
---
## 4. Component Architecture
### 4.1 FacetExtractor
Extracts file entries from container images based on facet definitions:
```csharp
public interface IFacetExtractor
{
Task<FacetManifest> ExtractAsync(
string imageRef,
FacetDefinition definition,
CancellationToken ct = default);
Task<ImmutableArray<FacetManifest>> ExtractAllAsync(
string imageRef,
ImmutableArray<FacetDefinition> definitions,
CancellationToken ct = default);
}
```
Implementation notes:
- Uses existing `ISurfaceReader` for container layer traversal
- Normalizes paths to POSIX format (forward slashes, no trailing slashes)
- Computes SHA-256 content hashes for each file
- Normalizes timestamps to UTC, mode to POSIX string
- Sorts files lexicographically for deterministic ordering
### 4.2 FacetHasher
Computes Merkle tree for facet file entries:
```csharp
public interface IFacetHasher
{
FacetMerkleResult ComputeMerkle(ImmutableArray<FacetFileEntry> files);
}
public sealed record FacetMerkleResult
{
public required string Root { get; init; }
public required ImmutableArray<string> LeafHashes { get; init; }
public required ImmutableArray<MerkleProofNode> Proof { get; init; }
}
```
Implementation notes:
- Leaf hash = SHA-256(path || contentHash || size || mode)
- Binary Merkle tree with lexicographic leaf ordering
- Empty facet produces well-known empty root hash
- Proof enables verification of individual file membership
### 4.3 FacetSealStore
PostgreSQL storage for sealed facet manifests:
```sql
-- Core seal storage
CREATE TABLE facet_seals (
seal_id UUID PRIMARY KEY,
tenant TEXT NOT NULL,
image_ref TEXT NOT NULL,
image_digest TEXT NOT NULL,
facet_id TEXT NOT NULL,
facet_type TEXT NOT NULL,
merkle_root TEXT NOT NULL,
file_count INTEGER NOT NULL,
total_bytes BIGINT NOT NULL,
sealed_at TIMESTAMPTZ NOT NULL,
sealed_by TEXT NOT NULL,
quota_json JSONB,
manifest_cas TEXT NOT NULL, -- CAS URI to full manifest
dsse_envelope JSONB NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
CONSTRAINT uq_facet_seal UNIQUE (tenant, image_digest, facet_id)
);
CREATE INDEX ix_facet_seals_image ON facet_seals (tenant, image_digest);
CREATE INDEX ix_facet_seals_merkle ON facet_seals (merkle_root);
-- Drift history
CREATE TABLE facet_drift_history (
drift_id UUID PRIMARY KEY,
tenant TEXT NOT NULL,
baseline_seal_id UUID NOT NULL REFERENCES facet_seals(seal_id),
current_seal_id UUID NOT NULL REFERENCES facet_seals(seal_id),
facet_id TEXT NOT NULL,
drift_score_json JSONB NOT NULL,
quota_verdict_json JSONB NOT NULL,
computed_at TIMESTAMPTZ NOT NULL,
CONSTRAINT uq_drift_pair UNIQUE (baseline_seal_id, current_seal_id)
);
```
### 4.4 DriftCalculator
Computes drift between baseline and current seals:
```csharp
public interface IDriftCalculator
{
Task<FacetDrift> CalculateAsync(
Guid baselineSealId,
Guid currentSealId,
CancellationToken ct = default);
Task<ImmutableArray<FacetDrift>> CalculateAllAsync(
string imageDigestBaseline,
string imageDigestCurrent,
CancellationToken ct = default);
}
```
Implementation notes:
- Retrieves manifests from CAS via seal metadata
- Performs set difference operations on file paths
- Detects modifications via content hash comparison
- Attributes drift causes where determinable (e.g., package manager metadata)
### 4.5 QuotaEnforcer
Evaluates drift against quota constraints:
```csharp
public interface IQuotaEnforcer
{
QuotaVerdict Evaluate(FacetDrift drift, FacetQuota quota);
Task<ImmutableArray<QuotaVerdict>> EvaluateAllAsync(
ImmutableArray<FacetDrift> drifts,
ImmutableDictionary<string, FacetQuota> quotas,
CancellationToken ct = default);
}
```
### 4.6 AdmissionValidator
Zastava webhook integration for admission control:
```csharp
public interface IFacetAdmissionValidator
{
Task<AdmissionResult> ValidateAsync(
AdmissionRequest request,
CancellationToken ct = default);
}
public sealed record AdmissionResult
{
public required bool Allowed { get; init; }
public string? Message { get; init; }
public ImmutableArray<QuotaViolation> Violations { get; init; } = [];
public string? RequiredVexStatement { get; init; }
}
```
---
## 5. DSSE Envelope Structure
Facet seals use DSSE (Dead Simple Signing Envelope) for cryptographic binding:
```json
{
"payloadType": "application/vnd.stellaops.facet-seal.v1+json",
"payload": "<base64url-encoded canonical JSON of FacetSeal>",
"signatures": [
{
"keyid": "sha256:abc123...",
"sig": "<base64url-encoded signature>"
}
]
}
```
Payload structure (canonical JSON, RFC 8785):
```json
{
"_type": "https://stellaops.io/FacetSeal/v1",
"facetId": "os",
"facetType": "OS",
"imageDigest": "sha256:abc123...",
"imageRef": "registry.example.com/app:v1.2.3",
"manifest": {
"extractedAt": "2026-01-05T10:00:00.000Z",
"extractorVersion": "1.0.0",
"fileCount": 1234,
"files": [
{
"contentHash": "sha256:...",
"mode": "0644",
"modTime": "2026-01-01T00:00:00.000Z",
"path": "/etc/os-release",
"size": 256
}
],
"merkleRoot": "sha256:def456...",
"totalBytes": 1048576
},
"quota": {
"maxAddedFiles": 25,
"maxChangedFiles": 50,
"maxChurnPercent": 5.0,
"maxRemovedFiles": 10,
"onExceed": "Warn"
},
"sealId": "550e8400-e29b-41d4-a716-446655440000",
"sealedAt": "2026-01-05T10:05:00.000Z",
"sealedBy": "scanner-worker-01"
}
```
---
## 6. Default Facet Definitions
Standard facet definitions applied when no custom configuration is provided:
```yaml
# Default facet configuration
facets:
- facetId: os
type: OS
includeGlobs:
- /var/lib/dpkg/**
- /var/lib/rpm/**
- /var/lib/pacman/**
- /var/lib/apk/**
- /var/cache/apt/**
- /etc/apt/**
- /etc/yum.repos.d/**
excludeGlobs:
- "**/*.log"
quota:
maxChurnPercent: 5.0
maxChangedFiles: 100
onExceed: Warn
- facetId: lang/node
type: LangNode
includeGlobs:
- "**/node_modules/**"
- "**/package.json"
- "**/package-lock.json"
- "**/yarn.lock"
- "**/pnpm-lock.yaml"
quota:
maxChurnPercent: 10.0
maxChangedFiles: 500
onExceed: RequireVex
- facetId: lang/python
type: LangPython
includeGlobs:
- "**/site-packages/**"
- "**/dist-packages/**"
- "**/requirements.txt"
- "**/Pipfile.lock"
- "**/poetry.lock"
quota:
maxChurnPercent: 10.0
maxChangedFiles: 200
onExceed: Warn
- facetId: lang/go
type: LangGo
includeGlobs:
- "**/go.mod"
- "**/go.sum"
- "**/vendor/**"
quota:
maxChurnPercent: 15.0
maxChangedFiles: 100
onExceed: Warn
- facetId: binary
type: Binary
includeGlobs:
- /usr/bin/*
- /usr/sbin/*
- /bin/*
- /sbin/*
- /usr/lib/**/*.so*
- /lib/**/*.so*
- /usr/local/bin/*
excludeGlobs:
- "**/*.py"
- "**/*.sh"
quota:
maxChurnPercent: 2.0
maxChangedFiles: 20
onExceed: Block
- facetId: config
type: Config
includeGlobs:
- /etc/**
- "**/*.conf"
- "**/*.cfg"
- "**/*.ini"
- "**/*.yaml"
- "**/*.yml"
- "**/*.json"
excludeGlobs:
- /etc/passwd
- /etc/shadow
- /etc/group
- "**/*.log"
quota:
maxChurnPercent: 20.0
maxChangedFiles: 50
onExceed: Warn
```
---
## 7. Integration Points
### 7.1 Scanner Integration
Scanner invokes facet extraction during scan:
```csharp
// In ScanOrchestrator
var facetDefs = await _facetConfigLoader.LoadAsync(scanRequest.FacetConfig, ct);
var manifests = await _facetExtractor.ExtractAllAsync(imageRef, facetDefs, ct);
foreach (var manifest in manifests)
{
var seal = await _facetSealer.SealAsync(manifest, scanRequest, ct);
await _facetSealStore.SaveAsync(seal, ct);
}
```
### 7.2 CLI Integration
```bash
# Seal all facets for an image
stella seal myregistry.io/app:v1.2.3 --output seals.json
# Seal specific facets
stella seal myregistry.io/app:v1.2.3 --facet os --facet lang/node
# Check drift between two image versions
stella drift myregistry.io/app:v1.2.3 myregistry.io/app:v1.2.4 --format json
# Generate VEX from drift
stella vex gen --from-drift myregistry.io/app:v1.2.3 myregistry.io/app:v1.2.4
```
### 7.3 Zastava Webhook Integration
```csharp
// In FacetAdmissionValidator
public async Task<AdmissionResult> ValidateAsync(AdmissionRequest request, CancellationToken ct)
{
// Find baseline seal (latest approved)
var baseline = await _sealStore.GetLatestApprovedAsync(request.ImageRef, ct);
if (baseline is null)
return AdmissionResult.Allowed("No baseline seal found, skipping facet check");
// Extract current facets
var currentManifests = await _extractor.ExtractAllAsync(request.ImageRef, _defaultFacets, ct);
// Calculate drift for each facet
var drifts = new List<FacetDrift>();
foreach (var manifest in currentManifests)
{
var baselineSeal = baseline.FirstOrDefault(s => s.FacetId == manifest.FacetId);
if (baselineSeal is not null)
{
var drift = await _driftCalculator.CalculateAsync(baselineSeal, manifest, ct);
drifts.Add(drift);
}
}
// Evaluate quotas
var violations = new List<QuotaViolation>();
QuotaAction maxAction = QuotaAction.Warn;
foreach (var drift in drifts)
{
var verdict = _quotaEnforcer.Evaluate(drift, drift.AppliedQuota);
if (!verdict.Passed)
{
violations.AddRange(verdict.Violations);
if (verdict.RecommendedAction > maxAction)
maxAction = verdict.RecommendedAction;
}
}
return maxAction switch
{
QuotaAction.Block => AdmissionResult.Denied(violations),
QuotaAction.RequireVex => AdmissionResult.RequiresVex(violations),
_ => AdmissionResult.Allowed(violations)
};
}
```
---
## 8. Observability
### 8.1 Metrics
| Metric | Type | Labels | Description |
|--------|------|--------|-------------|
| `facet_seal_total` | Counter | `tenant`, `facet_type`, `status` | Total seals created |
| `facet_seal_duration_seconds` | Histogram | `facet_type` | Time to create seal |
| `facet_drift_score` | Gauge | `tenant`, `facet_id`, `image` | Current drift score |
| `facet_quota_violations_total` | Counter | `tenant`, `facet_id`, `quota_field` | Quota violations |
| `facet_admission_decisions_total` | Counter | `tenant`, `decision`, `facet_id` | Admission decisions |
### 8.2 Traces
```
facet.extract - Facet file extraction from image
facet.hash - Merkle tree computation
facet.seal - DSSE signing
facet.drift.compute - Drift calculation
facet.quota.evaluate - Quota enforcement
facet.admission - Admission validation
```
### 8.3 Logs
Structured log fields:
- `facetId`: Facet identifier
- `imageRef`: Container image reference
- `imageDigest`: Image content digest
- `merkleRoot`: Facet Merkle root
- `driftScore`: Computed drift percentage
- `quotaVerdict`: Pass/fail status
---
## 9. Security Considerations
1. **Signature Verification**: All seals must be DSSE-signed with keys managed by Authority service
2. **Tenant Isolation**: Seals are scoped to tenants; cross-tenant access is prohibited
3. **Immutability**: Once created, seals cannot be modified; only superseded by new seals
4. **Audit Trail**: All seal operations are logged with correlation IDs
5. **Key Rotation**: Signing keys support rotation; old signatures remain valid with archived keys
---
## 10. References
- [DSSE Specification](https://github.com/secure-systems-lab/dsse)
- [RFC 8785 - JSON Canonicalization](https://tools.ietf.org/html/rfc8785)
- [Scanner Architecture](../scanner/architecture.md)
- [Attestor Architecture](../attestor/architecture.md)
- [Policy Engine Architecture](../policy/architecture.md)
- [Replay Architecture](../replay/architecture.md)
---
*Last updated: 2026-01-05*

622
docs-archived/modules/prov-cache/README.md Normal file
View File

@@ -0,0 +1,622 @@
# Provcache Module
> **Status: Implemented** — Core library shipped in Sprint 8200.0001.0001. API endpoints, caching, invalidation and write-behind queue are operational. Policy Engine integration pending architectural review.
> Provenance Cache — Maximizing Trust Evidence Density
## Overview
Provcache is a caching layer that maximizes "provenance density" — the amount of trustworthy evidence retained per byte — enabling faster security decisions, offline replays, and smaller air-gap bundles.
### Key Benefits
- **Trust Latency**: Warm cache lookups return in single-digit milliseconds
- **Bandwidth Efficiency**: Avoid re-fetching bulky SBOMs/attestations
- **Offline Operation**: Decisions usable without full SBOM/VEX payloads
- **Audit Transparency**: Full evidence chain verifiable via Merkle proofs
## Architecture
```
┌─────────────────────────────────────────────────────────────────┐
│ Policy Evaluator │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────┐ │
│ │ VeriKey │───▶│ Provcache │───▶│ TrustLatticeEngine │ │
│ │ Builder │ │ Service │ │ (if cache miss) │ │
│ └─────────────┘ └─────────────┘ └─────────────────────┘ │
└─────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────┐
│ Provcache Store │
│ ┌─────────────┐ ┌────────────────┐ │
│ │ Valkey │◀──▶│ Postgres │ │
│ │ (read-thru) │ │ (write-behind) │ │
│ └─────────────┘ └────────────────┘ │
└─────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────┐
│ Evidence Chunk Store │
│ ┌─────────────────────────────────────┐│
│ │ prov_evidence_chunks (Postgres) ││
│ │ - Chunked SBOM/VEX/CallGraph ││
│ │ - Merkle tree verification ││
│ └─────────────────────────────────────┘│
└─────────────────────────────────────────┘
```
## Core Concepts
### VeriKey (Provenance Identity Key)
A composite hash that uniquely identifies a provenance decision context:
```
VeriKey = SHA256(
"v1|" || // Version prefix for compatibility
source_hash || // Image/artifact digest
"|" ||
sbom_hash || // Canonical SBOM hash
"|" ||
vex_hash_set_hash || // Sorted VEX statement hashes
"|" ||
merge_policy_hash || // PolicyBundle hash
"|" ||
signer_set_hash || // Signer certificate hashes
"|" ||
time_window // Epoch bucket
)
```
**Why each component?**
| Component | Purpose |
|-----------|---------|
| `source_hash` | Different artifacts → different keys |
| `sbom_hash` | SBOM changes (new packages) → new key |
| `vex_hash_set` | VEX updates → new key |
| `policy_hash` | Policy changes → new key |
| `signer_set_hash` | Key rotation → new key (security) |
| `time_window` | Temporal bucketing → controlled expiry |
#### VeriKey Composition Rules
1. **Hash Normalization**: All input hashes are normalized to lowercase with `sha256:` prefix stripped if present
2. **Set Hash Computation**: For VEX statements and signer certificates:
- Individual hashes are sorted lexicographically (ordinal)
- Sorted hashes are concatenated with `|` delimiter
- Result is SHA256-hashed
- Empty sets use well-known sentinels (`"empty-vex-set"`, `"empty-signer-set"`)
3. **Time Window Computation**: `floor(timestamp.Ticks / bucket.Ticks) * bucket.Ticks` in UTC ISO-8601 format
4. **Output Format**: `sha256:<64-char-lowercase-hex>`
#### Code Example
```csharp
var veriKey = new VeriKeyBuilder(options)
.WithSourceHash("sha256:abc123...") // Image digest
.WithSbomHash("sha256:def456...") // SBOM digest
.WithVexStatementHashes(["sha256:v1", "sha256:v2"]) // Sorted automatically
.WithMergePolicyHash("sha256:policy...") // Policy bundle
.WithCertificateHashes(["sha256:cert1"]) // Signer certs
.WithTimeWindow(DateTimeOffset.UtcNow) // Auto-bucketed
.Build();
// Returns: "sha256:789abc..."
```
### DecisionDigest
Canonicalized representation of an evaluation result:
```json
{
"digestVersion": "v1",
"veriKey": "sha256:abc123...",
"verdictHash": "sha256:def456...",
"proofRoot": "sha256:789abc...",
"replaySeed": {
"feedIds": ["cve-2024", "ghsa-2024"],
"ruleIds": ["default-policy-v2"]
},
"trustScore": 85,
"createdAt": "2025-12-24T12:00:00Z",
"expiresAt": "2025-12-25T12:00:00Z"
}
```
### Trust Score
A composite score (0-100) indicating decision confidence:
| Component | Weight | Calculation |
|-----------|--------|-------------|
| Reachability | 25% | Call graph coverage, entry points analyzed |
| SBOM Completeness | 20% | Package count, license data presence |
| VEX Coverage | 20% | Vendor statements, justifications |
| Policy Freshness | 15% | Time since last policy update |
| Signer Trust | 20% | Key age, reputation, chain validity |
### Evidence Chunks
Large evidence (SBOM, VEX, call graphs) is stored in fixed-size chunks:
- **Default size**: 64 KB per chunk
- **Merkle verification**: Each chunk is a Merkle leaf
- **Lazy fetch**: Only fetch chunks needed for audit
- **LRU eviction**: Old chunks evicted under storage pressure
## API Reference
### Endpoints
| Method | Path | Description |
|--------|------|-------------|
| GET | `/v1/provcache/{veriKey}` | Lookup cached decision |
| POST | `/v1/provcache` | Store decision (idempotent) |
| POST | `/v1/provcache/invalidate` | Invalidate by pattern |
| GET | `/v1/proofs/{proofRoot}` | List evidence chunks |
| GET | `/v1/proofs/{proofRoot}/chunks/{index}` | Download chunk |
### Cache Lookup Flow
```mermaid
sequenceDiagram
participant Client
participant PolicyEngine
participant Provcache
participant Valkey
participant Postgres
participant TrustLattice
Client->>PolicyEngine: Evaluate(artifact)
PolicyEngine->>Provcache: Get(VeriKey)
Provcache->>Valkey: GET verikey
alt Cache Hit
Valkey-->>Provcache: DecisionDigest
Provcache-->>PolicyEngine: CacheResult(hit)
PolicyEngine-->>Client: Decision (cached)
else Cache Miss
Valkey-->>Provcache: null
Provcache->>Postgres: SELECT * FROM provcache_items
alt DB Hit
Postgres-->>Provcache: ProvcacheEntry
Provcache->>Valkey: SET (backfill)
Provcache-->>PolicyEngine: CacheResult(hit, source=postgres)
else DB Miss
Postgres-->>Provcache: null
Provcache-->>PolicyEngine: CacheResult(miss)
PolicyEngine->>TrustLattice: Evaluate
TrustLattice-->>PolicyEngine: EvaluationResult
PolicyEngine->>Provcache: Set(VeriKey, DecisionDigest)
Provcache->>Valkey: SET
Provcache->>Postgres: INSERT (async)
PolicyEngine-->>Client: Decision (computed)
end
end
```
## Invalidation
> **See also**: [architecture.md](architecture.md#invalidation-mechanisms) for detailed invalidation flow diagrams.
### Automatic Invalidation Triggers
| Trigger | Event | Scope | Implementation |
|---------|-------|-------|----------------|
| Signer Revocation | `SignerRevokedEvent` | All entries with matching `signer_set_hash` | `SignerSetInvalidator` |
| Feed Epoch Advance | `FeedEpochAdvancedEvent` | Entries with older `feed_epoch` | `FeedEpochInvalidator` |
| Policy Update | `PolicyUpdatedEvent` | Entries with matching `policy_hash` | `PolicyHashInvalidator` |
| TTL Expiry | Background job | Entries past `expires_at` | `TtlExpirationService` |
### Invalidation Interfaces
```csharp
// Main invalidator interface
public interface IProvcacheInvalidator
{
Task<int> InvalidateAsync(
InvalidationCriteria criteria,
string reason,
string? correlationId = null,
CancellationToken cancellationToken = default);
}
// Revocation ledger for audit trail
public interface IRevocationLedger
{
Task RecordAsync(RevocationEntry entry, CancellationToken ct = default);
Task<IReadOnlyList<RevocationEntry>> GetEntriesSinceAsync(long sinceSeqNo, int limit = 1000, CancellationToken ct = default);
Task<RevocationLedgerStats> GetStatsAsync(CancellationToken ct = default);
}
```
### Manual Invalidation
```bash
# Invalidate by signer
POST /v1/provcache/invalidate
{
"by": "signer_set_hash",
"value": "sha256:revoked-signer...",
"reason": "key-compromise"
}
# Invalidate by policy
POST /v1/provcache/invalidate
{
"by": "policy_hash",
"value": "sha256:old-policy...",
"reason": "policy-update"
}
```
### Revocation Replay
Nodes can replay missed revocation events after restart or network partition:
```csharp
var replayService = services.GetRequiredService<IRevocationReplayService>();
var checkpoint = await replayService.GetCheckpointAsync();
var result = await replayService.ReplayFromAsync(
sinceSeqNo: checkpoint,
new RevocationReplayOptions { BatchSize = 1000 });
// result.EntriesReplayed, result.TotalInvalidations
```
## Air-Gap Integration
> **See also**: [architecture.md](architecture.md#air-gap-exportimport) for bundle format specification and architecture diagrams.
### Export Workflow
```bash
# Export minimal proof (digest only)
stella prov export --verikey sha256:abc123 --density lite
# Export with evidence chunks
stella prov export --verikey sha256:abc123 --density standard
# Export full evidence
stella prov export --verikey sha256:abc123 --density strict --sign
```
### Import Workflow
```bash
# Import and verify Merkle root
stella prov import --input proof.bundle
# Import with lazy chunk fetch (connected mode)
stella prov import --input proof-lite.json --lazy-fetch --backend https://api.stellaops.com
# Import with lazy fetch from file directory (sneakernet mode)
stella prov import --input proof-lite.json --lazy-fetch --chunks-dir /mnt/usb/evidence
```
### Density Levels
| Level | Contents | Size | Use Case | Lazy Fetch Support |
|-------|----------|------|----------|--------------------|
| `lite` | DecisionDigest + ProofRoot + Manifest | ~2 KB | Quick verification | Required |
| `standard` | + First N chunks (~10%) | ~200 KB | Normal audit | Partial (remaining chunks) |
| `strict` | + All chunks | Variable | Full compliance | Not needed |
### Lazy Evidence Fetching
For `lite` and `standard` density exports, missing chunks can be fetched on-demand:
```csharp
// HTTP fetcher (connected mode)
var httpFetcher = new HttpChunkFetcher(
new Uri("https://api.stellaops.com"), logger);
// File fetcher (air-gapped/sneakernet mode)
var fileFetcher = new FileChunkFetcher(
basePath: "/mnt/usb/evidence", logger);
// Orchestrate fetch + verify + store
var orchestrator = new LazyFetchOrchestrator(repository, logger);
var result = await orchestrator.FetchAndStoreAsync(
proofRoot: "sha256:...",
fetcher,
new LazyFetchOptions
{
VerifyOnFetch = true,
BatchSize = 100,
MaxChunks = 1000
});
```
### Sneakernet Export for Chunked Evidence
```csharp
// Export evidence chunks to file system for transport
await fileFetcher.ExportEvidenceChunksToFilesAsync(
manifest,
chunks,
outputDirectory: "/mnt/usb/evidence");
```
## Configuration
### C# Configuration Class
The `ProvcacheOptions` class (section name: `"Provcache"`) exposes the following settings:
| Property | Type | Default | Validation | Description |
|----------|------|---------|------------|-------------|
| `DefaultTtl` | `TimeSpan` | 24h | 1min–7d | Default time-to-live for cache entries |
| `MaxTtl` | `TimeSpan` | 7d | 1min–30d | Maximum allowed TTL regardless of request |
| `TimeWindowBucket` | `TimeSpan` | 1h | 1min–24h | Time window bucket for VeriKey computation |
| `ValkeyKeyPrefix` | `string` | `"stellaops:prov:"` | — | Key prefix for Valkey storage |
| `EnableWriteBehind` | `bool` | `true` | — | Enable async Postgres persistence |
| `WriteBehindFlushInterval` | `TimeSpan` | 5s | 1s–5min | Interval for flushing write-behind queue |
| `WriteBehindMaxBatchSize` | `int` | 100 | 1–10000 | Maximum batch size per flush |
| `WriteBehindQueueCapacity` | `int` | 10000 | 100–1M | Max queue capacity (blocks when full) |
| `WriteBehindMaxRetries` | `int` | 3 | 0–10 | Retry attempts for failed writes |
| `ChunkSize` | `int` | 65536 | 1KB–1MB | Evidence chunk size in bytes |
| `MaxChunksPerEntry` | `int` | 1000 | 1–100000 | Max chunks per cache entry |
| `AllowCacheBypass` | `bool` | `true` | — | Allow clients to force re-evaluation |
| `DigestVersion` | `string` | `"v1"` | — | Serialization version for digests |
| `HashAlgorithm` | `string` | `"SHA256"` | — | Hash algorithm for VeriKey/digest |
| `EnableValkeyCache` | `bool` | `true` | — | Enable Valkey layer (false = Postgres only) |
| `SlidingExpiration` | `bool` | `false` | — | Refresh TTL on cache hits |
### appsettings.json Example
```json
{
"Provcache": {
"DefaultTtl": "24:00:00",
"MaxTtl": "7.00:00:00",
"TimeWindowBucket": "01:00:00",
"ValkeyKeyPrefix": "stellaops:prov:",
"EnableWriteBehind": true,
"WriteBehindFlushInterval": "00:00:05",
"WriteBehindMaxBatchSize": 100,
"WriteBehindQueueCapacity": 10000,
"WriteBehindMaxRetries": 3,
"ChunkSize": 65536,
"MaxChunksPerEntry": 1000,
"AllowCacheBypass": true,
"DigestVersion": "v1",
"HashAlgorithm": "SHA256",
"EnableValkeyCache": true,
"SlidingExpiration": false
}
}
```
### YAML Example (Helm/Kubernetes)
```yaml
provcache:
# TTL configuration
defaultTtl: 24h
maxTtl: 168h # 7 days
timeWindowBucket: 1h
# Storage
valkeyKeyPrefix: "stellaops:prov:"
enableWriteBehind: true
writeBehindFlushInterval: 5s
writeBehindMaxBatchSize: 100
# Evidence chunking
chunkSize: 65536 # 64 KB
maxChunksPerEntry: 1000
# Behavior
allowCacheBypass: true
digestVersion: "v1"
```
### Dependency Injection Registration
```csharp
// In Program.cs or Startup.cs
services.AddProvcache(configuration);
// Or with explicit configuration
services.AddProvcache(options =>
{
options.DefaultTtl = TimeSpan.FromHours(12);
options.EnableWriteBehind = true;
options.WriteBehindMaxBatchSize = 200;
});
```
## Observability
### Metrics
| Metric | Type | Description |
|--------|------|-------------|
| `provcache_requests_total` | Counter | Total cache requests |
| `provcache_hits_total` | Counter | Cache hits |
| `provcache_misses_total` | Counter | Cache misses |
| `provcache_latency_seconds` | Histogram | Operation latency |
| `provcache_items_count` | Gauge | Current item count |
| `provcache_invalidations_total` | Counter | Invalidation count |
### Alerts
```yaml
# Low cache hit rate
- alert: ProvcacheLowHitRate
expr: rate(provcache_hits_total[5m]) / rate(provcache_requests_total[5m]) < 0.5
for: 10m
labels:
severity: warning
annotations:
summary: "Provcache hit rate below 50%"
# High invalidation rate
- alert: ProvcacheHighInvalidationRate
expr: rate(provcache_invalidations_total[5m]) > 100
for: 5m
labels:
severity: warning
annotations:
summary: "High cache invalidation rate"
```
## Security Considerations
### Signer-Aware Caching
The `signer_set_hash` is part of the VeriKey, ensuring:
- Key rotation → new cache entries
- Key revocation → immediate invalidation
- No stale decisions from compromised signers
### Merkle Verification
All evidence chunks are Merkle-verified:
- `ProofRoot` = Merkle root of all chunks
- Individual chunks verifiable without full tree
- Tamper detection on import
### Audit Trail
All invalidations are logged to `prov_revocations` table:
```sql
SELECT * FROM provcache.prov_revocations
WHERE created_at > NOW() - INTERVAL '24 hours'
ORDER BY created_at DESC;
```
## Database Schema
### provcache_items
```sql
CREATE TABLE provcache.provcache_items (
verikey TEXT PRIMARY KEY,
digest_version TEXT NOT NULL,
verdict_hash TEXT NOT NULL,
proof_root TEXT NOT NULL,
replay_seed JSONB NOT NULL,
policy_hash TEXT NOT NULL,
signer_set_hash TEXT NOT NULL,
feed_epoch TEXT NOT NULL,
trust_score INTEGER NOT NULL,
hit_count BIGINT DEFAULT 0,
created_at TIMESTAMPTZ NOT NULL,
expires_at TIMESTAMPTZ NOT NULL,
updated_at TIMESTAMPTZ NOT NULL
);
```
### prov_evidence_chunks
```sql
CREATE TABLE provcache.prov_evidence_chunks (
chunk_id UUID PRIMARY KEY,
proof_root TEXT NOT NULL REFERENCES provcache_items(proof_root),
chunk_index INTEGER NOT NULL,
chunk_hash TEXT NOT NULL,
blob BYTEA NOT NULL,
blob_size INTEGER NOT NULL,
content_type TEXT NOT NULL,
created_at TIMESTAMPTZ NOT NULL
);
```
### prov_revocations
```sql
CREATE TABLE provcache.prov_revocations (
seq_no BIGSERIAL PRIMARY KEY,
revocation_id UUID NOT NULL UNIQUE,
revocation_type VARCHAR(32) NOT NULL, -- signer, feed_epoch, policy, explicit, expiration
revoked_key VARCHAR(512) NOT NULL,
reason VARCHAR(1024),
entries_invalidated INTEGER NOT NULL,
source VARCHAR(128) NOT NULL,
correlation_id VARCHAR(128),
revoked_at TIMESTAMPTZ NOT NULL,
metadata JSONB,
CONSTRAINT chk_revocation_type CHECK (
revocation_type IN ('signer', 'feed_epoch', 'policy', 'explicit', 'expiration')
)
);
CREATE INDEX idx_revocations_type ON provcache.prov_revocations(revocation_type);
CREATE INDEX idx_revocations_key ON provcache.prov_revocations(revoked_key);
CREATE INDEX idx_revocations_time ON provcache.prov_revocations(revoked_at);
```
## Implementation Status
### Completed (Sprint 8200.0001.0001 - Core Backend)
| Component | Path | Status |
|-----------|------|--------|
| Core Models | `src/__Libraries/StellaOps.Provcache/Models/` | ✅ Done |
| VeriKeyBuilder | `src/__Libraries/StellaOps.Provcache/VeriKeyBuilder.cs` | ✅ Done |
| DecisionDigest | `src/__Libraries/StellaOps.Provcache/DecisionDigest.cs` | ✅ Done |
| Caching Layer | `src/__Libraries/StellaOps.Provcache/Caching/` | ✅ Done |
| WriteBehindQueue | `src/__Libraries/StellaOps.Provcache/Persistence/` | ✅ Done |
| API Endpoints | `src/__Libraries/StellaOps.Provcache.Api/` | ✅ Done |
| Unit Tests (53) | `src/__Libraries/__Tests/StellaOps.Provcache.Tests/` | ✅ Done |
### Completed (Sprint 8200.0001.0002 - Invalidation & Air-Gap)
| Component | Path | Status |
|-----------|------|--------|
| Invalidation Interfaces | `src/__Libraries/StellaOps.Provcache/Invalidation/` | ✅ Done |
| Repository Invalidation Methods | `IEvidenceChunkRepository.Delete*Async()` | ✅ Done |
| Export Interfaces | `src/__Libraries/StellaOps.Provcache/Export/` | ✅ Done |
| IMinimalProofExporter | `Export/IMinimalProofExporter.cs` | ✅ Done |
| MinimalProofExporter | `Export/MinimalProofExporter.cs` | ✅ Done |
| Lazy Fetch - ILazyEvidenceFetcher | `LazyFetch/ILazyEvidenceFetcher.cs` | ✅ Done |
| Lazy Fetch - HttpChunkFetcher | `LazyFetch/HttpChunkFetcher.cs` | ✅ Done |
| Lazy Fetch - FileChunkFetcher | `LazyFetch/FileChunkFetcher.cs` | ✅ Done |
| Lazy Fetch - LazyFetchOrchestrator | `LazyFetch/LazyFetchOrchestrator.cs` | ✅ Done |
| Revocation - IRevocationLedger | `Revocation/IRevocationLedger.cs` | ✅ Done |
| Revocation - InMemoryRevocationLedger | `Revocation/InMemoryRevocationLedger.cs` | ✅ Done |
| Revocation - RevocationReplayService | `Revocation/RevocationReplayService.cs` | ✅ Done |
| ProvRevocationEntity | `Entities/ProvRevocationEntity.cs` | ✅ Done |
| Unit Tests (124 total) | `src/__Libraries/__Tests/StellaOps.Provcache.Tests/` | ✅ Done |
### Blocked
| Component | Reason |
|-----------|--------|
| Policy Engine Integration | `PolicyEvaluator` is `internal sealed`; requires architectural review to expose injection points for `IProvcacheService` |
| CLI e2e Tests | `AddSimRemoteCryptoProvider` method missing in CLI codebase |
### Pending
| Component | Sprint |
|-----------|--------|
| Authority Event Integration | 8200.0001.0002 (BLOCKED - Authority needs event publishing) |
| Concelier Event Integration | 8200.0001.0002 (BLOCKED - Concelier needs event publishing) |
| PostgresRevocationLedger | Future (requires EF Core integration) |
| UI Badges & Proof Tree | 8200.0001.0003 |
| Grafana Dashboards | 8200.0001.0003 |
## Implementation Sprints
| Sprint | Focus | Key Deliverables |
|--------|-------|------------------|
| [8200.0001.0001](../../implplan/SPRINT_8200_0001_0001_provcache_core_backend.md) | Core Backend | VeriKey, DecisionDigest, Valkey+Postgres, API |
| [8200.0001.0002](../../implplan/SPRINT_8200_0001_0002_provcache_invalidation_airgap.md) | Invalidation & Air-Gap | Signer revocation, feed epochs, CLI export/import |
| [8200.0001.0003](../../implplan/SPRINT_8200_0001_0003_provcache_ux_observability.md) | UX & Observability | UI badges, proof tree, Grafana, OCI attestation |
## Related Documentation
- **[Provcache Architecture Guide](architecture.md)** - Detailed architecture, invalidation flows, and API reference
- [Policy Engine Architecture](../policy/README.md)
- [TrustLattice Engine](../policy/design/policy-deterministic-evaluator.md)
- [Offline Kit Documentation](../../OFFLINE_KIT.md)
- [Air-Gap Controller](../airgap/README.md)
- [Authority Key Rotation](../authority/README.md)

451
docs-archived/modules/prov-cache/architecture.md Normal file
View File

@@ -0,0 +1,451 @@
# Provcache Architecture Guide
> **Status: Production (Shared Library Family).** Provcache is a mature, heavily-used shared library family — not a planned component. The implementation spans four libraries: `src/__Libraries/StellaOps.Provcache/` (77 core files: VeriKey, DecisionDigest, chunking, write-behind queue, invalidation, telemetry), `StellaOps.Provcache.Postgres/` (16 files: EF Core persistence with `provcache` schema), `StellaOps.Provcache.Valkey/` (hot-cache layer), and `StellaOps.Provcache.Api/` (HTTP endpoints). Consumed by 89+ files across Policy Engine, Concelier, ExportCenter, CLI, and other modules. Comprehensive test coverage (89 test files). Actively maintained with recent determinism refactoring (DET-005).
> Detailed architecture documentation for the Provenance Cache module
## Overview
Provcache provides a caching layer that maximizes "provenance density" — the amount of trustworthy evidence retained per byte. This document covers the internal architecture, invalidation mechanisms, air-gap support, and replay capabilities.
## Table of Contents
1. [Cache Architecture](#cache-architecture)
2. [Invalidation Mechanisms](#invalidation-mechanisms)
3. [Evidence Chunk Storage](#evidence-chunk-storage)
4. [Air-Gap Export/Import](#air-gap-exportimport)
5. [Lazy Evidence Fetching](#lazy-evidence-fetching)
6. [Revocation Ledger](#revocation-ledger)
7. [API Reference](#api-reference)
---
## Cache Architecture
### Storage Layers
```
┌───────────────────────────────────────────────────────────────┐
│ Application Layer │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────────┐ │
│ │ VeriKey │───▶│ Provcache │───▶│ Policy Engine │ │
│ │ Builder │ │ Service │ │ (cache miss) │ │
│ └─────────────┘ └─────────────┘ └─────────────────┘ │
└───────────────────────────────────────────────────────────────┘
│
▼
┌───────────────────────────────────────────────────────────────┐
│ Caching Layer │
│ ┌─────────────────┐ ┌──────────────────────────┐ │
│ │ Valkey │◀───────▶│ PostgreSQL │ │
│ │ (read-through) │ │ (write-behind queue) │ │
│ │ │ │ │ │
│ │ • Hot cache │ │ • provcache_items │ │
│ │ • Sub-ms reads │ │ • prov_evidence_chunks │ │
│ │ • TTL-based │ │ • prov_revocations │ │
│ └─────────────────┘ └──────────────────────────┘ │
└───────────────────────────────────────────────────────────────┘
```
### Key Components
| Component | Purpose |
|-----------|---------|
| `IProvcacheService` | Main service interface for cache operations |
| `IProvcacheStore` | Storage abstraction (Valkey + Postgres) |
| `WriteBehindQueue` | Async persistence to Postgres |
| `IEvidenceChunker` | Splits large evidence into Merkle-verified chunks |
| `IRevocationLedger` | Audit trail for all invalidation events |
---
## Invalidation Mechanisms
Provcache supports multiple invalidation triggers to ensure cache consistency when upstream data changes.
### Automatic Invalidation
#### 1. Signer Revocation
When a signing key is compromised or rotated:
```
┌─────────────┐ SignerRevokedEvent ┌──────────────────┐
│ Authority │ ──────────────────────────▶│ SignerSet │
│ Module │ │ Invalidator │
└─────────────┘ └────────┬─────────┘
│
▼
DELETE FROM provcache_items
WHERE signer_set_hash = ?
```
**Implementation**: `SignerSetInvalidator` subscribes to `SignerRevokedEvent` and invalidates all entries signed by the revoked key.
#### 2. Feed Epoch Advancement
When vulnerability feeds are updated:
```
┌─────────────┐ FeedEpochAdvancedEvent ┌──────────────────┐
│ Concelier │ ───────────────────────────▶│ FeedEpoch │
│ Module │ │ Invalidator │
└─────────────┘ └────────┬─────────┘
│
▼
DELETE FROM provcache_items
WHERE feed_epoch < ?
```
**Implementation**: `FeedEpochInvalidator` compares epochs using semantic versioning or ISO timestamps.
#### 3. Policy Updates
When policy bundles change:
```
┌─────────────┐ PolicyUpdatedEvent ┌──────────────────┐
│ Policy │ ───────────────────────────▶│ PolicyHash │
│ Engine │ │ Invalidator │
└─────────────┘ └────────┬─────────┘
│
▼
DELETE FROM provcache_items
WHERE policy_hash = ?
```
### Invalidation DI Wiring and Lifecycle
`AddProvcacheInvalidators()` registers the event-driven invalidation pipeline in dependency injection:
- Creates `IEventStream<SignerRevokedEvent>` using `IEventStreamFactory.Create<T>(new EventStreamOptions { StreamName = SignerRevokedEvent.StreamName })`
- Creates `IEventStream<FeedEpochAdvancedEvent>` using `IEventStreamFactory.Create<T>(new EventStreamOptions { StreamName = FeedEpochAdvancedEvent.StreamName })`
- Registers `SignerSetInvalidator` and `FeedEpochInvalidator` as singleton `IProvcacheInvalidator` implementations
- Registers `InvalidatorHostedService` as `IHostedService` to own invalidator startup/shutdown
`InvalidatorHostedService` starts all registered invalidators during host startup and stops them in reverse order during host shutdown. Each invalidator subscribes from `StreamPosition.End`, so only new events are consumed after process start.
### Invalidation Recording
All invalidation events are recorded in the revocation ledger for audit and replay:
```csharp
public interface IProvcacheInvalidator
{
Task<int> InvalidateAsync(
InvalidationCriteria criteria,
string reason,
string? correlationId = null,
CancellationToken cancellationToken = default);
}
```
The ledger entry includes:
- Revocation type (signer, feed_epoch, policy, explicit)
- The revoked key
- Number of entries invalidated
- Timestamp and correlation ID for tracing
---
## Evidence Chunk Storage
Large evidence (SBOMs, VEX documents, call graphs) is stored in fixed-size chunks with Merkle tree verification.
### Chunking Process
```
┌─────────────────────────────────────────────────────────────────┐
│ Original Evidence │
│ [ 2.3 MB SPDX SBOM JSON ] │
└─────────────────────────────────────────────────────────────────┘
│
▼ IEvidenceChunker.ChunkAsync()
┌─────────────────────────────────────────────────────────────────┐
│ Chunk 0 (64KB) │ Chunk 1 (64KB) │ ... │ Chunk N (partial) │
│ hash: abc123 │ hash: def456 │ │ hash: xyz789 │
└─────────────────────────────────────────────────────────────────┘
│
▼ Merkle tree construction
┌─────────────────────────────────────────────────────────────────┐
│ Proof Root │
│ sha256:merkle_root_of_all_chunks │
└─────────────────────────────────────────────────────────────────┘
```
### Database Schema
```sql
CREATE TABLE provcache.prov_evidence_chunks (
chunk_id UUID PRIMARY KEY,
proof_root VARCHAR(128) NOT NULL,
chunk_index INTEGER NOT NULL,
chunk_hash VARCHAR(128) NOT NULL,
blob BYTEA NOT NULL,
blob_size INTEGER NOT NULL,
content_type VARCHAR(64) NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
CONSTRAINT uk_proof_chunk UNIQUE (proof_root, chunk_index)
);
CREATE INDEX idx_evidence_proof_root ON provcache.prov_evidence_chunks(proof_root);
```
### Paging API
Evidence can be retrieved in pages to manage memory:
```http
GET /api/v1/proofs/{proofRoot}?page=0&pageSize=10
```
Response includes chunk metadata without blob data, allowing clients to fetch specific chunks on demand.
---
## Air-Gap Export/Import
Provcache supports air-gapped environments through minimal proof bundles.
### Bundle Format (v1)
```json
{
"version": "v1",
"exportedAt": "2025-01-15T10:30:00Z",
"density": "standard",
"digest": {
"veriKey": "sha256:...",
"verdictHash": "sha256:...",
"proofRoot": "sha256:...",
"trustScore": 85
},
"manifest": {
"proofRoot": "sha256:...",
"totalChunks": 42,
"totalSize": 2752512,
"chunks": [...]
},
"chunks": [
{
"index": 0,
"data": "base64...",
"hash": "sha256:..."
}
],
"signature": {
"algorithm": "ECDSA-P256",
"signature": "base64...",
"signedAt": "2025-01-15T10:30:01Z"
}
}
```
### Density Levels
| Level | Contents | Typical Size | Use Case |
|-------|----------|--------------|----------|
| **Lite** | Digest + ProofRoot + Manifest | ~2 KB | Quick verification, requires lazy fetch for full evidence |
| **Standard** | + First 10% of chunks | ~200 KB | Normal audits, balance of size vs completeness |
| **Strict** | + All chunks | Variable | Full compliance, no network needed |
### Export Example
```csharp
var exporter = serviceProvider.GetRequiredService<IMinimalProofExporter>();
// Lite export (manifest only)
var liteBundle = await exporter.ExportAsync(
veriKey: "sha256:abc123",
new MinimalProofExportOptions { Density = ProofDensity.Lite });
// Signed strict export
var strictBundle = await exporter.ExportAsync(
veriKey: "sha256:abc123",
new MinimalProofExportOptions
{
Density = ProofDensity.Strict,
SignBundle = true,
Signer = signerInstance
});
```
### Import and Verification
```csharp
var result = await exporter.ImportAsync(bundle);
if (result.DigestVerified && result.ChunksVerified)
{
// Bundle is authentic
await provcache.UpsertAsync(result.Entry);
}
```
---
## Lazy Evidence Fetching
For lite bundles, missing chunks can be fetched on-demand from connected or file sources.
### Fetcher Architecture
```
┌────────────────────┐
│ ILazyEvidenceFetcher│
└─────────┬──────────┘
│
┌─────┴─────┐
│ │
▼ ▼
┌─────────┐ ┌──────────┐
│ HTTP │ │ File │
│ Fetcher │ │ Fetcher │
└─────────┘ └──────────┘
```
### HTTP Fetcher (Connected Mode)
```csharp
var fetcher = new HttpChunkFetcher(
new Uri("https://api.stellaops.com"),
logger);
var orchestrator = new LazyFetchOrchestrator(repository, logger);
var result = await orchestrator.FetchAndStoreAsync(
proofRoot: "sha256:...",
fetcher,
new LazyFetchOptions
{
VerifyOnFetch = true,
BatchSize = 100
});
```
### File Fetcher (Sneakernet Mode)
For fully air-gapped environments:
1. Export full evidence to USB drive
2. Transport to isolated network
3. Import using file fetcher
```csharp
var fetcher = new FileChunkFetcher(
basePath: "/mnt/usb/evidence",
logger);
var result = await orchestrator.FetchAndStoreAsync(proofRoot, fetcher);
```
---
## Revocation Ledger
The revocation ledger provides a complete audit trail of all invalidation events.
### Schema
```sql
CREATE TABLE provcache.prov_revocations (
seq_no BIGSERIAL PRIMARY KEY,
revocation_id UUID NOT NULL,
revocation_type VARCHAR(32) NOT NULL,
revoked_key VARCHAR(512) NOT NULL,
reason VARCHAR(1024),
entries_invalidated INTEGER NOT NULL,
source VARCHAR(128) NOT NULL,
correlation_id VARCHAR(128),
revoked_at TIMESTAMPTZ NOT NULL,
metadata JSONB
);
```
### Replay for Catch-Up
After node restart or network partition, nodes can replay missed revocations:
```csharp
var replayService = serviceProvider.GetRequiredService<IRevocationReplayService>();
// Get last checkpoint
var checkpoint = await replayService.GetCheckpointAsync();
// Replay from checkpoint
var result = await replayService.ReplayFromAsync(
sinceSeqNo: checkpoint,
new RevocationReplayOptions
{
BatchSize = 1000,
SaveCheckpointPerBatch = true
});
Console.WriteLine($"Replayed {result.EntriesReplayed} revocations, {result.TotalInvalidations} entries invalidated");
```
### Statistics
```csharp
var ledger = serviceProvider.GetRequiredService<IRevocationLedger>();
var stats = await ledger.GetStatsAsync();
// stats.TotalEntries - total revocation events
// stats.EntriesByType - breakdown by type (signer, feed_epoch, etc.)
// stats.TotalEntriesInvalidated - sum of all invalidated cache entries
```
---
## API Reference
### Evidence Endpoints
| Endpoint | Method | Description |
|----------|--------|-------------|
| `/api/v1/proofs/{proofRoot}` | GET | Get paged evidence chunks |
| `/api/v1/proofs/{proofRoot}/manifest` | GET | Get chunk manifest |
| `/api/v1/proofs/{proofRoot}/chunks/{index}` | GET | Get specific chunk |
| `/api/v1/proofs/{proofRoot}/verify` | POST | Verify Merkle proof |
### Invalidation Endpoints
| Endpoint | Method | Description |
|----------|--------|-------------|
| `/api/v1/provcache/invalidate` | POST | Manual invalidation |
| `/api/v1/provcache/revocations` | GET | List revocation history |
| `/api/v1/provcache/stats` | GET | Cache statistics |
### CLI Commands
```bash
# Export commands
stella prov export --verikey <key> --density <lite|standard|strict> [--output <file>] [--sign]
# Import commands
stella prov import <file> [--lazy-fetch] [--backend <url>] [--chunks-dir <path>]
# Verify commands
stella prov verify <file> [--signer-cert <cert>]
```
---
## Configuration
Key settings in `appsettings.json`:
```json
{
"Provcache": {
"ChunkSize": 65536,
"MaxChunksPerEntry": 1000,
"DefaultTtl": "24:00:00",
"EnableWriteBehind": true,
"WriteBehindFlushInterval": "00:00:05"
}
}
```
See [README.md](README.md) for full configuration reference.

419
docs-archived/modules/prov-cache/metrics-alerting.md Normal file
View File

@@ -0,0 +1,419 @@
# Provcache Metrics and Alerting Guide
This document describes the Prometheus metrics exposed by the Provcache layer and recommended alerting configurations.
## Overview
Provcache emits metrics for monitoring cache performance, hit rates, latency, and invalidation patterns. These metrics enable operators to:
- Track cache effectiveness
- Identify performance degradation
- Detect anomalous invalidation patterns
- Capacity plan for cache infrastructure
## Prometheus Metrics
### Request Counters
#### `provcache_requests_total`
Total number of cache requests.
| Label | Values | Description |
|-------|--------|-------------|
| `source` | `valkey`, `postgres` | Cache tier that handled the request |
| `result` | `hit`, `miss`, `expired` | Request outcome |
```promql
# Total requests per minute
rate(provcache_requests_total[1m])
# Hit rate percentage
sum(rate(provcache_requests_total{result="hit"}[5m])) /
sum(rate(provcache_requests_total[5m])) * 100
```
#### `provcache_hits_total`
Total cache hits (subset of requests with `result="hit"`).
| Label | Values | Description |
|-------|--------|-------------|
| `source` | `valkey`, `postgres` | Cache tier |
```promql
# Valkey vs Postgres hit ratio
sum(rate(provcache_hits_total{source="valkey"}[5m])) /
sum(rate(provcache_hits_total[5m])) * 100
```
#### `provcache_misses_total`
Total cache misses.
| Label | Values | Description |
|-------|--------|-------------|
| `reason` | `not_found`, `expired`, `invalidated` | Miss reason |
```promql
# Miss rate by reason
sum by (reason) (rate(provcache_misses_total[5m]))
```
### Latency Histogram
#### `provcache_latency_seconds`
Latency distribution for cache operations.
| Label | Values | Description |
|-------|--------|-------------|
| `operation` | `get`, `set`, `invalidate` | Operation type |
| `source` | `valkey`, `postgres` | Cache tier |
Buckets: `0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0`
```promql
# P50 latency for cache gets
histogram_quantile(0.50, rate(provcache_latency_seconds_bucket{operation="get"}[5m]))
# P95 latency
histogram_quantile(0.95, rate(provcache_latency_seconds_bucket{operation="get"}[5m]))
# P99 latency
histogram_quantile(0.99, rate(provcache_latency_seconds_bucket{operation="get"}[5m]))
```
### Gauge Metrics
#### `provcache_items_count`
Current number of items in cache.
| Label | Values | Description |
|-------|--------|-------------|
| `source` | `valkey`, `postgres` | Cache tier |
```promql
# Total cached items
sum(provcache_items_count)
# Items by tier
sum by (source) (provcache_items_count)
```
### Invalidation Metrics
#### `provcache_invalidations_total`
Total invalidation events.
| Label | Values | Description |
|-------|--------|-------------|
| `reason` | `signer_revoked`, `epoch_advanced`, `ttl_expired`, `manual` | Invalidation trigger |
```promql
# Invalidation rate by reason
sum by (reason) (rate(provcache_invalidations_total[5m]))
# Security-related invalidations
sum(rate(provcache_invalidations_total{reason="signer_revoked"}[5m]))
```
### Trust Score Metrics
#### `provcache_trust_score_average`
Gauge showing average trust score across cached decisions.
```promql
# Current average trust score
provcache_trust_score_average
```
#### `provcache_trust_score_bucket`
Histogram of trust score distribution.
Buckets: `20, 40, 60, 80, 100`
```promql
# Percentage of decisions with trust score >= 80
sum(rate(provcache_trust_score_bucket{le="100"}[5m])) -
sum(rate(provcache_trust_score_bucket{le="80"}[5m]))
```
---
## Grafana Dashboard
A pre-built dashboard is available at `deploy/grafana/dashboards/provcache-overview.json`.
### Panels
| Panel | Type | Description |
|-------|------|-------------|
| Cache Hit Rate | Gauge | Current hit rate percentage |
| Hit Rate Over Time | Time series | Hit rate trend |
| Latency Percentiles | Time series | P50, P95, P99 latency |
| Invalidation Rate | Time series | Invalidations per minute |
| Cache Size | Time series | Item count over time |
| Hits by Source | Pie chart | Valkey vs Postgres distribution |
| Entry Size Distribution | Histogram | Size of cached entries |
| Trust Score Distribution | Histogram | Decision trust scores |
### Importing the Dashboard
```bash
# Via Grafana HTTP API
curl -X POST http://grafana:3000/api/dashboards/db \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $GRAFANA_API_KEY" \
-d @deploy/grafana/dashboards/provcache-overview.json
# Via Helm (auto-provisioned)
# Dashboard is auto-imported when using StellaOps Helm chart
helm upgrade stellaops ./devops/helm/stellaops \
--set grafana.dashboards.provcache.enabled=true
```
---
## Alerting Rules
### Recommended Alerts
#### Low Cache Hit Rate
```yaml
alert: ProvcacheLowHitRate
expr: |
sum(rate(provcache_requests_total{result="hit"}[5m])) /
sum(rate(provcache_requests_total[5m])) < 0.7
for: 10m
labels:
severity: warning
annotations:
summary: "Provcache hit rate below 70%"
description: "Cache hit rate is {{ $value | humanizePercentage }}. Check for invalidation storms or cold cache."
```
#### Critical Hit Rate Drop
```yaml
alert: ProvcacheCriticalHitRate
expr: |
sum(rate(provcache_requests_total{result="hit"}[5m])) /
sum(rate(provcache_requests_total[5m])) < 0.5
for: 5m
labels:
severity: critical
annotations:
summary: "Provcache hit rate critically low"
description: "Cache hit rate is {{ $value | humanizePercentage }}. Immediate investigation required."
```
#### High Latency
```yaml
alert: ProvcacheHighLatency
expr: |
histogram_quantile(0.95, rate(provcache_latency_seconds_bucket{operation="get"}[5m])) > 0.1
for: 5m
labels:
severity: warning
annotations:
summary: "Provcache P95 latency above 100ms"
description: "P95 get latency is {{ $value | humanizeDuration }}. Check Valkey/Postgres performance."
```
#### Excessive Invalidations
```yaml
alert: ProvcacheInvalidationStorm
expr: |
sum(rate(provcache_invalidations_total[5m])) > 100
for: 5m
labels:
severity: warning
annotations:
summary: "Provcache invalidation rate spike"
description: "Invalidations at {{ $value }} per second. Check for feed epoch changes or revocations."
```
#### Signer Revocation Spike
```yaml
alert: ProvcacheSignerRevocations
expr: |
sum(rate(provcache_invalidations_total{reason="signer_revoked"}[5m])) > 10
for: 2m
labels:
severity: critical
annotations:
summary: "Signer revocation causing mass invalidation"
description: "{{ $value }} invalidations/sec due to signer revocation. Security event investigation required."
```
#### Cache Size Approaching Limit
```yaml
alert: ProvcacheSizeHigh
expr: |
sum(provcache_items_count) > 900000
for: 15m
labels:
severity: warning
annotations:
summary: "Provcache size approaching limit"
description: "Cache has {{ $value }} items. Consider scaling or tuning TTL."
```
#### Low Trust Scores
```yaml
alert: ProvcacheLowTrustScores
expr: |
provcache_trust_score_average < 60
for: 30m
labels:
severity: info
annotations:
summary: "Average trust score below 60"
description: "Average trust score is {{ $value }}. Review SBOM completeness and VEX coverage."
```
### AlertManager Configuration
```yaml
# alertmanager.yml
route:
group_by: ['alertname', 'severity']
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
receiver: 'default-receiver'
routes:
- match:
severity: critical
receiver: 'pagerduty-critical'
- match:
alertname: ProvcacheSignerRevocations
receiver: 'security-team'
receivers:
- name: 'default-receiver'
slack_configs:
- channel: '#stellaops-alerts'
send_resolved: true
- name: 'pagerduty-critical'
pagerduty_configs:
- service_key: '<pagerduty-key>'
- name: 'security-team'
email_configs:
- to: 'security@example.com'
send_resolved: true
```
---
## Recording Rules
Pre-compute expensive queries for dashboard performance:
```yaml
# prometheus-rules.yml
groups:
- name: provcache-recording
interval: 30s
rules:
# Hit rate pre-computed
- record: provcache:hit_rate:5m
expr: |
sum(rate(provcache_requests_total{result="hit"}[5m])) /
sum(rate(provcache_requests_total[5m]))
# P95 latency pre-computed
- record: provcache:latency_p95:5m
expr: |
histogram_quantile(0.95, rate(provcache_latency_seconds_bucket{operation="get"}[5m]))
# Invalidation rate
- record: provcache:invalidation_rate:5m
expr: |
sum(rate(provcache_invalidations_total[5m]))
# Cache efficiency (hits per second vs misses)
- record: provcache:efficiency:5m
expr: |
sum(rate(provcache_hits_total[5m])) /
(sum(rate(provcache_hits_total[5m])) + sum(rate(provcache_misses_total[5m])))
```
---
## Operational Runbook
### Low Hit Rate Investigation
1. **Check invalidation metrics** — Is there an invalidation storm?
```promql
sum by (reason) (rate(provcache_invalidations_total[5m]))
```
2. **Check cache age** — Is the cache newly deployed (cold)?
```promql
sum(provcache_items_count)
```
3. **Check request patterns** — Are there many unique VeriKeys?
```promql
# High cardinality of unique requests suggests insufficient cache sharing
```
4. **Check TTL configuration** — Is TTL too aggressive?
- Review `Provcache:DefaultTtl` setting
- Consider increasing for stable workloads
### High Latency Investigation
1. **Check Valkey health**
```bash
valkey-cli -h valkey info stats
```
2. **Check Postgres connections**
```sql
SELECT count(*) FROM pg_stat_activity WHERE datname = 'stellaops';
```
3. **Check entry sizes**
```promql
histogram_quantile(0.95, rate(provcache_entry_size_bytes_bucket[5m]))
```
4. **Check network latency** between services
### Invalidation Storm Response
1. **Identify cause**
```promql
sum by (reason) (increase(provcache_invalidations_total[10m]))
```
2. **If epoch-related**: Expected during feed updates. Monitor duration.
3. **If signer-related**: Security event — escalate to security team.
4. **If manual**: Check audit logs for unauthorized invalidation.
---
## Related Documentation
- [Provcache Module README](../provcache/README.md) — Core concepts
- [Provcache Architecture](../provcache/architecture.md) — Technical details
- [Telemetry Architecture](../telemetry/architecture.md) — Observability patterns
- [Grafana Dashboard Guide](../../deploy/grafana/README.md) — Dashboard management

439
docs-archived/modules/prov-cache/oci-attestation-verification.md Normal file
View File

@@ -0,0 +1,439 @@
# Provcache OCI Attestation Verification Guide
This document describes how to verify Provcache decision attestations attached to OCI container images.
## Overview
StellaOps can attach provenance cache decisions as OCI-attached attestations to container images. These attestations enable:
- **Supply chain verification** — Verify security decisions were made by trusted evaluators
- **Audit trails** — Retrieve the exact decision state at image push time
- **Policy gates** — Admission controllers can verify attestations before deployment
- **Offline verification** — Decisions verifiable without calling StellaOps services
## Attestation Format
### Predicate Type
```
stella.ops/provcache@v1
```
### Predicate Schema
```json
{
"_type": "stella.ops/provcache@v1",
"veriKey": "sha256:abc123...",
"decision": {
"digestVersion": "v1",
"verdictHash": "sha256:def456...",
"proofRoot": "sha256:789abc...",
"trustScore": 85,
"createdAt": "2025-12-24T12:00:00Z",
"expiresAt": "2025-12-25T12:00:00Z"
},
"inputs": {
"sourceDigest": "sha256:image...",
"sbomDigest": "sha256:sbom...",
"policyDigest": "sha256:policy...",
"feedEpoch": "2024-W52"
},
"verdicts": {
"CVE-2024-1234": "mitigated",
"CVE-2024-5678": "affected"
}
}
```
### Field Descriptions
| Field | Type | Description |
|-------|------|-------------|
| `_type` | string | Predicate type URI |
| `veriKey` | string | VeriKey hash identifying this decision context |
| `decision.digestVersion` | string | Decision digest schema version |
| `decision.verdictHash` | string | Hash of all verdicts |
| `decision.proofRoot` | string | Merkle proof root hash |
| `decision.trustScore` | number | Overall trust score (0-100) |
| `decision.createdAt` | string | ISO-8601 creation timestamp |
| `decision.expiresAt` | string | ISO-8601 expiry timestamp |
| `inputs.sourceDigest` | string | Container image digest |
| `inputs.sbomDigest` | string | SBOM document digest |
| `inputs.policyDigest` | string | Policy bundle digest |
| `inputs.feedEpoch` | string | Feed epoch identifier |
| `verdicts` | object | Map of CVE IDs to verdict status |
---
## Verification with Cosign
### Prerequisites
```bash
# Install cosign
brew install cosign # macOS
# or
go install github.com/sigstore/cosign/v2/cmd/cosign@latest
```
### Basic Verification
```bash
# Verify attestation exists and is signed
cosign verify-attestation \
--type stella.ops/provcache@v1 \
registry.example.com/app:v1.2.3
```
### Verify with Identity Constraints
```bash
# Verify with signer identity (Fulcio)
cosign verify-attestation \
--type stella.ops/provcache@v1 \
--certificate-identity-regexp '.*@stellaops\.example\.com' \
--certificate-oidc-issuer https://auth.stellaops.example.com \
registry.example.com/app:v1.2.3
```
### Verify with Custom Trust Root
```bash
# Using enterprise CA
cosign verify-attestation \
--type stella.ops/provcache@v1 \
--certificate /path/to/enterprise-ca.crt \
--certificate-chain /path/to/ca-chain.crt \
registry.example.com/app:v1.2.3
```
### Extract Attestation Payload
```bash
# Get raw attestation JSON
cosign verify-attestation \
--type stella.ops/provcache@v1 \
--certificate-identity-regexp '.*@stellaops\.example\.com' \
--certificate-oidc-issuer https://auth.stellaops.example.com \
registry.example.com/app:v1.2.3 | jq '.payload' | base64 -d | jq .
```
---
## Verification with StellaOps CLI
### Verify Attestation
```bash
# Verify using StellaOps CLI
stella verify attestation \
--image registry.example.com/app:v1.2.3 \
--type provcache
# Output:
# ✓ Attestation found: stella.ops/provcache@v1
# ✓ Signature valid (Fulcio)
# ✓ Trust score: 85
# ✓ Decision created: 2025-12-24T12:00:00Z
# ✓ Decision expires: 2025-12-25T12:00:00Z
```
### Verify with Policy Requirements
```bash
# Verify with minimum trust score
stella verify attestation \
--image registry.example.com/app:v1.2.3 \
--type provcache \
--min-trust-score 80
# Verify with freshness requirement
stella verify attestation \
--image registry.example.com/app:v1.2.3 \
--type provcache \
--max-age 24h
```
### Extract Decision Details
```bash
# Get full decision details
stella verify attestation \
--image registry.example.com/app:v1.2.3 \
--type provcache \
--output json | jq .
# Get specific fields
stella verify attestation \
--image registry.example.com/app:v1.2.3 \
--type provcache \
--output json | jq '.predicate.verdicts'
```
---
## Kubernetes Admission Control
### Gatekeeper Policy
```yaml
# constraint-template.yaml
apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
name: provcacheattestation
spec:
crd:
spec:
names:
kind: ProvcacheAttestation
validation:
openAPIV3Schema:
type: object
properties:
minTrustScore:
type: integer
minimum: 0
maximum: 100
maxAgeHours:
type: integer
minimum: 1
targets:
- target: admission.k8s.gatekeeper.sh
rego: |
package provcacheattestation
violation[{"msg": msg}] {
container := input.review.object.spec.containers[_]
image := container.image
not has_valid_attestation(image)
msg := sprintf("Image %v missing valid provcache attestation", [image])
}
has_valid_attestation(image) {
attestation := get_attestation(image, "stella.ops/provcache@v1")
attestation.predicate.decision.trustScore >= input.parameters.minTrustScore
not is_expired(attestation.predicate.decision.expiresAt)
}
is_expired(expiry) {
time.parse_rfc3339_ns(expiry) < time.now_ns()
}
```
```yaml
# constraint.yaml
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: ProvcacheAttestation
metadata:
name: require-provcache-attestation
spec:
match:
kinds:
- apiGroups: [""]
kinds: ["Pod"]
namespaces:
- production
parameters:
minTrustScore: 80
maxAgeHours: 48
```
### Kyverno Policy
```yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: verify-provcache-attestation
spec:
validationFailureAction: enforce
background: true
rules:
- name: check-provcache-attestation
match:
any:
- resources:
kinds:
- Pod
verifyImages:
- imageReferences:
- "*"
attestations:
- predicateType: stella.ops/provcache@v1
conditions:
- all:
- key: "{{ decision.trustScore }}"
operator: GreaterThanOrEquals
value: 80
- key: "{{ decision.expiresAt }}"
operator: GreaterThan
value: "{{ time.Now() }}"
attestors:
- entries:
- keyless:
issuer: https://auth.stellaops.example.com
subject: ".*@stellaops\\.example\\.com"
```
---
## CI/CD Integration
### GitHub Actions
```yaml
# .github/workflows/verify-attestation.yml
name: Verify Provcache Attestation
on:
workflow_dispatch:
inputs:
image:
description: 'Image to verify'
required: true
jobs:
verify:
runs-on: ubuntu-latest
steps:
- name: Install cosign
uses: sigstore/cosign-installer@v3
- name: Verify attestation
run: |
cosign verify-attestation \
--type stella.ops/provcache@v1 \
--certificate-identity-regexp '.*@stellaops\.example\.com' \
--certificate-oidc-issuer https://auth.stellaops.example.com \
${{ inputs.image }}
- name: Check trust score
run: |
TRUST_SCORE=$(cosign verify-attestation \
--type stella.ops/provcache@v1 \
--certificate-identity-regexp '.*@stellaops\.example\.com' \
--certificate-oidc-issuer https://auth.stellaops.example.com \
${{ inputs.image }} | jq -r '.payload' | base64 -d | jq '.predicate.decision.trustScore')
if [ "$TRUST_SCORE" -lt 80 ]; then
echo "Trust score $TRUST_SCORE is below threshold (80)"
exit 1
fi
```
### GitLab CI
```yaml
# .gitlab-ci.yml
verify-attestation:
stage: verify
image: gcr.io/projectsigstore/cosign:latest
script:
- cosign verify-attestation
--type stella.ops/provcache@v1
--certificate-identity-regexp '.*@stellaops\.example\.com'
--certificate-oidc-issuer https://auth.stellaops.example.com
${CI_REGISTRY_IMAGE}:${CI_COMMIT_TAG}
rules:
- if: $CI_COMMIT_TAG
```
---
## Troubleshooting
### No Attestation Found
```bash
# List all attestations on image
cosign tree registry.example.com/app:v1.2.3
# Check if attestation was pushed
crane manifest registry.example.com/app:sha256-<digest>.att
```
### Signature Verification Failed
```bash
# Check certificate details
cosign verify-attestation \
--type stella.ops/provcache@v1 \
--output text \
registry.example.com/app:v1.2.3 2>&1 | grep -A5 "Certificate"
# Verify with verbose output
COSIGN_EXPERIMENTAL=1 cosign verify-attestation \
--type stella.ops/provcache@v1 \
registry.example.com/app:v1.2.3 -v
```
### Attestation Expired
```bash
# Check expiry timestamp
cosign verify-attestation \
--type stella.ops/provcache@v1 \
--certificate-identity-regexp '.*@stellaops\.example\.com' \
--certificate-oidc-issuer https://auth.stellaops.example.com \
registry.example.com/app:v1.2.3 | \
jq -r '.payload' | base64 -d | jq '.predicate.decision.expiresAt'
```
### Trust Score Below Threshold
```bash
# Check trust score breakdown
stella verify attestation \
--image registry.example.com/app:v1.2.3 \
--type provcache \
--output json | jq '.predicate.decision.trustScore'
# If score is low, check individual components:
# - SBOM completeness
# - VEX coverage
# - Reachability analysis
# - Policy freshness
# - Signer trust
```
---
## Security Considerations
### Key Management
- **Fulcio** — Ephemeral certificates tied to OIDC identity; recommended for public workflows
- **Enterprise CA** — Long-lived certificates for air-gapped environments
- **Self-signed** — Only for development/testing; not recommended for production
### Attestation Integrity
- Attestations are signed at push time
- Signature covers the entire predicate payload
- Modifying any field invalidates the signature
### Expiry Handling
- Attestations have `expiresAt` timestamps
- Expired attestations should be rejected by admission controllers
- Consider re-scanning images before deployment to get fresh attestations
### Verdict Reconciliation
- Verdicts in attestation reflect state at push time
- New vulnerabilities discovered after push won't appear
- Use `stella verify attestation --check-freshness` to compare against current feeds
---
## Related Documentation
- [Provcache Module README](./README.md) — Core concepts
- [Provcache Metrics and Alerting](./metrics-alerting.md) — Observability
- [Signer Module](../signer/architecture.md) — Signing infrastructure
- [Attestor Module](../attestor/architecture.md) — Attestation generation
- [OCI Artifact Spec](https://github.com/opencontainers/image-spec) — OCI standards
- [In-toto Attestation Spec](https://github.com/in-toto/attestation) — Attestation format
- [Sigstore Documentation](https://docs.sigstore.dev/) — Cosign and Fulcio

552
docs-archived/modules/reachability/architecture.md Normal file
View File

@@ -0,0 +1,552 @@
# Reachability Module Architecture
## Overview
The **Reachability** module provides a unified hybrid reachability analysis system that combines static call-graph analysis with runtime execution evidence to determine whether vulnerable code paths are actually exploitable in a given artifact. It serves as the **evidence backbone** for VEX (Vulnerability Exploitability eXchange) verdicts.
## Problem Statement
Vulnerability scanners generate excessive false positives:
- **Static analysis** over-approximates: flags code that is dead, feature-gated, or unreachable
- **Runtime analysis** under-approximates: misses rarely-executed but exploitable paths
- **No unified view** across static and runtime evidence sources
- **Symbol mismatch** between static extraction (Roslyn, ASM) and runtime observation (ETW, eBPF)
### Before Reachability Module
| Question | Answer Method | Limitation |
|----------|---------------|------------|
| Is CVE reachable statically? | Query ReachGraph | No runtime context |
| Was CVE executed at runtime? | Query Signals runtime facts | No static context |
| Should we mark CVE as NA? | Manual analysis | No evidence, no audit trail |
| What's the confidence? | Guesswork | No formal model |
### After Reachability Module
Single `IReachabilityIndex.QueryHybridAsync()` call returns:
- Lattice state (8-level certainty model)
- Confidence score (0.0-1.0)
- Evidence URIs (auditable, reproducible)
- Recommended VEX status + justification
---
## Module Location
```
src/__Libraries/StellaOps.Reachability.Core/
├── IReachabilityIndex.cs # Main facade interface
├── ReachabilityIndex.cs # Implementation
├── HybridQueryOptions.cs # Query configuration
├── SymbolRef.cs # Symbol reference
├── StaticReachabilityResult.cs # Static query result
├── RuntimeReachabilityResult.cs # Runtime query result
├── HybridReachabilityResult.cs # Combined result
├── LatticeState.cs # 8-state lattice enum
├── ReachabilityLattice.cs # Lattice state machine
├── ConfidenceCalculator.cs # Evidence-weighted confidence
├── EvidenceUriBuilder.cs # stella:// URI construction
├── IReachGraphAdapter.cs # ReachGraph integration interface
├── ISignalsAdapter.cs # Signals integration interface
├── ServiceCollectionExtensions.cs # DI registration
├── Symbols/
│ ├── ISymbolCanonicalizer.cs # Symbol normalization interface
│ ├── SymbolCanonicalizer.cs # Implementation
│ ├── ISymbolNormalizer.cs # Normalizer interface
│ ├── CanonicalSymbol.cs # Canonicalized symbol
│ ├── RawSymbol.cs # Raw input symbol
│ ├── SymbolMatchResult.cs # Match result
│ ├── SymbolMatchOptions.cs # Matching configuration
│ ├── SymbolMatcher.cs # Symbol matching logic
│ ├── SymbolSource.cs # Source enum
│ ├── ProgrammingLanguage.cs # Language enum
│ ├── DotNetSymbolNormalizer.cs # .NET symbols
│ ├── JavaSymbolNormalizer.cs # Java symbols
│ ├── NativeSymbolNormalizer.cs # C/C++/Rust
│ └── ScriptSymbolNormalizer.cs # JS/Python/PHP
└── CveMapping/
├── ICveSymbolMappingService.cs # CVE-symbol mapping interface
├── CveSymbolMappingService.cs # Implementation
├── CveSymbolMapping.cs # Mapping record
├── VulnerableSymbol.cs # Vulnerable symbol record
├── MappingSource.cs # Source enum
├── VulnerabilityType.cs # Vulnerability type enum
├── PatchAnalysisResult.cs # Patch analysis result
├── IPatchSymbolExtractor.cs # Patch analysis interface
├── IOsvEnricher.cs # OSV enricher interface
├── GitDiffExtractor.cs # Git diff parsing
├── UnifiedDiffParser.cs # Unified diff format parser
├── FunctionBoundaryDetector.cs # Function boundary detection
└── OsvEnricher.cs # OSV API enrichment
```
---
## Core Concepts
### 1. Reachability Lattice (8-State Model)
The lattice provides mathematically sound evidence aggregation:
```
X (Contested)
/ \
/ \
CR (Confirmed CU (Confirmed
Reachable) Unreachable)
| \ / |
| \ / |
RO (Runtime RU (Runtime
Observed) Unobserved)
| |
| |
SR (Static SU (Static
Reachable) Unreachable)
\ /
\ /
U (Unknown)
```
| State | Code | Description | Confidence Base |
|-------|------|-------------|-----------------|
| Unknown | U | No analysis performed | 0.00 |
| Static Reachable | SR | Call graph shows path exists | 0.30 |
| Static Unreachable | SU | Call graph proves no path | 0.40 |
| Runtime Observed | RO | Symbol executed at runtime | 0.70 |
| Runtime Unobserved | RU | Observation window passed, no execution | 0.60 |
| Confirmed Reachable | CR | Multiple sources confirm reachability | 0.90 |
| Confirmed Unreachable | CU | Multiple sources confirm no reachability | 0.95 |
| Contested | X | Evidence conflict | 0.20 (requires review) |
### 2. Symbol Canonicalization
Symbols from different sources must be normalized to enable matching:
| Source | Raw Format | Canonical Format |
|--------|-----------|------------------|
| Roslyn (.NET) | `StellaOps.Scanner.Core.SbomGenerator::GenerateAsync` | `stellaops.scanner.core/sbomgenerator/generateasync/(cancellationtoken)` |
| ASM (Java) | `org/apache/log4j/core/lookup/JndiLookup.lookup(Ljava/lang/String;)Ljava/lang/String;` | `org.apache.log4j.core.lookup/jndilookup/lookup/(string)` |
| eBPF (Native) | `_ZN4llvm12DenseMapBaseINS_...` | `llvm/densemapbase/operator[]/(keytype)` |
| ETW (.NET) | `MethodID=12345 ModuleID=67890` | (resolved via metadata) |
### 3. CVE-Symbol Mapping
Maps CVE identifiers to specific vulnerable symbols:
```json
{
"cveId": "CVE-2021-44228",
"symbols": [
{
"canonicalId": "sha256:abc123...",
"displayName": "org.apache.log4j.core.lookup/jndilookup/lookup/(string)",
"type": "Sink",
"condition": "When lookup string contains ${jndi:...}"
}
],
"source": "PatchAnalysis",
"confidence": 0.98,
"patchCommitUrl": "https://github.com/apache/logging-log4j2/commit/abc123"
}
```
### 4. Evidence URIs
Standardized `stella://` URI scheme for evidence references:
| Pattern | Example |
|---------|---------|
| `stella://reachgraph/{digest}` | `stella://reachgraph/blake3:abc123` |
| `stella://reachgraph/{digest}/slice?symbol={id}` | `stella://reachgraph/blake3:abc123/slice?symbol=sha256:def` |
| `stella://signals/runtime/{tenant}/{artifact}` | `stella://signals/runtime/acme/sha256:abc` |
| `stella://cvemap/{cveId}` | `stella://cvemap/CVE-2021-44228` |
| `stella://attestation/{digest}` | `stella://attestation/sha256:sig789` |
---
## Architecture Diagram
```
┌─────────────────────────────────────────────────────────────────────────────────┐
│ Reachability Core Library │
├─────────────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌────────────────────────────────────────────────────────────────────────────┐ │
│ │ IReachabilityIndex │ │
│ │ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────────────────┐ │ │
│ │ │ QueryStaticAsync │ │ QueryRuntimeAsync│ │ QueryHybridAsync │ │ │
│ │ └────────┬────────┘ └────────┬────────┘ └────────────┬───────────────┘ │ │
│ └───────────┼────────────────────┼─────────────────────────┼────────────────┘ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌─────────────────────────────────────────────────────────────────────────────┐│
│ │ Internal Components ││
│ │ ││
│ │ ┌────────────────┐ ┌────────────────┐ ┌────────────────────────────┐ ││
│ │ │ Symbol │ │ CVE-Symbol │ │ Reachability │ ││
│ │ │ Canonicalizer │ │ Mapping │ │ Lattice │ ││
│ │ │ │ │ │ │ │ ││
│ │ │ ┌────────────┐ │ │ ┌────────────┐ │ │ ┌───────────────────────┐ │ ││
│ │ │ │.NET Norm. │ │ │ │PatchExtract│ │ │ │ State Machine │ │ ││
│ │ │ │Java Norm. │ │ │ │OSV Enrich │ │ │ │ Confidence Calc │ │ ││
│ │ │ │Native Norm.│ │ │ │DeltaSig │ │ │ │ Transition Rules │ │ ││
│ │ │ │Script Norm.│ │ │ │Manual Input│ │ │ └───────────────────────┘ │ ││
│ │ │ └────────────┘ │ │ └────────────┘ │ │ │ ││
│ │ └────────────────┘ └────────────────┘ └────────────────────────────┘ ││
│ │ ││
│ └──────────────────────────────────────────────────────────────────────────────┘│
│ │ │
│ ▼ │
│ ┌──────────────────────────────────────────────────────────────────────────────┐│
│ │ Evidence Layer ││
│ │ ││
│ │ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────────────────┐ ││
│ │ │ Evidence URI │ │ Evidence Bundle │ │ Evidence Attestation │ ││
│ │ │ Builder │ │ (Collection) │ │ Service (DSSE) │ ││
│ │ └─────────────────┘ └─────────────────┘ └─────────────────────────────┘ ││
│ │ ││
│ └──────────────────────────────────────────────────────────────────────────────┘│
│ │
└──────────────────────────────────────────────────────────────────────────────────┘
│
┌────────────────────────┼────────────────────────┐
│ │ │
▼ ▼ ▼
┌────────────────┐ ┌────────────────┐ ┌────────────────┐
│ ReachGraph │ │ Signals │ │ Policy Engine │
│ Adapter │ │ Adapter │ │ Adapter │
└───────┬────────┘ └───────┬────────┘ └───────┬────────┘
│ │ │
▼ ▼ ▼
┌────────────────┐ ┌────────────────┐ ┌────────────────┐
│ ReachGraph │ │ Signals │ │ Policy Engine │
│ WebService │ │ WebService │ │ (VEX Emit) │
└────────────────┘ └────────────────┘ └────────────────┘
```
---
## Data Flow
### Query Flow
```
1. Consumer calls IReachabilityIndex.QueryHybridAsync(symbol, artifact, options)
│
▼
2. SymbolCanonicalizer normalizes input symbol to CanonicalSymbol
│
▼
3. Parallel queries:
├── ReachGraphAdapter.QueryAsync() → StaticReachabilityResult
└── SignalsAdapter.QueryRuntimeFactsAsync() → RuntimeReachabilityResult
│
▼
4. ReachabilityLattice computes combined state from evidence
│
▼
5. ConfidenceCalculator applies evidence weights and guardrails
│
▼
6. EvidenceBundle collects URIs for audit trail
│
▼
7. Return HybridReachabilityResult with verdict recommendation
```
### Ingestion Flow (CVE Mapping)
```
1. Patch commit detected (Concelier, Feedser, or manual)
│
▼
2. GitDiffExtractor parses diff to find changed functions
│
▼
3. SymbolCanonicalizer normalizes extracted symbols
│
▼
4. OsvEnricher adds context from OSV database
│
▼
5. CveSymbolMappingService persists mapping with provenance
│
▼
6. Mapping available for reachability queries
```
---
## API Contracts
### IReachabilityIndex
```csharp
public interface IReachabilityIndex
{
/// <summary>
/// Query static reachability from call graph.
/// </summary>
Task<StaticReachabilityResult> QueryStaticAsync(
SymbolRef symbol,
string artifactDigest,
CancellationToken ct);
/// <summary>
/// Query runtime reachability from observed facts.
/// </summary>
Task<RuntimeReachabilityResult> QueryRuntimeAsync(
SymbolRef symbol,
string artifactDigest,
TimeSpan observationWindow,
CancellationToken ct);
/// <summary>
/// Query hybrid reachability combining static + runtime.
/// </summary>
Task<HybridReachabilityResult> QueryHybridAsync(
SymbolRef symbol,
string artifactDigest,
HybridQueryOptions options,
CancellationToken ct);
/// <summary>
/// Batch query for CVE vulnerability analysis.
/// </summary>
Task<IReadOnlyList<HybridReachabilityResult>> QueryBatchAsync(
IEnumerable<SymbolRef> symbols,
string artifactDigest,
HybridQueryOptions options,
CancellationToken ct);
/// <summary>
/// Get vulnerable symbols for a CVE.
/// </summary>
Task<CveSymbolMapping?> GetCveMappingAsync(
string cveId,
CancellationToken ct);
}
```
### Result Types
```csharp
public sealed record HybridReachabilityResult
{
public required SymbolRef Symbol { get; init; }
public required string ArtifactDigest { get; init; }
public required LatticeState LatticeState { get; init; }
public required double Confidence { get; init; }
public required StaticEvidence? StaticEvidence { get; init; }
public required RuntimeEvidence? RuntimeEvidence { get; init; }
public required VerdictRecommendation Verdict { get; init; }
public required ImmutableArray<string> EvidenceUris { get; init; }
public required DateTimeOffset ComputedAt { get; init; }
public required string ComputedBy { get; init; }
}
public sealed record VerdictRecommendation
{
public required VexStatus Status { get; init; }
public VexJustification? Justification { get; init; }
public required ConfidenceBucket ConfidenceBucket { get; init; }
public string? ImpactStatement { get; init; }
public string? ActionStatement { get; init; }
}
public enum LatticeState
{
Unknown = 0,
StaticReachable = 1,
StaticUnreachable = 2,
RuntimeObserved = 3,
RuntimeUnobserved = 4,
ConfirmedReachable = 5,
ConfirmedUnreachable = 6,
Contested = 7
}
```
---
## Integration Points
### Upstream (Data Sources)
| Module | Interface | Data |
|--------|-----------|------|
| ReachGraph | `IReachGraphSliceService` | Static call-graph nodes/edges |
| Signals | `IRuntimeFactsService` | Runtime method observations |
| Scanner.CallGraph | `ICallGraphExtractor` | Per-artifact call graphs |
| Feedser | `IBackportProofService` | Patch analysis results |
### Downstream (Consumers)
| Module | Interface | Usage |
|--------|-----------|-------|
| Policy Engine | `IReachabilityAwareVexEmitter` | VEX verdict with evidence |
| VexLens | `IReachabilityIndex` | Consensus enrichment |
| Web Console | REST API | Evidence panel display |
| CLI | `stella reachability` | Command-line queries |
| ExportCenter | `IReachabilityExporter` | Offline bundles |
---
## Storage
### PostgreSQL Schema
```sql
-- CVE-Symbol Mappings
CREATE TABLE reachability.cve_symbol_mappings (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
tenant_id UUID NOT NULL,
cve_id TEXT NOT NULL,
symbol_canonical_id TEXT NOT NULL,
symbol_display_name TEXT NOT NULL,
vulnerability_type TEXT NOT NULL,
condition TEXT,
source TEXT NOT NULL,
confidence DECIMAL(3,2) NOT NULL,
patch_commit_url TEXT,
delta_sig_digest TEXT,
extracted_at TIMESTAMPTZ NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
UNIQUE (tenant_id, cve_id, symbol_canonical_id)
);
-- Query Cache
CREATE TABLE reachability.query_cache (
cache_key TEXT PRIMARY KEY,
artifact_digest TEXT NOT NULL,
symbol_canonical_id TEXT NOT NULL,
lattice_state INTEGER NOT NULL,
confidence DECIMAL(3,2) NOT NULL,
result_json JSONB NOT NULL,
computed_at TIMESTAMPTZ NOT NULL,
expires_at TIMESTAMPTZ NOT NULL
);
-- Audit Log
CREATE TABLE reachability.query_audit_log (
id BIGSERIAL PRIMARY KEY,
tenant_id UUID NOT NULL,
query_type TEXT NOT NULL,
artifact_digest TEXT NOT NULL,
symbol_count INTEGER NOT NULL,
lattice_state INTEGER NOT NULL,
confidence DECIMAL(3,2) NOT NULL,
duration_ms INTEGER NOT NULL,
queried_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
```
### Valkey (Redis) Caching
| Key Pattern | TTL | Purpose |
|-------------|-----|---------|
| `reach:static:{artifact}:{symbol}` | 1h | Static query cache |
| `reach:runtime:{artifact}:{symbol}` | 5m | Runtime query cache |
| `reach:hybrid:{artifact}:{symbol}:{options_hash}` | 15m | Hybrid query cache |
| `cvemap:{cve_id}` | 24h | CVE mapping cache |
---
## Determinism Guarantees
### Reproducibility Rules
1. **Canonical Symbol IDs:** SHA-256 of `purl|namespace|type|method|signature` (lowercase, sorted)
2. **Stable Lattice Transitions:** Deterministic state machine, no randomness
3. **Ordered Evidence:** Evidence URIs sorted lexicographically
4. **Time Injection:** All `ComputedAt` via `TimeProvider`
5. **Culture Invariance:** `InvariantCulture` for all string operations
### Replay Verification
```csharp
public interface IReachabilityReplayService
{
Task<ReplayResult> ReplayAsync(
HybridReachabilityInputs inputs,
HybridReachabilityResult expected,
CancellationToken ct);
}
```
---
## Performance Characteristics
| Operation | Target P95 | Notes |
|-----------|-----------|-------|
| Static query (cached) | <10ms | Valkey hit |
| Static query (uncached) | <100ms | ReachGraph slice |
| Runtime query (cached) | <5ms | Valkey hit |
| Runtime query (uncached) | <50ms | Signals lookup |
| Hybrid query | <50ms | Parallel static + runtime |
| Batch query (100 symbols) | <500ms | Parallelized |
| CVE mapping lookup | <10ms | Cached |
| Symbol canonicalization | <1ms | In-memory |
---
## Security Considerations
### Access Control
| Operation | Required Scope |
|-----------|---------------|
| Query reachability | `reachability:read` |
| Ingest CVE mapping | `reachability:write` |
| Admin CVE mapping | `reachability:admin` |
| Export bundles | `reachability:export` |
### Tenant Isolation
- All queries filtered by `tenant_id`
- RLS policies on all tables
- Cache keys include tenant prefix
### Data Sensitivity
- Symbol names may reveal internal architecture
- Runtime traces expose execution patterns
- CVE mappings are security-sensitive
---
## Observability
### Metrics
| Metric | Type | Labels |
|--------|------|--------|
| `reachability_query_duration_seconds` | histogram | query_type, cache_hit |
| `reachability_lattice_state_total` | counter | state |
| `reachability_cache_hit_ratio` | gauge | cache_type |
| `reachability_cvemap_count` | gauge | source |
### Traces
| Span | Description |
|------|-------------|
| `reachability.query.static` | Static graph query |
| `reachability.query.runtime` | Runtime facts query |
| `reachability.query.hybrid` | Combined computation |
| `reachability.canonicalize` | Symbol normalization |
| `reachability.lattice.compute` | State calculation |
---
## Related Documentation
- [Product Advisory: Hybrid Reachability](../../product/advisories/09-Jan-2026%20-%20Hybrid%20Reachability%20and%20VEX%20Integration%20(Revised).md)
- [ReachGraph Architecture](../reach-graph/architecture.md)
- [Signals Architecture](../signals/architecture.md)
- [VexLens Architecture](../vex-lens/architecture.md)
- [Sprint Index](../../implplan/SPRINT_20260109_009_000_INDEX_hybrid_reachability.md)
---
_Last updated: 10-Jan-2026_

588
docs-archived/modules/sarif-export/architecture.md Normal file
View File

@@ -0,0 +1,588 @@
# SARIF Export Module Architecture
> **Implementation Status:**
> - SARIF 2.1.0 Models: **Implemented** (`src/Scanner/__Libraries/StellaOps.Scanner.Sarif/`)
> - Export Service: **Implemented**
> - SmartDiff Integration: **Implemented**
> - Fingerprint Generator: **Implemented**
> - GitHub Upload Client: **Planned**
>
> There is no standalone `src/SarifExport/` module; SARIF export is a capability within the Scanner module.
## Overview
The **SARIF Export** module provides SARIF 2.1.0 compliant output for StellaOps Scanner findings, enabling integration with GitHub Code Scanning, GitLab SAST, Azure DevOps, and other platforms that consume SARIF.
## Current State
| Component | Status | Location |
|-----------|--------|----------|
| SARIF 2.1.0 Models | **Implemented** | `Scanner.Sarif/Models/SarifModels.cs` |
| SmartDiff SARIF Generator | **Implemented** | `Scanner.SmartDiff/Output/SarifOutputGenerator.cs` |
| SmartDiff SARIF Endpoint | **Implemented** | `GET /smart-diff/scans/{scanId}/sarif` |
| Findings SARIF Mapper | **Implemented** | `Scanner.Sarif/SarifExportService.cs` |
| SARIF Rule Registry | **Implemented** | `Scanner.Sarif/Rules/SarifRuleRegistry.cs` |
| Fingerprint Generator | **Implemented** | `Scanner.Sarif/Fingerprints/FingerprintGenerator.cs` |
| GitHub Upload Client | **Not Implemented** | Proposed |
---
## Module Location
```
src/Scanner/__Libraries/StellaOps.Scanner.Sarif/
├── ISarifExportService.cs # Main export interface
├── SarifExportService.cs # Implementation (DONE)
├── SarifExportOptions.cs # Configuration (DONE)
├── FindingInput.cs # Input model (DONE)
├── Models/
│ └── SarifModels.cs # Complete SARIF 2.1.0 types (DONE)
├── Rules/
│ ├── ISarifRuleRegistry.cs # Rule registry interface (DONE)
│ └── SarifRuleRegistry.cs # 21 rules implemented (DONE)
└── Fingerprints/
├── IFingerprintGenerator.cs # Fingerprint interface (DONE)
└── FingerprintGenerator.cs # SHA-256 fingerprints (DONE)
```
---
## Existing SmartDiff SARIF Implementation
The SmartDiff module provides a reference implementation:
### SarifModels.cs (Existing)
```csharp
// Already implemented record types
public sealed record SarifLog(
string Version,
string Schema,
ImmutableArray<SarifRun> Runs);
public sealed record SarifRun(
SarifTool Tool,
ImmutableArray<SarifResult> Results,
ImmutableArray<SarifArtifact> Artifacts,
ImmutableArray<SarifVersionControlDetails> VersionControlProvenance,
ImmutableDictionary<string, object> Properties);
public sealed record SarifResult(
string RuleId,
int? RuleIndex,
SarifLevel Level,
SarifMessage Message,
ImmutableArray<SarifLocation> Locations,
ImmutableDictionary<string, string> Fingerprints,
ImmutableDictionary<string, string> PartialFingerprints,
ImmutableDictionary<string, object> Properties);
```
### SarifOutputGenerator.cs (Existing)
```csharp
// Existing generator for SmartDiff findings
public class SarifOutputGenerator
{
public SarifLog Generate(
IEnumerable<MaterialRiskChangeResult> changes,
SarifOutputOptions options);
}
```
---
## New Findings SARIF Architecture
### ISarifExportService
```csharp
namespace StellaOps.Scanner.Sarif;
/// <summary>
/// Service for exporting scanner findings to SARIF format.
/// </summary>
public interface ISarifExportService
{
/// <summary>
/// Export findings to SARIF 2.1.0 format.
/// </summary>
/// <param name="findings">Scanner findings to export.</param>
/// <param name="options">Export options.</param>
/// <param name="ct">Cancellation token.</param>
/// <returns>SARIF log document.</returns>
Task<SarifLog> ExportAsync(
IEnumerable<Finding> findings,
SarifExportOptions options,
CancellationToken ct);
/// <summary>
/// Export findings to SARIF JSON string.
/// </summary>
Task<string> ExportToJsonAsync(
IEnumerable<Finding> findings,
SarifExportOptions options,
CancellationToken ct);
/// <summary>
/// Export findings to SARIF JSON stream.
/// </summary>
Task ExportToStreamAsync(
IEnumerable<Finding> findings,
SarifExportOptions options,
Stream outputStream,
CancellationToken ct);
/// <summary>
/// Validate SARIF output against schema.
/// </summary>
Task<SarifValidationResult> ValidateAsync(
SarifLog log,
CancellationToken ct);
}
```
### SarifExportOptions
```csharp
namespace StellaOps.Scanner.Sarif;
/// <summary>
/// Options for SARIF export.
/// </summary>
public sealed record SarifExportOptions
{
/// <summary>Tool name in SARIF output.</summary>
public string ToolName { get; init; } = "StellaOps Scanner";
/// <summary>Tool version.</summary>
public required string ToolVersion { get; init; }
/// <summary>Tool information URI.</summary>
public string ToolUri { get; init; } = "https://stellaops.io/scanner";
/// <summary>Minimum severity to include.</summary>
public Severity? MinimumSeverity { get; init; }
/// <summary>Include reachability evidence in properties.</summary>
public bool IncludeReachability { get; init; } = true;
/// <summary>Include VEX status in properties.</summary>
public bool IncludeVexStatus { get; init; } = true;
/// <summary>Include EPSS scores in properties.</summary>
public bool IncludeEpss { get; init; } = true;
/// <summary>Include KEV status in properties.</summary>
public bool IncludeKev { get; init; } = true;
/// <summary>Include evidence URIs in properties.</summary>
public bool IncludeEvidenceUris { get; init; } = false;
/// <summary>Include attestation reference in run properties.</summary>
public bool IncludeAttestation { get; init; } = true;
/// <summary>Version control provenance.</summary>
public VersionControlInfo? VersionControl { get; init; }
/// <summary>Pretty-print JSON output.</summary>
public bool IndentedJson { get; init; } = false;
/// <summary>Category for GitHub upload (distinguishes multiple tools).</summary>
public string? Category { get; init; }
/// <summary>Base URI for source files.</summary>
public string? SourceRoot { get; init; }
}
public sealed record VersionControlInfo
{
public required string RepositoryUri { get; init; }
public required string RevisionId { get; init; }
public string? Branch { get; init; }
}
```
---
## Rule Registry
### ISarifRuleRegistry
```csharp
namespace StellaOps.Scanner.Sarif.Rules;
/// <summary>
/// Registry of SARIF rules for StellaOps findings.
/// </summary>
public interface ISarifRuleRegistry
{
/// <summary>Get rule by ID.</summary>
SarifRule? GetRule(string ruleId);
/// <summary>Get rule for finding type and severity.</summary>
SarifRule GetRuleForFinding(FindingType type, Severity severity);
/// <summary>Get all registered rules.</summary>
IReadOnlyList<SarifRule> GetAllRules();
/// <summary>Get rules by category.</summary>
IReadOnlyList<SarifRule> GetRulesByCategory(string category);
}
```
### Rule Definitions
```csharp
namespace StellaOps.Scanner.Sarif.Rules;
public static class VulnerabilityRules
{
public static readonly SarifRule Critical = new()
{
Id = "STELLA-VULN-001",
Name = "CriticalVulnerability",
ShortDescription = "Critical vulnerability detected (CVSS >= 9.0)",
FullDescription = "A critical severity vulnerability was detected. " +
"This may be a known exploited vulnerability (KEV) or " +
"have a CVSS score of 9.0 or higher.",
HelpUri = "https://stellaops.io/rules/STELLA-VULN-001",
DefaultLevel = SarifLevel.Error,
Properties = new Dictionary<string, object>
{
["precision"] = "high",
["problem.severity"] = "error",
["security-severity"] = "10.0",
["tags"] = new[] { "security", "vulnerability", "critical" }
}.ToImmutableDictionary()
};
public static readonly SarifRule High = new()
{
Id = "STELLA-VULN-002",
Name = "HighVulnerability",
ShortDescription = "High severity vulnerability detected (CVSS 7.0-8.9)",
FullDescription = "A high severity vulnerability was detected with " +
"CVSS score between 7.0 and 8.9.",
HelpUri = "https://stellaops.io/rules/STELLA-VULN-002",
DefaultLevel = SarifLevel.Error,
Properties = new Dictionary<string, object>
{
["precision"] = "high",
["problem.severity"] = "error",
["security-severity"] = "8.0",
["tags"] = new[] { "security", "vulnerability", "high" }
}.ToImmutableDictionary()
};
public static readonly SarifRule Medium = new()
{
Id = "STELLA-VULN-003",
Name = "MediumVulnerability",
ShortDescription = "Medium severity vulnerability detected (CVSS 4.0-6.9)",
HelpUri = "https://stellaops.io/rules/STELLA-VULN-003",
DefaultLevel = SarifLevel.Warning,
Properties = new Dictionary<string, object>
{
["precision"] = "high",
["problem.severity"] = "warning",
["security-severity"] = "5.5",
["tags"] = new[] { "security", "vulnerability", "medium" }
}.ToImmutableDictionary()
};
public static readonly SarifRule Low = new()
{
Id = "STELLA-VULN-004",
Name = "LowVulnerability",
ShortDescription = "Low severity vulnerability detected (CVSS < 4.0)",
HelpUri = "https://stellaops.io/rules/STELLA-VULN-004",
DefaultLevel = SarifLevel.Note,
Properties = new Dictionary<string, object>
{
["precision"] = "high",
["problem.severity"] = "note",
["security-severity"] = "2.0",
["tags"] = new[] { "security", "vulnerability", "low" }
}.ToImmutableDictionary()
};
// Reachability-enhanced rules
public static readonly SarifRule RuntimeReachable = new()
{
Id = "STELLA-VULN-005",
Name = "ReachableVulnerability",
ShortDescription = "Runtime-confirmed reachable vulnerability",
FullDescription = "A vulnerability with runtime-confirmed reachability. " +
"The vulnerable code path was observed during execution.",
HelpUri = "https://stellaops.io/rules/STELLA-VULN-005",
DefaultLevel = SarifLevel.Error,
Properties = new Dictionary<string, object>
{
["precision"] = "very-high",
["problem.severity"] = "error",
["security-severity"] = "9.5",
["tags"] = new[] { "security", "vulnerability", "reachable", "runtime" }
}.ToImmutableDictionary()
};
}
```
---
## Fingerprint Generation
### IFingerprintGenerator
```csharp
namespace StellaOps.Scanner.Sarif.Fingerprints;
/// <summary>
/// Generates deterministic fingerprints for SARIF deduplication.
/// </summary>
public interface IFingerprintGenerator
{
/// <summary>
/// Generate primary fingerprint for a finding.
/// </summary>
string GeneratePrimary(Finding finding, FingerprintStrategy strategy);
/// <summary>
/// Generate partial fingerprints for GitHub fallback.
/// </summary>
ImmutableDictionary<string, string> GeneratePartial(
Finding finding,
string? sourceContent);
}
public enum FingerprintStrategy
{
/// <summary>Hash of ruleId + purl + vulnId + artifactDigest.</summary>
Standard,
/// <summary>Hash including file location for source-level findings.</summary>
WithLocation,
/// <summary>Hash including content hash for maximum stability.</summary>
ContentBased
}
```
### Implementation
```csharp
public class FingerprintGenerator : IFingerprintGenerator
{
public string GeneratePrimary(Finding finding, FingerprintStrategy strategy)
{
var input = strategy switch
{
FingerprintStrategy.Standard => string.Join("|",
finding.RuleId,
finding.ComponentPurl,
finding.VulnerabilityId ?? "",
finding.ArtifactDigest),
FingerprintStrategy.WithLocation => string.Join("|",
finding.RuleId,
finding.ComponentPurl,
finding.VulnerabilityId ?? "",
finding.ArtifactDigest,
finding.FilePath ?? "",
finding.LineNumber?.ToString(CultureInfo.InvariantCulture) ?? ""),
FingerprintStrategy.ContentBased => string.Join("|",
finding.RuleId,
finding.ComponentPurl,
finding.VulnerabilityId ?? "",
finding.ContentHash ?? finding.ArtifactDigest),
_ => throw new ArgumentOutOfRangeException(nameof(strategy))
};
return ComputeSha256(input);
}
public ImmutableDictionary<string, string> GeneratePartial(
Finding finding,
string? sourceContent)
{
var partial = new Dictionary<string, string>();
// Line hash for GitHub deduplication
if (!string.IsNullOrEmpty(sourceContent) && finding.LineNumber.HasValue)
{
var lines = sourceContent.Split('\n');
if (finding.LineNumber.Value <= lines.Length)
{
var line = lines[finding.LineNumber.Value - 1];
partial["primaryLocationLineHash"] = ComputeSha256(line.Trim());
}
}
return partial.ToImmutableDictionary();
}
private static string ComputeSha256(string input)
{
var bytes = SHA256.HashData(Encoding.UTF8.GetBytes(input));
return Convert.ToHexString(bytes).ToLowerInvariant();
}
}
```
---
## Severity Mapping
```csharp
public static class SeverityMapper
{
public static SarifLevel MapToSarifLevel(Severity severity, bool isReachable = false)
{
// Reachable vulnerabilities are always error level
if (isReachable && severity >= Severity.Medium)
return SarifLevel.Error;
return severity switch
{
Severity.Critical => SarifLevel.Error,
Severity.High => SarifLevel.Error,
Severity.Medium => SarifLevel.Warning,
Severity.Low => SarifLevel.Note,
Severity.Info => SarifLevel.Note,
_ => SarifLevel.None
};
}
public static double MapToSecuritySeverity(double cvssScore)
{
// GitHub uses security-severity for ordering
// Map CVSS 0-10 scale directly
return Math.Clamp(cvssScore, 0.0, 10.0);
}
}
```
---
## Determinism Requirements
Following CLAUDE.md rules:
1. **Canonical JSON:** RFC 8785 sorted keys, no nulls
2. **Stable Rule Ordering:** Rules sorted by ID
3. **Stable Result Ordering:** Results sorted by (ruleId, location, fingerprint)
4. **Time Injection:** Use `TimeProvider` for timestamps
5. **Culture Invariance:** `InvariantCulture` for all string operations
6. **Immutable Collections:** All outputs use `ImmutableArray`, `ImmutableDictionary`
---
## API Endpoints
### Scanner Export Endpoints
```csharp
public static class SarifExportEndpoints
{
public static void MapSarifEndpoints(this IEndpointRouteBuilder app)
{
var group = app.MapGroup("/v1/scans/{scanId}/exports")
.RequireAuthorization("scanner:read");
// SARIF export
group.MapGet("/sarif", ExportSarif)
.WithName("ExportScanSarif")
.Produces<string>(StatusCodes.Status200OK, "application/sarif+json");
// SARIF with options
group.MapPost("/sarif", ExportSarifWithOptions)
.WithName("ExportScanSarifWithOptions")
.Produces<string>(StatusCodes.Status200OK, "application/sarif+json");
}
private static async Task<IResult> ExportSarif(
Guid scanId,
[FromQuery] string? minSeverity,
[FromQuery] bool pretty = false,
[FromQuery] bool includeReachability = true,
ISarifExportService sarifService,
IFindingsService findingsService,
CancellationToken ct)
{
var findings = await findingsService.GetByScanIdAsync(scanId, ct);
var options = new SarifExportOptions
{
ToolVersion = GetToolVersion(),
MinimumSeverity = ParseSeverity(minSeverity),
IncludeReachability = includeReachability,
IndentedJson = pretty
};
var json = await sarifService.ExportToJsonAsync(findings, options, ct);
return Results.Content(json, "application/sarif+json");
}
}
```
---
## Integration with GitHub
See `src/Integrations/__Plugins/StellaOps.Integrations.Plugin.GitHubApp/` for GitHub connector.
New GitHub Code Scanning client extends existing infrastructure:
```csharp
public interface IGitHubCodeScanningClient
{
/// <summary>Upload SARIF to GitHub Code Scanning.</summary>
Task<SarifUploadResult> UploadSarifAsync(
string owner,
string repo,
SarifUploadRequest request,
CancellationToken ct);
/// <summary>Get upload status.</summary>
Task<SarifUploadStatus> GetUploadStatusAsync(
string owner,
string repo,
string sarifId,
CancellationToken ct);
/// <summary>List code scanning alerts.</summary>
Task<IReadOnlyList<CodeScanningAlert>> ListAlertsAsync(
string owner,
string repo,
AlertFilter? filter,
CancellationToken ct);
}
```
---
## Performance Targets
| Operation | Target P95 | Notes |
|-----------|-----------|-------|
| Export 100 findings | < 100ms | In-memory |
| Export 10,000 findings | < 5s | Streaming |
| SARIF serialization | < 50ms/MB | RFC 8785 |
| Schema validation | < 200ms | JSON Schema |
| Fingerprint generation | < 1ms/finding | SHA-256 |
---
## Related Documentation
- [Product Advisory](../../product/advisories/09-Jan-2026%20-%20GitHub%20Code%20Scanning%20Integration%20(Revised).md)
- [SARIF 2.1.0 Specification](https://docs.oasis-open.org/sarif/sarif/v2.1.0/sarif-v2.1.0.html)
- [GitHub SARIF Support](https://docs.github.com/en/code-security/code-scanning/integrating-with-code-scanning/sarif-support-for-code-scanning)
- [Existing SmartDiff SARIF](../../../src/Scanner/__Libraries/StellaOps.Scanner.SmartDiff/Output/)
---
_Last updated: 09-Jan-2026_

287
docs-archived/modules/triage/evidence-panel.md Normal file
View File

@@ -0,0 +1,287 @@
# Evidence Panel Component
> **Sprint:** SPRINT_20260107_006_001_FE
> **Module:** Triage UI
> **Version:** 1.0.0
## Overview
The Evidence Panel provides a unified tabbed interface for viewing all evidence related to a security finding. It consolidates five categories of evidence:
1. **Provenance** - DSSE attestation chain, signer identity, Rekor transparency
2. **Reachability** - Code path analysis showing if vulnerability is reachable
3. **Diff** - Source code changes introducing the vulnerability
4. **Runtime** - Runtime telemetry and execution evidence
5. **Policy** - OPA/Rego policy decisions and lattice trace
## Architecture
```
┌──────────────────────────────────────────────────────────────────────┐
│ TabbedEvidencePanelComponent │
├──────────────────────────────────────────────────────────────────────┤
│ [Provenance] [Reachability] [Diff] [Runtime] [Policy] │
├──────────────────────────────────────────────────────────────────────┤
│ ┌──────────────────────────────────────────────────────────────────┐ │
│ │ Tab Content (lazy-loaded) │ │
│ │ │ │
│ │ ProvenanceTabComponent / ReachabilityTabComponent / etc. │ │
│ └──────────────────────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────────────────┘
```
## Components
### TabbedEvidencePanelComponent
**Selector:** `app-tabbed-evidence-panel`
**Inputs:**
- `findingId: string` - The finding ID to load evidence for
**Outputs:**
- `tabChange: EventEmitter<EvidenceTabType>` - Emitted when tab changes
**Usage:**
```html
<app-tabbed-evidence-panel
[findingId]="selectedFindingId"
(tabChange)="onTabChange($event)"
/>
```
### ProvenanceTabComponent
Displays DSSE attestation information including:
- DSSE verification badge (verified/partial/missing)
- Attestation chain visualization (build → scan → triage → policy)
- Signer identity and key information
- Rekor log index with verification link
- Collapsible in-toto statement JSON
### DsseBadgeComponent
**Selector:** `app-dsse-badge`
Displays the DSSE verification status as a badge.
**Inputs:**
- `status: DsseBadgeStatus` - 'verified' | 'partial' | 'missing'
- `details?: DsseVerificationDetails` - Additional verification details
- `showTooltip?: boolean` - Show tooltip on hover (default: true)
- `animate?: boolean` - Enable hover animations (default: true)
**States:**
| State | Color | Icon | Meaning |
|-------|-------|------|---------|
| verified | Green | ✓ | Full DSSE chain verified |
| partial | Amber | ⚠ | Some attestations missing |
| missing | Red | ✗ | No valid attestation |
### AttestationChainComponent
**Selector:** `app-attestation-chain`
Visualizes the attestation chain as connected nodes.
**Inputs:**
- `nodes: AttestationChainNode[]` - Chain nodes to display
**Outputs:**
- `nodeClick: EventEmitter<AttestationChainNode>` - Emitted on node click
### PolicyTabComponent
Displays policy evaluation details including:
- Verdict badge (ALLOW/DENY/QUARANTINE/REVIEW)
- OPA/Rego rule path that matched
- K4 lattice merge trace visualization
- Counterfactual analysis ("What would change verdict?")
- Policy version and editor link
### ReachabilityTabComponent
Integrates the existing `ReachabilityContextComponent` with:
- Summary header with status badge
- Confidence percentage display
- Entry points list
- Link to full graph view
## Services
### EvidenceTabService
**Path:** `services/evidence-tab.service.ts`
Fetches evidence data for each tab with caching.
```typescript
interface EvidenceTabService {
getProvenanceEvidence(findingId: string, forceRefresh?: boolean): Observable<LoadState<ProvenanceEvidence>>;
getReachabilityEvidence(findingId: string, forceRefresh?: boolean): Observable<LoadState<ReachabilityData>>;
getDiffEvidence(findingId: string, forceRefresh?: boolean): Observable<LoadState<DiffEvidence>>;
getRuntimeEvidence(findingId: string, forceRefresh?: boolean): Observable<LoadState<RuntimeEvidence>>;
getPolicyEvidence(findingId: string, forceRefresh?: boolean): Observable<LoadState<PolicyEvidence>>;
clearCache(findingId?: string): void;
}
```
### TabUrlPersistenceService
**Path:** `services/tab-url-persistence.service.ts`
Manages URL query param persistence for selected tab.
```typescript
interface TabUrlPersistenceService {
readonly selectedTab$: Observable<EvidenceTabType>;
getCurrentTab(): EvidenceTabType;
setTab(tab: EvidenceTabType): void;
navigateToTab(tab: EvidenceTabType): void;
}
```
## Keyboard Shortcuts
| Key | Action |
|-----|--------|
| `1` | Go to Provenance tab |
| `2` | Go to Reachability tab |
| `3` | Go to Diff tab |
| `4` | Go to Runtime tab |
| `5` | Go to Policy tab |
| `→` | Next tab |
| `←` | Previous tab |
| `Home` | First tab |
| `End` | Last tab |
## URL Persistence
The selected tab is persisted in the URL query string:
```
/triage/findings/CVE-2024-1234?tab=provenance
/triage/findings/CVE-2024-1234?tab=reachability
/triage/findings/CVE-2024-1234?tab=diff
/triage/findings/CVE-2024-1234?tab=runtime
/triage/findings/CVE-2024-1234?tab=policy
```
This enables:
- Deep linking to specific evidence
- Browser history navigation
- Sharing links with colleagues
## Data Models
### ProvenanceEvidence
```typescript
interface ProvenanceEvidence {
dsseStatus: DsseBadgeStatus;
dsseDetails?: DsseVerificationDetails;
attestationChain: AttestationChainNode[];
signer?: SignerInfo;
rekorLogIndex?: number;
rekorVerifyUrl?: string;
inTotoStatement?: object;
}
```
### AttestationChainNode
```typescript
interface AttestationChainNode {
id: string;
type: 'build' | 'scan' | 'triage' | 'policy' | 'custom';
label: string;
status: 'verified' | 'pending' | 'missing' | 'failed';
predicateType?: string;
digest?: string;
timestamp?: string;
signer?: string;
details?: AttestationDetails;
}
```
### PolicyEvidence
```typescript
interface PolicyEvidence {
verdict: PolicyVerdict;
rulePath?: string;
latticeTrace?: LatticeTraceStep[];
counterfactuals?: PolicyCounterfactual[];
policyVersion?: string;
policyDigest?: string;
policyEditorUrl?: string;
evaluatedAt?: string;
}
```
## Accessibility
The Evidence Panel follows WAI-ARIA tabs pattern:
- `role="tablist"` on tab navigation
- `role="tab"` on each tab button
- `role="tabpanel"` on each panel
- `aria-selected` indicates active tab
- `aria-controls` links tabs to panels
- `aria-labelledby` links panels to tabs
- `tabindex` management for keyboard navigation
- Screen reader announcements on tab change
## Testing
### Unit Tests
Located in `evidence-panel/*.spec.ts`:
- Tab navigation behavior
- DSSE badge states and styling
- Attestation chain rendering
- Keyboard navigation
- URL persistence
- Loading/error states
### E2E Tests
Located in `e2e/evidence-panel.e2e.spec.ts`:
- Full tab switching workflow
- Evidence loading and display
- Copy JSON functionality
- URL persistence across reloads
- Accessibility compliance
## API Dependencies
The Evidence Panel depends on these API endpoints:
| Endpoint | Method | Description |
|----------|--------|-------------|
| `/api/evidence/provenance/{findingId}` | GET | Fetch provenance data |
| `/api/evidence/reachability/{findingId}` | GET | Fetch reachability data |
| `/api/evidence/diff/{findingId}` | GET | Fetch diff data |
| `/api/evidence/runtime/{findingId}` | GET | Fetch runtime data |
| `/api/evidence/policy/{findingId}` | GET | Fetch policy data |
See [Evidence API Reference](../../../api/evidence-api.md) for details.
## Screenshots
### Provenance Tab
![Provenance Tab](../../assets/screenshots/evidence-provenance.png)
### Policy Tab with Lattice Trace
![Policy Tab](../../assets/screenshots/evidence-policy.png)
### Attestation Chain Expanded
![Attestation Chain](../../assets/screenshots/attestation-chain.png)
## Changelog
| Version | Date | Changes |
|---------|------|---------|
| 1.0.0 | 2026-01-09 | Initial implementation |