Implement VEX document verification system with issuer management and signature verification
- Added IIssuerDirectory interface for managing VEX document issuers, including methods for registration, revocation, and trust validation. - Created InMemoryIssuerDirectory class as an in-memory implementation of IIssuerDirectory for testing and single-instance deployments. - Introduced ISignatureVerifier interface for verifying signatures on VEX documents, with support for multiple signature formats. - Developed SignatureVerifier class as the default implementation of ISignatureVerifier, allowing extensibility for different signature formats. - Implemented handlers for DSSE and JWS signature formats, including methods for verification and signature extraction. - Defined various records and enums for issuer and signature metadata, enhancing the structure and clarity of the verification process.
This commit is contained in:
StellaOps Bot
156
docs/modules/policy/design/policy-aoc-linting-rules.md
Normal file
156
docs/modules/policy/design/policy-aoc-linting-rules.md
Normal file
@@ -0,0 +1,156 @@
|
||||
# Policy AOC Linting Rules
|
||||
|
||||
**Document ID:** `DESIGN-POLICY-AOC-LINTING-001`
|
||||
**Version:** 1.0
|
||||
**Status:** Published
|
||||
**Last Updated:** 2025-12-06
|
||||
|
||||
## Overview
|
||||
|
||||
This document defines the linting and static analysis rules for Policy Engine and related library projects. These rules enforce determinism, nullability, async consistency, and JSON property ordering to ensure reproducible policy evaluation.
|
||||
|
||||
## Target Projects
|
||||
|
||||
| Project | Path | Notes |
|
||||
|---------|------|-------|
|
||||
| StellaOps.Policy.Engine | `src/Policy/StellaOps.Policy.Engine/` | Primary target |
|
||||
| StellaOps.Policy | `src/Policy/__Libraries/StellaOps.Policy/` | Core library |
|
||||
| StellaOps.PolicyDsl | `src/Policy/StellaOps.PolicyDsl/` | DSL compiler |
|
||||
| StellaOps.Policy.RiskProfile | `src/Policy/StellaOps.Policy.RiskProfile/` | Risk scoring |
|
||||
| StellaOps.Policy.Storage.Postgres | `src/Policy/__Libraries/StellaOps.Policy.Storage.Postgres/` | Storage layer |
|
||||
|
||||
### Excluded
|
||||
|
||||
- `**/obj/**` - Build artifacts
|
||||
- `**/bin/**` - Build outputs
|
||||
- `**/*.Tests/**` - Test projects (separate rules)
|
||||
- `**/Migrations/**` - Generated EF migrations
|
||||
|
||||
## Rule Categories
|
||||
|
||||
### 1. Determinism Rules (Error Severity)
|
||||
|
||||
Enforced by `ProhibitedPatternAnalyzer` at `src/Policy/StellaOps.Policy.Engine/DeterminismGuard/`.
|
||||
|
||||
| Rule ID | Pattern | Severity | Remediation |
|
||||
|---------|---------|----------|-------------|
|
||||
| DET-001 | `DateTime.Now` | Error | Use `TimeProvider.GetUtcNow()` |
|
||||
| DET-002 | `DateTime.UtcNow` | Error | Use `TimeProvider.GetUtcNow()` |
|
||||
| DET-003 | `DateTimeOffset.Now` | Error | Use `TimeProvider.GetUtcNow()` |
|
||||
| DET-004 | `DateTimeOffset.UtcNow` | Error | Use `TimeProvider.GetUtcNow()` |
|
||||
| DET-005 | `Guid.NewGuid()` | Error | Use `StableIdGenerator` or content hash |
|
||||
| DET-006 | `new Random()` | Error | Use seeded random or remove |
|
||||
| DET-007 | `RandomNumberGenerator` | Error | Remove from evaluation path |
|
||||
| DET-008 | `HttpClient` in eval | Critical | Remove network from eval path |
|
||||
| DET-009 | `File.Read*` in eval | Critical | Remove filesystem from eval path |
|
||||
| DET-010 | Dictionary iteration | Warning | Use `OrderBy` or `SortedDictionary` |
|
||||
| DET-011 | HashSet iteration | Warning | Use `OrderBy` or `SortedSet` |
|
||||
|
||||
### 2. Nullability Rules (Error Severity)
|
||||
|
||||
| Rule ID | Description | EditorConfig |
|
||||
|---------|-------------|--------------|
|
||||
| NUL-001 | Enable nullable reference types | `nullable = enable` |
|
||||
| NUL-002 | Nullable warnings as errors | `dotnet_diagnostic.CS8600-CS8609.severity = error` |
|
||||
| NUL-003 | Null parameter checks | `ArgumentNullException.ThrowIfNull()` |
|
||||
|
||||
### 3. Async/Sync Consistency Rules (Warning Severity)
|
||||
|
||||
| Rule ID | Description | EditorConfig |
|
||||
|---------|-------------|--------------|
|
||||
| ASY-001 | Async void methods | `dotnet_diagnostic.CA2012.severity = error` |
|
||||
| ASY-002 | Missing ConfigureAwait | `dotnet_diagnostic.CA2007.severity = warning` |
|
||||
| ASY-003 | Sync over async | `dotnet_diagnostic.MA0045.severity = warning` |
|
||||
| ASY-004 | Task.Result in async | `dotnet_diagnostic.MA0042.severity = error` |
|
||||
|
||||
### 4. JSON Property Ordering Rules
|
||||
|
||||
For deterministic JSON output, all DTOs must use explicit `[JsonPropertyOrder]` attributes.
|
||||
|
||||
| Rule ID | Description | Enforcement |
|
||||
|---------|-------------|-------------|
|
||||
| JSN-001 | Explicit property order | Code review + analyzer |
|
||||
| JSN-002 | Stable serialization | `JsonSerializerOptions.WriteIndented = false` |
|
||||
| JSN-003 | Key ordering | `JsonSerializerOptions.PropertyNamingPolicy` with stable order |
|
||||
|
||||
### 5. Code Style Rules
|
||||
|
||||
| Rule ID | Description | EditorConfig |
|
||||
|---------|-------------|--------------|
|
||||
| STY-001 | File-scoped namespaces | `csharp_style_namespace_declarations = file_scoped` |
|
||||
| STY-002 | Primary constructors | `csharp_style_prefer_primary_constructors = true` |
|
||||
| STY-003 | Collection expressions | `csharp_style_prefer_collection_expression = true` |
|
||||
| STY-004 | Implicit usings | `ImplicitUsings = enable` |
|
||||
|
||||
## Severity Levels
|
||||
|
||||
| Level | Behavior | CI Impact |
|
||||
|-------|----------|-----------|
|
||||
| Error | Build fails | Blocks merge |
|
||||
| Warning | Build succeeds, logged | Review required |
|
||||
| Info | Logged only | No action required |
|
||||
|
||||
## CI Integration
|
||||
|
||||
### Build-time Enforcement
|
||||
|
||||
Policy projects use `TreatWarningsAsErrors=true` in `.csproj`:
|
||||
|
||||
```xml
|
||||
<PropertyGroup>
|
||||
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
|
||||
<Nullable>enable</Nullable>
|
||||
</PropertyGroup>
|
||||
```
|
||||
|
||||
### Static Analysis Pipeline
|
||||
|
||||
The `.gitea/workflows/policy-lint.yml` workflow runs:
|
||||
|
||||
1. **dotnet build** with analyzer packages
|
||||
2. **DeterminismGuard analysis** via CLI
|
||||
3. **Format check** via `dotnet format --verify-no-changes`
|
||||
|
||||
### Required Analyzer Packages
|
||||
|
||||
```xml
|
||||
<ItemGroup>
|
||||
<PackageReference Include="Microsoft.CodeAnalysis.NetAnalyzers" Version="10.0.0-*" />
|
||||
<PackageReference Include="Meziantou.Analyzer" Version="2.0.*" />
|
||||
</ItemGroup>
|
||||
```
|
||||
|
||||
## Baseline Suppressions
|
||||
|
||||
Create `.globalconfig` for legacy code that cannot be immediately fixed:
|
||||
|
||||
```ini
|
||||
# Legacy suppressions - track issue for remediation
|
||||
[src/Policy/**/LegacyCode.cs]
|
||||
dotnet_diagnostic.DET-010.severity = suggestion
|
||||
```
|
||||
|
||||
## Runtime Enforcement
|
||||
|
||||
The `DeterminismGuardService` provides runtime monitoring:
|
||||
|
||||
```csharp
|
||||
using var scope = _determinismGuard.CreateScope(scopeId, timestamp);
|
||||
var result = await evaluation(scope);
|
||||
var analysis = scope.Complete();
|
||||
if (!analysis.Passed) { /* log/reject */ }
|
||||
```
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
1. All Policy projects build with zero errors
|
||||
2. `dotnet format` reports no changes needed
|
||||
3. DeterminismGuard analysis passes
|
||||
4. New code has no nullable warnings
|
||||
5. Async methods use `ConfigureAwait(false)`
|
||||
|
||||
## Related Documents
|
||||
|
||||
- [Deterministic Evaluator Design](./deterministic-evaluator.md)
|
||||
- [Policy Engine Architecture](../architecture.md)
|
||||
- [CONTRACT-POLICY-STUDIO-007](../../contracts/policy-studio.md)
|
||||
203
docs/modules/policy/design/policy-determinism-tests.md
Normal file
203
docs/modules/policy/design/policy-determinism-tests.md
Normal file
@@ -0,0 +1,203 @@
|
||||
# Policy Determinism Test Design
|
||||
|
||||
**Document ID:** `DESIGN-POLICY-DETERMINISM-TESTS-001`
|
||||
**Version:** 1.0
|
||||
**Status:** Published
|
||||
**Last Updated:** 2025-12-06
|
||||
|
||||
## Overview
|
||||
|
||||
This document defines the test expectations for ensuring deterministic output from Policy Engine scoring and decision APIs. Determinism is critical for reproducible policy evaluation across environments.
|
||||
|
||||
## Determinism Requirements
|
||||
|
||||
### Output Ordering
|
||||
|
||||
All collections in API responses must have stable, deterministic ordering:
|
||||
|
||||
| Collection | Ordering Rule |
|
||||
|------------|---------------|
|
||||
| Findings | By `finding_id` alphabetically |
|
||||
| Decisions | By `decision_id` (timestamp prefix) |
|
||||
| Signals | By signal name alphabetically |
|
||||
| Severity counts | By canonical severity order: critical → high → medium → low → info |
|
||||
| Contributions | By signal name alphabetically |
|
||||
|
||||
### JSON Serialization
|
||||
|
||||
1. **Property Order**: Use `[JsonPropertyOrder]` or declare properties in stable order
|
||||
2. **No Random Elements**: No GUIDs, random IDs, or timestamps unless from context
|
||||
3. **Stable Key Order**: Dictionary keys must serialize in consistent order
|
||||
|
||||
### Deprecated Field Absence
|
||||
|
||||
After v2.0, responses must NOT include:
|
||||
- `normalized_score`
|
||||
- `top_severity_sources`
|
||||
- `source_rank`
|
||||
|
||||
See [Normalized Field Removal](./policy-normalized-field-removal.md).
|
||||
|
||||
## Test Categories
|
||||
|
||||
### 1. Snapshot Equality Tests
|
||||
|
||||
Verify that identical inputs produce byte-for-byte identical JSON outputs.
|
||||
|
||||
```csharp
|
||||
[Theory]
|
||||
[MemberData(nameof(DeterminismFixtures))]
|
||||
public void Scoring_ShouldProduceDeterministicOutput(string inputFile)
|
||||
{
|
||||
// Arrange
|
||||
var input = LoadFixture(inputFile);
|
||||
|
||||
// Act - Run twice with same input
|
||||
var result1 = _scoringService.Score(input);
|
||||
var result2 = _scoringService.Score(input);
|
||||
|
||||
// Assert - Byte-for-byte equality
|
||||
var json1 = JsonSerializer.Serialize(result1);
|
||||
var json2 = JsonSerializer.Serialize(result2);
|
||||
Assert.Equal(json1, json2);
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Cross-Environment Tests
|
||||
|
||||
Verify output is identical across different environments (CI, local, prod).
|
||||
|
||||
```csharp
|
||||
[Theory]
|
||||
[InlineData("fixture-001")]
|
||||
public void Scoring_ShouldMatchGoldenFile(string fixtureId)
|
||||
{
|
||||
// Arrange
|
||||
var input = LoadFixture($"{fixtureId}-input.json");
|
||||
var expected = LoadFixture($"{fixtureId}-expected.json");
|
||||
|
||||
// Act
|
||||
var result = _scoringService.Score(input);
|
||||
|
||||
// Assert
|
||||
AssertJsonEqual(expected, result);
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Ordering Verification Tests
|
||||
|
||||
Verify collections are always in expected order.
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public void Decisions_ShouldBeOrderedByDecisionId()
|
||||
{
|
||||
// Arrange
|
||||
var input = CreateTestInput();
|
||||
|
||||
// Act
|
||||
var result = _decisionService.Evaluate(input);
|
||||
|
||||
// Assert
|
||||
Assert.True(result.Decisions.SequenceEqual(
|
||||
result.Decisions.OrderBy(d => d.DecisionId)));
|
||||
}
|
||||
```
|
||||
|
||||
### 4. Deprecated Field Absence Tests
|
||||
|
||||
Verify deprecated fields are not serialized (v2.0+).
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public void ScoringResult_ShouldNotIncludeNormalizedScore_InV2()
|
||||
{
|
||||
// Arrange
|
||||
var result = new RiskScoringResult(...);
|
||||
var options = new PolicyScoringOptions { IncludeLegacyNormalizedScore = false };
|
||||
|
||||
// Act
|
||||
var json = JsonSerializer.Serialize(result, CreateJsonOptions(options));
|
||||
var doc = JsonDocument.Parse(json);
|
||||
|
||||
// Assert
|
||||
Assert.False(doc.RootElement.TryGetProperty("normalized_score", out _));
|
||||
}
|
||||
```
|
||||
|
||||
## Fixture Structure
|
||||
|
||||
### Input Fixtures
|
||||
|
||||
Located at `docs/modules/policy/samples/policy-determinism-fixtures*.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"$schema": "https://stellaops.org/schemas/policy/determinism-fixture-v1.json",
|
||||
"fixture_id": "DET-001",
|
||||
"description": "Basic scoring determinism test",
|
||||
"input": {
|
||||
"finding_id": "CVE-2024-1234",
|
||||
"signals": {
|
||||
"cvss_base": 7.5,
|
||||
"exploitability": 2.8
|
||||
}
|
||||
},
|
||||
"expected_output": {
|
||||
"severity": "high",
|
||||
"signal_order": ["cvss_base", "exploitability"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Golden Files
|
||||
|
||||
Pre-computed expected outputs stored alongside inputs:
|
||||
- `policy-determinism-fixtures-input.json`
|
||||
- `policy-determinism-fixtures-expected.json`
|
||||
|
||||
## CI Integration
|
||||
|
||||
### Pipeline Steps
|
||||
|
||||
1. **Build**: Compile with analyzers
|
||||
2. **Unit Tests**: Run determinism unit tests
|
||||
3. **Snapshot Tests**: Compare against golden files
|
||||
4. **Diff Check**: Fail if any unexpected changes
|
||||
|
||||
### GitHub Action
|
||||
|
||||
```yaml
|
||||
- name: Run Determinism Tests
|
||||
run: |
|
||||
dotnet test --filter "Category=Determinism"
|
||||
|
||||
- name: Verify Snapshots
|
||||
run: |
|
||||
dotnet run --project tools/SnapshotVerifier -- \
|
||||
--fixtures docs/modules/policy/samples/policy-determinism-*.json
|
||||
```
|
||||
|
||||
## Maintenance
|
||||
|
||||
### Updating Golden Files
|
||||
|
||||
When intentionally changing output format:
|
||||
|
||||
1. Update design docs (this file, normalized-field-removal.md)
|
||||
2. Re-run tests with `UPDATE_GOLDEN=true`
|
||||
3. Review diffs
|
||||
4. Commit new golden files with explanation
|
||||
|
||||
### Adding New Fixtures
|
||||
|
||||
1. Create input fixture in `samples/`
|
||||
2. Run scoring to generate expected output
|
||||
3. Review for correctness
|
||||
4. Add to test data provider
|
||||
|
||||
## Related Documents
|
||||
|
||||
- [Policy AOC Linting Rules](./policy-aoc-linting-rules.md)
|
||||
- [Normalized Field Removal](./policy-normalized-field-removal.md)
|
||||
- [Deterministic Evaluator Design](./deterministic-evaluator.md)
|
||||
151
docs/modules/policy/design/policy-normalized-field-removal.md
Normal file
151
docs/modules/policy/design/policy-normalized-field-removal.md
Normal file
@@ -0,0 +1,151 @@
|
||||
# Policy Normalized Field Removal Design
|
||||
|
||||
**Document ID:** `DESIGN-POLICY-NORMALIZED-FIELD-REMOVAL-001`
|
||||
**Version:** 1.0
|
||||
**Status:** Draft
|
||||
**Last Updated:** 2025-12-06
|
||||
|
||||
## Overview
|
||||
|
||||
This document defines the migration plan for removing deprecated and legacy normalized fields from Policy Engine models. These fields were introduced for backwards compatibility but are now superseded by deterministic, canonical alternatives.
|
||||
|
||||
## Background
|
||||
|
||||
The Policy Engine currently includes several "normalized" fields that were designed for cross-source comparison but have been deprecated in favor of deterministic scoring:
|
||||
|
||||
1. **`normalized_score`** - Originally used for cross-vendor score comparison, now superseded by canonical severity scoring
|
||||
2. **`source_rank`** - Used for source prioritization, now handled by trust weighting service
|
||||
3. **Duplicated severity fields** - Multiple representations of the same severity data
|
||||
|
||||
## Fields Analysis
|
||||
|
||||
### Candidates for Removal
|
||||
|
||||
| Field | Location | Status | Migration Path |
|
||||
|-------|----------|--------|----------------|
|
||||
| `normalized_score` | `RiskScoringResult` | DEPRECATED | Use `severity` with canonical mapping |
|
||||
| `source_rank` in scoring | `PolicyDecisionSourceRank` | DEPRECATED | Use trust weighting service |
|
||||
| Legacy `severity_counts` | Multiple models | KEEP | Still used for aggregation |
|
||||
|
||||
### Fields to Retain
|
||||
|
||||
| Field | Location | Reason |
|
||||
|-------|----------|--------|
|
||||
| `severity` | All scoring models | Canonical severity (critical/high/medium/low/info) |
|
||||
| `profile_hash` | `RiskScoringResult` | Deterministic policy identification |
|
||||
| `trust_weight` | Decision models | Active trust weighting system |
|
||||
| `raw_score` | Scoring results | Needed for audit/debugging |
|
||||
|
||||
## Migration Plan
|
||||
|
||||
### Phase 1: Deprecation (Current State)
|
||||
|
||||
Fields marked with `[Obsolete]` attribute to warn consumers:
|
||||
|
||||
```csharp
|
||||
[Obsolete("Use severity with canonical mapping. Scheduled for removal in v2.0")]
|
||||
[JsonPropertyName("normalized_score")]
|
||||
public double NormalizedScore { get; init; }
|
||||
```
|
||||
|
||||
### Phase 2: Soft Removal (v1.5)
|
||||
|
||||
- Remove fields from serialization by default
|
||||
- Add configuration flag to re-enable for backwards compatibility
|
||||
- Update API documentation
|
||||
|
||||
### Phase 3: Hard Removal (v2.0)
|
||||
|
||||
- Remove fields from models entirely
|
||||
- Remove backwards compatibility flags
|
||||
|
||||
## API Impact
|
||||
|
||||
### Before (Current)
|
||||
|
||||
```json
|
||||
{
|
||||
"finding_id": "CVE-2024-1234",
|
||||
"raw_score": 7.5,
|
||||
"normalized_score": 0.75,
|
||||
"severity": "high",
|
||||
"source_ranks": [
|
||||
{"source": "nvd", "rank": 1},
|
||||
{"source": "vendor", "rank": 2}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### After (Target)
|
||||
|
||||
```json
|
||||
{
|
||||
"finding_id": "CVE-2024-1234",
|
||||
"raw_score": 7.5,
|
||||
"severity": "high",
|
||||
"trust_weights": {
|
||||
"nvd": 1.0,
|
||||
"vendor": 0.8
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Step 1: Add Deprecation Markers
|
||||
|
||||
Add `[Obsolete]` to target fields with clear migration guidance.
|
||||
|
||||
### Step 2: Create Compatibility Layer
|
||||
|
||||
Add opt-in flag for legacy serialization:
|
||||
|
||||
```csharp
|
||||
public class PolicyScoringOptions
|
||||
{
|
||||
/// <summary>
|
||||
/// Include deprecated normalized_score field for backwards compatibility.
|
||||
/// </summary>
|
||||
public bool IncludeLegacyNormalizedScore { get; set; } = false;
|
||||
}
|
||||
```
|
||||
|
||||
### Step 3: Update Serialization
|
||||
|
||||
Use conditional serialization based on options:
|
||||
|
||||
```csharp
|
||||
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
|
||||
public double? NormalizedScore =>
|
||||
_options.IncludeLegacyNormalizedScore ? _normalizedScore : null;
|
||||
```
|
||||
|
||||
### Step 4: Update Documentation
|
||||
|
||||
- Mark fields as deprecated in OpenAPI spec
|
||||
- Add migration guide to release notes
|
||||
|
||||
## Fixtures
|
||||
|
||||
Sample payloads are available in:
|
||||
- `docs/modules/policy/samples/policy-normalized-field-removal-before.json`
|
||||
- `docs/modules/policy/samples/policy-normalized-field-removal-after.json`
|
||||
|
||||
## Rollback Strategy
|
||||
|
||||
If issues arise during migration:
|
||||
1. Re-enable legacy fields via configuration
|
||||
2. No data loss - fields are computed, not stored
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
1. Deprecated fields marked with `[Obsolete]`
|
||||
2. Configuration option for backwards compatibility
|
||||
3. API documentation updated
|
||||
4. Migration guide published
|
||||
5. Fixtures validate before/after behavior
|
||||
|
||||
## Related Documents
|
||||
|
||||
- [Policy AOC Linting Rules](./policy-aoc-linting-rules.md)
|
||||
- [CONTRACT-AUTHORITY-EFFECTIVE-WRITE-008](../../contracts/authority-effective-write.md)
|
||||
165
docs/modules/policy/samples/policy-determinism-fixtures.json
Normal file
165
docs/modules/policy/samples/policy-determinism-fixtures.json
Normal file
@@ -0,0 +1,165 @@
|
||||
{
|
||||
"$schema": "https://stellaops.org/schemas/policy/determinism-fixture-v1.json",
|
||||
"version": "1.0.0",
|
||||
"description": "Determinism fixtures for Policy Engine scoring and decision APIs",
|
||||
"fixtures": [
|
||||
{
|
||||
"fixture_id": "DET-001",
|
||||
"name": "Basic Scoring Determinism",
|
||||
"description": "Verify that scoring produces identical output for identical input",
|
||||
"input": {
|
||||
"finding_id": "CVE-2024-0001",
|
||||
"tenant_id": "default",
|
||||
"profile_id": "risk-profile-001",
|
||||
"signals": {
|
||||
"cvss_base": 7.5,
|
||||
"exploitability": 2.8,
|
||||
"impact": 5.9
|
||||
}
|
||||
},
|
||||
"expected_output": {
|
||||
"severity": "high",
|
||||
"raw_score": 7.5,
|
||||
"signal_order": ["cvss_base", "exploitability", "impact"],
|
||||
"assertions": [
|
||||
"signal_contributions keys are alphabetically ordered",
|
||||
"scored_at is from context, not wall clock"
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"fixture_id": "DET-002",
|
||||
"name": "Multi-Finding Ordering",
|
||||
"description": "Verify that multiple findings are returned in stable order",
|
||||
"input": {
|
||||
"findings": [
|
||||
{"finding_id": "CVE-2024-0003", "cvss_base": 5.0},
|
||||
{"finding_id": "CVE-2024-0001", "cvss_base": 9.8},
|
||||
{"finding_id": "CVE-2024-0002", "cvss_base": 7.5}
|
||||
]
|
||||
},
|
||||
"expected_output": {
|
||||
"finding_order": ["CVE-2024-0001", "CVE-2024-0002", "CVE-2024-0003"],
|
||||
"assertions": [
|
||||
"findings sorted alphabetically by finding_id",
|
||||
"order is stable across multiple runs"
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"fixture_id": "DET-003",
|
||||
"name": "Decision Summary Ordering",
|
||||
"description": "Verify severity counts are in canonical order",
|
||||
"input": {
|
||||
"decisions": [
|
||||
{"severity": "low", "count": 5},
|
||||
{"severity": "critical", "count": 1},
|
||||
{"severity": "medium", "count": 3},
|
||||
{"severity": "high", "count": 2}
|
||||
]
|
||||
},
|
||||
"expected_output": {
|
||||
"severity_order": ["critical", "high", "medium", "low", "info"],
|
||||
"assertions": [
|
||||
"severity_counts keys follow canonical order",
|
||||
"missing severities are either omitted or zero-filled consistently"
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"fixture_id": "DET-004",
|
||||
"name": "Deprecated Field Absence (v2.0)",
|
||||
"description": "Verify deprecated fields are not present in v2.0 output",
|
||||
"input": {
|
||||
"finding_id": "CVE-2024-0001",
|
||||
"cvss_base": 7.5,
|
||||
"version": "2.0"
|
||||
},
|
||||
"expected_output": {
|
||||
"absent_fields": [
|
||||
"normalized_score",
|
||||
"top_severity_sources",
|
||||
"source_rank"
|
||||
],
|
||||
"present_fields": [
|
||||
"severity",
|
||||
"raw_score",
|
||||
"trust_weights"
|
||||
],
|
||||
"assertions": [
|
||||
"normalized_score is not serialized",
|
||||
"trust_weights replaces top_severity_sources"
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"fixture_id": "DET-005",
|
||||
"name": "Legacy Compatibility Mode (v1.5)",
|
||||
"description": "Verify deprecated fields are present when legacy mode enabled",
|
||||
"input": {
|
||||
"finding_id": "CVE-2024-0001",
|
||||
"cvss_base": 7.5,
|
||||
"options": {
|
||||
"include_legacy_normalized_score": true
|
||||
}
|
||||
},
|
||||
"expected_output": {
|
||||
"present_fields": [
|
||||
"normalized_score",
|
||||
"severity",
|
||||
"raw_score"
|
||||
],
|
||||
"assertions": [
|
||||
"normalized_score is present for backwards compatibility",
|
||||
"severity is canonical (high, not HIGH)"
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"fixture_id": "DET-006",
|
||||
"name": "Signal Contribution Ordering",
|
||||
"description": "Verify signal contributions maintain stable key order",
|
||||
"input": {
|
||||
"signals": {
|
||||
"zeta_factor": 0.5,
|
||||
"alpha_score": 1.0,
|
||||
"beta_weight": 0.75
|
||||
}
|
||||
},
|
||||
"expected_output": {
|
||||
"contribution_order": ["alpha_score", "beta_weight", "zeta_factor"],
|
||||
"assertions": [
|
||||
"signal_contributions keys are alphabetically sorted",
|
||||
"contribution values are deterministic decimals"
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"fixture_id": "DET-007",
|
||||
"name": "Timestamp Determinism",
|
||||
"description": "Verify timestamps come from context, not wall clock",
|
||||
"input": {
|
||||
"finding_id": "CVE-2024-0001",
|
||||
"context": {
|
||||
"evaluation_time": "2025-12-06T10:00:00Z"
|
||||
}
|
||||
},
|
||||
"expected_output": {
|
||||
"scored_at": "2025-12-06T10:00:00Z",
|
||||
"assertions": [
|
||||
"scored_at matches context.evaluation_time exactly",
|
||||
"no random GUIDs in output"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"test_requirements": {
|
||||
"snapshot_equality": "Identical inputs must produce byte-for-byte identical JSON",
|
||||
"cross_environment": "Output must match across CI, local, and production",
|
||||
"ordering_stability": "Collection order must be deterministic and documented"
|
||||
},
|
||||
"migration_notes": {
|
||||
"v1.5": "Enable legacy mode with include_legacy_normalized_score for backwards compatibility",
|
||||
"v2.0": "Remove all deprecated fields, trust_weights replaces source ranking"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,43 @@
|
||||
{
|
||||
"$schema": "https://stellaops.org/schemas/policy/scoring-result-v2.json",
|
||||
"description": "Sample scoring result AFTER normalized field removal (canonical format)",
|
||||
"scoring_result": {
|
||||
"finding_id": "CVE-2024-1234",
|
||||
"tenant_id": "default",
|
||||
"profile_id": "risk-profile-001",
|
||||
"profile_version": "1.2.0",
|
||||
"raw_score": 7.5,
|
||||
"severity": "high",
|
||||
"signal_values": {
|
||||
"cvss_base": 7.5,
|
||||
"exploitability": 2.8,
|
||||
"impact": 5.9
|
||||
},
|
||||
"scored_at": "2025-12-06T10:00:00Z",
|
||||
"profile_hash": "sha256:abc123def456..."
|
||||
},
|
||||
"decision_summary": {
|
||||
"total_decisions": 5,
|
||||
"total_conflicts": 1,
|
||||
"severity_counts": {
|
||||
"critical": 0,
|
||||
"high": 3,
|
||||
"medium": 2,
|
||||
"low": 0
|
||||
},
|
||||
"trust_weights": {
|
||||
"nvd": 1.0,
|
||||
"vendor-advisory": 0.8
|
||||
}
|
||||
},
|
||||
"migration_notes": {
|
||||
"removed_fields": ["normalized_score", "top_severity_sources"],
|
||||
"added_fields": ["profile_hash", "trust_weights"],
|
||||
"canonical_severity_mapping": {
|
||||
"0.0-3.9": "low",
|
||||
"4.0-6.9": "medium",
|
||||
"7.0-8.9": "high",
|
||||
"9.0-10.0": "critical"
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,41 @@
|
||||
{
|
||||
"$schema": "https://stellaops.org/schemas/policy/scoring-result-v1.json",
|
||||
"description": "Sample scoring result BEFORE normalized field removal (legacy format)",
|
||||
"scoring_result": {
|
||||
"finding_id": "CVE-2024-1234",
|
||||
"tenant_id": "default",
|
||||
"profile_id": "risk-profile-001",
|
||||
"profile_version": "1.2.0",
|
||||
"raw_score": 7.5,
|
||||
"normalized_score": 0.75,
|
||||
"severity": "high",
|
||||
"signal_values": {
|
||||
"cvss_base": 7.5,
|
||||
"exploitability": 2.8,
|
||||
"impact": 5.9
|
||||
},
|
||||
"scored_at": "2025-12-06T10:00:00Z"
|
||||
},
|
||||
"decision_summary": {
|
||||
"total_decisions": 5,
|
||||
"total_conflicts": 1,
|
||||
"severity_counts": {
|
||||
"critical": 0,
|
||||
"high": 3,
|
||||
"medium": 2,
|
||||
"low": 0
|
||||
},
|
||||
"top_severity_sources": [
|
||||
{
|
||||
"source": "nvd",
|
||||
"total_weight": 1.0,
|
||||
"finding_count": 3
|
||||
},
|
||||
{
|
||||
"source": "vendor-advisory",
|
||||
"total_weight": 0.8,
|
||||
"finding_count": 2
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
297
docs/modules/vex-lens/runbooks/operations.md
Normal file
297
docs/modules/vex-lens/runbooks/operations.md
Normal file
@@ -0,0 +1,297 @@
|
||||
# VexLens Operations Runbook
|
||||
|
||||
> VexLens provides VEX consensus computation across multiple issuer sources. This runbook covers deployment, configuration, operations, and troubleshooting.
|
||||
|
||||
## 1. Service scope
|
||||
|
||||
VexLens computes deterministic consensus over VEX (Vulnerability Exploitability eXchange) statements from multiple issuers. Operations owns:
|
||||
|
||||
- Consensus engine scaling, projection storage, and event bus connectivity.
|
||||
- Monitoring and alerts for consensus latency, conflict rates, and trust weight anomalies.
|
||||
- Runbook execution for recovery, offline bundle import, and issuer trust management.
|
||||
- Coordination with Policy Engine and Vuln Explorer consumers.
|
||||
|
||||
Related documentation:
|
||||
|
||||
- `docs/modules/vex-lens/README.md`
|
||||
- `docs/modules/vex-lens/architecture.md`
|
||||
- `docs/modules/vex-lens/implementation_plan.md`
|
||||
- `docs/modules/vex-lens/runbooks/observability.md`
|
||||
|
||||
## 2. Contacts & tooling
|
||||
|
||||
| Area | Owner(s) | Escalation |
|
||||
|------|----------|------------|
|
||||
| VexLens service | VEX Lens Guild | `#vex-lens-ops`, on-call rotation |
|
||||
| Issuer Directory | Issuer Directory Guild | `#issuer-directory` |
|
||||
| Policy Engine integration | Policy Guild | `#policy-engine` |
|
||||
| Offline Kit | Offline Kit Guild | `#offline-kit` |
|
||||
|
||||
Primary tooling:
|
||||
|
||||
- `stella vex consensus` CLI (query, export, verify).
|
||||
- VexLens API (`/api/v1/vex/consensus/*`) for automation.
|
||||
- Grafana dashboards (`VEX Lens / Consensus Health`, `VEX Lens / Conflicts`).
|
||||
- Alertmanager routes (`VexLens.ConsensusLatency`, `VexLens.Conflicts`).
|
||||
|
||||
## 3. Configuration
|
||||
|
||||
### 3.1 Options reference
|
||||
|
||||
Configure via `vexlens.yaml` or environment variables with `VEXLENS_` prefix:
|
||||
|
||||
```yaml
|
||||
VexLens:
|
||||
Storage:
|
||||
Driver: mongo # "memory" for testing, "mongo" for production
|
||||
ConnectionString: "mongodb://..."
|
||||
Database: stellaops
|
||||
ProjectionsCollection: vex_consensus
|
||||
HistoryCollection: vex_consensus_history
|
||||
MaxHistoryEntries: 100
|
||||
CommandTimeoutSeconds: 30
|
||||
|
||||
Trust:
|
||||
AuthoritativeWeight: 1.0
|
||||
TrustedWeight: 0.8
|
||||
KnownWeight: 0.5
|
||||
UnknownWeight: 0.3
|
||||
UntrustedWeight: 0.1
|
||||
SignedMultiplier: 1.2
|
||||
FreshnessDecayDays: 30
|
||||
MinFreshnessFactor: 0.5
|
||||
JustifiedNotAffectedBoost: 1.1
|
||||
FixedStatusBoost: 1.05
|
||||
|
||||
Consensus:
|
||||
DefaultMode: WeightedVote # HighestWeight, WeightedVote, Lattice, AuthoritativeFirst
|
||||
MinimumWeightThreshold: 0.1
|
||||
ConflictThreshold: 0.3
|
||||
RequireJustificationForNotAffected: false
|
||||
MaxStatementsPerComputation: 100
|
||||
EnableConflictDetection: true
|
||||
EmitEvents: true
|
||||
|
||||
Normalization:
|
||||
EnabledFormats:
|
||||
- OpenVEX
|
||||
- CSAF
|
||||
- CycloneDX
|
||||
StrictMode: false
|
||||
MaxDocumentSizeBytes: 10485760 # 10 MB
|
||||
MaxStatementsPerDocument: 10000
|
||||
|
||||
AirGap:
|
||||
SealedMode: false
|
||||
BundlePath: /var/lib/stellaops/vex-bundles
|
||||
VerifyBundleSignatures: true
|
||||
AllowedBundleSources: []
|
||||
ExportFormat: jsonl
|
||||
|
||||
Telemetry:
|
||||
MetricsEnabled: true
|
||||
TracingEnabled: true
|
||||
MeterName: StellaOps.VexLens
|
||||
ActivitySourceName: StellaOps.VexLens
|
||||
```
|
||||
|
||||
### 3.2 Environment variable overrides
|
||||
|
||||
```bash
|
||||
VEXLENS_STORAGE__DRIVER=mongo
|
||||
VEXLENS_STORAGE__CONNECTIONSTRING=mongodb://localhost:27017
|
||||
VEXLENS_CONSENSUS__DEFAULTMODE=WeightedVote
|
||||
VEXLENS_AIRGAP__SEALEDMODE=true
|
||||
```
|
||||
|
||||
### 3.3 Consensus mode selection
|
||||
|
||||
| Mode | Use case |
|
||||
|------|----------|
|
||||
| HighestWeight | Single authoritative source preferred |
|
||||
| WeightedVote | Democratic consensus from multiple sources |
|
||||
| Lattice | Formal lattice join (most conservative) |
|
||||
| AuthoritativeFirst | Short-circuit on authoritative issuer |
|
||||
|
||||
## 4. Monitoring & SLOs
|
||||
|
||||
Key metrics (exposed by VexLensMetrics):
|
||||
|
||||
| Metric | SLO / Alert | Notes |
|
||||
|--------|-------------|-------|
|
||||
| `vexlens.consensus.duration_seconds` | p95 < 500ms | Per-computation latency |
|
||||
| `vexlens.consensus.conflicts_total` | Monitor trend | Conflicts by reason |
|
||||
| `vexlens.consensus.confidence` | avg > 0.7 | Low confidence indicates issuer gaps |
|
||||
| `vexlens.normalization.duration_seconds` | p95 < 200ms | Per-document normalization |
|
||||
| `vexlens.normalization.errors_total` | Alert on spike | By format |
|
||||
| `vexlens.trust.weight_value` | Distribution | Trust weight distribution |
|
||||
| `vexlens.projection.query_duration_seconds` | p95 < 100ms | Projection lookups |
|
||||
|
||||
Dashboards must include:
|
||||
|
||||
- Consensus computation rate by mode and outcome.
|
||||
- Conflict breakdown (status disagreement, weight tie, insufficient data).
|
||||
- Trust weight distribution by issuer category.
|
||||
- Normalization success/failure by VEX format.
|
||||
- Projection query latency and throughput.
|
||||
|
||||
Alerts (Alertmanager):
|
||||
|
||||
- `VexLensConsensusLatencyHigh` - consensus duration p95 > 500ms for 5 minutes.
|
||||
- `VexLensConflictSpike` - conflict rate increase > 50% in 10 minutes.
|
||||
- `VexLensNormalizationFailures` - normalization error rate > 5% for 5 minutes.
|
||||
- `VexLensLowConfidence` - average confidence < 0.5 for 10 minutes.
|
||||
|
||||
## 5. Routine operations
|
||||
|
||||
### 5.1 Daily checklist
|
||||
|
||||
- Review dashboard for consensus latency and conflict rates.
|
||||
- Check normalization error logs for malformed VEX documents.
|
||||
- Verify projection storage growth is within capacity thresholds.
|
||||
- Review trust weight distribution for anomalies.
|
||||
- Scan logs for `issuer_not_found` or `signature_verification_failed`.
|
||||
|
||||
### 5.2 Weekly tasks
|
||||
|
||||
- Review issuer directory for new registrations or revocations.
|
||||
- Audit conflict queue for persistent disagreements.
|
||||
- Test consensus determinism with sample documents.
|
||||
- Verify Policy Engine and Vuln Explorer integrations are functional.
|
||||
|
||||
### 5.3 Monthly tasks
|
||||
|
||||
- Review and tune trust weights based on issuer performance.
|
||||
- Archive old projection history beyond retention period.
|
||||
- Update issuer trust tiers based on incident history.
|
||||
- Test offline bundle import/export workflow.
|
||||
|
||||
## 6. Offline operations
|
||||
|
||||
### 6.1 Bundle export
|
||||
|
||||
```bash
|
||||
# Export consensus projections to offline bundle
|
||||
stella vex consensus export \
|
||||
--format jsonl \
|
||||
--output /var/lib/stellaops/vex-bundles/consensus-2025-01.jsonl \
|
||||
--manifest /var/lib/stellaops/vex-bundles/manifest.json \
|
||||
--sign
|
||||
|
||||
# Verify bundle integrity
|
||||
stella vex consensus verify \
|
||||
--bundle /var/lib/stellaops/vex-bundles/consensus-2025-01.jsonl \
|
||||
--manifest /var/lib/stellaops/vex-bundles/manifest.json
|
||||
```
|
||||
|
||||
### 6.2 Bundle import (air-gapped)
|
||||
|
||||
```bash
|
||||
# Enable sealed mode
|
||||
export VEXLENS_AIRGAP__SEALEDMODE=true
|
||||
export VEXLENS_AIRGAP__BUNDLEPATH=/var/lib/stellaops/vex-bundles
|
||||
|
||||
# Import bundle
|
||||
stella vex consensus import \
|
||||
--bundle /var/lib/stellaops/vex-bundles/consensus-2025-01.jsonl \
|
||||
--verify-signatures
|
||||
|
||||
# Verify import
|
||||
stella vex consensus status
|
||||
```
|
||||
|
||||
### 6.3 Air-gap verification
|
||||
|
||||
1. Confirm `VEXLENS_AIRGAP__SEALEDMODE=true` in environment.
|
||||
2. Verify no external network calls in service logs.
|
||||
3. Check bundle manifest hashes match imported data.
|
||||
4. Run determinism check on imported projections.
|
||||
|
||||
## 7. Troubleshooting
|
||||
|
||||
### 7.1 High conflict rates
|
||||
|
||||
**Symptoms:** `vexlens.consensus.conflicts_total` spiking.
|
||||
|
||||
**Investigation:**
|
||||
1. Check conflict breakdown by reason in dashboard.
|
||||
2. Identify issuers with conflicting statements.
|
||||
3. Review issuer trust tiers and weights.
|
||||
|
||||
**Resolution:**
|
||||
- Adjust `ConflictThreshold` if legitimate disagreements.
|
||||
- Update issuer trust tiers based on authority.
|
||||
- Contact issuer owners to resolve source conflicts.
|
||||
|
||||
### 7.2 Normalization failures
|
||||
|
||||
**Symptoms:** `vexlens.normalization.errors_total` increasing.
|
||||
|
||||
**Investigation:**
|
||||
1. Check error logs for specific format failures.
|
||||
2. Identify malformed documents in input stream.
|
||||
3. Validate document against format schema.
|
||||
|
||||
**Resolution:**
|
||||
- Enable `StrictMode: false` for lenient parsing.
|
||||
- Report malformed documents to source issuers.
|
||||
- Update normalizers if format specification changed.
|
||||
|
||||
### 7.3 Low consensus confidence
|
||||
|
||||
**Symptoms:** Average confidence below 0.5.
|
||||
|
||||
**Investigation:**
|
||||
1. Check issuer coverage for affected vulnerabilities.
|
||||
2. Review trust weight distribution.
|
||||
3. Identify missing or untrusted issuers.
|
||||
|
||||
**Resolution:**
|
||||
- Register additional trusted issuers.
|
||||
- Adjust trust tier assignments.
|
||||
- Import offline bundles from authoritative sources.
|
||||
|
||||
### 7.4 Projection storage growth
|
||||
|
||||
**Symptoms:** Storage usage increasing beyond capacity.
|
||||
|
||||
**Investigation:**
|
||||
1. Check `MaxHistoryEntries` setting.
|
||||
2. Review projection count and history depth.
|
||||
3. Identify high-churn vulnerability/product pairs.
|
||||
|
||||
**Resolution:**
|
||||
- Reduce `MaxHistoryEntries`.
|
||||
- Implement history pruning job.
|
||||
- Archive old projections to cold storage.
|
||||
|
||||
## 8. Recovery procedures
|
||||
|
||||
### 8.1 Storage failover
|
||||
|
||||
1. Stop VexLens service instances.
|
||||
2. Switch storage connection to replica.
|
||||
3. Verify connectivity with health check.
|
||||
4. Restart service instances.
|
||||
5. Monitor for consensus recomputation.
|
||||
|
||||
### 8.2 Issuer directory sync
|
||||
|
||||
1. Export current issuer registry backup.
|
||||
2. Resync from authoritative issuer directory source.
|
||||
3. Verify issuer fingerprints and trust tiers.
|
||||
4. Restart VexLens to reload issuer cache.
|
||||
|
||||
### 8.3 Consensus recomputation
|
||||
|
||||
1. Trigger recomputation for affected vulnerability/product pairs.
|
||||
2. Monitor recomputation progress in logs.
|
||||
3. Verify consensus outcomes match expected state.
|
||||
4. Emit status change events if outcomes differ.
|
||||
|
||||
## 9. Evidence locations
|
||||
|
||||
- Sprint tracker: `docs/implplan/SPRINT_0129_0001_0001_policy_reasoning.md`
|
||||
- Module docs: `docs/modules/vex-lens/`
|
||||
- Source code: `src/VexLens/StellaOps.VexLens/`
|
||||
- Dashboard stub: `docs/modules/vex-lens/runbooks/dashboards/vex-lens-observability.json`
|
||||
Reference in New Issue
Block a user