502 lines
14 KiB
Markdown
502 lines
14 KiB
Markdown
# Cross-Cutting Testing Standards Guide
|
|
|
|
This guide documents the cross-cutting testing standards implemented for StellaOps, including blast-radius annotations, schema evolution testing, dead-path detection, and config-diff testing.
|
|
|
|
**Sprint Reference:** SPRINT_20260105_002_005_TEST_cross_cutting
|
|
|
|
---
|
|
|
|
## Table of Contents
|
|
|
|
1. [Overview](#overview)
|
|
2. [Blast-Radius Annotations](#blast-radius-annotations)
|
|
3. [Schema Evolution Testing](#schema-evolution-testing)
|
|
4. [Dead-Path Detection](#dead-path-detection)
|
|
5. [Config-Diff Testing](#config-diff-testing)
|
|
6. [CI Workflows](#ci-workflows)
|
|
7. [Best Practices](#best-practices)
|
|
|
|
---
|
|
|
|
## Overview
|
|
|
|
Cross-cutting testing standards ensure consistent test quality across all modules:
|
|
|
|
| Standard | Purpose | Enforcement |
|
|
|----------|---------|-------------|
|
|
| **Blast-Radius** | Categorize tests by operational surface | CI validation on PRs |
|
|
| **Schema Evolution** | Verify backward compatibility | CI on schema changes |
|
|
| **Dead-Path Detection** | Identify uncovered code | CI with baseline comparison |
|
|
| **Config-Diff** | Validate config behavioral isolation | Integration tests |
|
|
|
|
---
|
|
|
|
## Blast-Radius Annotations
|
|
|
|
### Purpose
|
|
|
|
Blast-radius annotations categorize tests by the operational surfaces they affect. During incidents, this enables targeted test runs for specific areas (e.g., run only Auth-related tests when investigating an authentication issue).
|
|
|
|
### Categories
|
|
|
|
| Category | Description | Examples |
|
|
|----------|-------------|----------|
|
|
| `Auth` | Authentication, authorization, tokens | Login, OAuth, DPoP |
|
|
| `Scanning` | SBOM generation, vulnerability scanning | Scanner, analyzers |
|
|
| `Evidence` | Attestation, evidence storage | EvidenceLocker, Attestor |
|
|
| `Compliance` | Audit, regulatory, GDPR | Compliance reports |
|
|
| `Advisories` | Advisory ingestion, VEX processing | Concelier, VexLens |
|
|
| `RiskPolicy` | Risk scoring, policy evaluation | RiskEngine, Policy |
|
|
| `Crypto` | Cryptographic operations | Signing, verification |
|
|
| `Integrations` | External systems, webhooks | Notifications, webhooks |
|
|
| `Persistence` | Database operations | Repositories, migrations |
|
|
| `Api` | API surface, contracts | Controllers, endpoints |
|
|
|
|
### Usage
|
|
|
|
```csharp
|
|
using StellaOps.TestKit;
|
|
using Xunit;
|
|
|
|
// Single blast-radius
|
|
[Trait("Category", TestCategories.Integration)]
|
|
[Trait("BlastRadius", TestCategories.BlastRadius.Auth)]
|
|
public class TokenValidationTests
|
|
{
|
|
[Fact]
|
|
public async Task ValidToken_ReturnsSuccess()
|
|
{
|
|
// Test implementation
|
|
}
|
|
}
|
|
|
|
// Multiple blast-radii (affects multiple surfaces)
|
|
[Trait("Category", TestCategories.Integration)]
|
|
[Trait("BlastRadius", TestCategories.BlastRadius.Auth)]
|
|
[Trait("BlastRadius", TestCategories.BlastRadius.Api)]
|
|
public class AuthenticatedApiTests
|
|
{
|
|
// Tests that affect both Auth and Api surfaces
|
|
}
|
|
```
|
|
|
|
### Requirements
|
|
|
|
- **Integration tests**: Must have at least one BlastRadius annotation
|
|
- **Contract tests**: Must have at least one BlastRadius annotation
|
|
- **Security tests**: Must have at least one BlastRadius annotation
|
|
- **Unit tests**: BlastRadius optional but recommended
|
|
|
|
### Running Tests by Blast-Radius
|
|
|
|
```bash
|
|
# Run all Auth-related tests
|
|
dotnet test --filter "BlastRadius=Auth"
|
|
|
|
# Run tests for multiple surfaces
|
|
dotnet test --filter "BlastRadius=Auth|BlastRadius=Api"
|
|
|
|
# Run incident response test suite
|
|
dotnet run --project src/__Libraries/StellaOps.TestKit \
|
|
-- run-blast-radius Auth,Api --fail-fast
|
|
```
|
|
|
|
---
|
|
|
|
## Schema Evolution Testing
|
|
|
|
### Purpose
|
|
|
|
Schema evolution tests verify that code remains compatible with previous database schema versions. This prevents breaking changes during:
|
|
|
|
- Rolling deployments (new code, old schema)
|
|
- Rollbacks (old code, new schema)
|
|
- Migration windows
|
|
|
|
### Schema Versions
|
|
|
|
| Version | Description |
|
|
|---------|-------------|
|
|
| `N` | Current schema (HEAD) |
|
|
| `N-1` | Previous schema version |
|
|
| `N-2` | Two versions back |
|
|
|
|
### Using SchemaEvolutionTestBase
|
|
|
|
```csharp
|
|
using StellaOps.Testing.SchemaEvolution;
|
|
using Testcontainers.PostgreSql;
|
|
using Xunit;
|
|
|
|
[Trait("Category", TestCategories.SchemaEvolution)]
|
|
public class ScannerSchemaEvolutionTests : PostgresSchemaEvolutionTestBase
|
|
{
|
|
public ScannerSchemaEvolutionTests()
|
|
: base(new SchemaEvolutionConfig
|
|
{
|
|
ModuleName = "Scanner",
|
|
CurrentVersion = new SchemaVersion("v2.1.0",
|
|
DateTimeOffset.Parse("2026-01-01")),
|
|
PreviousVersions =
|
|
[
|
|
new SchemaVersion("v2.0.0",
|
|
DateTimeOffset.Parse("2025-10-01")),
|
|
new SchemaVersion("v1.9.0",
|
|
DateTimeOffset.Parse("2025-07-01"))
|
|
],
|
|
ConnectionStringTemplate =
|
|
"Host={0};Port={1};Database={2};Username={3};Password={4}"
|
|
})
|
|
{
|
|
}
|
|
|
|
[Fact]
|
|
public async Task ReadOperations_CompatibleWithPreviousSchema()
|
|
{
|
|
var result = await TestReadBackwardCompatibilityAsync(
|
|
async (connection, version) =>
|
|
{
|
|
// Test read operations against old schema
|
|
var repository = new ScanRepository(connection);
|
|
var scans = await repository.GetRecentScansAsync(10);
|
|
return scans.Count >= 0;
|
|
});
|
|
|
|
Assert.True(result.IsSuccess);
|
|
}
|
|
|
|
[Fact]
|
|
public async Task WriteOperations_CompatibleWithPreviousSchema()
|
|
{
|
|
var result = await TestWriteForwardCompatibilityAsync(
|
|
async (connection, version) =>
|
|
{
|
|
// Test write operations
|
|
var repository = new ScanRepository(connection);
|
|
await repository.CreateScanAsync(new ScanRequest { /* ... */ });
|
|
return true;
|
|
});
|
|
|
|
Assert.True(result.IsSuccess);
|
|
}
|
|
}
|
|
```
|
|
|
|
### Versioned Container Images
|
|
|
|
Build versioned PostgreSQL images for testing:
|
|
|
|
```bash
|
|
# Build all versions for a module
|
|
./devops/docker/schema-versions/build-schema-images.sh scanner
|
|
|
|
# Build specific version
|
|
./devops/docker/schema-versions/build-schema-images.sh scanner v2.0.0
|
|
|
|
# Use in tests
|
|
docker run -d -p 5432:5432 ghcr.io/stellaops/schema-test:scanner-v2.0.0
|
|
```
|
|
|
|
---
|
|
|
|
## Dead-Path Detection
|
|
|
|
### Purpose
|
|
|
|
Dead-path detection identifies uncovered code branches. This helps:
|
|
|
|
- Find untested edge cases
|
|
- Identify potentially dead code
|
|
- Prevent coverage regression
|
|
|
|
### How It Works
|
|
|
|
1. Tests run with branch coverage collection (Coverlet)
|
|
2. Cobertura XML report is parsed
|
|
3. Uncovered branches are identified
|
|
4. New dead paths are compared against baseline
|
|
5. CI fails if new dead paths are introduced
|
|
|
|
### Baseline Management
|
|
|
|
The baseline file (`dead-paths-baseline.json`) tracks known dead paths:
|
|
|
|
```json
|
|
{
|
|
"version": "1.0.0",
|
|
"activeDeadPaths": 42,
|
|
"totalDeadPaths": 50,
|
|
"exemptedPaths": 8,
|
|
"entries": [
|
|
{
|
|
"file": "src/Scanner/Services/AnalyzerService.cs",
|
|
"line": 128,
|
|
"coverage": "1/2",
|
|
"isExempt": false
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
### Exemptions
|
|
|
|
Add exemptions for intentionally untested code in `coverage-exemptions.yaml`:
|
|
|
|
```yaml
|
|
exemptions:
|
|
- path: "src/Authority/Emergency/BreakGlassHandler.cs:42"
|
|
category: emergency
|
|
justification: "Emergency access bypass - tested in incident drills"
|
|
added: "2026-01-06"
|
|
owner: "security-team"
|
|
|
|
- path: "src/Scanner/Platform/WindowsRegistryScanner.cs:*"
|
|
category: platform
|
|
justification: "Windows-only code - CI runs on Linux"
|
|
added: "2026-01-06"
|
|
owner: "scanner-team"
|
|
|
|
ignore_patterns:
|
|
- "*.Generated.cs"
|
|
- "**/Migrations/*.cs"
|
|
```
|
|
|
|
### Using BranchCoverageEnforcer
|
|
|
|
```csharp
|
|
using StellaOps.Testing.Coverage;
|
|
|
|
var enforcer = new BranchCoverageEnforcer(new BranchCoverageConfig
|
|
{
|
|
MinimumBranchCoverage = 80,
|
|
FailOnNewDeadPaths = true,
|
|
ExemptionFiles = ["coverage-exemptions.yaml"]
|
|
});
|
|
|
|
// Parse coverage report
|
|
var parser = new CoberturaParser();
|
|
var coverage = await parser.ParseFileAsync("coverage.cobertura.xml");
|
|
|
|
// Validate
|
|
var result = enforcer.Validate(coverage);
|
|
if (!result.IsValid)
|
|
{
|
|
foreach (var violation in result.Violations)
|
|
{
|
|
Console.WriteLine($"Violation: {violation.File}:{violation.Line}");
|
|
}
|
|
}
|
|
|
|
// Generate dead-path report
|
|
var report = enforcer.GenerateDeadPathReport(coverage);
|
|
Console.WriteLine($"Active dead paths: {report.ActiveDeadPaths}");
|
|
```
|
|
|
|
---
|
|
|
|
## Config-Diff Testing
|
|
|
|
### Purpose
|
|
|
|
Config-diff tests verify that configuration changes produce only expected behavioral deltas. This prevents:
|
|
|
|
- Unintended side effects from config changes
|
|
- Config options affecting unrelated behaviors
|
|
- Regressions in config handling
|
|
|
|
### Using ConfigDiffTestBase
|
|
|
|
```csharp
|
|
using StellaOps.Testing.ConfigDiff;
|
|
using Xunit;
|
|
|
|
[Trait("Category", TestCategories.ConfigDiff)]
|
|
public class ConcelierConfigDiffTests : ConfigDiffTestBase
|
|
{
|
|
[Fact]
|
|
public async Task ChangingCacheTimeout_OnlyAffectsCacheBehavior()
|
|
{
|
|
var baselineConfig = new ConcelierOptions
|
|
{
|
|
CacheTimeoutMinutes = 30,
|
|
MaxConcurrentDownloads = 10
|
|
};
|
|
|
|
var changedConfig = baselineConfig with
|
|
{
|
|
CacheTimeoutMinutes = 60
|
|
};
|
|
|
|
var result = await TestConfigIsolationAsync(
|
|
baselineConfig,
|
|
changedConfig,
|
|
changedSetting: "CacheTimeoutMinutes",
|
|
unrelatedBehaviors:
|
|
[
|
|
async config => await GetDownloadBehavior(config),
|
|
async config => await GetParseBehavior(config),
|
|
async config => await GetMergeBehavior(config)
|
|
]);
|
|
|
|
Assert.True(result.IsSuccess,
|
|
$"Unexpected changes: {string.Join(", ", result.UnexpectedChanges)}");
|
|
}
|
|
|
|
[Fact]
|
|
public async Task ChangingRetryPolicy_ProducesExpectedDelta()
|
|
{
|
|
var baseline = new ConcelierOptions { MaxRetries = 3 };
|
|
var changed = new ConcelierOptions { MaxRetries = 5 };
|
|
|
|
var expectedDelta = new ConfigDelta(
|
|
ChangedBehaviors: ["RetryCount", "TotalRequestTime"],
|
|
BehaviorDeltas:
|
|
[
|
|
new BehaviorDelta("RetryCount", "3", "5", null),
|
|
new BehaviorDelta("TotalRequestTime", "increase", null,
|
|
"More retries = longer total time")
|
|
]);
|
|
|
|
var result = await TestConfigBehavioralDeltaAsync(
|
|
baseline,
|
|
changed,
|
|
getBehavior: async config => await CaptureRetryBehavior(config),
|
|
computeDelta: ComputeBehaviorSnapshotDelta,
|
|
expectedDelta: expectedDelta);
|
|
|
|
Assert.True(result.IsSuccess);
|
|
}
|
|
}
|
|
```
|
|
|
|
### Behavior Snapshots
|
|
|
|
Capture behavior at specific configuration states:
|
|
|
|
```csharp
|
|
var snapshot = CreateSnapshotBuilder("baseline-config")
|
|
.AddBehavior("CacheHitRate", cacheMetrics.HitRate)
|
|
.AddBehavior("ResponseTime", responseMetrics.P99)
|
|
.AddBehavior("ErrorRate", errorMetrics.Rate)
|
|
.WithCapturedAt(DateTimeOffset.UtcNow)
|
|
.Build();
|
|
```
|
|
|
|
---
|
|
|
|
## CI Workflows
|
|
|
|
### Available Workflows
|
|
|
|
| Workflow | File | Trigger |
|
|
|----------|------|---------|
|
|
| Blast-Radius Validation | `test-blast-radius.yml` | PRs with test changes |
|
|
| Dead-Path Detection | `dead-path-detection.yml` | Push to main, PRs |
|
|
| Schema Evolution | `schema-evolution.yml` | Schema/migration changes |
|
|
| Rollback Lag | `rollback-lag.yml` | Manual trigger, weekly |
|
|
| Test Infrastructure | `test-infrastructure.yml` | All changes, nightly |
|
|
|
|
### Workflow Outputs
|
|
|
|
Each workflow posts results as PR comments:
|
|
|
|
```markdown
|
|
## Test Infrastructure :white_check_mark: All checks passed
|
|
|
|
| Check | Status | Details |
|
|
|-------|--------|---------|
|
|
| Blast-Radius | :white_check_mark: | 0 violations |
|
|
| Dead-Path Detection | :white_check_mark: | Coverage: 82.5% |
|
|
| Schema Evolution | :white_check_mark: | Compatible: N-1,N-2 |
|
|
| Config-Diff | :white_check_mark: | Tested: Concelier,Authority,Scanner |
|
|
```
|
|
|
|
### Running Locally
|
|
|
|
```bash
|
|
# Blast-radius validation
|
|
dotnet test --filter "Category=Integration" | grep BlastRadius
|
|
|
|
# Dead-path detection
|
|
dotnet test /p:CollectCoverage=true /p:CoverletOutputFormat=cobertura
|
|
|
|
# Schema evolution (requires Docker)
|
|
docker-compose -f devops/compose/schema-test.yml up -d
|
|
dotnet test --filter "Category=SchemaEvolution"
|
|
|
|
# Config-diff
|
|
dotnet test --filter "Category=ConfigDiff"
|
|
```
|
|
|
|
---
|
|
|
|
## Best Practices
|
|
|
|
### General Guidelines
|
|
|
|
1. **Test categories**: Always categorize tests correctly
|
|
- Unit tests: Pure logic, no I/O
|
|
- Integration tests: Database, network, external systems
|
|
- Contract tests: API contracts, schemas
|
|
- Security tests: Authentication, authorization, injection
|
|
|
|
2. **Blast-radius**: Choose the narrowest applicable category
|
|
- If a test affects Auth only, use `BlastRadius.Auth`
|
|
- If it affects Auth and Api, use both
|
|
|
|
3. **Schema evolution**: Test both read and write paths
|
|
- Read compatibility: Old data readable by new code
|
|
- Write compatibility: New code writes valid old-schema data
|
|
|
|
4. **Dead-path exemptions**: Document thoroughly
|
|
- Include justification
|
|
- Set owner and review date
|
|
- Remove when no longer applicable
|
|
|
|
5. **Config-diff**: Focus on high-impact options
|
|
- Security-related configs
|
|
- Performance-related configs
|
|
- Feature flags
|
|
|
|
### Code Review Checklist
|
|
|
|
- [ ] Integration/Contract/Security tests have BlastRadius annotations
|
|
- [ ] Schema changes include evolution tests
|
|
- [ ] New branches have test coverage
|
|
- [ ] Config option tests verify isolation
|
|
- [ ] Exemptions have justifications
|
|
|
|
### Troubleshooting
|
|
|
|
**Blast-radius validation fails:**
|
|
```bash
|
|
# Find tests missing BlastRadius
|
|
dotnet test --filter "Category=Integration" --list-tests | \
|
|
xargs -I {} grep -L "BlastRadius" {}
|
|
```
|
|
|
|
**Dead-path baseline drift:**
|
|
```bash
|
|
# Regenerate baseline
|
|
dotnet test /p:CollectCoverage=true
|
|
python extract-dead-paths.py coverage.cobertura.xml
|
|
cp dead-paths-report.json dead-paths-baseline.json
|
|
```
|
|
|
|
**Schema evolution test fails:**
|
|
```bash
|
|
# Check schema version compatibility
|
|
docker run -it ghcr.io/stellaops/schema-test:scanner-v2.0.0 \
|
|
psql -U stellaops_test -d stellaops_schema_test \
|
|
-c "SELECT * FROM _schema_metadata;"
|
|
```
|
|
|
|
---
|
|
|
|
## Related Documentation
|
|
|
|
- [Test Infrastructure Overview](../testing/README.md)
|
|
- [Database Schema Specification](../db/SPECIFICATION.md)
|
|
- [CI/CD Workflows](../../.gitea/workflows/README.md)
|
|
- [Module Testing Agents](../../src/__Tests/AGENTS.md)
|