Merge branch 'main' of https://git.stella-ops.org/stella-ops.org/git.stella-ops.org
This commit is contained in:
master
501
docs/technical/testing/cross-cutting-testing-guide.md
Normal file
501
docs/technical/testing/cross-cutting-testing-guide.md
Normal file
@@ -0,0 +1,501 @@
|
||||
# Cross-Cutting Testing Standards Guide
|
||||
|
||||
This guide documents the cross-cutting testing standards implemented for StellaOps, including blast-radius annotations, schema evolution testing, dead-path detection, and config-diff testing.
|
||||
|
||||
**Sprint Reference:** SPRINT_20260105_002_005_TEST_cross_cutting
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [Overview](#overview)
|
||||
2. [Blast-Radius Annotations](#blast-radius-annotations)
|
||||
3. [Schema Evolution Testing](#schema-evolution-testing)
|
||||
4. [Dead-Path Detection](#dead-path-detection)
|
||||
5. [Config-Diff Testing](#config-diff-testing)
|
||||
6. [CI Workflows](#ci-workflows)
|
||||
7. [Best Practices](#best-practices)
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Cross-cutting testing standards ensure consistent test quality across all modules:
|
||||
|
||||
| Standard | Purpose | Enforcement |
|
||||
|----------|---------|-------------|
|
||||
| **Blast-Radius** | Categorize tests by operational surface | CI validation on PRs |
|
||||
| **Schema Evolution** | Verify backward compatibility | CI on schema changes |
|
||||
| **Dead-Path Detection** | Identify uncovered code | CI with baseline comparison |
|
||||
| **Config-Diff** | Validate config behavioral isolation | Integration tests |
|
||||
|
||||
---
|
||||
|
||||
## Blast-Radius Annotations
|
||||
|
||||
### Purpose
|
||||
|
||||
Blast-radius annotations categorize tests by the operational surfaces they affect. During incidents, this enables targeted test runs for specific areas (e.g., run only Auth-related tests when investigating an authentication issue).
|
||||
|
||||
### Categories
|
||||
|
||||
| Category | Description | Examples |
|
||||
|----------|-------------|----------|
|
||||
| `Auth` | Authentication, authorization, tokens | Login, OAuth, DPoP |
|
||||
| `Scanning` | SBOM generation, vulnerability scanning | Scanner, analyzers |
|
||||
| `Evidence` | Attestation, evidence storage | EvidenceLocker, Attestor |
|
||||
| `Compliance` | Audit, regulatory, GDPR | Compliance reports |
|
||||
| `Advisories` | Advisory ingestion, VEX processing | Concelier, VexLens |
|
||||
| `RiskPolicy` | Risk scoring, policy evaluation | RiskEngine, Policy |
|
||||
| `Crypto` | Cryptographic operations | Signing, verification |
|
||||
| `Integrations` | External systems, webhooks | Notifications, webhooks |
|
||||
| `Persistence` | Database operations | Repositories, migrations |
|
||||
| `Api` | API surface, contracts | Controllers, endpoints |
|
||||
|
||||
### Usage
|
||||
|
||||
```csharp
|
||||
using StellaOps.TestKit;
|
||||
using Xunit;
|
||||
|
||||
// Single blast-radius
|
||||
[Trait("Category", TestCategories.Integration)]
|
||||
[Trait("BlastRadius", TestCategories.BlastRadius.Auth)]
|
||||
public class TokenValidationTests
|
||||
{
|
||||
[Fact]
|
||||
public async Task ValidToken_ReturnsSuccess()
|
||||
{
|
||||
// Test implementation
|
||||
}
|
||||
}
|
||||
|
||||
// Multiple blast-radii (affects multiple surfaces)
|
||||
[Trait("Category", TestCategories.Integration)]
|
||||
[Trait("BlastRadius", TestCategories.BlastRadius.Auth)]
|
||||
[Trait("BlastRadius", TestCategories.BlastRadius.Api)]
|
||||
public class AuthenticatedApiTests
|
||||
{
|
||||
// Tests that affect both Auth and Api surfaces
|
||||
}
|
||||
```
|
||||
|
||||
### Requirements
|
||||
|
||||
- **Integration tests**: Must have at least one BlastRadius annotation
|
||||
- **Contract tests**: Must have at least one BlastRadius annotation
|
||||
- **Security tests**: Must have at least one BlastRadius annotation
|
||||
- **Unit tests**: BlastRadius optional but recommended
|
||||
|
||||
### Running Tests by Blast-Radius
|
||||
|
||||
```bash
|
||||
# Run all Auth-related tests
|
||||
dotnet test --filter "BlastRadius=Auth"
|
||||
|
||||
# Run tests for multiple surfaces
|
||||
dotnet test --filter "BlastRadius=Auth|BlastRadius=Api"
|
||||
|
||||
# Run incident response test suite
|
||||
dotnet run --project src/__Libraries/StellaOps.TestKit \
|
||||
-- run-blast-radius Auth,Api --fail-fast
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Schema Evolution Testing
|
||||
|
||||
### Purpose
|
||||
|
||||
Schema evolution tests verify that code remains compatible with previous database schema versions. This prevents breaking changes during:
|
||||
|
||||
- Rolling deployments (new code, old schema)
|
||||
- Rollbacks (old code, new schema)
|
||||
- Migration windows
|
||||
|
||||
### Schema Versions
|
||||
|
||||
| Version | Description |
|
||||
|---------|-------------|
|
||||
| `N` | Current schema (HEAD) |
|
||||
| `N-1` | Previous schema version |
|
||||
| `N-2` | Two versions back |
|
||||
|
||||
### Using SchemaEvolutionTestBase
|
||||
|
||||
```csharp
|
||||
using StellaOps.Testing.SchemaEvolution;
|
||||
using Testcontainers.PostgreSql;
|
||||
using Xunit;
|
||||
|
||||
[Trait("Category", TestCategories.SchemaEvolution)]
|
||||
public class ScannerSchemaEvolutionTests : PostgresSchemaEvolutionTestBase
|
||||
{
|
||||
public ScannerSchemaEvolutionTests()
|
||||
: base(new SchemaEvolutionConfig
|
||||
{
|
||||
ModuleName = "Scanner",
|
||||
CurrentVersion = new SchemaVersion("v2.1.0",
|
||||
DateTimeOffset.Parse("2026-01-01")),
|
||||
PreviousVersions =
|
||||
[
|
||||
new SchemaVersion("v2.0.0",
|
||||
DateTimeOffset.Parse("2025-10-01")),
|
||||
new SchemaVersion("v1.9.0",
|
||||
DateTimeOffset.Parse("2025-07-01"))
|
||||
],
|
||||
ConnectionStringTemplate =
|
||||
"Host={0};Port={1};Database={2};Username={3};Password={4}"
|
||||
})
|
||||
{
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ReadOperations_CompatibleWithPreviousSchema()
|
||||
{
|
||||
var result = await TestReadBackwardCompatibilityAsync(
|
||||
async (connection, version) =>
|
||||
{
|
||||
// Test read operations against old schema
|
||||
var repository = new ScanRepository(connection);
|
||||
var scans = await repository.GetRecentScansAsync(10);
|
||||
return scans.Count >= 0;
|
||||
});
|
||||
|
||||
Assert.True(result.IsSuccess);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task WriteOperations_CompatibleWithPreviousSchema()
|
||||
{
|
||||
var result = await TestWriteForwardCompatibilityAsync(
|
||||
async (connection, version) =>
|
||||
{
|
||||
// Test write operations
|
||||
var repository = new ScanRepository(connection);
|
||||
await repository.CreateScanAsync(new ScanRequest { /* ... */ });
|
||||
return true;
|
||||
});
|
||||
|
||||
Assert.True(result.IsSuccess);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Versioned Container Images
|
||||
|
||||
Build versioned PostgreSQL images for testing:
|
||||
|
||||
```bash
|
||||
# Build all versions for a module
|
||||
./devops/docker/schema-versions/build-schema-images.sh scanner
|
||||
|
||||
# Build specific version
|
||||
./devops/docker/schema-versions/build-schema-images.sh scanner v2.0.0
|
||||
|
||||
# Use in tests
|
||||
docker run -d -p 5432:5432 ghcr.io/stellaops/schema-test:scanner-v2.0.0
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Dead-Path Detection
|
||||
|
||||
### Purpose
|
||||
|
||||
Dead-path detection identifies uncovered code branches. This helps:
|
||||
|
||||
- Find untested edge cases
|
||||
- Identify potentially dead code
|
||||
- Prevent coverage regression
|
||||
|
||||
### How It Works
|
||||
|
||||
1. Tests run with branch coverage collection (Coverlet)
|
||||
2. Cobertura XML report is parsed
|
||||
3. Uncovered branches are identified
|
||||
4. New dead paths are compared against baseline
|
||||
5. CI fails if new dead paths are introduced
|
||||
|
||||
### Baseline Management
|
||||
|
||||
The baseline file (`dead-paths-baseline.json`) tracks known dead paths:
|
||||
|
||||
```json
|
||||
{
|
||||
"version": "1.0.0",
|
||||
"activeDeadPaths": 42,
|
||||
"totalDeadPaths": 50,
|
||||
"exemptedPaths": 8,
|
||||
"entries": [
|
||||
{
|
||||
"file": "src/Scanner/Services/AnalyzerService.cs",
|
||||
"line": 128,
|
||||
"coverage": "1/2",
|
||||
"isExempt": false
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Exemptions
|
||||
|
||||
Add exemptions for intentionally untested code in `coverage-exemptions.yaml`:
|
||||
|
||||
```yaml
|
||||
exemptions:
|
||||
- path: "src/Authority/Emergency/BreakGlassHandler.cs:42"
|
||||
category: emergency
|
||||
justification: "Emergency access bypass - tested in incident drills"
|
||||
added: "2026-01-06"
|
||||
owner: "security-team"
|
||||
|
||||
- path: "src/Scanner/Platform/WindowsRegistryScanner.cs:*"
|
||||
category: platform
|
||||
justification: "Windows-only code - CI runs on Linux"
|
||||
added: "2026-01-06"
|
||||
owner: "scanner-team"
|
||||
|
||||
ignore_patterns:
|
||||
- "*.Generated.cs"
|
||||
- "**/Migrations/*.cs"
|
||||
```
|
||||
|
||||
### Using BranchCoverageEnforcer
|
||||
|
||||
```csharp
|
||||
using StellaOps.Testing.Coverage;
|
||||
|
||||
var enforcer = new BranchCoverageEnforcer(new BranchCoverageConfig
|
||||
{
|
||||
MinimumBranchCoverage = 80,
|
||||
FailOnNewDeadPaths = true,
|
||||
ExemptionFiles = ["coverage-exemptions.yaml"]
|
||||
});
|
||||
|
||||
// Parse coverage report
|
||||
var parser = new CoberturaParser();
|
||||
var coverage = await parser.ParseFileAsync("coverage.cobertura.xml");
|
||||
|
||||
// Validate
|
||||
var result = enforcer.Validate(coverage);
|
||||
if (!result.IsValid)
|
||||
{
|
||||
foreach (var violation in result.Violations)
|
||||
{
|
||||
Console.WriteLine($"Violation: {violation.File}:{violation.Line}");
|
||||
}
|
||||
}
|
||||
|
||||
// Generate dead-path report
|
||||
var report = enforcer.GenerateDeadPathReport(coverage);
|
||||
Console.WriteLine($"Active dead paths: {report.ActiveDeadPaths}");
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Config-Diff Testing
|
||||
|
||||
### Purpose
|
||||
|
||||
Config-diff tests verify that configuration changes produce only expected behavioral deltas. This prevents:
|
||||
|
||||
- Unintended side effects from config changes
|
||||
- Config options affecting unrelated behaviors
|
||||
- Regressions in config handling
|
||||
|
||||
### Using ConfigDiffTestBase
|
||||
|
||||
```csharp
|
||||
using StellaOps.Testing.ConfigDiff;
|
||||
using Xunit;
|
||||
|
||||
[Trait("Category", TestCategories.ConfigDiff)]
|
||||
public class ConcelierConfigDiffTests : ConfigDiffTestBase
|
||||
{
|
||||
[Fact]
|
||||
public async Task ChangingCacheTimeout_OnlyAffectsCacheBehavior()
|
||||
{
|
||||
var baselineConfig = new ConcelierOptions
|
||||
{
|
||||
CacheTimeoutMinutes = 30,
|
||||
MaxConcurrentDownloads = 10
|
||||
};
|
||||
|
||||
var changedConfig = baselineConfig with
|
||||
{
|
||||
CacheTimeoutMinutes = 60
|
||||
};
|
||||
|
||||
var result = await TestConfigIsolationAsync(
|
||||
baselineConfig,
|
||||
changedConfig,
|
||||
changedSetting: "CacheTimeoutMinutes",
|
||||
unrelatedBehaviors:
|
||||
[
|
||||
async config => await GetDownloadBehavior(config),
|
||||
async config => await GetParseBehavior(config),
|
||||
async config => await GetMergeBehavior(config)
|
||||
]);
|
||||
|
||||
Assert.True(result.IsSuccess,
|
||||
$"Unexpected changes: {string.Join(", ", result.UnexpectedChanges)}");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ChangingRetryPolicy_ProducesExpectedDelta()
|
||||
{
|
||||
var baseline = new ConcelierOptions { MaxRetries = 3 };
|
||||
var changed = new ConcelierOptions { MaxRetries = 5 };
|
||||
|
||||
var expectedDelta = new ConfigDelta(
|
||||
ChangedBehaviors: ["RetryCount", "TotalRequestTime"],
|
||||
BehaviorDeltas:
|
||||
[
|
||||
new BehaviorDelta("RetryCount", "3", "5", null),
|
||||
new BehaviorDelta("TotalRequestTime", "increase", null,
|
||||
"More retries = longer total time")
|
||||
]);
|
||||
|
||||
var result = await TestConfigBehavioralDeltaAsync(
|
||||
baseline,
|
||||
changed,
|
||||
getBehavior: async config => await CaptureRetryBehavior(config),
|
||||
computeDelta: ComputeBehaviorSnapshotDelta,
|
||||
expectedDelta: expectedDelta);
|
||||
|
||||
Assert.True(result.IsSuccess);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Behavior Snapshots
|
||||
|
||||
Capture behavior at specific configuration states:
|
||||
|
||||
```csharp
|
||||
var snapshot = CreateSnapshotBuilder("baseline-config")
|
||||
.AddBehavior("CacheHitRate", cacheMetrics.HitRate)
|
||||
.AddBehavior("ResponseTime", responseMetrics.P99)
|
||||
.AddBehavior("ErrorRate", errorMetrics.Rate)
|
||||
.WithCapturedAt(DateTimeOffset.UtcNow)
|
||||
.Build();
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## CI Workflows
|
||||
|
||||
### Available Workflows
|
||||
|
||||
| Workflow | File | Trigger |
|
||||
|----------|------|---------|
|
||||
| Blast-Radius Validation | `test-blast-radius.yml` | PRs with test changes |
|
||||
| Dead-Path Detection | `dead-path-detection.yml` | Push to main, PRs |
|
||||
| Schema Evolution | `schema-evolution.yml` | Schema/migration changes |
|
||||
| Rollback Lag | `rollback-lag.yml` | Manual trigger, weekly |
|
||||
| Test Infrastructure | `test-infrastructure.yml` | All changes, nightly |
|
||||
|
||||
### Workflow Outputs
|
||||
|
||||
Each workflow posts results as PR comments:
|
||||
|
||||
```markdown
|
||||
## Test Infrastructure :white_check_mark: All checks passed
|
||||
|
||||
| Check | Status | Details |
|
||||
|-------|--------|---------|
|
||||
| Blast-Radius | :white_check_mark: | 0 violations |
|
||||
| Dead-Path Detection | :white_check_mark: | Coverage: 82.5% |
|
||||
| Schema Evolution | :white_check_mark: | Compatible: N-1,N-2 |
|
||||
| Config-Diff | :white_check_mark: | Tested: Concelier,Authority,Scanner |
|
||||
```
|
||||
|
||||
### Running Locally
|
||||
|
||||
```bash
|
||||
# Blast-radius validation
|
||||
dotnet test --filter "Category=Integration" | grep BlastRadius
|
||||
|
||||
# Dead-path detection
|
||||
dotnet test /p:CollectCoverage=true /p:CoverletOutputFormat=cobertura
|
||||
|
||||
# Schema evolution (requires Docker)
|
||||
docker-compose -f devops/compose/schema-test.yml up -d
|
||||
dotnet test --filter "Category=SchemaEvolution"
|
||||
|
||||
# Config-diff
|
||||
dotnet test --filter "Category=ConfigDiff"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Best Practices
|
||||
|
||||
### General Guidelines
|
||||
|
||||
1. **Test categories**: Always categorize tests correctly
|
||||
- Unit tests: Pure logic, no I/O
|
||||
- Integration tests: Database, network, external systems
|
||||
- Contract tests: API contracts, schemas
|
||||
- Security tests: Authentication, authorization, injection
|
||||
|
||||
2. **Blast-radius**: Choose the narrowest applicable category
|
||||
- If a test affects Auth only, use `BlastRadius.Auth`
|
||||
- If it affects Auth and Api, use both
|
||||
|
||||
3. **Schema evolution**: Test both read and write paths
|
||||
- Read compatibility: Old data readable by new code
|
||||
- Write compatibility: New code writes valid old-schema data
|
||||
|
||||
4. **Dead-path exemptions**: Document thoroughly
|
||||
- Include justification
|
||||
- Set owner and review date
|
||||
- Remove when no longer applicable
|
||||
|
||||
5. **Config-diff**: Focus on high-impact options
|
||||
- Security-related configs
|
||||
- Performance-related configs
|
||||
- Feature flags
|
||||
|
||||
### Code Review Checklist
|
||||
|
||||
- [ ] Integration/Contract/Security tests have BlastRadius annotations
|
||||
- [ ] Schema changes include evolution tests
|
||||
- [ ] New branches have test coverage
|
||||
- [ ] Config option tests verify isolation
|
||||
- [ ] Exemptions have justifications
|
||||
|
||||
### Troubleshooting
|
||||
|
||||
**Blast-radius validation fails:**
|
||||
```bash
|
||||
# Find tests missing BlastRadius
|
||||
dotnet test --filter "Category=Integration" --list-tests | \
|
||||
xargs -I {} grep -L "BlastRadius" {}
|
||||
```
|
||||
|
||||
**Dead-path baseline drift:**
|
||||
```bash
|
||||
# Regenerate baseline
|
||||
dotnet test /p:CollectCoverage=true
|
||||
python extract-dead-paths.py coverage.cobertura.xml
|
||||
cp dead-paths-report.json dead-paths-baseline.json
|
||||
```
|
||||
|
||||
**Schema evolution test fails:**
|
||||
```bash
|
||||
# Check schema version compatibility
|
||||
docker run -it ghcr.io/stellaops/schema-test:scanner-v2.0.0 \
|
||||
psql -U stellaops_test -d stellaops_schema_test \
|
||||
-c "SELECT * FROM _schema_metadata;"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Test Infrastructure Overview](../testing/README.md)
|
||||
- [Database Schema Specification](../db/SPECIFICATION.md)
|
||||
- [CI/CD Workflows](../../.gitea/workflows/README.md)
|
||||
- [Module Testing Agents](../../src/__Tests/AGENTS.md)
|
||||
Reference in New Issue
Block a user