Merge branch 'main' of https://git.stella-ops.org/stella-ops.org/git.stella-ops.org
This commit is contained in:
StellaOps Bot
646
docs/technical/testing/DETERMINISM_DEVELOPER_GUIDE.md
Normal file
646
docs/technical/testing/DETERMINISM_DEVELOPER_GUIDE.md
Normal file
@@ -0,0 +1,646 @@
|
||||
# Determinism Developer Guide
|
||||
|
||||
## Overview
|
||||
|
||||
This guide helps developers add new determinism tests to StellaOps. Deterministic behavior is critical for:
|
||||
- Reproducible verdicts
|
||||
- Auditable evidence chains
|
||||
- Cryptographic verification
|
||||
- Cross-platform consistency
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [Core Principles](#core-principles)
|
||||
2. [Test Structure](#test-structure)
|
||||
3. [Common Patterns](#common-patterns)
|
||||
4. [Anti-Patterns to Avoid](#anti-patterns-to-avoid)
|
||||
5. [Adding New Tests](#adding-new-tests)
|
||||
6. [Cross-Platform Considerations](#cross-platform-considerations)
|
||||
7. [Performance Guidelines](#performance-guidelines)
|
||||
8. [Troubleshooting](#troubleshooting)
|
||||
|
||||
## Core Principles
|
||||
|
||||
### 1. Determinism Guarantee
|
||||
|
||||
**Definition**: Same inputs always produce identical outputs, regardless of:
|
||||
- Platform (Windows, macOS, Linux, Alpine, Debian)
|
||||
- Runtime (.NET version, JIT compiler)
|
||||
- Execution order (parallel vs sequential)
|
||||
- Time of day
|
||||
- System locale
|
||||
|
||||
### 2. Golden File Philosophy
|
||||
|
||||
**Golden files** are baseline reference values that lock in correct behavior:
|
||||
- Established after careful verification
|
||||
- Never changed without ADR and migration plan
|
||||
- Verified on all platforms before acceptance
|
||||
|
||||
### 3. Test Independence
|
||||
|
||||
Each test must:
|
||||
- Not depend on other tests' execution or order
|
||||
- Clean up resources after completion
|
||||
- Use isolated data (no shared state)
|
||||
|
||||
## Test Structure
|
||||
|
||||
### Standard Test Template
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
[Trait("Category", TestCategories.Determinism)]
|
||||
[Trait("Category", TestCategories.Unit)]
|
||||
public async Task Feature_Behavior_ExpectedOutcome()
|
||||
{
|
||||
// Arrange - Create deterministic inputs
|
||||
var input = CreateDeterministicInput();
|
||||
|
||||
// Act - Execute feature
|
||||
var output1 = await ExecuteFeature(input);
|
||||
var output2 = await ExecuteFeature(input);
|
||||
|
||||
// Assert - Verify determinism
|
||||
output1.Should().Be(output2, "same input should produce identical output");
|
||||
}
|
||||
```
|
||||
|
||||
### Test Organization
|
||||
|
||||
```
|
||||
src/__Tests/Determinism/
|
||||
├── CgsDeterminismTests.cs # CGS hash tests
|
||||
├── LineageDeterminismTests.cs # SBOM lineage tests
|
||||
├── VexDeterminismTests.cs # VEX consensus tests (future)
|
||||
├── README.md # Test documentation
|
||||
└── Fixtures/ # Test data
|
||||
├── known-evidence-pack.json
|
||||
├── known-policy-lock.json
|
||||
└── golden-hashes/
|
||||
└── cgs-v1.txt
|
||||
```
|
||||
|
||||
## Common Patterns
|
||||
|
||||
### Pattern 1: 10-Iteration Stability Test
|
||||
|
||||
**Purpose**: Verify that executing the same operation 10 times produces identical results.
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
[Trait("Category", TestCategories.Determinism)]
|
||||
public async Task Feature_SameInput_ProducesIdenticalOutput_Across10Iterations()
|
||||
{
|
||||
// Arrange
|
||||
var input = CreateDeterministicInput();
|
||||
var service = CreateService();
|
||||
var outputs = new List<string>();
|
||||
|
||||
// Act - Execute 10 times
|
||||
for (int i = 0; i < 10; i++)
|
||||
{
|
||||
var result = await service.ProcessAsync(input, CancellationToken.None);
|
||||
outputs.Add(result.Hash);
|
||||
_output.WriteLine($"Iteration {i + 1}: {result.Hash}");
|
||||
}
|
||||
|
||||
// Assert - All hashes should be identical
|
||||
outputs.Distinct().Should().HaveCount(1,
|
||||
"same input should produce identical output across all iterations");
|
||||
}
|
||||
```
|
||||
|
||||
**Why 10 iterations?**
|
||||
- Catches non-deterministic behavior (e.g., GUID generation, random values)
|
||||
- Reasonable execution time (<5 seconds for most tests)
|
||||
- Industry standard for determinism verification
|
||||
|
||||
### Pattern 2: Golden File Test
|
||||
|
||||
**Purpose**: Verify output matches a known-good baseline value.
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
[Trait("Category", TestCategories.Determinism)]
|
||||
[Trait("Category", TestCategories.Golden)]
|
||||
public async Task Feature_WithKnownInput_MatchesGoldenHash()
|
||||
{
|
||||
// Arrange
|
||||
var input = CreateKnownInput(); // MUST be completely deterministic
|
||||
var service = CreateService();
|
||||
|
||||
// Act
|
||||
var result = await service.ProcessAsync(input, CancellationToken.None);
|
||||
|
||||
// Assert
|
||||
var goldenHash = "sha256:d4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3";
|
||||
|
||||
_output.WriteLine($"Computed Hash: {result.Hash}");
|
||||
_output.WriteLine($"Golden Hash: {goldenHash}");
|
||||
|
||||
result.Hash.Should().Be(goldenHash, "hash must match golden file");
|
||||
}
|
||||
```
|
||||
|
||||
**Golden file best practices:**
|
||||
- Document how golden value was established (date, platform, .NET version)
|
||||
- Include golden value directly in test code (not external file) for visibility
|
||||
- Add comment explaining what golden value represents
|
||||
- Test golden value on all platforms before merging
|
||||
|
||||
### Pattern 3: Order Independence Test
|
||||
|
||||
**Purpose**: Verify that input ordering doesn't affect output.
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
[Trait("Category", TestCategories.Determinism)]
|
||||
public async Task Feature_InputOrder_DoesNotAffectOutput()
|
||||
{
|
||||
// Arrange
|
||||
var item1 = CreateItem("A");
|
||||
var item2 = CreateItem("B");
|
||||
var item3 = CreateItem("C");
|
||||
|
||||
var service = CreateService();
|
||||
|
||||
// Act - Process items in different orders
|
||||
var result1 = await service.ProcessAsync(new[] { item1, item2, item3 }, CancellationToken.None);
|
||||
var result2 = await service.ProcessAsync(new[] { item3, item1, item2 }, CancellationToken.None);
|
||||
var result3 = await service.ProcessAsync(new[] { item2, item3, item1 }, CancellationToken.None);
|
||||
|
||||
// Assert - All should produce same hash
|
||||
result1.Hash.Should().Be(result2.Hash, "input order should not affect output");
|
||||
result1.Hash.Should().Be(result3.Hash, "input order should not affect output");
|
||||
|
||||
_output.WriteLine($"Order-independent hash: {result1.Hash}");
|
||||
}
|
||||
```
|
||||
|
||||
**When to use:**
|
||||
- Collections that should be sorted internally (VEX documents, rules, dependencies)
|
||||
- APIs that accept unordered inputs (dictionary keys, sets)
|
||||
- Parallel processing where order is undefined
|
||||
|
||||
### Pattern 4: Deterministic Timestamp Test
|
||||
|
||||
**Purpose**: Verify that fixed timestamps produce deterministic results.
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
[Trait("Category", TestCategories.Determinism)]
|
||||
public async Task Feature_WithFixedTimestamp_IsDeterministic()
|
||||
{
|
||||
// Arrange - Use FIXED timestamp (not DateTimeOffset.Now!)
|
||||
var timestamp = DateTimeOffset.Parse("2025-01-01T00:00:00Z");
|
||||
var input = CreateInputWithTimestamp(timestamp);
|
||||
var service = CreateService();
|
||||
|
||||
// Act
|
||||
var result1 = await service.ProcessAsync(input, CancellationToken.None);
|
||||
var result2 = await service.ProcessAsync(input, CancellationToken.None);
|
||||
|
||||
// Assert
|
||||
result1.Hash.Should().Be(result2.Hash, "fixed timestamp should produce deterministic output");
|
||||
}
|
||||
```
|
||||
|
||||
**Timestamp guidelines:**
|
||||
- ❌ **Never use**: `DateTimeOffset.Now`, `DateTime.UtcNow`, `Guid.NewGuid()`
|
||||
- ✅ **Always use**: `DateTimeOffset.Parse("2025-01-01T00:00:00Z")` for tests
|
||||
|
||||
### Pattern 5: Empty/Minimal Input Test
|
||||
|
||||
**Purpose**: Verify that minimal or empty inputs don't cause non-determinism.
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
[Trait("Category", TestCategories.Determinism)]
|
||||
public async Task Feature_EmptyInput_ProducesDeterministicHash()
|
||||
{
|
||||
// Arrange - Minimal input
|
||||
var input = CreateEmptyInput();
|
||||
var service = CreateService();
|
||||
|
||||
// Act
|
||||
var result = await service.ProcessAsync(input, CancellationToken.None);
|
||||
|
||||
// Assert - Verify format (hash may not be golden yet)
|
||||
result.Hash.Should().StartWith("sha256:");
|
||||
result.Hash.Length.Should().Be(71); // "sha256:" + 64 hex chars
|
||||
|
||||
_output.WriteLine($"Empty input hash: {result.Hash}");
|
||||
}
|
||||
```
|
||||
|
||||
**Edge cases to test:**
|
||||
- Empty collections (`Array.Empty<string>()`)
|
||||
- Null optional fields
|
||||
- Zero-length strings
|
||||
- Default values
|
||||
|
||||
## Anti-Patterns to Avoid
|
||||
|
||||
### ❌ Anti-Pattern 1: Using Current Time
|
||||
|
||||
```csharp
|
||||
// BAD - Non-deterministic!
|
||||
var input = new Input
|
||||
{
|
||||
Timestamp = DateTimeOffset.Now // ❌ Different every run!
|
||||
};
|
||||
```
|
||||
|
||||
**Fix:**
|
||||
```csharp
|
||||
// GOOD - Deterministic
|
||||
var input = new Input
|
||||
{
|
||||
Timestamp = DateTimeOffset.Parse("2025-01-01T00:00:00Z") // ✅ Same every run
|
||||
};
|
||||
```
|
||||
|
||||
### ❌ Anti-Pattern 2: Using Random Values
|
||||
|
||||
```csharp
|
||||
// BAD - Non-deterministic!
|
||||
var random = new Random();
|
||||
var input = new Input
|
||||
{
|
||||
Id = random.Next() // ❌ Different every run!
|
||||
};
|
||||
```
|
||||
|
||||
**Fix:**
|
||||
```csharp
|
||||
// GOOD - Deterministic
|
||||
var input = new Input
|
||||
{
|
||||
Id = 12345 // ✅ Same every run
|
||||
};
|
||||
```
|
||||
|
||||
### ❌ Anti-Pattern 3: Using GUID Generation
|
||||
|
||||
```csharp
|
||||
// BAD - Non-deterministic!
|
||||
var input = new Input
|
||||
{
|
||||
Id = Guid.NewGuid().ToString() // ❌ Different every run!
|
||||
};
|
||||
```
|
||||
|
||||
**Fix:**
|
||||
```csharp
|
||||
// GOOD - Deterministic
|
||||
var input = new Input
|
||||
{
|
||||
Id = "00000000-0000-0000-0000-000000000001" // ✅ Same every run
|
||||
};
|
||||
```
|
||||
|
||||
### ❌ Anti-Pattern 4: Using Unordered Collections
|
||||
|
||||
```csharp
|
||||
// BAD - Dictionary iteration order is NOT guaranteed!
|
||||
var dict = new Dictionary<string, string>
|
||||
{
|
||||
["key1"] = "value1",
|
||||
["key2"] = "value2"
|
||||
};
|
||||
|
||||
foreach (var kvp in dict) // ❌ Order may vary!
|
||||
{
|
||||
hash.Update(kvp.Key);
|
||||
}
|
||||
```
|
||||
|
||||
**Fix:**
|
||||
```csharp
|
||||
// GOOD - Explicit ordering
|
||||
var dict = new Dictionary<string, string>
|
||||
{
|
||||
["key1"] = "value1",
|
||||
["key2"] = "value2"
|
||||
};
|
||||
|
||||
foreach (var kvp in dict.OrderBy(x => x.Key, StringComparer.Ordinal)) // ✅ Consistent order
|
||||
{
|
||||
hash.Update(kvp.Key);
|
||||
}
|
||||
```
|
||||
|
||||
### ❌ Anti-Pattern 5: Platform-Specific Paths
|
||||
|
||||
```csharp
|
||||
// BAD - Platform-specific!
|
||||
var path = "dir\\file.txt"; // ❌ Windows-only!
|
||||
```
|
||||
|
||||
**Fix:**
|
||||
```csharp
|
||||
// GOOD - Cross-platform
|
||||
var path = Path.Combine("dir", "file.txt"); // ✅ Works everywhere
|
||||
```
|
||||
|
||||
### ❌ Anti-Pattern 6: Culture-Dependent Formatting
|
||||
|
||||
```csharp
|
||||
// BAD - Culture-dependent!
|
||||
var formatted = value.ToString(); // ❌ Locale-specific!
|
||||
```
|
||||
|
||||
**Fix:**
|
||||
```csharp
|
||||
// GOOD - Culture-invariant
|
||||
var formatted = value.ToString(CultureInfo.InvariantCulture); // ✅ Same everywhere
|
||||
```
|
||||
|
||||
## Adding New Tests
|
||||
|
||||
### Step 1: Identify Determinism Requirement
|
||||
|
||||
**Ask yourself:**
|
||||
- Does this feature produce a hash, signature, or cryptographic output?
|
||||
- Will this feature's output be stored and verified later?
|
||||
- Does this feature need to be reproducible across platforms?
|
||||
- Is this feature part of an audit trail?
|
||||
|
||||
If **YES** to any → Add determinism test.
|
||||
|
||||
### Step 2: Create Test File
|
||||
|
||||
```bash
|
||||
cd src/__Tests/Determinism
|
||||
touch MyFeatureDeterminismTests.cs
|
||||
```
|
||||
|
||||
### Step 3: Write Test Class
|
||||
|
||||
```csharp
|
||||
using FluentAssertions;
|
||||
using StellaOps.TestKit;
|
||||
using Xunit;
|
||||
using Xunit.Abstractions;
|
||||
|
||||
namespace StellaOps.Tests.Determinism;
|
||||
|
||||
/// <summary>
|
||||
/// Determinism tests for [Feature Name].
|
||||
/// Verifies that [specific behavior] is deterministic across platforms and runs.
|
||||
/// </summary>
|
||||
[Trait("Category", TestCategories.Determinism)]
|
||||
[Trait("Category", TestCategories.Unit)]
|
||||
public sealed class MyFeatureDeterminismTests
|
||||
{
|
||||
private readonly ITestOutputHelper _output;
|
||||
|
||||
public MyFeatureDeterminismTests(ITestOutputHelper output)
|
||||
{
|
||||
_output = output;
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task MyFeature_SameInput_ProducesIdenticalOutput_Across10Iterations()
|
||||
{
|
||||
// Arrange
|
||||
var input = CreateDeterministicInput();
|
||||
var service = CreateMyFeatureService();
|
||||
var outputs = new List<string>();
|
||||
|
||||
// Act - Execute 10 times
|
||||
for (int i = 0; i < 10; i++)
|
||||
{
|
||||
var result = await service.ProcessAsync(input, CancellationToken.None);
|
||||
outputs.Add(result.Hash);
|
||||
_output.WriteLine($"Iteration {i + 1}: {result.Hash}");
|
||||
}
|
||||
|
||||
// Assert - All hashes should be identical
|
||||
outputs.Distinct().Should().HaveCount(1,
|
||||
"same input should produce identical output across all iterations");
|
||||
}
|
||||
|
||||
#region Helper Methods
|
||||
|
||||
private static MyInput CreateDeterministicInput()
|
||||
{
|
||||
return new MyInput
|
||||
{
|
||||
// ✅ Use fixed values
|
||||
Id = "test-001",
|
||||
Timestamp = DateTimeOffset.Parse("2025-01-01T00:00:00Z"),
|
||||
Data = new[] { "item1", "item2", "item3" }
|
||||
};
|
||||
}
|
||||
|
||||
private static MyFeatureService CreateMyFeatureService()
|
||||
{
|
||||
return new MyFeatureService(NullLogger<MyFeatureService>.Instance);
|
||||
}
|
||||
|
||||
#endregion
|
||||
}
|
||||
```
|
||||
|
||||
### Step 4: Run Test Locally 10 Times
|
||||
|
||||
```bash
|
||||
for i in {1..10}; do
|
||||
echo "=== Run $i ==="
|
||||
dotnet test --filter "FullyQualifiedName~MyFeature_SameInput_ProducesIdenticalOutput_Across10Iterations"
|
||||
done
|
||||
```
|
||||
|
||||
**Expected:** All 10 runs pass with identical output.
|
||||
|
||||
### Step 5: Add to CI/CD
|
||||
|
||||
Test is automatically included when pushed (no configuration needed).
|
||||
|
||||
CI/CD workflow `.gitea/workflows/cross-platform-determinism.yml` runs all `Category=Determinism` tests on 5 platforms.
|
||||
|
||||
### Step 6: Document in README
|
||||
|
||||
Update `src/__Tests/Determinism/README.md`:
|
||||
|
||||
```markdown
|
||||
### MyFeature Determinism
|
||||
|
||||
Tests that verify [feature] hash computation is deterministic:
|
||||
|
||||
- **10-Iteration Stability**: Same input produces identical hash 10 times
|
||||
- **Order Independence**: Input ordering doesn't affect hash
|
||||
- **Empty Input**: Minimal input produces deterministic hash
|
||||
```
|
||||
|
||||
## Cross-Platform Considerations
|
||||
|
||||
### Platform Matrix
|
||||
|
||||
Tests run on:
|
||||
- **Windows** (windows-latest): glibc, CRLF line endings
|
||||
- **macOS** (macos-latest): BSD libc, LF line endings
|
||||
- **Linux Ubuntu** (ubuntu-latest): glibc, LF line endings
|
||||
- **Linux Alpine** (Alpine Docker): musl libc, LF line endings
|
||||
- **Linux Debian** (Debian Docker): glibc, LF line endings
|
||||
|
||||
### Common Cross-Platform Issues
|
||||
|
||||
#### Issue 1: String Sorting (musl vs glibc)
|
||||
|
||||
**Symptom**: Alpine produces different hash than Ubuntu.
|
||||
|
||||
**Cause**: `musl` libc has different `strcoll` implementation than `glibc`.
|
||||
|
||||
**Solution**: Always use `StringComparer.Ordinal` for sorting:
|
||||
|
||||
```csharp
|
||||
// ❌ Wrong - Platform-dependent sorting
|
||||
items.Sort();
|
||||
|
||||
// ✅ Correct - Culture-invariant sorting
|
||||
items = items.OrderBy(x => x, StringComparer.Ordinal).ToList();
|
||||
```
|
||||
|
||||
#### Issue 2: Path Separators
|
||||
|
||||
**Symptom**: Windows produces different hash than macOS/Linux.
|
||||
|
||||
**Cause**: Windows uses `\`, Unix uses `/`.
|
||||
|
||||
**Solution**: Use `Path.Combine` or normalize:
|
||||
|
||||
```csharp
|
||||
// ❌ Wrong - Hardcoded separator
|
||||
var path = "dir\\file.txt";
|
||||
|
||||
// ✅ Correct - Cross-platform
|
||||
var path = Path.Combine("dir", "file.txt");
|
||||
|
||||
// ✅ Alternative - Normalize to forward slash
|
||||
var normalizedPath = path.Replace('\\', '/');
|
||||
```
|
||||
|
||||
#### Issue 3: Line Endings
|
||||
|
||||
**Symptom**: Hash includes file content with different line endings.
|
||||
|
||||
**Cause**: Windows uses CRLF (`\r\n`), Unix uses LF (`\n`).
|
||||
|
||||
**Solution**: Normalize to LF:
|
||||
|
||||
```csharp
|
||||
// ❌ Wrong - Platform line endings
|
||||
var content = File.ReadAllText(path);
|
||||
|
||||
// ✅ Correct - Normalized to LF
|
||||
var content = File.ReadAllText(path).Replace("\r\n", "\n");
|
||||
```
|
||||
|
||||
#### Issue 4: Floating-Point Precision
|
||||
|
||||
**Symptom**: Different platforms produce slightly different floating-point values.
|
||||
|
||||
**Cause**: JIT compiler optimizations, FPU rounding modes.
|
||||
|
||||
**Solution**: Use `decimal` for exact arithmetic, or round explicitly:
|
||||
|
||||
```csharp
|
||||
// ❌ Wrong - Floating-point non-determinism
|
||||
var value = 0.1 + 0.2; // Might be 0.30000000000000004
|
||||
|
||||
// ✅ Correct - Decimal for exact values
|
||||
var value = 0.1m + 0.2m; // Always 0.3
|
||||
|
||||
// ✅ Alternative - Round explicitly
|
||||
var value = Math.Round(0.1 + 0.2, 2); // 0.30
|
||||
```
|
||||
|
||||
## Performance Guidelines
|
||||
|
||||
### Execution Time Targets
|
||||
|
||||
| Test Type | Target | Max |
|
||||
|-----------|--------|-----|
|
||||
| Single iteration | <100ms | <500ms |
|
||||
| 10-iteration stability | <1s | <3s |
|
||||
| Golden file test | <100ms | <500ms |
|
||||
| **Full test suite** | **<5s** | **<15s** |
|
||||
|
||||
### Optimization Tips
|
||||
|
||||
1. **Avoid unnecessary I/O**: Create test data in memory
|
||||
2. **Use Task.CompletedTask**: For synchronous operations
|
||||
3. **Minimize allocations**: Reuse test data across assertions
|
||||
4. **Parallel test execution**: xUnit runs tests in parallel by default
|
||||
|
||||
### Performance Regression Detection
|
||||
|
||||
If test execution time increases by >2x:
|
||||
1. Profile with `dotnet-trace` or BenchmarkDotNet
|
||||
2. Identify bottleneck (I/O, CPU, memory)
|
||||
3. Optimize or split into separate test
|
||||
4. Document performance expectations in test comments
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Problem: Test Passes 9/10 Times, Fails 1/10
|
||||
|
||||
**Cause**: Non-deterministic input or race condition.
|
||||
|
||||
**Debug Steps:**
|
||||
1. Add logging to each iteration:
|
||||
```csharp
|
||||
_output.WriteLine($"Iteration {i}: Input={JsonSerializer.Serialize(input)}, Output={output}");
|
||||
```
|
||||
2. Look for differences in input or output
|
||||
3. Check for `Guid.NewGuid()`, `Random`, `DateTimeOffset.Now`
|
||||
4. Check for unsynchronized parallel operations
|
||||
|
||||
### Problem: Test Fails on Alpine but Passes Elsewhere
|
||||
|
||||
**Cause**: musl libc vs glibc difference.
|
||||
|
||||
**Debug Steps:**
|
||||
1. Run test locally with Alpine Docker:
|
||||
```bash
|
||||
docker run -it --rm -v $(pwd):/app mcr.microsoft.com/dotnet/sdk:10.0-alpine sh
|
||||
cd /app
|
||||
dotnet test --filter "FullyQualifiedName~MyTest"
|
||||
```
|
||||
2. Compare output with local (glibc) output
|
||||
3. Check for string sorting, culture-dependent formatting
|
||||
4. Use `StringComparer.Ordinal` and `CultureInfo.InvariantCulture`
|
||||
|
||||
### Problem: Golden Hash Changes After .NET Upgrade
|
||||
|
||||
**Cause**: .NET runtime change in JSON serialization or hash algorithm.
|
||||
|
||||
**Debug Steps:**
|
||||
1. Compare .NET versions:
|
||||
```bash
|
||||
dotnet --version # Should be same in CI/CD
|
||||
```
|
||||
2. Check JsonSerializer behavior:
|
||||
```csharp
|
||||
var json1 = JsonSerializer.Serialize(input, options);
|
||||
var json2 = JsonSerializer.Serialize(input, options);
|
||||
json1.Should().Be(json2);
|
||||
```
|
||||
3. If intentional .NET change, follow [Breaking Change Process](./GOLDEN_FILE_ESTABLISHMENT_GUIDE.md#breaking-change-process)
|
||||
|
||||
## References
|
||||
|
||||
- **Test README**: `src/__Tests/Determinism/README.md`
|
||||
- **Golden File Guide**: `docs/implplan/archived/2025-12-29-completed-sprints/GOLDEN_FILE_ESTABLISHMENT_GUIDE.md`
|
||||
- **ADR 0042**: CGS Merkle Tree Implementation
|
||||
- **ADR 0043**: Fulcio Keyless Signing
|
||||
- **CI/CD Workflow**: `.gitea/workflows/cross-platform-determinism.yml`
|
||||
|
||||
## Getting Help
|
||||
|
||||
- **Slack**: #determinism-testing
|
||||
- **Issue Label**: `determinism`, `testing`
|
||||
- **Priority**: High (determinism bugs affect audit trails)
|
||||
553
docs/technical/testing/LOCAL_CI_GUIDE.md
Normal file
553
docs/technical/testing/LOCAL_CI_GUIDE.md
Normal file
@@ -0,0 +1,553 @@
|
||||
# Local CI Testing Guide
|
||||
|
||||
> Run CI/CD pipelines locally before pushing to verify your changes.
|
||||
|
||||
---
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- **Docker Desktop** (Windows/macOS) or Docker Engine (Linux)
|
||||
- **.NET 10 SDK** ([download](https://dot.net/download))
|
||||
- **Git**
|
||||
- **Bash** (Linux/macOS native, Windows via WSL2 or Git Bash)
|
||||
- **act** (optional, for workflow simulation) - [install](https://github.com/nektos/act)
|
||||
|
||||
### Installation
|
||||
|
||||
1. **Copy environment template:**
|
||||
```bash
|
||||
cp devops/ci-local/.env.local.sample devops/ci-local/.env.local
|
||||
```
|
||||
|
||||
2. **(Optional) Install act for workflow simulation:**
|
||||
```bash
|
||||
# macOS
|
||||
brew install act
|
||||
|
||||
# Linux
|
||||
curl -sSL https://raw.githubusercontent.com/nektos/act/master/install.sh | sudo bash
|
||||
|
||||
# Windows (via Chocolatey)
|
||||
choco install act-cli
|
||||
```
|
||||
|
||||
3. **(Optional) Build CI Docker image:**
|
||||
```bash
|
||||
docker build -t stellaops-ci:local -f devops/docker/Dockerfile.ci .
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Usage
|
||||
|
||||
### Basic Commands
|
||||
|
||||
```bash
|
||||
# Quick smoke test (~2 min)
|
||||
./devops/scripts/local-ci.sh smoke
|
||||
|
||||
# Smoke steps (isolate build vs unit tests)
|
||||
./devops/scripts/local-ci.sh smoke --smoke-step build
|
||||
./devops/scripts/local-ci.sh smoke --smoke-step unit
|
||||
./devops/scripts/local-ci.sh smoke --smoke-step unit-split
|
||||
./devops/scripts/local-ci.sh smoke --smoke-step unit-split --test-timeout 5m --progress-interval 60
|
||||
./devops/scripts/local-ci.sh smoke --smoke-step unit-split --project-start 1 --project-count 50
|
||||
|
||||
# Full PR-gating suite (~15 min)
|
||||
./devops/scripts/local-ci.sh pr
|
||||
|
||||
# Test specific module
|
||||
./devops/scripts/local-ci.sh module --module Scanner
|
||||
|
||||
# Auto-detect changed modules
|
||||
./devops/scripts/local-ci.sh module
|
||||
|
||||
# Simulate workflow
|
||||
./devops/scripts/local-ci.sh workflow --workflow test-matrix
|
||||
|
||||
# Release dry-run
|
||||
./devops/scripts/local-ci.sh release --dry-run
|
||||
|
||||
# Full test suite (~45 min)
|
||||
./devops/scripts/local-ci.sh full
|
||||
```
|
||||
|
||||
### Windows (PowerShell)
|
||||
|
||||
```powershell
|
||||
# Quick smoke test
|
||||
.\devops\scripts\local-ci.ps1 smoke
|
||||
|
||||
# Smoke steps (isolate build vs unit tests)
|
||||
.\devops\scripts\local-ci.ps1 smoke -SmokeStep build
|
||||
.\devops\scripts\local-ci.ps1 smoke -SmokeStep unit
|
||||
.\devops\scripts\local-ci.ps1 smoke -SmokeStep unit-split
|
||||
.\devops\scripts\local-ci.ps1 smoke -SmokeStep unit-split -TestTimeout 5m -ProgressInterval 60
|
||||
.\devops\scripts\local-ci.ps1 smoke -SmokeStep unit-split -ProjectStart 1 -ProjectCount 50
|
||||
|
||||
# Full PR check
|
||||
.\devops\scripts\local-ci.ps1 pr
|
||||
|
||||
# With options
|
||||
.\devops\scripts\local-ci.ps1 pr -Verbose -Docker
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Modes
|
||||
|
||||
### smoke (~2 minutes)
|
||||
Quick validation before pushing. Runs only Unit tests.
|
||||
|
||||
```bash
|
||||
./devops/scripts/local-ci.sh smoke
|
||||
```
|
||||
|
||||
Optional stepwise smoke (to isolate hangs):
|
||||
|
||||
```bash
|
||||
./devops/scripts/local-ci.sh smoke --smoke-step build
|
||||
./devops/scripts/local-ci.sh smoke --smoke-step unit
|
||||
./devops/scripts/local-ci.sh smoke --smoke-step unit-split
|
||||
```
|
||||
|
||||
**What it does:**
|
||||
1. Builds the solution
|
||||
2. Runs Unit tests
|
||||
3. Reports pass/fail
|
||||
|
||||
### pr (~15 minutes)
|
||||
Full PR-gating suite matching CI exactly. Run this before opening a PR.
|
||||
|
||||
```bash
|
||||
./devops/scripts/local-ci.sh pr
|
||||
```
|
||||
|
||||
**What it does:**
|
||||
1. Starts CI services (PostgreSQL, Valkey)
|
||||
2. Builds with warnings-as-errors
|
||||
3. Runs all PR-gating categories:
|
||||
- Unit
|
||||
- Architecture
|
||||
- Contract
|
||||
- Integration
|
||||
- Security
|
||||
- Golden
|
||||
4. Generates TRX reports
|
||||
5. Stops CI services
|
||||
|
||||
### module
|
||||
Test only the modules you've changed. Detects changes via git diff.
|
||||
|
||||
```bash
|
||||
# Auto-detect changed modules
|
||||
./devops/scripts/local-ci.sh module
|
||||
|
||||
# Test specific module
|
||||
./devops/scripts/local-ci.sh module --module Scanner
|
||||
./devops/scripts/local-ci.sh module --module Concelier
|
||||
./devops/scripts/local-ci.sh module --module Authority
|
||||
```
|
||||
|
||||
**Available modules:**
|
||||
Scanner, Concelier, Authority, Policy, Attestor, EvidenceLocker, ExportCenter,
|
||||
Findings, SbomService, Notify, Router, Cryptography, AirGap, Cli, AdvisoryAI,
|
||||
ReachGraph, Orchestrator, PacksRegistry, Replay, Aoc, IssuerDirectory, Telemetry, Signals
|
||||
|
||||
### workflow
|
||||
Simulate a specific Gitea Actions workflow using `act`.
|
||||
|
||||
```bash
|
||||
# Simulate test-matrix workflow
|
||||
./devops/scripts/local-ci.sh workflow --workflow test-matrix
|
||||
|
||||
# Simulate build-test-deploy
|
||||
./devops/scripts/local-ci.sh workflow --workflow build-test-deploy
|
||||
```
|
||||
|
||||
**Requirements:**
|
||||
- `act` must be installed
|
||||
- CI Docker image built (`stellaops-ci:local`)
|
||||
|
||||
### release
|
||||
Dry-run release pipeline to verify release artifacts.
|
||||
|
||||
```bash
|
||||
./devops/scripts/local-ci.sh release --dry-run
|
||||
```
|
||||
|
||||
**What it does:**
|
||||
1. Builds all modules
|
||||
2. Validates CLI packaging
|
||||
3. Validates Helm chart
|
||||
4. Shows what would be released
|
||||
5. **Does NOT publish anything**
|
||||
|
||||
### full (~45 minutes)
|
||||
Complete test suite including extended categories.
|
||||
|
||||
```bash
|
||||
./devops/scripts/local-ci.sh full
|
||||
```
|
||||
|
||||
**Categories run:**
|
||||
- PR-Gating: Unit, Architecture, Contract, Integration, Security, Golden
|
||||
- Extended: Performance, Benchmark, AirGap, Chaos, Determinism, Resilience, Observability
|
||||
|
||||
---
|
||||
|
||||
## Options
|
||||
|
||||
| Option | Description |
|
||||
|--------|-------------|
|
||||
| `--category <cat>` | Run specific test category |
|
||||
| `--module <name>` | Test specific module |
|
||||
| `--workflow <name>` | Workflow to simulate |
|
||||
| `--smoke-step <step>` | Smoke step: build, unit, unit-split |
|
||||
| `--test-timeout <t>` | Per-test timeout (e.g., 5m) using --blame-hang |
|
||||
| `--progress-interval <s>` | Progress heartbeat in seconds |
|
||||
| `--project-start <n>` | Start index (1-based) for unit-split slicing |
|
||||
| `--project-count <n>` | Limit number of projects for unit-split slicing |
|
||||
| `--docker` | Force Docker execution |
|
||||
| `--native` | Force native execution |
|
||||
| `--act` | Force act execution |
|
||||
| `--parallel <n>` | Parallel test runners |
|
||||
| `--verbose` | Verbose output |
|
||||
| `--dry-run` | Show what would run |
|
||||
| `--rebuild` | Force rebuild CI image |
|
||||
| `--no-services` | Skip CI services |
|
||||
| `--keep-services` | Keep services running after tests |
|
||||
|
||||
---
|
||||
|
||||
## Test Categories
|
||||
|
||||
### PR-Gating (Required for merge)
|
||||
|
||||
| Category | Purpose | Time |
|
||||
|----------|---------|------|
|
||||
| **Unit** | Component isolation tests | ~3 min |
|
||||
| **Architecture** | Dependency rules | ~2 min |
|
||||
| **Contract** | API compatibility | ~2 min |
|
||||
| **Integration** | Database/service tests | ~8 min |
|
||||
| **Security** | Security assertions | ~3 min |
|
||||
| **Golden** | Corpus-based validation | ~3 min |
|
||||
|
||||
### Extended (On-demand)
|
||||
|
||||
| Category | Purpose | Time |
|
||||
|----------|---------|------|
|
||||
| **Performance** | Latency/throughput | ~20 min |
|
||||
| **Benchmark** | BenchmarkDotNet | ~30 min |
|
||||
| **Determinism** | Reproducibility | ~15 min |
|
||||
| **AirGap** | Offline operation | ~15 min |
|
||||
| **Chaos** | Resilience testing | ~20 min |
|
||||
|
||||
---
|
||||
|
||||
## CI Services
|
||||
|
||||
The local CI uses Docker Compose to run required services.
|
||||
|
||||
### Services Started
|
||||
|
||||
| Service | Port | Purpose |
|
||||
|---------|------|---------|
|
||||
| postgres-ci | 5433 | PostgreSQL 16 for tests |
|
||||
| valkey-ci | 6380 | Cache/messaging tests |
|
||||
| nats-ci | 4223 | Message queue tests |
|
||||
| mock-registry | 5001 | Container registry |
|
||||
| minio-ci | 9100 | S3-compatible storage |
|
||||
|
||||
### Manual Service Management
|
||||
|
||||
```bash
|
||||
# Start services
|
||||
docker compose -f devops/compose/docker-compose.ci.yaml up -d
|
||||
|
||||
# Check status
|
||||
docker compose -f devops/compose/docker-compose.ci.yaml ps
|
||||
|
||||
# View logs
|
||||
docker compose -f devops/compose/docker-compose.ci.yaml logs postgres-ci
|
||||
|
||||
# Stop services
|
||||
docker compose -f devops/compose/docker-compose.ci.yaml down
|
||||
|
||||
# Stop and remove volumes
|
||||
docker compose -f devops/compose/docker-compose.ci.yaml down -v
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
### Environment Variables
|
||||
|
||||
Copy the template and customize:
|
||||
|
||||
```bash
|
||||
cp devops/ci-local/.env.local.sample devops/ci-local/.env.local
|
||||
```
|
||||
|
||||
Key variables:
|
||||
|
||||
```bash
|
||||
# Database
|
||||
STELLAOPS_TEST_POSTGRES_CONNECTION="Host=localhost;Port=5433;..."
|
||||
|
||||
# Cache
|
||||
VALKEY_CONNECTION_STRING="localhost:6380"
|
||||
|
||||
# Execution
|
||||
LOCAL_CI_MODE=docker # docker, native, act
|
||||
LOCAL_CI_PARALLEL=4 # Parallel jobs
|
||||
LOCAL_CI_VERBOSE=false # Verbose output
|
||||
```
|
||||
|
||||
### Act Configuration
|
||||
|
||||
The `.actrc` file configures `act` for workflow simulation:
|
||||
|
||||
```ini
|
||||
--platform ubuntu-22.04=stellaops-ci:local
|
||||
--env-file devops/ci-local/.env.local
|
||||
--bind
|
||||
--reuse
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Offline & Cache
|
||||
|
||||
Local CI can run in offline or rate-limited environments. These steps help ensure reliable builds.
|
||||
|
||||
### NuGet Cache Warmup
|
||||
|
||||
Before running tests offline or during rate-limited periods, warm the NuGet cache:
|
||||
|
||||
```bash
|
||||
# Warm cache with throttled requests to avoid 429 errors
|
||||
export NUGET_MAX_HTTP_REQUESTS=4
|
||||
dotnet restore src/StellaOps.sln --disable-parallel
|
||||
|
||||
# Verify cache is populated
|
||||
ls ~/.nuget/packages | wc -l
|
||||
```
|
||||
|
||||
```powershell
|
||||
# PowerShell equivalent
|
||||
$env:NUGET_MAX_HTTP_REQUESTS = "4"
|
||||
dotnet restore src\StellaOps.sln --disable-parallel
|
||||
|
||||
# Verify cache
|
||||
(Get-ChildItem "$env:USERPROFILE\.nuget\packages").Count
|
||||
```
|
||||
|
||||
### Rate Limiting Mitigation
|
||||
|
||||
If encountering NuGet 429 (Too Many Requests) errors from package sources:
|
||||
|
||||
1. **Reduce concurrency:**
|
||||
```bash
|
||||
export NUGET_MAX_HTTP_REQUESTS=2
|
||||
dotnet restore --disable-parallel
|
||||
```
|
||||
|
||||
2. **Retry off-peak hours** (avoid UTC 09:00-17:00 on weekdays)
|
||||
|
||||
3. **Use local cache fallback:**
|
||||
```bash
|
||||
# Configure fallback to local cache only (offline mode)
|
||||
dotnet restore --source ~/.nuget/packages
|
||||
```
|
||||
|
||||
### Docker Image Caching
|
||||
|
||||
Pre-pull required CI images to avoid network dependency during tests:
|
||||
|
||||
```bash
|
||||
# Pull CI services
|
||||
docker compose -f devops/compose/docker-compose.ci.yaml pull
|
||||
|
||||
# Build local CI image
|
||||
docker build -t stellaops-ci:local -f devops/docker/Dockerfile.ci .
|
||||
|
||||
# Verify images are cached
|
||||
docker images | grep -E "stellaops|postgres|valkey|nats"
|
||||
```
|
||||
|
||||
### Offline-Safe Test Execution
|
||||
|
||||
For fully offline validation:
|
||||
|
||||
```bash
|
||||
# 1. Ensure NuGet cache is warm (see above)
|
||||
# 2. Start local CI services (pre-pulled)
|
||||
docker compose -f devops/compose/docker-compose.ci.yaml up -d
|
||||
|
||||
# 3. Run smoke with no network dependency
|
||||
./devops/scripts/local-ci.sh smoke --no-restore
|
||||
|
||||
# 4. Or run specific category offline
|
||||
dotnet test src/StellaOps.sln \
|
||||
--filter "Category=Unit" \
|
||||
--no-restore \
|
||||
--no-build
|
||||
```
|
||||
|
||||
### Test Fixtures for Air-Gap
|
||||
|
||||
Some tests require fixture data that must be present locally:
|
||||
|
||||
| Fixture | Location | Purpose |
|
||||
|---------|----------|---------|
|
||||
| Golden bundles | `src/__Tests/__Datasets/EvidenceLocker/` | Evidence locker tests |
|
||||
| Reachability fixtures | `src/tests/reachability/` | Reachability drift tests |
|
||||
| Connector snapshots | `src/<Module>/__Tests/**/Fixtures/` | Connector replay tests |
|
||||
|
||||
To verify fixtures are present:
|
||||
```bash
|
||||
find src -type d -name "Fixtures" | head -20
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Docker Issues
|
||||
|
||||
```bash
|
||||
# Reset CI services
|
||||
docker compose -f devops/compose/docker-compose.ci.yaml down -v
|
||||
|
||||
# Rebuild CI image
|
||||
docker build --no-cache -t stellaops-ci:local -f devops/docker/Dockerfile.ci .
|
||||
|
||||
# Check Docker is running
|
||||
docker info
|
||||
```
|
||||
|
||||
### Test Failures
|
||||
|
||||
```bash
|
||||
# Run with verbose output
|
||||
./devops/scripts/local-ci.sh pr --verbose
|
||||
|
||||
# Run specific category
|
||||
./devops/scripts/local-ci.sh pr --category Unit
|
||||
|
||||
# Check logs
|
||||
cat out/local-ci/logs/Unit-*.log
|
||||
|
||||
# Check current test project during unit-split
|
||||
cat out/local-ci/active-test.txt
|
||||
```
|
||||
|
||||
### Act Issues
|
||||
|
||||
```bash
|
||||
# List available workflows
|
||||
act -l
|
||||
|
||||
# Dry run workflow
|
||||
act -n pull_request -W .gitea/workflows/test-matrix.yml
|
||||
|
||||
# Debug mode
|
||||
act --verbose pull_request
|
||||
```
|
||||
|
||||
### Windows Issues
|
||||
|
||||
```powershell
|
||||
# Check WSL is installed
|
||||
wsl --status
|
||||
|
||||
# Install WSL if needed
|
||||
wsl --install
|
||||
|
||||
# Check Git Bash
|
||||
& "C:\Program Files\Git\bin\bash.exe" --version
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Results
|
||||
|
||||
Test results are saved to `out/local-ci/`:
|
||||
|
||||
```
|
||||
out/local-ci/
|
||||
├── trx/ # TRX test reports
|
||||
│ ├── Unit-20250128_120000.trx
|
||||
│ ├── Integration-20250128_120000.trx
|
||||
│ └── ...
|
||||
└── logs/ # Execution logs
|
||||
├── Unit-20250128_120000.log
|
||||
└── ...
|
||||
```
|
||||
|
||||
### Viewing Results
|
||||
|
||||
```bash
|
||||
# List TRX files
|
||||
ls -la out/local-ci/trx/
|
||||
|
||||
# View test log
|
||||
cat out/local-ci/logs/Unit-*.log | less
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Examples
|
||||
|
||||
### Pre-Push Workflow
|
||||
|
||||
```bash
|
||||
# 1. Quick validation
|
||||
./devops/scripts/local-ci.sh smoke
|
||||
|
||||
# 2. If smoke passes, run full PR check
|
||||
./devops/scripts/local-ci.sh pr
|
||||
|
||||
# 3. Push your changes
|
||||
git push origin feature/my-branch
|
||||
```
|
||||
|
||||
### Module Development
|
||||
|
||||
```bash
|
||||
# 1. Make changes to Scanner module
|
||||
|
||||
# 2. Run module-specific tests
|
||||
./devops/scripts/local-ci.sh module --module Scanner
|
||||
|
||||
# 3. If passes, run full PR check before PR
|
||||
./devops/scripts/local-ci.sh pr
|
||||
```
|
||||
|
||||
### Debugging Test Failures
|
||||
|
||||
```bash
|
||||
# 1. Run with verbose output
|
||||
./devops/scripts/local-ci.sh pr --verbose --category Unit
|
||||
|
||||
# 2. Check the log
|
||||
cat out/local-ci/logs/Unit-*.log
|
||||
|
||||
# 3. Run specific test directly
|
||||
dotnet test src/Scanner/__Tests/StellaOps.Scanner.Tests/StellaOps.Scanner.Tests.csproj \
|
||||
--filter "Category=Unit" \
|
||||
--verbosity detailed
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [CI/CD Overview](../cicd/README.md)
|
||||
- [Test Strategy](../cicd/test-strategy.md)
|
||||
- [Workflow Triggers](../cicd/workflow-triggers.md)
|
||||
- [Path Filters](../cicd/path-filters.md)
|
||||
379
docs/technical/testing/PERFORMANCE_BASELINES.md
Normal file
379
docs/technical/testing/PERFORMANCE_BASELINES.md
Normal file
@@ -0,0 +1,379 @@
|
||||
# Performance Baselines - Determinism Tests
|
||||
|
||||
## Overview
|
||||
|
||||
This document tracks performance baselines for determinism tests. Baselines help detect performance regressions and ensure tests remain fast for rapid CI/CD feedback.
|
||||
|
||||
**Last Updated**: 2025-12-29
|
||||
**.NET Version**: 10.0.100
|
||||
**Hardware Reference**: GitHub Actions runners (windows-latest, ubuntu-latest, macos-latest)
|
||||
|
||||
## Baseline Metrics
|
||||
|
||||
### CGS (Canonical Graph Signature) Tests
|
||||
|
||||
**File**: `src/__Tests/Determinism/CgsDeterminismTests.cs`
|
||||
|
||||
| Test | Windows | macOS | Linux | Alpine | Debian | Notes |
|
||||
|------|---------|-------|-------|--------|--------|-------|
|
||||
| `CgsHash_WithKnownEvidence_MatchesGoldenHash` | 87ms | 92ms | 85ms | 135ms | 89ms | Single verdict build |
|
||||
| `CgsHash_EmptyEvidence_ProducesDeterministicHash` | 45ms | 48ms | 43ms | 68ms | 46ms | Minimal evidence pack |
|
||||
| `CgsHash_SameInput_ProducesIdenticalHash_Across10Iterations` | 850ms | 920ms | 830ms | 1,350ms | 870ms | 10 iterations |
|
||||
| `CgsHash_VexOrderIndependent_ProducesIdenticalHash` | 165ms | 178ms | 162ms | 254ms | 169ms | 3 evidence packs |
|
||||
| `CgsHash_WithReachability_IsDifferentFromWithout` | 112ms | 121ms | 109ms | 172ms | 115ms | 2 evidence packs |
|
||||
| `CgsHash_DifferentPolicyVersion_ProducesDifferentHash` | 108ms | 117ms | 105ms | 165ms | 110ms | 2 evidence packs |
|
||||
| **Total Suite** | **1,367ms** | **1,476ms** | **1,334ms** | **2,144ms** | **1,399ms** | All tests |
|
||||
|
||||
**Regression Threshold**: If any test exceeds baseline by >2x, investigate.
|
||||
|
||||
### SBOM Lineage Tests
|
||||
|
||||
**File**: `src/SbomService/__Tests/StellaOps.SbomService.Lineage.Tests/LineageDeterminismTests.cs`
|
||||
|
||||
| Test | Windows | macOS | Linux | Alpine | Debian | Notes |
|
||||
|------|---------|-------|-------|--------|--------|-------|
|
||||
| `LineageExport_SameGraph_ProducesIdenticalNdjson_Across10Iterations` | 920ms | 995ms | 895ms | 1,420ms | 935ms | 10 iterations |
|
||||
| `LineageGraph_WithCycles_DetectsDeterministically` | 245ms | 265ms | 238ms | 378ms | 248ms | 1,000 node graph |
|
||||
| `LineageGraph_LargeGraph_PaginatesDeterministically` | 485ms | 525ms | 472ms | 748ms | 492ms | 10,000 node graph |
|
||||
| **Total Suite** | **1,650ms** | **1,785ms** | **1,605ms** | **2,546ms** | **1,675ms** | All tests |
|
||||
|
||||
### VexLens Truth Table Tests
|
||||
|
||||
**File**: `src/VexLens/__Tests/StellaOps.VexLens.Tests/Consensus/VexLensTruthTableTests.cs`
|
||||
|
||||
| Test | Windows | macOS | Linux | Alpine | Debian | Notes |
|
||||
|------|---------|-------|-------|--------|--------|-------|
|
||||
| `SingleIssuer_ReturnsIdentity` (5 cases) | 125ms | 135ms | 122ms | 192ms | 127ms | TheoryData |
|
||||
| `TwoIssuers_SameTier_MergesCorrectly` (9 cases) | 225ms | 243ms | 219ms | 347ms | 228ms | TheoryData |
|
||||
| `TrustTier_PrecedenceApplied` (3 cases) | 75ms | 81ms | 73ms | 115ms | 76ms | TheoryData |
|
||||
| `SameInputs_ProducesIdenticalOutput_Across10Iterations` | 485ms | 524ms | 473ms | 748ms | 493ms | 10 iterations |
|
||||
| `VexOrder_DoesNotAffectConsensus` | 95ms | 103ms | 92ms | 146ms | 96ms | 3 orderings |
|
||||
| **Total Suite** | **1,005ms** | **1,086ms** | **979ms** | **1,548ms** | **1,020ms** | All tests |
|
||||
|
||||
### Scheduler Resilience Tests
|
||||
|
||||
**File**: `src/Scheduler/__Tests/StellaOps.Scheduler.Tests/`
|
||||
|
||||
| Test | Windows | macOS | Linux | Alpine | Debian | Notes |
|
||||
|------|---------|-------|-------|--------|--------|-------|
|
||||
| `IdempotentKey_PreventsDuplicateExecution` | 1,250ms | 1,350ms | 1,225ms | 1,940ms | 1,275ms | 10 jobs, Testcontainers |
|
||||
| `WorkerKilledMidRun_JobRecoveredByAnotherWorker` | 5,500ms | 5,950ms | 5,375ms | 8,515ms | 5,605ms | Chaos test, heartbeat timeout |
|
||||
| `HighLoad_AppliesBackpressureCorrectly` | 12,000ms | 12,980ms | 11,720ms | 18,575ms | 12,220ms | 1,000 jobs, concurrency limit |
|
||||
| **Total Suite** | **18,750ms** | **20,280ms** | **18,320ms** | **29,030ms** | **19,100ms** | All tests |
|
||||
|
||||
**Note**: Scheduler tests use Testcontainers (PostgreSQL), adding ~2s startup overhead.
|
||||
|
||||
## Platform Comparison
|
||||
|
||||
### Average Speed Factor (relative to Linux Ubuntu)
|
||||
|
||||
| Platform | Speed Factor | Notes |
|
||||
|----------|--------------|-------|
|
||||
| Linux Ubuntu | 1.00x | Baseline (fastest) |
|
||||
| Windows | 1.02x | ~2% slower |
|
||||
| macOS | 1.10x | ~10% slower |
|
||||
| Debian | 1.05x | ~5% slower |
|
||||
| Alpine | 1.60x | ~60% slower (musl libc overhead) |
|
||||
|
||||
**Alpine Performance**: Alpine is consistently ~60% slower due to musl libc differences. This is expected and acceptable.
|
||||
|
||||
## Historical Trends
|
||||
|
||||
### 2025-12-29 (Baseline Establishment)
|
||||
|
||||
- **.NET Version**: 10.0.100
|
||||
- **Total Tests**: 79
|
||||
- **Total Execution Time**: ~25 seconds (all platforms, sequential)
|
||||
- **Status**: ✅ All tests passing
|
||||
|
||||
**Key Metrics**:
|
||||
- CGS determinism tests: <3s per platform
|
||||
- Lineage determinism tests: <3s per platform
|
||||
- VexLens truth tables: <2s per platform
|
||||
- Scheduler resilience: <30s per platform (includes Testcontainers overhead)
|
||||
|
||||
## Regression Detection
|
||||
|
||||
### Automated Monitoring
|
||||
|
||||
CI/CD workflow `.gitea/workflows/cross-platform-determinism.yml` tracks execution time and fails if:
|
||||
|
||||
```yaml
|
||||
- name: Check for performance regression
|
||||
run: |
|
||||
# Fail if CGS test suite exceeds 3 seconds on Linux
|
||||
if [ $CGS_SUITE_TIME_MS -gt 3000 ]; then
|
||||
echo "ERROR: CGS test suite exceeded 3s baseline (${CGS_SUITE_TIME_MS}ms)"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Fail if Alpine is >3x slower than Linux (expected is ~1.6x)
|
||||
ALPINE_FACTOR=$(echo "$ALPINE_TIME_MS / $LINUX_TIME_MS" | bc -l)
|
||||
if (( $(echo "$ALPINE_FACTOR > 3.0" | bc -l) )); then
|
||||
echo "ERROR: Alpine is >3x slower than Linux (factor: $ALPINE_FACTOR)"
|
||||
exit 1
|
||||
fi
|
||||
```
|
||||
|
||||
### Manual Benchmarking
|
||||
|
||||
Run benchmarks locally to compare before/after changes:
|
||||
|
||||
```bash
|
||||
cd src/__Tests/Determinism
|
||||
|
||||
# Run with detailed timing
|
||||
dotnet test --logger "console;verbosity=detailed" | tee benchmark-$(date +%Y%m%d).log
|
||||
|
||||
# Extract timing
|
||||
grep -E "Test Name:|Duration:" benchmark-*.log
|
||||
```
|
||||
|
||||
**Example Output**:
|
||||
```
|
||||
Test Name: CgsHash_WithKnownEvidence_MatchesGoldenHash
|
||||
Duration: 87ms
|
||||
|
||||
Test Name: CgsHash_SameInput_ProducesIdenticalHash_Across10Iterations
|
||||
Duration: 850ms
|
||||
```
|
||||
|
||||
### BenchmarkDotNet Integration (Future)
|
||||
|
||||
For precise micro-benchmarks:
|
||||
|
||||
```csharp
|
||||
// src/__Tests/__Benchmarks/CgsHashBenchmarks.cs
|
||||
[MemoryDiagnoser]
|
||||
[MarkdownExporter]
|
||||
public class CgsHashBenchmarks
|
||||
{
|
||||
[Benchmark]
|
||||
public string ComputeCgsHash_SmallEvidence()
|
||||
{
|
||||
var evidence = CreateSmallEvidencePack();
|
||||
var policyLock = CreatePolicyLock();
|
||||
var service = new VerdictBuilderService(NullLogger.Instance);
|
||||
return service.ComputeCgsHash(evidence, policyLock);
|
||||
}
|
||||
|
||||
[Benchmark]
|
||||
public string ComputeCgsHash_LargeEvidence()
|
||||
{
|
||||
var evidence = CreateLargeEvidencePack(); // 100 VEX documents
|
||||
var policyLock = CreatePolicyLock();
|
||||
var service = new VerdictBuilderService(NullLogger.Instance);
|
||||
return service.ComputeCgsHash(evidence, policyLock);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Run**:
|
||||
```bash
|
||||
dotnet run -c Release --project src/__Tests/__Benchmarks/StellaOps.Benchmarks.Determinism.csproj
|
||||
```
|
||||
|
||||
## Optimization Strategies
|
||||
|
||||
### Strategy 1: Reduce Allocations
|
||||
|
||||
**Before**:
|
||||
```csharp
|
||||
for (int i = 0; i < 10; i++)
|
||||
{
|
||||
var leaves = new List<string>(); // ❌ Allocates every iteration
|
||||
leaves.Add(ComputeHash(input));
|
||||
}
|
||||
```
|
||||
|
||||
**After**:
|
||||
```csharp
|
||||
var leaves = new List<string>(capacity: 10); // ✅ Pre-allocate
|
||||
for (int i = 0; i < 10; i++)
|
||||
{
|
||||
leaves.Clear();
|
||||
leaves.Add(ComputeHash(input));
|
||||
}
|
||||
```
|
||||
|
||||
### Strategy 2: Use Span<T> for Hashing
|
||||
|
||||
**Before**:
|
||||
```csharp
|
||||
var bytes = Encoding.UTF8.GetBytes(input); // ❌ Allocates byte array
|
||||
var hash = SHA256.HashData(bytes);
|
||||
```
|
||||
|
||||
**After**:
|
||||
```csharp
|
||||
Span<byte> buffer = stackalloc byte[256]; // ✅ Stack allocation
|
||||
var bytesWritten = Encoding.UTF8.GetBytes(input, buffer);
|
||||
var hash = SHA256.HashData(buffer[..bytesWritten]);
|
||||
```
|
||||
|
||||
### Strategy 3: Cache Expensive Computations
|
||||
|
||||
**Before**:
|
||||
```csharp
|
||||
[Fact]
|
||||
public void Test()
|
||||
{
|
||||
var service = CreateService(); // ❌ Recreates every test
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
**After**:
|
||||
```csharp
|
||||
private readonly MyService _service; // ✅ Reuse across tests
|
||||
|
||||
public MyTests()
|
||||
{
|
||||
_service = CreateService();
|
||||
}
|
||||
```
|
||||
|
||||
### Strategy 4: Parallel Test Execution
|
||||
|
||||
xUnit runs tests in parallel by default. To disable for specific tests:
|
||||
|
||||
```csharp
|
||||
[Collection("Sequential")] // Disable parallelism
|
||||
public class MySlowTests
|
||||
{
|
||||
// Tests run sequentially within this class
|
||||
}
|
||||
```
|
||||
|
||||
## Performance Regression Examples
|
||||
|
||||
### Example 1: Unexpected Allocations
|
||||
|
||||
**Symptom**: Test time increased from 85ms to 450ms after refactoring.
|
||||
|
||||
**Cause**: Accidental string concatenation in loop:
|
||||
```csharp
|
||||
// Before: 85ms
|
||||
var hash = string.Join("", hashes);
|
||||
|
||||
// After: 450ms (BUG!)
|
||||
var result = "";
|
||||
foreach (var h in hashes)
|
||||
{
|
||||
result += h; // ❌ Creates new string every iteration!
|
||||
}
|
||||
```
|
||||
|
||||
**Fix**: Use `StringBuilder`:
|
||||
```csharp
|
||||
var sb = new StringBuilder();
|
||||
foreach (var h in hashes)
|
||||
{
|
||||
sb.Append(h); // ✅ Efficient
|
||||
}
|
||||
var result = sb.ToString();
|
||||
```
|
||||
|
||||
### Example 2: Excessive I/O
|
||||
|
||||
**Symptom**: Test time increased from 100ms to 2,500ms.
|
||||
|
||||
**Cause**: Reading file from disk every iteration:
|
||||
```csharp
|
||||
for (int i = 0; i < 10; i++)
|
||||
{
|
||||
var data = File.ReadAllText("test-data.json"); // ❌ Disk I/O every iteration!
|
||||
ProcessData(data);
|
||||
}
|
||||
```
|
||||
|
||||
**Fix**: Read once, reuse:
|
||||
```csharp
|
||||
var data = File.ReadAllText("test-data.json"); // ✅ Read once
|
||||
for (int i = 0; i < 10; i++)
|
||||
{
|
||||
ProcessData(data);
|
||||
}
|
||||
```
|
||||
|
||||
### Example 3: Inefficient Sorting
|
||||
|
||||
**Symptom**: Test time increased from 165ms to 950ms after adding VEX documents.
|
||||
|
||||
**Cause**: Sorting inside loop:
|
||||
```csharp
|
||||
for (int i = 0; i < 10; i++)
|
||||
{
|
||||
var sortedVex = vexDocuments.OrderBy(v => v).ToList(); // ❌ Sorts every iteration!
|
||||
ProcessVex(sortedVex);
|
||||
}
|
||||
```
|
||||
|
||||
**Fix**: Sort once, reuse:
|
||||
```csharp
|
||||
var sortedVex = vexDocuments.OrderBy(v => v).ToList(); // ✅ Sort once
|
||||
for (int i = 0; i < 10; i++)
|
||||
{
|
||||
ProcessVex(sortedVex);
|
||||
}
|
||||
```
|
||||
|
||||
## Monitoring and Alerts
|
||||
|
||||
### Slack Alerts
|
||||
|
||||
Configure alerts for performance regressions:
|
||||
|
||||
```yaml
|
||||
# .gitea/workflows/cross-platform-determinism.yml
|
||||
- name: Notify on regression
|
||||
if: failure() && steps.performance-check.outcome == 'failure'
|
||||
uses: slackapi/slack-github-action@v1
|
||||
with:
|
||||
payload: |
|
||||
{
|
||||
"text": "⚠️ Performance regression detected in determinism tests",
|
||||
"blocks": [
|
||||
{
|
||||
"type": "section",
|
||||
"text": {
|
||||
"type": "mrkdwn",
|
||||
"text": "*CGS Test Suite Exceeded Baseline*\n\nBaseline: 3s\nActual: ${{ steps.performance-check.outputs.duration }}s\n\nPlatform: Linux Ubuntu\nCommit: ${{ github.sha }}"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Grafana Dashboard
|
||||
|
||||
Track execution time over time:
|
||||
|
||||
```promql
|
||||
# Prometheus query
|
||||
histogram_quantile(0.95,
|
||||
rate(determinism_test_duration_seconds_bucket{test="CgsHash_10Iterations"}[5m])
|
||||
)
|
||||
```
|
||||
|
||||
**Dashboard Panels**:
|
||||
1. Test duration (p50, p95, p99) over time
|
||||
2. Platform comparison (Windows vs Linux vs macOS vs Alpine)
|
||||
3. Test failure rate by platform
|
||||
4. Execution time distribution (histogram)
|
||||
|
||||
## References
|
||||
|
||||
- **CI/CD Workflow**: `.gitea/workflows/cross-platform-determinism.yml`
|
||||
- **Test README**: `src/__Tests/Determinism/README.md`
|
||||
- **Developer Guide**: `docs/testing/DETERMINISM_DEVELOPER_GUIDE.md`
|
||||
- **Batch Summary**: `docs/implplan/archived/2025-12-29-completed-sprints/BATCH_20251229_BE_COMPLETION_SUMMARY.md`
|
||||
|
||||
## Changelog
|
||||
|
||||
### 2025-12-29 - Initial Baselines
|
||||
|
||||
- Established baselines for CGS, Lineage, VexLens, and Scheduler tests
|
||||
- Documented platform speed factors (Alpine 1.6x, macOS 1.1x, Windows 1.02x)
|
||||
- Set regression thresholds (>2x baseline triggers investigation)
|
||||
- Configured CI/CD performance monitoring
|
||||
164
docs/technical/testing/PRE_COMMIT_CHECKLIST.md
Normal file
164
docs/technical/testing/PRE_COMMIT_CHECKLIST.md
Normal file
@@ -0,0 +1,164 @@
|
||||
# Pre-Commit Checklist
|
||||
|
||||
> Quick reference for validating changes before committing.
|
||||
|
||||
---
|
||||
|
||||
## TL;DR - Minimum Validation
|
||||
|
||||
```bash
|
||||
# 1. Quick smoke test (2 min)
|
||||
./devops/scripts/local-ci.sh smoke
|
||||
|
||||
# 2. If smoke passes, full PR check (15 min)
|
||||
./devops/scripts/local-ci.sh pr
|
||||
|
||||
# 3. Commit
|
||||
git add -A && git commit -m "Your message"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Validation Levels
|
||||
|
||||
### Level 1: Quick Check (Always)
|
||||
|
||||
```bash
|
||||
./devops/scripts/local-ci.sh smoke
|
||||
```
|
||||
|
||||
**Time:** ~2 minutes
|
||||
**Validates:** Build + Unit tests
|
||||
|
||||
---
|
||||
|
||||
### Level 2: PR-Ready (Before opening PR)
|
||||
|
||||
```bash
|
||||
./devops/scripts/local-ci.sh pr
|
||||
```
|
||||
|
||||
**Time:** ~15 minutes
|
||||
**Validates:** Unit, Architecture, Contract, Integration, Security, Golden
|
||||
|
||||
---
|
||||
|
||||
### Level 3: Module-Focused (Changed modules only)
|
||||
|
||||
```bash
|
||||
./devops/scripts/local-ci.sh module
|
||||
```
|
||||
|
||||
**Time:** Varies by module
|
||||
**Validates:** Auto-detected or specified module
|
||||
|
||||
---
|
||||
|
||||
### Level 4: Full Suite (Major changes/releases)
|
||||
|
||||
```bash
|
||||
./devops/scripts/local-ci.sh full
|
||||
```
|
||||
|
||||
**Time:** ~45 minutes
|
||||
**Validates:** All categories including Performance, Benchmark, AirGap, Chaos
|
||||
|
||||
---
|
||||
|
||||
## Quick Commands
|
||||
|
||||
| Purpose | Command |
|
||||
|---------|---------|
|
||||
| Smoke test | `./devops/scripts/local-ci.sh smoke` |
|
||||
| PR check | `./devops/scripts/local-ci.sh pr` |
|
||||
| Module tests | `./devops/scripts/local-ci.sh module --module Scanner` |
|
||||
| Single category | `./devops/scripts/local-ci.sh pr --category Unit` |
|
||||
| Verbose output | `./devops/scripts/local-ci.sh pr --verbose` |
|
||||
| Dry run | `./devops/scripts/local-ci.sh pr --dry-run` |
|
||||
| Release check | `./devops/scripts/local-ci.sh release --dry-run` |
|
||||
|
||||
---
|
||||
|
||||
## Windows (PowerShell)
|
||||
|
||||
```powershell
|
||||
.\devops\scripts\local-ci.ps1 smoke
|
||||
.\devops\scripts\local-ci.ps1 pr
|
||||
.\devops\scripts\local-ci.ps1 module -Module Scanner
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Service Management
|
||||
|
||||
```bash
|
||||
# Start
|
||||
docker compose -f devops/compose/docker-compose.ci.yaml up -d
|
||||
|
||||
# Stop
|
||||
docker compose -f devops/compose/docker-compose.ci.yaml down
|
||||
|
||||
# Reset
|
||||
docker compose -f devops/compose/docker-compose.ci.yaml down -v
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Test Categories
|
||||
|
||||
| Category | When Required |
|
||||
|----------|---------------|
|
||||
| **Unit** | Always |
|
||||
| **Architecture** | PR-gating |
|
||||
| **Contract** | API changes |
|
||||
| **Integration** | Database/service changes |
|
||||
| **Security** | Any code changes |
|
||||
| **Golden** | Output format changes |
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Build fails
|
||||
```bash
|
||||
dotnet clean src/StellaOps.sln
|
||||
dotnet build src/StellaOps.sln
|
||||
```
|
||||
|
||||
### Tests fail
|
||||
```bash
|
||||
./devops/scripts/local-ci.sh pr --verbose
|
||||
cat out/local-ci/logs/Unit-*.log
|
||||
```
|
||||
|
||||
### Services won't start
|
||||
```bash
|
||||
docker compose -f devops/compose/docker-compose.ci.yaml down -v
|
||||
docker compose -f devops/compose/docker-compose.ci.yaml up -d
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Checklist
|
||||
|
||||
**Before every commit:**
|
||||
- [ ] `./devops/scripts/local-ci.sh smoke` passes
|
||||
|
||||
**Before opening PR:**
|
||||
- [ ] `./devops/scripts/local-ci.sh pr` passes
|
||||
- [ ] `git status` shows only intended changes
|
||||
|
||||
**Before release:**
|
||||
- [ ] `./devops/scripts/local-ci.sh full` passes
|
||||
- [ ] `./devops/scripts/local-ci.sh release --dry-run` succeeds
|
||||
|
||||
---
|
||||
|
||||
## Results Location
|
||||
|
||||
```
|
||||
out/local-ci/
|
||||
trx/ # Test reports (TRX format)
|
||||
logs/ # Execution logs
|
||||
```
|
||||
|
||||
170
docs/technical/testing/README.md
Normal file
170
docs/technical/testing/README.md
Normal file
@@ -0,0 +1,170 @@
|
||||
# Testing Documentation
|
||||
|
||||
> Comprehensive guides for testing StellaOps before and during development.
|
||||
|
||||
---
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Before Committing
|
||||
|
||||
```bash
|
||||
# Quick smoke test (~2 min)
|
||||
./devops/scripts/validate-before-commit.sh quick
|
||||
|
||||
# Full PR validation (~15 min)
|
||||
./devops/scripts/validate-before-commit.sh
|
||||
|
||||
# Windows
|
||||
.\devops\scripts\validate-before-commit.ps1
|
||||
```
|
||||
|
||||
See [PRE_COMMIT_CHECKLIST.md](PRE_COMMIT_CHECKLIST.md) for the validation checklist.
|
||||
|
||||
---
|
||||
|
||||
## Documentation Index
|
||||
|
||||
### Local Testing
|
||||
|
||||
| Document | Description |
|
||||
|----------|-------------|
|
||||
| [LOCAL_CI_GUIDE.md](LOCAL_CI_GUIDE.md) | Complete guide to local CI testing |
|
||||
| [PRE_COMMIT_CHECKLIST.md](PRE_COMMIT_CHECKLIST.md) | Quick checklist for pre-commit validation |
|
||||
|
||||
### Testing Strategy
|
||||
|
||||
| Document | Description |
|
||||
|----------|-------------|
|
||||
| [TESTING_MASTER_PLAN.md](TESTING_MASTER_PLAN.md) | Overall testing strategy and goals |
|
||||
| [testing-strategy-models.md](testing-strategy-models.md) | Testing strategy models and approaches |
|
||||
| [TEST_COVERAGE_MATRIX.md](TEST_COVERAGE_MATRIX.md) | Coverage matrix by module and category |
|
||||
| [testkit-usage-guide.md](testkit-usage-guide.md) | Guide to using the StellaOps TestKit |
|
||||
|
||||
### CI/CD Integration
|
||||
|
||||
| Document | Description |
|
||||
|----------|-------------|
|
||||
| [ci-quality-gates.md](ci-quality-gates.md) | Quality gates for CI/CD pipelines |
|
||||
| [ci-lane-filters.md](ci-lane-filters.md) | Test lane filter configuration |
|
||||
| [ci-lane-integration.md](ci-lane-integration.md) | CI lane integration guide |
|
||||
|
||||
### Specialized Testing
|
||||
|
||||
| Document | Description |
|
||||
|----------|-------------|
|
||||
| [determinism-gates.md](determinism-gates.md) | Determinism verification gates |
|
||||
| [determinism-verification.md](determinism-verification.md) | Determinism testing guide |
|
||||
| [security-testing-guide.md](security-testing-guide.md) | Security testing practices |
|
||||
| [mutation-testing-guide.md](mutation-testing-guide.md) | Mutation testing guide |
|
||||
| [mutation-testing-baselines.md](mutation-testing-baselines.md) | Mutation testing baselines |
|
||||
| [e2e-reproducibility.md](e2e-reproducibility.md) | End-to-end reproducibility testing |
|
||||
| [competitor-parity-testing.md](competitor-parity-testing.md) | Competitive parity testing |
|
||||
|
||||
### Component-Specific
|
||||
|
||||
| Document | Description |
|
||||
|----------|-------------|
|
||||
| [webservice-test-discipline.md](webservice-test-discipline.md) | Web service testing discipline |
|
||||
| [webservice-test-rollout-plan.md](webservice-test-rollout-plan.md) | Web service test rollout plan |
|
||||
| [connector-fixture-discipline.md](connector-fixture-discipline.md) | Connector fixture testing |
|
||||
| [schema-validation.md](schema-validation.md) | Schema validation testing |
|
||||
|
||||
### Sprint Planning
|
||||
|
||||
| Document | Description |
|
||||
|----------|-------------|
|
||||
| [SPRINT_DEPENDENCY_GRAPH.md](SPRINT_DEPENDENCY_GRAPH.md) | Sprint dependency visualization |
|
||||
| [SPRINT_EXECUTION_PLAYBOOK.md](SPRINT_EXECUTION_PLAYBOOK.md) | Sprint execution guide |
|
||||
| [testing-quality-guardrails-implementation.md](testing-quality-guardrails-implementation.md) | Quality guardrails implementation |
|
||||
|
||||
---
|
||||
|
||||
## Test Categories
|
||||
|
||||
| Category | Description | When Run |
|
||||
|----------|-------------|----------|
|
||||
| **Unit** | Component isolation tests | Always |
|
||||
| **Architecture** | Dependency and layering rules | PR-gating |
|
||||
| **Contract** | API compatibility validation | PR-gating |
|
||||
| **Integration** | Database and service tests | PR-gating |
|
||||
| **Security** | Security assertion tests | PR-gating |
|
||||
| **Golden** | Corpus-based regression tests | PR-gating |
|
||||
| **Performance** | Latency and throughput tests | Extended |
|
||||
| **Benchmark** | BenchmarkDotNet runs | Extended |
|
||||
| **Determinism** | Reproducibility tests | Extended |
|
||||
| **AirGap** | Offline operation tests | Extended |
|
||||
| **Chaos** | Resilience tests | Extended |
|
||||
|
||||
---
|
||||
|
||||
## Quick Commands
|
||||
|
||||
```bash
|
||||
# Local CI runner
|
||||
./devops/scripts/local-ci.sh smoke # Quick validation
|
||||
./devops/scripts/local-ci.sh pr # PR-gating suite
|
||||
./devops/scripts/local-ci.sh module # Module tests
|
||||
./devops/scripts/local-ci.sh full # All tests
|
||||
|
||||
# Pre-commit validation
|
||||
./devops/scripts/validate-before-commit.sh # PR-gating
|
||||
./devops/scripts/validate-before-commit.sh quick # Smoke only
|
||||
./devops/scripts/validate-before-commit.sh full # Everything
|
||||
|
||||
# Web/Angular tests
|
||||
./devops/scripts/local-ci.sh module --module Web # Web module tests
|
||||
./devops/scripts/local-ci.sh pr --category Web # Web as part of PR
|
||||
|
||||
# Service management
|
||||
docker compose -f devops/compose/docker-compose.ci.yaml up -d
|
||||
docker compose -f devops/compose/docker-compose.ci.yaml down
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Web/Angular Testing
|
||||
|
||||
The Angular frontend (`src/Web/StellaOps.Web`) has its own test suite:
|
||||
|
||||
```bash
|
||||
cd src/Web/StellaOps.Web
|
||||
|
||||
# Install dependencies
|
||||
npm ci
|
||||
|
||||
# Unit tests (Karma/Jasmine)
|
||||
npm run test:ci
|
||||
|
||||
# E2E tests (Playwright)
|
||||
npx playwright install --with-deps chromium
|
||||
npm run test:e2e
|
||||
|
||||
# Accessibility tests (Axe)
|
||||
npm run test:a11y
|
||||
|
||||
# Production build
|
||||
npm run build -- --configuration production
|
||||
|
||||
# Storybook build
|
||||
npm run storybook:build
|
||||
```
|
||||
|
||||
| Test Type | Framework | Command | Duration |
|
||||
|-----------|-----------|---------|----------|
|
||||
| Unit | Karma/Jasmine | `npm run test:ci` | ~3 min |
|
||||
| E2E | Playwright | `npm run test:e2e` | ~5 min |
|
||||
| A11y | Axe-core | `npm run test:a11y` | ~2 min |
|
||||
| Build | Angular CLI | `npm run build` | ~2 min |
|
||||
|
||||
---
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Test Suite Overview](../TEST_SUITE_OVERVIEW.md) - High-level entry point for testing
|
||||
- [CI/CD Overview](../cicd/README.md)
|
||||
- [CI/CD Test Strategy](../cicd/test-strategy.md) - Detailed CI/CD test integration
|
||||
- [Workflow Triggers](../cicd/workflow-triggers.md)
|
||||
- [Path Filters](../cicd/path-filters.md)
|
||||
- [Test Infrastructure](../../src/__Tests/AGENTS.md)
|
||||
|
||||
393
docs/technical/testing/SPRINT_DEPENDENCY_GRAPH.md
Normal file
393
docs/technical/testing/SPRINT_DEPENDENCY_GRAPH.md
Normal file
@@ -0,0 +1,393 @@
|
||||
# Testing Strategy Sprint Dependency Graph & Critical Path Analysis
|
||||
|
||||
> **Purpose:** Visualize sprint dependencies, identify critical path, optimize parallel execution, and coordinate cross-guild work.
|
||||
|
||||
---
|
||||
|
||||
## Executive Summary
|
||||
|
||||
**Total Sprints:** 22 sprints across 4 batches
|
||||
**Total Tasks:** ~370 tasks
|
||||
**Estimated Duration:** 26 weeks (6 months) if executed sequentially
|
||||
**Optimal Duration:** 14 weeks (3.5 months) with full parallelization
|
||||
**Critical Path:** 14 weeks (Foundation Epics → Module Tests)
|
||||
**Parallelization Opportunities:** Up to 15 sprints can run concurrently
|
||||
|
||||
---
|
||||
|
||||
## Sprint Inventory by Batch
|
||||
|
||||
### Batch 5100.0007: Foundation Epics (90 tasks, 6 sprints)
|
||||
| Sprint ID | Name | Tasks | Waves | Dependencies |
|
||||
|-----------|------|-------|-------|--------------|
|
||||
| 5100.0007.0001 | Master Testing Strategy | 18 | 4 | None (entry point) |
|
||||
| 5100.0007.0002 | Epic A: TestKit Foundations | 13 | 4 | 0001 (Wave 1 complete) |
|
||||
| 5100.0007.0003 | Epic B: Determinism Gate | 12 | 4 | 0002 (TestKit available) |
|
||||
| 5100.0007.0004 | Epic C: Storage Harness | 12 | 4 | 0002 (TestKit available) |
|
||||
| 5100.0007.0005 | Epic D: Connector Fixtures | 12 | 4 | 0002 (TestKit available) |
|
||||
| 5100.0007.0006 | Epic E: WebService Contract | 12 | 4 | 0002 (TestKit + OtelCapture) |
|
||||
| 5100.0007.0007 | Epic F: Architecture Tests | 17 | 6 | None (can start immediately) |
|
||||
|
||||
### Batch 5100.0008: Quality Gates (11 tasks, 1 sprint)
|
||||
| Sprint ID | Name | Tasks | Waves | Dependencies |
|
||||
|-----------|------|-------|-------|--------------|
|
||||
| 5100.0008.0001 | Competitor Parity Testing | 11 | 4 | 0007.0001 (Wave 1), 0007.0002 |
|
||||
|
||||
### Batch 5100.0009: Module Test Implementations (185 tasks, 11 sprints)
|
||||
| Sprint ID | Module | Models | Tasks | Waves | Dependencies |
|
||||
|-----------|--------|--------|-------|-------|--------------|
|
||||
| 5100.0009.0001 | Scanner | L0,AN1,S1,T1,W1,WK1,PERF | 25 | 4 | 0002,0003,0004,0006 |
|
||||
| 5100.0009.0002 | Concelier | C1,L0,S1,W1,AN1 | 18 | 4 | 0002,0003,0004,0005,0006,0007.0007 |
|
||||
| 5100.0009.0003 | Excititor | C1,L0,S1,W1,WK1 | 21 | 4 | 0002,0003,0004,0005,0006,0007.0007 |
|
||||
| 5100.0009.0004 | Policy | L0,S1,W1 | 15 | 3 | 0002,0003,0004,0006 |
|
||||
| 5100.0009.0005 | Authority | L0,W1,C1 | 17 | 3 | 0002,0005,0006 |
|
||||
| 5100.0009.0006 | Signer | L0,W1,C1 | 17 | 3 | 0002,0003,0005,0006 |
|
||||
| 5100.0009.0007 | Attestor | L0,W1 | 14 | 3 | 0002,0003,0006,0009.0006 |
|
||||
| 5100.0009.0008 | Scheduler | L0,S1,W1,WK1 | 14 | 3 | 0002,0004,0006 |
|
||||
| 5100.0009.0009 | Notify | L0,C1,S1,W1,WK1 | 18 | 3 | 0002,0004,0005,0006 |
|
||||
| 5100.0009.0010 | CLI | CLI1 | 13 | 3 | 0002,0003 |
|
||||
| 5100.0009.0011 | UI | W1 | 13 | 3 | 0002,0006 |
|
||||
|
||||
### Batch 5100.0010: Infrastructure Tests (62 tasks, 4 sprints)
|
||||
| Sprint ID | Module Family | Models | Tasks | Waves | Dependencies |
|
||||
|-----------|---------------|--------|-------|-------|--------------|
|
||||
| 5100.0010.0001 | EvidenceLocker + Findings + Replay | L0,S1,W1,WK1 | 16 | 3 | 0002,0004,0006 |
|
||||
| 5100.0010.0002 | Graph + TimelineIndexer | L0,S1,W1,WK1 | 15 | 3 | 0002,0004,0006 |
|
||||
| 5100.0010.0003 | Router + Messaging | L0,T1,W1,S1 | 14 | 3 | 0002,0004 |
|
||||
| 5100.0010.0004 | AirGap | L0,AN1,S1,W1,CLI1 | 17 | 3 | 0002,0003,0004,0006 |
|
||||
|
||||
---
|
||||
|
||||
## Dependency Visualization (ASCII Graph)
|
||||
|
||||
```
|
||||
CRITICAL PATH (14 weeks):
|
||||
Week 1-2: [5100.0007.0001] Master Strategy (Wave 1-4)
|
||||
│
|
||||
Week 3-4: [5100.0007.0002] TestKit Foundations ← CRITICAL BOTTLENECK
|
||||
│
|
||||
├──────────┬──────────┬──────────┬──────────┐
|
||||
Week 5-6: [0003] [0004] [0005] [0006] [0007.0007]
|
||||
Determ. Storage Connect. WebSvc Arch.Tests
|
||||
│ │ │ │
|
||||
└─────────┴─────────┴─────────┘
|
||||
│
|
||||
Week 7-10: ┌──────────┼──────────────────────────────────┐
|
||||
[5100.0009.xxxx] ALL MODULE TESTS (parallel) │
|
||||
11 sprints run concurrently │
|
||||
│ │
|
||||
Week 11-14:└────────────────────────────────────────────┘
|
||||
[5100.0010.xxxx] ALL INFRASTRUCTURE TESTS
|
||||
4 sprints run concurrently
|
||||
|
||||
|
||||
PARALLEL EXECUTION ZONES:
|
||||
|
||||
Zone 1 (Weeks 5-6): Epic Implementations
|
||||
- 5100.0007.0003 (Determinism) ─┐
|
||||
- 5100.0007.0004 (Storage) ├─ Can run in parallel
|
||||
- 5100.0007.0005 (Connectors) │ (all depend only on TestKit)
|
||||
- 5100.0007.0006 (WebService) │
|
||||
- 5100.0007.0007 (Architecture) ─┘
|
||||
|
||||
Zone 2 (Weeks 7-10): Module Tests
|
||||
- Scanner (5100.0009.0001) ─┐
|
||||
- Concelier (5100.0009.0002) │
|
||||
- Excititor (5100.0009.0003) │
|
||||
- Policy (5100.0009.0004) ├─ Can run in parallel
|
||||
- Authority (5100.0009.0005) │ (Epic dependencies met)
|
||||
- Signer (5100.0009.0006) │
|
||||
- Attestor (5100.0009.0007) │
|
||||
- Scheduler (5100.0009.0008) │
|
||||
- Notify (5100.0009.0009) │
|
||||
- CLI (5100.0009.0010) │
|
||||
- UI (5100.0009.0011) ─┘
|
||||
|
||||
Zone 3 (Weeks 11-14): Infrastructure Tests
|
||||
- EvidenceLocker (5100.0010.0001) ─┐
|
||||
- Graph (5100.0010.0002) ├─ Can run in parallel
|
||||
- Router/Messaging (5100.0010.0003)│
|
||||
- AirGap (5100.0010.0004) ─┘
|
||||
|
||||
Zone 4 (Weeks 3-14): Quality Gates (can overlap)
|
||||
- Competitor Parity (5100.0008.0001) runs after Week 3
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Critical Path Analysis
|
||||
|
||||
### Critical Path Sequence (14 weeks)
|
||||
1. **Week 1-2:** Master Strategy Sprint (5100.0007.0001)
|
||||
- Wave 1: Documentation sync
|
||||
- Wave 2: Quick wins (test runner scripts, trait standardization)
|
||||
- Wave 3: CI infrastructure
|
||||
- Wave 4: Epic sprint creation
|
||||
|
||||
2. **Week 3-4:** TestKit Foundations (5100.0007.0002) ← **CRITICAL BOTTLENECK**
|
||||
- ALL downstream sprints depend on TestKit
|
||||
- Must complete before any module tests can start
|
||||
- Priority: DeterministicTime, DeterministicRandom, CanonicalJsonAssert, SnapshotAssert, PostgresFixture, OtelCapture
|
||||
|
||||
3. **Week 5-6:** Epic Implementation (parallel zone)
|
||||
- 5 sprints run concurrently
|
||||
- Unblocks all module tests
|
||||
|
||||
4. **Week 7-10:** Module Test Implementation (parallel zone)
|
||||
- 11 sprints run concurrently
|
||||
- Longest pole: Scanner (25 tasks, 4 waves)
|
||||
|
||||
5. **Week 11-14:** Infrastructure Test Implementation (parallel zone)
|
||||
- 4 sprints run concurrently
|
||||
- Can overlap with late-stage module tests
|
||||
|
||||
### Critical Bottleneck: TestKit (Sprint 5100.0007.0002)
|
||||
**Impact:** Blocks 20 downstream sprints (all module + infrastructure tests)
|
||||
**Mitigation:**
|
||||
- Staff with 2-3 senior engineers
|
||||
- Prioritize DeterministicTime and SnapshotAssert (most commonly used)
|
||||
- Release incrementally (partial TestKit unlocks some modules)
|
||||
- Run daily check-ins to unblock consuming teams
|
||||
|
||||
---
|
||||
|
||||
## Dependency Matrix
|
||||
|
||||
### Epic → Module Dependencies
|
||||
|
||||
| Epic Sprint | Blocks Module Sprints | Reason |
|
||||
|-------------|----------------------|--------|
|
||||
| 5100.0007.0002 (TestKit) | ALL 15 module/infra sprints | Core test utilities required |
|
||||
| 5100.0007.0003 (Determinism) | Scanner, Excititor, Signer, CLI, AirGap | Determinism gate required |
|
||||
| 5100.0007.0004 (Storage) | Scanner, Concelier, Excititor, Policy, Scheduler, Notify, EvidenceLocker, Graph, Router, AirGap | PostgresFixture required |
|
||||
| 5100.0007.0005 (Connectors) | Concelier, Excititor, Authority, Signer, Notify | Fixture discipline required |
|
||||
| 5100.0007.0006 (WebService) | Scanner, Concelier, Excititor, Policy, Authority, Signer, Attestor, Scheduler, Notify, UI, EvidenceLocker, Graph, AirGap | WebServiceFixture required |
|
||||
| 5100.0007.0007 (Architecture) | Concelier, Excititor | Lattice boundary enforcement |
|
||||
|
||||
### Module → Module Dependencies
|
||||
|
||||
| Sprint | Depends On Other Modules | Reason |
|
||||
|--------|--------------------------|--------|
|
||||
| Attestor (0009.0007) | Signer (0009.0006) | Sign/verify integration tests |
|
||||
| (None other) | - | Modules are otherwise independent |
|
||||
|
||||
---
|
||||
|
||||
## Parallelization Strategy
|
||||
|
||||
### Maximum Parallel Execution (15 sprints)
|
||||
|
||||
**Week 5-6 (5 parallel):**
|
||||
- Determinism (2 eng)
|
||||
- Storage (3 eng)
|
||||
- Connectors (2 eng)
|
||||
- WebService (2 eng)
|
||||
- Architecture (1 eng)
|
||||
|
||||
**Week 7-10 (11 parallel):**
|
||||
- Scanner (3 eng) ← longest pole
|
||||
- Concelier (2 eng)
|
||||
- Excititor (2 eng)
|
||||
- Policy (2 eng)
|
||||
- Authority (2 eng)
|
||||
- Signer (2 eng)
|
||||
- Attestor (2 eng)
|
||||
- Scheduler (2 eng)
|
||||
- Notify (2 eng)
|
||||
- CLI (1 eng)
|
||||
- UI (2 eng)
|
||||
|
||||
**Week 11-14 (4 parallel):**
|
||||
- EvidenceLocker (2 eng)
|
||||
- Graph (2 eng)
|
||||
- Router/Messaging (2 eng)
|
||||
- AirGap (2 eng)
|
||||
|
||||
**Resource Requirement (Peak):**
|
||||
- Week 7-10: 22 engineers (11 sprints × avg 2 eng/sprint)
|
||||
- Realistic: 10-12 engineers with staggered starts
|
||||
|
||||
---
|
||||
|
||||
## Risk Hotspots
|
||||
|
||||
### High-Impact Delays
|
||||
|
||||
| Risk | Impact | Probability | Mitigation |
|
||||
|------|--------|-------------|------------|
|
||||
| TestKit delayed (5100.0007.0002) | Blocks ALL downstream sprints; +2-4 weeks delay | MEDIUM | Staff with senior engineers; daily standups; incremental releases |
|
||||
| Storage harness issues (5100.0007.0004) | Blocks 10 sprints | MEDIUM | Use Testcontainers early; validate Postgres 16 compatibility Week 1 |
|
||||
| Determinism gate drift (5100.0007.0003) | Scanner/Excititor blocked; compliance issues | LOW | Explicit canonical JSON contract; freeze schema early |
|
||||
| Attestor-Signer circular dependency (0009.0007 ↔ 0009.0006) | Integration tests blocked | MEDIUM | Mock signing for Attestor initial tests; coordinate guilds |
|
||||
| Concurrent module tests overwhelm CI | Test suite timeout; flaky tests | HIGH | Stagger module test starts; use CI parallelization; dedicated test runners |
|
||||
|
||||
### Critical Path Risks
|
||||
|
||||
| Sprint | Risk | Impact if Delayed |
|
||||
|--------|------|-------------------|
|
||||
| 5100.0007.0002 (TestKit) | DeterministicTime implementation complex | +2 weeks to critical path |
|
||||
| 5100.0009.0001 (Scanner) | 25 tasks, 4 waves; reachability tests complex | Delays integration tests; no impact on other modules |
|
||||
| 5100.0007.0004 (Storage) | Testcontainers Postgres compatibility issues | Blocks 10 sprints; +2-3 weeks |
|
||||
|
||||
---
|
||||
|
||||
## Recommended Execution Sequence
|
||||
|
||||
### Phase 1: Foundation (Weeks 1-4)
|
||||
**Goal:** Establish test infrastructure and strategy docs
|
||||
**Sprints:**
|
||||
1. 5100.0007.0001 (Master Strategy) — Week 1-2
|
||||
2. 5100.0007.0002 (TestKit) — Week 3-4 ← CRITICAL
|
||||
|
||||
**Exit Criteria:**
|
||||
- TestKit utilities available (DeterministicTime, SnapshotAssert, PostgresFixture, OtelCapture)
|
||||
- Test runner scripts operational
|
||||
- Trait standardization complete
|
||||
|
||||
### Phase 2: Epic Implementation (Weeks 5-6)
|
||||
**Goal:** Implement all foundation epics in parallel
|
||||
**Sprints (parallel):**
|
||||
1. 5100.0007.0003 (Determinism)
|
||||
2. 5100.0007.0004 (Storage)
|
||||
3. 5100.0007.0005 (Connectors)
|
||||
4. 5100.0007.0006 (WebService)
|
||||
5. 5100.0007.0007 (Architecture)
|
||||
|
||||
**Exit Criteria:**
|
||||
- PostgresFixture operational (Testcontainers)
|
||||
- Determinism manifest format defined
|
||||
- Connector fixture discipline documented
|
||||
- WebServiceFixture operational
|
||||
- Architecture tests in CI (PR gate)
|
||||
|
||||
### Phase 3: Module Tests — Priority Tier 1 (Weeks 7-8)
|
||||
**Goal:** Implement tests for critical security/compliance modules
|
||||
**Sprints (parallel):**
|
||||
1. 5100.0009.0001 (Scanner) — critical path, longest pole
|
||||
2. 5100.0009.0002 (Concelier)
|
||||
3. 5100.0009.0003 (Excititor)
|
||||
4. 5100.0009.0004 (Policy)
|
||||
5. 5100.0009.0005 (Authority)
|
||||
6. 5100.0009.0006 (Signer)
|
||||
|
||||
### Phase 4: Module Tests — Priority Tier 2 (Weeks 9-10)
|
||||
**Goal:** Complete remaining module tests
|
||||
**Sprints (parallel):**
|
||||
1. 5100.0009.0007 (Attestor)
|
||||
2. 5100.0009.0008 (Scheduler)
|
||||
3. 5100.0009.0009 (Notify)
|
||||
4. 5100.0009.0010 (CLI)
|
||||
5. 5100.0009.0011 (UI)
|
||||
|
||||
### Phase 5: Infrastructure Tests (Weeks 11-14)
|
||||
**Goal:** Complete platform infrastructure tests
|
||||
**Sprints (parallel):**
|
||||
1. 5100.0010.0001 (EvidenceLocker)
|
||||
2. 5100.0010.0002 (Graph)
|
||||
3. 5100.0010.0003 (Router/Messaging)
|
||||
4. 5100.0010.0004 (AirGap)
|
||||
|
||||
### Phase 6: Quality Gates (Overlapping Weeks 3-14)
|
||||
**Goal:** Establish ongoing parity testing
|
||||
**Sprint:**
|
||||
1. 5100.0008.0001 (Competitor Parity) — can start Week 3, run nightly thereafter
|
||||
|
||||
---
|
||||
|
||||
## Guild Coordination
|
||||
|
||||
### Cross-Guild Dependencies
|
||||
|
||||
| Guild | Owns Sprints | Depends On Guilds | Coordination Points |
|
||||
|-------|--------------|-------------------|---------------------|
|
||||
| Platform Guild | TestKit, Storage, Architecture, EvidenceLocker, Graph, Router | None | Week 3: TestKit readiness review |
|
||||
| Scanner Guild | Scanner | Platform (TestKit, Storage, Determinism, WebService) | Week 5: Storage harness validation |
|
||||
| Concelier Guild | Concelier | Platform (TestKit, Storage, Connectors, WebService), Architecture | Week 6: Connector fixture review |
|
||||
| Excititor Guild | Excititor | Platform (TestKit, Storage, Connectors, WebService), Architecture | Week 6: Preserve-prune test design |
|
||||
| Policy Guild | Policy, AirGap (analyzers) | Platform (TestKit, Storage, WebService) | Week 7: Unknown budget enforcement review |
|
||||
| Authority Guild | Authority | Platform (TestKit, Connectors, WebService) | Week 7: OIDC connector fixture validation |
|
||||
| Crypto Guild | Signer, Attestor | Platform (TestKit, Determinism, Connectors, WebService), Authority | Week 8: Canonical payload design; Week 9: Sign/verify integration |
|
||||
| Scheduler Guild | Scheduler | Platform (TestKit, Storage, WebService) | Week 9: DeterministicTime validation |
|
||||
| Notify Guild | Notify | Platform (TestKit, Storage, Connectors, WebService) | Week 9: Connector fixture templates |
|
||||
| CLI Guild | CLI | Platform (TestKit, Determinism) | Week 10: Exit code conventions |
|
||||
| UI Guild | UI | Platform (TestKit, WebService) | Week 10: API contract snapshot review |
|
||||
| AirGap Guild | AirGap | Platform (TestKit, Determinism, Storage, WebService), Policy | Week 11: Bundle determinism validation |
|
||||
| QA Guild | Competitor Parity | Platform (TestKit) | Week 3: Parity harness design |
|
||||
|
||||
### Weekly Sync Schedule
|
||||
|
||||
**Week 1-2:**
|
||||
- All guilds: Master strategy review, sprint assignment
|
||||
|
||||
**Week 3-4:**
|
||||
- Platform Guild: Daily standup (TestKit unblocking)
|
||||
- All guilds: TestKit API review (design feedback)
|
||||
|
||||
**Week 5-6:**
|
||||
- Epic guilds: Bi-weekly sync (Determinism, Storage, Connectors, WebService, Architecture)
|
||||
- Scanner/Concelier/Excititor guilds: Prepare for module test sprint kickoff
|
||||
|
||||
**Week 7-10:**
|
||||
- All module guilds: Weekly guild-specific standups
|
||||
- Cross-guild: Bi-weekly integration sync (Signer ↔ Attestor, Policy ↔ AirGap)
|
||||
|
||||
**Week 11-14:**
|
||||
- Infrastructure guilds: Weekly sync
|
||||
- All guilds: Bi-weekly retrospective
|
||||
|
||||
---
|
||||
|
||||
## Metrics & Tracking
|
||||
|
||||
### Sprint Completion Metrics
|
||||
|
||||
| Metric | Target | Measurement |
|
||||
|--------|--------|-------------|
|
||||
| Sprint on-time completion | >80% | Tasks complete by wave deadline |
|
||||
| Test coverage increase | +30% per module | Code coverage reports |
|
||||
| Determinism tests passing | 100% | Determinism gate CI job |
|
||||
| Contract tests in CI | 100% of WebServices | Contract lane CI job |
|
||||
| Architecture tests enforcing | 100% violations blocked | Architecture test failures = build failures |
|
||||
|
||||
### Quality Gates
|
||||
|
||||
| Gate | Criteria | Enforced By |
|
||||
|------|----------|-------------|
|
||||
| Determinism | SHA-256 hash stable across runs | Sprint 5100.0007.0003 tests |
|
||||
| Contract Stability | OpenAPI schema unchanged or explicitly versioned | Sprint 5100.0007.0006 tests |
|
||||
| Architecture Boundaries | Concelier/Excititor do NOT reference Scanner lattice | Sprint 5100.0007.0007 tests |
|
||||
| Preserve-Prune Source | Excititor preserves VEX source references and rationale | Sprint 5100.0009.0003 tests |
|
||||
| Unknown Budget Enforcement | Policy engine fails when unknowns > N | Sprint 5100.0009.0004 tests |
|
||||
|
||||
---
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Week 1 (2026-01-06):**
|
||||
- Kick off Sprint 5100.0007.0001 (Master Strategy)
|
||||
- Assign Platform Guild to TestKit (5100.0007.0002) for Week 3 start
|
||||
- Review this dependency graph with all guild leads
|
||||
|
||||
2. **Week 2 (2026-01-13):**
|
||||
- Complete Master Strategy Wave 1-2
|
||||
- Finalize TestKit API design (DeterministicTime, SnapshotAssert, etc.)
|
||||
|
||||
3. **Week 3 (2026-01-20):**
|
||||
- Start TestKit implementation (CRITICAL PATH)
|
||||
- Daily standup for TestKit unblocking
|
||||
- Prepare Epic sprint kickoffs for Week 5
|
||||
|
||||
4. **Week 4 (2026-01-27):**
|
||||
- Complete TestKit Wave 1-2 (DeterministicTime, SnapshotAssert)
|
||||
- Validate TestKit with pilot tests
|
||||
- Final Epic sprint preparation
|
||||
|
||||
5. **Week 5 (2026-02-03):**
|
||||
- Kick off 5 Epic sprints in parallel
|
||||
- Weekly Epic sync meeting (Fridays)
|
||||
|
||||
---
|
||||
|
||||
**Prepared by:** Project Management
|
||||
**Date:** 2025-12-23
|
||||
**Next Review:** 2026-01-06 (Week 1 kickoff)
|
||||
517
docs/technical/testing/SPRINT_EXECUTION_PLAYBOOK.md
Normal file
517
docs/technical/testing/SPRINT_EXECUTION_PLAYBOOK.md
Normal file
@@ -0,0 +1,517 @@
|
||||
# Testing Strategy Sprint Execution Playbook
|
||||
|
||||
> **Purpose:** Practical guide for executing testing sprints - coordination, Definition of Done, sign-off criteria, ceremonies, and troubleshooting.
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
1. [Sprint Lifecycle](#sprint-lifecycle)
|
||||
2. [Definition of Done (DoD)](#definition-of-done-dod)
|
||||
3. [Wave-Based Execution](#wave-based-execution)
|
||||
4. [Sign-Off Criteria](#sign-off-criteria)
|
||||
5. [Cross-Guild Coordination](#cross-guild-coordination)
|
||||
6. [Common Failure Patterns](#common-failure-patterns)
|
||||
7. [Troubleshooting Guide](#troubleshooting-guide)
|
||||
8. [Sprint Templates](#sprint-templates)
|
||||
|
||||
---
|
||||
|
||||
## Sprint Lifecycle
|
||||
|
||||
### Sprint States
|
||||
|
||||
```
|
||||
TODO → DOING → BLOCKED/IN_REVIEW → DONE
|
||||
│ │ │ │
|
||||
│ │ │ └─ All waves complete + sign-off
|
||||
│ │ └─ Waiting on dependency or approval
|
||||
│ └─ Active development (1+ waves in progress)
|
||||
└─ Not yet started
|
||||
```
|
||||
|
||||
### Standard Sprint Duration
|
||||
|
||||
- **Foundation Epics (5100.0007.*):** 2 weeks per sprint
|
||||
- **Module Tests (5100.0009.*):** 2 weeks per sprint
|
||||
- **Infrastructure Tests (5100.0010.*):** 2 weeks per sprint
|
||||
- **Competitor Parity (5100.0008.0001):** Initial setup 2 weeks; then ongoing (nightly/weekly)
|
||||
|
||||
### Ceremonies
|
||||
|
||||
#### Sprint Kickoff (Day 1)
|
||||
**Who:** Sprint owner + guild members + dependencies
|
||||
**Duration:** 60 min
|
||||
**Agenda:**
|
||||
1. Review sprint scope and deliverables (10 min)
|
||||
2. Review wave structure and task breakdown (15 min)
|
||||
3. Identify dependencies and blockers (15 min)
|
||||
4. Assign tasks to engineers (10 min)
|
||||
5. Schedule wave reviews (5 min)
|
||||
6. Q&A (5 min)
|
||||
|
||||
#### Wave Review (End of each wave)
|
||||
**Who:** Sprint owner + guild members
|
||||
**Duration:** 30 min
|
||||
**Agenda:**
|
||||
1. Demo completed tasks (10 min)
|
||||
2. Review DoD checklist for wave (10 min)
|
||||
3. Identify blockers for next wave (5 min)
|
||||
4. Update sprint status in `Delivery Tracker` (5 min)
|
||||
|
||||
#### Sprint Sign-Off (Final day)
|
||||
**Who:** Sprint owner + guild lead + architect (for critical sprints)
|
||||
**Duration:** 30 min
|
||||
**Agenda:**
|
||||
1. Review all wave completion (10 min)
|
||||
2. Verify sign-off criteria (10 min)
|
||||
3. Demo integration (if applicable) (5 min)
|
||||
4. Sign execution log (5 min)
|
||||
|
||||
#### Weekly Sync (Every Friday)
|
||||
**Who:** All active sprint owners + project manager
|
||||
**Duration:** 30 min
|
||||
**Agenda:**
|
||||
1. Sprint status updates (15 min)
|
||||
2. Blocker escalation (10 min)
|
||||
3. Next week preview (5 min)
|
||||
|
||||
---
|
||||
|
||||
## Definition of Done (DoD)
|
||||
|
||||
### Universal DoD (Applies to ALL sprints)
|
||||
|
||||
✅ **Code:**
|
||||
- [ ] All tasks in `Delivery Tracker` marked as `DONE`
|
||||
- [ ] Code reviewed by at least 1 other engineer
|
||||
- [ ] No pending TODOs or FIXMEs in committed code
|
||||
- [ ] Code follows StellaOps coding standards (SOLID, DRY, KISS)
|
||||
|
||||
✅ **Tests:**
|
||||
- [ ] All tests passing locally
|
||||
- [ ] All tests passing in CI (appropriate lane)
|
||||
- [ ] Code coverage increase ≥ target (see module-specific DoD)
|
||||
- [ ] No flaky tests (deterministic pass rate 100%)
|
||||
|
||||
✅ **Documentation:**
|
||||
- [ ] Sprint `Execution Log` updated with completion date
|
||||
- [ ] Module-specific `AGENTS.md` updated (if new patterns introduced)
|
||||
- [ ] API documentation updated (if endpoints changed)
|
||||
|
||||
✅ **Integration:**
|
||||
- [ ] Changes merged to `main` branch
|
||||
- [ ] CI lanes passing (Unit, Contract, Integration, Security as applicable)
|
||||
- [ ] No regressions introduced (existing tests still passing)
|
||||
|
||||
---
|
||||
|
||||
### Model-Specific DoD
|
||||
|
||||
#### L0 (Library/Core)
|
||||
- [ ] Unit tests covering all public methods
|
||||
- [ ] Property tests for key invariants (where applicable)
|
||||
- [ ] Snapshot tests for canonical outputs (SBOM, VEX, verdicts, etc.)
|
||||
- [ ] Code coverage: ≥80% for core libraries
|
||||
|
||||
#### S1 (Storage/Postgres)
|
||||
- [ ] Migration tests (apply from scratch, apply from N-1) passing
|
||||
- [ ] Idempotency tests passing (same operation twice → no duplicates)
|
||||
- [ ] Query determinism tests passing (explicit ORDER BY checks)
|
||||
- [ ] Testcontainers Postgres fixture operational
|
||||
|
||||
#### T1 (Transport/Queue)
|
||||
- [ ] Protocol roundtrip tests passing
|
||||
- [ ] Fuzz tests for invalid input passing
|
||||
- [ ] Delivery semantics tests passing (at-least-once, idempotency)
|
||||
- [ ] Backpressure tests passing
|
||||
|
||||
#### C1 (Connector/External)
|
||||
- [ ] Fixture folders created (`Fixtures/<source>/<case>.json`, `Expected/<case>.canonical.json`)
|
||||
- [ ] Parser tests passing (fixture → parse → snapshot)
|
||||
- [ ] Resilience tests passing (missing fields, invalid enums, etc.)
|
||||
- [ ] Security tests passing (URL allowlist, redirect handling, payload limits)
|
||||
|
||||
#### W1 (WebService/API)
|
||||
- [ ] Contract tests passing (OpenAPI snapshot validation)
|
||||
- [ ] Auth/authz tests passing (deny-by-default, token expiry, scope enforcement)
|
||||
- [ ] OTel trace assertions passing (spans emitted, tags present)
|
||||
- [ ] Negative tests passing (malformed requests, size limits, method mismatch)
|
||||
|
||||
#### WK1 (Worker/Indexer)
|
||||
- [ ] End-to-end tests passing (enqueue → worker → stored → events emitted)
|
||||
- [ ] Retry tests passing (transient failure → backoff; permanent → poison)
|
||||
- [ ] Idempotency tests passing (same job twice → single execution)
|
||||
- [ ] OTel correlation tests passing (trace spans across lifecycle)
|
||||
|
||||
#### AN1 (Analyzer/SourceGen)
|
||||
- [ ] Roslyn compilation tests passing (expected diagnostics, no false positives)
|
||||
- [ ] Golden generated code tests passing (if applicable)
|
||||
|
||||
#### CLI1 (Tool/CLI)
|
||||
- [ ] Exit code tests passing (0=success, 1=user error, 2=system error, etc.)
|
||||
- [ ] Golden output tests passing (stdout/stderr snapshots)
|
||||
- [ ] Determinism tests passing (same inputs → same outputs)
|
||||
|
||||
#### PERF (Benchmarks)
|
||||
- [ ] Benchmark tests operational
|
||||
- [ ] Perf smoke tests in CI (2× regression gate)
|
||||
- [ ] Baseline results documented
|
||||
|
||||
---
|
||||
|
||||
### Sprint-Specific DoD
|
||||
|
||||
#### Foundation Epic Sprints (5100.0007.*)
|
||||
|
||||
**Epic A (TestKit):**
|
||||
- [ ] `StellaOps.TestKit` NuGet package published internally
|
||||
- [ ] DeterministicTime, DeterministicRandom, CanonicalJsonAssert, SnapshotAssert, PostgresFixture, ValkeyFixture, OtelCapture, HttpFixtureServer all operational
|
||||
- [ ] Pilot adoption in 2+ modules (e.g., Scanner, Concelier)
|
||||
|
||||
**Epic B (Determinism):**
|
||||
- [ ] Determinism manifest JSON schema defined
|
||||
- [ ] `tests/integration/StellaOps.Integration.Determinism` expanded for SBOM, VEX, policy verdicts, evidence bundles, AirGap exports
|
||||
- [ ] Determinism tests in CI (merge gate)
|
||||
- [ ] Determinism artifacts stored in CI artifact repo
|
||||
|
||||
**Epic C (Storage):**
|
||||
- [ ] PostgresFixture operational (Testcontainers, automatic migrations, schema isolation)
|
||||
- [ ] ValkeyFixture operational
|
||||
- [ ] Pilot adoption in 2+ modules with S1 model (e.g., Scanner, Policy)
|
||||
|
||||
**Epic D (Connectors):**
|
||||
- [ ] Connector fixture discipline documented in `docs/testing/connector-fixture-discipline.md`
|
||||
- [ ] FixtureUpdater tool operational (with `UPDATE_CONNECTOR_FIXTURES=1` env var guard)
|
||||
- [ ] Pilot adoption in Concelier.Connector.NVD
|
||||
|
||||
**Epic E (WebService):**
|
||||
- [ ] WebServiceFixture<TProgram> operational (Microsoft.AspNetCore.Mvc.Testing)
|
||||
- [ ] Contract test pattern documented
|
||||
- [ ] Pilot adoption in Scanner.WebService
|
||||
|
||||
**Epic F (Architecture):**
|
||||
- [ ] `tests/architecture/StellaOps.Architecture.Tests` project operational
|
||||
- [ ] Lattice placement rules enforced (Concelier/Excititor must NOT reference Scanner lattice)
|
||||
- [ ] Architecture tests in CI (PR gate, Unit lane)
|
||||
|
||||
#### Module Test Sprints (5100.0009.*)
|
||||
|
||||
**Per Module:**
|
||||
- [ ] All model requirements from TEST_CATALOG.yml satisfied
|
||||
- [ ] Module-specific quality gates passing (see TEST_COVERAGE_MATRIX.md)
|
||||
- [ ] Code coverage increase: ≥30% from baseline
|
||||
- [ ] All wave deliverables complete
|
||||
|
||||
#### Infrastructure Test Sprints (5100.0010.*)
|
||||
|
||||
**Per Infrastructure Module:**
|
||||
- [ ] All integration tests passing
|
||||
- [ ] Cross-module dependencies validated (e.g., EvidenceLocker ↔ Scanner)
|
||||
|
||||
---
|
||||
|
||||
## Wave-Based Execution
|
||||
|
||||
### Wave Structure
|
||||
|
||||
Most sprints use a 3-4 wave structure:
|
||||
- **Wave 1:** Foundation / Core logic
|
||||
- **Wave 2:** Integration / Storage / Connectors
|
||||
- **Wave 3:** WebService / Workers / End-to-end
|
||||
- **Wave 4:** (Optional) Polish / Documentation / Edge cases
|
||||
|
||||
### Wave Execution Pattern
|
||||
|
||||
```
|
||||
Week 1:
|
||||
Day 1-2: Wave 1 development
|
||||
Day 3: Wave 1 review → APPROVED → proceed to Wave 2
|
||||
Day 4-5: Wave 2 development
|
||||
|
||||
Week 2:
|
||||
Day 1: Wave 2 review → APPROVED → proceed to Wave 3
|
||||
Day 2-4: Wave 3 development
|
||||
Day 5: Wave 3 review + Sprint Sign-Off
|
||||
```
|
||||
|
||||
### Wave Review Checklist
|
||||
|
||||
✅ **Before Wave Review:**
|
||||
- [ ] All tasks in wave marked as `DOING` → `DONE` in `Delivery Tracker`
|
||||
- [ ] All tests for wave passing in CI
|
||||
- [ ] Code reviewed
|
||||
|
||||
✅ **During Wave Review:**
|
||||
- [ ] Demo completed functionality
|
||||
- [ ] Review wave DoD checklist
|
||||
- [ ] Identify blockers for next wave
|
||||
- [ ] **Sign-off decision:** APPROVED / CHANGES_REQUIRED / BLOCKED
|
||||
|
||||
✅ **After Wave Review:**
|
||||
- [ ] Update sprint `Execution Log` with wave completion
|
||||
- [ ] Update task status in `Delivery Tracker`
|
||||
- [ ] If BLOCKED: escalate to project manager
|
||||
|
||||
---
|
||||
|
||||
## Sign-Off Criteria
|
||||
|
||||
### Sprint Sign-Off Levels
|
||||
|
||||
#### Level 1: Self-Sign-Off (Guild Lead)
|
||||
**Applies to:** Routine module test sprints without architectural changes
|
||||
**Criteria:**
|
||||
- All waves complete
|
||||
- All DoD items checked
|
||||
- Guild lead approval
|
||||
|
||||
#### Level 2: Architect Sign-Off
|
||||
**Applies to:** Foundation epics, architectural changes, cross-cutting concerns
|
||||
**Criteria:**
|
||||
- All waves complete
|
||||
- All DoD items checked
|
||||
- Guild lead approval
|
||||
- **Architect review and approval**
|
||||
|
||||
#### Level 3: Project Manager + Architect Sign-Off
|
||||
**Applies to:** Critical path sprints (TestKit, Determinism, Storage)
|
||||
**Criteria:**
|
||||
- All waves complete
|
||||
- All DoD items checked
|
||||
- Guild lead approval
|
||||
- Architect approval
|
||||
- **Project manager approval (validates dependencies unblocked)**
|
||||
|
||||
### Sign-Off Process
|
||||
|
||||
1. **Engineer completes final wave** → marks all tasks `DONE`
|
||||
2. **Guild lead reviews** → verifies DoD checklist
|
||||
3. **Sprint owner schedules sign-off meeting** (if Level 2/3)
|
||||
4. **Sign-off meeting** (30 min):
|
||||
- Demo final deliverables
|
||||
- Review DoD checklist
|
||||
- Verify integration (if applicable)
|
||||
- **Decision:** APPROVED / CHANGES_REQUIRED
|
||||
5. **Update Execution Log:**
|
||||
```markdown
|
||||
| 2026-XX-XX | Sprint signed off by [Guild Lead / Architect / PM]. | [Owner] |
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Cross-Guild Coordination
|
||||
|
||||
### Dependency Handoffs
|
||||
|
||||
When Sprint A depends on Sprint B:
|
||||
|
||||
**Sprint B (Provider):**
|
||||
1. **Week before completion:** Notify Sprint A owner of expected completion date
|
||||
2. **Wave 2-3 complete:** Provide preview build / early access to Sprint A
|
||||
3. **Sprint complete:** Formally notify Sprint A owner; provide integration guide
|
||||
|
||||
**Sprint A (Consumer):**
|
||||
1. **Sprint B Wave 2:** Begin integration planning; identify integration risks
|
||||
2. **Sprint B Wave 3:** Start integration development (against preview build)
|
||||
3. **Sprint B complete:** Complete integration; validate against final build
|
||||
|
||||
### Coordination Meetings
|
||||
|
||||
#### Epic → Module Handoff (Week 5)
|
||||
**Who:** Epic sprint owners + all module sprint owners
|
||||
**Duration:** 60 min
|
||||
**Agenda:**
|
||||
1. Epic deliverables review (TestKit, Storage, etc.) (20 min)
|
||||
2. Integration guide walkthrough (15 min)
|
||||
3. Module sprint kickoff previews (15 min)
|
||||
4. Q&A (10 min)
|
||||
|
||||
#### Module Integration Sync (Bi-weekly, Weeks 7-10)
|
||||
**Who:** Module sprint owners with cross-dependencies (e.g., Signer ↔ Attestor)
|
||||
**Duration:** 30 min
|
||||
**Agenda:**
|
||||
1. Integration status updates (10 min)
|
||||
2. Blocker resolution (15 min)
|
||||
3. Next steps (5 min)
|
||||
|
||||
### Blocked Sprint Protocol
|
||||
|
||||
If a sprint is BLOCKED:
|
||||
|
||||
1. **Sprint owner:** Update sprint status to `BLOCKED` in `Delivery Tracker`
|
||||
2. **Sprint owner:** Add blocker note to `Decisions & Risks` table
|
||||
3. **Sprint owner:** Notify project manager immediately (Slack + email)
|
||||
4. **Project manager:** Schedule blocker resolution meeting within 24 hours
|
||||
5. **Resolution meeting:** Decide:
|
||||
- **Workaround:** Continue with mock/stub dependency
|
||||
- **Re-sequence:** Defer sprint; start alternative sprint
|
||||
- **Escalate:** Assign additional resources to unblock dependency
|
||||
|
||||
---
|
||||
|
||||
## Common Failure Patterns
|
||||
|
||||
### Pattern 1: Testcontainers Failure (Storage Harness)
|
||||
|
||||
**Symptom:** Tests fail with "Docker not running" or "Container startup timeout"
|
||||
|
||||
**Root Cause:** Docker daemon not running, Docker Desktop not installed, or Testcontainers compatibility issue
|
||||
|
||||
**Fix:**
|
||||
1. Verify Docker Desktop installed and running
|
||||
2. Verify Testcontainers.Postgres version compatible with .NET 10
|
||||
3. Add explicit timeout: `new PostgreSqlBuilder().WithStartupTimeout(TimeSpan.FromMinutes(5))`
|
||||
4. For CI: ensure Docker available in CI runner environment
|
||||
|
||||
### Pattern 2: Determinism Test Drift
|
||||
|
||||
**Symptom:** Determinism tests fail with "Expected hash X, got hash Y"
|
||||
|
||||
**Root Cause:** Non-deterministic timestamps, GUIDs, or ordering
|
||||
|
||||
**Fix:**
|
||||
1. Use `DeterministicTime` instead of `DateTime.UtcNow`
|
||||
2. Use `DeterministicRandom` for random data
|
||||
3. Explicit `ORDER BY` clauses in all queries
|
||||
4. Strip timestamps from snapshots or use placeholders
|
||||
|
||||
### Pattern 3: Fixture Update Breaks Tests
|
||||
|
||||
**Symptom:** Connector tests fail after updating fixtures
|
||||
|
||||
**Root Cause:** Upstream schema drift (NVD, OSV, etc.)
|
||||
|
||||
**Fix:**
|
||||
1. Review schema changes in upstream source
|
||||
2. Update connector parser logic if needed
|
||||
3. Regenerate expected snapshots with `UPDATE_CONNECTOR_FIXTURES=1`
|
||||
4. Document schema version in fixture filename (e.g., `nvd_v1.1.json`)
|
||||
|
||||
### Pattern 4: WebService Contract Drift
|
||||
|
||||
**Symptom:** Contract tests fail with "OpenAPI schema mismatch"
|
||||
|
||||
**Root Cause:** Backend API schema changed (breaking change)
|
||||
|
||||
**Fix:**
|
||||
1. Review API changes in backend PR
|
||||
2. **If breaking:** Version API (e.g., `/api/v2/...`)
|
||||
3. **If non-breaking:** Update contract snapshot
|
||||
4. Coordinate with frontend/consumer teams
|
||||
|
||||
### Pattern 5: Circular Dependency (Attestor ↔ Signer)
|
||||
|
||||
**Symptom:** Integration tests blocked waiting for both Attestor and Signer
|
||||
|
||||
**Root Cause:** Attestor needs Signer for signing; Signer integration tests need Attestor
|
||||
|
||||
**Fix:**
|
||||
1. **Signer Sprint (5100.0009.0006):** Use mock signing for initial tests; defer Attestor integration
|
||||
2. **Attestor Sprint (5100.0009.0007):** Coordinate with Signer guild; run integration tests in Week 2
|
||||
3. **Integration Sprint (post-module):** Full Attestor ↔ Signer integration validation
|
||||
|
||||
### Pattern 6: Flaky Tests (Timing Issues)
|
||||
|
||||
**Symptom:** Tests pass locally but fail intermittently in CI
|
||||
|
||||
**Root Cause:** Race conditions, sleeps, non-deterministic timing
|
||||
|
||||
**Fix:**
|
||||
1. Use `DeterministicTime` instead of `Thread.Sleep` or `Task.Delay`
|
||||
2. Use explicit waits (e.g., `await condition.UntilAsync(...)`) instead of fixed delays
|
||||
3. Avoid hard-coded timeouts; use configurable timeouts
|
||||
4. Run tests 10× locally to verify determinism
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting Guide
|
||||
|
||||
### Issue: "My sprint depends on Epic X, but Epic X is delayed"
|
||||
|
||||
**Solution:**
|
||||
1. Check if partial Epic X deliverables available (e.g., TestKit Wave 1-2 complete → can start L0 tests)
|
||||
2. If not, use mock/stub implementation
|
||||
3. Coordinate with Epic X owner for preview build
|
||||
4. If critically blocked: escalate to project manager for re-sequencing
|
||||
|
||||
### Issue: "Tests passing locally but failing in CI"
|
||||
|
||||
**Checklist:**
|
||||
- [ ] Docker running in CI? (for Testcontainers)
|
||||
- [ ] Environment variables set? (e.g., `STELLAOPS_TEST_POSTGRES_CONNECTION`)
|
||||
- [ ] Correct .NET SDK version? (net10.0)
|
||||
- [ ] Test isolation? (each test resets state)
|
||||
- [ ] Deterministic? (run tests 10× locally)
|
||||
|
||||
### Issue: "Code coverage below target (80%)"
|
||||
|
||||
**Solution:**
|
||||
1. Identify uncovered lines: `dotnet test --collect:"XPlat Code Coverage"`
|
||||
2. Add unit tests for uncovered public methods
|
||||
3. Add property tests for invariants
|
||||
4. If coverage still low: review with guild lead (some boilerplate excluded from coverage)
|
||||
|
||||
### Issue: "Architecture tests failing (lattice boundary violation)"
|
||||
|
||||
**Solution:**
|
||||
1. Review failing types: which assembly is referencing Scanner lattice?
|
||||
2. **If legitimate:** Refactor to remove dependency (move logic to Scanner.WebService)
|
||||
3. **If test project:** Add to allowlist in architecture test
|
||||
|
||||
### Issue: "Snapshot test failing after refactor"
|
||||
|
||||
**Solution:**
|
||||
1. Review snapshot diff: is it intentional?
|
||||
2. **If intentional:** Update snapshot (re-run test with snapshot update flag)
|
||||
3. **If unintentional:** Revert refactor; investigate why output changed
|
||||
|
||||
---
|
||||
|
||||
## Sprint Templates
|
||||
|
||||
### Template: Task Status Update
|
||||
|
||||
```markdown
|
||||
## Delivery Tracker
|
||||
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| 1 | MODULE-5100-001 | DONE | None | John Doe | Add unit tests for... |
|
||||
| 2 | MODULE-5100-002 | DOING | Task 1 | Jane Smith | Add property tests for... |
|
||||
| 3 | MODULE-5100-003 | TODO | Task 2 | - | Add snapshot tests for... |
|
||||
```
|
||||
|
||||
### Template: Execution Log Entry
|
||||
|
||||
```markdown
|
||||
## Execution Log
|
||||
| Date (UTC) | Update | Owner |
|
||||
| --- | --- | --- |
|
||||
| 2026-01-20 | Sprint created. | Project Mgmt |
|
||||
| 2026-01-27 | Wave 1 complete (Tasks 1-5). | Guild Lead |
|
||||
| 2026-02-03 | Wave 2 complete (Tasks 6-10). | Guild Lead |
|
||||
| 2026-02-10 | Sprint signed off by Architect. | Project Mgmt |
|
||||
```
|
||||
|
||||
### Template: Blocker Note
|
||||
|
||||
```markdown
|
||||
## Decisions & Risks
|
||||
| Risk | Impact | Mitigation | Owner |
|
||||
| --- | --- | --- | --- |
|
||||
| [BLOCKER] TestKit delayed by 1 week | Cannot start module tests | Using mock TestKit for initial development; switch to real TestKit Week 5 | Module Guild |
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Week 1:** All guild leads review this playbook
|
||||
2. **Week 1:** Project manager schedules kickoff meetings for Foundation Epics (Week 3)
|
||||
3. **Week 2:** Epic sprint owners prepare kickoff materials (scope, wave breakdown, task assignments)
|
||||
4. **Week 3:** Foundation Epic sprints begin (5100.0007.0002-0007)
|
||||
|
||||
---
|
||||
|
||||
**Prepared by:** Project Management
|
||||
**Date:** 2025-12-23
|
||||
**Next Review:** 2026-01-06 (Week 1 kickoff)
|
||||
419
docs/technical/testing/TESTING_MASTER_PLAN.md
Normal file
419
docs/technical/testing/TESTING_MASTER_PLAN.md
Normal file
@@ -0,0 +1,419 @@
|
||||
# StellaOps Testing Strategy Master Plan (2026 H1)
|
||||
|
||||
> **Executive Summary:** Comprehensive 14-week testing initiative to establish model-driven test coverage across 15 modules, implement 6 foundation epics, and achieve 30%+ code coverage increase platform-wide.
|
||||
|
||||
---
|
||||
|
||||
## Document Control
|
||||
|
||||
| Attribute | Value |
|
||||
|-----------|-------|
|
||||
| **Program Name** | Testing Strategy Implementation 2026 H1 |
|
||||
| **Program ID** | SPRINT-5100 |
|
||||
| **Owner** | Project Management |
|
||||
| **Status** | PLANNING |
|
||||
| **Start Date** | 2026-01-06 (Week 1) |
|
||||
| **Target End Date** | 2026-04-14 (Week 14) |
|
||||
| **Budget** | TBD (resource model below) |
|
||||
| **Last Updated** | 2025-12-23 |
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
1. [Program Objectives](#program-objectives)
|
||||
2. [Scope & Deliverables](#scope--deliverables)
|
||||
3. [Timeline & Phases](#timeline--phases)
|
||||
4. [Resource Model](#resource-model)
|
||||
5. [Risk Register](#risk-register)
|
||||
6. [Success Metrics](#success-metrics)
|
||||
7. [Governance](#governance)
|
||||
8. [Communication Plan](#communication-plan)
|
||||
|
||||
---
|
||||
|
||||
## Program Objectives
|
||||
|
||||
### Primary Objectives
|
||||
|
||||
1. **Establish Model-Driven Testing:** Implement 9 test models (L0, S1, T1, C1, W1, WK1, AN1, CLI1, PERF) across 15 modules
|
||||
2. **Increase Code Coverage:** Achieve ≥30% code coverage increase from baseline (current ~40% → target 70%+)
|
||||
3. **Enforce Quality Gates:** Implement determinism, architecture, and module-specific quality gates
|
||||
4. **Build Test Infrastructure:** Deliver 6 foundation epics (TestKit, Determinism, Storage, Connectors, WebService, Architecture)
|
||||
5. **Enable CI/CD Confidence:** Establish PR-gating and merge-gating test lanes
|
||||
|
||||
### Secondary Objectives
|
||||
|
||||
1. **Reduce Test Flakiness:** Achieve 100% deterministic test pass rate (eliminate timing-based failures)
|
||||
2. **Improve Developer Experience:** Standardize test patterns, reduce test authoring friction
|
||||
3. **Establish Parity Monitoring:** Continuous validation against competitor tools (Syft, Grype, Trivy, Anchore)
|
||||
4. **Document Test Strategy:** Create comprehensive testing guides and playbooks
|
||||
|
||||
---
|
||||
|
||||
## Scope & Deliverables
|
||||
|
||||
### In-Scope
|
||||
|
||||
#### Foundation Epics (Batch 5100.0007, 90 tasks)
|
||||
| Sprint ID | Epic | Deliverables |
|
||||
|-----------|------|--------------|
|
||||
| 5100.0007.0001 | Master Testing Strategy | Strategy docs, test runner scripts, trait standardization, Epic sprint creation |
|
||||
| 5100.0007.0002 | TestKit Foundations | DeterministicTime, DeterministicRandom, CanonicalJsonAssert, SnapshotAssert, PostgresFixture, ValkeyFixture, OtelCapture, HttpFixtureServer |
|
||||
| 5100.0007.0003 | Determinism Gate | Determinism manifest format, expanded integration tests, CI artifact storage, drift detection |
|
||||
| 5100.0007.0004 | Storage Harness | PostgresFixture (Testcontainers), ValkeyFixture, automatic migrations, schema isolation |
|
||||
| 5100.0007.0005 | Connector Fixtures | Fixture discipline, FixtureUpdater tool, pilot adoption in Concelier.Connector.NVD |
|
||||
| 5100.0007.0006 | WebService Contract | WebServiceFixture<TProgram>, contract test pattern, pilot adoption in Scanner.WebService |
|
||||
| 5100.0007.0007 | Architecture Tests | NetArchTest.Rules, lattice placement enforcement, PR-gating architecture tests |
|
||||
|
||||
#### Module Test Implementations (Batch 5100.0009, 185 tasks)
|
||||
| Sprint ID | Module | Test Models | Deliverables |
|
||||
|-----------|--------|-------------|--------------|
|
||||
| 5100.0009.0001 | Scanner | L0, AN1, S1, T1, W1, WK1, PERF | 25 tasks: property tests, SBOM/reachability/verdict snapshots, determinism, WebService contract, Worker e2e, perf smoke |
|
||||
| 5100.0009.0002 | Concelier | C1, L0, S1, W1, AN1 | 18 tasks: connector fixtures (NVD/OSV/GHSA/CSAF), merge property tests, WebService contract, architecture enforcement |
|
||||
| 5100.0009.0003 | Excititor | C1, L0, S1, W1, WK1 | 21 tasks: connector fixtures (CSAF/OpenVEX), format export snapshots, preserve-prune tests, Worker e2e, architecture enforcement |
|
||||
| 5100.0009.0004 | Policy | L0, S1, W1 | 15 tasks: policy engine property tests, DSL roundtrip tests, verdict snapshots, unknown budget enforcement |
|
||||
| 5100.0009.0005 | Authority | L0, W1, C1 | 17 tasks: auth logic tests, connector fixtures (OIDC/SAML/LDAP), WebService contract, sign/verify integration |
|
||||
| 5100.0009.0006 | Signer | L0, W1, C1 | 17 tasks: canonical payload tests, crypto plugin tests (BouncyCastle/CryptoPro/eIDAS/SimRemote), WebService contract |
|
||||
| 5100.0009.0007 | Attestor | L0, W1 | 14 tasks: DSSE envelope tests, Rekor integration tests, attestation statement snapshots, WebService contract |
|
||||
| 5100.0009.0008 | Scheduler | L0, S1, W1, WK1 | 14 tasks: scheduling invariant property tests, storage idempotency, WebService contract, Worker e2e |
|
||||
| 5100.0009.0009 | Notify | L0, C1, S1, W1, WK1 | 18 tasks: connector fixtures (email/Slack/Teams/webhook), WebService contract, Worker e2e |
|
||||
| 5100.0009.0010 | CLI | CLI1 | 13 tasks: exit code tests, golden output tests, determinism tests |
|
||||
| 5100.0009.0011 | UI | W1 | 13 tasks: API contract tests, E2E smoke tests, accessibility tests |
|
||||
|
||||
#### Infrastructure Test Implementations (Batch 5100.0010, 62 tasks)
|
||||
| Sprint ID | Module Family | Deliverables |
|
||||
|-----------|---------------|--------------|
|
||||
| 5100.0010.0001 | EvidenceLocker + Findings + Replay | Immutability tests, ledger determinism, replay token security, WebService contract |
|
||||
| 5100.0010.0002 | Graph + TimelineIndexer | Graph construction/traversal tests, indexer e2e, query determinism, WebService contract |
|
||||
| 5100.0010.0003 | Router + Messaging | Transport compliance suite (in-memory/TCP/TLS/Valkey/RabbitMQ), routing determinism, fuzz tests |
|
||||
| 5100.0010.0004 | AirGap | Bundle export/import determinism, policy analyzer tests, WebService contract, CLI tool tests |
|
||||
|
||||
#### Quality Gates (Batch 5100.0008, 11 tasks)
|
||||
| Sprint ID | Purpose | Deliverables |
|
||||
|-----------|---------|--------------|
|
||||
| 5100.0008.0001 | Competitor Parity Testing | Parity test harness, fixture set (10-15 container images), comparison logic (SBOM/vuln/latency/errors), time-series storage, drift detection (>5% threshold) |
|
||||
|
||||
### Out-of-Scope
|
||||
|
||||
- ❌ **Performance optimization** (beyond PERF smoke tests for Scanner)
|
||||
- ❌ **UI/UX testing** (beyond W1 contract tests and E2E smoke tests)
|
||||
- ❌ **Load testing** (deferred to future sprint)
|
||||
- ❌ **Chaos engineering** (deferred to future sprint)
|
||||
- ❌ **Mobile/responsive testing** (not applicable - server-side platform)
|
||||
- ❌ **Penetration testing** (separate security initiative)
|
||||
|
||||
---
|
||||
|
||||
## Timeline & Phases
|
||||
|
||||
### Master Timeline (14 Weeks, 2026-01-06 to 2026-04-14)
|
||||
|
||||
```
|
||||
PHASE 1: FOUNDATION (Weeks 1-4)
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Week 1-2: Master Strategy (5100.0007.0001) │
|
||||
│ - Documentation sync │
|
||||
│ - Test runner scripts │
|
||||
│ - Trait standardization │
|
||||
│ - Epic sprint creation │
|
||||
│ │
|
||||
│ Week 3-4: TestKit Foundations (5100.0007.0002) ← CRITICAL │
|
||||
│ - DeterministicTime, DeterministicRandom │
|
||||
│ - CanonicalJsonAssert, SnapshotAssert │
|
||||
│ - PostgresFixture, ValkeyFixture, OtelCapture │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
|
||||
PHASE 2: EPIC IMPLEMENTATION (Weeks 5-6)
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Week 5-6: 5 Epic Sprints (PARALLEL) │
|
||||
│ - 5100.0007.0003 (Determinism Gate) │
|
||||
│ - 5100.0007.0004 (Storage Harness) │
|
||||
│ - 5100.0007.0005 (Connector Fixtures) │
|
||||
│ - 5100.0007.0006 (WebService Contract) │
|
||||
│ - 5100.0007.0007 (Architecture Tests) │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
|
||||
PHASE 3: MODULE TESTS - TIER 1 (Weeks 7-8)
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Week 7-8: 6 Module Sprints (PARALLEL) │
|
||||
│ - Scanner, Concelier, Excititor (core platform) │
|
||||
│ - Policy, Authority, Signer (security/compliance) │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
|
||||
PHASE 4: MODULE TESTS - TIER 2 (Weeks 9-10)
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Week 9-10: 5 Module Sprints (PARALLEL) │
|
||||
│ - Attestor, Scheduler, Notify (platform services) │
|
||||
│ - CLI, UI (client interfaces) │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
|
||||
PHASE 5: INFRASTRUCTURE TESTS (Weeks 11-14)
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Week 11-14: 4 Infrastructure Sprints (PARALLEL) │
|
||||
│ - EvidenceLocker, Graph, Router/Messaging, AirGap │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
|
||||
ONGOING: QUALITY GATES (Weeks 3-14+)
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Week 3: Competitor Parity harness setup │
|
||||
│ Week 4+: Nightly/weekly parity tests │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Critical Path (14 Weeks)
|
||||
|
||||
**Week 1-2:** Master Strategy → **Week 3-4:** TestKit ← **BOTTLENECK** → **Week 5-6:** Epic Implementation → **Week 7-10:** Module Tests → **Week 11-14:** Infrastructure Tests
|
||||
|
||||
**Critical Path Risks:**
|
||||
- TestKit delay → ALL downstream sprints blocked (+2-4 weeks)
|
||||
- Storage harness delay → 10 sprints blocked (+2-3 weeks)
|
||||
|
||||
### Milestones
|
||||
|
||||
| Milestone | Week | Deliverables | Sign-Off Criteria |
|
||||
|-----------|------|--------------|-------------------|
|
||||
| **M1: Foundation Ready** | Week 4 | TestKit operational | DeterministicTime, SnapshotAssert, PostgresFixture, OtelCapture available; pilot adoption in 2+ modules |
|
||||
| **M2: Epic Complete** | Week 6 | All 6 foundation epics complete | Determinism gate in CI; Storage harness operational; WebService contract tests in Scanner; Architecture tests PR-gating |
|
||||
| **M3: Core Modules Tested** | Week 8 | Scanner, Concelier, Excititor, Policy, Authority, Signer complete | Code coverage increase ≥30%; quality gates passing |
|
||||
| **M4: All Modules Tested** | Week 10 | All 11 module test sprints complete | All module-specific quality gates passing |
|
||||
| **M5: Program Complete** | Week 14 | All infrastructure tests complete; program retrospective | All sprints signed off; final metrics review |
|
||||
|
||||
---
|
||||
|
||||
## Resource Model
|
||||
|
||||
### Guild Allocation
|
||||
|
||||
| Guild | Assigned Sprints | Peak Staffing (Weeks 7-10) | Avg Sprint Ownership |
|
||||
|-------|------------------|----------------------------|----------------------|
|
||||
| **Platform Guild** | TestKit, Storage, Architecture, EvidenceLocker, Graph, Router | 10 engineers | 6 sprints |
|
||||
| **Scanner Guild** | Scanner | 3 engineers | 1 sprint |
|
||||
| **Concelier Guild** | Concelier | 2 engineers | 1 sprint |
|
||||
| **Excititor Guild** | Excititor | 2 engineers | 1 sprint |
|
||||
| **Policy Guild** | Policy, AirGap (analyzers) | 2-4 engineers | 2 sprints |
|
||||
| **Authority Guild** | Authority | 2 engineers | 1 sprint |
|
||||
| **Crypto Guild** | Signer, Attestor | 4 engineers | 2 sprints |
|
||||
| **Scheduler Guild** | Scheduler | 2 engineers | 1 sprint |
|
||||
| **Notify Guild** | Notify | 2 engineers | 1 sprint |
|
||||
| **CLI Guild** | CLI | 1 engineer | 1 sprint |
|
||||
| **UI Guild** | UI | 2 engineers | 1 sprint |
|
||||
| **AirGap Guild** | AirGap (core) | 2 engineers | 1 sprint |
|
||||
| **QA Guild** | Competitor Parity | 2 engineers | 1 sprint |
|
||||
|
||||
### Staffing Profile
|
||||
|
||||
**Peak Staffing (Weeks 7-10):** 22-26 engineers
|
||||
**Average Staffing (Weeks 1-14):** 12-16 engineers
|
||||
**Critical Path Sprints (TestKit, Storage):** 3-4 senior engineers each
|
||||
|
||||
### Resource Constraints
|
||||
|
||||
| Constraint | Impact | Mitigation |
|
||||
|------------|--------|------------|
|
||||
| Platform Guild oversubscribed (10 engineers, 6 sprints) | Burnout, delays | Stagger Epic sprints (Storage Week 5, Connectors Week 6); hire contractors for Weeks 5-10 |
|
||||
| Senior engineers limited (5-6 available) | TestKit/Storage quality risk | Assign 2 senior engineers to TestKit (critical path); 1 senior to Storage; rotate for reviews |
|
||||
| UI Guild availability (Angular expertise scarce) | UI sprint delayed | Start UI sprint Week 10 (after Tier 1/2 modules); hire Angular contractor if needed |
|
||||
|
||||
---
|
||||
|
||||
## Risk Register
|
||||
|
||||
### High-Impact Risks (Severity: CRITICAL)
|
||||
|
||||
| ID | Risk | Probability | Impact | Mitigation | Owner | Status |
|
||||
|----|------|-------------|--------|------------|-------|--------|
|
||||
| R1 | TestKit delayed by 2+ weeks | MEDIUM | Blocks ALL 15 module/infra sprints; +4-6 weeks program delay | Staff with 2 senior engineers; daily standups; incremental releases (partial TestKit unblocks some modules) | Platform Guild | OPEN |
|
||||
| R2 | Storage harness (Testcontainers) incompatible with .NET 10 | LOW | Blocks 10 sprints; +3-4 weeks delay | Validate Testcontainers compatibility Week 1; fallback to manual Postgres setup | Platform Guild | OPEN |
|
||||
| R3 | Determinism tests fail due to non-deterministic crypto signatures | MEDIUM | Scanner, Signer, Attestor blocked; compliance issues | Focus determinism tests on payload hash (not signature bytes); document non-deterministic algorithms | Crypto Guild | OPEN |
|
||||
| R4 | Concurrent module tests overwhelm CI infrastructure | HIGH | Test suite timeout, flaky tests, developer friction | Stagger module test starts (Tier 1 Week 7-8, Tier 2 Week 9-10); use dedicated CI runners; implement CI parallelization | Platform Guild | OPEN |
|
||||
|
||||
### Medium-Impact Risks
|
||||
|
||||
| ID | Risk | Probability | Impact | Mitigation | Owner | Status |
|
||||
|----|------|-------------|--------|------------|-------|--------|
|
||||
| R5 | Attestor-Signer circular dependency blocks integration tests | MEDIUM | Integration tests delayed 1-2 weeks | Signer uses mock attestation initially; coordinate integration in Week 9 | Crypto Guild | OPEN |
|
||||
| R6 | Upstream schema drift (NVD, OSV) breaks connector fixtures | MEDIUM | Connector tests fail; manual fixture regeneration required | FixtureUpdater tool automates regeneration; weekly live smoke tests detect drift early | Concelier Guild | OPEN |
|
||||
| R7 | WebService contract tests too brittle (fail on every API change) | MEDIUM | Developer friction, contract tests disabled | Version APIs explicitly; allow non-breaking changes; review contract test strategy Week 6 | Platform Guild | OPEN |
|
||||
|
||||
### Low-Impact Risks
|
||||
|
||||
| ID | Risk | Probability | Impact | Mitigation | Owner | Status |
|
||||
|----|------|-------------|--------|------------|-------|--------|
|
||||
| R8 | Property test generation too slow (FsCheck iterations high) | LOW | Test suite timeout | Limit property test iterations (default 100 → 50); profile and optimize generators | Scanner Guild | OPEN |
|
||||
| R9 | Architecture tests false positive (allowlist too restrictive) | LOW | Valid code blocked | Review architecture rules Week 5; explicit allowlist for test projects, benchmarks | Platform Guild | OPEN |
|
||||
| R10 | Competitor parity tests require paid Trivy/Anchore licenses | LOW | Parity testing incomplete | Use Trivy free tier; defer Anchore to future sprint; focus on Syft/Grype (OSS) | QA Guild | OPEN |
|
||||
|
||||
### Risk Burn-Down Plan
|
||||
|
||||
**Week 1:** Validate Testcontainers .NET 10 compatibility (R2)
|
||||
**Week 2:** TestKit API design review (R1)
|
||||
**Week 4:** Determinism test strategy review (R3)
|
||||
**Week 6:** CI infrastructure capacity review (R4)
|
||||
**Week 8:** Signer-Attestor integration coordination (R5)
|
||||
|
||||
---
|
||||
|
||||
## Success Metrics
|
||||
|
||||
### Quantitative Metrics
|
||||
|
||||
| Metric | Baseline | Target | Measurement Method | Tracked By |
|
||||
|--------|----------|--------|-------------------|------------|
|
||||
| **Code Coverage** | ~40% | ≥70% | `dotnet test --collect:"XPlat Code Coverage"` | Weekly (Fridays) |
|
||||
| **Test Count** | ~200 tests | ≥500 tests | Test suite execution count | Weekly |
|
||||
| **Determinism Pass Rate** | N/A (not tracked) | 100% (no flaky tests) | Determinism gate CI job | Daily (CI) |
|
||||
| **Contract Test Coverage** | 0 WebServices | 13 WebServices (100%) | Contract lane CI job | Weekly |
|
||||
| **Architecture Violations** | Unknown | 0 violations | Architecture test failures | Daily (CI, PR gate) |
|
||||
| **Sprint On-Time Completion** | N/A | ≥80% | Tasks complete by wave deadline | Weekly |
|
||||
|
||||
### Qualitative Metrics
|
||||
|
||||
| Metric | Success Criteria | Measurement Method | Tracked By |
|
||||
|--------|------------------|-------------------|------------|
|
||||
| **Developer Experience** | ≥80% of developers rate test authoring as "easy" or "very easy" | Post-sprint developer survey (Week 14) | Project Manager |
|
||||
| **Test Maintainability** | ≥75% of test failures are due to actual bugs (not test brittleness) | Monthly test failure classification | QA Guild |
|
||||
| **Integration Confidence** | ≥90% of PRs pass CI on first attempt (no test fixes required) | CI metrics (PR pass rate) | Platform Guild |
|
||||
|
||||
### Program Success Criteria
|
||||
|
||||
✅ **Program Successful If:**
|
||||
- All 22 sprints signed off (5100.0007.* + 5100.0008.0001 + 5100.0009.* + 5100.0010.*)
|
||||
- Code coverage ≥70% platform-wide
|
||||
- Determinism tests passing 100% in CI (no flaky tests)
|
||||
- Contract tests enforced for all 13 WebServices
|
||||
- Architecture tests PR-gating (lattice boundary violations blocked)
|
||||
|
||||
❌ **Program Failed If:**
|
||||
- <18 sprints signed off (<80% completion)
|
||||
- Code coverage increase <20% (baseline ~40% → <60%)
|
||||
- Critical quality gates missing (Determinism, Architecture, Contract)
|
||||
- TestKit not operational (blocking all module tests)
|
||||
|
||||
---
|
||||
|
||||
## Governance
|
||||
|
||||
### Steering Committee
|
||||
|
||||
| Role | Name | Responsibility |
|
||||
|------|------|----------------|
|
||||
| **Program Sponsor** | CTO | Final escalation; budget approval |
|
||||
| **Program Manager** | Project Management | Overall program coordination; risk management |
|
||||
| **Technical Lead** | Platform Guild Lead | Architecture decisions; technical escalation |
|
||||
| **QA Lead** | QA Guild Lead | Quality gate oversight; test strategy validation |
|
||||
|
||||
### Decision-Making Authority
|
||||
|
||||
| Decision Type | Authority | Escalation Path |
|
||||
|---------------|-----------|----------------|
|
||||
| **Sprint scope changes** | Sprint owner + Guild lead | Program Manager → Steering Committee |
|
||||
| **Architecture changes** | Platform Guild Lead | Steering Committee |
|
||||
| **Resource allocation** | Program Manager | CTO (if >10% budget impact) |
|
||||
| **Schedule changes (>1 week)** | Program Manager | Steering Committee |
|
||||
| **Risk acceptance** | Program Manager | Steering Committee (for HIGH/CRITICAL risks) |
|
||||
|
||||
### Status Reporting
|
||||
|
||||
**Weekly Status Report (Fridays):**
|
||||
- Sprint completion status (% tasks complete)
|
||||
- Blockers and risks (RED/YELLOW/GREEN)
|
||||
- Resource allocation (current vs. planned)
|
||||
- Next week preview
|
||||
|
||||
**Monthly Executive Summary:**
|
||||
- Program health (on-track / at-risk / off-track)
|
||||
- Milestone completion (M1-M5)
|
||||
- Budget vs. actuals
|
||||
- Key risks and mitigations
|
||||
|
||||
### Change Control
|
||||
|
||||
**Change Request Process:**
|
||||
1. **Requester submits change request** (scope, schedule, or resource change)
|
||||
2. **Program Manager reviews** (impact analysis: cost, schedule, quality)
|
||||
3. **Steering Committee approves/rejects** (for changes >1 week or >10% budget)
|
||||
4. **Program Manager updates plan** (timeline, resource model, risk register)
|
||||
|
||||
---
|
||||
|
||||
## Communication Plan
|
||||
|
||||
### Stakeholders
|
||||
|
||||
| Stakeholder Group | Interest | Communication Frequency | Method |
|
||||
|-------------------|----------|------------------------|--------|
|
||||
| **Engineering Teams (Guilds)** | Sprint execution, dependencies | Daily/Weekly | Slack #testing-strategy, guild standups |
|
||||
| **Guild Leads** | Sprint status, blockers | Weekly | Friday status sync (30 min) |
|
||||
| **Product Management** | Quality gates, feature readiness | Bi-weekly | Sprint demos, monthly exec summary |
|
||||
| **CTO / Executives** | Program health, budget | Monthly | Executive summary (email) |
|
||||
|
||||
### Meetings
|
||||
|
||||
#### Weekly Sync (Every Friday, 30 min)
|
||||
**Attendees:** All active sprint owners + program manager
|
||||
**Agenda:**
|
||||
1. Sprint status updates (green/yellow/red) (15 min)
|
||||
2. Blocker escalation (10 min)
|
||||
3. Next week preview (5 min)
|
||||
|
||||
#### Monthly Steering Committee (First Monday, 60 min)
|
||||
**Attendees:** Steering Committee (CTO, Program Manager, Platform Guild Lead, QA Lead)
|
||||
**Agenda:**
|
||||
1. Program health review (on-track / at-risk / off-track) (20 min)
|
||||
2. Milestone completion (M1-M5) (15 min)
|
||||
3. Budget vs. actuals (10 min)
|
||||
4. Risk review (top 3 risks) (10 min)
|
||||
5. Decisions required (5 min)
|
||||
|
||||
#### Retrospective (Week 14, 90 min)
|
||||
**Attendees:** All guild leads + program manager + steering committee
|
||||
**Agenda:**
|
||||
1. Program retrospective (what went well, what didn't, lessons learned) (60 min)
|
||||
2. Metrics review (code coverage, test count, determinism, etc.) (20 min)
|
||||
3. Future improvements (next testing initiatives) (10 min)
|
||||
|
||||
---
|
||||
|
||||
## Appendices
|
||||
|
||||
### Appendix A: Sprint Inventory
|
||||
|
||||
**Total Sprints:** 22
|
||||
- Foundation Epics: 7 (5100.0007.0001-0007)
|
||||
- Quality Gates: 1 (5100.0008.0001)
|
||||
- Module Tests: 11 (5100.0009.0001-0011)
|
||||
- Infrastructure Tests: 4 (5100.0010.0001-0004)
|
||||
|
||||
**Total Tasks:** ~370 tasks
|
||||
**Total Estimated Effort:** ~450 engineer-days (assuming avg 1.2 days/task)
|
||||
|
||||
### Appendix B: Reference Documents
|
||||
|
||||
1. **Advisory:** `docs/product-advisories/22-Dec-2026 - Better testing strategy.md`
|
||||
2. **Test Catalog:** `docs/testing/TEST_CATALOG.yml`
|
||||
3. **Test Models:** `docs/testing/testing-strategy-models.md`
|
||||
4. **Dependency Graph:** `docs/testing/SPRINT_DEPENDENCY_GRAPH.md`
|
||||
5. **Coverage Matrix:** `docs/testing/TEST_COVERAGE_MATRIX.md`
|
||||
6. **Execution Playbook:** `docs/testing/SPRINT_EXECUTION_PLAYBOOK.md`
|
||||
|
||||
### Appendix C: Budget Estimate (Preliminary)
|
||||
|
||||
**Assumptions:**
|
||||
- Average engineer cost: $150/hour (fully loaded)
|
||||
- Average sprint duration: 80 hours (2 weeks × 40 hours)
|
||||
- Peak staffing: 22 engineers (Weeks 7-10)
|
||||
|
||||
**Budget Estimate:**
|
||||
- Foundation Phase (Weeks 1-6): 12 engineers × 240 hours × $150 = $432,000
|
||||
- Module Tests Phase (Weeks 7-10): 22 engineers × 160 hours × $150 = $528,000
|
||||
- Infrastructure Phase (Weeks 11-14): 8 engineers × 160 hours × $150 = $192,000
|
||||
- **Total Estimated Cost:** $1,152,000
|
||||
|
||||
**Note:** Final budget requires approval from CTO/Finance. Contractor costs may reduce total if used strategically for peak staffing (Weeks 7-10).
|
||||
|
||||
---
|
||||
|
||||
**Prepared by:** Project Management
|
||||
**Approval Required From:** Steering Committee (CTO, Program Manager, Platform Guild Lead, QA Lead)
|
||||
**Date:** 2025-12-23
|
||||
**Next Review:** 2026-01-06 (Week 1 kickoff)
|
||||
262
docs/technical/testing/TEST_COVERAGE_MATRIX.md
Normal file
262
docs/technical/testing/TEST_COVERAGE_MATRIX.md
Normal file
@@ -0,0 +1,262 @@
|
||||
# Testing Strategy Coverage Matrix
|
||||
|
||||
> **Purpose:** Visual map of test model requirements per module, quality gates, and sprint-to-model relationships.
|
||||
|
||||
---
|
||||
|
||||
## Module-to-Model Coverage Map
|
||||
|
||||
### Legend
|
||||
- ✅ **Required** (from TEST_CATALOG.yml)
|
||||
- 🟡 **Optional** (recommended but not mandatory)
|
||||
- ⬜ **Not Applicable**
|
||||
|
||||
### Model Definitions (Quick Reference)
|
||||
| Model | Description | Key Tests |
|
||||
|-------|-------------|-----------|
|
||||
| **L0** | Library/Core | Unit, property, snapshot, determinism |
|
||||
| **S1** | Storage/Postgres | Integration, migrations, idempotency, query ordering |
|
||||
| **T1** | Transport/Queue | Protocol roundtrip, fuzz invalid, delivery semantics, backpressure |
|
||||
| **C1** | Connector/External | Fixtures, snapshot, resilience, security |
|
||||
| **W1** | WebService/API | Contract, authz, OTel, negative |
|
||||
| **WK1** | Worker/Indexer | End-to-end, retries, idempotency, OTel |
|
||||
| **AN1** | Analyzer/SourceGen | Diagnostics, codefixes, golden generated |
|
||||
| **CLI1** | Tool/CLI | Exit codes, golden output, determinism |
|
||||
| **PERF** | Benchmarks | Benchmark, perf smoke, regression thresholds |
|
||||
|
||||
---
|
||||
|
||||
## Coverage Matrix
|
||||
|
||||
### Core Modules
|
||||
|
||||
| Module | L0 | S1 | T1 | C1 | W1 | WK1 | AN1 | CLI1 | PERF | Sprint | Tasks |
|
||||
|--------|----|----|----|----|----|----|-----|------|------|--------|-------|
|
||||
| **Scanner** | ✅ | ✅ | ✅ | ⬜ | ✅ | ✅ | ✅ | ⬜ | ✅ | 5100.0009.0001 | 25 |
|
||||
| **Concelier** | ✅ | ✅ | ⬜ | ✅ | ✅ | ⬜ | ✅ | ⬜ | ⬜ | 5100.0009.0002 | 18 |
|
||||
| **Excititor** | ✅ | ✅ | ⬜ | ✅ | ✅ | ✅ | ⬜ | ⬜ | ⬜ | 5100.0009.0003 | 21 |
|
||||
| **Policy** | ✅ | ✅ | ⬜ | ⬜ | ✅ | ⬜ | ⬜ | ⬜ | ⬜ | 5100.0009.0004 | 15 |
|
||||
|
||||
### Security & Compliance Modules
|
||||
|
||||
| Module | L0 | S1 | T1 | C1 | W1 | WK1 | AN1 | CLI1 | PERF | Sprint | Tasks |
|
||||
|--------|----|----|----|----|----|----|-----|------|------|--------|-------|
|
||||
| **Authority** | ✅ | ⬜ | ⬜ | ✅ | ✅ | ⬜ | ⬜ | ⬜ | ⬜ | 5100.0009.0005 | 17 |
|
||||
| **Signer** | ✅ | ⬜ | ⬜ | ✅ | ✅ | ⬜ | ⬜ | ⬜ | ⬜ | 5100.0009.0006 | 17 |
|
||||
| **Attestor** | ✅ | ⬜ | ⬜ | ⬜ | ✅ | ⬜ | ⬜ | ⬜ | ⬜ | 5100.0009.0007 | 14 |
|
||||
|
||||
### Platform Services
|
||||
|
||||
| Module | L0 | S1 | T1 | C1 | W1 | WK1 | AN1 | CLI1 | PERF | Sprint | Tasks |
|
||||
|--------|----|----|----|----|----|----|-----|------|------|--------|-------|
|
||||
| **Scheduler** | ✅ | ✅ | ⬜ | ⬜ | ✅ | ✅ | ⬜ | ⬜ | ⬜ | 5100.0009.0008 | 14 |
|
||||
| **Notify** | ✅ | ✅ | ⬜ | ✅ | ✅ | ✅ | ⬜ | ⬜ | ⬜ | 5100.0009.0009 | 18 |
|
||||
|
||||
### Client Interfaces
|
||||
|
||||
| Module | L0 | S1 | T1 | C1 | W1 | WK1 | AN1 | CLI1 | PERF | Sprint | Tasks |
|
||||
|--------|----|----|----|----|----|----|-----|------|------|--------|-------|
|
||||
| **CLI** | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ | ✅ | ⬜ | 5100.0009.0010 | 13 |
|
||||
| **UI** | ⬜ | ⬜ | ⬜ | ⬜ | ✅ | ⬜ | ⬜ | ⬜ | ⬜ | 5100.0009.0011 | 13 |
|
||||
|
||||
### Infrastructure & Platform
|
||||
|
||||
| Module | L0 | S1 | T1 | C1 | W1 | WK1 | AN1 | CLI1 | PERF | Sprint | Tasks |
|
||||
|--------|----|----|----|----|----|----|-----|------|------|--------|-------|
|
||||
| **EvidenceLocker** | ✅ | ✅ | ⬜ | ⬜ | ✅ | ⬜ | ⬜ | ⬜ | ⬜ | 5100.0010.0001 | 16 |
|
||||
| **Graph/Timeline** | ✅ | ✅ | ⬜ | ⬜ | ✅ | ✅ | ⬜ | ⬜ | ⬜ | 5100.0010.0002 | 15 |
|
||||
| **Router/Messaging** | ✅ | ✅ | ✅ | ⬜ | ✅ | ⬜ | ⬜ | ⬜ | ⬜ | 5100.0010.0003 | 14 |
|
||||
| **AirGap** | ✅ | ✅ | ⬜ | ⬜ | ✅ | ⬜ | ✅ | ✅ | ⬜ | 5100.0010.0004 | 17 |
|
||||
|
||||
---
|
||||
|
||||
## Model Distribution Analysis
|
||||
|
||||
### Models by Usage Frequency
|
||||
|
||||
| Model | Modules Using | Percentage | Complexity |
|
||||
|-------|---------------|------------|------------|
|
||||
| **L0** (Library/Core) | 13/15 modules | 87% | HIGH (property tests, snapshots) |
|
||||
| **W1** (WebService) | 13/15 modules | 87% | MEDIUM (contract tests, auth) |
|
||||
| **S1** (Storage) | 10/15 modules | 67% | HIGH (migrations, idempotency) |
|
||||
| **C1** (Connectors) | 5/15 modules | 33% | MEDIUM (fixtures, resilience) |
|
||||
| **WK1** (Workers) | 5/15 modules | 33% | MEDIUM (end-to-end, retries) |
|
||||
| **AN1** (Analyzers) | 3/15 modules | 20% | HIGH (Roslyn, diagnostics) |
|
||||
| **T1** (Transport) | 2/15 modules | 13% | HIGH (protocol compliance) |
|
||||
| **CLI1** (CLI Tools) | 2/15 modules | 13% | LOW (exit codes, snapshots) |
|
||||
| **PERF** (Performance) | 1/15 modules | 7% | MEDIUM (benchmarks, regression) |
|
||||
|
||||
### Complexity Heatmap
|
||||
|
||||
**High Complexity (>15 tasks per sprint):**
|
||||
- Scanner (25 tasks: L0+AN1+S1+T1+W1+WK1+PERF)
|
||||
- Excititor (21 tasks: C1+L0+S1+W1+WK1)
|
||||
- Concelier (18 tasks: C1+L0+S1+W1+AN1)
|
||||
- Notify (18 tasks: L0+C1+S1+W1+WK1)
|
||||
- Authority (17 tasks: L0+W1+C1)
|
||||
- Signer (17 tasks: L0+W1+C1)
|
||||
- AirGap (17 tasks: L0+AN1+S1+W1+CLI1)
|
||||
|
||||
**Medium Complexity (10-15 tasks):**
|
||||
- Policy (15 tasks: L0+S1+W1)
|
||||
- EvidenceLocker (16 tasks: L0+S1+W1)
|
||||
- Graph/Timeline (15 tasks: L0+S1+W1+WK1)
|
||||
- Scheduler (14 tasks: L0+S1+W1+WK1)
|
||||
- Attestor (14 tasks: L0+W1)
|
||||
- Router/Messaging (14 tasks: L0+T1+W1+S1)
|
||||
- CLI (13 tasks: CLI1)
|
||||
- UI (13 tasks: W1)
|
||||
|
||||
---
|
||||
|
||||
## Quality Gate Coverage
|
||||
|
||||
### Module-Specific Quality Gates (from TEST_CATALOG.yml)
|
||||
|
||||
| Module | Quality Gates | Enforced By |
|
||||
|--------|---------------|-------------|
|
||||
| **Scanner** | determinism, reachability_evidence, proof_spine | Sprint 5100.0009.0001 Tasks 7-10, 23-25 |
|
||||
| **Concelier** | fixture_coverage, normalization_determinism, no_lattice_dependency | Sprint 5100.0009.0002 Tasks 1-7, 8-10, 18 |
|
||||
| **Excititor** | preserve_prune_source, format_snapshots, no_lattice_dependency | Sprint 5100.0009.0003 Tasks 6-11, 21 |
|
||||
| **Policy** | unknown_budget, verdict_snapshot | Sprint 5100.0009.0004 Tasks 2, 4, 14-15 |
|
||||
| **Authority** | scope_enforcement, sign_verify | Sprint 5100.0009.0005 Tasks 3-5, 16-17 |
|
||||
| **Signer** | canonical_payloads, sign_verify | Sprint 5100.0009.0006 Tasks 1-3, 15-17 |
|
||||
| **Attestor** | rekor_receipts, dsse_verify | Sprint 5100.0009.0007 Tasks 6-8, 2 |
|
||||
| **Scheduler** | idempotent_jobs, retry_backoff | Sprint 5100.0009.0008 Tasks 4, 3, 12 |
|
||||
| **Notify** | connector_snapshots, retry_semantics | Sprint 5100.0009.0009 Tasks 1-6, 16 |
|
||||
| **CLI** | exit_codes, stdout_snapshots | Sprint 5100.0009.0010 Tasks 1-4, 5-8 |
|
||||
| **UI** | contract_snapshots, e2e_smoke | Sprint 5100.0009.0011 Tasks 1-2, 7-10 |
|
||||
|
||||
### Cross-Cutting Quality Gates
|
||||
|
||||
| Gate | Applies To | Enforced By |
|
||||
|------|-----------|-------------|
|
||||
| **Determinism Contract** | Scanner, Excititor, Signer, CLI, AirGap, Concelier | Sprint 5100.0007.0003 (Determinism Gate) |
|
||||
| **Architecture Boundaries** | Concelier, Excititor (must NOT reference Scanner lattice) | Sprint 5100.0007.0007 (Architecture Tests) |
|
||||
| **Contract Stability** | All WebServices (13 modules) | Sprint 5100.0007.0006 (WebService Contract) |
|
||||
| **Storage Idempotency** | All S1 modules (10 modules) | Sprint 5100.0007.0004 (Storage Harness) |
|
||||
| **Connector Resilience** | All C1 modules (5 modules) | Sprint 5100.0007.0005 (Connector Fixtures) |
|
||||
|
||||
---
|
||||
|
||||
## CI Lane Coverage
|
||||
|
||||
### Test Distribution Across CI Lanes
|
||||
|
||||
| CI Lane | Models | Modules | Sprint Tasks | Est. Runtime |
|
||||
|---------|--------|---------|--------------|--------------|
|
||||
| **Unit** | L0, AN1, CLI1 | All 15 modules | ~120 tasks | <5 min |
|
||||
| **Contract** | W1 | 13 modules | ~50 tasks | <2 min |
|
||||
| **Integration** | S1, WK1, T1 | 12 modules | ~100 tasks | 10-15 min |
|
||||
| **Security** | C1 (security tests), W1 (auth tests) | 5 connectors + 13 WebServices | ~60 tasks | 5-10 min |
|
||||
| **Performance** | PERF | Scanner only | ~3 tasks | 3-5 min |
|
||||
| **Live** | C1 (live smoke tests) | Concelier, Excititor, Notify, Authority, Signer | ~5 tasks (opt-in) | 5-10 min (nightly) |
|
||||
|
||||
### CI Lane Dependencies
|
||||
|
||||
```
|
||||
PR Gate (Must Pass):
|
||||
├─ Unit Lane (L0, AN1, CLI1) ← Fast feedback
|
||||
├─ Contract Lane (W1) ← API stability
|
||||
├─ Architecture Lane (Sprint 5100.0007.0007) ← Boundary enforcement
|
||||
└─ Integration Lane (S1, WK1, T1) ← Testcontainers
|
||||
|
||||
Merge Gate (Must Pass):
|
||||
├─ All PR Gate lanes
|
||||
├─ Security Lane (C1 security, W1 auth)
|
||||
└─ Determinism Lane (Sprint 5100.0007.0003)
|
||||
|
||||
Nightly (Optional):
|
||||
├─ Performance Lane (PERF)
|
||||
└─ Live Lane (C1 live smoke)
|
||||
|
||||
Weekly (Optional):
|
||||
└─ Competitor Parity (Sprint 5100.0008.0001)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Epic-to-Model Coverage
|
||||
|
||||
### Epic Sprints Support Multiple Models
|
||||
|
||||
| Epic Sprint | Models Enabled | Consuming Modules | Tasks |
|
||||
|-------------|----------------|-------------------|-------|
|
||||
| **5100.0007.0002 (TestKit)** | ALL (L0, S1, T1, C1, W1, WK1, AN1, CLI1, PERF) | ALL 15 modules | 13 |
|
||||
| **5100.0007.0003 (Determinism)** | L0 (determinism), CLI1 (determinism) | Scanner, Excititor, Signer, CLI, AirGap, Concelier | 12 |
|
||||
| **5100.0007.0004 (Storage)** | S1 | 10 modules | 12 |
|
||||
| **5100.0007.0005 (Connectors)** | C1 | Concelier, Excititor, Authority, Signer, Notify | 12 |
|
||||
| **5100.0007.0006 (WebService)** | W1 | 13 modules | 12 |
|
||||
| **5100.0007.0007 (Architecture)** | (Cross-cutting) | Concelier, Excititor | 17 |
|
||||
|
||||
---
|
||||
|
||||
## Test Type Distribution
|
||||
|
||||
### By Test Category (Trait)
|
||||
|
||||
| Test Category | Model Coverage | Estimated Test Count | CI Lane |
|
||||
|---------------|----------------|----------------------|---------|
|
||||
| **Unit** | L0, AN1 | ~150 tests across 13 modules | Unit |
|
||||
| **Property** | L0 (subset) | ~40 tests (Scanner, Policy, Scheduler, Router) | Unit |
|
||||
| **Snapshot** | L0, C1, CLI1 | ~80 tests (all modules with canonical outputs) | Unit/Contract |
|
||||
| **Integration** | S1, WK1, T1 | ~120 tests across 12 modules | Integration |
|
||||
| **Contract** | W1 | ~50 tests (13 WebServices × avg 4 endpoints) | Contract |
|
||||
| **Security** | C1 (security), W1 (auth) | ~60 tests | Security |
|
||||
| **Performance** | PERF | ~3 tests (Scanner only) | Performance |
|
||||
| **Live** | C1 (live smoke) | ~5 tests (opt-in, nightly) | Live |
|
||||
|
||||
---
|
||||
|
||||
## Coverage Gaps & Recommendations
|
||||
|
||||
### Current Gaps
|
||||
|
||||
1. **Performance Testing:** Only Scanner has PERF model
|
||||
- **Recommendation:** Add PERF to Policy (policy evaluation latency), Concelier (merge performance), Scheduler (scheduling overhead)
|
||||
|
||||
2. **Transport Testing:** Only Router/Messaging has T1 model
|
||||
- **Recommendation:** Scanner has T1 in TEST_CATALOG.yml but should validate Valkey transport for job queues
|
||||
|
||||
3. **Live Connector Tests:** Only 5 modules have C1 live smoke tests (opt-in)
|
||||
- **Recommendation:** Run weekly, not nightly; treat as early warning system for schema drift
|
||||
|
||||
### Recommended Additions (Future Sprints)
|
||||
|
||||
| Module | Missing Model | Justification | Priority |
|
||||
|--------|---------------|---------------|----------|
|
||||
| Policy | PERF | Policy evaluation latency critical for real-time decisioning | HIGH |
|
||||
| Concelier | PERF | Merge performance affects ingestion throughput | MEDIUM |
|
||||
| Scheduler | PERF | Scheduling overhead affects job execution latency | MEDIUM |
|
||||
| Scanner | T1 (validate) | Job queue transport (Valkey) should have compliance tests | HIGH |
|
||||
| Authority | S1 | Token storage/revocation should have migration tests | MEDIUM |
|
||||
|
||||
---
|
||||
|
||||
## Summary Statistics
|
||||
|
||||
**Total Test Models:** 9
|
||||
**Total Modules Covered:** 15
|
||||
**Total Module Test Sprints:** 15 (11 module + 4 infrastructure)
|
||||
**Total Epic Sprints:** 6
|
||||
**Total Quality Gate Sprints:** 1 (Competitor Parity)
|
||||
|
||||
**Model Usage:**
|
||||
- L0: 13 modules (87%)
|
||||
- W1: 13 modules (87%)
|
||||
- S1: 10 modules (67%)
|
||||
- C1: 5 modules (33%)
|
||||
- WK1: 5 modules (33%)
|
||||
- AN1: 3 modules (20%)
|
||||
- T1: 2 modules (13%)
|
||||
- CLI1: 2 modules (13%)
|
||||
- PERF: 1 module (7%)
|
||||
|
||||
**Estimated Total Tests:** ~500 tests across all modules and models
|
||||
|
||||
---
|
||||
|
||||
**Prepared by:** Project Management
|
||||
**Date:** 2025-12-23
|
||||
**Next Review:** 2026-01-06 (Week 1 kickoff)
|
||||
**Source:** `docs/testing/TEST_CATALOG.yml`, Sprint files 5100.0009.* and 5100.0010.*
|
||||
245
docs/technical/testing/ci-lane-filters.md
Normal file
245
docs/technical/testing/ci-lane-filters.md
Normal file
@@ -0,0 +1,245 @@
|
||||
# CI Lane Filters and Test Traits
|
||||
|
||||
This document describes how to categorize tests by lane and test type for CI filtering.
|
||||
|
||||
## Test Lanes
|
||||
|
||||
StellaOps uses standardized test lanes based on `docs/testing/TEST_CATALOG.yml`:
|
||||
|
||||
| Lane | Purpose | Characteristics | PR Gating |
|
||||
|------|---------|-----------------|-----------|
|
||||
| **Unit** | Fast, isolated tests | No I/O, deterministic, offline | ✅ Yes |
|
||||
| **Contract** | API contract stability | Schema/OpenAPI validation | ✅ Yes |
|
||||
| **Integration** | Service and storage tests | Uses Testcontainers (Postgres/Valkey) | ✅ Yes |
|
||||
| **Security** | Security regression tests | authz, negative cases, injection tests | ✅ Yes |
|
||||
| **Performance** | Benchmark and perf smoke | Regression thresholds | ⚠️ Optional |
|
||||
| **Live** | External connector smoke tests | Requires network, upstream deps | ❌ No (opt-in only) |
|
||||
|
||||
## Using Test Traits
|
||||
|
||||
Add `StellaOps.TestKit` to your test project:
|
||||
|
||||
```xml
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="..\..\__Libraries\StellaOps.TestKit\StellaOps.TestKit.csproj" />
|
||||
</ItemGroup>
|
||||
```
|
||||
|
||||
### Lane Attributes
|
||||
|
||||
Mark tests with lane attributes:
|
||||
|
||||
```csharp
|
||||
using StellaOps.TestKit.Traits;
|
||||
using Xunit;
|
||||
|
||||
public class MyTests
|
||||
{
|
||||
[Fact]
|
||||
[UnitTest] // Runs in Unit lane
|
||||
public void FastIsolatedTest()
|
||||
{
|
||||
// No I/O, deterministic
|
||||
}
|
||||
|
||||
[Fact]
|
||||
[IntegrationTest] // Runs in Integration lane
|
||||
public async Task DatabaseTest()
|
||||
{
|
||||
// Uses Testcontainers PostgreSQL
|
||||
}
|
||||
|
||||
[Fact]
|
||||
[SecurityTest] // Runs in Security lane
|
||||
public void AuthorizationTest()
|
||||
{
|
||||
// Tests RBAC, scope enforcement
|
||||
}
|
||||
|
||||
[Fact]
|
||||
[LiveTest] // Runs in Live lane (opt-in only)
|
||||
public async Task ExternalApiTest()
|
||||
{
|
||||
// Calls external service
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Test Type Attributes
|
||||
|
||||
Mark tests with specific test types:
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
[UnitTest]
|
||||
[DeterminismTest] // Verifies deterministic output
|
||||
public void SbomGenerationIsDeterministic()
|
||||
{
|
||||
var sbom1 = GenerateSbom();
|
||||
var sbom2 = GenerateSbom();
|
||||
Assert.Equal(sbom1, sbom2); // Must be identical
|
||||
}
|
||||
|
||||
[Fact]
|
||||
[IntegrationTest]
|
||||
[SnapshotTest] // Compares against golden file
|
||||
public void ApiResponseMatchesSnapshot()
|
||||
{
|
||||
var response = CallApi();
|
||||
SnapshotHelper.VerifySnapshot(response, "api_response.json");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
[SecurityTest]
|
||||
[AuthzTest] // Tests authorization
|
||||
public void UnauthorizedRequestIsRejected()
|
||||
{
|
||||
// Test RBAC enforcement
|
||||
}
|
||||
```
|
||||
|
||||
## Running Tests by Lane
|
||||
|
||||
### Command Line
|
||||
|
||||
```bash
|
||||
# Run Unit tests only
|
||||
dotnet test --filter "Lane=Unit"
|
||||
|
||||
# Run Integration tests only
|
||||
dotnet test --filter "Lane=Integration"
|
||||
|
||||
# Run Security tests only
|
||||
dotnet test --filter "Lane=Security"
|
||||
|
||||
# Using the helper script
|
||||
./scripts/test-lane.sh Unit
|
||||
./scripts/test-lane.sh Integration --results-directory ./test-results
|
||||
```
|
||||
|
||||
### CI Workflows
|
||||
|
||||
Example job for Unit lane:
|
||||
|
||||
```yaml
|
||||
unit-tests:
|
||||
runs-on: ubuntu-22.04
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Setup .NET
|
||||
uses: actions/setup-dotnet@v4
|
||||
with:
|
||||
dotnet-version: '10.0.x'
|
||||
|
||||
- name: Run Unit tests
|
||||
run: |
|
||||
dotnet test \
|
||||
--filter "Lane=Unit" \
|
||||
--configuration Release \
|
||||
--logger "trx;LogFileName=unit-tests.trx" \
|
||||
--results-directory ./test-results
|
||||
```
|
||||
|
||||
Example job for Integration lane:
|
||||
|
||||
```yaml
|
||||
integration-tests:
|
||||
runs-on: ubuntu-22.04
|
||||
services:
|
||||
postgres:
|
||||
image: postgres:16-alpine
|
||||
# ...
|
||||
steps:
|
||||
- name: Run Integration tests
|
||||
run: |
|
||||
dotnet test \
|
||||
--filter "Lane=Integration" \
|
||||
--configuration Release \
|
||||
--logger "trx;LogFileName=integration-tests.trx" \
|
||||
--results-directory ./test-results
|
||||
```
|
||||
|
||||
## Lane Filtering Best Practices
|
||||
|
||||
### Unit Lane
|
||||
- ✅ Pure functions, logic tests
|
||||
- ✅ In-memory operations
|
||||
- ✅ Deterministic time/random (use `StellaOps.TestKit.Time.DeterministicClock`)
|
||||
- ❌ No file I/O (except snapshots in `__snapshots__/`)
|
||||
- ❌ No network calls
|
||||
- ❌ No databases
|
||||
|
||||
### Contract Lane
|
||||
- ✅ OpenAPI schema validation
|
||||
- ✅ API response envelope checks
|
||||
- ✅ Contract stability tests
|
||||
- ❌ No external dependencies
|
||||
|
||||
### Integration Lane
|
||||
- ✅ Testcontainers for Postgres/Valkey
|
||||
- ✅ End-to-end service flows
|
||||
- ✅ Multi-component interactions
|
||||
- ❌ No external/live services
|
||||
|
||||
### Security Lane
|
||||
- ✅ Authorization/authentication tests
|
||||
- ✅ Input validation (SQL injection, XSS prevention)
|
||||
- ✅ RBAC scope enforcement
|
||||
- ✅ Negative test cases
|
||||
|
||||
### Performance Lane
|
||||
- ✅ Benchmarks with `[Benchmark]` attribute
|
||||
- ✅ Performance smoke tests
|
||||
- ✅ Regression threshold checks
|
||||
- ⚠️ Not PR-gating by default (runs on schedule)
|
||||
|
||||
### Live Lane
|
||||
- ✅ External API smoke tests
|
||||
- ✅ Upstream connector validation
|
||||
- ❌ **Never PR-gating**
|
||||
- ⚠️ Opt-in only (schedule or manual trigger)
|
||||
|
||||
## Combining Traits
|
||||
|
||||
You can combine multiple traits:
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
[IntegrationTest]
|
||||
[TestType("idempotency")]
|
||||
[TestType("postgres")]
|
||||
public async Task JobExecutionIsIdempotent()
|
||||
{
|
||||
// Test uses Postgres fixture and verifies idempotency
|
||||
}
|
||||
```
|
||||
|
||||
## Migration Guide
|
||||
|
||||
If you have existing tests without lane attributes:
|
||||
|
||||
1. **Identify test characteristics**:
|
||||
- Does it use I/O? → `IntegrationTest`
|
||||
- Is it fast and isolated? → `UnitTest`
|
||||
- Does it test auth/security? → `SecurityTest`
|
||||
- Does it call external APIs? → `LiveTest`
|
||||
|
||||
2. **Add appropriate attributes**:
|
||||
```csharp
|
||||
[Fact]
|
||||
[UnitTest] // Add this
|
||||
public void ExistingTest() { ... }
|
||||
```
|
||||
|
||||
3. **Verify in CI**:
|
||||
```bash
|
||||
# Should only run newly tagged tests
|
||||
dotnet test --filter "Lane=Unit"
|
||||
```
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- Test Catalog: `docs/testing/TEST_CATALOG.yml`
|
||||
- Testing Strategy: `docs/testing/testing-strategy-models.md`
|
||||
- TestKit README: `src/__Libraries/StellaOps.TestKit/README.md`
|
||||
310
docs/technical/testing/ci-lane-integration.md
Normal file
310
docs/technical/testing/ci-lane-integration.md
Normal file
@@ -0,0 +1,310 @@
|
||||
# CI Lane Integration Guide
|
||||
|
||||
This guide explains how to integrate the standardized test lane filtering into CI workflows.
|
||||
|
||||
## Overview
|
||||
|
||||
StellaOps uses a lane-based test categorization system with six standardized lanes:
|
||||
- **Unit**: Fast, isolated, deterministic tests (PR-gating)
|
||||
- **Contract**: API contract stability tests (PR-gating)
|
||||
- **Integration**: Service and storage tests with Testcontainers (PR-gating)
|
||||
- **Security**: AuthZ, input validation, negative tests (PR-gating)
|
||||
- **Performance**: Benchmarks and regression thresholds (optional/scheduled)
|
||||
- **Live**: External API smoke tests (opt-in only, never PR-gating)
|
||||
|
||||
## Using Lane Filters in CI
|
||||
|
||||
### Using the Test Runner Script
|
||||
|
||||
The recommended approach is to use `scripts/test-lane.sh`:
|
||||
|
||||
```yaml
|
||||
- name: Run Unit lane tests
|
||||
run: |
|
||||
chmod +x scripts/test-lane.sh
|
||||
./scripts/test-lane.sh Unit \
|
||||
--logger "trx;LogFileName=unit-tests.trx" \
|
||||
--results-directory ./test-results \
|
||||
--verbosity normal
|
||||
```
|
||||
|
||||
### Direct dotnet test Filtering
|
||||
|
||||
Alternatively, use `dotnet test` with lane filters directly:
|
||||
|
||||
```yaml
|
||||
- name: Run Integration lane tests
|
||||
run: |
|
||||
dotnet test \
|
||||
--filter "Lane=Integration" \
|
||||
--configuration Release \
|
||||
--logger "trx;LogFileName=integration-tests.trx" \
|
||||
--results-directory ./test-results
|
||||
```
|
||||
|
||||
## Lane-Based Workflow Pattern
|
||||
|
||||
### Full Workflow Example
|
||||
|
||||
See `.gitea/workflows/test-lanes.yml` for a complete reference implementation.
|
||||
|
||||
Key features:
|
||||
- **Separate jobs per lane** for parallel execution
|
||||
- **PR-gating lanes** run on all PRs (Unit, Contract, Integration, Security)
|
||||
- **Optional lanes** run on schedule or manual trigger (Performance, Live)
|
||||
- **Test results summary** aggregates all lane results
|
||||
|
||||
### Job Structure
|
||||
|
||||
```yaml
|
||||
unit-tests:
|
||||
name: Unit Tests
|
||||
runs-on: ubuntu-22.04
|
||||
timeout-minutes: 15
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- name: Setup .NET
|
||||
uses: actions/setup-dotnet@v4
|
||||
with:
|
||||
dotnet-version: '10.0.100'
|
||||
- name: Build
|
||||
run: dotnet build src/StellaOps.sln --configuration Release
|
||||
- name: Run Unit lane
|
||||
run: ./scripts/test-lane.sh Unit --results-directory ./test-results
|
||||
- name: Upload results
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: unit-test-results
|
||||
path: ./test-results
|
||||
```
|
||||
|
||||
## Lane Execution Guidelines
|
||||
|
||||
### Unit Lane
|
||||
- **Timeout**: 10-15 minutes
|
||||
- **Dependencies**: None (no I/O, no network, no databases)
|
||||
- **PR gating**: ✅ Required
|
||||
- **Characteristics**: Deterministic, fast, offline
|
||||
|
||||
### Contract Lane
|
||||
- **Timeout**: 5-10 minutes
|
||||
- **Dependencies**: None (schema validation only)
|
||||
- **PR gating**: ✅ Required
|
||||
- **Characteristics**: OpenAPI/schema validation, no external calls
|
||||
|
||||
### Integration Lane
|
||||
- **Timeout**: 20-30 minutes
|
||||
- **Dependencies**: Testcontainers (Postgres, Valkey)
|
||||
- **PR gating**: ✅ Required
|
||||
- **Characteristics**: End-to-end service flows, database tests
|
||||
|
||||
### Security Lane
|
||||
- **Timeout**: 15-20 minutes
|
||||
- **Dependencies**: Testcontainers (if needed for auth tests)
|
||||
- **PR gating**: ✅ Required
|
||||
- **Characteristics**: RBAC, injection prevention, negative tests
|
||||
|
||||
### Performance Lane
|
||||
- **Timeout**: 30-45 minutes
|
||||
- **Dependencies**: Baseline data, historical metrics
|
||||
- **PR gating**: ❌ Optional (scheduled/manual)
|
||||
- **Characteristics**: Benchmarks, regression thresholds
|
||||
|
||||
### Live Lane
|
||||
- **Timeout**: 15-20 minutes
|
||||
- **Dependencies**: External APIs, upstream services
|
||||
- **PR gating**: ❌ Never (opt-in only)
|
||||
- **Characteristics**: Smoke tests, connector validation
|
||||
|
||||
## Migration from Per-Project to Lane-Based
|
||||
|
||||
### Before (Per-Project)
|
||||
```yaml
|
||||
- name: Run Concelier tests
|
||||
run: dotnet test src/Concelier/StellaOps.Concelier.sln
|
||||
|
||||
- name: Run Authority tests
|
||||
run: dotnet test src/Authority/StellaOps.Authority.sln
|
||||
|
||||
- name: Run Scanner tests
|
||||
run: dotnet test src/Scanner/StellaOps.Scanner.sln
|
||||
```
|
||||
|
||||
### After (Lane-Based)
|
||||
```yaml
|
||||
- name: Run Unit lane
|
||||
run: ./scripts/test-lane.sh Unit
|
||||
|
||||
- name: Run Integration lane
|
||||
run: ./scripts/test-lane.sh Integration
|
||||
|
||||
- name: Run Security lane
|
||||
run: ./scripts/test-lane.sh Security
|
||||
```
|
||||
|
||||
**Benefits**:
|
||||
- Run all unit tests across all modules in parallel
|
||||
- Clear separation of concerns by test type
|
||||
- Faster feedback (fast tests run first)
|
||||
- Better resource utilization (no Testcontainers for Unit tests)
|
||||
|
||||
## Best Practices
|
||||
|
||||
### 1. Parallel Execution
|
||||
Run PR-gating lanes in parallel for faster feedback:
|
||||
|
||||
```yaml
|
||||
jobs:
|
||||
unit-tests:
|
||||
# ...
|
||||
integration-tests:
|
||||
# ...
|
||||
security-tests:
|
||||
# ...
|
||||
```
|
||||
|
||||
### 2. Conditional Execution
|
||||
Use workflow inputs for optional lanes:
|
||||
|
||||
```yaml
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
run_performance:
|
||||
type: boolean
|
||||
default: false
|
||||
|
||||
jobs:
|
||||
performance-tests:
|
||||
if: github.event.inputs.run_performance == 'true'
|
||||
# ...
|
||||
```
|
||||
|
||||
### 3. Test Result Aggregation
|
||||
Create a summary job that depends on all lane jobs:
|
||||
|
||||
```yaml
|
||||
test-summary:
|
||||
needs: [unit-tests, contract-tests, integration-tests, security-tests]
|
||||
if: always()
|
||||
steps:
|
||||
- name: Download all results
|
||||
uses: actions/download-artifact@v4
|
||||
- name: Generate summary
|
||||
run: ./scripts/ci/aggregate-test-results.sh
|
||||
```
|
||||
|
||||
### 4. Timeout Configuration
|
||||
Set appropriate timeouts per lane:
|
||||
|
||||
```yaml
|
||||
unit-tests:
|
||||
timeout-minutes: 15 # Fast
|
||||
|
||||
integration-tests:
|
||||
timeout-minutes: 30 # Testcontainers startup
|
||||
|
||||
performance-tests:
|
||||
timeout-minutes: 45 # Benchmark execution
|
||||
```
|
||||
|
||||
### 5. Environment Isolation
|
||||
Use Testcontainers for Integration lane, not GitHub Actions services:
|
||||
|
||||
```yaml
|
||||
integration-tests:
|
||||
steps:
|
||||
- name: Run Integration tests
|
||||
env:
|
||||
POSTGRES_TEST_IMAGE: postgres:16-alpine
|
||||
run: ./scripts/test-lane.sh Integration
|
||||
```
|
||||
|
||||
Testcontainers provides:
|
||||
- Per-test isolation
|
||||
- Automatic cleanup
|
||||
- Consistent behavior across environments
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Tests Not Found
|
||||
**Problem**: `dotnet test --filter "Lane=Unit"` finds no tests
|
||||
|
||||
**Solution**: Ensure tests have lane attributes:
|
||||
```csharp
|
||||
[Fact]
|
||||
[UnitTest] // This attribute is required
|
||||
public void MyTest() { }
|
||||
```
|
||||
|
||||
### Wrong Lane Assignment
|
||||
**Problem**: Integration test running in Unit lane
|
||||
|
||||
**Solution**: Check test attributes:
|
||||
```csharp
|
||||
// Bad: No database in Unit lane
|
||||
[Fact]
|
||||
[UnitTest]
|
||||
public async Task DatabaseTest() { /* uses Postgres */ }
|
||||
|
||||
// Good: Use Integration lane for database tests
|
||||
[Fact]
|
||||
[IntegrationTest]
|
||||
public async Task DatabaseTest() { /* uses Testcontainers */ }
|
||||
```
|
||||
|
||||
### Testcontainers Timeout
|
||||
**Problem**: Integration tests timeout waiting for containers
|
||||
|
||||
**Solution**: Increase job timeout and ensure Docker is available:
|
||||
```yaml
|
||||
integration-tests:
|
||||
timeout-minutes: 30 # Increased from 15
|
||||
steps:
|
||||
- name: Verify Docker
|
||||
run: docker info
|
||||
```
|
||||
|
||||
### Live Tests in PR
|
||||
**Problem**: Live lane tests failing in PRs
|
||||
|
||||
**Solution**: Never run Live tests in PRs:
|
||||
```yaml
|
||||
live-tests:
|
||||
if: github.event_name == 'workflow_dispatch' && github.event.inputs.run_live == 'true'
|
||||
# Never runs automatically on PR
|
||||
```
|
||||
|
||||
## Integration with Existing Workflows
|
||||
|
||||
### Adding Lane-Based Testing to build-test-deploy.yml
|
||||
|
||||
Replace per-module test execution with lane-based execution:
|
||||
|
||||
```yaml
|
||||
# Old approach
|
||||
- name: Run Concelier tests
|
||||
run: dotnet test src/Concelier/StellaOps.Concelier.sln
|
||||
|
||||
# New approach (recommended)
|
||||
- name: Run all Unit tests
|
||||
run: ./scripts/test-lane.sh Unit
|
||||
|
||||
- name: Run all Integration tests
|
||||
run: ./scripts/test-lane.sh Integration
|
||||
```
|
||||
|
||||
### Gradual Migration Strategy
|
||||
|
||||
1. **Phase 1**: Add lane attributes to existing tests
|
||||
2. **Phase 2**: Add lane-based jobs alongside existing per-project jobs
|
||||
3. **Phase 3**: Monitor lane-based jobs for stability
|
||||
4. **Phase 4**: Remove per-project jobs once lane-based jobs proven stable
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- Test Lane Filters: `docs/testing/ci-lane-filters.md`
|
||||
- Testing Strategy: `docs/testing/testing-strategy-models.md`
|
||||
- Test Catalog: `docs/testing/TEST_CATALOG.yml`
|
||||
- TestKit README: `src/__Libraries/StellaOps.TestKit/README.md`
|
||||
- Example Workflow: `.gitea/workflows/test-lanes.yml`
|
||||
157
docs/technical/testing/ci-quality-gates.md
Normal file
157
docs/technical/testing/ci-quality-gates.md
Normal file
@@ -0,0 +1,157 @@
|
||||
# CI Quality Gates
|
||||
|
||||
Sprint: `SPRINT_0350_0001_0001_ci_quality_gates_foundation`
|
||||
Task: `QGATE-0350-009`
|
||||
|
||||
## Overview
|
||||
|
||||
StellaOps implements automated quality gates in CI to enforce:
|
||||
- **Reachability Quality** - Recall/precision thresholds for vulnerability detection
|
||||
- **TTFS Regression** - Time-to-First-Signal performance tracking
|
||||
- **Performance SLOs** - Scan time and compute budget enforcement
|
||||
|
||||
These gates run as part of the `build-test-deploy.yml` workflow after the main test suite completes.
|
||||
|
||||
## Quality Gate Jobs
|
||||
|
||||
### Reachability Quality Gate
|
||||
|
||||
**Script:** `scripts/ci/compute-reachability-metrics.sh`
|
||||
**Config:** `scripts/ci/reachability-thresholds.yaml`
|
||||
|
||||
Validates that the scanner meets recall/precision thresholds against the ground-truth corpus.
|
||||
|
||||
#### Metrics Computed
|
||||
|
||||
| Metric | Description | Threshold |
|
||||
|--------|-------------|-----------|
|
||||
| `runtime_dependency_recall` | % of runtime dep vulns detected | ≥ 95% |
|
||||
| `unreachable_false_positives` | FP rate for unreachable findings | ≤ 5% |
|
||||
| `reachability_underreport` | Underreporting rate | ≤ 10% |
|
||||
| `os_package_recall` | % of OS package vulns detected | ≥ 92% |
|
||||
| `code_vuln_recall` | % of code vulns detected | ≥ 88% |
|
||||
| `config_vuln_recall` | % of config vulns detected | ≥ 85% |
|
||||
|
||||
#### Running Locally
|
||||
|
||||
```bash
|
||||
# Dry run (no enforcement)
|
||||
./scripts/ci/compute-reachability-metrics.sh --dry-run
|
||||
|
||||
# Full run against corpus
|
||||
./scripts/ci/compute-reachability-metrics.sh
|
||||
```
|
||||
|
||||
### TTFS Regression Gate
|
||||
|
||||
**Script:** `scripts/ci/compute-ttfs-metrics.sh`
|
||||
**Baseline:** `bench/baselines/ttfs-baseline.json`
|
||||
|
||||
Detects performance regressions in Time-to-First-Signal.
|
||||
|
||||
#### Metrics Computed
|
||||
|
||||
| Metric | Description | Threshold |
|
||||
|--------|-------------|-----------|
|
||||
| `ttfs_p50_ms` | P50 time to first signal | ≤ baseline + 10% |
|
||||
| `ttfs_p95_ms` | P95 time to first signal | ≤ baseline + 15% |
|
||||
| `ttfs_max_ms` | Maximum TTFS | ≤ baseline + 25% |
|
||||
|
||||
#### Baseline Format
|
||||
|
||||
```json
|
||||
{
|
||||
"ttfs_p50_ms": 450,
|
||||
"ttfs_p95_ms": 1200,
|
||||
"ttfs_max_ms": 3000,
|
||||
"measured_at": "2025-12-16T00:00:00Z",
|
||||
"sample_count": 1000
|
||||
}
|
||||
```
|
||||
|
||||
### Performance SLO Gate
|
||||
|
||||
**Script:** `scripts/ci/enforce-performance-slos.sh`
|
||||
**Config:** `scripts/ci/performance-slos.yaml`
|
||||
|
||||
Enforces scan time and compute budget SLOs.
|
||||
|
||||
#### SLOs Enforced
|
||||
|
||||
| SLO | Description | Target |
|
||||
|-----|-------------|--------|
|
||||
| `scan_time_p50_ms` | P50 scan time | ≤ 120,000ms (2 min) |
|
||||
| `scan_time_p95_ms` | P95 scan time | ≤ 300,000ms (5 min) |
|
||||
| `memory_peak_mb` | Peak memory usage | ≤ 2048 MB |
|
||||
| `cpu_seconds` | Total CPU time | ≤ 120 seconds |
|
||||
|
||||
## Workflow Integration
|
||||
|
||||
Quality gates are integrated into the main CI workflow:
|
||||
|
||||
```yaml
|
||||
# .gitea/workflows/build-test-deploy.yml
|
||||
|
||||
quality-gates:
|
||||
runs-on: ubuntu-22.04
|
||||
needs: build-test
|
||||
steps:
|
||||
- name: Reachability quality gate
|
||||
run: ./scripts/ci/compute-reachability-metrics.sh
|
||||
|
||||
- name: TTFS regression gate
|
||||
run: ./scripts/ci/compute-ttfs-metrics.sh
|
||||
|
||||
- name: Performance SLO gate
|
||||
run: ./scripts/ci/enforce-performance-slos.sh --warn-only
|
||||
```
|
||||
|
||||
## Failure Modes
|
||||
|
||||
### Hard Failure (Blocks Merge)
|
||||
|
||||
- Reachability recall below threshold
|
||||
- TTFS regression exceeds 25%
|
||||
- Memory budget exceeded by 50%
|
||||
|
||||
### Soft Failure (Warning Only)
|
||||
|
||||
- Minor TTFS regression (< 15%)
|
||||
- Memory near budget limit
|
||||
- Missing baseline data (new fixtures)
|
||||
|
||||
## Adding New Quality Gates
|
||||
|
||||
1. Create computation script in `scripts/ci/`
|
||||
2. Add threshold configuration (YAML or JSON)
|
||||
3. Integrate into workflow as a new step
|
||||
4. Update this documentation
|
||||
5. Add to sprint tracking
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Gate Fails on PR but Passes on Main
|
||||
|
||||
Check for:
|
||||
- Non-deterministic test execution
|
||||
- Timing-sensitive assertions
|
||||
- Missing test fixtures in PR branch
|
||||
|
||||
### Baseline Drift
|
||||
|
||||
If baselines become stale:
|
||||
|
||||
```bash
|
||||
# Regenerate baselines
|
||||
./scripts/ci/compute-ttfs-metrics.sh --update-baseline
|
||||
./scripts/ci/compute-reachability-metrics.sh --update-baseline
|
||||
```
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Test Suite Overview](../TEST_SUITE_OVERVIEW.md)
|
||||
- [Testing Strategy Models](./testing-strategy-models.md)
|
||||
- [Test Catalog](./TEST_CATALOG.yml)
|
||||
- [Reachability Corpus Plan](../reachability/corpus-plan.md)
|
||||
- [Performance Workbook](../PERFORMANCE_WORKBOOK.md)
|
||||
- [Testing Quality Guardrails](./testing-quality-guardrails-implementation.md)
|
||||
277
docs/technical/testing/competitor-parity-testing.md
Normal file
277
docs/technical/testing/competitor-parity-testing.md
Normal file
@@ -0,0 +1,277 @@
|
||||
# Competitor Parity Testing Guide
|
||||
|
||||
This document describes StellaOps' competitor parity testing methodology, which ensures we maintain feature parity and performance competitiveness with industry-standard scanners.
|
||||
|
||||
## Overview
|
||||
|
||||
Parity testing compares StellaOps against three primary competitors:
|
||||
|
||||
| Tool | Type | Primary Function |
|
||||
|------|------|------------------|
|
||||
| **Syft** (Anchore) | SBOM Generator | Package extraction and SBOM creation |
|
||||
| **Grype** (Anchore) | Vulnerability Scanner | CVE detection using Syft SBOMs |
|
||||
| **Trivy** (Aqua) | Multi-scanner | SBOM + vulnerability + secrets + misconfig |
|
||||
|
||||
## Goals
|
||||
|
||||
1. **Prevent regression**: Detect when StellaOps falls behind competitors on key metrics
|
||||
2. **Track trends**: Monitor parity metrics over time to identify drift patterns
|
||||
3. **Guide development**: Use competitor gaps to prioritize feature work
|
||||
4. **Validate claims**: Ensure marketing claims are backed by measurable data
|
||||
|
||||
## Fixture Set
|
||||
|
||||
Tests run against a standardized set of container images that represent diverse workloads:
|
||||
|
||||
### Quick Set (Nightly)
|
||||
- Alpine 3.19
|
||||
- Debian 12
|
||||
- Ubuntu 24.04
|
||||
- Python 3.12-slim
|
||||
- Node.js 22-alpine
|
||||
- nginx:1.25 (known vulnerabilities)
|
||||
|
||||
### Full Set (Weekly)
|
||||
All quick set images plus:
|
||||
- RHEL 9 UBI
|
||||
- Go 1.22-alpine
|
||||
- Rust 1.79-slim
|
||||
- OpenJDK 21-slim
|
||||
- .NET 8.0-alpine
|
||||
- postgres:16 (known vulnerabilities)
|
||||
- wordpress:6.5 (complex application)
|
||||
- redis:7-alpine
|
||||
- Multi-layer image (test depth)
|
||||
|
||||
## Metrics Collected
|
||||
|
||||
### SBOM Metrics
|
||||
|
||||
| Metric | Description | Target |
|
||||
|--------|-------------|--------|
|
||||
| Package Count | Total packages detected | ≥95% of Syft |
|
||||
| PURL Completeness | Packages with valid PURLs | ≥98% |
|
||||
| License Detection | Packages with license info | ≥90% |
|
||||
| CPE Mapping | Packages with CPE identifiers | ≥85% |
|
||||
|
||||
### Vulnerability Metrics
|
||||
|
||||
| Metric | Description | Target |
|
||||
|--------|-------------|--------|
|
||||
| Recall | CVEs found vs. union of all scanners | ≥95% |
|
||||
| Precision | True positives vs. total findings | ≥90% |
|
||||
| F1 Score | Harmonic mean of precision/recall | ≥92% |
|
||||
| Severity Distribution | Breakdown by Critical/High/Medium/Low | Match competitors ±10% |
|
||||
|
||||
### Latency Metrics
|
||||
|
||||
| Metric | Description | Target |
|
||||
|--------|-------------|--------|
|
||||
| P50 | Median scan time | ≤1.5x Grype |
|
||||
| P95 | 95th percentile scan time | ≤2x Grype |
|
||||
| P99 | 99th percentile scan time | ≤3x Grype |
|
||||
| Time-to-First-Signal | Time to first vulnerability found | ≤Grype |
|
||||
|
||||
### Error Handling Metrics
|
||||
|
||||
| Scenario | Expected Behavior |
|
||||
|----------|------------------|
|
||||
| Malformed manifest | Graceful degradation with partial results |
|
||||
| Network timeout | Clear error message; cached feed fallback |
|
||||
| Large image (>5GB) | Streaming extraction; no OOM |
|
||||
| Corrupt layer | Skip layer; report warning |
|
||||
| Missing base layer | Report incomplete scan |
|
||||
| Registry auth failure | Clear auth error; suggest remediation |
|
||||
| Rate limiting | Backoff + retry; clear message |
|
||||
|
||||
## Running Parity Tests
|
||||
|
||||
### Locally
|
||||
|
||||
```bash
|
||||
# Build the parity test project
|
||||
dotnet build tests/parity/StellaOps.Parity.Tests
|
||||
|
||||
# Run quick fixture set
|
||||
dotnet test tests/parity/StellaOps.Parity.Tests \
|
||||
-e PARITY_FIXTURE_SET=quick \
|
||||
-e PARITY_OUTPUT_PATH=./parity-results
|
||||
|
||||
# Run full fixture set
|
||||
dotnet test tests/parity/StellaOps.Parity.Tests \
|
||||
-e PARITY_FIXTURE_SET=full \
|
||||
-e PARITY_OUTPUT_PATH=./parity-results
|
||||
```
|
||||
|
||||
### Prerequisites
|
||||
|
||||
Ensure competitor tools are installed and in PATH:
|
||||
|
||||
```bash
|
||||
# Install Syft
|
||||
curl -sSfL https://raw.githubusercontent.com/anchore/syft/main/install.sh | sh -s -- -b /usr/local/bin v1.9.0
|
||||
|
||||
# Install Grype
|
||||
curl -sSfL https://raw.githubusercontent.com/anchore/grype/main/install.sh | sh -s -- -b /usr/local/bin v0.79.3
|
||||
|
||||
# Install Trivy
|
||||
curl -sfL https://raw.githubusercontent.com/aquasecurity/trivy/main/contrib/install.sh | sh -s -- -b /usr/local/bin v0.54.1
|
||||
```
|
||||
|
||||
### CI/CD
|
||||
|
||||
Parity tests run automatically:
|
||||
- **Nightly (02:00 UTC)**: Quick fixture set
|
||||
- **Weekly (Sunday 00:00 UTC)**: Full fixture set
|
||||
|
||||
Results are stored as workflow artifacts and optionally pushed to Prometheus.
|
||||
|
||||
## Drift Detection
|
||||
|
||||
The parity system includes automated drift detection that alerts when StellaOps falls behind competitors:
|
||||
|
||||
### Thresholds
|
||||
|
||||
| Metric | Threshold | Trend Period |
|
||||
|--------|-----------|--------------|
|
||||
| SBOM Completeness | >5% decline | 3 days |
|
||||
| Vulnerability Recall | >5% decline | 3 days |
|
||||
| Latency vs Grype | >10% increase | 3 days |
|
||||
| PURL Completeness | >5% decline | 3 days |
|
||||
| F1 Score | >5% decline | 3 days |
|
||||
|
||||
### Alert Severity
|
||||
|
||||
| Severity | Condition | Action |
|
||||
|----------|-----------|--------|
|
||||
| Low | 1-1.5x threshold | Monitor |
|
||||
| Medium | 1.5-2x threshold | Investigate within sprint |
|
||||
| High | 2-3x threshold | Prioritize fix |
|
||||
| Critical | >3x threshold | Immediate action required |
|
||||
|
||||
### Analyzing Drift
|
||||
|
||||
```bash
|
||||
# Run drift analysis on stored results
|
||||
dotnet run --project tests/parity/StellaOps.Parity.Tests \
|
||||
-- analyze-drift \
|
||||
--results-path ./parity-results \
|
||||
--threshold 0.05 \
|
||||
--trend-days 3
|
||||
```
|
||||
|
||||
## Result Storage
|
||||
|
||||
### JSON Format
|
||||
|
||||
Results are stored as timestamped JSON files:
|
||||
|
||||
```
|
||||
parity-results/
|
||||
├── parity-20250115T020000Z-abc123.json
|
||||
├── parity-20250116T020000Z-def456.json
|
||||
└── parity-20250122T000000Z-ghi789.json # Weekly
|
||||
```
|
||||
|
||||
### Retention Policy
|
||||
|
||||
- **Last 90 days**: Full detail retained
|
||||
- **Older than 90 days**: Aggregated to weekly summaries
|
||||
- **Artifacts**: Workflow artifacts retained for 90 days
|
||||
|
||||
### Prometheus Export
|
||||
|
||||
Results can be exported to Prometheus for dashboarding:
|
||||
|
||||
```
|
||||
stellaops_parity_sbom_completeness_ratio{run_id="..."} 0.97
|
||||
stellaops_parity_vuln_recall{run_id="..."} 0.95
|
||||
stellaops_parity_latency_p95_ms{scanner="stellaops",run_id="..."} 1250
|
||||
```
|
||||
|
||||
### InfluxDB Export
|
||||
|
||||
For InfluxDB time-series storage:
|
||||
|
||||
```
|
||||
parity_sbom,run_id=abc123 completeness_ratio=0.97,matched_count=142i 1705280400000000000
|
||||
parity_vuln,run_id=abc123 recall=0.95,precision=0.92,f1=0.935 1705280400000000000
|
||||
```
|
||||
|
||||
## Competitor Version Tracking
|
||||
|
||||
Competitor tools are pinned to specific versions to ensure reproducibility:
|
||||
|
||||
| Tool | Current Version | Last Updated |
|
||||
|------|-----------------|--------------|
|
||||
| Syft | 1.9.0 | 2025-01-15 |
|
||||
| Grype | 0.79.3 | 2025-01-15 |
|
||||
| Trivy | 0.54.1 | 2025-01-15 |
|
||||
|
||||
### Updating Versions
|
||||
|
||||
1. Update version in `.gitea/workflows/parity-tests.yml`
|
||||
2. Update version in `tests/parity/StellaOps.Parity.Tests/ParityHarness.cs`
|
||||
3. Run full parity test to establish new baseline
|
||||
4. Document version change in sprint execution log
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
**Syft not found**
|
||||
```
|
||||
Error: Syft executable not found in PATH
|
||||
```
|
||||
Solution: Install Syft or set `SYFT_PATH` environment variable.
|
||||
|
||||
**Grype DB outdated**
|
||||
```
|
||||
Warning: Grype vulnerability database is 7+ days old
|
||||
```
|
||||
Solution: Run `grype db update` to refresh the database.
|
||||
|
||||
**Image pull rate limit**
|
||||
```
|
||||
Error: docker pull rate limit exceeded
|
||||
```
|
||||
Solution: Use authenticated Docker Hub credentials or local registry mirror.
|
||||
|
||||
**Test timeout**
|
||||
```
|
||||
Error: Test exceeded 30 minute timeout
|
||||
```
|
||||
Solution: Increase timeout or use quick fixture set.
|
||||
|
||||
### Debug Mode
|
||||
|
||||
Enable verbose logging:
|
||||
|
||||
```bash
|
||||
dotnet test tests/parity/StellaOps.Parity.Tests \
|
||||
-e PARITY_DEBUG=true \
|
||||
-e PARITY_KEEP_OUTPUTS=true
|
||||
```
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
tests/parity/StellaOps.Parity.Tests/
|
||||
├── StellaOps.Parity.Tests.csproj # Project file
|
||||
├── ParityTestFixtureSet.cs # Container image fixtures
|
||||
├── ParityHarness.cs # Scanner execution harness
|
||||
├── SbomComparisonLogic.cs # SBOM comparison
|
||||
├── VulnerabilityComparisonLogic.cs # Vulnerability comparison
|
||||
├── LatencyComparisonLogic.cs # Latency comparison
|
||||
├── ErrorModeComparisonLogic.cs # Error handling comparison
|
||||
└── Storage/
|
||||
├── ParityResultStore.cs # Time-series storage
|
||||
└── ParityDriftDetector.cs # Drift detection
|
||||
```
|
||||
|
||||
## See Also
|
||||
|
||||
- [Scanner Architecture](../modules/scanner/architecture.md)
|
||||
- [Test Suite Overview](../TEST_SUITE_OVERVIEW.md)
|
||||
- [CI/CD Workflows](.gitea/workflows/parity-tests.yml)
|
||||
- [Competitive Benchmark](.gitea/workflows/benchmark-vs-competitors.yml)
|
||||
425
docs/technical/testing/connector-fixture-discipline.md
Normal file
425
docs/technical/testing/connector-fixture-discipline.md
Normal file
@@ -0,0 +1,425 @@
|
||||
# Connector Fixture Discipline
|
||||
|
||||
This document defines the testing discipline for StellaOps Concelier and Excititor connectors. All connectors must follow these patterns to ensure consistent, deterministic, and offline-capable testing.
|
||||
|
||||
## Overview
|
||||
|
||||
Connector tests follow **Model C1 (Connector/External)** from the testing strategy:
|
||||
|
||||
1. **Fixture-based parser tests** — Raw upstream payload → normalized internal model (offline)
|
||||
2. **Resilience tests** — Partial/bad input → deterministic failure classification
|
||||
3. **Security tests** — URL allowlist, redirect handling, payload limits
|
||||
4. **Live smoke tests** — Schema drift detection (opt-in, non-gating)
|
||||
|
||||
---
|
||||
|
||||
## 1. Directory Structure
|
||||
|
||||
Each connector test project follows this structure:
|
||||
|
||||
```
|
||||
src/<Module>/__Tests/StellaOps.<Module>.Connector.<Source>.Tests/
|
||||
├── StellaOps.<Module>.Connector.<Source>.Tests.csproj
|
||||
├── Fixtures/
|
||||
│ ├── <source>-typical.json # Typical advisory payload
|
||||
│ ├── <source>-edge-<case>.json # Edge case payloads
|
||||
│ ├── <source>-error-<type>.json # Malformed/invalid payloads
|
||||
│ └── expected-<id>.json # Expected normalized output
|
||||
├── <Source>/
|
||||
│ ├── <Source>ParserTests.cs # Parser unit tests
|
||||
│ ├── <Source>ConnectorTests.cs # Connector integration tests
|
||||
│ └── <Source>ResilienceTests.cs # Resilience/security tests
|
||||
└── Expected/
|
||||
└── <source>-<id>.canonical.json # Canonical JSON snapshots
|
||||
```
|
||||
|
||||
### Example: NVD Connector
|
||||
|
||||
```
|
||||
src/Concelier/__Tests/StellaOps.Concelier.Connector.Nvd.Tests/
|
||||
├── Fixtures/
|
||||
│ ├── nvd-window-1.json
|
||||
│ ├── nvd-window-2.json
|
||||
│ ├── nvd-multipage-1.json
|
||||
│ ├── nvd-multipage-2.json
|
||||
│ ├── nvd-invalid-schema.json
|
||||
│ └── expected-CVE-2024-0001.json
|
||||
├── Nvd/
|
||||
│ ├── NvdParserTests.cs
|
||||
│ ├── NvdConnectorTests.cs
|
||||
│ └── NvdConnectorHarnessTests.cs
|
||||
└── Expected/
|
||||
└── conflict-nvd.canonical.json
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Fixture-Based Parser Tests
|
||||
|
||||
### Purpose
|
||||
|
||||
Test that the parser correctly transforms raw upstream payloads into normalized internal models without network access.
|
||||
|
||||
### Pattern
|
||||
|
||||
```csharp
|
||||
using StellaOps.TestKit.Connectors;
|
||||
|
||||
public class NvdParserTests : ConnectorParserTestBase<NvdRawAdvisory, ConcelierAdvisory>
|
||||
{
|
||||
public NvdParserTests()
|
||||
: base(new NvdParser(), "Nvd/Fixtures")
|
||||
{
|
||||
}
|
||||
|
||||
[Fact]
|
||||
[Trait("Lane", "Unit")]
|
||||
public async Task ParseTypicalAdvisory_ProducesExpectedModel()
|
||||
{
|
||||
// Arrange
|
||||
var raw = await LoadFixture<NvdRawAdvisory>("nvd-window-1.json");
|
||||
|
||||
// Act
|
||||
var result = Parser.Parse(raw);
|
||||
|
||||
// Assert
|
||||
await AssertMatchesSnapshot(result, "expected-CVE-2024-0001.json");
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[Trait("Lane", "Unit")]
|
||||
[InlineData("nvd-multipage-1.json", "expected-multipage-1.json")]
|
||||
[InlineData("nvd-multipage-2.json", "expected-multipage-2.json")]
|
||||
public async Task ParseAllFixtures_ProducesExpectedModels(string input, string expected)
|
||||
{
|
||||
var raw = await LoadFixture<NvdRawAdvisory>(input);
|
||||
var result = Parser.Parse(raw);
|
||||
await AssertMatchesSnapshot(result, expected);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Fixture Requirements
|
||||
|
||||
| Type | Naming Convention | Purpose |
|
||||
|------|-------------------|---------|
|
||||
| Typical | `<source>-typical.json` | Normal advisory with all common fields |
|
||||
| Edge case | `<source>-edge-<case>.json` | Unusual but valid payloads |
|
||||
| Error | `<source>-error-<type>.json` | Malformed/invalid payloads |
|
||||
| Expected | `expected-<id>.json` | Expected normalized output |
|
||||
| Canonical | `<source>-<id>.canonical.json` | Deterministic JSON snapshot |
|
||||
|
||||
### Minimum Coverage
|
||||
|
||||
Each connector must have fixtures for:
|
||||
|
||||
- [ ] At least 1 typical payload
|
||||
- [ ] At least 2 edge cases (e.g., multi-vendor, unusual CVSS, missing optional fields)
|
||||
- [ ] At least 2 error cases (e.g., missing required fields, invalid schema)
|
||||
|
||||
---
|
||||
|
||||
## 3. Resilience Tests
|
||||
|
||||
### Purpose
|
||||
|
||||
Verify that connectors handle malformed input gracefully with deterministic failure classification.
|
||||
|
||||
### Pattern
|
||||
|
||||
```csharp
|
||||
public class NvdResilienceTests : ConnectorResilienceTestBase<NvdConnector>
|
||||
{
|
||||
public NvdResilienceTests()
|
||||
: base(new NvdConnector(CreateTestHttpClient()))
|
||||
{
|
||||
}
|
||||
|
||||
[Fact]
|
||||
[Trait("Lane", "Unit")]
|
||||
public async Task MissingRequiredField_ReturnsParseError()
|
||||
{
|
||||
// Arrange
|
||||
var payload = await LoadFixture("nvd-error-missing-cve-id.json");
|
||||
|
||||
// Act
|
||||
var result = await Connector.ParseAsync(payload);
|
||||
|
||||
// Assert
|
||||
Assert.False(result.IsSuccess);
|
||||
Assert.Equal(ConnectorErrorKind.ParseError, result.Error.Kind);
|
||||
Assert.Contains("cve_id", result.Error.Message, StringComparison.OrdinalIgnoreCase);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
[Trait("Lane", "Unit")]
|
||||
public async Task InvalidDateFormat_ReturnsParseError()
|
||||
{
|
||||
var payload = await LoadFixture("nvd-error-invalid-date.json");
|
||||
|
||||
var result = await Connector.ParseAsync(payload);
|
||||
|
||||
Assert.False(result.IsSuccess);
|
||||
Assert.Equal(ConnectorErrorKind.ParseError, result.Error.Kind);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
[Trait("Lane", "Unit")]
|
||||
public async Task UnexpectedEnumValue_LogsWarningAndContinues()
|
||||
{
|
||||
var payload = await LoadFixture("nvd-edge-unknown-severity.json");
|
||||
|
||||
var result = await Connector.ParseAsync(payload);
|
||||
|
||||
Assert.True(result.IsSuccess);
|
||||
Assert.Contains(result.Warnings, w => w.Contains("unknown severity"));
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Required Test Cases
|
||||
|
||||
| Case | Expected Behavior | Trait |
|
||||
|------|-------------------|-------|
|
||||
| Missing required field | `ConnectorErrorKind.ParseError` | Unit |
|
||||
| Invalid date format | `ConnectorErrorKind.ParseError` | Unit |
|
||||
| Invalid JSON structure | `ConnectorErrorKind.ParseError` | Unit |
|
||||
| Unknown enum value | Warning logged, continues | Unit |
|
||||
| Empty response | `ConnectorErrorKind.EmptyResponse` | Unit |
|
||||
| Truncated payload | `ConnectorErrorKind.ParseError` | Unit |
|
||||
|
||||
---
|
||||
|
||||
## 4. Security Tests
|
||||
|
||||
### Purpose
|
||||
|
||||
Verify that connectors enforce security boundaries for network operations.
|
||||
|
||||
### Pattern
|
||||
|
||||
```csharp
|
||||
public class NvdSecurityTests : ConnectorSecurityTestBase<NvdConnector>
|
||||
{
|
||||
[Fact]
|
||||
[Trait("Lane", "Security")]
|
||||
public async Task UrlOutsideAllowlist_RejectsRequest()
|
||||
{
|
||||
// Arrange
|
||||
var connector = CreateConnector(allowedHosts: ["services.nvd.nist.gov"]);
|
||||
|
||||
// Act & Assert
|
||||
await Assert.ThrowsAsync<SecurityException>(
|
||||
() => connector.FetchAsync("https://evil.example.com/api"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
[Trait("Lane", "Security")]
|
||||
public async Task RedirectToDisallowedHost_RejectsRequest()
|
||||
{
|
||||
var handler = CreateMockHandler(redirectTo: "https://evil.example.com");
|
||||
var connector = CreateConnector(handler, allowedHosts: ["services.nvd.nist.gov"]);
|
||||
|
||||
await Assert.ThrowsAsync<SecurityException>(
|
||||
() => connector.FetchAsync("https://services.nvd.nist.gov/api"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
[Trait("Lane", "Security")]
|
||||
public async Task PayloadExceedsMaxSize_RejectsPayload()
|
||||
{
|
||||
var handler = CreateMockHandler(responseSize: 100_000_001); // 100MB
|
||||
var connector = CreateConnector(handler, maxPayloadBytes: 100_000_000);
|
||||
|
||||
await Assert.ThrowsAsync<PayloadTooLargeException>(
|
||||
() => connector.FetchAsync("https://services.nvd.nist.gov/api"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
[Trait("Lane", "Security")]
|
||||
public async Task DecompressionBomb_RejectsPayload()
|
||||
{
|
||||
// 1KB compressed, 1GB decompressed
|
||||
var handler = CreateMockHandler(compressedBomb: true);
|
||||
var connector = CreateConnector(handler);
|
||||
|
||||
await Assert.ThrowsAsync<PayloadTooLargeException>(
|
||||
() => connector.FetchAsync("https://services.nvd.nist.gov/api"));
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Required Security Tests
|
||||
|
||||
| Test | Purpose | Trait |
|
||||
|------|---------|-------|
|
||||
| URL allowlist | Block requests to unauthorized hosts | Security |
|
||||
| Redirect validation | Block redirects to unauthorized hosts | Security |
|
||||
| Max payload size | Reject oversized responses | Security |
|
||||
| Decompression bomb | Reject zip bombs | Security |
|
||||
| Rate limiting | Respect upstream rate limits | Security |
|
||||
|
||||
---
|
||||
|
||||
## 5. Live Smoke Tests (Opt-In)
|
||||
|
||||
### Purpose
|
||||
|
||||
Detect upstream schema drift by comparing live responses against known fixtures.
|
||||
|
||||
### Pattern
|
||||
|
||||
```csharp
|
||||
public class NvdLiveTests : ConnectorLiveTestBase<NvdConnector>
|
||||
{
|
||||
[Fact]
|
||||
[Trait("Lane", "Live")]
|
||||
[Trait("Category", "SchemaDrift")]
|
||||
public async Task LiveSchema_MatchesFixtureSchema()
|
||||
{
|
||||
// Skip if not in Live lane
|
||||
Skip.IfNot(IsLiveLaneEnabled());
|
||||
|
||||
// Fetch live response
|
||||
var live = await Connector.FetchLatestAsync();
|
||||
|
||||
// Compare schema (not values) against fixture
|
||||
var fixture = await LoadFixture<NvdResponse>("nvd-typical.json");
|
||||
|
||||
AssertSchemaMatches(live, fixture);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
[Trait("Lane", "Live")]
|
||||
public async Task LiveFetch_ReturnsValidAdvisories()
|
||||
{
|
||||
Skip.IfNot(IsLiveLaneEnabled());
|
||||
|
||||
var result = await Connector.FetchLatestAsync();
|
||||
|
||||
Assert.True(result.IsSuccess);
|
||||
Assert.NotEmpty(result.Value.Advisories);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Configuration
|
||||
|
||||
Live tests are:
|
||||
- **Never PR-gating** — run only in scheduled/nightly jobs
|
||||
- **Opt-in** — require explicit `LIVE_TESTS_ENABLED=true` environment variable
|
||||
- **Alerting** — schema drift triggers notification, not failure
|
||||
|
||||
---
|
||||
|
||||
## 6. Fixture Updater
|
||||
|
||||
### Purpose
|
||||
|
||||
Refresh fixtures from live sources when upstream schemas change intentionally.
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
# Update all fixtures for NVD connector
|
||||
dotnet run --project tools/FixtureUpdater -- \
|
||||
--connector nvd \
|
||||
--output src/Concelier/__Tests/StellaOps.Concelier.Connector.Nvd.Tests/Nvd/Fixtures/
|
||||
|
||||
# Update specific fixture
|
||||
dotnet run --project tools/FixtureUpdater -- \
|
||||
--connector nvd \
|
||||
--cve CVE-2024-0001 \
|
||||
--output src/Concelier/__Tests/.../Fixtures/nvd-CVE-2024-0001.json
|
||||
|
||||
# Dry-run mode (show diff without writing)
|
||||
dotnet run --project tools/FixtureUpdater -- \
|
||||
--connector nvd \
|
||||
--dry-run
|
||||
```
|
||||
|
||||
### Workflow
|
||||
|
||||
1. Live test detects schema drift
|
||||
2. CI creates draft PR with fixture update
|
||||
3. Developer reviews diff for intentional vs accidental changes
|
||||
4. If intentional: update parser and merge
|
||||
5. If accidental: investigate upstream API issue
|
||||
|
||||
---
|
||||
|
||||
## 7. Test Traits and CI Integration
|
||||
|
||||
### Trait Assignment
|
||||
|
||||
| Test Category | Trait | Lane | PR-Gating |
|
||||
|---------------|-------|------|-----------|
|
||||
| Parser tests | `[Trait("Lane", "Unit")]` | Unit | Yes |
|
||||
| Resilience tests | `[Trait("Lane", "Unit")]` | Unit | Yes |
|
||||
| Security tests | `[Trait("Lane", "Security")]` | Security | Yes |
|
||||
| Live tests | `[Trait("Lane", "Live")]` | Live | No |
|
||||
|
||||
### Running Tests
|
||||
|
||||
```bash
|
||||
# Run all connector unit tests
|
||||
dotnet test --filter "Lane=Unit" src/Concelier/__Tests/
|
||||
|
||||
# Run security tests
|
||||
dotnet test --filter "Lane=Security" src/Concelier/__Tests/
|
||||
|
||||
# Run live tests (requires LIVE_TESTS_ENABLED=true)
|
||||
LIVE_TESTS_ENABLED=true dotnet test --filter "Lane=Live" src/Concelier/__Tests/
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8. Connector Inventory
|
||||
|
||||
### Concelier Connectors (Advisory Sources)
|
||||
|
||||
| Connector | Fixtures | Parser Tests | Resilience | Security | Live |
|
||||
|-----------|----------|--------------|------------|----------|------|
|
||||
| NVD | ✅ | ✅ | ✅ | ⬜ | ⬜ |
|
||||
| OSV | ✅ | ✅ | ⬜ | ⬜ | ⬜ |
|
||||
| GHSA | ✅ | ✅ | ⬜ | ⬜ | ⬜ |
|
||||
| CVE | ✅ | ✅ | ⬜ | ⬜ | ⬜ |
|
||||
| KEV | ✅ | ✅ | ⬜ | ⬜ | ⬜ |
|
||||
| EPSS | ✅ | ✅ | ⬜ | ⬜ | ⬜ |
|
||||
| Distro.Alpine | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ |
|
||||
| Distro.Debian | ✅ | ✅ | ⬜ | ⬜ | ⬜ |
|
||||
| Distro.RedHat | ✅ | ✅ | ⬜ | ⬜ | ⬜ |
|
||||
| Distro.Suse | ✅ | ✅ | ⬜ | ⬜ | ⬜ |
|
||||
| Distro.Ubuntu | ✅ | ✅ | ⬜ | ⬜ | ⬜ |
|
||||
| Vndr.Adobe | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ |
|
||||
| Vndr.Apple | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ |
|
||||
| Vndr.Cisco | ✅ | ✅ | ⬜ | ⬜ | ⬜ |
|
||||
| Vndr.Msrc | ✅ | ✅ | ⬜ | ⬜ | ⬜ |
|
||||
| Vndr.Oracle | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ |
|
||||
| Vndr.Vmware | ✅ | ✅ | ⬜ | ⬜ | ⬜ |
|
||||
| Cert.Bund | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ |
|
||||
| Cert.Cc | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ |
|
||||
| Cert.Fr | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ |
|
||||
| ICS.Cisa | ✅ | ✅ | ⬜ | ⬜ | ⬜ |
|
||||
|
||||
### Excititor Connectors (VEX Sources)
|
||||
|
||||
| Connector | Fixtures | Parser Tests | Resilience | Security | Live |
|
||||
|-----------|----------|--------------|------------|----------|------|
|
||||
| OpenVEX | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ |
|
||||
| CSAF/VEX | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ |
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- [ConnectorHttpFixture](../../src/__Libraries/StellaOps.TestKit/Connectors/ConnectorHttpFixture.cs)
|
||||
- [ConnectorTestBase](../../src/__Libraries/StellaOps.TestKit/Connectors/ConnectorTestBase.cs)
|
||||
- [ConnectorResilienceTestBase](../../src/__Libraries/StellaOps.TestKit/Connectors/ConnectorResilienceTestBase.cs)
|
||||
- [FixtureUpdater](../../src/__Libraries/StellaOps.TestKit/Connectors/FixtureUpdater.cs)
|
||||
- [Testing Strategy Models](./testing-strategy-models.md)
|
||||
- [CI Lane Filters](./ci-lane-filters.md)
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2025-12-24 · Sprint 5100.0007.0005*
|
||||
501
docs/technical/testing/cross-cutting-testing-guide.md
Normal file
501
docs/technical/testing/cross-cutting-testing-guide.md
Normal file
@@ -0,0 +1,501 @@
|
||||
# Cross-Cutting Testing Standards Guide
|
||||
|
||||
This guide documents the cross-cutting testing standards implemented for StellaOps, including blast-radius annotations, schema evolution testing, dead-path detection, and config-diff testing.
|
||||
|
||||
**Sprint Reference:** SPRINT_20260105_002_005_TEST_cross_cutting
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [Overview](#overview)
|
||||
2. [Blast-Radius Annotations](#blast-radius-annotations)
|
||||
3. [Schema Evolution Testing](#schema-evolution-testing)
|
||||
4. [Dead-Path Detection](#dead-path-detection)
|
||||
5. [Config-Diff Testing](#config-diff-testing)
|
||||
6. [CI Workflows](#ci-workflows)
|
||||
7. [Best Practices](#best-practices)
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Cross-cutting testing standards ensure consistent test quality across all modules:
|
||||
|
||||
| Standard | Purpose | Enforcement |
|
||||
|----------|---------|-------------|
|
||||
| **Blast-Radius** | Categorize tests by operational surface | CI validation on PRs |
|
||||
| **Schema Evolution** | Verify backward compatibility | CI on schema changes |
|
||||
| **Dead-Path Detection** | Identify uncovered code | CI with baseline comparison |
|
||||
| **Config-Diff** | Validate config behavioral isolation | Integration tests |
|
||||
|
||||
---
|
||||
|
||||
## Blast-Radius Annotations
|
||||
|
||||
### Purpose
|
||||
|
||||
Blast-radius annotations categorize tests by the operational surfaces they affect. During incidents, this enables targeted test runs for specific areas (e.g., run only Auth-related tests when investigating an authentication issue).
|
||||
|
||||
### Categories
|
||||
|
||||
| Category | Description | Examples |
|
||||
|----------|-------------|----------|
|
||||
| `Auth` | Authentication, authorization, tokens | Login, OAuth, DPoP |
|
||||
| `Scanning` | SBOM generation, vulnerability scanning | Scanner, analyzers |
|
||||
| `Evidence` | Attestation, evidence storage | EvidenceLocker, Attestor |
|
||||
| `Compliance` | Audit, regulatory, GDPR | Compliance reports |
|
||||
| `Advisories` | Advisory ingestion, VEX processing | Concelier, VexLens |
|
||||
| `RiskPolicy` | Risk scoring, policy evaluation | RiskEngine, Policy |
|
||||
| `Crypto` | Cryptographic operations | Signing, verification |
|
||||
| `Integrations` | External systems, webhooks | Notifications, webhooks |
|
||||
| `Persistence` | Database operations | Repositories, migrations |
|
||||
| `Api` | API surface, contracts | Controllers, endpoints |
|
||||
|
||||
### Usage
|
||||
|
||||
```csharp
|
||||
using StellaOps.TestKit;
|
||||
using Xunit;
|
||||
|
||||
// Single blast-radius
|
||||
[Trait("Category", TestCategories.Integration)]
|
||||
[Trait("BlastRadius", TestCategories.BlastRadius.Auth)]
|
||||
public class TokenValidationTests
|
||||
{
|
||||
[Fact]
|
||||
public async Task ValidToken_ReturnsSuccess()
|
||||
{
|
||||
// Test implementation
|
||||
}
|
||||
}
|
||||
|
||||
// Multiple blast-radii (affects multiple surfaces)
|
||||
[Trait("Category", TestCategories.Integration)]
|
||||
[Trait("BlastRadius", TestCategories.BlastRadius.Auth)]
|
||||
[Trait("BlastRadius", TestCategories.BlastRadius.Api)]
|
||||
public class AuthenticatedApiTests
|
||||
{
|
||||
// Tests that affect both Auth and Api surfaces
|
||||
}
|
||||
```
|
||||
|
||||
### Requirements
|
||||
|
||||
- **Integration tests**: Must have at least one BlastRadius annotation
|
||||
- **Contract tests**: Must have at least one BlastRadius annotation
|
||||
- **Security tests**: Must have at least one BlastRadius annotation
|
||||
- **Unit tests**: BlastRadius optional but recommended
|
||||
|
||||
### Running Tests by Blast-Radius
|
||||
|
||||
```bash
|
||||
# Run all Auth-related tests
|
||||
dotnet test --filter "BlastRadius=Auth"
|
||||
|
||||
# Run tests for multiple surfaces
|
||||
dotnet test --filter "BlastRadius=Auth|BlastRadius=Api"
|
||||
|
||||
# Run incident response test suite
|
||||
dotnet run --project src/__Libraries/StellaOps.TestKit \
|
||||
-- run-blast-radius Auth,Api --fail-fast
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Schema Evolution Testing
|
||||
|
||||
### Purpose
|
||||
|
||||
Schema evolution tests verify that code remains compatible with previous database schema versions. This prevents breaking changes during:
|
||||
|
||||
- Rolling deployments (new code, old schema)
|
||||
- Rollbacks (old code, new schema)
|
||||
- Migration windows
|
||||
|
||||
### Schema Versions
|
||||
|
||||
| Version | Description |
|
||||
|---------|-------------|
|
||||
| `N` | Current schema (HEAD) |
|
||||
| `N-1` | Previous schema version |
|
||||
| `N-2` | Two versions back |
|
||||
|
||||
### Using SchemaEvolutionTestBase
|
||||
|
||||
```csharp
|
||||
using StellaOps.Testing.SchemaEvolution;
|
||||
using Testcontainers.PostgreSql;
|
||||
using Xunit;
|
||||
|
||||
[Trait("Category", TestCategories.SchemaEvolution)]
|
||||
public class ScannerSchemaEvolutionTests : PostgresSchemaEvolutionTestBase
|
||||
{
|
||||
public ScannerSchemaEvolutionTests()
|
||||
: base(new SchemaEvolutionConfig
|
||||
{
|
||||
ModuleName = "Scanner",
|
||||
CurrentVersion = new SchemaVersion("v2.1.0",
|
||||
DateTimeOffset.Parse("2026-01-01")),
|
||||
PreviousVersions =
|
||||
[
|
||||
new SchemaVersion("v2.0.0",
|
||||
DateTimeOffset.Parse("2025-10-01")),
|
||||
new SchemaVersion("v1.9.0",
|
||||
DateTimeOffset.Parse("2025-07-01"))
|
||||
],
|
||||
ConnectionStringTemplate =
|
||||
"Host={0};Port={1};Database={2};Username={3};Password={4}"
|
||||
})
|
||||
{
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ReadOperations_CompatibleWithPreviousSchema()
|
||||
{
|
||||
var result = await TestReadBackwardCompatibilityAsync(
|
||||
async (connection, version) =>
|
||||
{
|
||||
// Test read operations against old schema
|
||||
var repository = new ScanRepository(connection);
|
||||
var scans = await repository.GetRecentScansAsync(10);
|
||||
return scans.Count >= 0;
|
||||
});
|
||||
|
||||
Assert.True(result.IsSuccess);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task WriteOperations_CompatibleWithPreviousSchema()
|
||||
{
|
||||
var result = await TestWriteForwardCompatibilityAsync(
|
||||
async (connection, version) =>
|
||||
{
|
||||
// Test write operations
|
||||
var repository = new ScanRepository(connection);
|
||||
await repository.CreateScanAsync(new ScanRequest { /* ... */ });
|
||||
return true;
|
||||
});
|
||||
|
||||
Assert.True(result.IsSuccess);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Versioned Container Images
|
||||
|
||||
Build versioned PostgreSQL images for testing:
|
||||
|
||||
```bash
|
||||
# Build all versions for a module
|
||||
./devops/docker/schema-versions/build-schema-images.sh scanner
|
||||
|
||||
# Build specific version
|
||||
./devops/docker/schema-versions/build-schema-images.sh scanner v2.0.0
|
||||
|
||||
# Use in tests
|
||||
docker run -d -p 5432:5432 ghcr.io/stellaops/schema-test:scanner-v2.0.0
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Dead-Path Detection
|
||||
|
||||
### Purpose
|
||||
|
||||
Dead-path detection identifies uncovered code branches. This helps:
|
||||
|
||||
- Find untested edge cases
|
||||
- Identify potentially dead code
|
||||
- Prevent coverage regression
|
||||
|
||||
### How It Works
|
||||
|
||||
1. Tests run with branch coverage collection (Coverlet)
|
||||
2. Cobertura XML report is parsed
|
||||
3. Uncovered branches are identified
|
||||
4. New dead paths are compared against baseline
|
||||
5. CI fails if new dead paths are introduced
|
||||
|
||||
### Baseline Management
|
||||
|
||||
The baseline file (`dead-paths-baseline.json`) tracks known dead paths:
|
||||
|
||||
```json
|
||||
{
|
||||
"version": "1.0.0",
|
||||
"activeDeadPaths": 42,
|
||||
"totalDeadPaths": 50,
|
||||
"exemptedPaths": 8,
|
||||
"entries": [
|
||||
{
|
||||
"file": "src/Scanner/Services/AnalyzerService.cs",
|
||||
"line": 128,
|
||||
"coverage": "1/2",
|
||||
"isExempt": false
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Exemptions
|
||||
|
||||
Add exemptions for intentionally untested code in `coverage-exemptions.yaml`:
|
||||
|
||||
```yaml
|
||||
exemptions:
|
||||
- path: "src/Authority/Emergency/BreakGlassHandler.cs:42"
|
||||
category: emergency
|
||||
justification: "Emergency access bypass - tested in incident drills"
|
||||
added: "2026-01-06"
|
||||
owner: "security-team"
|
||||
|
||||
- path: "src/Scanner/Platform/WindowsRegistryScanner.cs:*"
|
||||
category: platform
|
||||
justification: "Windows-only code - CI runs on Linux"
|
||||
added: "2026-01-06"
|
||||
owner: "scanner-team"
|
||||
|
||||
ignore_patterns:
|
||||
- "*.Generated.cs"
|
||||
- "**/Migrations/*.cs"
|
||||
```
|
||||
|
||||
### Using BranchCoverageEnforcer
|
||||
|
||||
```csharp
|
||||
using StellaOps.Testing.Coverage;
|
||||
|
||||
var enforcer = new BranchCoverageEnforcer(new BranchCoverageConfig
|
||||
{
|
||||
MinimumBranchCoverage = 80,
|
||||
FailOnNewDeadPaths = true,
|
||||
ExemptionFiles = ["coverage-exemptions.yaml"]
|
||||
});
|
||||
|
||||
// Parse coverage report
|
||||
var parser = new CoberturaParser();
|
||||
var coverage = await parser.ParseFileAsync("coverage.cobertura.xml");
|
||||
|
||||
// Validate
|
||||
var result = enforcer.Validate(coverage);
|
||||
if (!result.IsValid)
|
||||
{
|
||||
foreach (var violation in result.Violations)
|
||||
{
|
||||
Console.WriteLine($"Violation: {violation.File}:{violation.Line}");
|
||||
}
|
||||
}
|
||||
|
||||
// Generate dead-path report
|
||||
var report = enforcer.GenerateDeadPathReport(coverage);
|
||||
Console.WriteLine($"Active dead paths: {report.ActiveDeadPaths}");
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Config-Diff Testing
|
||||
|
||||
### Purpose
|
||||
|
||||
Config-diff tests verify that configuration changes produce only expected behavioral deltas. This prevents:
|
||||
|
||||
- Unintended side effects from config changes
|
||||
- Config options affecting unrelated behaviors
|
||||
- Regressions in config handling
|
||||
|
||||
### Using ConfigDiffTestBase
|
||||
|
||||
```csharp
|
||||
using StellaOps.Testing.ConfigDiff;
|
||||
using Xunit;
|
||||
|
||||
[Trait("Category", TestCategories.ConfigDiff)]
|
||||
public class ConcelierConfigDiffTests : ConfigDiffTestBase
|
||||
{
|
||||
[Fact]
|
||||
public async Task ChangingCacheTimeout_OnlyAffectsCacheBehavior()
|
||||
{
|
||||
var baselineConfig = new ConcelierOptions
|
||||
{
|
||||
CacheTimeoutMinutes = 30,
|
||||
MaxConcurrentDownloads = 10
|
||||
};
|
||||
|
||||
var changedConfig = baselineConfig with
|
||||
{
|
||||
CacheTimeoutMinutes = 60
|
||||
};
|
||||
|
||||
var result = await TestConfigIsolationAsync(
|
||||
baselineConfig,
|
||||
changedConfig,
|
||||
changedSetting: "CacheTimeoutMinutes",
|
||||
unrelatedBehaviors:
|
||||
[
|
||||
async config => await GetDownloadBehavior(config),
|
||||
async config => await GetParseBehavior(config),
|
||||
async config => await GetMergeBehavior(config)
|
||||
]);
|
||||
|
||||
Assert.True(result.IsSuccess,
|
||||
$"Unexpected changes: {string.Join(", ", result.UnexpectedChanges)}");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ChangingRetryPolicy_ProducesExpectedDelta()
|
||||
{
|
||||
var baseline = new ConcelierOptions { MaxRetries = 3 };
|
||||
var changed = new ConcelierOptions { MaxRetries = 5 };
|
||||
|
||||
var expectedDelta = new ConfigDelta(
|
||||
ChangedBehaviors: ["RetryCount", "TotalRequestTime"],
|
||||
BehaviorDeltas:
|
||||
[
|
||||
new BehaviorDelta("RetryCount", "3", "5", null),
|
||||
new BehaviorDelta("TotalRequestTime", "increase", null,
|
||||
"More retries = longer total time")
|
||||
]);
|
||||
|
||||
var result = await TestConfigBehavioralDeltaAsync(
|
||||
baseline,
|
||||
changed,
|
||||
getBehavior: async config => await CaptureRetryBehavior(config),
|
||||
computeDelta: ComputeBehaviorSnapshotDelta,
|
||||
expectedDelta: expectedDelta);
|
||||
|
||||
Assert.True(result.IsSuccess);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Behavior Snapshots
|
||||
|
||||
Capture behavior at specific configuration states:
|
||||
|
||||
```csharp
|
||||
var snapshot = CreateSnapshotBuilder("baseline-config")
|
||||
.AddBehavior("CacheHitRate", cacheMetrics.HitRate)
|
||||
.AddBehavior("ResponseTime", responseMetrics.P99)
|
||||
.AddBehavior("ErrorRate", errorMetrics.Rate)
|
||||
.WithCapturedAt(DateTimeOffset.UtcNow)
|
||||
.Build();
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## CI Workflows
|
||||
|
||||
### Available Workflows
|
||||
|
||||
| Workflow | File | Trigger |
|
||||
|----------|------|---------|
|
||||
| Blast-Radius Validation | `test-blast-radius.yml` | PRs with test changes |
|
||||
| Dead-Path Detection | `dead-path-detection.yml` | Push to main, PRs |
|
||||
| Schema Evolution | `schema-evolution.yml` | Schema/migration changes |
|
||||
| Rollback Lag | `rollback-lag.yml` | Manual trigger, weekly |
|
||||
| Test Infrastructure | `test-infrastructure.yml` | All changes, nightly |
|
||||
|
||||
### Workflow Outputs
|
||||
|
||||
Each workflow posts results as PR comments:
|
||||
|
||||
```markdown
|
||||
## Test Infrastructure :white_check_mark: All checks passed
|
||||
|
||||
| Check | Status | Details |
|
||||
|-------|--------|---------|
|
||||
| Blast-Radius | :white_check_mark: | 0 violations |
|
||||
| Dead-Path Detection | :white_check_mark: | Coverage: 82.5% |
|
||||
| Schema Evolution | :white_check_mark: | Compatible: N-1,N-2 |
|
||||
| Config-Diff | :white_check_mark: | Tested: Concelier,Authority,Scanner |
|
||||
```
|
||||
|
||||
### Running Locally
|
||||
|
||||
```bash
|
||||
# Blast-radius validation
|
||||
dotnet test --filter "Category=Integration" | grep BlastRadius
|
||||
|
||||
# Dead-path detection
|
||||
dotnet test /p:CollectCoverage=true /p:CoverletOutputFormat=cobertura
|
||||
|
||||
# Schema evolution (requires Docker)
|
||||
docker-compose -f devops/compose/schema-test.yml up -d
|
||||
dotnet test --filter "Category=SchemaEvolution"
|
||||
|
||||
# Config-diff
|
||||
dotnet test --filter "Category=ConfigDiff"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Best Practices
|
||||
|
||||
### General Guidelines
|
||||
|
||||
1. **Test categories**: Always categorize tests correctly
|
||||
- Unit tests: Pure logic, no I/O
|
||||
- Integration tests: Database, network, external systems
|
||||
- Contract tests: API contracts, schemas
|
||||
- Security tests: Authentication, authorization, injection
|
||||
|
||||
2. **Blast-radius**: Choose the narrowest applicable category
|
||||
- If a test affects Auth only, use `BlastRadius.Auth`
|
||||
- If it affects Auth and Api, use both
|
||||
|
||||
3. **Schema evolution**: Test both read and write paths
|
||||
- Read compatibility: Old data readable by new code
|
||||
- Write compatibility: New code writes valid old-schema data
|
||||
|
||||
4. **Dead-path exemptions**: Document thoroughly
|
||||
- Include justification
|
||||
- Set owner and review date
|
||||
- Remove when no longer applicable
|
||||
|
||||
5. **Config-diff**: Focus on high-impact options
|
||||
- Security-related configs
|
||||
- Performance-related configs
|
||||
- Feature flags
|
||||
|
||||
### Code Review Checklist
|
||||
|
||||
- [ ] Integration/Contract/Security tests have BlastRadius annotations
|
||||
- [ ] Schema changes include evolution tests
|
||||
- [ ] New branches have test coverage
|
||||
- [ ] Config option tests verify isolation
|
||||
- [ ] Exemptions have justifications
|
||||
|
||||
### Troubleshooting
|
||||
|
||||
**Blast-radius validation fails:**
|
||||
```bash
|
||||
# Find tests missing BlastRadius
|
||||
dotnet test --filter "Category=Integration" --list-tests | \
|
||||
xargs -I {} grep -L "BlastRadius" {}
|
||||
```
|
||||
|
||||
**Dead-path baseline drift:**
|
||||
```bash
|
||||
# Regenerate baseline
|
||||
dotnet test /p:CollectCoverage=true
|
||||
python extract-dead-paths.py coverage.cobertura.xml
|
||||
cp dead-paths-report.json dead-paths-baseline.json
|
||||
```
|
||||
|
||||
**Schema evolution test fails:**
|
||||
```bash
|
||||
# Check schema version compatibility
|
||||
docker run -it ghcr.io/stellaops/schema-test:scanner-v2.0.0 \
|
||||
psql -U stellaops_test -d stellaops_schema_test \
|
||||
-c "SELECT * FROM _schema_metadata;"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Test Infrastructure Overview](../testing/README.md)
|
||||
- [Database Schema Specification](../db/SPECIFICATION.md)
|
||||
- [CI/CD Workflows](../../.gitea/workflows/README.md)
|
||||
- [Module Testing Agents](../../src/__Tests/AGENTS.md)
|
||||
291
docs/technical/testing/determinism-gates.md
Normal file
291
docs/technical/testing/determinism-gates.md
Normal file
@@ -0,0 +1,291 @@
|
||||
# Determinism Gates
|
||||
|
||||
Determinism is a core principle of StellaOps - all artifact generation (SBOM, VEX, attestations) must be reproducible. This document describes how to test for determinism.
|
||||
|
||||
## Why Determinism Matters
|
||||
|
||||
- **Reproducible builds**: Same input → same output, always
|
||||
- **Cryptographic verification**: Hash-based integrity depends on byte-for-byte reproducibility
|
||||
- **Audit trails**: Deterministic timestamps and ordering for compliance
|
||||
- **Offline operation**: No reliance on external randomness or timestamps
|
||||
|
||||
## Using Determinism Gates
|
||||
|
||||
Add `StellaOps.TestKit` to your test project and use the `DeterminismGate` class:
|
||||
|
||||
```csharp
|
||||
using StellaOps.TestKit.Determinism;
|
||||
using StellaOps.TestKit.Traits;
|
||||
using Xunit;
|
||||
|
||||
public class SbomGeneratorTests
|
||||
{
|
||||
[Fact]
|
||||
[UnitTest]
|
||||
[DeterminismTest]
|
||||
public void SbomGenerationIsDeterministic()
|
||||
{
|
||||
// Verify that calling the function 3 times produces identical output
|
||||
DeterminismGate.AssertDeterministic(() =>
|
||||
{
|
||||
return GenerateSbom();
|
||||
}, iterations: 3);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
[UnitTest]
|
||||
[DeterminismTest]
|
||||
public void SbomBinaryIsDeterministic()
|
||||
{
|
||||
// Verify binary reproducibility
|
||||
DeterminismGate.AssertDeterministic(() =>
|
||||
{
|
||||
return GenerateSbomBytes();
|
||||
}, iterations: 3);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## JSON Determinism
|
||||
|
||||
JSON output must have:
|
||||
- Stable property ordering (alphabetical or schema-defined)
|
||||
- Consistent whitespace/formatting
|
||||
- No random IDs or timestamps (unless explicitly from deterministic clock)
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
[UnitTest]
|
||||
[DeterminismTest]
|
||||
public void VexDocumentJsonIsDeterministic()
|
||||
{
|
||||
// Verifies JSON canonicalization and property ordering
|
||||
DeterminismGate.AssertJsonDeterministic(() =>
|
||||
{
|
||||
var vex = GenerateVexDocument();
|
||||
return JsonSerializer.Serialize(vex);
|
||||
});
|
||||
}
|
||||
|
||||
[Fact]
|
||||
[UnitTest]
|
||||
[DeterminismTest]
|
||||
public void VerdictObjectIsDeterministic()
|
||||
{
|
||||
// Verifies object serialization is deterministic
|
||||
DeterminismGate.AssertJsonDeterministic(() =>
|
||||
{
|
||||
return GenerateVerdict();
|
||||
});
|
||||
}
|
||||
```
|
||||
|
||||
## Canonical Equality
|
||||
|
||||
Compare two objects for canonical equivalence:
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
[UnitTest]
|
||||
public void VerdictFromDifferentPathsAreCanonicallyEqual()
|
||||
{
|
||||
var verdict1 = GenerateVerdictFromSbom();
|
||||
var verdict2 = GenerateVerdictFromCache();
|
||||
|
||||
// Asserts that both produce identical canonical JSON
|
||||
DeterminismGate.AssertCanonicallyEqual(verdict1, verdict2);
|
||||
}
|
||||
```
|
||||
|
||||
## Hash-Based Regression Testing
|
||||
|
||||
Compute stable hashes for regression detection:
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
[UnitTest]
|
||||
[DeterminismTest]
|
||||
public void SbomHashMatchesBaseline()
|
||||
{
|
||||
var sbom = GenerateSbom();
|
||||
var hash = DeterminismGate.ComputeHash(sbom);
|
||||
|
||||
// This hash should NEVER change unless SBOM format changes intentionally
|
||||
const string expectedHash = "abc123...";
|
||||
Assert.Equal(expectedHash, hash);
|
||||
}
|
||||
```
|
||||
|
||||
## Path Ordering
|
||||
|
||||
File paths in manifests must be sorted:
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
[UnitTest]
|
||||
[DeterminismTest]
|
||||
public void SbomFilePathsAreSorted()
|
||||
{
|
||||
var sbom = GenerateSbom();
|
||||
var filePaths = ExtractFilePaths(sbom);
|
||||
|
||||
// Asserts paths are in deterministic (lexicographic) order
|
||||
DeterminismGate.AssertSortedPaths(filePaths);
|
||||
}
|
||||
```
|
||||
|
||||
## Timestamp Validation
|
||||
|
||||
All timestamps must be UTC ISO 8601:
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
[UnitTest]
|
||||
[DeterminismTest]
|
||||
public void AttestationTimestampIsUtcIso8601()
|
||||
{
|
||||
var attestation = GenerateAttestation();
|
||||
|
||||
// Asserts timestamp is UTC with 'Z' suffix
|
||||
DeterminismGate.AssertUtcIso8601(attestation.Timestamp);
|
||||
}
|
||||
```
|
||||
|
||||
## Determin
|
||||
|
||||
istic Time in Tests
|
||||
|
||||
Use `DeterministicClock` for reproducible timestamps:
|
||||
|
||||
```csharp
|
||||
using StellaOps.TestKit.Time;
|
||||
|
||||
[Fact]
|
||||
[UnitTest]
|
||||
[DeterminismTest]
|
||||
public void AttestationWithDeterministicTime()
|
||||
{
|
||||
var clock = new DeterministicClock();
|
||||
|
||||
// All operations using this clock will get the same time
|
||||
var attestation1 = GenerateAttestation(clock);
|
||||
var attestation2 = GenerateAttestation(clock);
|
||||
|
||||
Assert.Equal(attestation1.Timestamp, attestation2.Timestamp);
|
||||
}
|
||||
```
|
||||
|
||||
## Deterministic Random in Tests
|
||||
|
||||
Use `DeterministicRandom` for reproducible randomness:
|
||||
|
||||
```csharp
|
||||
using StellaOps.TestKit.Random;
|
||||
|
||||
[Fact]
|
||||
[UnitTest]
|
||||
[DeterminismTest]
|
||||
public void GeneratedIdsAreReproducible()
|
||||
{
|
||||
var rng1 = DeterministicRandomExtensions.WithTestSeed();
|
||||
var id1 = GenerateId(rng1);
|
||||
|
||||
var rng2 = DeterministicRandomExtensions.WithTestSeed();
|
||||
var id2 = GenerateId(rng2);
|
||||
|
||||
// Same seed → same output
|
||||
Assert.Equal(id1, id2);
|
||||
}
|
||||
```
|
||||
|
||||
## Module-Specific Gates
|
||||
|
||||
### Scanner Determinism
|
||||
- SBOM file path ordering
|
||||
- Component hash stability
|
||||
- Dependency graph ordering
|
||||
|
||||
### Concelier Determinism
|
||||
- Advisory normalization (same advisory → same canonical form)
|
||||
- Vulnerability merge determinism
|
||||
- No lattice ordering dependencies
|
||||
|
||||
### Excititor Determinism
|
||||
- VEX document format stability
|
||||
- Preserve/prune decision ordering
|
||||
- No lattice dependencies
|
||||
|
||||
### Policy Determinism
|
||||
- Verdict reproducibility (same inputs → same verdict)
|
||||
- Policy evaluation ordering
|
||||
- Unknown budget calculations
|
||||
|
||||
### Attestor Determinism
|
||||
- DSSE envelope canonical bytes
|
||||
- Signature ordering (multiple signers)
|
||||
- Rekor receipt stability
|
||||
|
||||
## Common Determinism Violations
|
||||
|
||||
❌ **Timestamps from system clock**
|
||||
```csharp
|
||||
// Bad: Uses system time
|
||||
var timestamp = DateTimeOffset.UtcNow.ToString("o");
|
||||
|
||||
// Good: Uses injected clock
|
||||
var timestamp = clock.UtcNow.ToString("o");
|
||||
```
|
||||
|
||||
❌ **Random GUIDs**
|
||||
```csharp
|
||||
// Bad: Non-deterministic
|
||||
var id = Guid.NewGuid().ToString();
|
||||
|
||||
// Good: Deterministic or content-addressed
|
||||
var id = ComputeContentHash(data);
|
||||
```
|
||||
|
||||
❌ **Unordered collections**
|
||||
```csharp
|
||||
// Bad: Dictionary iteration order is undefined
|
||||
foreach (var (key, value) in dict) { ... }
|
||||
|
||||
// Good: Explicit ordering
|
||||
foreach (var (key, value) in dict.OrderBy(x => x.Key)) { ... }
|
||||
```
|
||||
|
||||
❌ **Floating-point comparisons**
|
||||
```csharp
|
||||
// Bad: Floating-point can differ across platforms
|
||||
var score = 0.1 + 0.2; // Might not equal 0.3 exactly
|
||||
|
||||
// Good: Use fixed-point or integers
|
||||
var scoreInt = (int)((0.1 + 0.2) * 1000);
|
||||
```
|
||||
|
||||
❌ **Non-UTC timestamps**
|
||||
```csharp
|
||||
// Bad: Timezone-dependent
|
||||
var timestamp = DateTime.Now.ToString();
|
||||
|
||||
// Good: Always UTC with 'Z'
|
||||
var timestamp = DateTimeOffset.UtcNow.ToString("o");
|
||||
```
|
||||
|
||||
## Determinism Test Checklist
|
||||
|
||||
When writing determinism tests, verify:
|
||||
|
||||
- [ ] Multiple invocations produce identical output
|
||||
- [ ] JSON has stable property ordering
|
||||
- [ ] File paths are sorted lexicographically
|
||||
- [ ] Timestamps are UTC ISO 8601 with 'Z' suffix
|
||||
- [ ] No random GUIDs (use content-addressing)
|
||||
- [ ] Collections are explicitly ordered
|
||||
- [ ] No system time/random usage (use DeterministicClock/DeterministicRandom)
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- TestKit README: `src/__Libraries/StellaOps.TestKit/README.md`
|
||||
- Testing Strategy: `docs/testing/testing-strategy-models.md`
|
||||
- Test Catalog: `docs/testing/TEST_CATALOG.yml`
|
||||
362
docs/technical/testing/determinism-verification.md
Normal file
362
docs/technical/testing/determinism-verification.md
Normal file
@@ -0,0 +1,362 @@
|
||||
# Determinism Verification Guide
|
||||
|
||||
**Sprint:** 5100.0007.0003 (Epic B)
|
||||
**Last Updated:** 2025-12-23
|
||||
|
||||
## Overview
|
||||
|
||||
StellaOps enforces deterministic artifact generation across all exported formats. This ensures:
|
||||
|
||||
1. **Reproducibility**: Given the same inputs, outputs are byte-for-byte identical
|
||||
2. **Auditability**: Hash verification proves artifact integrity
|
||||
3. **Compliance**: Regulated environments can replay and verify builds
|
||||
4. **CI Gating**: Drift detection prevents unintended changes
|
||||
|
||||
## Supported Artifact Types
|
||||
|
||||
| Type | Format(s) | Test File |
|
||||
|------|-----------|-----------|
|
||||
| SBOM | SPDX 3.0.1, CycloneDX 1.6, CycloneDX 1.7 | `SbomDeterminismTests.cs` |
|
||||
| VEX | OpenVEX, CSAF 2.0 | `VexDeterminismTests.cs` |
|
||||
| Policy Verdicts | JSON | `PolicyDeterminismTests.cs` |
|
||||
| Evidence Bundles | JSON, DSSE, in-toto | `EvidenceBundleDeterminismTests.cs` |
|
||||
| AirGap Bundles | NDJSON | `AirGapBundleDeterminismTests.cs` |
|
||||
| Advisory Normalization | Canonical JSON | `IngestionDeterminismTests.cs` |
|
||||
|
||||
## Determinism Manifest Format
|
||||
|
||||
Every deterministic artifact can produce a manifest describing its content hash and generation context.
|
||||
|
||||
### Schema (v1.0)
|
||||
|
||||
```json
|
||||
{
|
||||
"schemaVersion": "1.0",
|
||||
"artifact": {
|
||||
"type": "sbom | vex | policy-verdict | evidence-bundle | airgap-bundle",
|
||||
"name": "artifact-identifier",
|
||||
"version": "1.0.0",
|
||||
"format": "SPDX 3.0.1 | CycloneDX 1.6 | OpenVEX | CSAF 2.0 | ..."
|
||||
},
|
||||
"canonicalHash": {
|
||||
"algorithm": "SHA-256",
|
||||
"value": "abc123..."
|
||||
},
|
||||
"toolchain": {
|
||||
"platform": ".NET 10.0",
|
||||
"components": [
|
||||
{ "name": "StellaOps.Scanner", "version": "1.0.0" }
|
||||
]
|
||||
},
|
||||
"inputs": {
|
||||
"feedSnapshotHash": "def456...",
|
||||
"policyManifestHash": "ghi789...",
|
||||
"configHash": "jkl012..."
|
||||
},
|
||||
"generatedAt": "2025-12-23T18:00:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
### Field Descriptions
|
||||
|
||||
| Field | Description |
|
||||
|-------|-------------|
|
||||
| `schemaVersion` | Manifest schema version (currently `1.0`) |
|
||||
| `artifact.type` | Category of the artifact |
|
||||
| `artifact.name` | Identifier for the artifact |
|
||||
| `artifact.version` | Version of the artifact (if applicable) |
|
||||
| `artifact.format` | Specific format/spec version |
|
||||
| `canonicalHash.algorithm` | Hash algorithm (always `SHA-256`) |
|
||||
| `canonicalHash.value` | Lowercase hex hash of canonical bytes |
|
||||
| `toolchain.platform` | Runtime platform |
|
||||
| `toolchain.components` | List of generating components with versions |
|
||||
| `inputs` | Hashes of input artifacts (feed snapshots, policies, etc.) |
|
||||
| `generatedAt` | ISO-8601 UTC timestamp of generation |
|
||||
|
||||
## Creating a Determinism Manifest
|
||||
|
||||
Use `DeterminismManifestWriter` from `StellaOps.Testing.Determinism`:
|
||||
|
||||
```csharp
|
||||
using StellaOps.Testing.Determinism;
|
||||
|
||||
// Generate artifact bytes
|
||||
var sbomBytes = GenerateSbom(input, frozenTime);
|
||||
|
||||
// Create artifact info
|
||||
var artifactInfo = new ArtifactInfo
|
||||
{
|
||||
Type = "sbom",
|
||||
Name = "my-container-sbom",
|
||||
Version = "1.0.0",
|
||||
Format = "CycloneDX 1.6"
|
||||
};
|
||||
|
||||
// Create toolchain info
|
||||
var toolchain = new ToolchainInfo
|
||||
{
|
||||
Platform = ".NET 10.0",
|
||||
Components = new[]
|
||||
{
|
||||
new ComponentInfo { Name = "StellaOps.Scanner", Version = "1.0.0" }
|
||||
}
|
||||
};
|
||||
|
||||
// Create manifest
|
||||
var manifest = DeterminismManifestWriter.CreateManifest(
|
||||
sbomBytes,
|
||||
artifactInfo,
|
||||
toolchain);
|
||||
|
||||
// Save manifest
|
||||
DeterminismManifestWriter.Save(manifest, "determinism.json");
|
||||
```
|
||||
|
||||
## Reading and Verifying Manifests
|
||||
|
||||
```csharp
|
||||
// Load manifest
|
||||
var manifest = DeterminismManifestReader.Load("determinism.json");
|
||||
|
||||
// Verify artifact bytes match manifest hash
|
||||
var currentBytes = File.ReadAllBytes("artifact.json");
|
||||
var isValid = DeterminismManifestReader.Verify(manifest, currentBytes);
|
||||
|
||||
if (!isValid)
|
||||
{
|
||||
throw new DeterminismDriftException(
|
||||
$"Artifact hash mismatch. Expected: {manifest.CanonicalHash.Value}");
|
||||
}
|
||||
```
|
||||
|
||||
## Determinism Rules
|
||||
|
||||
### 1. Canonical JSON Serialization
|
||||
|
||||
All JSON output must use canonical serialization via `StellaOps.Canonical.Json`:
|
||||
|
||||
```csharp
|
||||
using StellaOps.Canonical.Json;
|
||||
|
||||
var json = CanonJson.Serialize(myObject);
|
||||
var hash = CanonJson.Sha256Hex(Encoding.UTF8.GetBytes(json));
|
||||
```
|
||||
|
||||
Rules:
|
||||
- Keys sorted lexicographically
|
||||
- No trailing whitespace
|
||||
- Unix line endings (`\n`)
|
||||
- No BOM
|
||||
- UTF-8 encoding
|
||||
|
||||
### 2. Frozen Timestamps
|
||||
|
||||
All timestamps must be provided externally or use `DeterministicTime`:
|
||||
|
||||
```csharp
|
||||
// ❌ BAD - Non-deterministic
|
||||
var timestamp = DateTimeOffset.UtcNow;
|
||||
|
||||
// ✅ GOOD - Deterministic
|
||||
var timestamp = frozenTime; // Passed as parameter
|
||||
```
|
||||
|
||||
### 3. Deterministic IDs
|
||||
|
||||
UUIDs and IDs must be derived from content, not random:
|
||||
|
||||
```csharp
|
||||
// ❌ BAD - Random UUID
|
||||
var id = Guid.NewGuid();
|
||||
|
||||
// ✅ GOOD - Content-derived ID
|
||||
var seed = $"{input.Name}:{input.Version}:{timestamp:O}";
|
||||
var hash = CanonJson.Sha256Hex(Encoding.UTF8.GetBytes(seed));
|
||||
var id = new Guid(Convert.FromHexString(hash[..32]));
|
||||
```
|
||||
|
||||
### 4. Stable Ordering
|
||||
|
||||
Collections must be sorted before serialization:
|
||||
|
||||
```csharp
|
||||
// ❌ BAD - Non-deterministic order
|
||||
var items = dictionary.Values;
|
||||
|
||||
// ✅ GOOD - Sorted order
|
||||
var items = dictionary.Values
|
||||
.OrderBy(v => v.Key, StringComparer.Ordinal);
|
||||
```
|
||||
|
||||
### 5. Parallel Safety
|
||||
|
||||
Determinism must hold under parallel execution:
|
||||
|
||||
```csharp
|
||||
var tasks = Enumerable.Range(0, 20)
|
||||
.Select(_ => Task.Run(() => GenerateArtifact(input, frozenTime)))
|
||||
.ToArray();
|
||||
|
||||
var results = await Task.WhenAll(tasks);
|
||||
results.Should().AllBe(results[0]); // All identical
|
||||
```
|
||||
|
||||
## CI Integration
|
||||
|
||||
### PR Merge Gate
|
||||
|
||||
The determinism gate runs on PR merge:
|
||||
|
||||
```yaml
|
||||
# .gitea/workflows/determinism-gate.yaml
|
||||
name: Determinism Gate
|
||||
on:
|
||||
pull_request:
|
||||
types: [synchronize, ready_for_review]
|
||||
jobs:
|
||||
determinism:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: actions/setup-dotnet@v4
|
||||
with:
|
||||
dotnet-version: '10.0.x'
|
||||
- name: Run Determinism Tests
|
||||
run: |
|
||||
dotnet test tests/integration/StellaOps.Integration.Determinism \
|
||||
--logger "trx;LogFileName=determinism.trx"
|
||||
- name: Generate Determinism Manifest
|
||||
run: |
|
||||
dotnet run --project tools/DeterminismManifestGenerator \
|
||||
--output determinism.json
|
||||
- name: Upload Determinism Artifact
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: determinism-manifest
|
||||
path: determinism.json
|
||||
```
|
||||
|
||||
### Baseline Storage
|
||||
|
||||
Determinism baselines are stored as CI artifacts:
|
||||
|
||||
```
|
||||
ci-artifacts/
|
||||
determinism/
|
||||
baseline/
|
||||
sbom-spdx-3.0.1.json
|
||||
sbom-cyclonedx-1.6.json
|
||||
sbom-cyclonedx-1.7.json
|
||||
vex-openvex.json
|
||||
vex-csaf.json
|
||||
policy-verdict.json
|
||||
evidence-bundle.json
|
||||
airgap-bundle.json
|
||||
```
|
||||
|
||||
### Drift Detection
|
||||
|
||||
When a PR changes artifact output:
|
||||
|
||||
1. CI compares new manifest hash against baseline
|
||||
2. If different, CI fails with diff report
|
||||
3. Developer must either:
|
||||
- Fix the regression (restore determinism)
|
||||
- Update the baseline (if change is intentional)
|
||||
|
||||
### Baseline Update Process
|
||||
|
||||
To intentionally update a baseline:
|
||||
|
||||
```bash
|
||||
# 1. Run determinism tests to generate new manifests
|
||||
dotnet test tests/integration/StellaOps.Integration.Determinism
|
||||
|
||||
# 2. Update baseline files
|
||||
cp determinism/*.json ci-artifacts/determinism/baseline/
|
||||
|
||||
# 3. Commit with explicit message
|
||||
git add ci-artifacts/determinism/baseline/
|
||||
git commit -m "chore(determinism): update baselines for [reason]
|
||||
|
||||
Breaking: [explain what changed]
|
||||
Justification: [explain why this is correct]"
|
||||
```
|
||||
|
||||
## Replay Verification
|
||||
|
||||
To verify an artifact was produced deterministically:
|
||||
|
||||
```bash
|
||||
# 1. Get the manifest
|
||||
curl -O https://releases.stellaops.io/v1.0.0/sbom.determinism.json
|
||||
|
||||
# 2. Get the artifact
|
||||
curl -O https://releases.stellaops.io/v1.0.0/sbom.cdx.json
|
||||
|
||||
# 3. Verify
|
||||
dotnet run --project tools/DeterminismVerifier \
|
||||
--manifest sbom.determinism.json \
|
||||
--artifact sbom.cdx.json
|
||||
```
|
||||
|
||||
Output:
|
||||
```
|
||||
Determinism Verification
|
||||
========================
|
||||
Artifact: sbom.cdx.json
|
||||
Manifest: sbom.determinism.json
|
||||
Expected Hash: abc123...
|
||||
Actual Hash: abc123...
|
||||
Status: ✅ VERIFIED
|
||||
```
|
||||
|
||||
## Test Files Reference
|
||||
|
||||
All determinism tests are in `tests/integration/StellaOps.Integration.Determinism/`:
|
||||
|
||||
| File | Tests | Description |
|
||||
|------|-------|-------------|
|
||||
| `DeterminismValidationTests.cs` | 16 | Manifest format and reader/writer |
|
||||
| `SbomDeterminismTests.cs` | 14 | SPDX 3.0.1, CycloneDX 1.6/1.7 |
|
||||
| `VexDeterminismTests.cs` | 17 | OpenVEX, CSAF 2.0 |
|
||||
| `PolicyDeterminismTests.cs` | 18 | Policy verdict artifacts |
|
||||
| `EvidenceBundleDeterminismTests.cs` | 15 | DSSE, in-toto attestations |
|
||||
| `AirGapBundleDeterminismTests.cs` | 14 | NDJSON bundles, manifests |
|
||||
| `IngestionDeterminismTests.cs` | 17 | NVD/OSV/GHSA/CSAF normalization |
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Hash Mismatch
|
||||
|
||||
If you see a hash mismatch:
|
||||
|
||||
1. **Check timestamps**: Ensure frozen time is used
|
||||
2. **Check ordering**: Ensure all collections are sorted
|
||||
3. **Check IDs**: Ensure IDs are content-derived
|
||||
4. **Check encoding**: Ensure UTF-8 without BOM
|
||||
|
||||
### Flaky Tests
|
||||
|
||||
If determinism tests are flaky:
|
||||
|
||||
1. **Check parallelism**: Ensure no shared mutable state
|
||||
2. **Check time zones**: Use UTC explicitly
|
||||
3. **Check random sources**: Remove all random number generation
|
||||
4. **Check hash inputs**: Ensure all inputs are captured
|
||||
|
||||
### CI Failures
|
||||
|
||||
If CI determinism gate fails:
|
||||
|
||||
1. Compare the diff between expected and actual
|
||||
2. Identify which field changed
|
||||
3. Track back to the code change that caused it
|
||||
4. Either fix the regression or update baseline with justification
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Testing Strategy Models](testing-strategy-models.md) - Overview of testing models
|
||||
- [Canonical JSON Specification](../DATA_SCHEMAS.md#canonical-json) - JSON serialization rules
|
||||
- [CI/CD Workflows](../modules/devops/architecture.md) - CI pipeline details
|
||||
- [Evidence Bundle Schema](../modules/evidence-locker/architecture.md) - Bundle format reference
|
||||
354
docs/technical/testing/e2e-reproducibility.md
Normal file
354
docs/technical/testing/e2e-reproducibility.md
Normal file
@@ -0,0 +1,354 @@
|
||||
# End-to-End Reproducibility Testing Guide
|
||||
|
||||
> **Sprint:** SPRINT_8200_0001_0004_e2e_reproducibility_test
|
||||
> **Tasks:** E2E-8200-025, E2E-8200-026
|
||||
> **Last Updated:** 2025-06-15
|
||||
|
||||
## Overview
|
||||
|
||||
StellaOps implements comprehensive end-to-end (E2E) reproducibility testing to ensure that identical inputs always produce identical outputs across:
|
||||
|
||||
- Sequential pipeline runs
|
||||
- Parallel pipeline runs
|
||||
- Different execution environments (Ubuntu, Windows, macOS)
|
||||
- Different points in time (using frozen timestamps)
|
||||
|
||||
This document describes the E2E test structure, how to run tests, and how to troubleshoot reproducibility failures.
|
||||
|
||||
## Test Architecture
|
||||
|
||||
### Pipeline Stages
|
||||
|
||||
The E2E reproducibility tests cover the full security scanning pipeline:
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────────┐
|
||||
│ Full E2E Pipeline │
|
||||
├─────────────────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ ┌──────────┐ ┌───────────┐ ┌──────┐ ┌────────┐ ┌──────────┐ │
|
||||
│ │ Ingest │───▶│ Normalize │───▶│ Diff │───▶│ Decide │───▶│ Attest │ │
|
||||
│ │ Advisory │ │ Merge & │ │ SBOM │ │ Policy │ │ DSSE │ │
|
||||
│ │ Feeds │ │ Dedup │ │ vs │ │ Verdict│ │ Envelope │ │
|
||||
│ └──────────┘ └───────────┘ │Adviso│ └────────┘ └──────────┘ │
|
||||
│ │ries │ │ │
|
||||
│ └──────┘ ▼ │
|
||||
│ ┌──────────┐ │
|
||||
│ │ Bundle │ │
|
||||
│ │ Package │ │
|
||||
│ └──────────┘ │
|
||||
└─────────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Key Components
|
||||
|
||||
| Component | File | Purpose |
|
||||
|-----------|------|---------|
|
||||
| Test Project | `StellaOps.Integration.E2E.csproj` | MSBuild project for E2E tests |
|
||||
| Test Fixture | `E2EReproducibilityTestFixture.cs` | Pipeline composition and execution |
|
||||
| Tests | `E2EReproducibilityTests.cs` | Reproducibility verification tests |
|
||||
| Comparer | `ManifestComparer.cs` | Byte-for-byte manifest comparison |
|
||||
| CI Workflow | `.gitea/workflows/e2e-reproducibility.yml` | Cross-platform CI pipeline |
|
||||
|
||||
## Running E2E Tests
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- .NET 10.0 SDK
|
||||
- Docker (for PostgreSQL container)
|
||||
- At least 4GB RAM available
|
||||
|
||||
### Local Execution
|
||||
|
||||
```bash
|
||||
# Run all E2E reproducibility tests
|
||||
dotnet test tests/integration/StellaOps.Integration.E2E/ \
|
||||
--logger "console;verbosity=detailed"
|
||||
|
||||
# Run specific test category
|
||||
dotnet test tests/integration/StellaOps.Integration.E2E/ \
|
||||
--filter "Category=Integration" \
|
||||
--logger "console;verbosity=detailed"
|
||||
|
||||
# Run with code coverage
|
||||
dotnet test tests/integration/StellaOps.Integration.E2E/ \
|
||||
--collect:"XPlat Code Coverage" \
|
||||
--results-directory ./TestResults
|
||||
```
|
||||
|
||||
### CI Execution
|
||||
|
||||
E2E tests run automatically on:
|
||||
|
||||
- Pull requests affecting `src/**` or `tests/integration/**`
|
||||
- Pushes to `main` and `develop` branches
|
||||
- Nightly at 2:00 AM UTC (full cross-platform suite)
|
||||
- Manual trigger with optional cross-platform flag
|
||||
|
||||
## Test Categories
|
||||
|
||||
### 1. Sequential Reproducibility (Tasks 11-14)
|
||||
|
||||
Tests that the pipeline produces identical results when run multiple times:
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public async Task FullPipeline_ProducesIdenticalVerdictHash_AcrossRuns()
|
||||
{
|
||||
// Arrange
|
||||
var inputs = await _fixture.SnapshotInputsAsync();
|
||||
|
||||
// Act - Run twice
|
||||
var result1 = await _fixture.RunFullPipelineAsync(inputs);
|
||||
var result2 = await _fixture.RunFullPipelineAsync(inputs);
|
||||
|
||||
// Assert
|
||||
result1.VerdictId.Should().Be(result2.VerdictId);
|
||||
result1.BundleManifestHash.Should().Be(result2.BundleManifestHash);
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Parallel Reproducibility (Task 14)
|
||||
|
||||
Tests that concurrent execution produces identical results:
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public async Task FullPipeline_ParallelExecution_10Concurrent_AllIdentical()
|
||||
{
|
||||
var inputs = await _fixture.SnapshotInputsAsync();
|
||||
const int concurrentRuns = 10;
|
||||
|
||||
var tasks = Enumerable.Range(0, concurrentRuns)
|
||||
.Select(_ => _fixture.RunFullPipelineAsync(inputs));
|
||||
|
||||
var results = await Task.WhenAll(tasks);
|
||||
var comparison = ManifestComparer.CompareMultiple(results.ToList());
|
||||
|
||||
comparison.AllMatch.Should().BeTrue();
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Cross-Platform Reproducibility (Tasks 15-18)
|
||||
|
||||
Tests that identical inputs produce identical outputs on different operating systems:
|
||||
|
||||
| Platform | Runner | Status |
|
||||
|----------|--------|--------|
|
||||
| Ubuntu | `ubuntu-latest` | Primary (runs on every PR) |
|
||||
| Windows | `windows-latest` | Nightly / On-demand |
|
||||
| macOS | `macos-latest` | Nightly / On-demand |
|
||||
|
||||
### 4. Golden Baseline Verification (Tasks 19-21)
|
||||
|
||||
Tests that current results match a pre-approved baseline:
|
||||
|
||||
```json
|
||||
// bench/determinism/golden-baseline/e2e-hashes.json
|
||||
{
|
||||
"verdict_hash": "sha256:abc123...",
|
||||
"manifest_hash": "sha256:def456...",
|
||||
"envelope_hash": "sha256:ghi789...",
|
||||
"updated_at": "2025-06-15T12:00:00Z",
|
||||
"updated_by": "ci",
|
||||
"commit": "abc123def456"
|
||||
}
|
||||
```
|
||||
|
||||
## Troubleshooting Reproducibility Failures
|
||||
|
||||
### Common Causes
|
||||
|
||||
#### 1. Non-Deterministic Ordering
|
||||
|
||||
**Symptom:** Different verdict hashes despite identical inputs.
|
||||
|
||||
**Diagnosis:**
|
||||
```csharp
|
||||
// Check if collections are being ordered
|
||||
var comparison = ManifestComparer.Compare(result1, result2);
|
||||
var report = ManifestComparer.GenerateDiffReport(comparison);
|
||||
Console.WriteLine(report);
|
||||
```
|
||||
|
||||
**Solution:** Ensure all collections are sorted before hashing:
|
||||
```csharp
|
||||
// Bad - non-deterministic
|
||||
var findings = results.ToList();
|
||||
|
||||
// Good - deterministic
|
||||
var findings = results.OrderBy(f => f.CveId, StringComparer.Ordinal)
|
||||
.ThenBy(f => f.Purl, StringComparer.Ordinal)
|
||||
.ToList();
|
||||
```
|
||||
|
||||
#### 2. Timestamp Drift
|
||||
|
||||
**Symptom:** Bundle manifests differ in `createdAt` field.
|
||||
|
||||
**Diagnosis:**
|
||||
```csharp
|
||||
var jsonComparison = ManifestComparer.CompareJson(
|
||||
result1.BundleManifest,
|
||||
result2.BundleManifest);
|
||||
```
|
||||
|
||||
**Solution:** Use frozen timestamps in tests:
|
||||
```csharp
|
||||
// In test fixture
|
||||
public DateTimeOffset FrozenTimestamp { get; } =
|
||||
new DateTimeOffset(2025, 6, 15, 12, 0, 0, TimeSpan.Zero);
|
||||
```
|
||||
|
||||
#### 3. Platform-Specific Behavior
|
||||
|
||||
**Symptom:** Tests pass on Ubuntu but fail on Windows/macOS.
|
||||
|
||||
**Common causes:**
|
||||
- Line ending differences (`\n` vs `\r\n`)
|
||||
- Path separator differences (`/` vs `\`)
|
||||
- Unicode normalization differences
|
||||
- Floating-point representation differences
|
||||
|
||||
**Diagnosis:**
|
||||
```bash
|
||||
# Download artifacts from all platforms
|
||||
# Compare hex dumps
|
||||
xxd ubuntu-manifest.bin > ubuntu.hex
|
||||
xxd windows-manifest.bin > windows.hex
|
||||
diff ubuntu.hex windows.hex
|
||||
```
|
||||
|
||||
**Solution:** Use platform-agnostic serialization:
|
||||
```csharp
|
||||
// Use canonical JSON
|
||||
var json = CanonJson.Serialize(data);
|
||||
|
||||
// Normalize line endings
|
||||
var normalized = content.Replace("\r\n", "\n");
|
||||
```
|
||||
|
||||
#### 4. Key/Signature Differences
|
||||
|
||||
**Symptom:** Envelope hashes differ despite identical payloads.
|
||||
|
||||
**Diagnosis:**
|
||||
```csharp
|
||||
// Compare envelope structure
|
||||
var envelope1 = JsonSerializer.Deserialize<DsseEnvelope>(result1.EnvelopeBytes);
|
||||
var envelope2 = JsonSerializer.Deserialize<DsseEnvelope>(result2.EnvelopeBytes);
|
||||
|
||||
// Check if payloads match
|
||||
envelope1.Payload.SequenceEqual(envelope2.Payload).Should().BeTrue();
|
||||
```
|
||||
|
||||
**Solution:** Use deterministic key generation:
|
||||
```csharp
|
||||
// Generate key from fixed seed for reproducibility
|
||||
private static ECDsa GenerateDeterministicKey(int seed)
|
||||
{
|
||||
var rng = new DeterministicRng(seed);
|
||||
var keyBytes = new byte[32];
|
||||
rng.GetBytes(keyBytes);
|
||||
// ... create key from bytes
|
||||
}
|
||||
```
|
||||
|
||||
### Debugging Tools
|
||||
|
||||
#### ManifestComparer
|
||||
|
||||
```csharp
|
||||
// Full comparison
|
||||
var comparison = ManifestComparer.Compare(expected, actual);
|
||||
|
||||
// Multiple results
|
||||
var multiComparison = ManifestComparer.CompareMultiple(results);
|
||||
|
||||
// Detailed report
|
||||
var report = ManifestComparer.GenerateDiffReport(comparison);
|
||||
|
||||
// Hex dump for byte-level debugging
|
||||
var hexDump = ManifestComparer.GenerateHexDump(expected.BundleManifest, actual.BundleManifest);
|
||||
```
|
||||
|
||||
#### JSON Comparison
|
||||
|
||||
```csharp
|
||||
var jsonComparison = ManifestComparer.CompareJson(
|
||||
expected.BundleManifest,
|
||||
actual.BundleManifest);
|
||||
|
||||
foreach (var diff in jsonComparison.Differences)
|
||||
{
|
||||
Console.WriteLine($"Path: {diff.Path}");
|
||||
Console.WriteLine($"Expected: {diff.Expected}");
|
||||
Console.WriteLine($"Actual: {diff.Actual}");
|
||||
}
|
||||
```
|
||||
|
||||
## Updating the Golden Baseline
|
||||
|
||||
When intentional changes affect reproducibility (e.g., new fields, algorithm changes):
|
||||
|
||||
### 1. Manual Update
|
||||
|
||||
```bash
|
||||
# Run tests and capture new hashes
|
||||
dotnet test tests/integration/StellaOps.Integration.E2E/ \
|
||||
--results-directory ./TestResults
|
||||
|
||||
# Update baseline
|
||||
cp ./TestResults/verdict_hash.txt ./bench/determinism/golden-baseline/
|
||||
# ... update e2e-hashes.json
|
||||
```
|
||||
|
||||
### 2. CI Update (Recommended)
|
||||
|
||||
```bash
|
||||
# Trigger workflow with update flag
|
||||
# Via Gitea UI: Actions → E2E Reproducibility → Run workflow
|
||||
# Set update_baseline = true
|
||||
```
|
||||
|
||||
### 3. Approval Process
|
||||
|
||||
1. Create PR with baseline update
|
||||
2. Explain why the change is intentional
|
||||
3. Verify all platforms produce consistent results
|
||||
4. Get approval from Platform Guild lead
|
||||
5. Merge after CI passes
|
||||
|
||||
## CI Workflow Reference
|
||||
|
||||
### Jobs
|
||||
|
||||
| Job | Runs On | Trigger | Purpose |
|
||||
|-----|---------|---------|---------|
|
||||
| `reproducibility-ubuntu` | Every PR | PR/Push | Primary reproducibility check |
|
||||
| `reproducibility-windows` | Nightly | Schedule/Manual | Cross-platform Windows |
|
||||
| `reproducibility-macos` | Nightly | Schedule/Manual | Cross-platform macOS |
|
||||
| `cross-platform-compare` | After platform jobs | Schedule/Manual | Compare hashes |
|
||||
| `golden-baseline` | After Ubuntu | Always | Baseline verification |
|
||||
| `reproducibility-gate` | After all | Always | Final status check |
|
||||
|
||||
### Artifacts
|
||||
|
||||
| Artifact | Retention | Contents |
|
||||
|----------|-----------|----------|
|
||||
| `e2e-results-{platform}` | 14 days | Test results (.trx), logs |
|
||||
| `hashes-{platform}` | 14 days | Hash files for comparison |
|
||||
| `cross-platform-report` | 30 days | Markdown comparison report |
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Reproducibility Architecture](../reproducibility.md)
|
||||
- [VerdictId Content-Addressing](../modules/policy/architecture.md#verdictid)
|
||||
- [DSSE Envelope Format](../modules/attestor/architecture.md#dsse)
|
||||
- [Determinism Testing](./determinism-verification.md)
|
||||
|
||||
## Sprint History
|
||||
|
||||
- **8200.0001.0004** - Initial E2E reproducibility test implementation
|
||||
- **8200.0001.0001** - VerdictId content-addressing (dependency)
|
||||
- **8200.0001.0002** - DSSE round-trip testing (dependency)
|
||||
80
docs/technical/testing/mutation-testing-baselines.md
Normal file
80
docs/technical/testing/mutation-testing-baselines.md
Normal file
@@ -0,0 +1,80 @@
|
||||
# Mutation Testing Baselines
|
||||
|
||||
> Sprint: SPRINT_0353_0001_0001_mutation_testing_integration
|
||||
> Task: MUT-0353-005
|
||||
|
||||
This document tracks mutation testing baselines for critical modules.
|
||||
|
||||
## Baseline Scores
|
||||
|
||||
| Module | Initial Score | Target Score | Date Established |
|
||||
|--------|--------------|--------------|------------------|
|
||||
| Scanner.Core | 72% | ≥ 80% | 2025-12-16 |
|
||||
| Policy.Engine | 68% | ≥ 80% | 2025-12-16 |
|
||||
| Authority.Core | 75% | ≥ 85% | 2025-12-16 |
|
||||
| Signer.Core | 70% | ≥ 80% | TBD |
|
||||
| Attestor.Core | 65% | ≥ 80% | TBD |
|
||||
| Reachability.Core | 60% | ≥ 75% | TBD |
|
||||
|
||||
## Threshold Configuration
|
||||
|
||||
See `stryker-thresholds.json` for per-module threshold configuration.
|
||||
|
||||
## Mutation Operators Applied
|
||||
|
||||
| Operator | Description | Enabled |
|
||||
|----------|-------------|---------|
|
||||
| Arithmetic | Replace +, -, *, /, % | ✓ |
|
||||
| Boolean | Flip true/false | ✓ |
|
||||
| Comparison | Replace <, >, <=, >=, ==, != | ✓ |
|
||||
| Logical | Replace &&, ||, ! | ✓ |
|
||||
| String | Mutate string literals | ✓ |
|
||||
| Linq | Mutate LINQ methods | ✓ |
|
||||
| NullCoalescing | Mutate ?? operators | ✓ |
|
||||
| Assignment | Mutate assignment operators | ✓ |
|
||||
|
||||
## Exclusions
|
||||
|
||||
The following patterns are excluded from mutation testing:
|
||||
|
||||
- `**/Migrations/**` - Database migrations (tested via integration tests)
|
||||
- `**/Generated/**` - Generated code
|
||||
- `**/*.g.cs` - Source-generated files
|
||||
- `**/Models/**` - Simple data transfer objects
|
||||
- `**/Exceptions/**` - Exception types (tested via integration)
|
||||
|
||||
## Running Mutation Tests
|
||||
|
||||
### Local Execution
|
||||
|
||||
```bash
|
||||
# Run mutation tests for a specific module
|
||||
cd src/Scanner/__Libraries/StellaOps.Scanner.Core
|
||||
dotnet stryker
|
||||
|
||||
# Run with specific configuration
|
||||
dotnet stryker -f stryker-config.json --reporter html
|
||||
|
||||
# Quick mode (fewer mutations, faster feedback)
|
||||
dotnet stryker --since:main
|
||||
```
|
||||
|
||||
### CI Execution
|
||||
|
||||
Mutation tests run on:
|
||||
- Merge requests targeting main
|
||||
- Weekly scheduled runs (comprehensive)
|
||||
|
||||
Results are uploaded as artifacts and published to the mutation testing dashboard.
|
||||
|
||||
## Improving Mutation Score
|
||||
|
||||
1. **Add missing test cases** - Cover edge cases revealed by surviving mutants
|
||||
2. **Strengthen assertions** - Replace weak assertions with specific ones
|
||||
3. **Test boundary conditions** - Cover off-by-one and boundary scenarios
|
||||
4. **Add negative tests** - Test that invalid inputs are rejected
|
||||
|
||||
## References
|
||||
|
||||
- [Stryker.NET Documentation](https://stryker-mutator.io/docs/stryker-net/)
|
||||
- [Mutation Testing Guide](../testing/mutation-testing-guide.md)
|
||||
210
docs/technical/testing/mutation-testing-guide.md
Normal file
210
docs/technical/testing/mutation-testing-guide.md
Normal file
@@ -0,0 +1,210 @@
|
||||
# Mutation Testing Guide
|
||||
|
||||
This guide documents the integration and usage of Stryker.NET mutation testing in StellaOps.
|
||||
|
||||
## Overview
|
||||
|
||||
Mutation testing measures test suite effectiveness by introducing small code changes (mutants) and verifying that tests detect them. Unlike line coverage, mutation testing answers: **"Would my tests catch this bug?"**
|
||||
|
||||
## Installation
|
||||
|
||||
Stryker.NET is configured as a local dotnet tool:
|
||||
|
||||
```bash
|
||||
# Restore tools (includes Stryker.NET)
|
||||
dotnet tool restore
|
||||
|
||||
# Verify installation
|
||||
dotnet stryker --version
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
### Solution-Level Configuration
|
||||
|
||||
Base configuration is at `stryker-config.json` in the solution root. Module-specific configs override these settings.
|
||||
|
||||
### Module Configurations
|
||||
|
||||
| Module | Config Path | Mutation Break Threshold |
|
||||
|--------|-------------|-------------------------|
|
||||
| Scanner.Core | `src/Scanner/__Libraries/StellaOps.Scanner.Core/stryker-config.json` | 60% |
|
||||
| Policy.Engine | `src/Policy/StellaOps.Policy.Engine/stryker-config.json` | 60% |
|
||||
| Authority | `src/Authority/StellaOps.Authority/stryker-config.json` | 65% |
|
||||
|
||||
## Running Mutation Tests
|
||||
|
||||
### Single Module
|
||||
|
||||
```bash
|
||||
# Navigate to module directory
|
||||
cd src/Scanner/__Libraries/StellaOps.Scanner.Core
|
||||
|
||||
# Run mutation testing
|
||||
dotnet stryker
|
||||
|
||||
# With specific config
|
||||
dotnet stryker --config-file stryker-config.json
|
||||
```
|
||||
|
||||
### All Configured Modules
|
||||
|
||||
```bash
|
||||
# From solution root
|
||||
dotnet stryker --solution StellaOps.Router.slnx
|
||||
```
|
||||
|
||||
### CI Mode (Threshold Enforcement)
|
||||
|
||||
```bash
|
||||
# Fails if mutation score below threshold
|
||||
dotnet stryker --break-at-score 60
|
||||
```
|
||||
|
||||
## Understanding Results
|
||||
|
||||
### Mutation Score
|
||||
|
||||
```
|
||||
Mutation Score = (Killed Mutants / Total Mutants) × 100
|
||||
```
|
||||
|
||||
- **Killed**: Test failed when mutant was introduced (good!)
|
||||
- **Survived**: Test passed with mutant present (test gap!)
|
||||
- **No Coverage**: No test covered the mutated code
|
||||
- **Timeout**: Test timed out (usually treated as killed)
|
||||
|
||||
### Thresholds
|
||||
|
||||
| Level | Score | Meaning |
|
||||
|-------|-------|---------|
|
||||
| High | ≥80% | Excellent test effectiveness |
|
||||
| Low | ≥60% | Acceptable, improvements needed |
|
||||
| Break | <50% | Build fails, critical gaps |
|
||||
|
||||
### Example Output
|
||||
|
||||
```
|
||||
All mutants have been tested, and your mutation score has been calculated
|
||||
╔═══════════════════════════════════════════════════════════════════════╗
|
||||
║ Mutation Testing Report ║
|
||||
╠═══════════════════════════════════════════════════════════════════════╣
|
||||
║ Mutants tested: 156 ║
|
||||
║ Mutants killed: 134 ║
|
||||
║ Mutants survived: 18 ║
|
||||
║ Mutants no coverage: 4 ║
|
||||
║ Mutation score: 85.90% ║
|
||||
╚═══════════════════════════════════════════════════════════════════════╝
|
||||
```
|
||||
|
||||
## Common Mutators
|
||||
|
||||
| Mutator | Original | Mutant |
|
||||
|---------|----------|--------|
|
||||
| Comparison | `>=` | `>` |
|
||||
| Equality | `==` | `!=` |
|
||||
| Boolean | `true` | `false` |
|
||||
| Logical | `&&` | `\|\|` |
|
||||
| Arithmetic | `+` | `-` |
|
||||
| NullCoalescing | `??` | ` ` (remove) |
|
||||
|
||||
## Fixing Survived Mutants
|
||||
|
||||
### 1. Analyze the Report
|
||||
|
||||
Open the HTML report in `.stryker/output/<module>/mutation-report.html`.
|
||||
|
||||
### 2. Identify the Gap
|
||||
|
||||
Look at the survived mutant:
|
||||
|
||||
```csharp
|
||||
// Original
|
||||
if (score >= threshold) { return "PASS"; }
|
||||
|
||||
// Mutant (survived!)
|
||||
if (score > threshold) { return "PASS"; }
|
||||
```
|
||||
|
||||
### 3. Add Missing Test
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public void Should_Pass_When_Score_Equals_Threshold()
|
||||
{
|
||||
var score = 60;
|
||||
var threshold = 60;
|
||||
|
||||
var result = EvaluateScore(score, threshold);
|
||||
|
||||
result.Should().Be("PASS"); // Now kills the >= to > mutant
|
||||
}
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
### 1. Focus on Critical Modules First
|
||||
|
||||
Prioritize mutation testing for:
|
||||
- Security-critical code (Authority, Signer)
|
||||
- Business logic (Policy decisions, Scanner matching)
|
||||
- Boundary conditions
|
||||
|
||||
### 2. Don't Chase 100%
|
||||
|
||||
Some mutants are false positives or equivalent mutants. Aim for 80%+ on critical modules.
|
||||
|
||||
### 3. Use Baseline Mode
|
||||
|
||||
Enable baseline to only test changed files:
|
||||
|
||||
```bash
|
||||
dotnet stryker --with-baseline:main
|
||||
```
|
||||
|
||||
### 4. Exclude Non-Critical Code
|
||||
|
||||
Exclude from mutation testing:
|
||||
- DTOs and models
|
||||
- Generated code
|
||||
- Migrations
|
||||
- UI components
|
||||
|
||||
## CI Integration
|
||||
|
||||
Mutation testing runs in CI:
|
||||
|
||||
```yaml
|
||||
mutation-test:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- name: Run Stryker
|
||||
run: |
|
||||
dotnet tool restore
|
||||
dotnet stryker --break-at-score 60
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Slow Execution
|
||||
|
||||
- Use `--concurrency` to control parallelism
|
||||
- Enable `coverage-analysis: perTest` for smarter mutant selection
|
||||
- Use `--since:main` to only test changed code
|
||||
|
||||
### Out of Memory
|
||||
|
||||
- Reduce `--concurrency` value
|
||||
- Exclude large test projects
|
||||
|
||||
### Timeout Issues
|
||||
|
||||
- Adjust `--timeout` setting
|
||||
- Some infinite loop mutants may timeout (this is expected)
|
||||
|
||||
## References
|
||||
|
||||
- [Stryker.NET Documentation](https://stryker-mutator.io/docs/stryker-net/introduction/)
|
||||
- [Mutation Testing Theory](https://en.wikipedia.org/wiki/Mutation_testing)
|
||||
- StellaOps Test Suite Overview: `docs/19_TEST_SUITE_OVERVIEW.md`
|
||||
202
docs/technical/testing/schema-validation.md
Normal file
202
docs/technical/testing/schema-validation.md
Normal file
@@ -0,0 +1,202 @@
|
||||
# SBOM Schema Validation
|
||||
|
||||
This document describes the schema validation system for SBOM (Software Bill of Materials) fixtures in StellaOps.
|
||||
|
||||
## Overview
|
||||
|
||||
StellaOps validates all SBOM fixtures against official JSON schemas to detect schema drift before runtime. This ensures:
|
||||
|
||||
- CycloneDX 1.6 fixtures are compliant with the official schema
|
||||
- SPDX 3.0.1 fixtures meet specification requirements
|
||||
- OpenVEX fixtures follow the 0.2.0 specification
|
||||
- Invalid fixtures are detected early in the CI pipeline
|
||||
|
||||
## Supported Formats
|
||||
|
||||
| Format | Version | Schema Location | Validator |
|
||||
|--------|---------|-----------------|-----------|
|
||||
| CycloneDX | 1.6 | `docs/schemas/cyclonedx-bom-1.6.schema.json` | sbom-utility |
|
||||
| SPDX | 3.0.1 | `docs/schemas/spdx-jsonld-3.0.1.schema.json` | pyspdxtools / check-jsonschema |
|
||||
| OpenVEX | 0.2.0 | `docs/schemas/openvex-0.2.0.schema.json` | ajv-cli |
|
||||
|
||||
## CI Workflows
|
||||
|
||||
### Schema Validation Workflow
|
||||
|
||||
**File:** `.gitea/workflows/schema-validation.yml`
|
||||
|
||||
Runs on:
|
||||
- Pull requests touching `bench/golden-corpus/**`, `src/Scanner/**`, `docs/schemas/**`, or `scripts/validate-*.sh`
|
||||
- Push to `main` branch
|
||||
|
||||
Jobs:
|
||||
1. **validate-cyclonedx** - Validates all CycloneDX 1.6 fixtures
|
||||
2. **validate-spdx** - Validates all SPDX 3.0.1 fixtures
|
||||
3. **validate-vex** - Validates all OpenVEX 0.2.0 fixtures
|
||||
4. **validate-negative** - Verifies invalid fixtures are correctly rejected
|
||||
5. **summary** - Aggregates results
|
||||
|
||||
### Determinism Gate Integration
|
||||
|
||||
**File:** `.gitea/workflows/determinism-gate.yml`
|
||||
|
||||
The determinism gate includes schema validation as a prerequisite step. If schema validation fails, determinism checks are blocked.
|
||||
|
||||
To skip schema validation (e.g., during debugging):
|
||||
```bash
|
||||
# Via workflow_dispatch
|
||||
skip_schema_validation: true
|
||||
```
|
||||
|
||||
## Fixture Directories
|
||||
|
||||
Validation scans these directories for SBOM fixtures:
|
||||
|
||||
| Directory | Purpose |
|
||||
|-----------|---------|
|
||||
| `src/__Tests/__Benchmarks/golden-corpus/` | Golden reference fixtures for reproducibility testing |
|
||||
| `src/__Tests/fixtures/` | Test fixtures for unit and integration tests |
|
||||
| `src/__Tests/__Datasets/seed-data/` | Initial seed data for development environments |
|
||||
| `src/__Tests/fixtures/invalid/` | **Excluded** - Contains intentionally invalid fixtures for negative testing |
|
||||
|
||||
## Local Validation
|
||||
|
||||
### Using the Validation Scripts
|
||||
|
||||
```bash
|
||||
# Validate a single CycloneDX file
|
||||
./scripts/validate-sbom.sh path/to/sbom.json
|
||||
|
||||
# Validate all CycloneDX files in a directory
|
||||
./scripts/validate-sbom.sh --all path/to/directory
|
||||
|
||||
# Validate SPDX file
|
||||
./scripts/validate-spdx.sh path/to/sbom.spdx.json
|
||||
|
||||
# Validate OpenVEX file
|
||||
./scripts/validate-vex.sh path/to/vex.openvex.json
|
||||
```
|
||||
|
||||
### Using sbom-utility Directly
|
||||
|
||||
```bash
|
||||
# Install sbom-utility
|
||||
curl -sSfL "https://github.com/CycloneDX/sbom-utility/releases/download/v0.16.0/sbom-utility-v0.16.0-linux-amd64.tar.gz" | tar xz
|
||||
sudo mv sbom-utility /usr/local/bin/
|
||||
|
||||
# Validate
|
||||
sbom-utility validate --input-file sbom.json --schema docs/schemas/cyclonedx-bom-1.6.schema.json
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Validation Errors
|
||||
|
||||
#### 1. Invalid specVersion
|
||||
|
||||
**Error:** `enum: must be equal to one of the allowed values`
|
||||
|
||||
**Cause:** The `specVersion` field contains an invalid or unsupported version.
|
||||
|
||||
**Solution:**
|
||||
```json
|
||||
// Invalid
|
||||
"specVersion": "2.0"
|
||||
|
||||
// Valid
|
||||
"specVersion": "1.6"
|
||||
```
|
||||
|
||||
#### 2. Missing Required Fields
|
||||
|
||||
**Error:** `required: must have required property 'name'`
|
||||
|
||||
**Cause:** A component is missing required fields.
|
||||
|
||||
**Solution:** Ensure all components have required fields:
|
||||
```json
|
||||
{
|
||||
"type": "library",
|
||||
"name": "example-package",
|
||||
"version": "1.0.0"
|
||||
}
|
||||
```
|
||||
|
||||
#### 3. Invalid Component Type
|
||||
|
||||
**Error:** `enum: type must be equal to one of the allowed values`
|
||||
|
||||
**Cause:** The component type is not a valid CycloneDX type.
|
||||
|
||||
**Solution:** Use valid types: `application`, `framework`, `library`, `container`, `operating-system`, `device`, `firmware`, `file`, `data`
|
||||
|
||||
#### 4. Invalid PURL Format
|
||||
|
||||
**Error:** `format: must match format "purl"`
|
||||
|
||||
**Cause:** The package URL (purl) is malformed.
|
||||
|
||||
**Solution:** Use correct purl format:
|
||||
```json
|
||||
// Invalid
|
||||
"purl": "npm:example@1.0.0"
|
||||
|
||||
// Valid
|
||||
"purl": "pkg:npm/example@1.0.0"
|
||||
```
|
||||
|
||||
### CI Failure Recovery
|
||||
|
||||
1. **Identify the failing fixture:** Check CI logs for the specific file
|
||||
2. **Download the fixture:** `cat path/to/failing-fixture.json`
|
||||
3. **Run local validation:** `./scripts/validate-sbom.sh path/to/failing-fixture.json`
|
||||
4. **Fix the schema issues:** Use the error messages to guide corrections
|
||||
5. **Verify the fix:** Re-run local validation
|
||||
6. **Push and verify CI passes**
|
||||
|
||||
### Negative Test Failures
|
||||
|
||||
If negative tests fail with "UNEXPECTED PASS":
|
||||
|
||||
1. The invalid fixture in `tests/fixtures/invalid/` somehow passed validation
|
||||
2. Review the fixture to ensure it contains actual schema violations
|
||||
3. Update the fixture to include more obvious violations
|
||||
4. Document the expected error in `tests/fixtures/invalid/README.md`
|
||||
|
||||
## Adding New Fixtures
|
||||
|
||||
### Valid Fixtures
|
||||
|
||||
1. Create fixture in appropriate directory (`bench/golden-corpus/`, `tests/fixtures/`)
|
||||
2. Ensure it contains the format marker:
|
||||
- CycloneDX: `"bomFormat": "CycloneDX"`
|
||||
- SPDX: `"spdxVersion"` or `"@context"` with SPDX
|
||||
- OpenVEX: `"@context"` with openvex
|
||||
3. Run local validation before committing
|
||||
4. CI will automatically validate on PR
|
||||
|
||||
### Invalid Fixtures (Negative Testing)
|
||||
|
||||
1. Create fixture in `tests/fixtures/invalid/`
|
||||
2. Add `$comment` field explaining the defect
|
||||
3. Update `tests/fixtures/invalid/README.md` with expected error
|
||||
4. Ensure the fixture has the correct format marker
|
||||
5. CI will verify it fails validation
|
||||
|
||||
## Schema Updates
|
||||
|
||||
When updating schema versions:
|
||||
|
||||
1. Download new schema to `docs/schemas/`
|
||||
2. Update `SBOM_UTILITY_VERSION` in workflows if needed
|
||||
3. Run full validation to check for new violations
|
||||
4. Update documentation with new version
|
||||
5. Update `docs/reproducibility.md` with schema version changes
|
||||
|
||||
## References
|
||||
|
||||
- [CycloneDX Specification](https://cyclonedx.org/specification/overview/)
|
||||
- [CycloneDX sbom-utility](https://github.com/CycloneDX/sbom-utility)
|
||||
- [SPDX Specification](https://spdx.github.io/spdx-spec/v3.0.1/)
|
||||
- [SPDX Python Tools](https://github.com/spdx/tools-python)
|
||||
- [OpenVEX Specification](https://github.com/openvex/spec)
|
||||
229
docs/technical/testing/security-testing-guide.md
Normal file
229
docs/technical/testing/security-testing-guide.md
Normal file
@@ -0,0 +1,229 @@
|
||||
# Security Testing Guide
|
||||
|
||||
> Sprint: SPRINT_0352_0001_0001_security_testing_framework
|
||||
> Task: SEC-0352-010
|
||||
|
||||
This guide describes the security testing framework used in StellaOps, aligned with OWASP Top 10 categories.
|
||||
|
||||
## Overview
|
||||
|
||||
The security testing framework provides automated tests for common security vulnerabilities organized by OWASP category:
|
||||
|
||||
| OWASP Category | Directory | Status |
|
||||
|----------------|-----------|--------|
|
||||
| A01: Broken Access Control | `A01_BrokenAccessControl/` | ✓ Implemented |
|
||||
| A02: Cryptographic Failures | `A02_CryptographicFailures/` | ✓ Implemented |
|
||||
| A03: Injection | `A03_Injection/` | ✓ Implemented |
|
||||
| A05: Security Misconfiguration | `A05_SecurityMisconfiguration/` | ✓ Implemented |
|
||||
| A07: Authentication Failures | `A07_AuthenticationFailures/` | ✓ Implemented |
|
||||
| A08: Software/Data Integrity | `A08_SoftwareDataIntegrity/` | ✓ Implemented |
|
||||
| A10: SSRF | `A10_SSRF/` | ✓ Implemented |
|
||||
|
||||
## Directory Structure
|
||||
|
||||
```
|
||||
tests/
|
||||
└── security/
|
||||
├── README.md
|
||||
└── StellaOps.Security.Tests/
|
||||
├── Infrastructure/
|
||||
│ ├── SecurityTestBase.cs
|
||||
│ ├── MaliciousPayloads.cs
|
||||
│ └── SecurityAssertions.cs
|
||||
├── A01_BrokenAccessControl/
|
||||
├── A02_CryptographicFailures/
|
||||
├── A03_Injection/
|
||||
├── A05_SecurityMisconfiguration/
|
||||
├── A07_AuthenticationFailures/
|
||||
├── A08_SoftwareDataIntegrity/
|
||||
└── A10_SSRF/
|
||||
```
|
||||
|
||||
## Running Security Tests
|
||||
|
||||
### Local Execution
|
||||
|
||||
```bash
|
||||
# Run all security tests
|
||||
cd tests/security/StellaOps.Security.Tests
|
||||
dotnet test --filter "Category=Security"
|
||||
|
||||
# Run specific OWASP category
|
||||
dotnet test --filter "OWASP=A01"
|
||||
|
||||
# Run with detailed output
|
||||
dotnet test --filter "Category=Security" --verbosity detailed
|
||||
```
|
||||
|
||||
### CI Integration
|
||||
|
||||
Security tests run automatically on:
|
||||
- All pull requests to `main` or `develop`
|
||||
- Scheduled nightly builds
|
||||
|
||||
Results are uploaded as artifacts and any failures block the PR.
|
||||
|
||||
## Test Categories
|
||||
|
||||
### A01: Broken Access Control
|
||||
|
||||
Tests for authorization bypass vulnerabilities:
|
||||
- Tenant isolation violations
|
||||
- RBAC enforcement
|
||||
- Privilege escalation
|
||||
- IDOR (Insecure Direct Object References)
|
||||
|
||||
### A02: Cryptographic Failures
|
||||
|
||||
Tests for cryptographic weaknesses:
|
||||
- Key material exposure in logs
|
||||
- Weak algorithm usage
|
||||
- TLS configuration
|
||||
- Secure random generation
|
||||
|
||||
### A03: Injection
|
||||
|
||||
Tests for injection vulnerabilities:
|
||||
- SQL injection (parameterization)
|
||||
- Command injection
|
||||
- ORM injection
|
||||
- Path traversal
|
||||
|
||||
### A05: Security Misconfiguration
|
||||
|
||||
Tests for configuration errors:
|
||||
- Debug mode in production
|
||||
- Error detail leakage
|
||||
- Security headers
|
||||
- CORS configuration
|
||||
|
||||
### A07: Authentication Failures
|
||||
|
||||
Tests for authentication weaknesses:
|
||||
- Brute force protection
|
||||
- Weak password acceptance
|
||||
- Session management
|
||||
- Account lockout
|
||||
|
||||
### A08: Software/Data Integrity
|
||||
|
||||
Tests for integrity verification:
|
||||
- Artifact signature verification
|
||||
- SBOM integrity
|
||||
- Attestation chain validation
|
||||
- DSSE envelope validation
|
||||
|
||||
### A10: SSRF
|
||||
|
||||
Tests for server-side request forgery:
|
||||
- Internal network access
|
||||
- Cloud metadata endpoint blocking
|
||||
- URL validation
|
||||
|
||||
## Writing Security Tests
|
||||
|
||||
### Base Class
|
||||
|
||||
All security tests should extend `SecurityTestBase`:
|
||||
|
||||
```csharp
|
||||
using StellaOps.Security.Tests.Infrastructure;
|
||||
|
||||
[Trait("Category", "Security")]
|
||||
[Trait("OWASP", "A01")]
|
||||
public sealed class MySecurityTests : SecurityTestBase
|
||||
{
|
||||
[Fact(DisplayName = "A01-XXX: Descriptive test name")]
|
||||
public void TestMethod()
|
||||
{
|
||||
// Arrange, Act, Assert
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Naming Convention
|
||||
|
||||
- Test display names: `A{category}-{number}: {description}`
|
||||
- Example: `A01-001: Admin endpoints should require authentication`
|
||||
|
||||
### Test Traits
|
||||
|
||||
Always include these traits:
|
||||
- `Category = Security`
|
||||
- `OWASP = A{category}`
|
||||
|
||||
## Security Test Guidelines
|
||||
|
||||
1. **Test both positive and negative cases** - Verify both allowed and denied actions
|
||||
2. **Use realistic payloads** - Include common attack patterns from `MaliciousPayloads.cs`
|
||||
3. **Don't rely on security by obscurity** - Assume attackers know the system
|
||||
4. **Test boundaries** - Check edge cases and boundary conditions
|
||||
5. **Document expected behavior** - Use descriptive test names and assertions
|
||||
|
||||
## Malicious Payloads
|
||||
|
||||
The `MaliciousPayloads.cs` file contains common attack patterns:
|
||||
|
||||
```csharp
|
||||
public static class MaliciousPayloads
|
||||
{
|
||||
public static readonly string[] SqlInjection = new[]
|
||||
{
|
||||
"' OR '1'='1",
|
||||
"1; DROP TABLE users--",
|
||||
"admin'--"
|
||||
};
|
||||
|
||||
public static readonly string[] CommandInjection = new[]
|
||||
{
|
||||
"; rm -rf /",
|
||||
"| cat /etc/passwd",
|
||||
"$(whoami)"
|
||||
};
|
||||
|
||||
public static readonly string[] PathTraversal = new[]
|
||||
{
|
||||
"../../../etc/passwd",
|
||||
"..\\..\\..\\windows\\system32\\config\\sam"
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
## CI Integration
|
||||
|
||||
### Workflow Configuration
|
||||
|
||||
The security test job runs after build-test completes:
|
||||
|
||||
```yaml
|
||||
security-testing:
|
||||
runs-on: ubuntu-22.04
|
||||
needs: build-test
|
||||
steps:
|
||||
- name: Run OWASP security tests
|
||||
run: |
|
||||
dotnet test tests/security/StellaOps.Security.Tests \
|
||||
--filter "Category=Security" \
|
||||
--logger "trx;LogFileName=security-tests.trx"
|
||||
```
|
||||
|
||||
### Failure Handling
|
||||
|
||||
Security test failures:
|
||||
- Block PR merge
|
||||
- Generate detailed report
|
||||
- Notify security team via webhook
|
||||
|
||||
## Reporting
|
||||
|
||||
Security test results are:
|
||||
- Uploaded as CI artifacts
|
||||
- Included in quality gate summary
|
||||
- Tracked for trend analysis
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [OWASP Top 10](https://owasp.org/Top10/)
|
||||
- [OWASP Testing Guide](https://owasp.org/www-project-web-security-testing-guide/)
|
||||
- [Mutation Testing Guide](./mutation-testing-guide.md)
|
||||
- [CI Quality Gates](./ci-quality-gates.md)
|
||||
@@ -0,0 +1,344 @@
|
||||
# Testing Quality Guardrails Implementation
|
||||
|
||||
## Overview
|
||||
|
||||
This document provides the master implementation plan for the Testing Quality Guardrails system derived from the `14-Dec-2025 - Testing and Quality Guardrails Technical Reference.md` product advisory.
|
||||
|
||||
**Source Advisory:** `docs/product-advisories/14-Dec-2025 - Testing and Quality Guardrails Technical Reference.md`
|
||||
|
||||
**Implementation Status:** Planning Complete, Execution Pending
|
||||
|
||||
---
|
||||
|
||||
## Executive Summary
|
||||
|
||||
The Testing Quality Guardrails implementation addresses gaps between the product advisory and the current StellaOps codebase. After analysis, we identified 6 high-value items to implement across 4 focused sprints.
|
||||
|
||||
### What We're Implementing
|
||||
|
||||
| Item | Sprint | Value | Effort |
|
||||
|------|--------|-------|--------|
|
||||
| Reachability quality gates in CI | 0350 | HIGH | LOW |
|
||||
| TTFS regression tracking | 0350 | HIGH | LOW |
|
||||
| Performance SLO enforcement | 0350 | HIGH | LOW |
|
||||
| SCA Failure Catalogue (FC6-FC10) | 0351 | HIGH | MEDIUM |
|
||||
| Security testing (OWASP Top 10) | 0352 | HIGH | MEDIUM |
|
||||
| Mutation testing (Stryker.NET) | 0353 | MEDIUM | MEDIUM |
|
||||
|
||||
### What We're NOT Implementing (and Why)
|
||||
|
||||
| Item | Reason |
|
||||
|------|--------|
|
||||
| `toys/svc-XX/` directory restructure | Already have equivalent in `tests/reachability/corpus/` |
|
||||
| `labels.yaml` per-service format | Already have `reachgraph.truth.json` with same semantics |
|
||||
| Canonical TAG format | Can adopt incrementally, not blocking |
|
||||
| Fix validation 100% pass rate | Too rigid; changed to 90% for fixable cases |
|
||||
| Automated reviewer rejection | Over-engineering; human judgment needed |
|
||||
|
||||
---
|
||||
|
||||
## Sprint Roadmap
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ TESTING QUALITY GUARDRAILS │
|
||||
├─────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ Sprint 0350 Sprint 0351 │
|
||||
│ CI Quality Gates SCA Failure Catalogue │
|
||||
│ ┌─────────────┐ ┌─────────────┐ │
|
||||
│ │ Reachability│ │ FC6: Java │ │
|
||||
│ │ TTFS │ │ FC7: .NET │ │
|
||||
│ │ Performance │ │ FC8: Docker │ │
|
||||
│ └─────────────┘ │ FC9: PURL │ │
|
||||
│ │ │ FC10: CVE │ │
|
||||
│ │ └─────────────┘ │
|
||||
│ │ │ │
|
||||
│ └──────────┬───────────────┘ │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ Sprint 0352 Sprint 0353 │
|
||||
│ Security Testing Mutation Testing │
|
||||
│ ┌─────────────┐ ┌─────────────┐ │
|
||||
│ │ OWASP Top 10│ │ Stryker.NET │ │
|
||||
│ │ A01-A10 │ │ Scanner │ │
|
||||
│ │ 50+ tests │ │ Policy │ │
|
||||
│ └─────────────┘ │ Authority │ │
|
||||
│ └─────────────┘ │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Execution Order
|
||||
|
||||
1. **Sprint 0350** and **Sprint 0351** can run in parallel (no dependencies)
|
||||
2. **Sprint 0352** can run in parallel with 0350/0351
|
||||
3. **Sprint 0353** should start after 0352 (security tests should be stable first)
|
||||
|
||||
### Estimated Duration
|
||||
|
||||
| Sprint | Tasks | Estimated Effort |
|
||||
|--------|-------|------------------|
|
||||
| 0350 | 10 | 2-3 developer-days |
|
||||
| 0351 | 10 | 3-4 developer-days |
|
||||
| 0352 | 10 | 4-5 developer-days |
|
||||
| 0353 | 10 | 3-4 developer-days |
|
||||
| **Total** | **40** | **12-16 developer-days** |
|
||||
|
||||
---
|
||||
|
||||
## Sprint Details
|
||||
|
||||
### Sprint 0350: CI Quality Gates Foundation
|
||||
|
||||
**File:** `docs/implplan/SPRINT_0350_0001_0001_ci_quality_gates_foundation.md`
|
||||
|
||||
**Objective:** Connect existing test infrastructure to CI enforcement
|
||||
|
||||
**Key Deliverables:**
|
||||
- `scripts/ci/compute-reachability-metrics.sh` - Compute recall/precision from corpus
|
||||
- `scripts/ci/reachability-thresholds.yaml` - Enforcement thresholds
|
||||
- `scripts/ci/compute-ttfs-metrics.sh` - TTFS extraction from test runs
|
||||
- `bench/baselines/ttfs-baseline.json` - TTFS targets
|
||||
- `scripts/ci/enforce-performance-slos.sh` - Performance SLO checks
|
||||
- CI workflow modifications for quality gates
|
||||
|
||||
**Quality Thresholds:**
|
||||
```yaml
|
||||
thresholds:
|
||||
runtime_dependency_recall: >= 0.95
|
||||
unreachable_false_positives: <= 0.05
|
||||
reachability_underreport: <= 0.10
|
||||
ttfs_regression: <= +10% vs main
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Sprint 0351: SCA Failure Catalogue Completion
|
||||
|
||||
**File:** `docs/implplan/SPRINT_0351_0001_0001_sca_failure_catalogue_completion.md`
|
||||
|
||||
**Objective:** Complete FC6-FC10 test cases for scanner regression testing
|
||||
|
||||
**New Failure Cases:**
|
||||
| ID | Name | Failure Mode |
|
||||
|----|------|--------------|
|
||||
| FC6 | Java Shadow JAR | Shaded dependencies not detected |
|
||||
| FC7 | .NET Transitive Pinning | CPM pins to vulnerable version |
|
||||
| FC8 | Docker Multi-Stage Leakage | Build-time deps in runtime analysis |
|
||||
| FC9 | PURL Namespace Collision | npm vs pypi same package name |
|
||||
| FC10 | CVE Split/Merge | Single vuln with multiple CVE IDs |
|
||||
|
||||
**Key Deliverables:**
|
||||
- 5 new fixture directories under `tests/fixtures/sca/catalogue/`
|
||||
- DSSE manifests for integrity verification
|
||||
- xUnit test project for failure catalogue
|
||||
- Updated documentation
|
||||
|
||||
---
|
||||
|
||||
### Sprint 0352: Security Testing Framework
|
||||
|
||||
**File:** `docs/implplan/SPRINT_0352_0001_0001_security_testing_framework.md`
|
||||
|
||||
**Objective:** Systematic OWASP Top 10 coverage for StellaOps
|
||||
|
||||
**Coverage Matrix:**
|
||||
| OWASP | Category | Test Count |
|
||||
|-------|----------|------------|
|
||||
| A01 | Broken Access Control | 8+ |
|
||||
| A02 | Cryptographic Failures | 6+ |
|
||||
| A03 | Injection | 10+ |
|
||||
| A05 | Security Misconfiguration | 6+ |
|
||||
| A07 | Authentication Failures | 8+ |
|
||||
| A08 | Integrity Failures | 5+ |
|
||||
| A10 | SSRF | 8+ |
|
||||
| **Total** | **7 categories** | **50+ tests** |
|
||||
|
||||
**Key Deliverables:**
|
||||
- `tests/security/StellaOps.Security.Tests/` - Security test project
|
||||
- `MaliciousPayloads.cs` - Common attack patterns
|
||||
- `SecurityTestBase.cs` - Test infrastructure
|
||||
- `.gitea/workflows/security-tests.yml` - Dedicated CI workflow
|
||||
- `docs/testing/security-testing-guide.md` - Documentation
|
||||
|
||||
---
|
||||
|
||||
### Sprint 0353: Mutation Testing Integration
|
||||
|
||||
**File:** `docs/implplan/SPRINT_0353_0001_0001_mutation_testing_integration.md`
|
||||
|
||||
**Objective:** Measure test suite effectiveness with Stryker.NET
|
||||
|
||||
**Target Modules:**
|
||||
| Module | Threshold (Break) | Threshold (High) |
|
||||
|--------|-------------------|------------------|
|
||||
| Scanner.Core | 60% | 85% |
|
||||
| Policy.Engine | 60% | 85% |
|
||||
| Authority.Core | 65% | 90% |
|
||||
|
||||
**Key Deliverables:**
|
||||
- Stryker.NET configuration for each target module
|
||||
- `scripts/ci/mutation-thresholds.yaml` - Threshold configuration
|
||||
- `.gitea/workflows/mutation-testing.yml` - Weekly mutation runs
|
||||
- `bench/baselines/mutation-baselines.json` - Baseline scores
|
||||
- `docs/testing/mutation-testing-guide.md` - Developer guide
|
||||
|
||||
---
|
||||
|
||||
## Existing Infrastructure Mapping
|
||||
|
||||
The advisory describes structures that already exist under different names:
|
||||
|
||||
| Advisory Structure | Existing Equivalent | Notes |
|
||||
|-------------------|---------------------|-------|
|
||||
| `toys/svc-XX/` | `tests/reachability/corpus/` | Same purpose, different path |
|
||||
| `labels.yaml` | `reachgraph.truth.json` | Different schema, same semantics |
|
||||
| `evidence/trace.json` | Evidence in attestor module | Already implemented |
|
||||
| `PostgresFixture` | `PostgresIntegrationFixture` | Already implemented |
|
||||
| `FakeTimeProvider` | Authority tests, ConnectorTestHarness | Already used |
|
||||
| `inputs.lock` | Exists in acceptance/guardrails | Already implemented |
|
||||
|
||||
---
|
||||
|
||||
## Quality Gate Summary
|
||||
|
||||
After implementation, CI will enforce:
|
||||
|
||||
### Reachability Gates
|
||||
- Runtime dependency recall ≥ 95%
|
||||
- Unreachable false positives ≤ 5%
|
||||
- Reachability underreport ≤ 10%
|
||||
|
||||
### Performance Gates
|
||||
- Medium service scan < 2 minutes
|
||||
- Reachability compute < 30 seconds
|
||||
- SBOM ingestion < 5 seconds
|
||||
|
||||
### TTFS Gates
|
||||
- p50 < 2 seconds
|
||||
- p95 < 5 seconds
|
||||
- Regression ≤ +10% vs main
|
||||
|
||||
### Coverage Gates
|
||||
- Line coverage ≥ 70% (existing)
|
||||
- Branch coverage ≥ 60% (existing)
|
||||
- Mutation score ≥ 60-65% (break threshold)
|
||||
|
||||
### Security Gates
|
||||
- All security tests pass
|
||||
- No OWASP Top 10 violations
|
||||
|
||||
---
|
||||
|
||||
## File Changes Summary
|
||||
|
||||
### New Files
|
||||
|
||||
```
|
||||
scripts/ci/
|
||||
├── compute-reachability-metrics.sh
|
||||
├── compute-ttfs-metrics.sh
|
||||
├── enforce-performance-slos.sh
|
||||
├── enforce-thresholds.sh
|
||||
├── enforce-mutation-thresholds.sh
|
||||
├── extract-mutation-score.sh
|
||||
├── reachability-thresholds.yaml
|
||||
└── mutation-thresholds.yaml
|
||||
|
||||
bench/baselines/
|
||||
├── ttfs-baseline.json
|
||||
└── mutation-baselines.json
|
||||
|
||||
tests/
|
||||
├── fixtures/sca/catalogue/
|
||||
│ ├── fc6-java-shadow-jar/
|
||||
│ ├── fc7-dotnet-transitive-pinning/
|
||||
│ ├── fc8-docker-multistage-leakage/
|
||||
│ ├── fc9-purl-namespace-collision/
|
||||
│ └── fc10-cve-split-merge/
|
||||
└── security/
|
||||
└── StellaOps.Security.Tests/
|
||||
|
||||
.gitea/workflows/
|
||||
├── security-tests.yml
|
||||
└── mutation-testing.yml
|
||||
|
||||
.config/
|
||||
└── dotnet-tools.json (stryker)
|
||||
|
||||
stryker-config.json (root)
|
||||
|
||||
src/Scanner/__Libraries/StellaOps.Scanner.Core/stryker-config.json
|
||||
src/Policy/StellaOps.Policy.Engine/stryker-config.json
|
||||
src/Authority/StellaOps.Authority.Core/stryker-config.json
|
||||
|
||||
docs/testing/
|
||||
├── ci-quality-gates.md
|
||||
├── security-testing-guide.md
|
||||
└── mutation-testing-guide.md
|
||||
```
|
||||
|
||||
### Modified Files
|
||||
|
||||
```
|
||||
.gitea/workflows/build-test-deploy.yml
|
||||
tests/fixtures/sca/catalogue/inputs.lock
|
||||
tests/fixtures/sca/catalogue/README.md
|
||||
README.md (badges)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Rollback Strategy
|
||||
|
||||
If quality gates cause CI instability:
|
||||
|
||||
1. **Immediate:** Set `failure_mode: warn` in threshold configs
|
||||
2. **Short-term:** Remove `needs:` dependencies to unblock other jobs
|
||||
3. **Investigation:** Create issue with specific threshold that failed
|
||||
4. **Resolution:** Either fix underlying issue or adjust threshold
|
||||
5. **Re-enable:** Set `failure_mode: block` after verification
|
||||
|
||||
---
|
||||
|
||||
## Success Metrics
|
||||
|
||||
| Metric | Current | Target | Measurement |
|
||||
|--------|---------|--------|-------------|
|
||||
| FC Catalogue coverage | 5 cases | 10 cases | Count of fixtures |
|
||||
| Security test coverage | Partial | 50+ tests | OWASP categories |
|
||||
| Mutation score (Scanner) | Unknown | ≥ 70% | Stryker weekly |
|
||||
| Mutation score (Policy) | Unknown | ≥ 70% | Stryker weekly |
|
||||
| Mutation score (Authority) | Unknown | ≥ 80% | Stryker weekly |
|
||||
| Quality gate pass rate | N/A | ≥ 95% | CI runs |
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
### Sprint Files
|
||||
- `docs/implplan/SPRINT_0350_0001_0001_ci_quality_gates_foundation.md`
|
||||
- `docs/implplan/SPRINT_0351_0001_0001_sca_failure_catalogue_completion.md`
|
||||
- `docs/implplan/SPRINT_0352_0001_0001_security_testing_framework.md`
|
||||
- `docs/implplan/SPRINT_0353_0001_0001_mutation_testing_integration.md`
|
||||
|
||||
### Source Advisory
|
||||
- `docs/product-advisories/14-Dec-2025 - Testing and Quality Guardrails Technical Reference.md`
|
||||
|
||||
### Existing Documentation
|
||||
- `docs/19_TEST_SUITE_OVERVIEW.md`
|
||||
- `docs/modules/reach-graph/schemas/ground-truth-schema.md`
|
||||
- `docs/modules/reach-graph/guides/corpus-plan.md`
|
||||
- `tests/reachability/README.md`
|
||||
|
||||
---
|
||||
|
||||
## Document Version
|
||||
|
||||
| Field | Value |
|
||||
|-------|-------|
|
||||
| Version | 1.0 |
|
||||
| Created | 2025-12-14 |
|
||||
| Author | Platform Team |
|
||||
| Status | Planning Complete |
|
||||
52
docs/technical/testing/testing-strategy-models.md
Normal file
52
docs/technical/testing/testing-strategy-models.md
Normal file
@@ -0,0 +1,52 @@
|
||||
# Testing Strategy Models and Lanes (2026)
|
||||
|
||||
Source advisory: `docs/product-advisories/22-Dec-2026 - Better testing strategy.md`
|
||||
Supersedes/extends: `docs/product-advisories/archived/2025-12-21-testing-strategy/20-Dec-2025 - Testing strategy.md`
|
||||
|
||||
## Purpose
|
||||
- Define a single testing taxonomy for all StellaOps project types.
|
||||
- Make determinism, offline readiness, and evidence integrity testable by default.
|
||||
- Align CI lanes with a shared catalog so coverage is visible and enforceable.
|
||||
|
||||
## Strategy in brief
|
||||
- Use test models (L0, S1, C1, W1, WK1, T1, AN1, CLI1, PERF) to encode required test types.
|
||||
- Map every module to one or more models in `docs/testing/TEST_CATALOG.yml`.
|
||||
- Run tests through standardized CI lanes (Unit, Contract, Integration, Security, Performance, Live).
|
||||
|
||||
## Test models (requirements)
|
||||
- L0 (Library/Core): unit + property + snapshot + determinism.
|
||||
- S1 (Storage/Postgres): migrations + idempotency + concurrency + query ordering.
|
||||
- T1 (Transport/Queue): protocol roundtrip + fuzz invalid + delivery semantics.
|
||||
- C1 (Connector/External): fixtures + snapshot + resilience + security; optional Live smoke.
|
||||
- W1 (WebService/API): contract + authz + OTel trace assertions + negative cases.
|
||||
- WK1 (Worker/Indexer): end-to-end job flow + retries + idempotency + telemetry.
|
||||
- AN1 (Analyzer/SourceGen): Roslyn harness + diagnostics + golden generated output.
|
||||
- CLI1 (Tool/CLI): exit codes + golden output + deterministic formatting.
|
||||
- PERF (Benchmark): perf smoke subset + regression thresholds (relative).
|
||||
|
||||
## Repository foundations
|
||||
- TestKit primitives: deterministic time/random, canonical JSON asserts, snapshot helpers, Postgres/Valkey fixtures, OTel capture.
|
||||
- Determinism gate: canonical bytes and stable hashes for SBOM/VEX/verdict artifacts.
|
||||
- Hybrid reachability posture: graph DSSE mandatory; edge-bundle DSSE optional/targeted with deterministic ordering.
|
||||
- Architecture guards: enforce cross-module dependency boundaries (no lattice in Concelier/Excititor).
|
||||
- Offline defaults: no network access unless explicitly tagged `Live`.
|
||||
|
||||
## CI lanes (standard filters)
|
||||
- Unit: fast, offline; includes property and snapshot sub-traits.
|
||||
- Contract: schema/OpenAPI stability and response envelopes.
|
||||
- Integration: Testcontainers-backed service and storage tests.
|
||||
- Security: authz/negative tests and security regressions.
|
||||
- Performance: perf smoke and benchmark guards.
|
||||
- Live: opt-in upstream connector checks (never PR gating by default).
|
||||
|
||||
## Documentation moments (when to update)
|
||||
- New model or required test type: update `docs/testing/TEST_CATALOG.yml`.
|
||||
- New lane or gate: update `docs/TEST_SUITE_OVERVIEW.md` and `docs/testing/ci-quality-gates.md`.
|
||||
- Module-specific test policy change: update the module dossier under `docs/modules/<module>/`.
|
||||
- New fixtures or runnable harnesses: place under `docs/benchmarks/**` or `tests/**` and link here.
|
||||
|
||||
## Related artifacts
|
||||
- Test catalog (source of truth): `docs/testing/TEST_CATALOG.yml`
|
||||
- Test suite overview: `docs/TEST_SUITE_OVERVIEW.md`
|
||||
- Quality guardrails: `docs/testing/testing-quality-guardrails-implementation.md`
|
||||
- Code samples from the advisory: `docs/benchmarks/testing/better-testing-strategy-samples.md`
|
||||
613
docs/technical/testing/testkit-usage-guide.md
Normal file
613
docs/technical/testing/testkit-usage-guide.md
Normal file
@@ -0,0 +1,613 @@
|
||||
# StellaOps.TestKit Usage Guide
|
||||
|
||||
**Version:** 1.0
|
||||
**Status:** Pilot Release (Wave 4 Complete)
|
||||
**Audience:** StellaOps developers writing unit, integration, and contract tests
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
`StellaOps.TestKit` provides deterministic testing infrastructure for StellaOps modules. It eliminates flaky tests, provides reproducible test primitives, and standardizes fixtures for integration testing.
|
||||
|
||||
### Key Features
|
||||
|
||||
- **Deterministic Time**: Freeze and advance time for reproducible tests
|
||||
- **Deterministic Random**: Seeded random number generation
|
||||
- **Canonical JSON Assertions**: SHA-256 hash verification for determinism
|
||||
- **Snapshot Testing**: Golden master regression testing
|
||||
- **PostgreSQL Fixture**: Testcontainers-based PostgreSQL 16 for integration tests
|
||||
- **Valkey Fixture**: Redis-compatible caching tests
|
||||
- **HTTP Fixture**: In-memory API contract testing
|
||||
- **OpenTelemetry Capture**: Trace and span assertion helpers
|
||||
- **Test Categories**: Standardized trait constants for CI filtering
|
||||
|
||||
---
|
||||
|
||||
## Installation
|
||||
|
||||
Add `StellaOps.TestKit` as a project reference to your test project:
|
||||
|
||||
```xml
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="../../../__Libraries/StellaOps.TestKit/StellaOps.TestKit.csproj" />
|
||||
</ItemGroup>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Quick Start Examples
|
||||
|
||||
### 1. Deterministic Time
|
||||
|
||||
Eliminate flaky tests caused by time-dependent logic:
|
||||
|
||||
```csharp
|
||||
using StellaOps.TestKit.Deterministic;
|
||||
using Xunit;
|
||||
|
||||
[Fact]
|
||||
public void Test_ExpirationLogic()
|
||||
{
|
||||
// Arrange: Fix time at a known UTC timestamp
|
||||
using var time = new DeterministicTime(new DateTime(2026, 1, 15, 10, 30, 0, DateTimeKind.Utc));
|
||||
|
||||
var expiresAt = time.UtcNow.AddHours(24);
|
||||
|
||||
// Act: Advance time to just before expiration
|
||||
time.Advance(TimeSpan.FromHours(23));
|
||||
Assert.False(time.UtcNow > expiresAt);
|
||||
|
||||
// Advance past expiration
|
||||
time.Advance(TimeSpan.FromHours(2));
|
||||
Assert.True(time.UtcNow > expiresAt);
|
||||
}
|
||||
```
|
||||
|
||||
**API Reference:**
|
||||
- `DeterministicTime(DateTime initialUtc)` - Create with fixed start time
|
||||
- `UtcNow` - Get current deterministic time
|
||||
- `Advance(TimeSpan duration)` - Move time forward
|
||||
- `SetTo(DateTime newUtc)` - Jump to specific time
|
||||
|
||||
---
|
||||
|
||||
### 2. Deterministic Random
|
||||
|
||||
Reproducible random sequences for property tests and fuzzing:
|
||||
|
||||
```csharp
|
||||
using StellaOps.TestKit.Deterministic;
|
||||
|
||||
[Fact]
|
||||
public void Test_RandomIdGeneration()
|
||||
{
|
||||
// Arrange: Same seed produces same sequence
|
||||
var random1 = new DeterministicRandom(seed: 42);
|
||||
var random2 = new DeterministicRandom(seed: 42);
|
||||
|
||||
// Act
|
||||
var guid1 = random1.NextGuid();
|
||||
var guid2 = random2.NextGuid();
|
||||
|
||||
// Assert: Reproducible GUIDs
|
||||
Assert.Equal(guid1, guid2);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Test_Shuffling()
|
||||
{
|
||||
var random = new DeterministicRandom(seed: 100);
|
||||
var array = new[] { 1, 2, 3, 4, 5 };
|
||||
|
||||
random.Shuffle(array);
|
||||
|
||||
// Deterministic shuffle order
|
||||
Assert.NotEqual(new[] { 1, 2, 3, 4, 5 }, array);
|
||||
}
|
||||
```
|
||||
|
||||
**API Reference:**
|
||||
- `DeterministicRandom(int seed)` - Create with seed
|
||||
- `NextGuid()` - Generate deterministic GUID
|
||||
- `NextString(int length)` - Generate alphanumeric string
|
||||
- `NextInt(int min, int max)` - Generate integer in range
|
||||
- `Shuffle<T>(T[] array)` - Fisher-Yates shuffle
|
||||
|
||||
---
|
||||
|
||||
### 3. Canonical JSON Assertions
|
||||
|
||||
Verify JSON determinism for SBOM, VEX, and attestation outputs:
|
||||
|
||||
```csharp
|
||||
using StellaOps.TestKit.Assertions;
|
||||
|
||||
[Fact]
|
||||
public void Test_SbomDeterminism()
|
||||
{
|
||||
var sbom = new
|
||||
{
|
||||
SpdxVersion = "SPDX-3.0.1",
|
||||
Name = "MySbom",
|
||||
Packages = new[] { new { Name = "Pkg1", Version = "1.0" } }
|
||||
};
|
||||
|
||||
// Verify deterministic serialization
|
||||
CanonicalJsonAssert.IsDeterministic(sbom, iterations: 100);
|
||||
|
||||
// Verify expected hash (golden master)
|
||||
var expectedHash = "abc123..."; // Precomputed SHA-256
|
||||
CanonicalJsonAssert.HasExpectedHash(sbom, expectedHash);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Test_JsonPropertyExists()
|
||||
{
|
||||
var vex = new
|
||||
{
|
||||
Document = new { Id = "VEX-2026-001" },
|
||||
Statements = new[] { new { Vulnerability = "CVE-2026-1234" } }
|
||||
};
|
||||
|
||||
// Deep property verification
|
||||
CanonicalJsonAssert.ContainsProperty(vex, "Document.Id", "VEX-2026-001");
|
||||
CanonicalJsonAssert.ContainsProperty(vex, "Statements[0].Vulnerability", "CVE-2026-1234");
|
||||
}
|
||||
```
|
||||
|
||||
**API Reference:**
|
||||
- `IsDeterministic<T>(T value, int iterations)` - Verify N serializations match
|
||||
- `HasExpectedHash<T>(T value, string expectedSha256Hex)` - Verify SHA-256 hash
|
||||
- `ComputeCanonicalHash<T>(T value)` - Compute hash for golden master
|
||||
- `AreCanonicallyEqual<T>(T expected, T actual)` - Compare canonical JSON
|
||||
- `ContainsProperty<T>(T value, string propertyPath, object expectedValue)` - Deep search
|
||||
|
||||
---
|
||||
|
||||
### 4. Snapshot Testing
|
||||
|
||||
Golden master regression testing for complex outputs:
|
||||
|
||||
```csharp
|
||||
using StellaOps.TestKit.Assertions;
|
||||
|
||||
[Fact, Trait("Category", TestCategories.Snapshot)]
|
||||
public void Test_SbomGeneration()
|
||||
{
|
||||
var sbom = GenerateSbom(); // Your SBOM generation logic
|
||||
|
||||
// Snapshot will be stored in Snapshots/TestSbomGeneration.json
|
||||
SnapshotAssert.MatchesSnapshot(sbom, "TestSbomGeneration");
|
||||
}
|
||||
|
||||
// Update snapshots when intentional changes occur:
|
||||
// UPDATE_SNAPSHOTS=1 dotnet test
|
||||
```
|
||||
|
||||
**Text and Binary Snapshots:**
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public void Test_LicenseText()
|
||||
{
|
||||
var licenseText = GenerateLicenseNotice();
|
||||
SnapshotAssert.MatchesTextSnapshot(licenseText, "LicenseNotice");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Test_SignatureBytes()
|
||||
{
|
||||
var signature = SignDocument(document);
|
||||
SnapshotAssert.MatchesBinarySnapshot(signature, "DocumentSignature");
|
||||
}
|
||||
```
|
||||
|
||||
**API Reference:**
|
||||
- `MatchesSnapshot<T>(T value, string snapshotName)` - JSON snapshot
|
||||
- `MatchesTextSnapshot(string value, string snapshotName)` - Text snapshot
|
||||
- `MatchesBinarySnapshot(byte[] value, string snapshotName)` - Binary snapshot
|
||||
- Environment variable: `UPDATE_SNAPSHOTS=1` to update baselines
|
||||
|
||||
---
|
||||
|
||||
### 5. PostgreSQL Fixture
|
||||
|
||||
Testcontainers-based PostgreSQL 16 for integration tests:
|
||||
|
||||
```csharp
|
||||
using StellaOps.TestKit.Fixtures;
|
||||
using Xunit;
|
||||
|
||||
public class DatabaseTests : IClassFixture<PostgresFixture>
|
||||
{
|
||||
private readonly PostgresFixture _fixture;
|
||||
|
||||
public DatabaseTests(PostgresFixture fixture)
|
||||
{
|
||||
_fixture = fixture;
|
||||
}
|
||||
|
||||
[Fact, Trait("Category", TestCategories.Integration)]
|
||||
public async Task Test_DatabaseOperations()
|
||||
{
|
||||
// Use _fixture.ConnectionString to connect
|
||||
using var connection = new NpgsqlConnection(_fixture.ConnectionString);
|
||||
await connection.OpenAsync();
|
||||
|
||||
// Run migrations
|
||||
await _fixture.RunMigrationsAsync(connection);
|
||||
|
||||
// Test database operations
|
||||
var result = await connection.QueryAsync("SELECT version()");
|
||||
Assert.NotEmpty(result);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**API Reference:**
|
||||
- `PostgresFixture` - xUnit class fixture
|
||||
- `ConnectionString` - PostgreSQL connection string
|
||||
- `RunMigrationsAsync(DbConnection)` - Apply migrations
|
||||
- Requires Docker running locally
|
||||
|
||||
---
|
||||
|
||||
### 6. Valkey Fixture
|
||||
|
||||
Redis-compatible caching for integration tests:
|
||||
|
||||
```csharp
|
||||
using StellaOps.TestKit.Fixtures;
|
||||
|
||||
public class CacheTests : IClassFixture<ValkeyFixture>
|
||||
{
|
||||
private readonly ValkeyFixture _fixture;
|
||||
|
||||
[Fact, Trait("Category", TestCategories.Integration)]
|
||||
public async Task Test_CachingLogic()
|
||||
{
|
||||
var connection = await ConnectionMultiplexer.Connect(_fixture.ConnectionString);
|
||||
var db = connection.GetDatabase();
|
||||
|
||||
await db.StringSetAsync("key", "value");
|
||||
var result = await db.StringGetAsync("key");
|
||||
|
||||
Assert.Equal("value", result.ToString());
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**API Reference:**
|
||||
- `ValkeyFixture` - xUnit class fixture
|
||||
- `ConnectionString` - Redis connection string (host:port)
|
||||
- `Host`, `Port` - Connection details
|
||||
- Uses `redis:7-alpine` image (Valkey-compatible)
|
||||
|
||||
---
|
||||
|
||||
### 7. HTTP Fixture Server
|
||||
|
||||
In-memory API contract testing:
|
||||
|
||||
```csharp
|
||||
using StellaOps.TestKit.Fixtures;
|
||||
|
||||
public class ApiTests : IClassFixture<HttpFixtureServer<Program>>
|
||||
{
|
||||
private readonly HttpClient _client;
|
||||
|
||||
public ApiTests(HttpFixtureServer<Program> fixture)
|
||||
{
|
||||
_client = fixture.CreateClient();
|
||||
}
|
||||
|
||||
[Fact, Trait("Category", TestCategories.Contract)]
|
||||
public async Task Test_HealthEndpoint()
|
||||
{
|
||||
var response = await _client.GetAsync("/health");
|
||||
response.EnsureSuccessStatusCode();
|
||||
|
||||
var body = await response.Content.ReadAsStringAsync();
|
||||
Assert.Contains("healthy", body);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**HTTP Message Handler Stub (Hermetic Tests):**
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public async Task Test_ExternalApiCall()
|
||||
{
|
||||
var handler = new HttpMessageHandlerStub()
|
||||
.WhenRequest("https://api.example.com/data", HttpStatusCode.OK, "{\"status\":\"ok\"}");
|
||||
|
||||
var httpClient = new HttpClient(handler);
|
||||
var response = await httpClient.GetAsync("https://api.example.com/data");
|
||||
|
||||
Assert.Equal(HttpStatusCode.OK, response.StatusCode);
|
||||
}
|
||||
```
|
||||
|
||||
**API Reference:**
|
||||
- `HttpFixtureServer<TProgram>` - WebApplicationFactory wrapper
|
||||
- `CreateClient()` - Get HttpClient for test server
|
||||
- `HttpMessageHandlerStub` - Stub external HTTP dependencies
|
||||
- `WhenRequest(url, statusCode, content)` - Configure stub responses
|
||||
|
||||
---
|
||||
|
||||
### 8. OpenTelemetry Capture
|
||||
|
||||
Trace and span assertion helpers:
|
||||
|
||||
```csharp
|
||||
using StellaOps.TestKit.Observability;
|
||||
|
||||
[Fact]
|
||||
public async Task Test_TracingBehavior()
|
||||
{
|
||||
using var capture = new OtelCapture();
|
||||
|
||||
// Execute code that emits traces
|
||||
await MyService.DoWorkAsync();
|
||||
|
||||
// Assert traces
|
||||
capture.AssertHasSpan("MyService.DoWork");
|
||||
capture.AssertHasTag("user_id", "123");
|
||||
capture.AssertSpanCount(expectedCount: 3);
|
||||
|
||||
// Verify parent-child hierarchy
|
||||
capture.AssertHierarchy("ParentSpan", "ChildSpan");
|
||||
}
|
||||
```
|
||||
|
||||
**API Reference:**
|
||||
- `OtelCapture(string? activitySourceName = null)` - Create capture
|
||||
- `AssertHasSpan(string spanName)` - Verify span exists
|
||||
- `AssertHasTag(string tagKey, string expectedValue)` - Verify tag
|
||||
- `AssertSpanCount(int expectedCount)` - Verify span count
|
||||
- `AssertHierarchy(string parentSpanName, string childSpanName)` - Verify parent-child
|
||||
- `CapturedActivities` - Get all captured spans
|
||||
|
||||
---
|
||||
|
||||
### 9. Test Categories
|
||||
|
||||
Standardized trait constants for CI lane filtering:
|
||||
|
||||
```csharp
|
||||
using StellaOps.TestKit;
|
||||
|
||||
[Fact, Trait("Category", TestCategories.Unit)]
|
||||
public void FastUnitTest() { }
|
||||
|
||||
[Fact, Trait("Category", TestCategories.Integration)]
|
||||
public async Task SlowIntegrationTest() { }
|
||||
|
||||
[Fact, Trait("Category", TestCategories.Live)]
|
||||
public async Task RequiresExternalServices() { }
|
||||
```
|
||||
|
||||
**CI Lane Filtering:**
|
||||
|
||||
```bash
|
||||
# Run only unit tests (fast, no dependencies)
|
||||
dotnet test --filter "Category=Unit"
|
||||
|
||||
# Run all tests except Live
|
||||
dotnet test --filter "Category!=Live"
|
||||
|
||||
# Run Integration + Contract tests
|
||||
dotnet test --filter "Category=Integration|Category=Contract"
|
||||
```
|
||||
|
||||
**Available Categories:**
|
||||
- `Unit` - Fast, in-memory, no external dependencies
|
||||
- `Property` - FsCheck/generative testing
|
||||
- `Snapshot` - Golden master regression
|
||||
- `Integration` - Testcontainers (PostgreSQL, Valkey)
|
||||
- `Contract` - API/WebService contract tests
|
||||
- `Security` - Cryptographic validation
|
||||
- `Performance` - Benchmarking, load tests
|
||||
- `Live` - Requires external services (disabled in CI by default)
|
||||
|
||||
---
|
||||
|
||||
## Best Practices
|
||||
|
||||
### 1. Always Use TestCategories
|
||||
|
||||
Tag every test with the appropriate category:
|
||||
|
||||
```csharp
|
||||
[Fact, Trait("Category", TestCategories.Unit)]
|
||||
public void MyUnitTest() { }
|
||||
```
|
||||
|
||||
This enables CI lane filtering and improves test discoverability.
|
||||
|
||||
### 2. Prefer Deterministic Primitives
|
||||
|
||||
Avoid `DateTime.UtcNow`, `Guid.NewGuid()`, `Random` in tests. Use TestKit alternatives:
|
||||
|
||||
```csharp
|
||||
// ❌ Flaky test (time-dependent)
|
||||
var expiration = DateTime.UtcNow.AddHours(1);
|
||||
|
||||
// ✅ Deterministic test
|
||||
using var time = new DeterministicTime(DateTime.UtcNow);
|
||||
var expiration = time.UtcNow.AddHours(1);
|
||||
```
|
||||
|
||||
### 3. Use Snapshot Tests for Complex Outputs
|
||||
|
||||
For large JSON outputs (SBOM, VEX, attestations), snapshot testing is more maintainable than manual assertions:
|
||||
|
||||
```csharp
|
||||
// ❌ Brittle manual assertions
|
||||
Assert.Equal("SPDX-3.0.1", sbom.SpdxVersion);
|
||||
Assert.Equal(42, sbom.Packages.Count);
|
||||
// ...hundreds of assertions...
|
||||
|
||||
// ✅ Snapshot testing
|
||||
SnapshotAssert.MatchesSnapshot(sbom, "MySbomSnapshot");
|
||||
```
|
||||
|
||||
### 4. Isolate Integration Tests
|
||||
|
||||
Use TestCategories to separate fast unit tests from slow integration tests:
|
||||
|
||||
```csharp
|
||||
[Fact, Trait("Category", TestCategories.Unit)]
|
||||
public void FastTest() { /* no external dependencies */ }
|
||||
|
||||
[Fact, Trait("Category", TestCategories.Integration)]
|
||||
public async Task SlowTest() { /* uses PostgresFixture */ }
|
||||
```
|
||||
|
||||
In CI, run Unit tests first for fast feedback, then Integration tests in parallel.
|
||||
|
||||
### 5. Document Snapshot Baselines
|
||||
|
||||
When updating snapshots (`UPDATE_SNAPSHOTS=1`), add a commit message explaining why:
|
||||
|
||||
```bash
|
||||
git commit -m "Update SBOM snapshot: added new package metadata fields"
|
||||
```
|
||||
|
||||
This helps reviewers understand intentional vs. accidental changes.
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Snapshot Mismatch
|
||||
|
||||
**Error:** `Snapshot 'MySbomSnapshot' does not match expected.`
|
||||
|
||||
**Solution:**
|
||||
1. Review diff manually (check `Snapshots/MySbomSnapshot.json`)
|
||||
2. If change is intentional: `UPDATE_SNAPSHOTS=1 dotnet test`
|
||||
3. Commit updated snapshot with explanation
|
||||
|
||||
### Testcontainers Failure
|
||||
|
||||
**Error:** `Docker daemon not running`
|
||||
|
||||
**Solution:**
|
||||
- Ensure Docker Desktop is running
|
||||
- Verify `docker ps` works in terminal
|
||||
- Check Testcontainers logs: `TESTCONTAINERS_DEBUG=1 dotnet test`
|
||||
|
||||
### Determinism Failure
|
||||
|
||||
**Error:** `CanonicalJsonAssert.IsDeterministic failed: byte arrays differ`
|
||||
|
||||
**Root Cause:** Non-deterministic data in serialization (e.g., random GUIDs, timestamps)
|
||||
|
||||
**Solution:**
|
||||
- Use `DeterministicTime` and `DeterministicRandom`
|
||||
- Ensure all data is seeded or mocked
|
||||
- Check for `DateTime.UtcNow` or `Guid.NewGuid()` calls
|
||||
|
||||
---
|
||||
|
||||
## Migration Guide (Existing Tests)
|
||||
|
||||
### Step 1: Add TestKit Reference
|
||||
|
||||
```xml
|
||||
<ProjectReference Include="../../../__Libraries/StellaOps.TestKit/StellaOps.TestKit.csproj" />
|
||||
```
|
||||
|
||||
### Step 2: Replace Time-Dependent Code
|
||||
|
||||
**Before:**
|
||||
```csharp
|
||||
var now = DateTime.UtcNow;
|
||||
```
|
||||
|
||||
**After:**
|
||||
```csharp
|
||||
using var time = new DeterministicTime(DateTime.UtcNow);
|
||||
var now = time.UtcNow;
|
||||
```
|
||||
|
||||
### Step 3: Add Test Categories
|
||||
|
||||
```csharp
|
||||
[Fact] // Old
|
||||
[Fact, Trait("Category", TestCategories.Unit)] // New
|
||||
```
|
||||
|
||||
### Step 4: Adopt Snapshot Testing (Optional)
|
||||
|
||||
For complex JSON assertions, replace manual checks with snapshots:
|
||||
|
||||
```csharp
|
||||
// Old
|
||||
Assert.Equal(expected.SpdxVersion, actual.SpdxVersion);
|
||||
// ...
|
||||
|
||||
// New
|
||||
SnapshotAssert.MatchesSnapshot(actual, "TestName");
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## CI Integration
|
||||
|
||||
### Example `.gitea/workflows/test.yml`
|
||||
|
||||
```yaml
|
||||
name: Test Suite
|
||||
on: [push, pull_request]
|
||||
|
||||
jobs:
|
||||
unit:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: actions/setup-dotnet@v4
|
||||
with:
|
||||
dotnet-version: '10.0.x'
|
||||
- name: Unit Tests (Fast)
|
||||
run: dotnet test --filter "Category=Unit" --logger "trx;LogFileName=unit-results.trx"
|
||||
- name: Upload Results
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: unit-test-results
|
||||
path: '**/unit-results.trx'
|
||||
|
||||
integration:
|
||||
runs-on: ubuntu-latest
|
||||
services:
|
||||
docker:
|
||||
image: docker:dind
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- name: Integration Tests
|
||||
run: dotnet test --filter "Category=Integration" --logger "trx;LogFileName=integration-results.trx"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Support and Feedback
|
||||
|
||||
- **Issues:** Report bugs in sprint tracking files under `docs/implplan/`
|
||||
- **Questions:** Contact Platform Guild
|
||||
- **Documentation:** `src/__Libraries/StellaOps.TestKit/README.md`
|
||||
|
||||
---
|
||||
|
||||
## Changelog
|
||||
|
||||
### v1.0 (2025-12-23)
|
||||
- Initial release: DeterministicTime, DeterministicRandom
|
||||
- CanonicalJsonAssert, SnapshotAssert
|
||||
- PostgresFixture, ValkeyFixture, HttpFixtureServer
|
||||
- OtelCapture for OpenTelemetry traces
|
||||
- TestCategories for CI lane filtering
|
||||
- Pilot adoption in Scanner.Core.Tests
|
||||
366
docs/technical/testing/webservice-test-discipline.md
Normal file
366
docs/technical/testing/webservice-test-discipline.md
Normal file
@@ -0,0 +1,366 @@
|
||||
# WebService Test Discipline
|
||||
|
||||
This document defines the testing discipline for StellaOps WebService projects. All web services must follow these patterns to ensure consistent test coverage, contract stability, telemetry verification, and security hardening.
|
||||
|
||||
## Overview
|
||||
|
||||
WebService tests use `WebServiceFixture<TProgram>` from `StellaOps.TestKit` and `WebApplicationFactory<TProgram>` from `Microsoft.AspNetCore.Mvc.Testing`. Tests are organized into four categories:
|
||||
|
||||
1. **Contract Tests** — OpenAPI schema stability
|
||||
2. **OTel Trace Tests** — Telemetry verification
|
||||
3. **Negative Tests** — Error handling validation
|
||||
4. **Auth/AuthZ Tests** — Security boundary enforcement
|
||||
|
||||
---
|
||||
|
||||
## 1. Test Infrastructure
|
||||
|
||||
### WebServiceFixture Pattern
|
||||
|
||||
```csharp
|
||||
using StellaOps.TestKit.Fixtures;
|
||||
|
||||
public class ScannerWebServiceTests : WebServiceTestBase<ScannerProgram>
|
||||
{
|
||||
public ScannerWebServiceTests() : base(new WebServiceFixture<ScannerProgram>())
|
||||
{
|
||||
}
|
||||
|
||||
// Tests inherit shared fixture setup
|
||||
}
|
||||
```
|
||||
|
||||
### Fixture Configuration
|
||||
|
||||
Each web service should have a dedicated fixture class that configures test-specific settings:
|
||||
|
||||
```csharp
|
||||
public sealed class ScannerTestFixture : WebServiceFixture<ScannerProgram>
|
||||
{
|
||||
protected override void ConfigureTestServices(IServiceCollection services)
|
||||
{
|
||||
// Replace external dependencies with test doubles
|
||||
services.AddSingleton<IStorageClient, InMemoryStorageClient>();
|
||||
services.AddSingleton<IQueueClient, InMemoryQueueClient>();
|
||||
}
|
||||
|
||||
protected override void ConfigureTestConfiguration(IDictionary<string, string?> config)
|
||||
{
|
||||
config["scanner:storage:driver"] = "inmemory";
|
||||
config["scanner:events:enabled"] = "false";
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Contract Tests
|
||||
|
||||
Contract tests ensure OpenAPI schema stability and detect breaking changes.
|
||||
|
||||
### Pattern
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
[Trait("Lane", "Contract")]
|
||||
public async Task OpenApi_Schema_MatchesSnapshot()
|
||||
{
|
||||
// Arrange
|
||||
using var client = Fixture.CreateClient();
|
||||
|
||||
// Act
|
||||
var response = await client.GetAsync("/swagger/v1/swagger.json");
|
||||
var schema = await response.Content.ReadAsStringAsync();
|
||||
|
||||
// Assert
|
||||
await ContractTestHelper.AssertSchemaMatchesSnapshot(schema, "scanner-v1");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
[Trait("Lane", "Contract")]
|
||||
public async Task Api_Response_MatchesContract()
|
||||
{
|
||||
// Arrange
|
||||
using var client = Fixture.CreateClient();
|
||||
var request = new ScanRequest { /* test data */ };
|
||||
|
||||
// Act
|
||||
var response = await client.PostAsJsonAsync("/api/v1/scans", request);
|
||||
var result = await response.Content.ReadFromJsonAsync<ScanResponse>();
|
||||
|
||||
// Assert
|
||||
ContractTestHelper.AssertResponseMatchesSchema(result, "ScanResponse");
|
||||
}
|
||||
```
|
||||
|
||||
### Snapshot Management
|
||||
|
||||
- Snapshots stored in `Snapshots/` directory relative to test project
|
||||
- Schema format: `<service>-<version>.json`
|
||||
- Update snapshots intentionally when breaking changes are approved
|
||||
|
||||
---
|
||||
|
||||
## 3. OTel Trace Tests
|
||||
|
||||
OTel tests verify that telemetry spans are emitted correctly with required tags.
|
||||
|
||||
### Pattern
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
[Trait("Lane", "Integration")]
|
||||
public async Task ScanEndpoint_EmitsOtelTrace()
|
||||
{
|
||||
// Arrange
|
||||
using var otelCapture = Fixture.CaptureOtelTraces();
|
||||
using var client = Fixture.CreateClient();
|
||||
var request = new ScanRequest { ImageRef = "nginx:1.25" };
|
||||
|
||||
// Act
|
||||
await client.PostAsJsonAsync("/api/v1/scans", request);
|
||||
|
||||
// Assert
|
||||
otelCapture.AssertHasSpan("scanner.scan");
|
||||
otelCapture.AssertHasTag("scanner.scan", "scan.image_ref", "nginx:1.25");
|
||||
otelCapture.AssertHasTag("scanner.scan", "tenant.id", ExpectedTenantId);
|
||||
}
|
||||
```
|
||||
|
||||
### Required Tags
|
||||
|
||||
All WebService endpoints must emit these tags:
|
||||
|
||||
| Tag | Description | Example |
|
||||
|-----|-------------|---------|
|
||||
| `tenant.id` | Tenant identifier | `tenant-a` |
|
||||
| `request.id` | Correlation ID | `req-abc123` |
|
||||
| `http.route` | Endpoint route | `/api/v1/scans` |
|
||||
| `http.status_code` | Response code | `200` |
|
||||
|
||||
Service-specific tags are documented in each module's architecture doc.
|
||||
|
||||
---
|
||||
|
||||
## 4. Negative Tests
|
||||
|
||||
Negative tests verify proper error handling for invalid inputs.
|
||||
|
||||
### Pattern
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
[Trait("Lane", "Security")]
|
||||
public async Task MalformedContentType_Returns415()
|
||||
{
|
||||
// Arrange
|
||||
using var client = Fixture.CreateClient();
|
||||
var content = new StringContent("{}", Encoding.UTF8, "text/plain");
|
||||
|
||||
// Act
|
||||
var response = await client.PostAsync("/api/v1/scans", content);
|
||||
|
||||
// Assert
|
||||
Assert.Equal(HttpStatusCode.UnsupportedMediaType, response.StatusCode);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
[Trait("Lane", "Security")]
|
||||
public async Task OversizedPayload_Returns413()
|
||||
{
|
||||
// Arrange
|
||||
using var client = Fixture.CreateClient();
|
||||
var payload = new string('x', 10_000_001); // Exceeds 10MB limit
|
||||
var content = new StringContent(payload, Encoding.UTF8, "application/json");
|
||||
|
||||
// Act
|
||||
var response = await client.PostAsync("/api/v1/scans", content);
|
||||
|
||||
// Assert
|
||||
Assert.Equal(HttpStatusCode.RequestEntityTooLarge, response.StatusCode);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
[Trait("Lane", "Unit")]
|
||||
public async Task MethodMismatch_Returns405()
|
||||
{
|
||||
// Arrange
|
||||
using var client = Fixture.CreateClient();
|
||||
|
||||
// Act (POST endpoint, but using GET)
|
||||
var response = await client.GetAsync("/api/v1/scans");
|
||||
|
||||
// Assert
|
||||
Assert.Equal(HttpStatusCode.MethodNotAllowed, response.StatusCode);
|
||||
}
|
||||
```
|
||||
|
||||
### Required Coverage
|
||||
|
||||
| Negative Case | Expected Status | Test Trait |
|
||||
|--------------|-----------------|------------|
|
||||
| Malformed content type | 415 | Security |
|
||||
| Oversized payload | 413 | Security |
|
||||
| Method mismatch | 405 | Unit |
|
||||
| Missing required field | 400 | Unit |
|
||||
| Invalid field value | 400 | Unit |
|
||||
| Unknown route | 404 | Unit |
|
||||
|
||||
---
|
||||
|
||||
## 5. Auth/AuthZ Tests
|
||||
|
||||
Auth tests verify security boundaries and tenant isolation.
|
||||
|
||||
### Pattern
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
[Trait("Lane", "Security")]
|
||||
public async Task AnonymousRequest_Returns401()
|
||||
{
|
||||
// Arrange
|
||||
using var client = Fixture.CreateClient(); // No auth
|
||||
|
||||
// Act
|
||||
var response = await client.GetAsync("/api/v1/scans");
|
||||
|
||||
// Assert
|
||||
Assert.Equal(HttpStatusCode.Unauthorized, response.StatusCode);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
[Trait("Lane", "Security")]
|
||||
public async Task ExpiredToken_Returns401()
|
||||
{
|
||||
// Arrange
|
||||
using var client = Fixture.CreateAuthenticatedClient(tokenExpired: true);
|
||||
|
||||
// Act
|
||||
var response = await client.GetAsync("/api/v1/scans");
|
||||
|
||||
// Assert
|
||||
Assert.Equal(HttpStatusCode.Unauthorized, response.StatusCode);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
[Trait("Lane", "Security")]
|
||||
public async Task TenantIsolation_CannotAccessOtherTenantData()
|
||||
{
|
||||
// Arrange
|
||||
using var tenantAClient = Fixture.CreateTenantClient("tenant-a");
|
||||
using var tenantBClient = Fixture.CreateTenantClient("tenant-b");
|
||||
|
||||
// Create scan as tenant A
|
||||
var scanResponse = await tenantAClient.PostAsJsonAsync("/api/v1/scans", new ScanRequest { /* */ });
|
||||
var scan = await scanResponse.Content.ReadFromJsonAsync<ScanResponse>();
|
||||
|
||||
// Act: Try to access as tenant B
|
||||
var response = await tenantBClient.GetAsync($"/api/v1/scans/{scan!.Id}");
|
||||
|
||||
// Assert
|
||||
Assert.Equal(HttpStatusCode.NotFound, response.StatusCode); // Tenant isolation
|
||||
}
|
||||
```
|
||||
|
||||
### Required Coverage
|
||||
|
||||
| Auth Case | Expected Behavior | Test Trait |
|
||||
|-----------|-------------------|------------|
|
||||
| No token | 401 Unauthorized | Security |
|
||||
| Expired token | 401 Unauthorized | Security |
|
||||
| Invalid signature | 401 Unauthorized | Security |
|
||||
| Wrong audience | 401 Unauthorized | Security |
|
||||
| Missing scope | 403 Forbidden | Security |
|
||||
| Cross-tenant access | 404 Not Found or 403 Forbidden | Security |
|
||||
|
||||
---
|
||||
|
||||
## 6. Test Organization
|
||||
|
||||
### Directory Structure
|
||||
|
||||
```
|
||||
src/<Module>/__Tests/StellaOps.<Module>.WebService.Tests/
|
||||
├── StellaOps.<Module>.WebService.Tests.csproj
|
||||
├── <Module>ApplicationFactory.cs # WebApplicationFactory implementation
|
||||
├── <Module>TestFixture.cs # Shared test fixture
|
||||
├── Contract/
|
||||
│ └── OpenApiSchemaTests.cs
|
||||
├── Telemetry/
|
||||
│ └── OtelTraceTests.cs
|
||||
├── Negative/
|
||||
│ ├── ContentTypeTests.cs
|
||||
│ ├── PayloadLimitTests.cs
|
||||
│ └── MethodMismatchTests.cs
|
||||
├── Auth/
|
||||
│ ├── AuthenticationTests.cs
|
||||
│ ├── AuthorizationTests.cs
|
||||
│ └── TenantIsolationTests.cs
|
||||
└── Snapshots/
|
||||
└── <module>-v1.json # OpenAPI schema snapshot
|
||||
```
|
||||
|
||||
### Test Trait Assignment
|
||||
|
||||
| Category | Trait | CI Lane | PR-Gating |
|
||||
|----------|-------|---------|-----------|
|
||||
| Contract | `[Trait("Lane", "Contract")]` | Contract | Yes |
|
||||
| OTel | `[Trait("Lane", "Integration")]` | Integration | Yes |
|
||||
| Negative (security) | `[Trait("Lane", "Security")]` | Security | Yes |
|
||||
| Negative (validation) | `[Trait("Lane", "Unit")]` | Unit | Yes |
|
||||
| Auth/AuthZ | `[Trait("Lane", "Security")]` | Security | Yes |
|
||||
|
||||
---
|
||||
|
||||
## 7. CI Integration
|
||||
|
||||
WebService tests run in the appropriate CI lanes:
|
||||
|
||||
```yaml
|
||||
# .gitea/workflows/test-lanes.yml
|
||||
jobs:
|
||||
contract-tests:
|
||||
steps:
|
||||
- run: ./scripts/test-lane.sh Contract
|
||||
|
||||
security-tests:
|
||||
steps:
|
||||
- run: ./scripts/test-lane.sh Security
|
||||
|
||||
integration-tests:
|
||||
steps:
|
||||
- run: ./scripts/test-lane.sh Integration
|
||||
```
|
||||
|
||||
All lanes are PR-gating. Failed tests block merge.
|
||||
|
||||
---
|
||||
|
||||
## 8. Rollout Checklist
|
||||
|
||||
When adding WebService tests to a new module:
|
||||
|
||||
- [ ] Create `<Module>ApplicationFactory` extending `WebApplicationFactory<TProgram>`
|
||||
- [ ] Create `<Module>TestFixture` extending `WebServiceFixture<TProgram>` if needed
|
||||
- [ ] Add contract tests with OpenAPI schema snapshot
|
||||
- [ ] Add OTel trace tests for key endpoints
|
||||
- [ ] Add negative tests (content type, payload, method)
|
||||
- [ ] Add auth/authz tests (anonymous, expired, tenant isolation)
|
||||
- [ ] Verify all tests have appropriate `[Trait("Lane", "...")]` attributes
|
||||
- [ ] Run locally: `dotnet test --filter "Lane=Contract|Lane=Security|Lane=Integration"`
|
||||
- [ ] Verify CI passes on PR
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- [WebServiceFixture Implementation](../../src/__Libraries/StellaOps.TestKit/Fixtures/WebServiceFixture.cs)
|
||||
- [ContractTestHelper Implementation](../../src/__Libraries/StellaOps.TestKit/Fixtures/ContractTestHelper.cs)
|
||||
- [WebServiceTestBase Implementation](../../src/__Libraries/StellaOps.TestKit/Templates/WebServiceTestBase.cs)
|
||||
- [Test Lanes CI Workflow](../../.gitea/workflows/test-lanes.yml)
|
||||
- [CI Lane Filters Documentation](./ci-lane-filters.md)
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2025-06-30 · Sprint 5100.0007.0006*
|
||||
230
docs/technical/testing/webservice-test-rollout-plan.md
Normal file
230
docs/technical/testing/webservice-test-rollout-plan.md
Normal file
@@ -0,0 +1,230 @@
|
||||
# WebService Test Rollout Plan
|
||||
|
||||
This document defines the rollout plan for applying the WebService test discipline to all StellaOps web services.
|
||||
|
||||
## Overview
|
||||
|
||||
Following the pilot implementation on Scanner.WebService (Sprint 5100.0007.0006), this plan defines the order and timeline for rolling out comprehensive WebService tests to all remaining services.
|
||||
|
||||
---
|
||||
|
||||
## Service Inventory
|
||||
|
||||
| Service | Module Path | Priority | Status | Sprint |
|
||||
|---------|-------------|----------|--------|--------|
|
||||
| Scanner.WebService | `src/Scanner/StellaOps.Scanner.WebService` | P0 (Pilot) | ✅ Existing tests | 5100.0007.0006 |
|
||||
| Concelier.WebService | `src/Concelier/StellaOps.Concelier.WebService` | P1 | Pending | TBD |
|
||||
| Excititor.WebService | `src/Excititor/StellaOps.Excititor.WebService` | P1 | Pending | TBD |
|
||||
| Policy.Engine | `src/Policy/StellaOps.Policy.Engine` | P1 | Pending | TBD |
|
||||
| Scheduler.WebService | `src/Scheduler/StellaOps.Scheduler.WebService` | P2 | Pending | TBD |
|
||||
| Notify.WebService | `src/Notify/StellaOps.Notify.WebService` | P2 | Pending | TBD |
|
||||
| Authority | `src/Authority/StellaOps.Authority` | P2 | Pending | TBD |
|
||||
| Signer | `src/Signer/StellaOps.Signer` | P3 | Pending | TBD |
|
||||
| Attestor | `src/Attestor/StellaOps.Attestor` | P3 | Pending | TBD |
|
||||
| ExportCenter.WebService | `src/ExportCenter/StellaOps.ExportCenter.WebService` | P3 | Pending | TBD |
|
||||
| Registry.TokenService | `src/Registry/StellaOps.Registry.TokenService` | P3 | Pending | TBD |
|
||||
| VulnExplorer.Api | `src/VulnExplorer/StellaOps.VulnExplorer.Api` | P3 | Pending | TBD |
|
||||
| Graph.Api | `src/Graph/StellaOps.Graph.Api` | P3 | Pending | TBD |
|
||||
| Orchestrator | `src/Orchestrator/StellaOps.Orchestrator` | P4 | Pending | TBD |
|
||||
|
||||
---
|
||||
|
||||
## Rollout Phases
|
||||
|
||||
### Phase 1: Core Data Flow Services (P1)
|
||||
|
||||
**Timeline**: Sprint 5100.0008.* (Q1 2026)
|
||||
|
||||
**Services**:
|
||||
- **Concelier.WebService** — Primary advisory ingestion service
|
||||
- **Excititor.WebService** — Enrichment and correlation service
|
||||
- **Policy.Engine** — Policy evaluation service
|
||||
|
||||
**Rationale**: These services form the core data flow pipeline. They have high traffic, complex contracts, and critical security boundaries.
|
||||
|
||||
**Test Requirements**:
|
||||
| Test Type | Concelier | Excititor | Policy |
|
||||
|-----------|-----------|-----------|--------|
|
||||
| Contract (OpenAPI) | Required | Required | Required |
|
||||
| OTel traces | Required | Required | Required |
|
||||
| Negative tests | Required | Required | Required |
|
||||
| Auth/AuthZ | Required | Required | Required |
|
||||
| Tenant isolation | Required | Required | Required |
|
||||
|
||||
---
|
||||
|
||||
### Phase 2: Scheduling & Notification Services (P2)
|
||||
|
||||
**Timeline**: Sprint 5100.0009.* (Q2 2026)
|
||||
|
||||
**Services**:
|
||||
- **Scheduler.WebService** — Job scheduling and orchestration
|
||||
- **Notify.WebService** — Notification dispatch
|
||||
- **Authority** — Authentication/authorization service
|
||||
|
||||
**Rationale**: These services support operational workflows. Authority is critical for security testing of all other services.
|
||||
|
||||
**Test Requirements**:
|
||||
| Test Type | Scheduler | Notify | Authority |
|
||||
|-----------|-----------|--------|-----------|
|
||||
| Contract (OpenAPI) | Required | Required | Required |
|
||||
| OTel traces | Required | Required | Required |
|
||||
| Negative tests | Required | Required | Required |
|
||||
| Auth/AuthZ | N/A (system) | Required | N/A (self) |
|
||||
| Token issuance | N/A | N/A | Required |
|
||||
|
||||
---
|
||||
|
||||
### Phase 3: Signing & Attestation Services (P3)
|
||||
|
||||
**Timeline**: Sprint 5100.0010.* (Q2-Q3 2026)
|
||||
|
||||
**Services**:
|
||||
- **Signer** — Cryptographic signing service
|
||||
- **Attestor** — Attestation generation/verification
|
||||
- **ExportCenter.WebService** — Report export service
|
||||
- **Registry.TokenService** — OCI registry token service
|
||||
- **VulnExplorer.Api** — Vulnerability exploration API
|
||||
- **Graph.Api** — Graph query API
|
||||
|
||||
**Rationale**: These services have specialized contracts and lower traffic. They require careful security testing due to cryptographic operations.
|
||||
|
||||
**Test Requirements**:
|
||||
| Test Type | Signer | Attestor | Others |
|
||||
|-----------|--------|----------|--------|
|
||||
| Contract (OpenAPI) | Required | Required | Required |
|
||||
| OTel traces | Required | Required | Required |
|
||||
| Negative tests | Required | Required | Required |
|
||||
| Crypto validation | Required | Required | N/A |
|
||||
|
||||
---
|
||||
|
||||
### Phase 4: Orchestration Services (P4)
|
||||
|
||||
**Timeline**: Sprint 5100.0011.* (Q3 2026)
|
||||
|
||||
**Services**:
|
||||
- **Orchestrator** — Workflow orchestration
|
||||
|
||||
**Rationale**: Orchestrator is a meta-service that coordinates other services. Testing depends on other services being testable first.
|
||||
|
||||
---
|
||||
|
||||
## Test Coverage Targets
|
||||
|
||||
### Minimum Requirements (PR-Gating)
|
||||
|
||||
| Test Category | Min Coverage | Lane |
|
||||
|---------------|-------------|------|
|
||||
| Contract (OpenAPI) | 100% of public endpoints | Contract |
|
||||
| Negative (4xx errors) | 100% of error codes | Unit/Security |
|
||||
| Auth/AuthZ | 100% of protected endpoints | Security |
|
||||
|
||||
### Recommended (Quality Gate)
|
||||
|
||||
| Test Category | Target Coverage | Lane |
|
||||
|---------------|-----------------|------|
|
||||
| OTel traces | 80% of endpoints | Integration |
|
||||
| Tenant isolation | 100% of data endpoints | Security |
|
||||
| Performance baselines | Key endpoints | Performance |
|
||||
|
||||
---
|
||||
|
||||
## Implementation Checklist per Service
|
||||
|
||||
```markdown
|
||||
## <Service Name> WebService Tests
|
||||
|
||||
### Setup
|
||||
- [ ] Create `<Service>ApplicationFactory` (WebApplicationFactory)
|
||||
- [ ] Create `<Service>TestFixture` if custom setup needed
|
||||
- [ ] Add test project: `StellaOps.<Service>.WebService.Tests`
|
||||
- [ ] Add reference to `StellaOps.TestKit`
|
||||
|
||||
### Contract Tests
|
||||
- [ ] Extract OpenAPI schema snapshot (`Snapshots/<service>-v1.json`)
|
||||
- [ ] Add schema stability test
|
||||
- [ ] Add response contract tests for key endpoints
|
||||
|
||||
### OTel Tests
|
||||
- [ ] Add trace assertion tests for key endpoints
|
||||
- [ ] Verify required tags (tenant.id, request.id, http.route)
|
||||
|
||||
### Negative Tests
|
||||
- [ ] Malformed content type → 415
|
||||
- [ ] Oversized payload → 413
|
||||
- [ ] Method mismatch → 405
|
||||
- [ ] Missing required field → 400
|
||||
- [ ] Invalid field value → 400
|
||||
|
||||
### Auth Tests
|
||||
- [ ] Anonymous request → 401
|
||||
- [ ] Expired token → 401
|
||||
- [ ] Missing scope → 403
|
||||
- [ ] Cross-tenant access → 404/403
|
||||
|
||||
### CI Integration
|
||||
- [ ] Verify traits assigned: Contract, Security, Integration, Unit
|
||||
- [ ] PR passes all lanes
|
||||
- [ ] Add to TEST_COVERAGE_MATRIX.md
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Sprint Planning Template
|
||||
|
||||
When creating sprints for new service tests:
|
||||
|
||||
```markdown
|
||||
# Sprint 5100.XXXX.YYYY - <Service> WebService Tests
|
||||
|
||||
## Topic & Scope
|
||||
- Apply WebService test discipline to <Service>.WebService
|
||||
- Contract tests, OTel traces, negative tests, auth tests
|
||||
- **Working directory:** `src/<Module>/__Tests/StellaOps.<Module>.WebService.Tests`
|
||||
|
||||
## Delivery Tracker
|
||||
| # | Task ID | Status | Task Definition |
|
||||
|---|---------|--------|-----------------|
|
||||
| 1 | WEBSVC-XXXX-001 | TODO | Create <Service>ApplicationFactory |
|
||||
| 2 | WEBSVC-XXXX-002 | TODO | Add OpenAPI contract tests |
|
||||
| 3 | WEBSVC-XXXX-003 | TODO | Add OTel trace tests |
|
||||
| 4 | WEBSVC-XXXX-004 | TODO | Add negative tests (4xx) |
|
||||
| 5 | WEBSVC-XXXX-005 | TODO | Add auth/authz tests |
|
||||
| 6 | WEBSVC-XXXX-006 | TODO | Update TEST_COVERAGE_MATRIX.md |
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Success Metrics
|
||||
|
||||
| Metric | Target | Measurement |
|
||||
|--------|--------|-------------|
|
||||
| Services with contract tests | 100% | Count of services with OpenAPI snapshot tests |
|
||||
| Services with auth tests | 100% | Count of services with auth boundary tests |
|
||||
| Contract test failures in production | 0 | Breaking changes detected in staging |
|
||||
| Security test coverage | 100% of auth endpoints | Audit of protected routes vs tests |
|
||||
|
||||
---
|
||||
|
||||
## Risks & Mitigations
|
||||
|
||||
| Risk | Impact | Mitigation |
|
||||
|------|--------|------------|
|
||||
| Services lack OpenAPI spec | Cannot do contract testing | Generate spec via Swashbuckle/NSwag |
|
||||
| OTel not configured in service | Cannot verify traces | Add OTel middleware as prerequisite |
|
||||
| Auth disabled in test mode | False confidence | Test with auth enabled, use test tokens |
|
||||
| Test fixtures are slow | CI timeout | Share fixtures, use in-memory providers |
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- [WebService Test Discipline](./webservice-test-discipline.md)
|
||||
- [Test Coverage Matrix](./TEST_COVERAGE_MATRIX.md)
|
||||
- [CI Lane Filters](./ci-lane-filters.md)
|
||||
- [Testing Strategy Models](./testing-strategy-models.md)
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2025-06-30 · Sprint 5100.0007.0006*
|
||||
Reference in New Issue
Block a user