Add tests for SBOM generation determinism across multiple formats

- Created `StellaOps.TestKit.Tests` project for unit tests related to determinism.
- Implemented `DeterminismManifestTests` to validate deterministic output for canonical bytes and strings, file read/write operations, and error handling for invalid schema versions.
- Added `SbomDeterminismTests` to ensure identical inputs produce consistent SBOMs across SPDX 3.0.1 and CycloneDX 1.6/1.7 formats, including parallel execution tests.
- Updated project references in `StellaOps.Integration.Determinism` to include the new determinism testing library.
This commit is contained in:
master
2025-12-23 18:56:12 +02:00
committed by StellaOps Bot
parent 7ac70ece71
commit 491e883653
409 changed files with 23797 additions and 17779 deletions

View File

@@ -0,0 +1,425 @@
# Connector Fixture Discipline
This document defines the testing discipline for StellaOps Concelier and Excititor connectors. All connectors must follow these patterns to ensure consistent, deterministic, and offline-capable testing.
## Overview
Connector tests follow **Model C1 (Connector/External)** from the testing strategy:
1. **Fixture-based parser tests** — Raw upstream payload → normalized internal model (offline)
2. **Resilience tests** — Partial/bad input → deterministic failure classification
3. **Security tests** — URL allowlist, redirect handling, payload limits
4. **Live smoke tests** — Schema drift detection (opt-in, non-gating)
---
## 1. Directory Structure
Each connector test project follows this structure:
```
src/<Module>/__Tests/StellaOps.<Module>.Connector.<Source>.Tests/
├── StellaOps.<Module>.Connector.<Source>.Tests.csproj
├── Fixtures/
│ ├── <source>-typical.json # Typical advisory payload
│ ├── <source>-edge-<case>.json # Edge case payloads
│ ├── <source>-error-<type>.json # Malformed/invalid payloads
│ └── expected-<id>.json # Expected normalized output
├── <Source>/
│ ├── <Source>ParserTests.cs # Parser unit tests
│ ├── <Source>ConnectorTests.cs # Connector integration tests
│ └── <Source>ResilienceTests.cs # Resilience/security tests
└── Expected/
└── <source>-<id>.canonical.json # Canonical JSON snapshots
```
### Example: NVD Connector
```
src/Concelier/__Tests/StellaOps.Concelier.Connector.Nvd.Tests/
├── Fixtures/
│ ├── nvd-window-1.json
│ ├── nvd-window-2.json
│ ├── nvd-multipage-1.json
│ ├── nvd-multipage-2.json
│ ├── nvd-invalid-schema.json
│ └── expected-CVE-2024-0001.json
├── Nvd/
│ ├── NvdParserTests.cs
│ ├── NvdConnectorTests.cs
│ └── NvdConnectorHarnessTests.cs
└── Expected/
└── conflict-nvd.canonical.json
```
---
## 2. Fixture-Based Parser Tests
### Purpose
Test that the parser correctly transforms raw upstream payloads into normalized internal models without network access.
### Pattern
```csharp
using StellaOps.TestKit.Connectors;
public class NvdParserTests : ConnectorParserTestBase<NvdRawAdvisory, ConcelierAdvisory>
{
public NvdParserTests()
: base(new NvdParser(), "Nvd/Fixtures")
{
}
[Fact]
[Trait("Lane", "Unit")]
public async Task ParseTypicalAdvisory_ProducesExpectedModel()
{
// Arrange
var raw = await LoadFixture<NvdRawAdvisory>("nvd-window-1.json");
// Act
var result = Parser.Parse(raw);
// Assert
await AssertMatchesSnapshot(result, "expected-CVE-2024-0001.json");
}
[Theory]
[Trait("Lane", "Unit")]
[InlineData("nvd-multipage-1.json", "expected-multipage-1.json")]
[InlineData("nvd-multipage-2.json", "expected-multipage-2.json")]
public async Task ParseAllFixtures_ProducesExpectedModels(string input, string expected)
{
var raw = await LoadFixture<NvdRawAdvisory>(input);
var result = Parser.Parse(raw);
await AssertMatchesSnapshot(result, expected);
}
}
```
### Fixture Requirements
| Type | Naming Convention | Purpose |
|------|-------------------|---------|
| Typical | `<source>-typical.json` | Normal advisory with all common fields |
| Edge case | `<source>-edge-<case>.json` | Unusual but valid payloads |
| Error | `<source>-error-<type>.json` | Malformed/invalid payloads |
| Expected | `expected-<id>.json` | Expected normalized output |
| Canonical | `<source>-<id>.canonical.json` | Deterministic JSON snapshot |
### Minimum Coverage
Each connector must have fixtures for:
- [ ] At least 1 typical payload
- [ ] At least 2 edge cases (e.g., multi-vendor, unusual CVSS, missing optional fields)
- [ ] At least 2 error cases (e.g., missing required fields, invalid schema)
---
## 3. Resilience Tests
### Purpose
Verify that connectors handle malformed input gracefully with deterministic failure classification.
### Pattern
```csharp
public class NvdResilienceTests : ConnectorResilienceTestBase<NvdConnector>
{
public NvdResilienceTests()
: base(new NvdConnector(CreateTestHttpClient()))
{
}
[Fact]
[Trait("Lane", "Unit")]
public async Task MissingRequiredField_ReturnsParseError()
{
// Arrange
var payload = await LoadFixture("nvd-error-missing-cve-id.json");
// Act
var result = await Connector.ParseAsync(payload);
// Assert
Assert.False(result.IsSuccess);
Assert.Equal(ConnectorErrorKind.ParseError, result.Error.Kind);
Assert.Contains("cve_id", result.Error.Message, StringComparison.OrdinalIgnoreCase);
}
[Fact]
[Trait("Lane", "Unit")]
public async Task InvalidDateFormat_ReturnsParseError()
{
var payload = await LoadFixture("nvd-error-invalid-date.json");
var result = await Connector.ParseAsync(payload);
Assert.False(result.IsSuccess);
Assert.Equal(ConnectorErrorKind.ParseError, result.Error.Kind);
}
[Fact]
[Trait("Lane", "Unit")]
public async Task UnexpectedEnumValue_LogsWarningAndContinues()
{
var payload = await LoadFixture("nvd-edge-unknown-severity.json");
var result = await Connector.ParseAsync(payload);
Assert.True(result.IsSuccess);
Assert.Contains(result.Warnings, w => w.Contains("unknown severity"));
}
}
```
### Required Test Cases
| Case | Expected Behavior | Trait |
|------|-------------------|-------|
| Missing required field | `ConnectorErrorKind.ParseError` | Unit |
| Invalid date format | `ConnectorErrorKind.ParseError` | Unit |
| Invalid JSON structure | `ConnectorErrorKind.ParseError` | Unit |
| Unknown enum value | Warning logged, continues | Unit |
| Empty response | `ConnectorErrorKind.EmptyResponse` | Unit |
| Truncated payload | `ConnectorErrorKind.ParseError` | Unit |
---
## 4. Security Tests
### Purpose
Verify that connectors enforce security boundaries for network operations.
### Pattern
```csharp
public class NvdSecurityTests : ConnectorSecurityTestBase<NvdConnector>
{
[Fact]
[Trait("Lane", "Security")]
public async Task UrlOutsideAllowlist_RejectsRequest()
{
// Arrange
var connector = CreateConnector(allowedHosts: ["services.nvd.nist.gov"]);
// Act & Assert
await Assert.ThrowsAsync<SecurityException>(
() => connector.FetchAsync("https://evil.example.com/api"));
}
[Fact]
[Trait("Lane", "Security")]
public async Task RedirectToDisallowedHost_RejectsRequest()
{
var handler = CreateMockHandler(redirectTo: "https://evil.example.com");
var connector = CreateConnector(handler, allowedHosts: ["services.nvd.nist.gov"]);
await Assert.ThrowsAsync<SecurityException>(
() => connector.FetchAsync("https://services.nvd.nist.gov/api"));
}
[Fact]
[Trait("Lane", "Security")]
public async Task PayloadExceedsMaxSize_RejectsPayload()
{
var handler = CreateMockHandler(responseSize: 100_000_001); // 100MB
var connector = CreateConnector(handler, maxPayloadBytes: 100_000_000);
await Assert.ThrowsAsync<PayloadTooLargeException>(
() => connector.FetchAsync("https://services.nvd.nist.gov/api"));
}
[Fact]
[Trait("Lane", "Security")]
public async Task DecompressionBomb_RejectsPayload()
{
// 1KB compressed, 1GB decompressed
var handler = CreateMockHandler(compressedBomb: true);
var connector = CreateConnector(handler);
await Assert.ThrowsAsync<PayloadTooLargeException>(
() => connector.FetchAsync("https://services.nvd.nist.gov/api"));
}
}
```
### Required Security Tests
| Test | Purpose | Trait |
|------|---------|-------|
| URL allowlist | Block requests to unauthorized hosts | Security |
| Redirect validation | Block redirects to unauthorized hosts | Security |
| Max payload size | Reject oversized responses | Security |
| Decompression bomb | Reject zip bombs | Security |
| Rate limiting | Respect upstream rate limits | Security |
---
## 5. Live Smoke Tests (Opt-In)
### Purpose
Detect upstream schema drift by comparing live responses against known fixtures.
### Pattern
```csharp
public class NvdLiveTests : ConnectorLiveTestBase<NvdConnector>
{
[Fact]
[Trait("Lane", "Live")]
[Trait("Category", "SchemaDrift")]
public async Task LiveSchema_MatchesFixtureSchema()
{
// Skip if not in Live lane
Skip.IfNot(IsLiveLaneEnabled());
// Fetch live response
var live = await Connector.FetchLatestAsync();
// Compare schema (not values) against fixture
var fixture = await LoadFixture<NvdResponse>("nvd-typical.json");
AssertSchemaMatches(live, fixture);
}
[Fact]
[Trait("Lane", "Live")]
public async Task LiveFetch_ReturnsValidAdvisories()
{
Skip.IfNot(IsLiveLaneEnabled());
var result = await Connector.FetchLatestAsync();
Assert.True(result.IsSuccess);
Assert.NotEmpty(result.Value.Advisories);
}
}
```
### Configuration
Live tests are:
- **Never PR-gating** — run only in scheduled/nightly jobs
- **Opt-in** — require explicit `LIVE_TESTS_ENABLED=true` environment variable
- **Alerting** — schema drift triggers notification, not failure
---
## 6. Fixture Updater
### Purpose
Refresh fixtures from live sources when upstream schemas change intentionally.
### Usage
```bash
# Update all fixtures for NVD connector
dotnet run --project tools/FixtureUpdater -- \
--connector nvd \
--output src/Concelier/__Tests/StellaOps.Concelier.Connector.Nvd.Tests/Nvd/Fixtures/
# Update specific fixture
dotnet run --project tools/FixtureUpdater -- \
--connector nvd \
--cve CVE-2024-0001 \
--output src/Concelier/__Tests/.../Fixtures/nvd-CVE-2024-0001.json
# Dry-run mode (show diff without writing)
dotnet run --project tools/FixtureUpdater -- \
--connector nvd \
--dry-run
```
### Workflow
1. Live test detects schema drift
2. CI creates draft PR with fixture update
3. Developer reviews diff for intentional vs accidental changes
4. If intentional: update parser and merge
5. If accidental: investigate upstream API issue
---
## 7. Test Traits and CI Integration
### Trait Assignment
| Test Category | Trait | Lane | PR-Gating |
|---------------|-------|------|-----------|
| Parser tests | `[Trait("Lane", "Unit")]` | Unit | Yes |
| Resilience tests | `[Trait("Lane", "Unit")]` | Unit | Yes |
| Security tests | `[Trait("Lane", "Security")]` | Security | Yes |
| Live tests | `[Trait("Lane", "Live")]` | Live | No |
### Running Tests
```bash
# Run all connector unit tests
dotnet test --filter "Lane=Unit" src/Concelier/__Tests/
# Run security tests
dotnet test --filter "Lane=Security" src/Concelier/__Tests/
# Run live tests (requires LIVE_TESTS_ENABLED=true)
LIVE_TESTS_ENABLED=true dotnet test --filter "Lane=Live" src/Concelier/__Tests/
```
---
## 8. Connector Inventory
### Concelier Connectors (Advisory Sources)
| Connector | Fixtures | Parser Tests | Resilience | Security | Live |
|-----------|----------|--------------|------------|----------|------|
| NVD | ✅ | ✅ | ✅ | ⬜ | ⬜ |
| OSV | ✅ | ✅ | ⬜ | ⬜ | ⬜ |
| GHSA | ✅ | ✅ | ⬜ | ⬜ | ⬜ |
| CVE | ✅ | ✅ | ⬜ | ⬜ | ⬜ |
| KEV | ✅ | ✅ | ⬜ | ⬜ | ⬜ |
| EPSS | ✅ | ✅ | ⬜ | ⬜ | ⬜ |
| Distro.Alpine | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ |
| Distro.Debian | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ |
| Distro.RedHat | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ |
| Distro.Suse | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ |
| Distro.Ubuntu | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ |
| Vndr.Adobe | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ |
| Vndr.Apple | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ |
| Vndr.Cisco | ✅ | ✅ | ⬜ | ⬜ | ⬜ |
| Vndr.Msrc | ✅ | ✅ | ⬜ | ⬜ | ⬜ |
| Vndr.Oracle | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ |
| Vndr.Vmware | ✅ | ✅ | ⬜ | ⬜ | ⬜ |
| Cert.Bund | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ |
| Cert.Cc | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ |
| Cert.Fr | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ |
| ICS.Cisa | ✅ | ✅ | ⬜ | ⬜ | ⬜ |
### Excititor Connectors (VEX Sources)
| Connector | Fixtures | Parser Tests | Resilience | Security | Live |
|-----------|----------|--------------|------------|----------|------|
| OpenVEX | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ |
| CSAF/VEX | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ |
---
## References
- [ConnectorHttpFixture](../../src/__Libraries/StellaOps.TestKit/Connectors/ConnectorHttpFixture.cs)
- [ConnectorTestBase](../../src/__Libraries/StellaOps.TestKit/Connectors/ConnectorTestBase.cs)
- [ConnectorResilienceTestBase](../../src/__Libraries/StellaOps.TestKit/Connectors/ConnectorResilienceTestBase.cs)
- [FixtureUpdater](../../src/__Libraries/StellaOps.TestKit/Connectors/FixtureUpdater.cs)
- [Testing Strategy Models](./testing-strategy-models.md)
- [CI Lane Filters](./ci-lane-filters.md)
---
*Last updated: 2025-06-30 · Sprint 5100.0007.0005*

View File

@@ -0,0 +1,362 @@
# Determinism Verification Guide
**Sprint:** 5100.0007.0003 (Epic B)
**Last Updated:** 2025-12-23
## Overview
StellaOps enforces deterministic artifact generation across all exported formats. This ensures:
1. **Reproducibility**: Given the same inputs, outputs are byte-for-byte identical
2. **Auditability**: Hash verification proves artifact integrity
3. **Compliance**: Regulated environments can replay and verify builds
4. **CI Gating**: Drift detection prevents unintended changes
## Supported Artifact Types
| Type | Format(s) | Test File |
|------|-----------|-----------|
| SBOM | SPDX 3.0.1, CycloneDX 1.6, CycloneDX 1.7 | `SbomDeterminismTests.cs` |
| VEX | OpenVEX, CSAF 2.0 | `VexDeterminismTests.cs` |
| Policy Verdicts | JSON | `PolicyDeterminismTests.cs` |
| Evidence Bundles | JSON, DSSE, in-toto | `EvidenceBundleDeterminismTests.cs` |
| AirGap Bundles | NDJSON | `AirGapBundleDeterminismTests.cs` |
| Advisory Normalization | Canonical JSON | `IngestionDeterminismTests.cs` |
## Determinism Manifest Format
Every deterministic artifact can produce a manifest describing its content hash and generation context.
### Schema (v1.0)
```json
{
"schemaVersion": "1.0",
"artifact": {
"type": "sbom | vex | policy-verdict | evidence-bundle | airgap-bundle",
"name": "artifact-identifier",
"version": "1.0.0",
"format": "SPDX 3.0.1 | CycloneDX 1.6 | OpenVEX | CSAF 2.0 | ..."
},
"canonicalHash": {
"algorithm": "SHA-256",
"value": "abc123..."
},
"toolchain": {
"platform": ".NET 10.0",
"components": [
{ "name": "StellaOps.Scanner", "version": "1.0.0" }
]
},
"inputs": {
"feedSnapshotHash": "def456...",
"policyManifestHash": "ghi789...",
"configHash": "jkl012..."
},
"generatedAt": "2025-12-23T18:00:00Z"
}
```
### Field Descriptions
| Field | Description |
|-------|-------------|
| `schemaVersion` | Manifest schema version (currently `1.0`) |
| `artifact.type` | Category of the artifact |
| `artifact.name` | Identifier for the artifact |
| `artifact.version` | Version of the artifact (if applicable) |
| `artifact.format` | Specific format/spec version |
| `canonicalHash.algorithm` | Hash algorithm (always `SHA-256`) |
| `canonicalHash.value` | Lowercase hex hash of canonical bytes |
| `toolchain.platform` | Runtime platform |
| `toolchain.components` | List of generating components with versions |
| `inputs` | Hashes of input artifacts (feed snapshots, policies, etc.) |
| `generatedAt` | ISO-8601 UTC timestamp of generation |
## Creating a Determinism Manifest
Use `DeterminismManifestWriter` from `StellaOps.Testing.Determinism`:
```csharp
using StellaOps.Testing.Determinism;
// Generate artifact bytes
var sbomBytes = GenerateSbom(input, frozenTime);
// Create artifact info
var artifactInfo = new ArtifactInfo
{
Type = "sbom",
Name = "my-container-sbom",
Version = "1.0.0",
Format = "CycloneDX 1.6"
};
// Create toolchain info
var toolchain = new ToolchainInfo
{
Platform = ".NET 10.0",
Components = new[]
{
new ComponentInfo { Name = "StellaOps.Scanner", Version = "1.0.0" }
}
};
// Create manifest
var manifest = DeterminismManifestWriter.CreateManifest(
sbomBytes,
artifactInfo,
toolchain);
// Save manifest
DeterminismManifestWriter.Save(manifest, "determinism.json");
```
## Reading and Verifying Manifests
```csharp
// Load manifest
var manifest = DeterminismManifestReader.Load("determinism.json");
// Verify artifact bytes match manifest hash
var currentBytes = File.ReadAllBytes("artifact.json");
var isValid = DeterminismManifestReader.Verify(manifest, currentBytes);
if (!isValid)
{
throw new DeterminismDriftException(
$"Artifact hash mismatch. Expected: {manifest.CanonicalHash.Value}");
}
```
## Determinism Rules
### 1. Canonical JSON Serialization
All JSON output must use canonical serialization via `StellaOps.Canonical.Json`:
```csharp
using StellaOps.Canonical.Json;
var json = CanonJson.Serialize(myObject);
var hash = CanonJson.Sha256Hex(Encoding.UTF8.GetBytes(json));
```
Rules:
- Keys sorted lexicographically
- No trailing whitespace
- Unix line endings (`\n`)
- No BOM
- UTF-8 encoding
### 2. Frozen Timestamps
All timestamps must be provided externally or use `DeterministicTime`:
```csharp
// ❌ BAD - Non-deterministic
var timestamp = DateTimeOffset.UtcNow;
// ✅ GOOD - Deterministic
var timestamp = frozenTime; // Passed as parameter
```
### 3. Deterministic IDs
UUIDs and IDs must be derived from content, not random:
```csharp
// ❌ BAD - Random UUID
var id = Guid.NewGuid();
// ✅ GOOD - Content-derived ID
var seed = $"{input.Name}:{input.Version}:{timestamp:O}";
var hash = CanonJson.Sha256Hex(Encoding.UTF8.GetBytes(seed));
var id = new Guid(Convert.FromHexString(hash[..32]));
```
### 4. Stable Ordering
Collections must be sorted before serialization:
```csharp
// ❌ BAD - Non-deterministic order
var items = dictionary.Values;
// ✅ GOOD - Sorted order
var items = dictionary.Values
.OrderBy(v => v.Key, StringComparer.Ordinal);
```
### 5. Parallel Safety
Determinism must hold under parallel execution:
```csharp
var tasks = Enumerable.Range(0, 20)
.Select(_ => Task.Run(() => GenerateArtifact(input, frozenTime)))
.ToArray();
var results = await Task.WhenAll(tasks);
results.Should().AllBe(results[0]); // All identical
```
## CI Integration
### PR Merge Gate
The determinism gate runs on PR merge:
```yaml
# .gitea/workflows/determinism-gate.yaml
name: Determinism Gate
on:
pull_request:
types: [synchronize, ready_for_review]
jobs:
determinism:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-dotnet@v4
with:
dotnet-version: '10.0.x'
- name: Run Determinism Tests
run: |
dotnet test tests/integration/StellaOps.Integration.Determinism \
--logger "trx;LogFileName=determinism.trx"
- name: Generate Determinism Manifest
run: |
dotnet run --project tools/DeterminismManifestGenerator \
--output determinism.json
- name: Upload Determinism Artifact
uses: actions/upload-artifact@v4
with:
name: determinism-manifest
path: determinism.json
```
### Baseline Storage
Determinism baselines are stored as CI artifacts:
```
ci-artifacts/
determinism/
baseline/
sbom-spdx-3.0.1.json
sbom-cyclonedx-1.6.json
sbom-cyclonedx-1.7.json
vex-openvex.json
vex-csaf.json
policy-verdict.json
evidence-bundle.json
airgap-bundle.json
```
### Drift Detection
When a PR changes artifact output:
1. CI compares new manifest hash against baseline
2. If different, CI fails with diff report
3. Developer must either:
- Fix the regression (restore determinism)
- Update the baseline (if change is intentional)
### Baseline Update Process
To intentionally update a baseline:
```bash
# 1. Run determinism tests to generate new manifests
dotnet test tests/integration/StellaOps.Integration.Determinism
# 2. Update baseline files
cp determinism/*.json ci-artifacts/determinism/baseline/
# 3. Commit with explicit message
git add ci-artifacts/determinism/baseline/
git commit -m "chore(determinism): update baselines for [reason]
Breaking: [explain what changed]
Justification: [explain why this is correct]"
```
## Replay Verification
To verify an artifact was produced deterministically:
```bash
# 1. Get the manifest
curl -O https://releases.stellaops.io/v1.0.0/sbom.determinism.json
# 2. Get the artifact
curl -O https://releases.stellaops.io/v1.0.0/sbom.cdx.json
# 3. Verify
dotnet run --project tools/DeterminismVerifier \
--manifest sbom.determinism.json \
--artifact sbom.cdx.json
```
Output:
```
Determinism Verification
========================
Artifact: sbom.cdx.json
Manifest: sbom.determinism.json
Expected Hash: abc123...
Actual Hash: abc123...
Status: ✅ VERIFIED
```
## Test Files Reference
All determinism tests are in `tests/integration/StellaOps.Integration.Determinism/`:
| File | Tests | Description |
|------|-------|-------------|
| `DeterminismValidationTests.cs` | 16 | Manifest format and reader/writer |
| `SbomDeterminismTests.cs` | 14 | SPDX 3.0.1, CycloneDX 1.6/1.7 |
| `VexDeterminismTests.cs` | 17 | OpenVEX, CSAF 2.0 |
| `PolicyDeterminismTests.cs` | 18 | Policy verdict artifacts |
| `EvidenceBundleDeterminismTests.cs` | 15 | DSSE, in-toto attestations |
| `AirGapBundleDeterminismTests.cs` | 14 | NDJSON bundles, manifests |
| `IngestionDeterminismTests.cs` | 17 | NVD/OSV/GHSA/CSAF normalization |
## Troubleshooting
### Hash Mismatch
If you see a hash mismatch:
1. **Check timestamps**: Ensure frozen time is used
2. **Check ordering**: Ensure all collections are sorted
3. **Check IDs**: Ensure IDs are content-derived
4. **Check encoding**: Ensure UTF-8 without BOM
### Flaky Tests
If determinism tests are flaky:
1. **Check parallelism**: Ensure no shared mutable state
2. **Check time zones**: Use UTC explicitly
3. **Check random sources**: Remove all random number generation
4. **Check hash inputs**: Ensure all inputs are captured
### CI Failures
If CI determinism gate fails:
1. Compare the diff between expected and actual
2. Identify which field changed
3. Track back to the code change that caused it
4. Either fix the regression or update baseline with justification
## Related Documentation
- [Testing Strategy Models](testing-strategy-models.md) - Overview of testing models
- [Canonical JSON Specification](../11_DATA_SCHEMAS.md#canonical-json) - JSON serialization rules
- [CI/CD Workflows](../modules/devops/architecture.md) - CI pipeline details
- [Evidence Bundle Schema](../modules/evidence-locker/architecture.md) - Bundle format reference

View File

@@ -0,0 +1,267 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://stella-ops.org/schemas/determinism-manifest/v1.json",
"title": "StellaOps Determinism Manifest",
"description": "Manifest tracking artifact reproducibility with canonical bytes hash, version stamps, and toolchain information",
"type": "object",
"required": [
"schemaVersion",
"artifact",
"canonicalHash",
"toolchain",
"generatedAt"
],
"properties": {
"schemaVersion": {
"type": "string",
"const": "1.0",
"description": "Version of this manifest schema"
},
"artifact": {
"type": "object",
"description": "Artifact being tracked for determinism",
"required": ["type", "name", "version"],
"properties": {
"type": {
"type": "string",
"enum": [
"sbom",
"vex",
"csaf",
"verdict",
"evidence-bundle",
"airgap-bundle",
"advisory-normalized",
"attestation",
"other"
],
"description": "Type of artifact"
},
"name": {
"type": "string",
"description": "Artifact identifier or name",
"minLength": 1
},
"version": {
"type": "string",
"description": "Artifact version or timestamp",
"minLength": 1
},
"format": {
"type": "string",
"description": "Artifact format (e.g., 'SPDX 3.0.1', 'CycloneDX 1.6', 'OpenVEX')",
"examples": ["SPDX 3.0.1", "CycloneDX 1.6", "OpenVEX", "CSAF 2.0"]
},
"metadata": {
"type": "object",
"description": "Additional artifact-specific metadata",
"additionalProperties": true
}
}
},
"canonicalHash": {
"type": "object",
"description": "Hash of the canonical representation of the artifact",
"required": ["algorithm", "value", "encoding"],
"properties": {
"algorithm": {
"type": "string",
"enum": ["SHA-256", "SHA-384", "SHA-512"],
"description": "Hash algorithm used"
},
"value": {
"type": "string",
"description": "Hex-encoded hash value",
"pattern": "^[0-9a-f]{64,128}$"
},
"encoding": {
"type": "string",
"enum": ["hex", "base64"],
"description": "Encoding of the hash value"
}
}
},
"inputs": {
"type": "object",
"description": "Version stamps of all inputs used to generate the artifact",
"properties": {
"feedSnapshotHash": {
"type": "string",
"description": "SHA-256 hash of the vulnerability feed snapshot used",
"pattern": "^[0-9a-f]{64}$"
},
"policyManifestHash": {
"type": "string",
"description": "SHA-256 hash of the policy manifest used",
"pattern": "^[0-9a-f]{64}$"
},
"sourceCodeHash": {
"type": "string",
"description": "Git commit SHA or source code hash",
"pattern": "^[0-9a-f]{40,64}$"
},
"dependencyLockfileHash": {
"type": "string",
"description": "Hash of dependency lockfile (e.g., package-lock.json, Cargo.lock)",
"pattern": "^[0-9a-f]{64}$"
},
"baseImageDigest": {
"type": "string",
"description": "Container base image digest (sha256:...)",
"pattern": "^sha256:[0-9a-f]{64}$"
},
"vexDocumentHashes": {
"type": "array",
"description": "Hashes of all VEX documents used as input",
"items": {
"type": "string",
"pattern": "^[0-9a-f]{64}$"
}
},
"custom": {
"type": "object",
"description": "Custom input hashes specific to artifact type",
"additionalProperties": {
"type": "string"
}
}
},
"additionalProperties": false
},
"toolchain": {
"type": "object",
"description": "Toolchain version information",
"required": ["platform", "components"],
"properties": {
"platform": {
"type": "string",
"description": "Runtime platform (e.g., '.NET 10.0', 'Node.js 20.0')",
"examples": [".NET 10.0.0", "Node.js 20.11.0", "Python 3.12.1"]
},
"components": {
"type": "array",
"description": "Toolchain component versions",
"items": {
"type": "object",
"required": ["name", "version"],
"properties": {
"name": {
"type": "string",
"description": "Component name",
"examples": ["StellaOps.Scanner", "StellaOps.Policy.Engine", "CycloneDX Generator"]
},
"version": {
"type": "string",
"description": "Semantic version or git SHA",
"examples": ["1.2.3", "2.0.0-beta.1", "abc123def"]
},
"hash": {
"type": "string",
"description": "Optional: SHA-256 hash of the component binary",
"pattern": "^[0-9a-f]{64}$"
}
}
}
},
"compiler": {
"type": "object",
"description": "Compiler information if applicable",
"properties": {
"name": {
"type": "string",
"description": "Compiler name (e.g., 'Roslyn', 'rustc')"
},
"version": {
"type": "string",
"description": "Compiler version"
}
}
}
}
},
"generatedAt": {
"type": "string",
"format": "date-time",
"description": "UTC timestamp when artifact was generated (ISO 8601)",
"examples": ["2025-12-23T17:45:00Z"]
},
"reproducibility": {
"type": "object",
"description": "Reproducibility metadata",
"properties": {
"deterministicSeed": {
"type": "integer",
"description": "Deterministic random seed if used",
"minimum": 0
},
"clockFixed": {
"type": "boolean",
"description": "Whether system clock was fixed during generation"
},
"orderingGuarantee": {
"type": "string",
"enum": ["stable", "sorted", "insertion", "unspecified"],
"description": "Ordering guarantee for collections in output"
},
"normalizationRules": {
"type": "array",
"description": "Normalization rules applied (e.g., 'UTF-8', 'LF line endings', 'no whitespace')",
"items": {
"type": "string"
},
"examples": [
["UTF-8 encoding", "LF line endings", "sorted JSON keys", "no trailing whitespace"]
]
}
}
},
"verification": {
"type": "object",
"description": "Verification instructions for reproducing the artifact",
"properties": {
"command": {
"type": "string",
"description": "Command to regenerate the artifact",
"examples": ["dotnet run --project Scanner -- scan container alpine:3.18"]
},
"expectedHash": {
"type": "string",
"description": "Expected SHA-256 hash after reproduction",
"pattern": "^[0-9a-f]{64}$"
},
"baseline": {
"type": "string",
"description": "Baseline manifest file path for regression testing",
"examples": ["tests/baselines/sbom-alpine-3.18.determinism.json"]
}
}
},
"signatures": {
"type": "array",
"description": "Optional cryptographic signatures of this manifest",
"items": {
"type": "object",
"required": ["algorithm", "keyId", "signature"],
"properties": {
"algorithm": {
"type": "string",
"description": "Signature algorithm (e.g., 'ES256', 'RS256')"
},
"keyId": {
"type": "string",
"description": "Key identifier used for signing"
},
"signature": {
"type": "string",
"description": "Base64-encoded signature"
},
"timestamp": {
"type": "string",
"format": "date-time",
"description": "UTC timestamp when signature was created"
}
}
}
}
}
}

View File

@@ -0,0 +1,613 @@
# StellaOps.TestKit Usage Guide
**Version:** 1.0
**Status:** Pilot Release (Wave 4 Complete)
**Audience:** StellaOps developers writing unit, integration, and contract tests
---
## Overview
`StellaOps.TestKit` provides deterministic testing infrastructure for StellaOps modules. It eliminates flaky tests, provides reproducible test primitives, and standardizes fixtures for integration testing.
### Key Features
- **Deterministic Time**: Freeze and advance time for reproducible tests
- **Deterministic Random**: Seeded random number generation
- **Canonical JSON Assertions**: SHA-256 hash verification for determinism
- **Snapshot Testing**: Golden master regression testing
- **PostgreSQL Fixture**: Testcontainers-based PostgreSQL 16 for integration tests
- **Valkey Fixture**: Redis-compatible caching tests
- **HTTP Fixture**: In-memory API contract testing
- **OpenTelemetry Capture**: Trace and span assertion helpers
- **Test Categories**: Standardized trait constants for CI filtering
---
## Installation
Add `StellaOps.TestKit` as a project reference to your test project:
```xml
<ItemGroup>
<ProjectReference Include="../../../__Libraries/StellaOps.TestKit/StellaOps.TestKit.csproj" />
</ItemGroup>
```
---
## Quick Start Examples
### 1. Deterministic Time
Eliminate flaky tests caused by time-dependent logic:
```csharp
using StellaOps.TestKit.Deterministic;
using Xunit;
[Fact]
public void Test_ExpirationLogic()
{
// Arrange: Fix time at a known UTC timestamp
using var time = new DeterministicTime(new DateTime(2026, 1, 15, 10, 30, 0, DateTimeKind.Utc));
var expiresAt = time.UtcNow.AddHours(24);
// Act: Advance time to just before expiration
time.Advance(TimeSpan.FromHours(23));
Assert.False(time.UtcNow > expiresAt);
// Advance past expiration
time.Advance(TimeSpan.FromHours(2));
Assert.True(time.UtcNow > expiresAt);
}
```
**API Reference:**
- `DeterministicTime(DateTime initialUtc)` - Create with fixed start time
- `UtcNow` - Get current deterministic time
- `Advance(TimeSpan duration)` - Move time forward
- `SetTo(DateTime newUtc)` - Jump to specific time
---
### 2. Deterministic Random
Reproducible random sequences for property tests and fuzzing:
```csharp
using StellaOps.TestKit.Deterministic;
[Fact]
public void Test_RandomIdGeneration()
{
// Arrange: Same seed produces same sequence
var random1 = new DeterministicRandom(seed: 42);
var random2 = new DeterministicRandom(seed: 42);
// Act
var guid1 = random1.NextGuid();
var guid2 = random2.NextGuid();
// Assert: Reproducible GUIDs
Assert.Equal(guid1, guid2);
}
[Fact]
public void Test_Shuffling()
{
var random = new DeterministicRandom(seed: 100);
var array = new[] { 1, 2, 3, 4, 5 };
random.Shuffle(array);
// Deterministic shuffle order
Assert.NotEqual(new[] { 1, 2, 3, 4, 5 }, array);
}
```
**API Reference:**
- `DeterministicRandom(int seed)` - Create with seed
- `NextGuid()` - Generate deterministic GUID
- `NextString(int length)` - Generate alphanumeric string
- `NextInt(int min, int max)` - Generate integer in range
- `Shuffle<T>(T[] array)` - Fisher-Yates shuffle
---
### 3. Canonical JSON Assertions
Verify JSON determinism for SBOM, VEX, and attestation outputs:
```csharp
using StellaOps.TestKit.Assertions;
[Fact]
public void Test_SbomDeterminism()
{
var sbom = new
{
SpdxVersion = "SPDX-3.0.1",
Name = "MySbom",
Packages = new[] { new { Name = "Pkg1", Version = "1.0" } }
};
// Verify deterministic serialization
CanonicalJsonAssert.IsDeterministic(sbom, iterations: 100);
// Verify expected hash (golden master)
var expectedHash = "abc123..."; // Precomputed SHA-256
CanonicalJsonAssert.HasExpectedHash(sbom, expectedHash);
}
[Fact]
public void Test_JsonPropertyExists()
{
var vex = new
{
Document = new { Id = "VEX-2026-001" },
Statements = new[] { new { Vulnerability = "CVE-2026-1234" } }
};
// Deep property verification
CanonicalJsonAssert.ContainsProperty(vex, "Document.Id", "VEX-2026-001");
CanonicalJsonAssert.ContainsProperty(vex, "Statements[0].Vulnerability", "CVE-2026-1234");
}
```
**API Reference:**
- `IsDeterministic<T>(T value, int iterations)` - Verify N serializations match
- `HasExpectedHash<T>(T value, string expectedSha256Hex)` - Verify SHA-256 hash
- `ComputeCanonicalHash<T>(T value)` - Compute hash for golden master
- `AreCanonicallyEqual<T>(T expected, T actual)` - Compare canonical JSON
- `ContainsProperty<T>(T value, string propertyPath, object expectedValue)` - Deep search
---
### 4. Snapshot Testing
Golden master regression testing for complex outputs:
```csharp
using StellaOps.TestKit.Assertions;
[Fact, Trait("Category", TestCategories.Snapshot)]
public void Test_SbomGeneration()
{
var sbom = GenerateSbom(); // Your SBOM generation logic
// Snapshot will be stored in Snapshots/TestSbomGeneration.json
SnapshotAssert.MatchesSnapshot(sbom, "TestSbomGeneration");
}
// Update snapshots when intentional changes occur:
// UPDATE_SNAPSHOTS=1 dotnet test
```
**Text and Binary Snapshots:**
```csharp
[Fact]
public void Test_LicenseText()
{
var licenseText = GenerateLicenseNotice();
SnapshotAssert.MatchesTextSnapshot(licenseText, "LicenseNotice");
}
[Fact]
public void Test_SignatureBytes()
{
var signature = SignDocument(document);
SnapshotAssert.MatchesBinarySnapshot(signature, "DocumentSignature");
}
```
**API Reference:**
- `MatchesSnapshot<T>(T value, string snapshotName)` - JSON snapshot
- `MatchesTextSnapshot(string value, string snapshotName)` - Text snapshot
- `MatchesBinarySnapshot(byte[] value, string snapshotName)` - Binary snapshot
- Environment variable: `UPDATE_SNAPSHOTS=1` to update baselines
---
### 5. PostgreSQL Fixture
Testcontainers-based PostgreSQL 16 for integration tests:
```csharp
using StellaOps.TestKit.Fixtures;
using Xunit;
public class DatabaseTests : IClassFixture<PostgresFixture>
{
private readonly PostgresFixture _fixture;
public DatabaseTests(PostgresFixture fixture)
{
_fixture = fixture;
}
[Fact, Trait("Category", TestCategories.Integration)]
public async Task Test_DatabaseOperations()
{
// Use _fixture.ConnectionString to connect
using var connection = new NpgsqlConnection(_fixture.ConnectionString);
await connection.OpenAsync();
// Run migrations
await _fixture.RunMigrationsAsync(connection);
// Test database operations
var result = await connection.QueryAsync("SELECT version()");
Assert.NotEmpty(result);
}
}
```
**API Reference:**
- `PostgresFixture` - xUnit class fixture
- `ConnectionString` - PostgreSQL connection string
- `RunMigrationsAsync(DbConnection)` - Apply migrations
- Requires Docker running locally
---
### 6. Valkey Fixture
Redis-compatible caching for integration tests:
```csharp
using StellaOps.TestKit.Fixtures;
public class CacheTests : IClassFixture<ValkeyFixture>
{
private readonly ValkeyFixture _fixture;
[Fact, Trait("Category", TestCategories.Integration)]
public async Task Test_CachingLogic()
{
var connection = await ConnectionMultiplexer.Connect(_fixture.ConnectionString);
var db = connection.GetDatabase();
await db.StringSetAsync("key", "value");
var result = await db.StringGetAsync("key");
Assert.Equal("value", result.ToString());
}
}
```
**API Reference:**
- `ValkeyFixture` - xUnit class fixture
- `ConnectionString` - Redis connection string (host:port)
- `Host`, `Port` - Connection details
- Uses `redis:7-alpine` image (Valkey-compatible)
---
### 7. HTTP Fixture Server
In-memory API contract testing:
```csharp
using StellaOps.TestKit.Fixtures;
public class ApiTests : IClassFixture<HttpFixtureServer<Program>>
{
private readonly HttpClient _client;
public ApiTests(HttpFixtureServer<Program> fixture)
{
_client = fixture.CreateClient();
}
[Fact, Trait("Category", TestCategories.Contract)]
public async Task Test_HealthEndpoint()
{
var response = await _client.GetAsync("/health");
response.EnsureSuccessStatusCode();
var body = await response.Content.ReadAsStringAsync();
Assert.Contains("healthy", body);
}
}
```
**HTTP Message Handler Stub (Hermetic Tests):**
```csharp
[Fact]
public async Task Test_ExternalApiCall()
{
var handler = new HttpMessageHandlerStub()
.WhenRequest("https://api.example.com/data", HttpStatusCode.OK, "{\"status\":\"ok\"}");
var httpClient = new HttpClient(handler);
var response = await httpClient.GetAsync("https://api.example.com/data");
Assert.Equal(HttpStatusCode.OK, response.StatusCode);
}
```
**API Reference:**
- `HttpFixtureServer<TProgram>` - WebApplicationFactory wrapper
- `CreateClient()` - Get HttpClient for test server
- `HttpMessageHandlerStub` - Stub external HTTP dependencies
- `WhenRequest(url, statusCode, content)` - Configure stub responses
---
### 8. OpenTelemetry Capture
Trace and span assertion helpers:
```csharp
using StellaOps.TestKit.Observability;
[Fact]
public async Task Test_TracingBehavior()
{
using var capture = new OtelCapture();
// Execute code that emits traces
await MyService.DoWorkAsync();
// Assert traces
capture.AssertHasSpan("MyService.DoWork");
capture.AssertHasTag("user_id", "123");
capture.AssertSpanCount(expectedCount: 3);
// Verify parent-child hierarchy
capture.AssertHierarchy("ParentSpan", "ChildSpan");
}
```
**API Reference:**
- `OtelCapture(string? activitySourceName = null)` - Create capture
- `AssertHasSpan(string spanName)` - Verify span exists
- `AssertHasTag(string tagKey, string expectedValue)` - Verify tag
- `AssertSpanCount(int expectedCount)` - Verify span count
- `AssertHierarchy(string parentSpanName, string childSpanName)` - Verify parent-child
- `CapturedActivities` - Get all captured spans
---
### 9. Test Categories
Standardized trait constants for CI lane filtering:
```csharp
using StellaOps.TestKit;
[Fact, Trait("Category", TestCategories.Unit)]
public void FastUnitTest() { }
[Fact, Trait("Category", TestCategories.Integration)]
public async Task SlowIntegrationTest() { }
[Fact, Trait("Category", TestCategories.Live)]
public async Task RequiresExternalServices() { }
```
**CI Lane Filtering:**
```bash
# Run only unit tests (fast, no dependencies)
dotnet test --filter "Category=Unit"
# Run all tests except Live
dotnet test --filter "Category!=Live"
# Run Integration + Contract tests
dotnet test --filter "Category=Integration|Category=Contract"
```
**Available Categories:**
- `Unit` - Fast, in-memory, no external dependencies
- `Property` - FsCheck/generative testing
- `Snapshot` - Golden master regression
- `Integration` - Testcontainers (PostgreSQL, Valkey)
- `Contract` - API/WebService contract tests
- `Security` - Cryptographic validation
- `Performance` - Benchmarking, load tests
- `Live` - Requires external services (disabled in CI by default)
---
## Best Practices
### 1. Always Use TestCategories
Tag every test with the appropriate category:
```csharp
[Fact, Trait("Category", TestCategories.Unit)]
public void MyUnitTest() { }
```
This enables CI lane filtering and improves test discoverability.
### 2. Prefer Deterministic Primitives
Avoid `DateTime.UtcNow`, `Guid.NewGuid()`, `Random` in tests. Use TestKit alternatives:
```csharp
// ❌ Flaky test (time-dependent)
var expiration = DateTime.UtcNow.AddHours(1);
// ✅ Deterministic test
using var time = new DeterministicTime(DateTime.UtcNow);
var expiration = time.UtcNow.AddHours(1);
```
### 3. Use Snapshot Tests for Complex Outputs
For large JSON outputs (SBOM, VEX, attestations), snapshot testing is more maintainable than manual assertions:
```csharp
// ❌ Brittle manual assertions
Assert.Equal("SPDX-3.0.1", sbom.SpdxVersion);
Assert.Equal(42, sbom.Packages.Count);
// ...hundreds of assertions...
// ✅ Snapshot testing
SnapshotAssert.MatchesSnapshot(sbom, "MySbomSnapshot");
```
### 4. Isolate Integration Tests
Use TestCategories to separate fast unit tests from slow integration tests:
```csharp
[Fact, Trait("Category", TestCategories.Unit)]
public void FastTest() { /* no external dependencies */ }
[Fact, Trait("Category", TestCategories.Integration)]
public async Task SlowTest() { /* uses PostgresFixture */ }
```
In CI, run Unit tests first for fast feedback, then Integration tests in parallel.
### 5. Document Snapshot Baselines
When updating snapshots (`UPDATE_SNAPSHOTS=1`), add a commit message explaining why:
```bash
git commit -m "Update SBOM snapshot: added new package metadata fields"
```
This helps reviewers understand intentional vs. accidental changes.
---
## Troubleshooting
### Snapshot Mismatch
**Error:** `Snapshot 'MySbomSnapshot' does not match expected.`
**Solution:**
1. Review diff manually (check `Snapshots/MySbomSnapshot.json`)
2. If change is intentional: `UPDATE_SNAPSHOTS=1 dotnet test`
3. Commit updated snapshot with explanation
### Testcontainers Failure
**Error:** `Docker daemon not running`
**Solution:**
- Ensure Docker Desktop is running
- Verify `docker ps` works in terminal
- Check Testcontainers logs: `TESTCONTAINERS_DEBUG=1 dotnet test`
### Determinism Failure
**Error:** `CanonicalJsonAssert.IsDeterministic failed: byte arrays differ`
**Root Cause:** Non-deterministic data in serialization (e.g., random GUIDs, timestamps)
**Solution:**
- Use `DeterministicTime` and `DeterministicRandom`
- Ensure all data is seeded or mocked
- Check for `DateTime.UtcNow` or `Guid.NewGuid()` calls
---
## Migration Guide (Existing Tests)
### Step 1: Add TestKit Reference
```xml
<ProjectReference Include="../../../__Libraries/StellaOps.TestKit/StellaOps.TestKit.csproj" />
```
### Step 2: Replace Time-Dependent Code
**Before:**
```csharp
var now = DateTime.UtcNow;
```
**After:**
```csharp
using var time = new DeterministicTime(DateTime.UtcNow);
var now = time.UtcNow;
```
### Step 3: Add Test Categories
```csharp
[Fact] // Old
[Fact, Trait("Category", TestCategories.Unit)] // New
```
### Step 4: Adopt Snapshot Testing (Optional)
For complex JSON assertions, replace manual checks with snapshots:
```csharp
// Old
Assert.Equal(expected.SpdxVersion, actual.SpdxVersion);
// ...
// New
SnapshotAssert.MatchesSnapshot(actual, "TestName");
```
---
## CI Integration
### Example `.gitea/workflows/test.yml`
```yaml
name: Test Suite
on: [push, pull_request]
jobs:
unit:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-dotnet@v4
with:
dotnet-version: '10.0.x'
- name: Unit Tests (Fast)
run: dotnet test --filter "Category=Unit" --logger "trx;LogFileName=unit-results.trx"
- name: Upload Results
uses: actions/upload-artifact@v4
with:
name: unit-test-results
path: '**/unit-results.trx'
integration:
runs-on: ubuntu-latest
services:
docker:
image: docker:dind
steps:
- uses: actions/checkout@v4
- name: Integration Tests
run: dotnet test --filter "Category=Integration" --logger "trx;LogFileName=integration-results.trx"
```
---
## Support and Feedback
- **Issues:** Report bugs in sprint tracking files under `docs/implplan/`
- **Questions:** Contact Platform Guild
- **Documentation:** `src/__Libraries/StellaOps.TestKit/README.md`
---
## Changelog
### v1.0 (2025-12-23)
- Initial release: DeterministicTime, DeterministicRandom
- CanonicalJsonAssert, SnapshotAssert
- PostgresFixture, ValkeyFixture, HttpFixtureServer
- OtelCapture for OpenTelemetry traces
- TestCategories for CI lane filtering
- Pilot adoption in Scanner.Core.Tests

View File

@@ -0,0 +1,366 @@
# WebService Test Discipline
This document defines the testing discipline for StellaOps WebService projects. All web services must follow these patterns to ensure consistent test coverage, contract stability, telemetry verification, and security hardening.
## Overview
WebService tests use `WebServiceFixture<TProgram>` from `StellaOps.TestKit` and `WebApplicationFactory<TProgram>` from `Microsoft.AspNetCore.Mvc.Testing`. Tests are organized into four categories:
1. **Contract Tests** — OpenAPI schema stability
2. **OTel Trace Tests** — Telemetry verification
3. **Negative Tests** — Error handling validation
4. **Auth/AuthZ Tests** — Security boundary enforcement
---
## 1. Test Infrastructure
### WebServiceFixture Pattern
```csharp
using StellaOps.TestKit.Fixtures;
public class ScannerWebServiceTests : WebServiceTestBase<ScannerProgram>
{
public ScannerWebServiceTests() : base(new WebServiceFixture<ScannerProgram>())
{
}
// Tests inherit shared fixture setup
}
```
### Fixture Configuration
Each web service should have a dedicated fixture class that configures test-specific settings:
```csharp
public sealed class ScannerTestFixture : WebServiceFixture<ScannerProgram>
{
protected override void ConfigureTestServices(IServiceCollection services)
{
// Replace external dependencies with test doubles
services.AddSingleton<IStorageClient, InMemoryStorageClient>();
services.AddSingleton<IQueueClient, InMemoryQueueClient>();
}
protected override void ConfigureTestConfiguration(IDictionary<string, string?> config)
{
config["scanner:storage:driver"] = "inmemory";
config["scanner:events:enabled"] = "false";
}
}
```
---
## 2. Contract Tests
Contract tests ensure OpenAPI schema stability and detect breaking changes.
### Pattern
```csharp
[Fact]
[Trait("Lane", "Contract")]
public async Task OpenApi_Schema_MatchesSnapshot()
{
// Arrange
using var client = Fixture.CreateClient();
// Act
var response = await client.GetAsync("/swagger/v1/swagger.json");
var schema = await response.Content.ReadAsStringAsync();
// Assert
await ContractTestHelper.AssertSchemaMatchesSnapshot(schema, "scanner-v1");
}
[Fact]
[Trait("Lane", "Contract")]
public async Task Api_Response_MatchesContract()
{
// Arrange
using var client = Fixture.CreateClient();
var request = new ScanRequest { /* test data */ };
// Act
var response = await client.PostAsJsonAsync("/api/v1/scans", request);
var result = await response.Content.ReadFromJsonAsync<ScanResponse>();
// Assert
ContractTestHelper.AssertResponseMatchesSchema(result, "ScanResponse");
}
```
### Snapshot Management
- Snapshots stored in `Snapshots/` directory relative to test project
- Schema format: `<service>-<version>.json`
- Update snapshots intentionally when breaking changes are approved
---
## 3. OTel Trace Tests
OTel tests verify that telemetry spans are emitted correctly with required tags.
### Pattern
```csharp
[Fact]
[Trait("Lane", "Integration")]
public async Task ScanEndpoint_EmitsOtelTrace()
{
// Arrange
using var otelCapture = Fixture.CaptureOtelTraces();
using var client = Fixture.CreateClient();
var request = new ScanRequest { ImageRef = "nginx:1.25" };
// Act
await client.PostAsJsonAsync("/api/v1/scans", request);
// Assert
otelCapture.AssertHasSpan("scanner.scan");
otelCapture.AssertHasTag("scanner.scan", "scan.image_ref", "nginx:1.25");
otelCapture.AssertHasTag("scanner.scan", "tenant.id", ExpectedTenantId);
}
```
### Required Tags
All WebService endpoints must emit these tags:
| Tag | Description | Example |
|-----|-------------|---------|
| `tenant.id` | Tenant identifier | `tenant-a` |
| `request.id` | Correlation ID | `req-abc123` |
| `http.route` | Endpoint route | `/api/v1/scans` |
| `http.status_code` | Response code | `200` |
Service-specific tags are documented in each module's architecture doc.
---
## 4. Negative Tests
Negative tests verify proper error handling for invalid inputs.
### Pattern
```csharp
[Fact]
[Trait("Lane", "Security")]
public async Task MalformedContentType_Returns415()
{
// Arrange
using var client = Fixture.CreateClient();
var content = new StringContent("{}", Encoding.UTF8, "text/plain");
// Act
var response = await client.PostAsync("/api/v1/scans", content);
// Assert
Assert.Equal(HttpStatusCode.UnsupportedMediaType, response.StatusCode);
}
[Fact]
[Trait("Lane", "Security")]
public async Task OversizedPayload_Returns413()
{
// Arrange
using var client = Fixture.CreateClient();
var payload = new string('x', 10_000_001); // Exceeds 10MB limit
var content = new StringContent(payload, Encoding.UTF8, "application/json");
// Act
var response = await client.PostAsync("/api/v1/scans", content);
// Assert
Assert.Equal(HttpStatusCode.RequestEntityTooLarge, response.StatusCode);
}
[Fact]
[Trait("Lane", "Unit")]
public async Task MethodMismatch_Returns405()
{
// Arrange
using var client = Fixture.CreateClient();
// Act (POST endpoint, but using GET)
var response = await client.GetAsync("/api/v1/scans");
// Assert
Assert.Equal(HttpStatusCode.MethodNotAllowed, response.StatusCode);
}
```
### Required Coverage
| Negative Case | Expected Status | Test Trait |
|--------------|-----------------|------------|
| Malformed content type | 415 | Security |
| Oversized payload | 413 | Security |
| Method mismatch | 405 | Unit |
| Missing required field | 400 | Unit |
| Invalid field value | 400 | Unit |
| Unknown route | 404 | Unit |
---
## 5. Auth/AuthZ Tests
Auth tests verify security boundaries and tenant isolation.
### Pattern
```csharp
[Fact]
[Trait("Lane", "Security")]
public async Task AnonymousRequest_Returns401()
{
// Arrange
using var client = Fixture.CreateClient(); // No auth
// Act
var response = await client.GetAsync("/api/v1/scans");
// Assert
Assert.Equal(HttpStatusCode.Unauthorized, response.StatusCode);
}
[Fact]
[Trait("Lane", "Security")]
public async Task ExpiredToken_Returns401()
{
// Arrange
using var client = Fixture.CreateAuthenticatedClient(tokenExpired: true);
// Act
var response = await client.GetAsync("/api/v1/scans");
// Assert
Assert.Equal(HttpStatusCode.Unauthorized, response.StatusCode);
}
[Fact]
[Trait("Lane", "Security")]
public async Task TenantIsolation_CannotAccessOtherTenantData()
{
// Arrange
using var tenantAClient = Fixture.CreateTenantClient("tenant-a");
using var tenantBClient = Fixture.CreateTenantClient("tenant-b");
// Create scan as tenant A
var scanResponse = await tenantAClient.PostAsJsonAsync("/api/v1/scans", new ScanRequest { /* */ });
var scan = await scanResponse.Content.ReadFromJsonAsync<ScanResponse>();
// Act: Try to access as tenant B
var response = await tenantBClient.GetAsync($"/api/v1/scans/{scan!.Id}");
// Assert
Assert.Equal(HttpStatusCode.NotFound, response.StatusCode); // Tenant isolation
}
```
### Required Coverage
| Auth Case | Expected Behavior | Test Trait |
|-----------|-------------------|------------|
| No token | 401 Unauthorized | Security |
| Expired token | 401 Unauthorized | Security |
| Invalid signature | 401 Unauthorized | Security |
| Wrong audience | 401 Unauthorized | Security |
| Missing scope | 403 Forbidden | Security |
| Cross-tenant access | 404 Not Found or 403 Forbidden | Security |
---
## 6. Test Organization
### Directory Structure
```
src/<Module>/__Tests/StellaOps.<Module>.WebService.Tests/
├── StellaOps.<Module>.WebService.Tests.csproj
├── <Module>ApplicationFactory.cs # WebApplicationFactory implementation
├── <Module>TestFixture.cs # Shared test fixture
├── Contract/
│ └── OpenApiSchemaTests.cs
├── Telemetry/
│ └── OtelTraceTests.cs
├── Negative/
│ ├── ContentTypeTests.cs
│ ├── PayloadLimitTests.cs
│ └── MethodMismatchTests.cs
├── Auth/
│ ├── AuthenticationTests.cs
│ ├── AuthorizationTests.cs
│ └── TenantIsolationTests.cs
└── Snapshots/
└── <module>-v1.json # OpenAPI schema snapshot
```
### Test Trait Assignment
| Category | Trait | CI Lane | PR-Gating |
|----------|-------|---------|-----------|
| Contract | `[Trait("Lane", "Contract")]` | Contract | Yes |
| OTel | `[Trait("Lane", "Integration")]` | Integration | Yes |
| Negative (security) | `[Trait("Lane", "Security")]` | Security | Yes |
| Negative (validation) | `[Trait("Lane", "Unit")]` | Unit | Yes |
| Auth/AuthZ | `[Trait("Lane", "Security")]` | Security | Yes |
---
## 7. CI Integration
WebService tests run in the appropriate CI lanes:
```yaml
# .gitea/workflows/test-lanes.yml
jobs:
contract-tests:
steps:
- run: ./scripts/test-lane.sh Contract
security-tests:
steps:
- run: ./scripts/test-lane.sh Security
integration-tests:
steps:
- run: ./scripts/test-lane.sh Integration
```
All lanes are PR-gating. Failed tests block merge.
---
## 8. Rollout Checklist
When adding WebService tests to a new module:
- [ ] Create `<Module>ApplicationFactory` extending `WebApplicationFactory<TProgram>`
- [ ] Create `<Module>TestFixture` extending `WebServiceFixture<TProgram>` if needed
- [ ] Add contract tests with OpenAPI schema snapshot
- [ ] Add OTel trace tests for key endpoints
- [ ] Add negative tests (content type, payload, method)
- [ ] Add auth/authz tests (anonymous, expired, tenant isolation)
- [ ] Verify all tests have appropriate `[Trait("Lane", "...")]` attributes
- [ ] Run locally: `dotnet test --filter "Lane=Contract|Lane=Security|Lane=Integration"`
- [ ] Verify CI passes on PR
---
## References
- [WebServiceFixture Implementation](../../src/__Libraries/StellaOps.TestKit/Fixtures/WebServiceFixture.cs)
- [ContractTestHelper Implementation](../../src/__Libraries/StellaOps.TestKit/Fixtures/ContractTestHelper.cs)
- [WebServiceTestBase Implementation](../../src/__Libraries/StellaOps.TestKit/Templates/WebServiceTestBase.cs)
- [Test Lanes CI Workflow](../../.gitea/workflows/test-lanes.yml)
- [CI Lane Filters Documentation](./ci-lane-filters.md)
---
*Last updated: 2025-06-30 · Sprint 5100.0007.0006*

View File

@@ -0,0 +1,230 @@
# WebService Test Rollout Plan
This document defines the rollout plan for applying the WebService test discipline to all StellaOps web services.
## Overview
Following the pilot implementation on Scanner.WebService (Sprint 5100.0007.0006), this plan defines the order and timeline for rolling out comprehensive WebService tests to all remaining services.
---
## Service Inventory
| Service | Module Path | Priority | Status | Sprint |
|---------|-------------|----------|--------|--------|
| Scanner.WebService | `src/Scanner/StellaOps.Scanner.WebService` | P0 (Pilot) | ✅ Existing tests | 5100.0007.0006 |
| Concelier.WebService | `src/Concelier/StellaOps.Concelier.WebService` | P1 | Pending | TBD |
| Excititor.WebService | `src/Excititor/StellaOps.Excititor.WebService` | P1 | Pending | TBD |
| Policy.Engine | `src/Policy/StellaOps.Policy.Engine` | P1 | Pending | TBD |
| Scheduler.WebService | `src/Scheduler/StellaOps.Scheduler.WebService` | P2 | Pending | TBD |
| Notify.WebService | `src/Notify/StellaOps.Notify.WebService` | P2 | Pending | TBD |
| Authority | `src/Authority/StellaOps.Authority` | P2 | Pending | TBD |
| Signer | `src/Signer/StellaOps.Signer` | P3 | Pending | TBD |
| Attestor | `src/Attestor/StellaOps.Attestor` | P3 | Pending | TBD |
| ExportCenter.WebService | `src/ExportCenter/StellaOps.ExportCenter.WebService` | P3 | Pending | TBD |
| Registry.TokenService | `src/Registry/StellaOps.Registry.TokenService` | P3 | Pending | TBD |
| VulnExplorer.Api | `src/VulnExplorer/StellaOps.VulnExplorer.Api` | P3 | Pending | TBD |
| Graph.Api | `src/Graph/StellaOps.Graph.Api` | P3 | Pending | TBD |
| Orchestrator | `src/Orchestrator/StellaOps.Orchestrator` | P4 | Pending | TBD |
---
## Rollout Phases
### Phase 1: Core Data Flow Services (P1)
**Timeline**: Sprint 5100.0008.* (Q1 2026)
**Services**:
- **Concelier.WebService** — Primary advisory ingestion service
- **Excititor.WebService** — Enrichment and correlation service
- **Policy.Engine** — Policy evaluation service
**Rationale**: These services form the core data flow pipeline. They have high traffic, complex contracts, and critical security boundaries.
**Test Requirements**:
| Test Type | Concelier | Excititor | Policy |
|-----------|-----------|-----------|--------|
| Contract (OpenAPI) | Required | Required | Required |
| OTel traces | Required | Required | Required |
| Negative tests | Required | Required | Required |
| Auth/AuthZ | Required | Required | Required |
| Tenant isolation | Required | Required | Required |
---
### Phase 2: Scheduling & Notification Services (P2)
**Timeline**: Sprint 5100.0009.* (Q2 2026)
**Services**:
- **Scheduler.WebService** — Job scheduling and orchestration
- **Notify.WebService** — Notification dispatch
- **Authority** — Authentication/authorization service
**Rationale**: These services support operational workflows. Authority is critical for security testing of all other services.
**Test Requirements**:
| Test Type | Scheduler | Notify | Authority |
|-----------|-----------|--------|-----------|
| Contract (OpenAPI) | Required | Required | Required |
| OTel traces | Required | Required | Required |
| Negative tests | Required | Required | Required |
| Auth/AuthZ | N/A (system) | Required | N/A (self) |
| Token issuance | N/A | N/A | Required |
---
### Phase 3: Signing & Attestation Services (P3)
**Timeline**: Sprint 5100.0010.* (Q2-Q3 2026)
**Services**:
- **Signer** — Cryptographic signing service
- **Attestor** — Attestation generation/verification
- **ExportCenter.WebService** — Report export service
- **Registry.TokenService** — OCI registry token service
- **VulnExplorer.Api** — Vulnerability exploration API
- **Graph.Api** — Graph query API
**Rationale**: These services have specialized contracts and lower traffic. They require careful security testing due to cryptographic operations.
**Test Requirements**:
| Test Type | Signer | Attestor | Others |
|-----------|--------|----------|--------|
| Contract (OpenAPI) | Required | Required | Required |
| OTel traces | Required | Required | Required |
| Negative tests | Required | Required | Required |
| Crypto validation | Required | Required | N/A |
---
### Phase 4: Orchestration Services (P4)
**Timeline**: Sprint 5100.0011.* (Q3 2026)
**Services**:
- **Orchestrator** — Workflow orchestration
**Rationale**: Orchestrator is a meta-service that coordinates other services. Testing depends on other services being testable first.
---
## Test Coverage Targets
### Minimum Requirements (PR-Gating)
| Test Category | Min Coverage | Lane |
|---------------|-------------|------|
| Contract (OpenAPI) | 100% of public endpoints | Contract |
| Negative (4xx errors) | 100% of error codes | Unit/Security |
| Auth/AuthZ | 100% of protected endpoints | Security |
### Recommended (Quality Gate)
| Test Category | Target Coverage | Lane |
|---------------|-----------------|------|
| OTel traces | 80% of endpoints | Integration |
| Tenant isolation | 100% of data endpoints | Security |
| Performance baselines | Key endpoints | Performance |
---
## Implementation Checklist per Service
```markdown
## <Service Name> WebService Tests
### Setup
- [ ] Create `<Service>ApplicationFactory` (WebApplicationFactory)
- [ ] Create `<Service>TestFixture` if custom setup needed
- [ ] Add test project: `StellaOps.<Service>.WebService.Tests`
- [ ] Add reference to `StellaOps.TestKit`
### Contract Tests
- [ ] Extract OpenAPI schema snapshot (`Snapshots/<service>-v1.json`)
- [ ] Add schema stability test
- [ ] Add response contract tests for key endpoints
### OTel Tests
- [ ] Add trace assertion tests for key endpoints
- [ ] Verify required tags (tenant.id, request.id, http.route)
### Negative Tests
- [ ] Malformed content type → 415
- [ ] Oversized payload → 413
- [ ] Method mismatch → 405
- [ ] Missing required field → 400
- [ ] Invalid field value → 400
### Auth Tests
- [ ] Anonymous request → 401
- [ ] Expired token → 401
- [ ] Missing scope → 403
- [ ] Cross-tenant access → 404/403
### CI Integration
- [ ] Verify traits assigned: Contract, Security, Integration, Unit
- [ ] PR passes all lanes
- [ ] Add to TEST_COVERAGE_MATRIX.md
```
---
## Sprint Planning Template
When creating sprints for new service tests:
```markdown
# Sprint 5100.XXXX.YYYY - <Service> WebService Tests
## Topic & Scope
- Apply WebService test discipline to <Service>.WebService
- Contract tests, OTel traces, negative tests, auth tests
- **Working directory:** `src/<Module>/__Tests/StellaOps.<Module>.WebService.Tests`
## Delivery Tracker
| # | Task ID | Status | Task Definition |
|---|---------|--------|-----------------|
| 1 | WEBSVC-XXXX-001 | TODO | Create <Service>ApplicationFactory |
| 2 | WEBSVC-XXXX-002 | TODO | Add OpenAPI contract tests |
| 3 | WEBSVC-XXXX-003 | TODO | Add OTel trace tests |
| 4 | WEBSVC-XXXX-004 | TODO | Add negative tests (4xx) |
| 5 | WEBSVC-XXXX-005 | TODO | Add auth/authz tests |
| 6 | WEBSVC-XXXX-006 | TODO | Update TEST_COVERAGE_MATRIX.md |
```
---
## Success Metrics
| Metric | Target | Measurement |
|--------|--------|-------------|
| Services with contract tests | 100% | Count of services with OpenAPI snapshot tests |
| Services with auth tests | 100% | Count of services with auth boundary tests |
| Contract test failures in production | 0 | Breaking changes detected in staging |
| Security test coverage | 100% of auth endpoints | Audit of protected routes vs tests |
---
## Risks & Mitigations
| Risk | Impact | Mitigation |
|------|--------|------------|
| Services lack OpenAPI spec | Cannot do contract testing | Generate spec via Swashbuckle/NSwag |
| OTel not configured in service | Cannot verify traces | Add OTel middleware as prerequisite |
| Auth disabled in test mode | False confidence | Test with auth enabled, use test tokens |
| Test fixtures are slow | CI timeout | Share fixtures, use in-memory providers |
---
## References
- [WebService Test Discipline](./webservice-test-discipline.md)
- [Test Coverage Matrix](./TEST_COVERAGE_MATRIX.md)
- [CI Lane Filters](./ci-lane-filters.md)
- [Testing Strategy Models](./testing-strategy-models.md)
---
*Last updated: 2025-06-30 · Sprint 5100.0007.0006*