Add tests for SBOM generation determinism across multiple formats
- Created `StellaOps.TestKit.Tests` project for unit tests related to determinism. - Implemented `DeterminismManifestTests` to validate deterministic output for canonical bytes and strings, file read/write operations, and error handling for invalid schema versions. - Added `SbomDeterminismTests` to ensure identical inputs produce consistent SBOMs across SPDX 3.0.1 and CycloneDX 1.6/1.7 formats, including parallel execution tests. - Updated project references in `StellaOps.Integration.Determinism` to include the new determinism testing library.
This commit is contained in:
425
docs/testing/connector-fixture-discipline.md
Normal file
425
docs/testing/connector-fixture-discipline.md
Normal file
@@ -0,0 +1,425 @@
|
||||
# Connector Fixture Discipline
|
||||
|
||||
This document defines the testing discipline for StellaOps Concelier and Excititor connectors. All connectors must follow these patterns to ensure consistent, deterministic, and offline-capable testing.
|
||||
|
||||
## Overview
|
||||
|
||||
Connector tests follow **Model C1 (Connector/External)** from the testing strategy:
|
||||
|
||||
1. **Fixture-based parser tests** — Raw upstream payload → normalized internal model (offline)
|
||||
2. **Resilience tests** — Partial/bad input → deterministic failure classification
|
||||
3. **Security tests** — URL allowlist, redirect handling, payload limits
|
||||
4. **Live smoke tests** — Schema drift detection (opt-in, non-gating)
|
||||
|
||||
---
|
||||
|
||||
## 1. Directory Structure
|
||||
|
||||
Each connector test project follows this structure:
|
||||
|
||||
```
|
||||
src/<Module>/__Tests/StellaOps.<Module>.Connector.<Source>.Tests/
|
||||
├── StellaOps.<Module>.Connector.<Source>.Tests.csproj
|
||||
├── Fixtures/
|
||||
│ ├── <source>-typical.json # Typical advisory payload
|
||||
│ ├── <source>-edge-<case>.json # Edge case payloads
|
||||
│ ├── <source>-error-<type>.json # Malformed/invalid payloads
|
||||
│ └── expected-<id>.json # Expected normalized output
|
||||
├── <Source>/
|
||||
│ ├── <Source>ParserTests.cs # Parser unit tests
|
||||
│ ├── <Source>ConnectorTests.cs # Connector integration tests
|
||||
│ └── <Source>ResilienceTests.cs # Resilience/security tests
|
||||
└── Expected/
|
||||
└── <source>-<id>.canonical.json # Canonical JSON snapshots
|
||||
```
|
||||
|
||||
### Example: NVD Connector
|
||||
|
||||
```
|
||||
src/Concelier/__Tests/StellaOps.Concelier.Connector.Nvd.Tests/
|
||||
├── Fixtures/
|
||||
│ ├── nvd-window-1.json
|
||||
│ ├── nvd-window-2.json
|
||||
│ ├── nvd-multipage-1.json
|
||||
│ ├── nvd-multipage-2.json
|
||||
│ ├── nvd-invalid-schema.json
|
||||
│ └── expected-CVE-2024-0001.json
|
||||
├── Nvd/
|
||||
│ ├── NvdParserTests.cs
|
||||
│ ├── NvdConnectorTests.cs
|
||||
│ └── NvdConnectorHarnessTests.cs
|
||||
└── Expected/
|
||||
└── conflict-nvd.canonical.json
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Fixture-Based Parser Tests
|
||||
|
||||
### Purpose
|
||||
|
||||
Test that the parser correctly transforms raw upstream payloads into normalized internal models without network access.
|
||||
|
||||
### Pattern
|
||||
|
||||
```csharp
|
||||
using StellaOps.TestKit.Connectors;
|
||||
|
||||
public class NvdParserTests : ConnectorParserTestBase<NvdRawAdvisory, ConcelierAdvisory>
|
||||
{
|
||||
public NvdParserTests()
|
||||
: base(new NvdParser(), "Nvd/Fixtures")
|
||||
{
|
||||
}
|
||||
|
||||
[Fact]
|
||||
[Trait("Lane", "Unit")]
|
||||
public async Task ParseTypicalAdvisory_ProducesExpectedModel()
|
||||
{
|
||||
// Arrange
|
||||
var raw = await LoadFixture<NvdRawAdvisory>("nvd-window-1.json");
|
||||
|
||||
// Act
|
||||
var result = Parser.Parse(raw);
|
||||
|
||||
// Assert
|
||||
await AssertMatchesSnapshot(result, "expected-CVE-2024-0001.json");
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[Trait("Lane", "Unit")]
|
||||
[InlineData("nvd-multipage-1.json", "expected-multipage-1.json")]
|
||||
[InlineData("nvd-multipage-2.json", "expected-multipage-2.json")]
|
||||
public async Task ParseAllFixtures_ProducesExpectedModels(string input, string expected)
|
||||
{
|
||||
var raw = await LoadFixture<NvdRawAdvisory>(input);
|
||||
var result = Parser.Parse(raw);
|
||||
await AssertMatchesSnapshot(result, expected);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Fixture Requirements
|
||||
|
||||
| Type | Naming Convention | Purpose |
|
||||
|------|-------------------|---------|
|
||||
| Typical | `<source>-typical.json` | Normal advisory with all common fields |
|
||||
| Edge case | `<source>-edge-<case>.json` | Unusual but valid payloads |
|
||||
| Error | `<source>-error-<type>.json` | Malformed/invalid payloads |
|
||||
| Expected | `expected-<id>.json` | Expected normalized output |
|
||||
| Canonical | `<source>-<id>.canonical.json` | Deterministic JSON snapshot |
|
||||
|
||||
### Minimum Coverage
|
||||
|
||||
Each connector must have fixtures for:
|
||||
|
||||
- [ ] At least 1 typical payload
|
||||
- [ ] At least 2 edge cases (e.g., multi-vendor, unusual CVSS, missing optional fields)
|
||||
- [ ] At least 2 error cases (e.g., missing required fields, invalid schema)
|
||||
|
||||
---
|
||||
|
||||
## 3. Resilience Tests
|
||||
|
||||
### Purpose
|
||||
|
||||
Verify that connectors handle malformed input gracefully with deterministic failure classification.
|
||||
|
||||
### Pattern
|
||||
|
||||
```csharp
|
||||
public class NvdResilienceTests : ConnectorResilienceTestBase<NvdConnector>
|
||||
{
|
||||
public NvdResilienceTests()
|
||||
: base(new NvdConnector(CreateTestHttpClient()))
|
||||
{
|
||||
}
|
||||
|
||||
[Fact]
|
||||
[Trait("Lane", "Unit")]
|
||||
public async Task MissingRequiredField_ReturnsParseError()
|
||||
{
|
||||
// Arrange
|
||||
var payload = await LoadFixture("nvd-error-missing-cve-id.json");
|
||||
|
||||
// Act
|
||||
var result = await Connector.ParseAsync(payload);
|
||||
|
||||
// Assert
|
||||
Assert.False(result.IsSuccess);
|
||||
Assert.Equal(ConnectorErrorKind.ParseError, result.Error.Kind);
|
||||
Assert.Contains("cve_id", result.Error.Message, StringComparison.OrdinalIgnoreCase);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
[Trait("Lane", "Unit")]
|
||||
public async Task InvalidDateFormat_ReturnsParseError()
|
||||
{
|
||||
var payload = await LoadFixture("nvd-error-invalid-date.json");
|
||||
|
||||
var result = await Connector.ParseAsync(payload);
|
||||
|
||||
Assert.False(result.IsSuccess);
|
||||
Assert.Equal(ConnectorErrorKind.ParseError, result.Error.Kind);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
[Trait("Lane", "Unit")]
|
||||
public async Task UnexpectedEnumValue_LogsWarningAndContinues()
|
||||
{
|
||||
var payload = await LoadFixture("nvd-edge-unknown-severity.json");
|
||||
|
||||
var result = await Connector.ParseAsync(payload);
|
||||
|
||||
Assert.True(result.IsSuccess);
|
||||
Assert.Contains(result.Warnings, w => w.Contains("unknown severity"));
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Required Test Cases
|
||||
|
||||
| Case | Expected Behavior | Trait |
|
||||
|------|-------------------|-------|
|
||||
| Missing required field | `ConnectorErrorKind.ParseError` | Unit |
|
||||
| Invalid date format | `ConnectorErrorKind.ParseError` | Unit |
|
||||
| Invalid JSON structure | `ConnectorErrorKind.ParseError` | Unit |
|
||||
| Unknown enum value | Warning logged, continues | Unit |
|
||||
| Empty response | `ConnectorErrorKind.EmptyResponse` | Unit |
|
||||
| Truncated payload | `ConnectorErrorKind.ParseError` | Unit |
|
||||
|
||||
---
|
||||
|
||||
## 4. Security Tests
|
||||
|
||||
### Purpose
|
||||
|
||||
Verify that connectors enforce security boundaries for network operations.
|
||||
|
||||
### Pattern
|
||||
|
||||
```csharp
|
||||
public class NvdSecurityTests : ConnectorSecurityTestBase<NvdConnector>
|
||||
{
|
||||
[Fact]
|
||||
[Trait("Lane", "Security")]
|
||||
public async Task UrlOutsideAllowlist_RejectsRequest()
|
||||
{
|
||||
// Arrange
|
||||
var connector = CreateConnector(allowedHosts: ["services.nvd.nist.gov"]);
|
||||
|
||||
// Act & Assert
|
||||
await Assert.ThrowsAsync<SecurityException>(
|
||||
() => connector.FetchAsync("https://evil.example.com/api"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
[Trait("Lane", "Security")]
|
||||
public async Task RedirectToDisallowedHost_RejectsRequest()
|
||||
{
|
||||
var handler = CreateMockHandler(redirectTo: "https://evil.example.com");
|
||||
var connector = CreateConnector(handler, allowedHosts: ["services.nvd.nist.gov"]);
|
||||
|
||||
await Assert.ThrowsAsync<SecurityException>(
|
||||
() => connector.FetchAsync("https://services.nvd.nist.gov/api"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
[Trait("Lane", "Security")]
|
||||
public async Task PayloadExceedsMaxSize_RejectsPayload()
|
||||
{
|
||||
var handler = CreateMockHandler(responseSize: 100_000_001); // 100MB
|
||||
var connector = CreateConnector(handler, maxPayloadBytes: 100_000_000);
|
||||
|
||||
await Assert.ThrowsAsync<PayloadTooLargeException>(
|
||||
() => connector.FetchAsync("https://services.nvd.nist.gov/api"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
[Trait("Lane", "Security")]
|
||||
public async Task DecompressionBomb_RejectsPayload()
|
||||
{
|
||||
// 1KB compressed, 1GB decompressed
|
||||
var handler = CreateMockHandler(compressedBomb: true);
|
||||
var connector = CreateConnector(handler);
|
||||
|
||||
await Assert.ThrowsAsync<PayloadTooLargeException>(
|
||||
() => connector.FetchAsync("https://services.nvd.nist.gov/api"));
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Required Security Tests
|
||||
|
||||
| Test | Purpose | Trait |
|
||||
|------|---------|-------|
|
||||
| URL allowlist | Block requests to unauthorized hosts | Security |
|
||||
| Redirect validation | Block redirects to unauthorized hosts | Security |
|
||||
| Max payload size | Reject oversized responses | Security |
|
||||
| Decompression bomb | Reject zip bombs | Security |
|
||||
| Rate limiting | Respect upstream rate limits | Security |
|
||||
|
||||
---
|
||||
|
||||
## 5. Live Smoke Tests (Opt-In)
|
||||
|
||||
### Purpose
|
||||
|
||||
Detect upstream schema drift by comparing live responses against known fixtures.
|
||||
|
||||
### Pattern
|
||||
|
||||
```csharp
|
||||
public class NvdLiveTests : ConnectorLiveTestBase<NvdConnector>
|
||||
{
|
||||
[Fact]
|
||||
[Trait("Lane", "Live")]
|
||||
[Trait("Category", "SchemaDrift")]
|
||||
public async Task LiveSchema_MatchesFixtureSchema()
|
||||
{
|
||||
// Skip if not in Live lane
|
||||
Skip.IfNot(IsLiveLaneEnabled());
|
||||
|
||||
// Fetch live response
|
||||
var live = await Connector.FetchLatestAsync();
|
||||
|
||||
// Compare schema (not values) against fixture
|
||||
var fixture = await LoadFixture<NvdResponse>("nvd-typical.json");
|
||||
|
||||
AssertSchemaMatches(live, fixture);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
[Trait("Lane", "Live")]
|
||||
public async Task LiveFetch_ReturnsValidAdvisories()
|
||||
{
|
||||
Skip.IfNot(IsLiveLaneEnabled());
|
||||
|
||||
var result = await Connector.FetchLatestAsync();
|
||||
|
||||
Assert.True(result.IsSuccess);
|
||||
Assert.NotEmpty(result.Value.Advisories);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Configuration
|
||||
|
||||
Live tests are:
|
||||
- **Never PR-gating** — run only in scheduled/nightly jobs
|
||||
- **Opt-in** — require explicit `LIVE_TESTS_ENABLED=true` environment variable
|
||||
- **Alerting** — schema drift triggers notification, not failure
|
||||
|
||||
---
|
||||
|
||||
## 6. Fixture Updater
|
||||
|
||||
### Purpose
|
||||
|
||||
Refresh fixtures from live sources when upstream schemas change intentionally.
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
# Update all fixtures for NVD connector
|
||||
dotnet run --project tools/FixtureUpdater -- \
|
||||
--connector nvd \
|
||||
--output src/Concelier/__Tests/StellaOps.Concelier.Connector.Nvd.Tests/Nvd/Fixtures/
|
||||
|
||||
# Update specific fixture
|
||||
dotnet run --project tools/FixtureUpdater -- \
|
||||
--connector nvd \
|
||||
--cve CVE-2024-0001 \
|
||||
--output src/Concelier/__Tests/.../Fixtures/nvd-CVE-2024-0001.json
|
||||
|
||||
# Dry-run mode (show diff without writing)
|
||||
dotnet run --project tools/FixtureUpdater -- \
|
||||
--connector nvd \
|
||||
--dry-run
|
||||
```
|
||||
|
||||
### Workflow
|
||||
|
||||
1. Live test detects schema drift
|
||||
2. CI creates draft PR with fixture update
|
||||
3. Developer reviews diff for intentional vs accidental changes
|
||||
4. If intentional: update parser and merge
|
||||
5. If accidental: investigate upstream API issue
|
||||
|
||||
---
|
||||
|
||||
## 7. Test Traits and CI Integration
|
||||
|
||||
### Trait Assignment
|
||||
|
||||
| Test Category | Trait | Lane | PR-Gating |
|
||||
|---------------|-------|------|-----------|
|
||||
| Parser tests | `[Trait("Lane", "Unit")]` | Unit | Yes |
|
||||
| Resilience tests | `[Trait("Lane", "Unit")]` | Unit | Yes |
|
||||
| Security tests | `[Trait("Lane", "Security")]` | Security | Yes |
|
||||
| Live tests | `[Trait("Lane", "Live")]` | Live | No |
|
||||
|
||||
### Running Tests
|
||||
|
||||
```bash
|
||||
# Run all connector unit tests
|
||||
dotnet test --filter "Lane=Unit" src/Concelier/__Tests/
|
||||
|
||||
# Run security tests
|
||||
dotnet test --filter "Lane=Security" src/Concelier/__Tests/
|
||||
|
||||
# Run live tests (requires LIVE_TESTS_ENABLED=true)
|
||||
LIVE_TESTS_ENABLED=true dotnet test --filter "Lane=Live" src/Concelier/__Tests/
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8. Connector Inventory
|
||||
|
||||
### Concelier Connectors (Advisory Sources)
|
||||
|
||||
| Connector | Fixtures | Parser Tests | Resilience | Security | Live |
|
||||
|-----------|----------|--------------|------------|----------|------|
|
||||
| NVD | ✅ | ✅ | ✅ | ⬜ | ⬜ |
|
||||
| OSV | ✅ | ✅ | ⬜ | ⬜ | ⬜ |
|
||||
| GHSA | ✅ | ✅ | ⬜ | ⬜ | ⬜ |
|
||||
| CVE | ✅ | ✅ | ⬜ | ⬜ | ⬜ |
|
||||
| KEV | ✅ | ✅ | ⬜ | ⬜ | ⬜ |
|
||||
| EPSS | ✅ | ✅ | ⬜ | ⬜ | ⬜ |
|
||||
| Distro.Alpine | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ |
|
||||
| Distro.Debian | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ |
|
||||
| Distro.RedHat | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ |
|
||||
| Distro.Suse | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ |
|
||||
| Distro.Ubuntu | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ |
|
||||
| Vndr.Adobe | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ |
|
||||
| Vndr.Apple | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ |
|
||||
| Vndr.Cisco | ✅ | ✅ | ⬜ | ⬜ | ⬜ |
|
||||
| Vndr.Msrc | ✅ | ✅ | ⬜ | ⬜ | ⬜ |
|
||||
| Vndr.Oracle | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ |
|
||||
| Vndr.Vmware | ✅ | ✅ | ⬜ | ⬜ | ⬜ |
|
||||
| Cert.Bund | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ |
|
||||
| Cert.Cc | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ |
|
||||
| Cert.Fr | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ |
|
||||
| ICS.Cisa | ✅ | ✅ | ⬜ | ⬜ | ⬜ |
|
||||
|
||||
### Excititor Connectors (VEX Sources)
|
||||
|
||||
| Connector | Fixtures | Parser Tests | Resilience | Security | Live |
|
||||
|-----------|----------|--------------|------------|----------|------|
|
||||
| OpenVEX | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ |
|
||||
| CSAF/VEX | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ |
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- [ConnectorHttpFixture](../../src/__Libraries/StellaOps.TestKit/Connectors/ConnectorHttpFixture.cs)
|
||||
- [ConnectorTestBase](../../src/__Libraries/StellaOps.TestKit/Connectors/ConnectorTestBase.cs)
|
||||
- [ConnectorResilienceTestBase](../../src/__Libraries/StellaOps.TestKit/Connectors/ConnectorResilienceTestBase.cs)
|
||||
- [FixtureUpdater](../../src/__Libraries/StellaOps.TestKit/Connectors/FixtureUpdater.cs)
|
||||
- [Testing Strategy Models](./testing-strategy-models.md)
|
||||
- [CI Lane Filters](./ci-lane-filters.md)
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2025-06-30 · Sprint 5100.0007.0005*
|
||||
362
docs/testing/determinism-verification.md
Normal file
362
docs/testing/determinism-verification.md
Normal file
@@ -0,0 +1,362 @@
|
||||
# Determinism Verification Guide
|
||||
|
||||
**Sprint:** 5100.0007.0003 (Epic B)
|
||||
**Last Updated:** 2025-12-23
|
||||
|
||||
## Overview
|
||||
|
||||
StellaOps enforces deterministic artifact generation across all exported formats. This ensures:
|
||||
|
||||
1. **Reproducibility**: Given the same inputs, outputs are byte-for-byte identical
|
||||
2. **Auditability**: Hash verification proves artifact integrity
|
||||
3. **Compliance**: Regulated environments can replay and verify builds
|
||||
4. **CI Gating**: Drift detection prevents unintended changes
|
||||
|
||||
## Supported Artifact Types
|
||||
|
||||
| Type | Format(s) | Test File |
|
||||
|------|-----------|-----------|
|
||||
| SBOM | SPDX 3.0.1, CycloneDX 1.6, CycloneDX 1.7 | `SbomDeterminismTests.cs` |
|
||||
| VEX | OpenVEX, CSAF 2.0 | `VexDeterminismTests.cs` |
|
||||
| Policy Verdicts | JSON | `PolicyDeterminismTests.cs` |
|
||||
| Evidence Bundles | JSON, DSSE, in-toto | `EvidenceBundleDeterminismTests.cs` |
|
||||
| AirGap Bundles | NDJSON | `AirGapBundleDeterminismTests.cs` |
|
||||
| Advisory Normalization | Canonical JSON | `IngestionDeterminismTests.cs` |
|
||||
|
||||
## Determinism Manifest Format
|
||||
|
||||
Every deterministic artifact can produce a manifest describing its content hash and generation context.
|
||||
|
||||
### Schema (v1.0)
|
||||
|
||||
```json
|
||||
{
|
||||
"schemaVersion": "1.0",
|
||||
"artifact": {
|
||||
"type": "sbom | vex | policy-verdict | evidence-bundle | airgap-bundle",
|
||||
"name": "artifact-identifier",
|
||||
"version": "1.0.0",
|
||||
"format": "SPDX 3.0.1 | CycloneDX 1.6 | OpenVEX | CSAF 2.0 | ..."
|
||||
},
|
||||
"canonicalHash": {
|
||||
"algorithm": "SHA-256",
|
||||
"value": "abc123..."
|
||||
},
|
||||
"toolchain": {
|
||||
"platform": ".NET 10.0",
|
||||
"components": [
|
||||
{ "name": "StellaOps.Scanner", "version": "1.0.0" }
|
||||
]
|
||||
},
|
||||
"inputs": {
|
||||
"feedSnapshotHash": "def456...",
|
||||
"policyManifestHash": "ghi789...",
|
||||
"configHash": "jkl012..."
|
||||
},
|
||||
"generatedAt": "2025-12-23T18:00:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
### Field Descriptions
|
||||
|
||||
| Field | Description |
|
||||
|-------|-------------|
|
||||
| `schemaVersion` | Manifest schema version (currently `1.0`) |
|
||||
| `artifact.type` | Category of the artifact |
|
||||
| `artifact.name` | Identifier for the artifact |
|
||||
| `artifact.version` | Version of the artifact (if applicable) |
|
||||
| `artifact.format` | Specific format/spec version |
|
||||
| `canonicalHash.algorithm` | Hash algorithm (always `SHA-256`) |
|
||||
| `canonicalHash.value` | Lowercase hex hash of canonical bytes |
|
||||
| `toolchain.platform` | Runtime platform |
|
||||
| `toolchain.components` | List of generating components with versions |
|
||||
| `inputs` | Hashes of input artifacts (feed snapshots, policies, etc.) |
|
||||
| `generatedAt` | ISO-8601 UTC timestamp of generation |
|
||||
|
||||
## Creating a Determinism Manifest
|
||||
|
||||
Use `DeterminismManifestWriter` from `StellaOps.Testing.Determinism`:
|
||||
|
||||
```csharp
|
||||
using StellaOps.Testing.Determinism;
|
||||
|
||||
// Generate artifact bytes
|
||||
var sbomBytes = GenerateSbom(input, frozenTime);
|
||||
|
||||
// Create artifact info
|
||||
var artifactInfo = new ArtifactInfo
|
||||
{
|
||||
Type = "sbom",
|
||||
Name = "my-container-sbom",
|
||||
Version = "1.0.0",
|
||||
Format = "CycloneDX 1.6"
|
||||
};
|
||||
|
||||
// Create toolchain info
|
||||
var toolchain = new ToolchainInfo
|
||||
{
|
||||
Platform = ".NET 10.0",
|
||||
Components = new[]
|
||||
{
|
||||
new ComponentInfo { Name = "StellaOps.Scanner", Version = "1.0.0" }
|
||||
}
|
||||
};
|
||||
|
||||
// Create manifest
|
||||
var manifest = DeterminismManifestWriter.CreateManifest(
|
||||
sbomBytes,
|
||||
artifactInfo,
|
||||
toolchain);
|
||||
|
||||
// Save manifest
|
||||
DeterminismManifestWriter.Save(manifest, "determinism.json");
|
||||
```
|
||||
|
||||
## Reading and Verifying Manifests
|
||||
|
||||
```csharp
|
||||
// Load manifest
|
||||
var manifest = DeterminismManifestReader.Load("determinism.json");
|
||||
|
||||
// Verify artifact bytes match manifest hash
|
||||
var currentBytes = File.ReadAllBytes("artifact.json");
|
||||
var isValid = DeterminismManifestReader.Verify(manifest, currentBytes);
|
||||
|
||||
if (!isValid)
|
||||
{
|
||||
throw new DeterminismDriftException(
|
||||
$"Artifact hash mismatch. Expected: {manifest.CanonicalHash.Value}");
|
||||
}
|
||||
```
|
||||
|
||||
## Determinism Rules
|
||||
|
||||
### 1. Canonical JSON Serialization
|
||||
|
||||
All JSON output must use canonical serialization via `StellaOps.Canonical.Json`:
|
||||
|
||||
```csharp
|
||||
using StellaOps.Canonical.Json;
|
||||
|
||||
var json = CanonJson.Serialize(myObject);
|
||||
var hash = CanonJson.Sha256Hex(Encoding.UTF8.GetBytes(json));
|
||||
```
|
||||
|
||||
Rules:
|
||||
- Keys sorted lexicographically
|
||||
- No trailing whitespace
|
||||
- Unix line endings (`\n`)
|
||||
- No BOM
|
||||
- UTF-8 encoding
|
||||
|
||||
### 2. Frozen Timestamps
|
||||
|
||||
All timestamps must be provided externally or use `DeterministicTime`:
|
||||
|
||||
```csharp
|
||||
// ❌ BAD - Non-deterministic
|
||||
var timestamp = DateTimeOffset.UtcNow;
|
||||
|
||||
// ✅ GOOD - Deterministic
|
||||
var timestamp = frozenTime; // Passed as parameter
|
||||
```
|
||||
|
||||
### 3. Deterministic IDs
|
||||
|
||||
UUIDs and IDs must be derived from content, not random:
|
||||
|
||||
```csharp
|
||||
// ❌ BAD - Random UUID
|
||||
var id = Guid.NewGuid();
|
||||
|
||||
// ✅ GOOD - Content-derived ID
|
||||
var seed = $"{input.Name}:{input.Version}:{timestamp:O}";
|
||||
var hash = CanonJson.Sha256Hex(Encoding.UTF8.GetBytes(seed));
|
||||
var id = new Guid(Convert.FromHexString(hash[..32]));
|
||||
```
|
||||
|
||||
### 4. Stable Ordering
|
||||
|
||||
Collections must be sorted before serialization:
|
||||
|
||||
```csharp
|
||||
// ❌ BAD - Non-deterministic order
|
||||
var items = dictionary.Values;
|
||||
|
||||
// ✅ GOOD - Sorted order
|
||||
var items = dictionary.Values
|
||||
.OrderBy(v => v.Key, StringComparer.Ordinal);
|
||||
```
|
||||
|
||||
### 5. Parallel Safety
|
||||
|
||||
Determinism must hold under parallel execution:
|
||||
|
||||
```csharp
|
||||
var tasks = Enumerable.Range(0, 20)
|
||||
.Select(_ => Task.Run(() => GenerateArtifact(input, frozenTime)))
|
||||
.ToArray();
|
||||
|
||||
var results = await Task.WhenAll(tasks);
|
||||
results.Should().AllBe(results[0]); // All identical
|
||||
```
|
||||
|
||||
## CI Integration
|
||||
|
||||
### PR Merge Gate
|
||||
|
||||
The determinism gate runs on PR merge:
|
||||
|
||||
```yaml
|
||||
# .gitea/workflows/determinism-gate.yaml
|
||||
name: Determinism Gate
|
||||
on:
|
||||
pull_request:
|
||||
types: [synchronize, ready_for_review]
|
||||
jobs:
|
||||
determinism:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: actions/setup-dotnet@v4
|
||||
with:
|
||||
dotnet-version: '10.0.x'
|
||||
- name: Run Determinism Tests
|
||||
run: |
|
||||
dotnet test tests/integration/StellaOps.Integration.Determinism \
|
||||
--logger "trx;LogFileName=determinism.trx"
|
||||
- name: Generate Determinism Manifest
|
||||
run: |
|
||||
dotnet run --project tools/DeterminismManifestGenerator \
|
||||
--output determinism.json
|
||||
- name: Upload Determinism Artifact
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: determinism-manifest
|
||||
path: determinism.json
|
||||
```
|
||||
|
||||
### Baseline Storage
|
||||
|
||||
Determinism baselines are stored as CI artifacts:
|
||||
|
||||
```
|
||||
ci-artifacts/
|
||||
determinism/
|
||||
baseline/
|
||||
sbom-spdx-3.0.1.json
|
||||
sbom-cyclonedx-1.6.json
|
||||
sbom-cyclonedx-1.7.json
|
||||
vex-openvex.json
|
||||
vex-csaf.json
|
||||
policy-verdict.json
|
||||
evidence-bundle.json
|
||||
airgap-bundle.json
|
||||
```
|
||||
|
||||
### Drift Detection
|
||||
|
||||
When a PR changes artifact output:
|
||||
|
||||
1. CI compares new manifest hash against baseline
|
||||
2. If different, CI fails with diff report
|
||||
3. Developer must either:
|
||||
- Fix the regression (restore determinism)
|
||||
- Update the baseline (if change is intentional)
|
||||
|
||||
### Baseline Update Process
|
||||
|
||||
To intentionally update a baseline:
|
||||
|
||||
```bash
|
||||
# 1. Run determinism tests to generate new manifests
|
||||
dotnet test tests/integration/StellaOps.Integration.Determinism
|
||||
|
||||
# 2. Update baseline files
|
||||
cp determinism/*.json ci-artifacts/determinism/baseline/
|
||||
|
||||
# 3. Commit with explicit message
|
||||
git add ci-artifacts/determinism/baseline/
|
||||
git commit -m "chore(determinism): update baselines for [reason]
|
||||
|
||||
Breaking: [explain what changed]
|
||||
Justification: [explain why this is correct]"
|
||||
```
|
||||
|
||||
## Replay Verification
|
||||
|
||||
To verify an artifact was produced deterministically:
|
||||
|
||||
```bash
|
||||
# 1. Get the manifest
|
||||
curl -O https://releases.stellaops.io/v1.0.0/sbom.determinism.json
|
||||
|
||||
# 2. Get the artifact
|
||||
curl -O https://releases.stellaops.io/v1.0.0/sbom.cdx.json
|
||||
|
||||
# 3. Verify
|
||||
dotnet run --project tools/DeterminismVerifier \
|
||||
--manifest sbom.determinism.json \
|
||||
--artifact sbom.cdx.json
|
||||
```
|
||||
|
||||
Output:
|
||||
```
|
||||
Determinism Verification
|
||||
========================
|
||||
Artifact: sbom.cdx.json
|
||||
Manifest: sbom.determinism.json
|
||||
Expected Hash: abc123...
|
||||
Actual Hash: abc123...
|
||||
Status: ✅ VERIFIED
|
||||
```
|
||||
|
||||
## Test Files Reference
|
||||
|
||||
All determinism tests are in `tests/integration/StellaOps.Integration.Determinism/`:
|
||||
|
||||
| File | Tests | Description |
|
||||
|------|-------|-------------|
|
||||
| `DeterminismValidationTests.cs` | 16 | Manifest format and reader/writer |
|
||||
| `SbomDeterminismTests.cs` | 14 | SPDX 3.0.1, CycloneDX 1.6/1.7 |
|
||||
| `VexDeterminismTests.cs` | 17 | OpenVEX, CSAF 2.0 |
|
||||
| `PolicyDeterminismTests.cs` | 18 | Policy verdict artifacts |
|
||||
| `EvidenceBundleDeterminismTests.cs` | 15 | DSSE, in-toto attestations |
|
||||
| `AirGapBundleDeterminismTests.cs` | 14 | NDJSON bundles, manifests |
|
||||
| `IngestionDeterminismTests.cs` | 17 | NVD/OSV/GHSA/CSAF normalization |
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Hash Mismatch
|
||||
|
||||
If you see a hash mismatch:
|
||||
|
||||
1. **Check timestamps**: Ensure frozen time is used
|
||||
2. **Check ordering**: Ensure all collections are sorted
|
||||
3. **Check IDs**: Ensure IDs are content-derived
|
||||
4. **Check encoding**: Ensure UTF-8 without BOM
|
||||
|
||||
### Flaky Tests
|
||||
|
||||
If determinism tests are flaky:
|
||||
|
||||
1. **Check parallelism**: Ensure no shared mutable state
|
||||
2. **Check time zones**: Use UTC explicitly
|
||||
3. **Check random sources**: Remove all random number generation
|
||||
4. **Check hash inputs**: Ensure all inputs are captured
|
||||
|
||||
### CI Failures
|
||||
|
||||
If CI determinism gate fails:
|
||||
|
||||
1. Compare the diff between expected and actual
|
||||
2. Identify which field changed
|
||||
3. Track back to the code change that caused it
|
||||
4. Either fix the regression or update baseline with justification
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Testing Strategy Models](testing-strategy-models.md) - Overview of testing models
|
||||
- [Canonical JSON Specification](../11_DATA_SCHEMAS.md#canonical-json) - JSON serialization rules
|
||||
- [CI/CD Workflows](../modules/devops/architecture.md) - CI pipeline details
|
||||
- [Evidence Bundle Schema](../modules/evidence-locker/architecture.md) - Bundle format reference
|
||||
@@ -0,0 +1,267 @@
|
||||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"$id": "https://stella-ops.org/schemas/determinism-manifest/v1.json",
|
||||
"title": "StellaOps Determinism Manifest",
|
||||
"description": "Manifest tracking artifact reproducibility with canonical bytes hash, version stamps, and toolchain information",
|
||||
"type": "object",
|
||||
"required": [
|
||||
"schemaVersion",
|
||||
"artifact",
|
||||
"canonicalHash",
|
||||
"toolchain",
|
||||
"generatedAt"
|
||||
],
|
||||
"properties": {
|
||||
"schemaVersion": {
|
||||
"type": "string",
|
||||
"const": "1.0",
|
||||
"description": "Version of this manifest schema"
|
||||
},
|
||||
"artifact": {
|
||||
"type": "object",
|
||||
"description": "Artifact being tracked for determinism",
|
||||
"required": ["type", "name", "version"],
|
||||
"properties": {
|
||||
"type": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"sbom",
|
||||
"vex",
|
||||
"csaf",
|
||||
"verdict",
|
||||
"evidence-bundle",
|
||||
"airgap-bundle",
|
||||
"advisory-normalized",
|
||||
"attestation",
|
||||
"other"
|
||||
],
|
||||
"description": "Type of artifact"
|
||||
},
|
||||
"name": {
|
||||
"type": "string",
|
||||
"description": "Artifact identifier or name",
|
||||
"minLength": 1
|
||||
},
|
||||
"version": {
|
||||
"type": "string",
|
||||
"description": "Artifact version or timestamp",
|
||||
"minLength": 1
|
||||
},
|
||||
"format": {
|
||||
"type": "string",
|
||||
"description": "Artifact format (e.g., 'SPDX 3.0.1', 'CycloneDX 1.6', 'OpenVEX')",
|
||||
"examples": ["SPDX 3.0.1", "CycloneDX 1.6", "OpenVEX", "CSAF 2.0"]
|
||||
},
|
||||
"metadata": {
|
||||
"type": "object",
|
||||
"description": "Additional artifact-specific metadata",
|
||||
"additionalProperties": true
|
||||
}
|
||||
}
|
||||
},
|
||||
"canonicalHash": {
|
||||
"type": "object",
|
||||
"description": "Hash of the canonical representation of the artifact",
|
||||
"required": ["algorithm", "value", "encoding"],
|
||||
"properties": {
|
||||
"algorithm": {
|
||||
"type": "string",
|
||||
"enum": ["SHA-256", "SHA-384", "SHA-512"],
|
||||
"description": "Hash algorithm used"
|
||||
},
|
||||
"value": {
|
||||
"type": "string",
|
||||
"description": "Hex-encoded hash value",
|
||||
"pattern": "^[0-9a-f]{64,128}$"
|
||||
},
|
||||
"encoding": {
|
||||
"type": "string",
|
||||
"enum": ["hex", "base64"],
|
||||
"description": "Encoding of the hash value"
|
||||
}
|
||||
}
|
||||
},
|
||||
"inputs": {
|
||||
"type": "object",
|
||||
"description": "Version stamps of all inputs used to generate the artifact",
|
||||
"properties": {
|
||||
"feedSnapshotHash": {
|
||||
"type": "string",
|
||||
"description": "SHA-256 hash of the vulnerability feed snapshot used",
|
||||
"pattern": "^[0-9a-f]{64}$"
|
||||
},
|
||||
"policyManifestHash": {
|
||||
"type": "string",
|
||||
"description": "SHA-256 hash of the policy manifest used",
|
||||
"pattern": "^[0-9a-f]{64}$"
|
||||
},
|
||||
"sourceCodeHash": {
|
||||
"type": "string",
|
||||
"description": "Git commit SHA or source code hash",
|
||||
"pattern": "^[0-9a-f]{40,64}$"
|
||||
},
|
||||
"dependencyLockfileHash": {
|
||||
"type": "string",
|
||||
"description": "Hash of dependency lockfile (e.g., package-lock.json, Cargo.lock)",
|
||||
"pattern": "^[0-9a-f]{64}$"
|
||||
},
|
||||
"baseImageDigest": {
|
||||
"type": "string",
|
||||
"description": "Container base image digest (sha256:...)",
|
||||
"pattern": "^sha256:[0-9a-f]{64}$"
|
||||
},
|
||||
"vexDocumentHashes": {
|
||||
"type": "array",
|
||||
"description": "Hashes of all VEX documents used as input",
|
||||
"items": {
|
||||
"type": "string",
|
||||
"pattern": "^[0-9a-f]{64}$"
|
||||
}
|
||||
},
|
||||
"custom": {
|
||||
"type": "object",
|
||||
"description": "Custom input hashes specific to artifact type",
|
||||
"additionalProperties": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
},
|
||||
"additionalProperties": false
|
||||
},
|
||||
"toolchain": {
|
||||
"type": "object",
|
||||
"description": "Toolchain version information",
|
||||
"required": ["platform", "components"],
|
||||
"properties": {
|
||||
"platform": {
|
||||
"type": "string",
|
||||
"description": "Runtime platform (e.g., '.NET 10.0', 'Node.js 20.0')",
|
||||
"examples": [".NET 10.0.0", "Node.js 20.11.0", "Python 3.12.1"]
|
||||
},
|
||||
"components": {
|
||||
"type": "array",
|
||||
"description": "Toolchain component versions",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"required": ["name", "version"],
|
||||
"properties": {
|
||||
"name": {
|
||||
"type": "string",
|
||||
"description": "Component name",
|
||||
"examples": ["StellaOps.Scanner", "StellaOps.Policy.Engine", "CycloneDX Generator"]
|
||||
},
|
||||
"version": {
|
||||
"type": "string",
|
||||
"description": "Semantic version or git SHA",
|
||||
"examples": ["1.2.3", "2.0.0-beta.1", "abc123def"]
|
||||
},
|
||||
"hash": {
|
||||
"type": "string",
|
||||
"description": "Optional: SHA-256 hash of the component binary",
|
||||
"pattern": "^[0-9a-f]{64}$"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"compiler": {
|
||||
"type": "object",
|
||||
"description": "Compiler information if applicable",
|
||||
"properties": {
|
||||
"name": {
|
||||
"type": "string",
|
||||
"description": "Compiler name (e.g., 'Roslyn', 'rustc')"
|
||||
},
|
||||
"version": {
|
||||
"type": "string",
|
||||
"description": "Compiler version"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"generatedAt": {
|
||||
"type": "string",
|
||||
"format": "date-time",
|
||||
"description": "UTC timestamp when artifact was generated (ISO 8601)",
|
||||
"examples": ["2025-12-23T17:45:00Z"]
|
||||
},
|
||||
"reproducibility": {
|
||||
"type": "object",
|
||||
"description": "Reproducibility metadata",
|
||||
"properties": {
|
||||
"deterministicSeed": {
|
||||
"type": "integer",
|
||||
"description": "Deterministic random seed if used",
|
||||
"minimum": 0
|
||||
},
|
||||
"clockFixed": {
|
||||
"type": "boolean",
|
||||
"description": "Whether system clock was fixed during generation"
|
||||
},
|
||||
"orderingGuarantee": {
|
||||
"type": "string",
|
||||
"enum": ["stable", "sorted", "insertion", "unspecified"],
|
||||
"description": "Ordering guarantee for collections in output"
|
||||
},
|
||||
"normalizationRules": {
|
||||
"type": "array",
|
||||
"description": "Normalization rules applied (e.g., 'UTF-8', 'LF line endings', 'no whitespace')",
|
||||
"items": {
|
||||
"type": "string"
|
||||
},
|
||||
"examples": [
|
||||
["UTF-8 encoding", "LF line endings", "sorted JSON keys", "no trailing whitespace"]
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"verification": {
|
||||
"type": "object",
|
||||
"description": "Verification instructions for reproducing the artifact",
|
||||
"properties": {
|
||||
"command": {
|
||||
"type": "string",
|
||||
"description": "Command to regenerate the artifact",
|
||||
"examples": ["dotnet run --project Scanner -- scan container alpine:3.18"]
|
||||
},
|
||||
"expectedHash": {
|
||||
"type": "string",
|
||||
"description": "Expected SHA-256 hash after reproduction",
|
||||
"pattern": "^[0-9a-f]{64}$"
|
||||
},
|
||||
"baseline": {
|
||||
"type": "string",
|
||||
"description": "Baseline manifest file path for regression testing",
|
||||
"examples": ["tests/baselines/sbom-alpine-3.18.determinism.json"]
|
||||
}
|
||||
}
|
||||
},
|
||||
"signatures": {
|
||||
"type": "array",
|
||||
"description": "Optional cryptographic signatures of this manifest",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"required": ["algorithm", "keyId", "signature"],
|
||||
"properties": {
|
||||
"algorithm": {
|
||||
"type": "string",
|
||||
"description": "Signature algorithm (e.g., 'ES256', 'RS256')"
|
||||
},
|
||||
"keyId": {
|
||||
"type": "string",
|
||||
"description": "Key identifier used for signing"
|
||||
},
|
||||
"signature": {
|
||||
"type": "string",
|
||||
"description": "Base64-encoded signature"
|
||||
},
|
||||
"timestamp": {
|
||||
"type": "string",
|
||||
"format": "date-time",
|
||||
"description": "UTC timestamp when signature was created"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
613
docs/testing/testkit-usage-guide.md
Normal file
613
docs/testing/testkit-usage-guide.md
Normal file
@@ -0,0 +1,613 @@
|
||||
# StellaOps.TestKit Usage Guide
|
||||
|
||||
**Version:** 1.0
|
||||
**Status:** Pilot Release (Wave 4 Complete)
|
||||
**Audience:** StellaOps developers writing unit, integration, and contract tests
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
`StellaOps.TestKit` provides deterministic testing infrastructure for StellaOps modules. It eliminates flaky tests, provides reproducible test primitives, and standardizes fixtures for integration testing.
|
||||
|
||||
### Key Features
|
||||
|
||||
- **Deterministic Time**: Freeze and advance time for reproducible tests
|
||||
- **Deterministic Random**: Seeded random number generation
|
||||
- **Canonical JSON Assertions**: SHA-256 hash verification for determinism
|
||||
- **Snapshot Testing**: Golden master regression testing
|
||||
- **PostgreSQL Fixture**: Testcontainers-based PostgreSQL 16 for integration tests
|
||||
- **Valkey Fixture**: Redis-compatible caching tests
|
||||
- **HTTP Fixture**: In-memory API contract testing
|
||||
- **OpenTelemetry Capture**: Trace and span assertion helpers
|
||||
- **Test Categories**: Standardized trait constants for CI filtering
|
||||
|
||||
---
|
||||
|
||||
## Installation
|
||||
|
||||
Add `StellaOps.TestKit` as a project reference to your test project:
|
||||
|
||||
```xml
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="../../../__Libraries/StellaOps.TestKit/StellaOps.TestKit.csproj" />
|
||||
</ItemGroup>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Quick Start Examples
|
||||
|
||||
### 1. Deterministic Time
|
||||
|
||||
Eliminate flaky tests caused by time-dependent logic:
|
||||
|
||||
```csharp
|
||||
using StellaOps.TestKit.Deterministic;
|
||||
using Xunit;
|
||||
|
||||
[Fact]
|
||||
public void Test_ExpirationLogic()
|
||||
{
|
||||
// Arrange: Fix time at a known UTC timestamp
|
||||
using var time = new DeterministicTime(new DateTime(2026, 1, 15, 10, 30, 0, DateTimeKind.Utc));
|
||||
|
||||
var expiresAt = time.UtcNow.AddHours(24);
|
||||
|
||||
// Act: Advance time to just before expiration
|
||||
time.Advance(TimeSpan.FromHours(23));
|
||||
Assert.False(time.UtcNow > expiresAt);
|
||||
|
||||
// Advance past expiration
|
||||
time.Advance(TimeSpan.FromHours(2));
|
||||
Assert.True(time.UtcNow > expiresAt);
|
||||
}
|
||||
```
|
||||
|
||||
**API Reference:**
|
||||
- `DeterministicTime(DateTime initialUtc)` - Create with fixed start time
|
||||
- `UtcNow` - Get current deterministic time
|
||||
- `Advance(TimeSpan duration)` - Move time forward
|
||||
- `SetTo(DateTime newUtc)` - Jump to specific time
|
||||
|
||||
---
|
||||
|
||||
### 2. Deterministic Random
|
||||
|
||||
Reproducible random sequences for property tests and fuzzing:
|
||||
|
||||
```csharp
|
||||
using StellaOps.TestKit.Deterministic;
|
||||
|
||||
[Fact]
|
||||
public void Test_RandomIdGeneration()
|
||||
{
|
||||
// Arrange: Same seed produces same sequence
|
||||
var random1 = new DeterministicRandom(seed: 42);
|
||||
var random2 = new DeterministicRandom(seed: 42);
|
||||
|
||||
// Act
|
||||
var guid1 = random1.NextGuid();
|
||||
var guid2 = random2.NextGuid();
|
||||
|
||||
// Assert: Reproducible GUIDs
|
||||
Assert.Equal(guid1, guid2);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Test_Shuffling()
|
||||
{
|
||||
var random = new DeterministicRandom(seed: 100);
|
||||
var array = new[] { 1, 2, 3, 4, 5 };
|
||||
|
||||
random.Shuffle(array);
|
||||
|
||||
// Deterministic shuffle order
|
||||
Assert.NotEqual(new[] { 1, 2, 3, 4, 5 }, array);
|
||||
}
|
||||
```
|
||||
|
||||
**API Reference:**
|
||||
- `DeterministicRandom(int seed)` - Create with seed
|
||||
- `NextGuid()` - Generate deterministic GUID
|
||||
- `NextString(int length)` - Generate alphanumeric string
|
||||
- `NextInt(int min, int max)` - Generate integer in range
|
||||
- `Shuffle<T>(T[] array)` - Fisher-Yates shuffle
|
||||
|
||||
---
|
||||
|
||||
### 3. Canonical JSON Assertions
|
||||
|
||||
Verify JSON determinism for SBOM, VEX, and attestation outputs:
|
||||
|
||||
```csharp
|
||||
using StellaOps.TestKit.Assertions;
|
||||
|
||||
[Fact]
|
||||
public void Test_SbomDeterminism()
|
||||
{
|
||||
var sbom = new
|
||||
{
|
||||
SpdxVersion = "SPDX-3.0.1",
|
||||
Name = "MySbom",
|
||||
Packages = new[] { new { Name = "Pkg1", Version = "1.0" } }
|
||||
};
|
||||
|
||||
// Verify deterministic serialization
|
||||
CanonicalJsonAssert.IsDeterministic(sbom, iterations: 100);
|
||||
|
||||
// Verify expected hash (golden master)
|
||||
var expectedHash = "abc123..."; // Precomputed SHA-256
|
||||
CanonicalJsonAssert.HasExpectedHash(sbom, expectedHash);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Test_JsonPropertyExists()
|
||||
{
|
||||
var vex = new
|
||||
{
|
||||
Document = new { Id = "VEX-2026-001" },
|
||||
Statements = new[] { new { Vulnerability = "CVE-2026-1234" } }
|
||||
};
|
||||
|
||||
// Deep property verification
|
||||
CanonicalJsonAssert.ContainsProperty(vex, "Document.Id", "VEX-2026-001");
|
||||
CanonicalJsonAssert.ContainsProperty(vex, "Statements[0].Vulnerability", "CVE-2026-1234");
|
||||
}
|
||||
```
|
||||
|
||||
**API Reference:**
|
||||
- `IsDeterministic<T>(T value, int iterations)` - Verify N serializations match
|
||||
- `HasExpectedHash<T>(T value, string expectedSha256Hex)` - Verify SHA-256 hash
|
||||
- `ComputeCanonicalHash<T>(T value)` - Compute hash for golden master
|
||||
- `AreCanonicallyEqual<T>(T expected, T actual)` - Compare canonical JSON
|
||||
- `ContainsProperty<T>(T value, string propertyPath, object expectedValue)` - Deep search
|
||||
|
||||
---
|
||||
|
||||
### 4. Snapshot Testing
|
||||
|
||||
Golden master regression testing for complex outputs:
|
||||
|
||||
```csharp
|
||||
using StellaOps.TestKit.Assertions;
|
||||
|
||||
[Fact, Trait("Category", TestCategories.Snapshot)]
|
||||
public void Test_SbomGeneration()
|
||||
{
|
||||
var sbom = GenerateSbom(); // Your SBOM generation logic
|
||||
|
||||
// Snapshot will be stored in Snapshots/TestSbomGeneration.json
|
||||
SnapshotAssert.MatchesSnapshot(sbom, "TestSbomGeneration");
|
||||
}
|
||||
|
||||
// Update snapshots when intentional changes occur:
|
||||
// UPDATE_SNAPSHOTS=1 dotnet test
|
||||
```
|
||||
|
||||
**Text and Binary Snapshots:**
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public void Test_LicenseText()
|
||||
{
|
||||
var licenseText = GenerateLicenseNotice();
|
||||
SnapshotAssert.MatchesTextSnapshot(licenseText, "LicenseNotice");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Test_SignatureBytes()
|
||||
{
|
||||
var signature = SignDocument(document);
|
||||
SnapshotAssert.MatchesBinarySnapshot(signature, "DocumentSignature");
|
||||
}
|
||||
```
|
||||
|
||||
**API Reference:**
|
||||
- `MatchesSnapshot<T>(T value, string snapshotName)` - JSON snapshot
|
||||
- `MatchesTextSnapshot(string value, string snapshotName)` - Text snapshot
|
||||
- `MatchesBinarySnapshot(byte[] value, string snapshotName)` - Binary snapshot
|
||||
- Environment variable: `UPDATE_SNAPSHOTS=1` to update baselines
|
||||
|
||||
---
|
||||
|
||||
### 5. PostgreSQL Fixture
|
||||
|
||||
Testcontainers-based PostgreSQL 16 for integration tests:
|
||||
|
||||
```csharp
|
||||
using StellaOps.TestKit.Fixtures;
|
||||
using Xunit;
|
||||
|
||||
public class DatabaseTests : IClassFixture<PostgresFixture>
|
||||
{
|
||||
private readonly PostgresFixture _fixture;
|
||||
|
||||
public DatabaseTests(PostgresFixture fixture)
|
||||
{
|
||||
_fixture = fixture;
|
||||
}
|
||||
|
||||
[Fact, Trait("Category", TestCategories.Integration)]
|
||||
public async Task Test_DatabaseOperations()
|
||||
{
|
||||
// Use _fixture.ConnectionString to connect
|
||||
using var connection = new NpgsqlConnection(_fixture.ConnectionString);
|
||||
await connection.OpenAsync();
|
||||
|
||||
// Run migrations
|
||||
await _fixture.RunMigrationsAsync(connection);
|
||||
|
||||
// Test database operations
|
||||
var result = await connection.QueryAsync("SELECT version()");
|
||||
Assert.NotEmpty(result);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**API Reference:**
|
||||
- `PostgresFixture` - xUnit class fixture
|
||||
- `ConnectionString` - PostgreSQL connection string
|
||||
- `RunMigrationsAsync(DbConnection)` - Apply migrations
|
||||
- Requires Docker running locally
|
||||
|
||||
---
|
||||
|
||||
### 6. Valkey Fixture
|
||||
|
||||
Redis-compatible caching for integration tests:
|
||||
|
||||
```csharp
|
||||
using StellaOps.TestKit.Fixtures;
|
||||
|
||||
public class CacheTests : IClassFixture<ValkeyFixture>
|
||||
{
|
||||
private readonly ValkeyFixture _fixture;
|
||||
|
||||
[Fact, Trait("Category", TestCategories.Integration)]
|
||||
public async Task Test_CachingLogic()
|
||||
{
|
||||
var connection = await ConnectionMultiplexer.Connect(_fixture.ConnectionString);
|
||||
var db = connection.GetDatabase();
|
||||
|
||||
await db.StringSetAsync("key", "value");
|
||||
var result = await db.StringGetAsync("key");
|
||||
|
||||
Assert.Equal("value", result.ToString());
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**API Reference:**
|
||||
- `ValkeyFixture` - xUnit class fixture
|
||||
- `ConnectionString` - Redis connection string (host:port)
|
||||
- `Host`, `Port` - Connection details
|
||||
- Uses `redis:7-alpine` image (Valkey-compatible)
|
||||
|
||||
---
|
||||
|
||||
### 7. HTTP Fixture Server
|
||||
|
||||
In-memory API contract testing:
|
||||
|
||||
```csharp
|
||||
using StellaOps.TestKit.Fixtures;
|
||||
|
||||
public class ApiTests : IClassFixture<HttpFixtureServer<Program>>
|
||||
{
|
||||
private readonly HttpClient _client;
|
||||
|
||||
public ApiTests(HttpFixtureServer<Program> fixture)
|
||||
{
|
||||
_client = fixture.CreateClient();
|
||||
}
|
||||
|
||||
[Fact, Trait("Category", TestCategories.Contract)]
|
||||
public async Task Test_HealthEndpoint()
|
||||
{
|
||||
var response = await _client.GetAsync("/health");
|
||||
response.EnsureSuccessStatusCode();
|
||||
|
||||
var body = await response.Content.ReadAsStringAsync();
|
||||
Assert.Contains("healthy", body);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**HTTP Message Handler Stub (Hermetic Tests):**
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public async Task Test_ExternalApiCall()
|
||||
{
|
||||
var handler = new HttpMessageHandlerStub()
|
||||
.WhenRequest("https://api.example.com/data", HttpStatusCode.OK, "{\"status\":\"ok\"}");
|
||||
|
||||
var httpClient = new HttpClient(handler);
|
||||
var response = await httpClient.GetAsync("https://api.example.com/data");
|
||||
|
||||
Assert.Equal(HttpStatusCode.OK, response.StatusCode);
|
||||
}
|
||||
```
|
||||
|
||||
**API Reference:**
|
||||
- `HttpFixtureServer<TProgram>` - WebApplicationFactory wrapper
|
||||
- `CreateClient()` - Get HttpClient for test server
|
||||
- `HttpMessageHandlerStub` - Stub external HTTP dependencies
|
||||
- `WhenRequest(url, statusCode, content)` - Configure stub responses
|
||||
|
||||
---
|
||||
|
||||
### 8. OpenTelemetry Capture
|
||||
|
||||
Trace and span assertion helpers:
|
||||
|
||||
```csharp
|
||||
using StellaOps.TestKit.Observability;
|
||||
|
||||
[Fact]
|
||||
public async Task Test_TracingBehavior()
|
||||
{
|
||||
using var capture = new OtelCapture();
|
||||
|
||||
// Execute code that emits traces
|
||||
await MyService.DoWorkAsync();
|
||||
|
||||
// Assert traces
|
||||
capture.AssertHasSpan("MyService.DoWork");
|
||||
capture.AssertHasTag("user_id", "123");
|
||||
capture.AssertSpanCount(expectedCount: 3);
|
||||
|
||||
// Verify parent-child hierarchy
|
||||
capture.AssertHierarchy("ParentSpan", "ChildSpan");
|
||||
}
|
||||
```
|
||||
|
||||
**API Reference:**
|
||||
- `OtelCapture(string? activitySourceName = null)` - Create capture
|
||||
- `AssertHasSpan(string spanName)` - Verify span exists
|
||||
- `AssertHasTag(string tagKey, string expectedValue)` - Verify tag
|
||||
- `AssertSpanCount(int expectedCount)` - Verify span count
|
||||
- `AssertHierarchy(string parentSpanName, string childSpanName)` - Verify parent-child
|
||||
- `CapturedActivities` - Get all captured spans
|
||||
|
||||
---
|
||||
|
||||
### 9. Test Categories
|
||||
|
||||
Standardized trait constants for CI lane filtering:
|
||||
|
||||
```csharp
|
||||
using StellaOps.TestKit;
|
||||
|
||||
[Fact, Trait("Category", TestCategories.Unit)]
|
||||
public void FastUnitTest() { }
|
||||
|
||||
[Fact, Trait("Category", TestCategories.Integration)]
|
||||
public async Task SlowIntegrationTest() { }
|
||||
|
||||
[Fact, Trait("Category", TestCategories.Live)]
|
||||
public async Task RequiresExternalServices() { }
|
||||
```
|
||||
|
||||
**CI Lane Filtering:**
|
||||
|
||||
```bash
|
||||
# Run only unit tests (fast, no dependencies)
|
||||
dotnet test --filter "Category=Unit"
|
||||
|
||||
# Run all tests except Live
|
||||
dotnet test --filter "Category!=Live"
|
||||
|
||||
# Run Integration + Contract tests
|
||||
dotnet test --filter "Category=Integration|Category=Contract"
|
||||
```
|
||||
|
||||
**Available Categories:**
|
||||
- `Unit` - Fast, in-memory, no external dependencies
|
||||
- `Property` - FsCheck/generative testing
|
||||
- `Snapshot` - Golden master regression
|
||||
- `Integration` - Testcontainers (PostgreSQL, Valkey)
|
||||
- `Contract` - API/WebService contract tests
|
||||
- `Security` - Cryptographic validation
|
||||
- `Performance` - Benchmarking, load tests
|
||||
- `Live` - Requires external services (disabled in CI by default)
|
||||
|
||||
---
|
||||
|
||||
## Best Practices
|
||||
|
||||
### 1. Always Use TestCategories
|
||||
|
||||
Tag every test with the appropriate category:
|
||||
|
||||
```csharp
|
||||
[Fact, Trait("Category", TestCategories.Unit)]
|
||||
public void MyUnitTest() { }
|
||||
```
|
||||
|
||||
This enables CI lane filtering and improves test discoverability.
|
||||
|
||||
### 2. Prefer Deterministic Primitives
|
||||
|
||||
Avoid `DateTime.UtcNow`, `Guid.NewGuid()`, `Random` in tests. Use TestKit alternatives:
|
||||
|
||||
```csharp
|
||||
// ❌ Flaky test (time-dependent)
|
||||
var expiration = DateTime.UtcNow.AddHours(1);
|
||||
|
||||
// ✅ Deterministic test
|
||||
using var time = new DeterministicTime(DateTime.UtcNow);
|
||||
var expiration = time.UtcNow.AddHours(1);
|
||||
```
|
||||
|
||||
### 3. Use Snapshot Tests for Complex Outputs
|
||||
|
||||
For large JSON outputs (SBOM, VEX, attestations), snapshot testing is more maintainable than manual assertions:
|
||||
|
||||
```csharp
|
||||
// ❌ Brittle manual assertions
|
||||
Assert.Equal("SPDX-3.0.1", sbom.SpdxVersion);
|
||||
Assert.Equal(42, sbom.Packages.Count);
|
||||
// ...hundreds of assertions...
|
||||
|
||||
// ✅ Snapshot testing
|
||||
SnapshotAssert.MatchesSnapshot(sbom, "MySbomSnapshot");
|
||||
```
|
||||
|
||||
### 4. Isolate Integration Tests
|
||||
|
||||
Use TestCategories to separate fast unit tests from slow integration tests:
|
||||
|
||||
```csharp
|
||||
[Fact, Trait("Category", TestCategories.Unit)]
|
||||
public void FastTest() { /* no external dependencies */ }
|
||||
|
||||
[Fact, Trait("Category", TestCategories.Integration)]
|
||||
public async Task SlowTest() { /* uses PostgresFixture */ }
|
||||
```
|
||||
|
||||
In CI, run Unit tests first for fast feedback, then Integration tests in parallel.
|
||||
|
||||
### 5. Document Snapshot Baselines
|
||||
|
||||
When updating snapshots (`UPDATE_SNAPSHOTS=1`), add a commit message explaining why:
|
||||
|
||||
```bash
|
||||
git commit -m "Update SBOM snapshot: added new package metadata fields"
|
||||
```
|
||||
|
||||
This helps reviewers understand intentional vs. accidental changes.
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Snapshot Mismatch
|
||||
|
||||
**Error:** `Snapshot 'MySbomSnapshot' does not match expected.`
|
||||
|
||||
**Solution:**
|
||||
1. Review diff manually (check `Snapshots/MySbomSnapshot.json`)
|
||||
2. If change is intentional: `UPDATE_SNAPSHOTS=1 dotnet test`
|
||||
3. Commit updated snapshot with explanation
|
||||
|
||||
### Testcontainers Failure
|
||||
|
||||
**Error:** `Docker daemon not running`
|
||||
|
||||
**Solution:**
|
||||
- Ensure Docker Desktop is running
|
||||
- Verify `docker ps` works in terminal
|
||||
- Check Testcontainers logs: `TESTCONTAINERS_DEBUG=1 dotnet test`
|
||||
|
||||
### Determinism Failure
|
||||
|
||||
**Error:** `CanonicalJsonAssert.IsDeterministic failed: byte arrays differ`
|
||||
|
||||
**Root Cause:** Non-deterministic data in serialization (e.g., random GUIDs, timestamps)
|
||||
|
||||
**Solution:**
|
||||
- Use `DeterministicTime` and `DeterministicRandom`
|
||||
- Ensure all data is seeded or mocked
|
||||
- Check for `DateTime.UtcNow` or `Guid.NewGuid()` calls
|
||||
|
||||
---
|
||||
|
||||
## Migration Guide (Existing Tests)
|
||||
|
||||
### Step 1: Add TestKit Reference
|
||||
|
||||
```xml
|
||||
<ProjectReference Include="../../../__Libraries/StellaOps.TestKit/StellaOps.TestKit.csproj" />
|
||||
```
|
||||
|
||||
### Step 2: Replace Time-Dependent Code
|
||||
|
||||
**Before:**
|
||||
```csharp
|
||||
var now = DateTime.UtcNow;
|
||||
```
|
||||
|
||||
**After:**
|
||||
```csharp
|
||||
using var time = new DeterministicTime(DateTime.UtcNow);
|
||||
var now = time.UtcNow;
|
||||
```
|
||||
|
||||
### Step 3: Add Test Categories
|
||||
|
||||
```csharp
|
||||
[Fact] // Old
|
||||
[Fact, Trait("Category", TestCategories.Unit)] // New
|
||||
```
|
||||
|
||||
### Step 4: Adopt Snapshot Testing (Optional)
|
||||
|
||||
For complex JSON assertions, replace manual checks with snapshots:
|
||||
|
||||
```csharp
|
||||
// Old
|
||||
Assert.Equal(expected.SpdxVersion, actual.SpdxVersion);
|
||||
// ...
|
||||
|
||||
// New
|
||||
SnapshotAssert.MatchesSnapshot(actual, "TestName");
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## CI Integration
|
||||
|
||||
### Example `.gitea/workflows/test.yml`
|
||||
|
||||
```yaml
|
||||
name: Test Suite
|
||||
on: [push, pull_request]
|
||||
|
||||
jobs:
|
||||
unit:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: actions/setup-dotnet@v4
|
||||
with:
|
||||
dotnet-version: '10.0.x'
|
||||
- name: Unit Tests (Fast)
|
||||
run: dotnet test --filter "Category=Unit" --logger "trx;LogFileName=unit-results.trx"
|
||||
- name: Upload Results
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: unit-test-results
|
||||
path: '**/unit-results.trx'
|
||||
|
||||
integration:
|
||||
runs-on: ubuntu-latest
|
||||
services:
|
||||
docker:
|
||||
image: docker:dind
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- name: Integration Tests
|
||||
run: dotnet test --filter "Category=Integration" --logger "trx;LogFileName=integration-results.trx"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Support and Feedback
|
||||
|
||||
- **Issues:** Report bugs in sprint tracking files under `docs/implplan/`
|
||||
- **Questions:** Contact Platform Guild
|
||||
- **Documentation:** `src/__Libraries/StellaOps.TestKit/README.md`
|
||||
|
||||
---
|
||||
|
||||
## Changelog
|
||||
|
||||
### v1.0 (2025-12-23)
|
||||
- Initial release: DeterministicTime, DeterministicRandom
|
||||
- CanonicalJsonAssert, SnapshotAssert
|
||||
- PostgresFixture, ValkeyFixture, HttpFixtureServer
|
||||
- OtelCapture for OpenTelemetry traces
|
||||
- TestCategories for CI lane filtering
|
||||
- Pilot adoption in Scanner.Core.Tests
|
||||
366
docs/testing/webservice-test-discipline.md
Normal file
366
docs/testing/webservice-test-discipline.md
Normal file
@@ -0,0 +1,366 @@
|
||||
# WebService Test Discipline
|
||||
|
||||
This document defines the testing discipline for StellaOps WebService projects. All web services must follow these patterns to ensure consistent test coverage, contract stability, telemetry verification, and security hardening.
|
||||
|
||||
## Overview
|
||||
|
||||
WebService tests use `WebServiceFixture<TProgram>` from `StellaOps.TestKit` and `WebApplicationFactory<TProgram>` from `Microsoft.AspNetCore.Mvc.Testing`. Tests are organized into four categories:
|
||||
|
||||
1. **Contract Tests** — OpenAPI schema stability
|
||||
2. **OTel Trace Tests** — Telemetry verification
|
||||
3. **Negative Tests** — Error handling validation
|
||||
4. **Auth/AuthZ Tests** — Security boundary enforcement
|
||||
|
||||
---
|
||||
|
||||
## 1. Test Infrastructure
|
||||
|
||||
### WebServiceFixture Pattern
|
||||
|
||||
```csharp
|
||||
using StellaOps.TestKit.Fixtures;
|
||||
|
||||
public class ScannerWebServiceTests : WebServiceTestBase<ScannerProgram>
|
||||
{
|
||||
public ScannerWebServiceTests() : base(new WebServiceFixture<ScannerProgram>())
|
||||
{
|
||||
}
|
||||
|
||||
// Tests inherit shared fixture setup
|
||||
}
|
||||
```
|
||||
|
||||
### Fixture Configuration
|
||||
|
||||
Each web service should have a dedicated fixture class that configures test-specific settings:
|
||||
|
||||
```csharp
|
||||
public sealed class ScannerTestFixture : WebServiceFixture<ScannerProgram>
|
||||
{
|
||||
protected override void ConfigureTestServices(IServiceCollection services)
|
||||
{
|
||||
// Replace external dependencies with test doubles
|
||||
services.AddSingleton<IStorageClient, InMemoryStorageClient>();
|
||||
services.AddSingleton<IQueueClient, InMemoryQueueClient>();
|
||||
}
|
||||
|
||||
protected override void ConfigureTestConfiguration(IDictionary<string, string?> config)
|
||||
{
|
||||
config["scanner:storage:driver"] = "inmemory";
|
||||
config["scanner:events:enabled"] = "false";
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Contract Tests
|
||||
|
||||
Contract tests ensure OpenAPI schema stability and detect breaking changes.
|
||||
|
||||
### Pattern
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
[Trait("Lane", "Contract")]
|
||||
public async Task OpenApi_Schema_MatchesSnapshot()
|
||||
{
|
||||
// Arrange
|
||||
using var client = Fixture.CreateClient();
|
||||
|
||||
// Act
|
||||
var response = await client.GetAsync("/swagger/v1/swagger.json");
|
||||
var schema = await response.Content.ReadAsStringAsync();
|
||||
|
||||
// Assert
|
||||
await ContractTestHelper.AssertSchemaMatchesSnapshot(schema, "scanner-v1");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
[Trait("Lane", "Contract")]
|
||||
public async Task Api_Response_MatchesContract()
|
||||
{
|
||||
// Arrange
|
||||
using var client = Fixture.CreateClient();
|
||||
var request = new ScanRequest { /* test data */ };
|
||||
|
||||
// Act
|
||||
var response = await client.PostAsJsonAsync("/api/v1/scans", request);
|
||||
var result = await response.Content.ReadFromJsonAsync<ScanResponse>();
|
||||
|
||||
// Assert
|
||||
ContractTestHelper.AssertResponseMatchesSchema(result, "ScanResponse");
|
||||
}
|
||||
```
|
||||
|
||||
### Snapshot Management
|
||||
|
||||
- Snapshots stored in `Snapshots/` directory relative to test project
|
||||
- Schema format: `<service>-<version>.json`
|
||||
- Update snapshots intentionally when breaking changes are approved
|
||||
|
||||
---
|
||||
|
||||
## 3. OTel Trace Tests
|
||||
|
||||
OTel tests verify that telemetry spans are emitted correctly with required tags.
|
||||
|
||||
### Pattern
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
[Trait("Lane", "Integration")]
|
||||
public async Task ScanEndpoint_EmitsOtelTrace()
|
||||
{
|
||||
// Arrange
|
||||
using var otelCapture = Fixture.CaptureOtelTraces();
|
||||
using var client = Fixture.CreateClient();
|
||||
var request = new ScanRequest { ImageRef = "nginx:1.25" };
|
||||
|
||||
// Act
|
||||
await client.PostAsJsonAsync("/api/v1/scans", request);
|
||||
|
||||
// Assert
|
||||
otelCapture.AssertHasSpan("scanner.scan");
|
||||
otelCapture.AssertHasTag("scanner.scan", "scan.image_ref", "nginx:1.25");
|
||||
otelCapture.AssertHasTag("scanner.scan", "tenant.id", ExpectedTenantId);
|
||||
}
|
||||
```
|
||||
|
||||
### Required Tags
|
||||
|
||||
All WebService endpoints must emit these tags:
|
||||
|
||||
| Tag | Description | Example |
|
||||
|-----|-------------|---------|
|
||||
| `tenant.id` | Tenant identifier | `tenant-a` |
|
||||
| `request.id` | Correlation ID | `req-abc123` |
|
||||
| `http.route` | Endpoint route | `/api/v1/scans` |
|
||||
| `http.status_code` | Response code | `200` |
|
||||
|
||||
Service-specific tags are documented in each module's architecture doc.
|
||||
|
||||
---
|
||||
|
||||
## 4. Negative Tests
|
||||
|
||||
Negative tests verify proper error handling for invalid inputs.
|
||||
|
||||
### Pattern
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
[Trait("Lane", "Security")]
|
||||
public async Task MalformedContentType_Returns415()
|
||||
{
|
||||
// Arrange
|
||||
using var client = Fixture.CreateClient();
|
||||
var content = new StringContent("{}", Encoding.UTF8, "text/plain");
|
||||
|
||||
// Act
|
||||
var response = await client.PostAsync("/api/v1/scans", content);
|
||||
|
||||
// Assert
|
||||
Assert.Equal(HttpStatusCode.UnsupportedMediaType, response.StatusCode);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
[Trait("Lane", "Security")]
|
||||
public async Task OversizedPayload_Returns413()
|
||||
{
|
||||
// Arrange
|
||||
using var client = Fixture.CreateClient();
|
||||
var payload = new string('x', 10_000_001); // Exceeds 10MB limit
|
||||
var content = new StringContent(payload, Encoding.UTF8, "application/json");
|
||||
|
||||
// Act
|
||||
var response = await client.PostAsync("/api/v1/scans", content);
|
||||
|
||||
// Assert
|
||||
Assert.Equal(HttpStatusCode.RequestEntityTooLarge, response.StatusCode);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
[Trait("Lane", "Unit")]
|
||||
public async Task MethodMismatch_Returns405()
|
||||
{
|
||||
// Arrange
|
||||
using var client = Fixture.CreateClient();
|
||||
|
||||
// Act (POST endpoint, but using GET)
|
||||
var response = await client.GetAsync("/api/v1/scans");
|
||||
|
||||
// Assert
|
||||
Assert.Equal(HttpStatusCode.MethodNotAllowed, response.StatusCode);
|
||||
}
|
||||
```
|
||||
|
||||
### Required Coverage
|
||||
|
||||
| Negative Case | Expected Status | Test Trait |
|
||||
|--------------|-----------------|------------|
|
||||
| Malformed content type | 415 | Security |
|
||||
| Oversized payload | 413 | Security |
|
||||
| Method mismatch | 405 | Unit |
|
||||
| Missing required field | 400 | Unit |
|
||||
| Invalid field value | 400 | Unit |
|
||||
| Unknown route | 404 | Unit |
|
||||
|
||||
---
|
||||
|
||||
## 5. Auth/AuthZ Tests
|
||||
|
||||
Auth tests verify security boundaries and tenant isolation.
|
||||
|
||||
### Pattern
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
[Trait("Lane", "Security")]
|
||||
public async Task AnonymousRequest_Returns401()
|
||||
{
|
||||
// Arrange
|
||||
using var client = Fixture.CreateClient(); // No auth
|
||||
|
||||
// Act
|
||||
var response = await client.GetAsync("/api/v1/scans");
|
||||
|
||||
// Assert
|
||||
Assert.Equal(HttpStatusCode.Unauthorized, response.StatusCode);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
[Trait("Lane", "Security")]
|
||||
public async Task ExpiredToken_Returns401()
|
||||
{
|
||||
// Arrange
|
||||
using var client = Fixture.CreateAuthenticatedClient(tokenExpired: true);
|
||||
|
||||
// Act
|
||||
var response = await client.GetAsync("/api/v1/scans");
|
||||
|
||||
// Assert
|
||||
Assert.Equal(HttpStatusCode.Unauthorized, response.StatusCode);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
[Trait("Lane", "Security")]
|
||||
public async Task TenantIsolation_CannotAccessOtherTenantData()
|
||||
{
|
||||
// Arrange
|
||||
using var tenantAClient = Fixture.CreateTenantClient("tenant-a");
|
||||
using var tenantBClient = Fixture.CreateTenantClient("tenant-b");
|
||||
|
||||
// Create scan as tenant A
|
||||
var scanResponse = await tenantAClient.PostAsJsonAsync("/api/v1/scans", new ScanRequest { /* */ });
|
||||
var scan = await scanResponse.Content.ReadFromJsonAsync<ScanResponse>();
|
||||
|
||||
// Act: Try to access as tenant B
|
||||
var response = await tenantBClient.GetAsync($"/api/v1/scans/{scan!.Id}");
|
||||
|
||||
// Assert
|
||||
Assert.Equal(HttpStatusCode.NotFound, response.StatusCode); // Tenant isolation
|
||||
}
|
||||
```
|
||||
|
||||
### Required Coverage
|
||||
|
||||
| Auth Case | Expected Behavior | Test Trait |
|
||||
|-----------|-------------------|------------|
|
||||
| No token | 401 Unauthorized | Security |
|
||||
| Expired token | 401 Unauthorized | Security |
|
||||
| Invalid signature | 401 Unauthorized | Security |
|
||||
| Wrong audience | 401 Unauthorized | Security |
|
||||
| Missing scope | 403 Forbidden | Security |
|
||||
| Cross-tenant access | 404 Not Found or 403 Forbidden | Security |
|
||||
|
||||
---
|
||||
|
||||
## 6. Test Organization
|
||||
|
||||
### Directory Structure
|
||||
|
||||
```
|
||||
src/<Module>/__Tests/StellaOps.<Module>.WebService.Tests/
|
||||
├── StellaOps.<Module>.WebService.Tests.csproj
|
||||
├── <Module>ApplicationFactory.cs # WebApplicationFactory implementation
|
||||
├── <Module>TestFixture.cs # Shared test fixture
|
||||
├── Contract/
|
||||
│ └── OpenApiSchemaTests.cs
|
||||
├── Telemetry/
|
||||
│ └── OtelTraceTests.cs
|
||||
├── Negative/
|
||||
│ ├── ContentTypeTests.cs
|
||||
│ ├── PayloadLimitTests.cs
|
||||
│ └── MethodMismatchTests.cs
|
||||
├── Auth/
|
||||
│ ├── AuthenticationTests.cs
|
||||
│ ├── AuthorizationTests.cs
|
||||
│ └── TenantIsolationTests.cs
|
||||
└── Snapshots/
|
||||
└── <module>-v1.json # OpenAPI schema snapshot
|
||||
```
|
||||
|
||||
### Test Trait Assignment
|
||||
|
||||
| Category | Trait | CI Lane | PR-Gating |
|
||||
|----------|-------|---------|-----------|
|
||||
| Contract | `[Trait("Lane", "Contract")]` | Contract | Yes |
|
||||
| OTel | `[Trait("Lane", "Integration")]` | Integration | Yes |
|
||||
| Negative (security) | `[Trait("Lane", "Security")]` | Security | Yes |
|
||||
| Negative (validation) | `[Trait("Lane", "Unit")]` | Unit | Yes |
|
||||
| Auth/AuthZ | `[Trait("Lane", "Security")]` | Security | Yes |
|
||||
|
||||
---
|
||||
|
||||
## 7. CI Integration
|
||||
|
||||
WebService tests run in the appropriate CI lanes:
|
||||
|
||||
```yaml
|
||||
# .gitea/workflows/test-lanes.yml
|
||||
jobs:
|
||||
contract-tests:
|
||||
steps:
|
||||
- run: ./scripts/test-lane.sh Contract
|
||||
|
||||
security-tests:
|
||||
steps:
|
||||
- run: ./scripts/test-lane.sh Security
|
||||
|
||||
integration-tests:
|
||||
steps:
|
||||
- run: ./scripts/test-lane.sh Integration
|
||||
```
|
||||
|
||||
All lanes are PR-gating. Failed tests block merge.
|
||||
|
||||
---
|
||||
|
||||
## 8. Rollout Checklist
|
||||
|
||||
When adding WebService tests to a new module:
|
||||
|
||||
- [ ] Create `<Module>ApplicationFactory` extending `WebApplicationFactory<TProgram>`
|
||||
- [ ] Create `<Module>TestFixture` extending `WebServiceFixture<TProgram>` if needed
|
||||
- [ ] Add contract tests with OpenAPI schema snapshot
|
||||
- [ ] Add OTel trace tests for key endpoints
|
||||
- [ ] Add negative tests (content type, payload, method)
|
||||
- [ ] Add auth/authz tests (anonymous, expired, tenant isolation)
|
||||
- [ ] Verify all tests have appropriate `[Trait("Lane", "...")]` attributes
|
||||
- [ ] Run locally: `dotnet test --filter "Lane=Contract|Lane=Security|Lane=Integration"`
|
||||
- [ ] Verify CI passes on PR
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- [WebServiceFixture Implementation](../../src/__Libraries/StellaOps.TestKit/Fixtures/WebServiceFixture.cs)
|
||||
- [ContractTestHelper Implementation](../../src/__Libraries/StellaOps.TestKit/Fixtures/ContractTestHelper.cs)
|
||||
- [WebServiceTestBase Implementation](../../src/__Libraries/StellaOps.TestKit/Templates/WebServiceTestBase.cs)
|
||||
- [Test Lanes CI Workflow](../../.gitea/workflows/test-lanes.yml)
|
||||
- [CI Lane Filters Documentation](./ci-lane-filters.md)
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2025-06-30 · Sprint 5100.0007.0006*
|
||||
230
docs/testing/webservice-test-rollout-plan.md
Normal file
230
docs/testing/webservice-test-rollout-plan.md
Normal file
@@ -0,0 +1,230 @@
|
||||
# WebService Test Rollout Plan
|
||||
|
||||
This document defines the rollout plan for applying the WebService test discipline to all StellaOps web services.
|
||||
|
||||
## Overview
|
||||
|
||||
Following the pilot implementation on Scanner.WebService (Sprint 5100.0007.0006), this plan defines the order and timeline for rolling out comprehensive WebService tests to all remaining services.
|
||||
|
||||
---
|
||||
|
||||
## Service Inventory
|
||||
|
||||
| Service | Module Path | Priority | Status | Sprint |
|
||||
|---------|-------------|----------|--------|--------|
|
||||
| Scanner.WebService | `src/Scanner/StellaOps.Scanner.WebService` | P0 (Pilot) | ✅ Existing tests | 5100.0007.0006 |
|
||||
| Concelier.WebService | `src/Concelier/StellaOps.Concelier.WebService` | P1 | Pending | TBD |
|
||||
| Excititor.WebService | `src/Excititor/StellaOps.Excititor.WebService` | P1 | Pending | TBD |
|
||||
| Policy.Engine | `src/Policy/StellaOps.Policy.Engine` | P1 | Pending | TBD |
|
||||
| Scheduler.WebService | `src/Scheduler/StellaOps.Scheduler.WebService` | P2 | Pending | TBD |
|
||||
| Notify.WebService | `src/Notify/StellaOps.Notify.WebService` | P2 | Pending | TBD |
|
||||
| Authority | `src/Authority/StellaOps.Authority` | P2 | Pending | TBD |
|
||||
| Signer | `src/Signer/StellaOps.Signer` | P3 | Pending | TBD |
|
||||
| Attestor | `src/Attestor/StellaOps.Attestor` | P3 | Pending | TBD |
|
||||
| ExportCenter.WebService | `src/ExportCenter/StellaOps.ExportCenter.WebService` | P3 | Pending | TBD |
|
||||
| Registry.TokenService | `src/Registry/StellaOps.Registry.TokenService` | P3 | Pending | TBD |
|
||||
| VulnExplorer.Api | `src/VulnExplorer/StellaOps.VulnExplorer.Api` | P3 | Pending | TBD |
|
||||
| Graph.Api | `src/Graph/StellaOps.Graph.Api` | P3 | Pending | TBD |
|
||||
| Orchestrator | `src/Orchestrator/StellaOps.Orchestrator` | P4 | Pending | TBD |
|
||||
|
||||
---
|
||||
|
||||
## Rollout Phases
|
||||
|
||||
### Phase 1: Core Data Flow Services (P1)
|
||||
|
||||
**Timeline**: Sprint 5100.0008.* (Q1 2026)
|
||||
|
||||
**Services**:
|
||||
- **Concelier.WebService** — Primary advisory ingestion service
|
||||
- **Excititor.WebService** — Enrichment and correlation service
|
||||
- **Policy.Engine** — Policy evaluation service
|
||||
|
||||
**Rationale**: These services form the core data flow pipeline. They have high traffic, complex contracts, and critical security boundaries.
|
||||
|
||||
**Test Requirements**:
|
||||
| Test Type | Concelier | Excititor | Policy |
|
||||
|-----------|-----------|-----------|--------|
|
||||
| Contract (OpenAPI) | Required | Required | Required |
|
||||
| OTel traces | Required | Required | Required |
|
||||
| Negative tests | Required | Required | Required |
|
||||
| Auth/AuthZ | Required | Required | Required |
|
||||
| Tenant isolation | Required | Required | Required |
|
||||
|
||||
---
|
||||
|
||||
### Phase 2: Scheduling & Notification Services (P2)
|
||||
|
||||
**Timeline**: Sprint 5100.0009.* (Q2 2026)
|
||||
|
||||
**Services**:
|
||||
- **Scheduler.WebService** — Job scheduling and orchestration
|
||||
- **Notify.WebService** — Notification dispatch
|
||||
- **Authority** — Authentication/authorization service
|
||||
|
||||
**Rationale**: These services support operational workflows. Authority is critical for security testing of all other services.
|
||||
|
||||
**Test Requirements**:
|
||||
| Test Type | Scheduler | Notify | Authority |
|
||||
|-----------|-----------|--------|-----------|
|
||||
| Contract (OpenAPI) | Required | Required | Required |
|
||||
| OTel traces | Required | Required | Required |
|
||||
| Negative tests | Required | Required | Required |
|
||||
| Auth/AuthZ | N/A (system) | Required | N/A (self) |
|
||||
| Token issuance | N/A | N/A | Required |
|
||||
|
||||
---
|
||||
|
||||
### Phase 3: Signing & Attestation Services (P3)
|
||||
|
||||
**Timeline**: Sprint 5100.0010.* (Q2-Q3 2026)
|
||||
|
||||
**Services**:
|
||||
- **Signer** — Cryptographic signing service
|
||||
- **Attestor** — Attestation generation/verification
|
||||
- **ExportCenter.WebService** — Report export service
|
||||
- **Registry.TokenService** — OCI registry token service
|
||||
- **VulnExplorer.Api** — Vulnerability exploration API
|
||||
- **Graph.Api** — Graph query API
|
||||
|
||||
**Rationale**: These services have specialized contracts and lower traffic. They require careful security testing due to cryptographic operations.
|
||||
|
||||
**Test Requirements**:
|
||||
| Test Type | Signer | Attestor | Others |
|
||||
|-----------|--------|----------|--------|
|
||||
| Contract (OpenAPI) | Required | Required | Required |
|
||||
| OTel traces | Required | Required | Required |
|
||||
| Negative tests | Required | Required | Required |
|
||||
| Crypto validation | Required | Required | N/A |
|
||||
|
||||
---
|
||||
|
||||
### Phase 4: Orchestration Services (P4)
|
||||
|
||||
**Timeline**: Sprint 5100.0011.* (Q3 2026)
|
||||
|
||||
**Services**:
|
||||
- **Orchestrator** — Workflow orchestration
|
||||
|
||||
**Rationale**: Orchestrator is a meta-service that coordinates other services. Testing depends on other services being testable first.
|
||||
|
||||
---
|
||||
|
||||
## Test Coverage Targets
|
||||
|
||||
### Minimum Requirements (PR-Gating)
|
||||
|
||||
| Test Category | Min Coverage | Lane |
|
||||
|---------------|-------------|------|
|
||||
| Contract (OpenAPI) | 100% of public endpoints | Contract |
|
||||
| Negative (4xx errors) | 100% of error codes | Unit/Security |
|
||||
| Auth/AuthZ | 100% of protected endpoints | Security |
|
||||
|
||||
### Recommended (Quality Gate)
|
||||
|
||||
| Test Category | Target Coverage | Lane |
|
||||
|---------------|-----------------|------|
|
||||
| OTel traces | 80% of endpoints | Integration |
|
||||
| Tenant isolation | 100% of data endpoints | Security |
|
||||
| Performance baselines | Key endpoints | Performance |
|
||||
|
||||
---
|
||||
|
||||
## Implementation Checklist per Service
|
||||
|
||||
```markdown
|
||||
## <Service Name> WebService Tests
|
||||
|
||||
### Setup
|
||||
- [ ] Create `<Service>ApplicationFactory` (WebApplicationFactory)
|
||||
- [ ] Create `<Service>TestFixture` if custom setup needed
|
||||
- [ ] Add test project: `StellaOps.<Service>.WebService.Tests`
|
||||
- [ ] Add reference to `StellaOps.TestKit`
|
||||
|
||||
### Contract Tests
|
||||
- [ ] Extract OpenAPI schema snapshot (`Snapshots/<service>-v1.json`)
|
||||
- [ ] Add schema stability test
|
||||
- [ ] Add response contract tests for key endpoints
|
||||
|
||||
### OTel Tests
|
||||
- [ ] Add trace assertion tests for key endpoints
|
||||
- [ ] Verify required tags (tenant.id, request.id, http.route)
|
||||
|
||||
### Negative Tests
|
||||
- [ ] Malformed content type → 415
|
||||
- [ ] Oversized payload → 413
|
||||
- [ ] Method mismatch → 405
|
||||
- [ ] Missing required field → 400
|
||||
- [ ] Invalid field value → 400
|
||||
|
||||
### Auth Tests
|
||||
- [ ] Anonymous request → 401
|
||||
- [ ] Expired token → 401
|
||||
- [ ] Missing scope → 403
|
||||
- [ ] Cross-tenant access → 404/403
|
||||
|
||||
### CI Integration
|
||||
- [ ] Verify traits assigned: Contract, Security, Integration, Unit
|
||||
- [ ] PR passes all lanes
|
||||
- [ ] Add to TEST_COVERAGE_MATRIX.md
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Sprint Planning Template
|
||||
|
||||
When creating sprints for new service tests:
|
||||
|
||||
```markdown
|
||||
# Sprint 5100.XXXX.YYYY - <Service> WebService Tests
|
||||
|
||||
## Topic & Scope
|
||||
- Apply WebService test discipline to <Service>.WebService
|
||||
- Contract tests, OTel traces, negative tests, auth tests
|
||||
- **Working directory:** `src/<Module>/__Tests/StellaOps.<Module>.WebService.Tests`
|
||||
|
||||
## Delivery Tracker
|
||||
| # | Task ID | Status | Task Definition |
|
||||
|---|---------|--------|-----------------|
|
||||
| 1 | WEBSVC-XXXX-001 | TODO | Create <Service>ApplicationFactory |
|
||||
| 2 | WEBSVC-XXXX-002 | TODO | Add OpenAPI contract tests |
|
||||
| 3 | WEBSVC-XXXX-003 | TODO | Add OTel trace tests |
|
||||
| 4 | WEBSVC-XXXX-004 | TODO | Add negative tests (4xx) |
|
||||
| 5 | WEBSVC-XXXX-005 | TODO | Add auth/authz tests |
|
||||
| 6 | WEBSVC-XXXX-006 | TODO | Update TEST_COVERAGE_MATRIX.md |
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Success Metrics
|
||||
|
||||
| Metric | Target | Measurement |
|
||||
|--------|--------|-------------|
|
||||
| Services with contract tests | 100% | Count of services with OpenAPI snapshot tests |
|
||||
| Services with auth tests | 100% | Count of services with auth boundary tests |
|
||||
| Contract test failures in production | 0 | Breaking changes detected in staging |
|
||||
| Security test coverage | 100% of auth endpoints | Audit of protected routes vs tests |
|
||||
|
||||
---
|
||||
|
||||
## Risks & Mitigations
|
||||
|
||||
| Risk | Impact | Mitigation |
|
||||
|------|--------|------------|
|
||||
| Services lack OpenAPI spec | Cannot do contract testing | Generate spec via Swashbuckle/NSwag |
|
||||
| OTel not configured in service | Cannot verify traces | Add OTel middleware as prerequisite |
|
||||
| Auth disabled in test mode | False confidence | Test with auth enabled, use test tokens |
|
||||
| Test fixtures are slow | CI timeout | Share fixtures, use in-memory providers |
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- [WebService Test Discipline](./webservice-test-discipline.md)
|
||||
- [Test Coverage Matrix](./TEST_COVERAGE_MATRIX.md)
|
||||
- [CI Lane Filters](./ci-lane-filters.md)
|
||||
- [Testing Strategy Models](./testing-strategy-models.md)
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2025-06-30 · Sprint 5100.0007.0006*
|
||||
Reference in New Issue
Block a user