stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 2 Releases Wiki Activity

Add tests for SBOM generation determinism across multiple formats

Browse Source 
- Created `StellaOps.TestKit.Tests` project for unit tests related to determinism.
- Implemented `DeterminismManifestTests` to validate deterministic output for canonical bytes and strings, file read/write operations, and error handling for invalid schema versions.
- Added `SbomDeterminismTests` to ensure identical inputs produce consistent SBOMs across SPDX 3.0.1 and CycloneDX 1.6/1.7 formats, including parallel execution tests.
- Updated project references in `StellaOps.Integration.Determinism` to include the new determinism testing library.
This commit is contained in:
master
2025-12-23 18:56:12 +02:00
committed by StellaOps Bot
parent 7ac70ece71
commit 491e883653
409 changed files with 23797 additions and 17779 deletions
Download Patch File Download Diff File Expand all files Collapse all files

425
docs/testing/connector-fixture-discipline.md Normal file
View File

@@ -0,0 +1,425 @@
# Connector Fixture Discipline
This document defines the testing discipline for StellaOps Concelier and Excititor connectors. All connectors must follow these patterns to ensure consistent, deterministic, and offline-capable testing.
## Overview
Connector tests follow **Model C1 (Connector/External)** from the testing strategy:
1. **Fixture-based parser tests** — Raw upstream payload → normalized internal model (offline)
2. **Resilience tests** — Partial/bad input → deterministic failure classification
3. **Security tests** — URL allowlist, redirect handling, payload limits
4. **Live smoke tests** — Schema drift detection (opt-in, non-gating)
---
## 1. Directory Structure
Each connector test project follows this structure:
```
src/<Module>/__Tests/StellaOps.<Module>.Connector.<Source>.Tests/
├── StellaOps.<Module>.Connector.<Source>.Tests.csproj
├── Fixtures/
│ ├── <source>-typical.json # Typical advisory payload
│ ├── <source>-edge-<case>.json # Edge case payloads
│ ├── <source>-error-<type>.json # Malformed/invalid payloads
│ └── expected-<id>.json # Expected normalized output
├── <Source>/
│ ├── <Source>ParserTests.cs # Parser unit tests
│ ├── <Source>ConnectorTests.cs # Connector integration tests
│ └── <Source>ResilienceTests.cs # Resilience/security tests
└── Expected/
└── <source>-<id>.canonical.json # Canonical JSON snapshots
```
### Example: NVD Connector
```
src/Concelier/__Tests/StellaOps.Concelier.Connector.Nvd.Tests/
├── Fixtures/
│ ├── nvd-window-1.json
│ ├── nvd-window-2.json
│ ├── nvd-multipage-1.json
│ ├── nvd-multipage-2.json
│ ├── nvd-invalid-schema.json
│ └── expected-CVE-2024-0001.json
├── Nvd/
│ ├── NvdParserTests.cs
│ ├── NvdConnectorTests.cs
│ └── NvdConnectorHarnessTests.cs
└── Expected/
└── conflict-nvd.canonical.json
```
---
## 2. Fixture-Based Parser Tests
### Purpose
Test that the parser correctly transforms raw upstream payloads into normalized internal models without network access.
### Pattern
```csharp
using StellaOps.TestKit.Connectors;
public class NvdParserTests : ConnectorParserTestBase<NvdRawAdvisory, ConcelierAdvisory>
{
public NvdParserTests()
: base(new NvdParser(), "Nvd/Fixtures")
{
}
[Fact]
[Trait("Lane", "Unit")]
public async Task ParseTypicalAdvisory_ProducesExpectedModel()
{
// Arrange
var raw = await LoadFixture<NvdRawAdvisory>("nvd-window-1.json");
// Act
var result = Parser.Parse(raw);
// Assert
await AssertMatchesSnapshot(result, "expected-CVE-2024-0001.json");
}
[Theory]
[Trait("Lane", "Unit")]
[InlineData("nvd-multipage-1.json", "expected-multipage-1.json")]
[InlineData("nvd-multipage-2.json", "expected-multipage-2.json")]
public async Task ParseAllFixtures_ProducesExpectedModels(string input, string expected)
{
var raw = await LoadFixture<NvdRawAdvisory>(input);
var result = Parser.Parse(raw);
await AssertMatchesSnapshot(result, expected);
}
}
```
### Fixture Requirements
| Type | Naming Convention | Purpose |
|------|-------------------|---------|
| Typical | `<source>-typical.json` | Normal advisory with all common fields |
| Edge case | `<source>-edge-<case>.json` | Unusual but valid payloads |
| Error | `<source>-error-<type>.json` | Malformed/invalid payloads |
| Expected | `expected-<id>.json` | Expected normalized output |
| Canonical | `<source>-<id>.canonical.json` | Deterministic JSON snapshot |
### Minimum Coverage
Each connector must have fixtures for:
- [ ] At least 1 typical payload
- [ ] At least 2 edge cases (e.g., multi-vendor, unusual CVSS, missing optional fields)
- [ ] At least 2 error cases (e.g., missing required fields, invalid schema)
---
## 3. Resilience Tests
### Purpose
Verify that connectors handle malformed input gracefully with deterministic failure classification.
### Pattern
```csharp
public class NvdResilienceTests : ConnectorResilienceTestBase<NvdConnector>
{
public NvdResilienceTests()
: base(new NvdConnector(CreateTestHttpClient()))
{
}
[Fact]
[Trait("Lane", "Unit")]
public async Task MissingRequiredField_ReturnsParseError()
{
// Arrange
var payload = await LoadFixture("nvd-error-missing-cve-id.json");
// Act
var result = await Connector.ParseAsync(payload);
// Assert
Assert.False(result.IsSuccess);
Assert.Equal(ConnectorErrorKind.ParseError, result.Error.Kind);
Assert.Contains("cve_id", result.Error.Message, StringComparison.OrdinalIgnoreCase);
}
[Fact]
[Trait("Lane", "Unit")]
public async Task InvalidDateFormat_ReturnsParseError()
{
var payload = await LoadFixture("nvd-error-invalid-date.json");
var result = await Connector.ParseAsync(payload);
Assert.False(result.IsSuccess);
Assert.Equal(ConnectorErrorKind.ParseError, result.Error.Kind);
}
[Fact]
[Trait("Lane", "Unit")]
public async Task UnexpectedEnumValue_LogsWarningAndContinues()
{
var payload = await LoadFixture("nvd-edge-unknown-severity.json");
var result = await Connector.ParseAsync(payload);
Assert.True(result.IsSuccess);
Assert.Contains(result.Warnings, w => w.Contains("unknown severity"));
}
}
```
### Required Test Cases
| Case | Expected Behavior | Trait |
|------|-------------------|-------|
| Missing required field | `ConnectorErrorKind.ParseError` | Unit |
| Invalid date format | `ConnectorErrorKind.ParseError` | Unit |
| Invalid JSON structure | `ConnectorErrorKind.ParseError` | Unit |
| Unknown enum value | Warning logged, continues | Unit |
| Empty response | `ConnectorErrorKind.EmptyResponse` | Unit |
| Truncated payload | `ConnectorErrorKind.ParseError` | Unit |
---
## 4. Security Tests
### Purpose
Verify that connectors enforce security boundaries for network operations.
### Pattern
```csharp
public class NvdSecurityTests : ConnectorSecurityTestBase<NvdConnector>
{
[Fact]
[Trait("Lane", "Security")]
public async Task UrlOutsideAllowlist_RejectsRequest()
{
// Arrange
var connector = CreateConnector(allowedHosts: ["services.nvd.nist.gov"]);
// Act & Assert
await Assert.ThrowsAsync<SecurityException>(
() => connector.FetchAsync("https://evil.example.com/api"));
}
[Fact]
[Trait("Lane", "Security")]
public async Task RedirectToDisallowedHost_RejectsRequest()
{
var handler = CreateMockHandler(redirectTo: "https://evil.example.com");
var connector = CreateConnector(handler, allowedHosts: ["services.nvd.nist.gov"]);
await Assert.ThrowsAsync<SecurityException>(
() => connector.FetchAsync("https://services.nvd.nist.gov/api"));
}
[Fact]
[Trait("Lane", "Security")]
public async Task PayloadExceedsMaxSize_RejectsPayload()
{
var handler = CreateMockHandler(responseSize: 100_000_001); // 100MB
var connector = CreateConnector(handler, maxPayloadBytes: 100_000_000);
await Assert.ThrowsAsync<PayloadTooLargeException>(
() => connector.FetchAsync("https://services.nvd.nist.gov/api"));
}
[Fact]
[Trait("Lane", "Security")]
public async Task DecompressionBomb_RejectsPayload()
{
// 1KB compressed, 1GB decompressed
var handler = CreateMockHandler(compressedBomb: true);
var connector = CreateConnector(handler);
await Assert.ThrowsAsync<PayloadTooLargeException>(
() => connector.FetchAsync("https://services.nvd.nist.gov/api"));
}
}
```
### Required Security Tests
| Test | Purpose | Trait |
|------|---------|-------|
| URL allowlist | Block requests to unauthorized hosts | Security |
| Redirect validation | Block redirects to unauthorized hosts | Security |
| Max payload size | Reject oversized responses | Security |
| Decompression bomb | Reject zip bombs | Security |
| Rate limiting | Respect upstream rate limits | Security |
---
## 5. Live Smoke Tests (Opt-In)
### Purpose
Detect upstream schema drift by comparing live responses against known fixtures.
### Pattern
```csharp
public class NvdLiveTests : ConnectorLiveTestBase<NvdConnector>
{
[Fact]
[Trait("Lane", "Live")]
[Trait("Category", "SchemaDrift")]
public async Task LiveSchema_MatchesFixtureSchema()
{
// Skip if not in Live lane
Skip.IfNot(IsLiveLaneEnabled());
// Fetch live response
var live = await Connector.FetchLatestAsync();
// Compare schema (not values) against fixture
var fixture = await LoadFixture<NvdResponse>("nvd-typical.json");
AssertSchemaMatches(live, fixture);
}
[Fact]
[Trait("Lane", "Live")]
public async Task LiveFetch_ReturnsValidAdvisories()
{
Skip.IfNot(IsLiveLaneEnabled());
var result = await Connector.FetchLatestAsync();
Assert.True(result.IsSuccess);
Assert.NotEmpty(result.Value.Advisories);
}
}
```
### Configuration
Live tests are:
- **Never PR-gating** — run only in scheduled/nightly jobs
- **Opt-in** — require explicit `LIVE_TESTS_ENABLED=true` environment variable
- **Alerting** — schema drift triggers notification, not failure
---
## 6. Fixture Updater
### Purpose
Refresh fixtures from live sources when upstream schemas change intentionally.
### Usage
```bash
# Update all fixtures for NVD connector
dotnet run --project tools/FixtureUpdater -- \
--connector nvd \
--output src/Concelier/__Tests/StellaOps.Concelier.Connector.Nvd.Tests/Nvd/Fixtures/
# Update specific fixture
dotnet run --project tools/FixtureUpdater -- \
--connector nvd \
--cve CVE-2024-0001 \
--output src/Concelier/__Tests/.../Fixtures/nvd-CVE-2024-0001.json
# Dry-run mode (show diff without writing)
dotnet run --project tools/FixtureUpdater -- \
--connector nvd \
--dry-run
```
### Workflow
1. Live test detects schema drift
2. CI creates draft PR with fixture update
3. Developer reviews diff for intentional vs accidental changes
4. If intentional: update parser and merge
5. If accidental: investigate upstream API issue
---
## 7. Test Traits and CI Integration
### Trait Assignment
| Test Category | Trait | Lane | PR-Gating |
|---------------|-------|------|-----------|
| Parser tests | `[Trait("Lane", "Unit")]` | Unit | Yes |
| Resilience tests | `[Trait("Lane", "Unit")]` | Unit | Yes |
| Security tests | `[Trait("Lane", "Security")]` | Security | Yes |
| Live tests | `[Trait("Lane", "Live")]` | Live | No |
### Running Tests
```bash
# Run all connector unit tests
dotnet test --filter "Lane=Unit" src/Concelier/__Tests/
# Run security tests
dotnet test --filter "Lane=Security" src/Concelier/__Tests/
# Run live tests (requires LIVE_TESTS_ENABLED=true)
LIVE_TESTS_ENABLED=true dotnet test --filter "Lane=Live" src/Concelier/__Tests/
```
---
## 8. Connector Inventory
### Concelier Connectors (Advisory Sources)
| Connector | Fixtures | Parser Tests | Resilience | Security | Live |
|-----------|----------|--------------|------------|----------|------|
| NVD | ✅ | ✅ | ✅ | ⬜ | ⬜ |
| OSV | ✅ | ✅ | ⬜ | ⬜ | ⬜ |
| GHSA | ✅ | ✅ | ⬜ | ⬜ | ⬜ |
| CVE | ✅ | ✅ | ⬜ | ⬜ | ⬜ |
| KEV | ✅ | ✅ | ⬜ | ⬜ | ⬜ |
| EPSS | ✅ | ✅ | ⬜ | ⬜ | ⬜ |
| Distro.Alpine | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ |
| Distro.Debian | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ |
| Distro.RedHat | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ |
| Distro.Suse | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ |
| Distro.Ubuntu | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ |
| Vndr.Adobe | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ |
| Vndr.Apple | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ |
| Vndr.Cisco | ✅ | ✅ | ⬜ | ⬜ | ⬜ |
| Vndr.Msrc | ✅ | ✅ | ⬜ | ⬜ | ⬜ |
| Vndr.Oracle | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ |
| Vndr.Vmware | ✅ | ✅ | ⬜ | ⬜ | ⬜ |
| Cert.Bund | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ |
| Cert.Cc | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ |
| Cert.Fr | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ |
| ICS.Cisa | ✅ | ✅ | ⬜ | ⬜ | ⬜ |
### Excititor Connectors (VEX Sources)
| Connector | Fixtures | Parser Tests | Resilience | Security | Live |
|-----------|----------|--------------|------------|----------|------|
| OpenVEX | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ |
| CSAF/VEX | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ |
---
## References
- [ConnectorHttpFixture](../../src/__Libraries/StellaOps.TestKit/Connectors/ConnectorHttpFixture.cs)
- [ConnectorTestBase](../../src/__Libraries/StellaOps.TestKit/Connectors/ConnectorTestBase.cs)
- [ConnectorResilienceTestBase](../../src/__Libraries/StellaOps.TestKit/Connectors/ConnectorResilienceTestBase.cs)
- [FixtureUpdater](../../src/__Libraries/StellaOps.TestKit/Connectors/FixtureUpdater.cs)
- [Testing Strategy Models](./testing-strategy-models.md)
- [CI Lane Filters](./ci-lane-filters.md)
---
*Last updated: 2025-06-30 · Sprint 5100.0007.0005*