git.stella-ops.org/docs/testing/connector-fixture-discipline.md

Connector Fixture Discipline

This document defines the testing discipline for StellaOps Concelier and Excititor connectors. All connectors must follow these patterns to ensure consistent, deterministic, and offline-capable testing.

Overview

Connector tests follow Model C1 (Connector/External) from the testing strategy:

1. Fixture-based parser tests — Raw upstream payload → normalized internal model (offline)
2. Resilience tests — Partial/bad input → deterministic failure classification
3. Security tests — URL allowlist, redirect handling, payload limits
4. Live smoke tests — Schema drift detection (opt-in, non-gating)

---

1. Directory Structure

Each connector test project follows this structure:

src/<Module>/__Tests/StellaOps.<Module>.Connector.<Source>.Tests/
├── StellaOps.<Module>.Connector.<Source>.Tests.csproj
├── Fixtures/
│   ├── <source>-typical.json          # Typical advisory payload
│   ├── <source>-edge-<case>.json      # Edge case payloads
│   ├── <source>-error-<type>.json     # Malformed/invalid payloads
│   └── expected-<id>.json             # Expected normalized output
├── <Source>/
│   ├── <Source>ParserTests.cs         # Parser unit tests
│   ├── <Source>ConnectorTests.cs      # Connector integration tests
│   └── <Source>ResilienceTests.cs     # Resilience/security tests
└── Expected/
    └── <source>-<id>.canonical.json   # Canonical JSON snapshots

Example: NVD Connector

src/Concelier/__Tests/StellaOps.Concelier.Connector.Nvd.Tests/
├── Fixtures/
│   ├── nvd-window-1.json
│   ├── nvd-window-2.json
│   ├── nvd-multipage-1.json
│   ├── nvd-multipage-2.json
│   ├── nvd-invalid-schema.json
│   └── expected-CVE-2024-0001.json
├── Nvd/
│   ├── NvdParserTests.cs
│   ├── NvdConnectorTests.cs
│   └── NvdConnectorHarnessTests.cs
└── Expected/
    └── conflict-nvd.canonical.json

---

2. Fixture-Based Parser Tests

Purpose

Test that the parser correctly transforms raw upstream payloads into normalized internal models without network access.

Pattern

using StellaOps.TestKit.Connectors;

public class NvdParserTests : ConnectorParserTestBase<NvdRawAdvisory, ConcelierAdvisory>
{
    public NvdParserTests() 
        : base(new NvdParser(), "Nvd/Fixtures")
    {
    }
    
    [Fact]
    [Trait("Lane", "Unit")]
    public async Task ParseTypicalAdvisory_ProducesExpectedModel()
    {
        // Arrange
        var raw = await LoadFixture<NvdRawAdvisory>("nvd-window-1.json");
        
        // Act
        var result = Parser.Parse(raw);
        
        // Assert
        await AssertMatchesSnapshot(result, "expected-CVE-2024-0001.json");
    }
    
    [Theory]
    [Trait("Lane", "Unit")]
    [InlineData("nvd-multipage-1.json", "expected-multipage-1.json")]
    [InlineData("nvd-multipage-2.json", "expected-multipage-2.json")]
    public async Task ParseAllFixtures_ProducesExpectedModels(string input, string expected)
    {
        var raw = await LoadFixture<NvdRawAdvisory>(input);
        var result = Parser.Parse(raw);
        await AssertMatchesSnapshot(result, expected);
    }
}

Fixture Requirements

Type	Naming Convention	Purpose
Typical	<source>-typical.json	Normal advisory with all common fields
Edge case	<source>-edge-<case>.json	Unusual but valid payloads
Error	<source>-error-<type>.json	Malformed/invalid payloads
Expected	expected-<id>.json	Expected normalized output
Canonical	<source>-<id>.canonical.json	Deterministic JSON snapshot

Minimum Coverage

Each connector must have fixtures for:

• At least 1 typical payload
• At least 2 edge cases (e.g., multi-vendor, unusual CVSS, missing optional fields)
• At least 2 error cases (e.g., missing required fields, invalid schema)

---

3. Resilience Tests

Purpose

Verify that connectors handle malformed input gracefully with deterministic failure classification.

Pattern

public class NvdResilienceTests : ConnectorResilienceTestBase<NvdConnector>
{
    public NvdResilienceTests()
        : base(new NvdConnector(CreateTestHttpClient()))
    {
    }
    
    [Fact]
    [Trait("Lane", "Unit")]
    public async Task MissingRequiredField_ReturnsParseError()
    {
        // Arrange
        var payload = await LoadFixture("nvd-error-missing-cve-id.json");
        
        // Act
        var result = await Connector.ParseAsync(payload);
        
        // Assert
        Assert.False(result.IsSuccess);
        Assert.Equal(ConnectorErrorKind.ParseError, result.Error.Kind);
        Assert.Contains("cve_id", result.Error.Message, StringComparison.OrdinalIgnoreCase);
    }
    
    [Fact]
    [Trait("Lane", "Unit")]
    public async Task InvalidDateFormat_ReturnsParseError()
    {
        var payload = await LoadFixture("nvd-error-invalid-date.json");
        
        var result = await Connector.ParseAsync(payload);
        
        Assert.False(result.IsSuccess);
        Assert.Equal(ConnectorErrorKind.ParseError, result.Error.Kind);
    }
    
    [Fact]
    [Trait("Lane", "Unit")]
    public async Task UnexpectedEnumValue_LogsWarningAndContinues()
    {
        var payload = await LoadFixture("nvd-edge-unknown-severity.json");
        
        var result = await Connector.ParseAsync(payload);
        
        Assert.True(result.IsSuccess);
        Assert.Contains(result.Warnings, w => w.Contains("unknown severity"));
    }
}

Required Test Cases

Case	Expected Behavior	Trait
Missing required field	ConnectorErrorKind.ParseError	Unit
Invalid date format	ConnectorErrorKind.ParseError	Unit
Invalid JSON structure	ConnectorErrorKind.ParseError	Unit
Unknown enum value	Warning logged, continues	Unit
Empty response	ConnectorErrorKind.EmptyResponse	Unit
Truncated payload	ConnectorErrorKind.ParseError	Unit

---

4. Security Tests

Purpose

Verify that connectors enforce security boundaries for network operations.

Pattern

public class NvdSecurityTests : ConnectorSecurityTestBase<NvdConnector>
{
    [Fact]
    [Trait("Lane", "Security")]
    public async Task UrlOutsideAllowlist_RejectsRequest()
    {
        // Arrange
        var connector = CreateConnector(allowedHosts: ["services.nvd.nist.gov"]);
        
        // Act & Assert
        await Assert.ThrowsAsync<SecurityException>(
            () => connector.FetchAsync("https://evil.example.com/api"));
    }
    
    [Fact]
    [Trait("Lane", "Security")]
    public async Task RedirectToDisallowedHost_RejectsRequest()
    {
        var handler = CreateMockHandler(redirectTo: "https://evil.example.com");
        var connector = CreateConnector(handler, allowedHosts: ["services.nvd.nist.gov"]);
        
        await Assert.ThrowsAsync<SecurityException>(
            () => connector.FetchAsync("https://services.nvd.nist.gov/api"));
    }
    
    [Fact]
    [Trait("Lane", "Security")]
    public async Task PayloadExceedsMaxSize_RejectsPayload()
    {
        var handler = CreateMockHandler(responseSize: 100_000_001); // 100MB
        var connector = CreateConnector(handler, maxPayloadBytes: 100_000_000);
        
        await Assert.ThrowsAsync<PayloadTooLargeException>(
            () => connector.FetchAsync("https://services.nvd.nist.gov/api"));
    }
    
    [Fact]
    [Trait("Lane", "Security")]
    public async Task DecompressionBomb_RejectsPayload()
    {
        // 1KB compressed, 1GB decompressed
        var handler = CreateMockHandler(compressedBomb: true);
        var connector = CreateConnector(handler);
        
        await Assert.ThrowsAsync<PayloadTooLargeException>(
            () => connector.FetchAsync("https://services.nvd.nist.gov/api"));
    }
}

Required Security Tests

Test	Purpose	Trait
URL allowlist	Block requests to unauthorized hosts	Security
Redirect validation	Block redirects to unauthorized hosts	Security
Max payload size	Reject oversized responses	Security
Decompression bomb	Reject zip bombs	Security
Rate limiting	Respect upstream rate limits	Security

---

5. Live Smoke Tests (Opt-In)

Purpose

Detect upstream schema drift by comparing live responses against known fixtures.

Pattern

public class NvdLiveTests : ConnectorLiveTestBase<NvdConnector>
{
    [Fact]
    [Trait("Lane", "Live")]
    [Trait("Category", "SchemaDrift")]
    public async Task LiveSchema_MatchesFixtureSchema()
    {
        // Skip if not in Live lane
        Skip.IfNot(IsLiveLaneEnabled());
        
        // Fetch live response
        var live = await Connector.FetchLatestAsync();
        
        // Compare schema (not values) against fixture
        var fixture = await LoadFixture<NvdResponse>("nvd-typical.json");
        
        AssertSchemaMatches(live, fixture);
    }
    
    [Fact]
    [Trait("Lane", "Live")]
    public async Task LiveFetch_ReturnsValidAdvisories()
    {
        Skip.IfNot(IsLiveLaneEnabled());
        
        var result = await Connector.FetchLatestAsync();
        
        Assert.True(result.IsSuccess);
        Assert.NotEmpty(result.Value.Advisories);
    }
}

Configuration

Live tests are:

• Never PR-gating — run only in scheduled/nightly jobs
• Opt-in — require explicit LIVE_TESTS_ENABLED=true environment variable
• Alerting — schema drift triggers notification, not failure

---

6. Fixture Updater

Purpose

Refresh fixtures from live sources when upstream schemas change intentionally.

Usage

# Update all fixtures for NVD connector
dotnet run --project tools/FixtureUpdater -- \
  --connector nvd \
  --output src/Concelier/__Tests/StellaOps.Concelier.Connector.Nvd.Tests/Nvd/Fixtures/

# Update specific fixture
dotnet run --project tools/FixtureUpdater -- \
  --connector nvd \
  --cve CVE-2024-0001 \
  --output src/Concelier/__Tests/.../Fixtures/nvd-CVE-2024-0001.json

# Dry-run mode (show diff without writing)
dotnet run --project tools/FixtureUpdater -- \
  --connector nvd \
  --dry-run

Workflow

1. Live test detects schema drift
2. CI creates draft PR with fixture update
3. Developer reviews diff for intentional vs accidental changes
4. If intentional: update parser and merge
5. If accidental: investigate upstream API issue

---

7. Test Traits and CI Integration

Trait Assignment

Test Category	Trait	Lane	PR-Gating
Parser tests	[Trait("Lane", "Unit")]	Unit	Yes
Resilience tests	[Trait("Lane", "Unit")]	Unit	Yes
Security tests	[Trait("Lane", "Security")]	Security	Yes
Live tests	[Trait("Lane", "Live")]	Live	No

Running Tests

# Run all connector unit tests
dotnet test --filter "Lane=Unit" src/Concelier/__Tests/

# Run security tests
dotnet test --filter "Lane=Security" src/Concelier/__Tests/

# Run live tests (requires LIVE_TESTS_ENABLED=true)
LIVE_TESTS_ENABLED=true dotnet test --filter "Lane=Live" src/Concelier/__Tests/

---

8. Connector Inventory

Concelier Connectors (Advisory Sources)

Connector	Fixtures	Parser Tests	Resilience	Security	Live
NVD	✅	✅	✅	⬜	⬜
OSV	✅	✅	⬜	⬜	⬜
GHSA	✅	✅	⬜	⬜	⬜
CVE	✅	✅	⬜	⬜	⬜
KEV	✅	✅	⬜	⬜	⬜
EPSS	✅	✅	⬜	⬜	⬜
Distro.Alpine	⬜	⬜	⬜	⬜	⬜
Distro.Debian	⬜	⬜	⬜	⬜	⬜
Distro.RedHat	⬜	⬜	⬜	⬜	⬜
Distro.Suse	⬜	⬜	⬜	⬜	⬜
Distro.Ubuntu	⬜	⬜	⬜	⬜	⬜
Vndr.Adobe	⬜	⬜	⬜	⬜	⬜
Vndr.Apple	⬜	⬜	⬜	⬜	⬜
Vndr.Cisco	✅	✅	⬜	⬜	⬜
Vndr.Msrc	✅	✅	⬜	⬜	⬜
Vndr.Oracle	⬜	⬜	⬜	⬜	⬜
Vndr.Vmware	✅	✅	⬜	⬜	⬜
Cert.Bund	⬜	⬜	⬜	⬜	⬜
Cert.Cc	⬜	⬜	⬜	⬜	⬜
Cert.Fr	⬜	⬜	⬜	⬜	⬜
ICS.Cisa	✅	✅	⬜	⬜	⬜

Excititor Connectors (VEX Sources)

Connector	Fixtures	Parser Tests	Resilience	Security	Live
OpenVEX	⬜	⬜	⬜	⬜	⬜
CSAF/VEX	⬜	⬜	⬜	⬜	⬜

---

References

• ConnectorHttpFixture
• ConnectorTestBase
• ConnectorResilienceTestBase
• FixtureUpdater
• Testing Strategy Models
• CI Lane Filters

---

Last updated: 2025-06-30 · Sprint 5100.0007.0005