Files

master 4789027317 docs consolidation and others

2026-01-06 19:07:48 +02:00

13 KiB

Raw Blame History

Connector Fixture Discipline

This document defines the testing discipline for StellaOps Concelier and Excititor connectors. All connectors must follow these patterns to ensure consistent, deterministic, and offline-capable testing.

Overview

Connector tests follow Model C1 (Connector/External) from the testing strategy:

Fixture-based parser tests — Raw upstream payload → normalized internal model (offline)
Resilience tests — Partial/bad input → deterministic failure classification
Security tests — URL allowlist, redirect handling, payload limits
Live smoke tests — Schema drift detection (opt-in, non-gating)

1. Directory Structure

Each connector test project follows this structure:

src/<Module>/__Tests/StellaOps.<Module>.Connector.<Source>.Tests/
├── StellaOps.<Module>.Connector.<Source>.Tests.csproj
├── Fixtures/
│   ├── <source>-typical.json          # Typical advisory payload
│   ├── <source>-edge-<case>.json      # Edge case payloads
│   ├── <source>-error-<type>.json     # Malformed/invalid payloads
│   └── expected-<id>.json             # Expected normalized output
├── <Source>/
│   ├── <Source>ParserTests.cs         # Parser unit tests
│   ├── <Source>ConnectorTests.cs      # Connector integration tests
│   └── <Source>ResilienceTests.cs     # Resilience/security tests
└── Expected/
    └── <source>-<id>.canonical.json   # Canonical JSON snapshots

Example: NVD Connector

src/Concelier/__Tests/StellaOps.Concelier.Connector.Nvd.Tests/
├── Fixtures/
│   ├── nvd-window-1.json
│   ├── nvd-window-2.json
│   ├── nvd-multipage-1.json
│   ├── nvd-multipage-2.json
│   ├── nvd-invalid-schema.json
│   └── expected-CVE-2024-0001.json
├── Nvd/
│   ├── NvdParserTests.cs
│   ├── NvdConnectorTests.cs
│   └── NvdConnectorHarnessTests.cs
└── Expected/
    └── conflict-nvd.canonical.json

2. Fixture-Based Parser Tests

Purpose

Test that the parser correctly transforms raw upstream payloads into normalized internal models without network access.

Pattern

using StellaOps.TestKit.Connectors;

public class NvdParserTests : ConnectorParserTestBase<NvdRawAdvisory, ConcelierAdvisory>
{
    public NvdParserTests() 
        : base(new NvdParser(), "Nvd/Fixtures")
    {
    }
    
    [Fact]
    [Trait("Lane", "Unit")]
    public async Task ParseTypicalAdvisory_ProducesExpectedModel()
    {
        // Arrange
        var raw = await LoadFixture<NvdRawAdvisory>("nvd-window-1.json");
        
        // Act
        var result = Parser.Parse(raw);
        
        // Assert
        await AssertMatchesSnapshot(result, "expected-CVE-2024-0001.json");
    }
    
    [Theory]
    [Trait("Lane", "Unit")]
    [InlineData("nvd-multipage-1.json", "expected-multipage-1.json")]
    [InlineData("nvd-multipage-2.json", "expected-multipage-2.json")]
    public async Task ParseAllFixtures_ProducesExpectedModels(string input, string expected)
    {
        var raw = await LoadFixture<NvdRawAdvisory>(input);
        var result = Parser.Parse(raw);
        await AssertMatchesSnapshot(result, expected);
    }
}

Fixture Requirements

Type	Naming Convention	Purpose
Typical	`<source>-typical.json`	Normal advisory with all common fields
Edge case	`<source>-edge-<case>.json`	Unusual but valid payloads
Error	`<source>-error-<type>.json`	Malformed/invalid payloads
Expected	`expected-<id>.json`	Expected normalized output
Canonical	`<source>-<id>.canonical.json`	Deterministic JSON snapshot

Minimum Coverage

Each connector must have fixtures for:

At least 1 typical payload
At least 2 edge cases (e.g., multi-vendor, unusual CVSS, missing optional fields)
At least 2 error cases (e.g., missing required fields, invalid schema)

3. Resilience Tests

Purpose

Verify that connectors handle malformed input gracefully with deterministic failure classification.

Pattern

public class NvdResilienceTests : ConnectorResilienceTestBase<NvdConnector>
{
    public NvdResilienceTests()
        : base(new NvdConnector(CreateTestHttpClient()))
    {
    }
    
    [Fact]
    [Trait("Lane", "Unit")]
    public async Task MissingRequiredField_ReturnsParseError()
    {
        // Arrange
        var payload = await LoadFixture("nvd-error-missing-cve-id.json");
        
        // Act
        var result = await Connector.ParseAsync(payload);
        
        // Assert
        Assert.False(result.IsSuccess);
        Assert.Equal(ConnectorErrorKind.ParseError, result.Error.Kind);
        Assert.Contains("cve_id", result.Error.Message, StringComparison.OrdinalIgnoreCase);
    }
    
    [Fact]
    [Trait("Lane", "Unit")]
    public async Task InvalidDateFormat_ReturnsParseError()
    {
        var payload = await LoadFixture("nvd-error-invalid-date.json");
        
        var result = await Connector.ParseAsync(payload);
        
        Assert.False(result.IsSuccess);
        Assert.Equal(ConnectorErrorKind.ParseError, result.Error.Kind);
    }
    
    [Fact]
    [Trait("Lane", "Unit")]
    public async Task UnexpectedEnumValue_LogsWarningAndContinues()
    {
        var payload = await LoadFixture("nvd-edge-unknown-severity.json");
        
        var result = await Connector.ParseAsync(payload);
        
        Assert.True(result.IsSuccess);
        Assert.Contains(result.Warnings, w => w.Contains("unknown severity"));
    }
}

Required Test Cases

Case	Expected Behavior	Trait
Missing required field	`ConnectorErrorKind.ParseError`	Unit
Invalid date format	`ConnectorErrorKind.ParseError`	Unit
Invalid JSON structure	`ConnectorErrorKind.ParseError`	Unit
Unknown enum value	Warning logged, continues	Unit
Empty response	`ConnectorErrorKind.EmptyResponse`	Unit
Truncated payload	`ConnectorErrorKind.ParseError`	Unit

4. Security Tests

Purpose

Verify that connectors enforce security boundaries for network operations.

Pattern

public class NvdSecurityTests : ConnectorSecurityTestBase<NvdConnector>
{
    [Fact]
    [Trait("Lane", "Security")]
    public async Task UrlOutsideAllowlist_RejectsRequest()
    {
        // Arrange
        var connector = CreateConnector(allowedHosts: ["services.nvd.nist.gov"]);
        
        // Act & Assert
        await Assert.ThrowsAsync<SecurityException>(
            () => connector.FetchAsync("https://evil.example.com/api"));
    }
    
    [Fact]
    [Trait("Lane", "Security")]
    public async Task RedirectToDisallowedHost_RejectsRequest()
    {
        var handler = CreateMockHandler(redirectTo: "https://evil.example.com");
        var connector = CreateConnector(handler, allowedHosts: ["services.nvd.nist.gov"]);
        
        await Assert.ThrowsAsync<SecurityException>(
            () => connector.FetchAsync("https://services.nvd.nist.gov/api"));
    }
    
    [Fact]
    [Trait("Lane", "Security")]
    public async Task PayloadExceedsMaxSize_RejectsPayload()
    {
        var handler = CreateMockHandler(responseSize: 100_000_001); // 100MB
        var connector = CreateConnector(handler, maxPayloadBytes: 100_000_000);
        
        await Assert.ThrowsAsync<PayloadTooLargeException>(
            () => connector.FetchAsync("https://services.nvd.nist.gov/api"));
    }
    
    [Fact]
    [Trait("Lane", "Security")]
    public async Task DecompressionBomb_RejectsPayload()
    {
        // 1KB compressed, 1GB decompressed
        var handler = CreateMockHandler(compressedBomb: true);
        var connector = CreateConnector(handler);
        
        await Assert.ThrowsAsync<PayloadTooLargeException>(
            () => connector.FetchAsync("https://services.nvd.nist.gov/api"));
    }
}

Required Security Tests

Test	Purpose	Trait
URL allowlist	Block requests to unauthorized hosts	Security
Redirect validation	Block redirects to unauthorized hosts	Security
Max payload size	Reject oversized responses	Security
Decompression bomb	Reject zip bombs	Security
Rate limiting	Respect upstream rate limits	Security

5. Live Smoke Tests (Opt-In)

Purpose

Detect upstream schema drift by comparing live responses against known fixtures.

Pattern

public class NvdLiveTests : ConnectorLiveTestBase<NvdConnector>
{
    [Fact]
    [Trait("Lane", "Live")]
    [Trait("Category", "SchemaDrift")]
    public async Task LiveSchema_MatchesFixtureSchema()
    {
        // Skip if not in Live lane
        Skip.IfNot(IsLiveLaneEnabled());
        
        // Fetch live response
        var live = await Connector.FetchLatestAsync();
        
        // Compare schema (not values) against fixture
        var fixture = await LoadFixture<NvdResponse>("nvd-typical.json");
        
        AssertSchemaMatches(live, fixture);
    }
    
    [Fact]
    [Trait("Lane", "Live")]
    public async Task LiveFetch_ReturnsValidAdvisories()
    {
        Skip.IfNot(IsLiveLaneEnabled());
        
        var result = await Connector.FetchLatestAsync();
        
        Assert.True(result.IsSuccess);
        Assert.NotEmpty(result.Value.Advisories);
    }
}

Configuration

Live tests are:

Never PR-gating — run only in scheduled/nightly jobs
Opt-in — require explicit LIVE_TESTS_ENABLED=true environment variable
Alerting — schema drift triggers notification, not failure

6. Fixture Updater

Purpose

Refresh fixtures from live sources when upstream schemas change intentionally.

Usage

# Update all fixtures for NVD connector
dotnet run --project tools/FixtureUpdater -- \
  --connector nvd \
  --output src/Concelier/__Tests/StellaOps.Concelier.Connector.Nvd.Tests/Nvd/Fixtures/

# Update specific fixture
dotnet run --project tools/FixtureUpdater -- \
  --connector nvd \
  --cve CVE-2024-0001 \
  --output src/Concelier/__Tests/.../Fixtures/nvd-CVE-2024-0001.json

# Dry-run mode (show diff without writing)
dotnet run --project tools/FixtureUpdater -- \
  --connector nvd \
  --dry-run

Workflow

Live test detects schema drift
CI creates draft PR with fixture update
Developer reviews diff for intentional vs accidental changes
If intentional: update parser and merge
If accidental: investigate upstream API issue

7. Test Traits and CI Integration

Trait Assignment

Test Category	Trait	Lane	PR-Gating
Parser tests	`[Trait("Lane", "Unit")]`	Unit	Yes
Resilience tests	`[Trait("Lane", "Unit")]`	Unit	Yes
Security tests	`[Trait("Lane", "Security")]`	Security	Yes
Live tests	`[Trait("Lane", "Live")]`	Live	No

Running Tests

# Run all connector unit tests
dotnet test --filter "Lane=Unit" src/Concelier/__Tests/

# Run security tests
dotnet test --filter "Lane=Security" src/Concelier/__Tests/

# Run live tests (requires LIVE_TESTS_ENABLED=true)
LIVE_TESTS_ENABLED=true dotnet test --filter "Lane=Live" src/Concelier/__Tests/

8. Connector Inventory

Concelier Connectors (Advisory Sources)

Connector	Fixtures	Parser Tests	Resilience	Security	Live
NVD	✅	✅	✅	⬜	⬜
OSV	✅	✅	⬜	⬜	⬜
GHSA	✅	✅	⬜	⬜	⬜
CVE	✅	✅	⬜	⬜	⬜
KEV	✅	✅	⬜	⬜	⬜
EPSS	✅	✅	⬜	⬜	⬜
Distro.Alpine	⬜	⬜	⬜	⬜	⬜
Distro.Debian	✅	✅	⬜	⬜	⬜
Distro.RedHat	✅	✅	⬜	⬜	⬜
Distro.Suse	✅	✅	⬜	⬜	⬜
Distro.Ubuntu	✅	✅	⬜	⬜	⬜
Vndr.Adobe	⬜	⬜	⬜	⬜	⬜
Vndr.Apple	⬜	⬜	⬜	⬜	⬜
Vndr.Cisco	✅	✅	⬜	⬜	⬜
Vndr.Msrc	✅	✅	⬜	⬜	⬜
Vndr.Oracle	⬜	⬜	⬜	⬜	⬜
Vndr.Vmware	✅	✅	⬜	⬜	⬜
Cert.Bund	⬜	⬜	⬜	⬜	⬜
Cert.Cc	⬜	⬜	⬜	⬜	⬜
Cert.Fr	⬜	⬜	⬜	⬜	⬜
ICS.Cisa	✅	✅	⬜	⬜	⬜

Excititor Connectors (VEX Sources)

Connector	Fixtures	Parser Tests	Resilience	Security	Live
OpenVEX	⬜	⬜	⬜	⬜	⬜
CSAF/VEX	⬜	⬜	⬜	⬜	⬜

References

Last updated: 2025-12-24 · Sprint 5100.0007.0005

13 KiB Raw Blame History

Connector Fixture Discipline

Overview

1. Directory Structure

Example: NVD Connector

2. Fixture-Based Parser Tests

Purpose

Pattern

Fixture Requirements

Minimum Coverage

3. Resilience Tests

Purpose

Pattern

Required Test Cases

4. Security Tests

Purpose

Pattern

Required Security Tests

5. Live Smoke Tests (Opt-In)

Purpose

Pattern

Configuration

6. Fixture Updater

Purpose

Usage

Workflow

7. Test Traits and CI Integration

Trait Assignment

Running Tests

8. Connector Inventory

Concelier Connectors (Advisory Sources)

Excititor Connectors (VEX Sources)

References

13 KiB

Raw Blame History