git.stella-ops.org/docs/testing/connector-fixture-discipline.md

# Connector Fixture Discipline

This document defines the testing discipline for StellaOps Concelier and Excititor connectors. All connectors must follow these patterns to ensure consistent, deterministic, and offline-capable testing.

## Overview

Connector tests follow **Model C1 (Connector/External)** from the testing strategy:

1. **Fixture-based parser tests** — Raw upstream payload → normalized internal model (offline)
2. **Resilience tests** — Partial/bad input → deterministic failure classification
3. **Security tests** — URL allowlist, redirect handling, payload limits
4. **Live smoke tests** — Schema drift detection (opt-in, non-gating)

---

## 1. Directory Structure

Each connector test project follows this structure:

```
src/<Module>/__Tests/StellaOps.<Module>.Connector.<Source>.Tests/
├── StellaOps.<Module>.Connector.<Source>.Tests.csproj
├── Fixtures/
│   ├── <source>-typical.json          # Typical advisory payload
│   ├── <source>-edge-<case>.json      # Edge case payloads
│   ├── <source>-error-<type>.json     # Malformed/invalid payloads
│   └── expected-<id>.json             # Expected normalized output
├── <Source>/
│   ├── <Source>ParserTests.cs         # Parser unit tests
│   ├── <Source>ConnectorTests.cs      # Connector integration tests
│   └── <Source>ResilienceTests.cs     # Resilience/security tests
└── Expected/
    └── <source>-<id>.canonical.json   # Canonical JSON snapshots
```

### Example: NVD Connector

```
src/Concelier/__Tests/StellaOps.Concelier.Connector.Nvd.Tests/
├── Fixtures/
│   ├── nvd-window-1.json
│   ├── nvd-window-2.json
│   ├── nvd-multipage-1.json
│   ├── nvd-multipage-2.json
│   ├── nvd-invalid-schema.json
│   └── expected-CVE-2024-0001.json
├── Nvd/
│   ├── NvdParserTests.cs
│   ├── NvdConnectorTests.cs
│   └── NvdConnectorHarnessTests.cs
└── Expected/
    └── conflict-nvd.canonical.json
```

---

## 2. Fixture-Based Parser Tests

### Purpose

Test that the parser correctly transforms raw upstream payloads into normalized internal models without network access.

### Pattern

```csharp
using StellaOps.TestKit.Connectors;

public class NvdParserTests : ConnectorParserTestBase<NvdRawAdvisory, ConcelierAdvisory>
{
    public NvdParserTests() 
        : base(new NvdParser(), "Nvd/Fixtures")
    {
    }
    
    [Fact]
    [Trait("Lane", "Unit")]
    public async Task ParseTypicalAdvisory_ProducesExpectedModel()
    {
        // Arrange
        var raw = await LoadFixture<NvdRawAdvisory>("nvd-window-1.json");
        
        // Act
        var result = Parser.Parse(raw);
        
        // Assert
        await AssertMatchesSnapshot(result, "expected-CVE-2024-0001.json");
    }
    
    [Theory]
    [Trait("Lane", "Unit")]
    [InlineData("nvd-multipage-1.json", "expected-multipage-1.json")]
    [InlineData("nvd-multipage-2.json", "expected-multipage-2.json")]
    public async Task ParseAllFixtures_ProducesExpectedModels(string input, string expected)
    {
        var raw = await LoadFixture<NvdRawAdvisory>(input);
        var result = Parser.Parse(raw);
        await AssertMatchesSnapshot(result, expected);
    }
}
```

### Fixture Requirements

| Type | Naming Convention | Purpose |
|------|-------------------|---------|
| Typical | `<source>-typical.json` | Normal advisory with all common fields |
| Edge case | `<source>-edge-<case>.json` | Unusual but valid payloads |
| Error | `<source>-error-<type>.json` | Malformed/invalid payloads |
| Expected | `expected-<id>.json` | Expected normalized output |
| Canonical | `<source>-<id>.canonical.json` | Deterministic JSON snapshot |

### Minimum Coverage

Each connector must have fixtures for:

- [ ] At least 1 typical payload
- [ ] At least 2 edge cases (e.g., multi-vendor, unusual CVSS, missing optional fields)
- [ ] At least 2 error cases (e.g., missing required fields, invalid schema)

---

## 3. Resilience Tests

### Purpose

Verify that connectors handle malformed input gracefully with deterministic failure classification.

### Pattern

```csharp
public class NvdResilienceTests : ConnectorResilienceTestBase<NvdConnector>
{
    public NvdResilienceTests()
        : base(new NvdConnector(CreateTestHttpClient()))
    {
    }
    
    [Fact]
    [Trait("Lane", "Unit")]
    public async Task MissingRequiredField_ReturnsParseError()
    {
        // Arrange
        var payload = await LoadFixture("nvd-error-missing-cve-id.json");
        
        // Act
        var result = await Connector.ParseAsync(payload);
        
        // Assert
        Assert.False(result.IsSuccess);
        Assert.Equal(ConnectorErrorKind.ParseError, result.Error.Kind);
        Assert.Contains("cve_id", result.Error.Message, StringComparison.OrdinalIgnoreCase);
    }
    
    [Fact]
    [Trait("Lane", "Unit")]
    public async Task InvalidDateFormat_ReturnsParseError()
    {
        var payload = await LoadFixture("nvd-error-invalid-date.json");
        
        var result = await Connector.ParseAsync(payload);
        
        Assert.False(result.IsSuccess);
        Assert.Equal(ConnectorErrorKind.ParseError, result.Error.Kind);
    }
    
    [Fact]
    [Trait("Lane", "Unit")]
    public async Task UnexpectedEnumValue_LogsWarningAndContinues()
    {
        var payload = await LoadFixture("nvd-edge-unknown-severity.json");
        
        var result = await Connector.ParseAsync(payload);
        
        Assert.True(result.IsSuccess);
        Assert.Contains(result.Warnings, w => w.Contains("unknown severity"));
    }
}
```

### Required Test Cases

| Case | Expected Behavior | Trait |
|------|-------------------|-------|
| Missing required field | `ConnectorErrorKind.ParseError` | Unit |
| Invalid date format | `ConnectorErrorKind.ParseError` | Unit |
| Invalid JSON structure | `ConnectorErrorKind.ParseError` | Unit |
| Unknown enum value | Warning logged, continues | Unit |
| Empty response | `ConnectorErrorKind.EmptyResponse` | Unit |
| Truncated payload | `ConnectorErrorKind.ParseError` | Unit |

---

## 4. Security Tests

### Purpose

Verify that connectors enforce security boundaries for network operations.

### Pattern

```csharp
public class NvdSecurityTests : ConnectorSecurityTestBase<NvdConnector>
{
    [Fact]
    [Trait("Lane", "Security")]
    public async Task UrlOutsideAllowlist_RejectsRequest()
    {
        // Arrange
        var connector = CreateConnector(allowedHosts: ["services.nvd.nist.gov"]);
        
        // Act & Assert
        await Assert.ThrowsAsync<SecurityException>(
            () => connector.FetchAsync("https://evil.example.com/api"));
    }
    
    [Fact]
    [Trait("Lane", "Security")]
    public async Task RedirectToDisallowedHost_RejectsRequest()
    {
        var handler = CreateMockHandler(redirectTo: "https://evil.example.com");
        var connector = CreateConnector(handler, allowedHosts: ["services.nvd.nist.gov"]);
        
        await Assert.ThrowsAsync<SecurityException>(
            () => connector.FetchAsync("https://services.nvd.nist.gov/api"));
    }
    
    [Fact]
    [Trait("Lane", "Security")]
    public async Task PayloadExceedsMaxSize_RejectsPayload()
    {
        var handler = CreateMockHandler(responseSize: 100_000_001); // 100MB
        var connector = CreateConnector(handler, maxPayloadBytes: 100_000_000);
        
        await Assert.ThrowsAsync<PayloadTooLargeException>(
            () => connector.FetchAsync("https://services.nvd.nist.gov/api"));
    }
    
    [Fact]
    [Trait("Lane", "Security")]
    public async Task DecompressionBomb_RejectsPayload()
    {
        // 1KB compressed, 1GB decompressed
        var handler = CreateMockHandler(compressedBomb: true);
        var connector = CreateConnector(handler);
        
        await Assert.ThrowsAsync<PayloadTooLargeException>(
            () => connector.FetchAsync("https://services.nvd.nist.gov/api"));
    }
}
```

### Required Security Tests

| Test | Purpose | Trait |
|------|---------|-------|
| URL allowlist | Block requests to unauthorized hosts | Security |
| Redirect validation | Block redirects to unauthorized hosts | Security |
| Max payload size | Reject oversized responses | Security |
| Decompression bomb | Reject zip bombs | Security |
| Rate limiting | Respect upstream rate limits | Security |

---

## 5. Live Smoke Tests (Opt-In)

### Purpose

Detect upstream schema drift by comparing live responses against known fixtures.

### Pattern

```csharp
public class NvdLiveTests : ConnectorLiveTestBase<NvdConnector>
{
    [Fact]
    [Trait("Lane", "Live")]
    [Trait("Category", "SchemaDrift")]
    public async Task LiveSchema_MatchesFixtureSchema()
    {
        // Skip if not in Live lane
        Skip.IfNot(IsLiveLaneEnabled());
        
        // Fetch live response
        var live = await Connector.FetchLatestAsync();
        
        // Compare schema (not values) against fixture
        var fixture = await LoadFixture<NvdResponse>("nvd-typical.json");
        
        AssertSchemaMatches(live, fixture);
    }
    
    [Fact]
    [Trait("Lane", "Live")]
    public async Task LiveFetch_ReturnsValidAdvisories()
    {
        Skip.IfNot(IsLiveLaneEnabled());
        
        var result = await Connector.FetchLatestAsync();
        
        Assert.True(result.IsSuccess);
        Assert.NotEmpty(result.Value.Advisories);
    }
}
```

### Configuration

Live tests are:
- **Never PR-gating** — run only in scheduled/nightly jobs
- **Opt-in** — require explicit `LIVE_TESTS_ENABLED=true` environment variable
- **Alerting** — schema drift triggers notification, not failure

---

## 6. Fixture Updater

### Purpose

Refresh fixtures from live sources when upstream schemas change intentionally.

### Usage

```bash
# Update all fixtures for NVD connector
dotnet run --project tools/FixtureUpdater -- \
  --connector nvd \
  --output src/Concelier/__Tests/StellaOps.Concelier.Connector.Nvd.Tests/Nvd/Fixtures/

# Update specific fixture
dotnet run --project tools/FixtureUpdater -- \
  --connector nvd \
  --cve CVE-2024-0001 \
  --output src/Concelier/__Tests/.../Fixtures/nvd-CVE-2024-0001.json

# Dry-run mode (show diff without writing)
dotnet run --project tools/FixtureUpdater -- \
  --connector nvd \
  --dry-run
```

### Workflow

1. Live test detects schema drift
2. CI creates draft PR with fixture update
3. Developer reviews diff for intentional vs accidental changes
4. If intentional: update parser and merge
5. If accidental: investigate upstream API issue

---

## 7. Test Traits and CI Integration

### Trait Assignment

| Test Category | Trait | Lane | PR-Gating |
|---------------|-------|------|-----------|
| Parser tests | `[Trait("Lane", "Unit")]` | Unit | Yes |
| Resilience tests | `[Trait("Lane", "Unit")]` | Unit | Yes |
| Security tests | `[Trait("Lane", "Security")]` | Security | Yes |
| Live tests | `[Trait("Lane", "Live")]` | Live | No |

### Running Tests

```bash
# Run all connector unit tests
dotnet test --filter "Lane=Unit" src/Concelier/__Tests/

# Run security tests
dotnet test --filter "Lane=Security" src/Concelier/__Tests/

# Run live tests (requires LIVE_TESTS_ENABLED=true)
LIVE_TESTS_ENABLED=true dotnet test --filter "Lane=Live" src/Concelier/__Tests/
```

---

## 8. Connector Inventory

### Concelier Connectors (Advisory Sources)

| Connector | Fixtures | Parser Tests | Resilience | Security | Live |
|-----------|----------|--------------|------------|----------|------|
| NVD | ✅ | ✅ | ✅ | ⬜ | ⬜ |
| OSV | ✅ | ✅ | ⬜ | ⬜ | ⬜ |
| GHSA | ✅ | ✅ | ⬜ | ⬜ | ⬜ |
| CVE | ✅ | ✅ | ⬜ | ⬜ | ⬜ |
| KEV | ✅ | ✅ | ⬜ | ⬜ | ⬜ |
| EPSS | ✅ | ✅ | ⬜ | ⬜ | ⬜ |
| Distro.Alpine | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ |
| Distro.Debian | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ |
| Distro.RedHat | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ |
| Distro.Suse | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ |
| Distro.Ubuntu | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ |
| Vndr.Adobe | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ |
| Vndr.Apple | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ |
| Vndr.Cisco | ✅ | ✅ | ⬜ | ⬜ | ⬜ |
| Vndr.Msrc | ✅ | ✅ | ⬜ | ⬜ | ⬜ |
| Vndr.Oracle | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ |
| Vndr.Vmware | ✅ | ✅ | ⬜ | ⬜ | ⬜ |
| Cert.Bund | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ |
| Cert.Cc | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ |
| Cert.Fr | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ |
| ICS.Cisa | ✅ | ✅ | ⬜ | ⬜ | ⬜ |

### Excititor Connectors (VEX Sources)

| Connector | Fixtures | Parser Tests | Resilience | Security | Live |
|-----------|----------|--------------|------------|----------|------|
| OpenVEX | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ |
| CSAF/VEX | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ |

---

## References

- [ConnectorHttpFixture](../../src/__Libraries/StellaOps.TestKit/Connectors/ConnectorHttpFixture.cs)
- [ConnectorTestBase](../../src/__Libraries/StellaOps.TestKit/Connectors/ConnectorTestBase.cs)
- [ConnectorResilienceTestBase](../../src/__Libraries/StellaOps.TestKit/Connectors/ConnectorResilienceTestBase.cs)
- [FixtureUpdater](../../src/__Libraries/StellaOps.TestKit/Connectors/FixtureUpdater.cs)
- [Testing Strategy Models](./testing-strategy-models.md)
- [CI Lane Filters](./ci-lane-filters.md)

---

*Last updated: 2025-06-30 · Sprint 5100.0007.0005*