git.stella-ops.org/docs/modules/scanner/design/competitor-anomaly-tests.md

Competitor Ingest Anomaly Regression Tests (CM4)

Status: Draft · Date: 2025-12-04 Scope: Define anomaly regression test suite for ingest pipeline covering schema drift, nullables, encoding, and ordering anomalies.

Objectives

• Detect schema drift in upstream tool outputs.
• Validate handling of nullable/missing fields.
• Ensure proper encoding handling (UTF-8, escaping).
• Verify deterministic ordering is maintained.
• Provide golden fixtures with expected hashes.

Test Categories

1. Schema Drift Tests

Detect when upstream tools change their output schema.

tests/anomaly/schema-drift/
├── syft/
│   ├── v1.0.0-baseline.json        # Known good output
│   ├── v1.5.0-new-fields.json      # Added fields
│   ├── v1.5.0-removed-fields.json  # Removed fields
│   ├── v1.5.0-type-change.json     # Field type changed
│   └── expected-results.json
├── trivy/
│   └── ... (same structure)
└── clair/
    └── ... (same structure)

Test Cases

Test	Input	Expected Behavior
new_optional_field	Output with new field	Accept, ignore new field
new_required_field	Output with new required field	Warn, map if possible
removed_optional_field	Output missing optional field	Accept
removed_required_field	Output missing required field	Reject
field_type_change	Field type differs from schema	Reject or coerce
field_rename	Field renamed without mapping	Warn, check mapping

Schema Drift Fixture

{
  "test": "new_optional_field",
  "tool": "syft",
  "inputVersion": "1.5.0",
  "baselineVersion": "1.0.0",
  "input": {
    "artifacts": [
      {
        "name": "lib-a",
        "version": "1.0.0",
        "purl": "pkg:npm/lib-a@1.0.0",
        "newField": "unexpected value"
      }
    ]
  },
  "expected": {
    "status": "accepted",
    "warnings": ["unknown_field:newField"],
    "normalizedHash": "b3:..."
  }
}

2. Nullable/Missing Field Tests

Validate handling of null, empty, and missing values.

tests/anomaly/nullables/
├── null-values.json
├── empty-strings.json
├── empty-arrays.json
├── missing-optional.json
├── missing-required.json
└── expected-results.json

Test Cases

Test	Input	Expected Behavior
null_optional	Optional field is null	Accept, omit from output
null_required	Required field is null	Reject
empty_string	String field is ""	Accept, preserve or omit
empty_array	Array field is []	Accept, preserve
missing_optional	Optional field absent	Accept
missing_required	Required field absent	Reject

Nullable Fixture

{
  "test": "null_optional",
  "tool": "syft",
  "input": {
    "artifacts": [
      {
        "name": "lib-a",
        "version": "1.0.0",
        "purl": "pkg:npm/lib-a@1.0.0",
        "licenses": null
      }
    ]
  },
  "expected": {
    "status": "accepted",
    "output": {
      "components": [
        {
          "name": "lib-a",
          "version": "1.0.0",
          "purl": "pkg:npm/lib-a@1.0.0"
        }
      ]
    },
    "normalizedHash": "b3:..."
  }
}

3. Encoding Tests

Validate proper handling of character encoding and escaping.

tests/anomaly/encoding/
├── utf8-valid.json
├── utf8-bom.json
├── latin1-fallback.json
├── unicode-escapes.json
├── special-chars.json
├── json-escaping.json
└── expected-results.json

Test Cases

Test	Input	Expected Behavior
utf8_valid	Standard UTF-8	Accept
utf8_bom	UTF-8 with BOM	Accept, strip BOM
unicode_escapes	\u0041 style escapes	Accept, decode
special_chars	Tabs, newlines in strings	Accept, preserve or escape
control_chars	Control characters (0x00-0x1F)	Reject or sanitize
surrogate_pairs	Emoji and supplementary chars	Accept

Encoding Fixture

{
  "test": "special_chars",
  "tool": "syft",
  "input": {
    "artifacts": [
      {
        "name": "lib-with-tab\ttab",
        "version": "1.0.0",
        "description": "Line1\nLine2"
      }
    ]
  },
  "expected": {
    "status": "accepted",
    "output": {
      "components": [
        {
          "name": "lib-with-tab\ttab",
          "version": "1.0.0"
        }
      ]
    },
    "normalizedHash": "b3:..."
  }
}

4. Ordering Tests

Verify deterministic ordering is maintained across inputs.

tests/anomaly/ordering/
├── unsorted-components.json
├── reversed-components.json
├── random-order.json
├── unicode-sort.json
├── case-sensitivity.json
└── expected-results.json

Test Cases

Test	Input	Expected Behavior
unsorted_input	Components in random order	Sort deterministically
reversed_input	Components in reverse order	Sort deterministically
same_after_sort	Pre-sorted input	Same output as unsorted
unicode_sort	Unicode component names	Locale-invariant sort
case_sensitivity	Mixed case names	Case-insensitive sort

Ordering Fixture

{
  "test": "unsorted_input",
  "tool": "syft",
  "input": {
    "artifacts": [
      {"name": "zebra", "version": "1.0.0", "purl": "pkg:npm/zebra@1.0.0"},
      {"name": "apple", "version": "1.0.0", "purl": "pkg:npm/apple@1.0.0"},
      {"name": "mango", "version": "1.0.0", "purl": "pkg:npm/mango@1.0.0"}
    ]
  },
  "expected": {
    "status": "accepted",
    "output": {
      "components": [
        {"name": "apple", "version": "1.0.0", "purl": "pkg:npm/apple@1.0.0"},
        {"name": "mango", "version": "1.0.0", "purl": "pkg:npm/mango@1.0.0"},
        {"name": "zebra", "version": "1.0.0", "purl": "pkg:npm/zebra@1.0.0"}
      ]
    },
    "normalizedHash": "b3:..."
  }
}

Golden Fixtures

Hash File Format

# tests/anomaly/hashes.txt
schema-drift/syft/v1.0.0-baseline.json: BLAKE3=... SHA256=...
schema-drift/syft/expected-results.json: BLAKE3=... SHA256=...
nullables/null-values.json: BLAKE3=... SHA256=...
nullables/expected-results.json: BLAKE3=... SHA256=...
encoding/utf8-valid.json: BLAKE3=... SHA256=...
encoding/expected-results.json: BLAKE3=... SHA256=...
ordering/unsorted-components.json: BLAKE3=... SHA256=...
ordering/expected-results.json: BLAKE3=... SHA256=...

CI Integration

Test Workflow

# .gitea/workflows/anomaly-tests.yml
name: Anomaly Regression Tests

on:
  push:
    paths:
      - 'src/Scanner/Adapters/**'
      - 'tests/anomaly/**'
  pull_request:

jobs:
  anomaly-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup .NET
        uses: actions/setup-dotnet@v4
        with:
          dotnet-version: '10.0.x'

      - name: Verify fixture hashes
        run: scripts/scanner/verify-anomaly-fixtures.sh

      - name: Run schema drift tests
        run: |
          dotnet test src/Scanner/__Tests/StellaOps.Scanner.Anomaly.Tests \
            --filter "Category=SchemaDrift"

      - name: Run nullable tests
        run: |
          dotnet test src/Scanner/__Tests/StellaOps.Scanner.Anomaly.Tests \
            --filter "Category=Nullable"

      - name: Run encoding tests
        run: |
          dotnet test src/Scanner/__Tests/StellaOps.Scanner.Anomaly.Tests \
            --filter "Category=Encoding"

      - name: Run ordering tests
        run: |
          dotnet test src/Scanner/__Tests/StellaOps.Scanner.Anomaly.Tests \
            --filter "Category=Ordering"

Test Runner

// src/Scanner/__Tests/StellaOps.Scanner.Anomaly.Tests/AnomalyTestRunner.cs
[Category("SchemaDrift")]
[Theory]
[MemberData(nameof(GetSchemaDriftTestCases))]
public async Task SchemaDrift_HandledCorrectly(AnomalyTestCase testCase)
{
    // Arrange
    var adapter = _adapterFactory.Create(testCase.Tool);

    // Act
    var result = await adapter.NormalizeAsync(testCase.Input);

    // Assert
    Assert.Equal(testCase.Expected.Status, result.Status);
    Assert.Equal(testCase.Expected.Warnings, result.Warnings);

    if (testCase.Expected.NormalizedHash != null)
    {
        var hash = Blake3.HashData(Encoding.UTF8.GetBytes(
            JsonSerializer.Serialize(result.Output)));
        Assert.Equal(testCase.Expected.NormalizedHash,
            $"b3:{Convert.ToHexString(hash).ToLowerInvariant()}");
    }
}

Failure Handling

On Test Failure

1. Schema Drift: Create issue, update adapter mapping
2. Nullable Handling: Fix normalization logic
3. Encoding Error: Fix encoding detection/conversion
4. Ordering Violation: Fix sort comparator

Failure Report

{
  "failure": {
    "category": "schema_drift",
    "test": "new_required_field",
    "tool": "syft",
    "input": {...},
    "expected": {...},
    "actual": {...},
    "diff": [
      {"path": "/status", "expected": "accepted", "actual": "rejected"}
    ],
    "timestamp": "2025-12-04T12:00:00Z"
  }
}

Links

• Sprint: docs/implplan/SPRINT_0186_0001_0001_record_deterministic_execution.md (CM4)
• Normalization: docs/modules/scanner/design/competitor-ingest-normalization.md (CM1)
• Fixtures: docs/modules/scanner/fixtures/competitor-adapters/