152 lines
7.7 KiB
Markdown
152 lines
7.7 KiB
Markdown
# Concelier Fixture Maintenance
|
|
|
|
Concelier uses a handful of deterministic fixtures to keep connector regressions in check. This guide lists the
|
|
fixture sets, where they live, and how to regenerate them safely.
|
|
|
|
---
|
|
|
|
## GHSA ↔ OSV parity fixtures
|
|
|
|
- **Location:** `src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Osv.Tests/Fixtures/osv-ghsa.*.json`
|
|
- **Purpose:** Exercised by `OsvGhsaParityRegressionTests` to ensure OSV + GHSA outputs stay aligned on aliases,
|
|
ranges, references, and credits.
|
|
- **Regeneration:** Either run the test harness with online regeneration (`UPDATE_PARITY_FIXTURES=1 dotnet test src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Osv.Tests/StellaOps.Concelier.Connector.Osv.Tests.csproj`)
|
|
or execute the fixture updater (`dotnet run --project src/Tools/FixtureUpdater/FixtureUpdater.csproj`). Both paths
|
|
normalise timestamps and canonical ordering.
|
|
- **SemVer provenance:** The regenerated fixtures should show `normalizedVersions[].notes` in the
|
|
`osv:{ecosystem}:{advisoryId}:{identifier}` shape emitted by `SemVerRangeRuleBuilder`. Confirm the
|
|
constraints and notes line up with GHSA/NVD composites before committing.
|
|
- **Verification:** Inspect the diff, then re-run `dotnet test src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Osv.Tests/StellaOps.Concelier.Connector.Osv.Tests.csproj` to confirm parity.
|
|
|
|
## GHSA credit parity fixtures
|
|
|
|
- **Location:** `src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ghsa.Tests/Fixtures/credit-parity.{ghsa,osv,nvd}.json`
|
|
- **Purpose:** Exercised by `GhsaCreditParityRegressionTests` to guarantee GHSA/NVD/OSV acknowledgements remain in lockstep.
|
|
- **Regeneration:** `dotnet run --project src/Tools/FixtureUpdater/FixtureUpdater.csproj` rewrites all three canonical snapshots.
|
|
- **Verification:** `dotnet test src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ghsa.Tests/StellaOps.Concelier.Connector.Ghsa.Tests.csproj`.
|
|
|
|
> Always commit fixture changes together with the code that motivated them and reference the regression test that guards the behaviour.
|
|
|
|
## Apple security update fixtures
|
|
|
|
- **Location:** `src/Concelier/__Tests/StellaOps.Concelier.Connector.Vndr.Apple.Tests/Apple/Fixtures/*.html` and `.expected.json`.
|
|
- **Purpose:** Exercised by `AppleLiveRegressionTests` to guarantee the Apple HTML parser and mapper stay deterministic while covering Rapid Security Responses and multi-device advisories.
|
|
- **Regeneration:** Use the helper scripts (`scripts/update-apple-fixtures.sh` or `scripts/update-apple-fixtures.ps1`). They export `UPDATE_APPLE_FIXTURES=1`, propagate the flag through `WSLENV`, touch `.update-apple-fixtures`, and then run the Apple test project. This keeps WSL/VSCode test invocations in sync while the refresh workflow fetches live Apple support pages, sanitises them, and rewrites both the HTML and expected DTO snapshots with normalised ordering.
|
|
- **Verification:** Inspect the generated diffs and re-run `dotnet test src/Concelier/__Tests/StellaOps.Concelier.Connector.Vndr.Apple.Tests/StellaOps.Concelier.Connector.Vndr.Apple.Tests.csproj` without the env var to confirm determinism.
|
|
|
|
> **Tip for other connector owners:** mirror the sentinel + `WSLENV` pattern (`touch .update-<connector>-fixtures`, append the env var via `WSLENV`) when you add fixture refresh scripts so contributors running under WSL inherit the regeneration flag automatically.
|
|
|
|
## KISA advisory fixtures
|
|
|
|
- **Location:** `src/Concelier/__Tests/StellaOps.Concelier.Connector.Kisa.Tests/Fixtures/kisa-{feed,detail}.(xml|json)`
|
|
- **Purpose:** Used by `KisaConnectorTests` to verify Hangul-aware fetch → parse → map flows and to assert telemetry counters stay wired.
|
|
- **Regeneration:** `UPDATE_KISA_FIXTURES=1 dotnet test src/Concelier/__Tests/StellaOps.Concelier.Connector.Kisa.Tests/StellaOps.Concelier.Connector.Kisa.Tests.csproj`
|
|
- **Verification:** Re-run the same test suite without the env var; confirm advisory content remains NFC-normalised and HTML is sanitised. Metrics assertions will fail if counters drift.
|
|
- **Localisation note:** RSS `category` values (e.g. `취약점정보`) remain in Hangul—do not translate them in fixtures; they feed directly into metrics/log tags.
|
|
|
|
---
|
|
|
|
## Fixture Tiers & Retention Rules
|
|
|
|
> Added in Sprint: SPRINT_20251229_004_LIB_fixture_harvester (FH-010)
|
|
|
|
### Tier Classification
|
|
|
|
Test fixtures in StellaOps are classified into tiers based on their source, purpose, and maintenance requirements:
|
|
|
|
| Tier | Name | Purpose | Retention |
|
|
|------|------|---------|-----------|
|
|
| **T0** | Synthetic | Minimal, deterministic fixtures for unit testing | Permanent |
|
|
| **T1** | Specification Examples | Reference fixtures from CycloneDX, SPDX, OpenVEX specs | Per spec version |
|
|
| **T2** | Real-World Samples | Production-representative fixtures from real ecosystems | 6 months + current |
|
|
| **T3** | Regression | Fixtures capturing specific bugs or production incidents | Permanent |
|
|
|
|
### Tier Details
|
|
|
|
**T0 - Synthetic Fixtures**
|
|
- Source: Generated/hand-crafted
|
|
- Size: Minimal (< 100 KB)
|
|
- External Dependencies: None
|
|
- Refresh Policy: Manual only
|
|
- Use Cases: Unit tests, schema validation, edge cases, air-gap testing
|
|
|
|
**T1 - Specification Examples**
|
|
- Source: CycloneDX, SPDX, OpenVEX official specs
|
|
- Size: Small to medium (< 1 MB)
|
|
- External Dependencies: Spec repositories
|
|
- Refresh Policy: Quarterly (with spec updates)
|
|
- Use Cases: Format compliance, parser validation, interoperability
|
|
|
|
**T2 - Real-World Samples**
|
|
- Source: Public registries, OSS projects
|
|
- Size: Medium (< 10 MB)
|
|
- External Dependencies: Public APIs
|
|
- Refresh Policy: Monthly or on-demand
|
|
- Use Cases: Integration testing, performance benchmarks
|
|
|
|
**T3 - Regression Fixtures**
|
|
- Source: Bug reports, production incidents
|
|
- Size: Varies
|
|
- External Dependencies: None (self-contained)
|
|
- Refresh Policy: Never (historical record)
|
|
- Use Cases: Regression prevention, bug reproduction
|
|
|
|
### Storage Guidelines
|
|
|
|
| Tier | Git Storage | LFS Required | Archive |
|
|
|------|-------------|--------------|---------|
|
|
| T0 | Direct | No | No |
|
|
| T1 | Direct | Optional | Spec releases |
|
|
| T2 | Via LFS | Yes (> 1MB) | Monthly snapshots |
|
|
| T3 | Direct | If > 1MB | Incident reports |
|
|
|
|
### Fixture Harvester Tool
|
|
|
|
The `FixtureHarvester` CLI tool manages fixture acquisition and validation:
|
|
|
|
```bash
|
|
# Harvest a new fixture
|
|
dotnet run --project src/__Tests/Tools/FixtureHarvester harvest --type sbom --id my-fixture --source https://example.com/sbom.json
|
|
|
|
# Validate all fixtures
|
|
dotnet run --project src/__Tests/Tools/FixtureHarvester validate
|
|
|
|
# Regenerate expected outputs (requires confirmation)
|
|
dotnet run --project src/__Tests/Tools/FixtureHarvester regen --fixture my-fixture --confirm
|
|
```
|
|
|
|
### Fixture Directory Structure
|
|
|
|
```
|
|
src/__Tests/fixtures/
|
|
├── fixtures.manifest.yml # Root manifest
|
|
├── sbom/
|
|
│ └── <fixture-id>/
|
|
│ ├── meta.json # Provenance and metadata
|
|
│ ├── raw/ # Original files
|
|
│ ├── normalized/ # Processed files
|
|
│ └── expected/ # Expected outputs
|
|
├── feeds/
|
|
│ └── <fixture-id>/...
|
|
└── vex/
|
|
└── <fixture-id>/...
|
|
```
|
|
|
|
### meta.json Schema
|
|
|
|
```json
|
|
{
|
|
"id": "fixture-id",
|
|
"source": "local-build | url | api | manual",
|
|
"sourceUrl": "https://...",
|
|
"retrievedAt": "2025-12-29T00:00:00Z",
|
|
"license": "CC0-1.0",
|
|
"sha256": "sha256:...",
|
|
"refreshPolicy": "manual | monthly | quarterly",
|
|
"tier": "T0 | T1 | T2 | T3",
|
|
"notes": "Additional context"
|
|
}
|
|
```
|
|
|
|
See also: [FixtureHarvester README](../src/__Tests/Tools/FixtureHarvester/README.md)
|