git.stella-ops.org/docs/interop/README.md

SBOM Interoperability Testing

Overview

StellaOps SBOM interoperability tests ensure compatibility with third-party security tools in the ecosystem. The tests validate that StellaOps-generated SBOMs can be consumed by popular tools like Grype, and that findings parity remains above 95%.

Test Coverage

SBOM Formats

Format	Version	Status	Parity Target
CycloneDX	1.6	✅ Supported	95%+
SPDX	3.0.1	✅ Supported	95%+

Third-Party Tools

Tool	Purpose	Version	Status
Syft	SBOM Generation	Latest	✅ Compatible
Grype	Vulnerability Scanning	Latest	✅ Compatible
cosign	Attestation	Latest	✅ Compatible

Parity Expectations

What is Parity?

Parity measures how closely StellaOps vulnerability findings match those from third-party tools like Grype when scanning the same SBOM.

Formula:

Parity % = (Matching Findings / Total Unique Findings) × 100

Target: ≥95% parity for both CycloneDX and SPDX formats

Known Differences

The following differences are acceptable and expected:

1. VEX Application

• Difference: StellaOps applies VEX documents, Grype may not
• Impact: StellaOps may show fewer vulnerabilities
• Acceptable: Yes - this is a feature, not a bug

2. Feed Coverage

• Difference: Tool-specific vulnerability databases
• Examples:
  • StellaOps may have distro-specific feeds Grype lacks
  • Grype may have GitHub Advisory feeds StellaOps doesn't prioritize
• Acceptable: Within 5% tolerance

3. Version Matching Semantics

• Difference: Interpretation of version ranges
• Examples:
  • SemVer vs non-SemVer handling
  • Epoch handling in RPM/Debian packages
• Acceptable: When using distro-native comparators

4. Package Identification (PURL)

• Difference: PURL generation strategies
• Examples:
  • pkg:npm/package vs pkg:npm/package@version
  • Namespace handling
• Acceptable: When functionally equivalent

Running Interop Tests

Prerequisites

Install required tools:

# Install Syft
curl -sSfL https://raw.githubusercontent.com/anchore/syft/main/install.sh | sh -s -- -b /usr/local/bin

# Install Grype
curl -sSfL https://raw.githubusercontent.com/anchore/grype/main/install.sh | sh -s -- -b /usr/local/bin

# Install cosign
curl -sSfL https://github.com/sigstore/cosign/releases/latest/download/cosign-linux-amd64 -o /usr/local/bin/cosign
chmod +x /usr/local/bin/cosign

Local Execution

# Run all interop tests
dotnet test tests/interop/StellaOps.Interop.Tests

# Run CycloneDX tests only
dotnet test tests/interop/StellaOps.Interop.Tests --filter "Format=CycloneDX"

# Run SPDX tests only
dotnet test tests/interop/StellaOps.Interop.Tests --filter "Format=SPDX"

# Run parity tests
dotnet test tests/interop/StellaOps.Interop.Tests --filter "Category=Parity"

CI Execution

Interop tests run automatically on:

• Pull requests affecting scanner or SBOM code
• Nightly schedule (6 AM UTC)
• Manual workflow dispatch

See .gitea/workflows/interop-e2e.yml for CI configuration.

Test Images

The following container images are used for interop testing:

Image	Purpose	Characteristics
alpine:3.18	Distro packages	APK packages, minimal
debian:12-slim	Distro packages	DEB packages, medium
ubuntu:22.04	Distro packages	DEB packages, larger
node:20-alpine	Language packages	NPM packages
python:3.12-slim	Language packages	Pip packages
golang:1.22-alpine	Language packages	Go modules

Troubleshooting

Parity Below Threshold

If parity drops below 95%:

1. Check for feed updates

   • Grype may have newer vulnerability data
   • Update StellaOps feeds

2. Review differences

   • Run parity analysis: dotnet test --filter "Category=Parity" --logger "console;verbosity=detailed"
   • Categorize differences using FindingsParityAnalyzer

3. Validate with golden corpus

   • Compare against known-good results in bench/golden-corpus/categories/interop/

4. Update acceptable differences

   • Document new acceptable differences in this README
   • Adjust tolerance if justified

Tool Installation Failures

If Syft/Grype/cosign fail to install:

# Check versions
syft --version
grype --version
cosign version

# Reinstall if needed
rm /usr/local/bin/{syft,grype,cosign}
# Re-run installation commands

SBOM Validation Failures

If SBOMs fail schema validation:

1. Verify format version:

   jq '.specVersion' sbom-cyclonedx.json  # Should be "1.6"
   jq '.spdxVersion' sbom-spdx.json       # Should be "SPDX-3.0"
   

2. Validate against official schemas:

   # CycloneDX
   npm install -g @cyclonedx/cdx-cli
   cdx-cli validate --input-file sbom-cyclonedx.json
   
   # SPDX (TODO: Add SPDX validation tool)
   

Continuous Improvement

Adding New Test Cases

1. Add new image to test matrix in *RoundTripTests.cs
2. Update TestImages member data
3. Run locally to verify
4. Submit PR with updated tests

Updating Parity Thresholds

Current threshold: 95%

To adjust:

1. Document justification in sprint file
2. Update tolerancePercent parameter in test calls
3. Update this README

Tool Version Pinning

Tools are currently installed from latest. To pin versions:

1. Update .gitea/workflows/interop-e2e.yml
2. Specify version in install commands
3. Document version compatibility in this README

References

• CycloneDX 1.6 Specification
• SPDX 3.0.1 Specification
• Syft Documentation
• Grype Documentation
• cosign Documentation

Contacts

For questions about interop testing:

• Sprint: SPRINT_5100_0003_0001
• Owner: QA Team
• Dependencies: Sprint 5100.0001.0002 (Evidence Index)