git.stella-ops.org/docs/interop/README.md

# SBOM Interoperability Testing

## Overview

StellaOps SBOM interoperability tests ensure compatibility with third-party security tools in the ecosystem. The tests validate that StellaOps-generated SBOMs can be consumed by popular tools like Grype, and that findings parity remains above 95%.

## Test Coverage

### SBOM Formats

| Format | Version | Status | Parity Target |
|--------|---------|--------|---------------|
| CycloneDX | 1.6 | ✅ Supported | 95%+ |
| SPDX | 3.0.1 | ✅ Supported | 95%+ |

### Third-Party Tools

| Tool | Purpose | Version | Status |
|------|---------|---------|--------|
| Syft | SBOM Generation | Latest | ✅ Compatible |
| Grype | Vulnerability Scanning | Latest | ✅ Compatible |
| cosign | Attestation | Latest | ✅ Compatible |

## Parity Expectations

### What is Parity?

Parity measures how closely StellaOps vulnerability findings match those from third-party tools like Grype when scanning the same SBOM.

**Formula:**
```
Parity % = (Matching Findings / Total Unique Findings) × 100
```

**Target:** ≥95% parity for both CycloneDX and SPDX formats

### Known Differences

The following differences are **acceptable** and expected:

#### 1. VEX Application
- **Difference:** StellaOps applies VEX documents, Grype may not
- **Impact:** StellaOps may show fewer vulnerabilities
- **Acceptable:** Yes - this is a feature, not a bug

#### 2. Feed Coverage
- **Difference:** Tool-specific vulnerability databases
- **Examples:**
  - StellaOps may have distro-specific feeds Grype lacks
  - Grype may have GitHub Advisory feeds StellaOps doesn't prioritize
- **Acceptable:** Within 5% tolerance

#### 3. Version Matching Semantics
- **Difference:** Interpretation of version ranges
- **Examples:**
  - SemVer vs non-SemVer handling
  - Epoch handling in RPM/Debian packages
- **Acceptable:** When using distro-native comparators

#### 4. Package Identification (PURL)
- **Difference:** PURL generation strategies
- **Examples:**
  - `pkg:npm/package` vs `pkg:npm/package@version`
  - Namespace handling
- **Acceptable:** When functionally equivalent

## Running Interop Tests

### Prerequisites

Install required tools:

```bash
# Install Syft
curl -sSfL https://raw.githubusercontent.com/anchore/syft/main/install.sh | sh -s -- -b /usr/local/bin

# Install Grype
curl -sSfL https://raw.githubusercontent.com/anchore/grype/main/install.sh | sh -s -- -b /usr/local/bin

# Install cosign
curl -sSfL https://github.com/sigstore/cosign/releases/latest/download/cosign-linux-amd64 -o /usr/local/bin/cosign
chmod +x /usr/local/bin/cosign
```

### Local Execution

```bash
# Run all interop tests
dotnet test tests/interop/StellaOps.Interop.Tests

# Run CycloneDX tests only
dotnet test tests/interop/StellaOps.Interop.Tests --filter "Format=CycloneDX"

# Run SPDX tests only
dotnet test tests/interop/StellaOps.Interop.Tests --filter "Format=SPDX"

# Run parity tests
dotnet test tests/interop/StellaOps.Interop.Tests --filter "Category=Parity"
```

### CI Execution

Interop tests run automatically on:
- Pull requests affecting scanner or SBOM code
- Nightly schedule (6 AM UTC)
- Manual workflow dispatch

See `.gitea/workflows/interop-e2e.yml` for CI configuration.

## Test Images

The following container images are used for interop testing:

| Image | Purpose | Characteristics |
|-------|---------|-----------------|
| `alpine:3.18` | Distro packages | APK packages, minimal |
| `debian:12-slim` | Distro packages | DEB packages, medium |
| `ubuntu:22.04` | Distro packages | DEB packages, larger |
| `node:20-alpine` | Language packages | NPM packages |
| `python:3.12-slim` | Language packages | Pip packages |
| `golang:1.22-alpine` | Language packages | Go modules |

## Troubleshooting

### Parity Below Threshold

If parity drops below 95%:

1. **Check for feed updates**
   - Grype may have newer vulnerability data
   - Update StellaOps feeds

2. **Review differences**
   - Run parity analysis: `dotnet test --filter "Category=Parity" --logger "console;verbosity=detailed"`
   - Categorize differences using `FindingsParityAnalyzer`

3. **Validate with golden corpus**
   - Compare against known-good results in `bench/golden-corpus/categories/interop/`

4. **Update acceptable differences**
   - Document new acceptable differences in this README
   - Adjust tolerance if justified

### Tool Installation Failures

If Syft/Grype/cosign fail to install:

```bash
# Check versions
syft --version
grype --version
cosign version

# Reinstall if needed
rm /usr/local/bin/{syft,grype,cosign}
# Re-run installation commands
```

### SBOM Validation Failures

If SBOMs fail schema validation:

1. Verify format version:
   ```bash
   jq '.specVersion' sbom-cyclonedx.json  # Should be "1.6"
   jq '.spdxVersion' sbom-spdx.json       # Should be "SPDX-3.0"
   ```

2. Validate against official schemas:
   ```bash
   # CycloneDX
   npm install -g @cyclonedx/cdx-cli
   cdx-cli validate --input-file sbom-cyclonedx.json

   # SPDX (TODO: Add SPDX validation tool)
   ```

## Continuous Improvement

### Adding New Test Cases

1. Add new image to test matrix in `*RoundTripTests.cs`
2. Update `TestImages` member data
3. Run locally to verify
4. Submit PR with updated tests

### Updating Parity Thresholds

Current threshold: **95%**

To adjust:
1. Document justification in sprint file
2. Update `tolerancePercent` parameter in test calls
3. Update this README

### Tool Version Pinning

Tools are currently installed from `latest`. To pin versions:

1. Update `.gitea/workflows/interop-e2e.yml`
2. Specify version in install commands
3. Document version compatibility in this README

## References

- [CycloneDX 1.6 Specification](https://cyclonedx.org/docs/1.6/)
- [SPDX 3.0.1 Specification](https://spdx.github.io/spdx-spec/v3.0/)
- [Syft Documentation](https://github.com/anchore/syft)
- [Grype Documentation](https://github.com/anchore/grype)
- [cosign Documentation](https://github.com/sigstore/cosign)

## Contacts

For questions about interop testing:
- **Sprint:** SPRINT_5100_0003_0001
- **Owner:** QA Team
- **Dependencies:** Sprint 5100.0001.0002 (Evidence Index)