Refactor code structure for improved readability and maintainability; optimize performance in key functions.
This commit is contained in:
master
217
docs/interop/README.md
Normal file
217
docs/interop/README.md
Normal file
@@ -0,0 +1,217 @@
|
||||
# SBOM Interoperability Testing
|
||||
|
||||
## Overview
|
||||
|
||||
StellaOps SBOM interoperability tests ensure compatibility with third-party security tools in the ecosystem. The tests validate that StellaOps-generated SBOMs can be consumed by popular tools like Grype, and that findings parity remains above 95%.
|
||||
|
||||
## Test Coverage
|
||||
|
||||
### SBOM Formats
|
||||
|
||||
| Format | Version | Status | Parity Target |
|
||||
|--------|---------|--------|---------------|
|
||||
| CycloneDX | 1.6 | ✅ Supported | 95%+ |
|
||||
| SPDX | 3.0.1 | ✅ Supported | 95%+ |
|
||||
|
||||
### Third-Party Tools
|
||||
|
||||
| Tool | Purpose | Version | Status |
|
||||
|------|---------|---------|--------|
|
||||
| Syft | SBOM Generation | Latest | ✅ Compatible |
|
||||
| Grype | Vulnerability Scanning | Latest | ✅ Compatible |
|
||||
| cosign | Attestation | Latest | ✅ Compatible |
|
||||
|
||||
## Parity Expectations
|
||||
|
||||
### What is Parity?
|
||||
|
||||
Parity measures how closely StellaOps vulnerability findings match those from third-party tools like Grype when scanning the same SBOM.
|
||||
|
||||
**Formula:**
|
||||
```
|
||||
Parity % = (Matching Findings / Total Unique Findings) × 100
|
||||
```
|
||||
|
||||
**Target:** ≥95% parity for both CycloneDX and SPDX formats
|
||||
|
||||
### Known Differences
|
||||
|
||||
The following differences are **acceptable** and expected:
|
||||
|
||||
#### 1. VEX Application
|
||||
- **Difference:** StellaOps applies VEX documents, Grype may not
|
||||
- **Impact:** StellaOps may show fewer vulnerabilities
|
||||
- **Acceptable:** Yes - this is a feature, not a bug
|
||||
|
||||
#### 2. Feed Coverage
|
||||
- **Difference:** Tool-specific vulnerability databases
|
||||
- **Examples:**
|
||||
- StellaOps may have distro-specific feeds Grype lacks
|
||||
- Grype may have GitHub Advisory feeds StellaOps doesn't prioritize
|
||||
- **Acceptable:** Within 5% tolerance
|
||||
|
||||
#### 3. Version Matching Semantics
|
||||
- **Difference:** Interpretation of version ranges
|
||||
- **Examples:**
|
||||
- SemVer vs non-SemVer handling
|
||||
- Epoch handling in RPM/Debian packages
|
||||
- **Acceptable:** When using distro-native comparators
|
||||
|
||||
#### 4. Package Identification (PURL)
|
||||
- **Difference:** PURL generation strategies
|
||||
- **Examples:**
|
||||
- `pkg:npm/package` vs `pkg:npm/package@version`
|
||||
- Namespace handling
|
||||
- **Acceptable:** When functionally equivalent
|
||||
|
||||
## Running Interop Tests
|
||||
|
||||
### Prerequisites
|
||||
|
||||
Install required tools:
|
||||
|
||||
```bash
|
||||
# Install Syft
|
||||
curl -sSfL https://raw.githubusercontent.com/anchore/syft/main/install.sh | sh -s -- -b /usr/local/bin
|
||||
|
||||
# Install Grype
|
||||
curl -sSfL https://raw.githubusercontent.com/anchore/grype/main/install.sh | sh -s -- -b /usr/local/bin
|
||||
|
||||
# Install cosign
|
||||
curl -sSfL https://github.com/sigstore/cosign/releases/latest/download/cosign-linux-amd64 -o /usr/local/bin/cosign
|
||||
chmod +x /usr/local/bin/cosign
|
||||
```
|
||||
|
||||
### Local Execution
|
||||
|
||||
```bash
|
||||
# Run all interop tests
|
||||
dotnet test tests/interop/StellaOps.Interop.Tests
|
||||
|
||||
# Run CycloneDX tests only
|
||||
dotnet test tests/interop/StellaOps.Interop.Tests --filter "Format=CycloneDX"
|
||||
|
||||
# Run SPDX tests only
|
||||
dotnet test tests/interop/StellaOps.Interop.Tests --filter "Format=SPDX"
|
||||
|
||||
# Run parity tests
|
||||
dotnet test tests/interop/StellaOps.Interop.Tests --filter "Category=Parity"
|
||||
```
|
||||
|
||||
### CI Execution
|
||||
|
||||
Interop tests run automatically on:
|
||||
- Pull requests affecting scanner or SBOM code
|
||||
- Nightly schedule (6 AM UTC)
|
||||
- Manual workflow dispatch
|
||||
|
||||
See `.gitea/workflows/interop-e2e.yml` for CI configuration.
|
||||
|
||||
## Test Images
|
||||
|
||||
The following container images are used for interop testing:
|
||||
|
||||
| Image | Purpose | Characteristics |
|
||||
|-------|---------|-----------------|
|
||||
| `alpine:3.18` | Distro packages | APK packages, minimal |
|
||||
| `debian:12-slim` | Distro packages | DEB packages, medium |
|
||||
| `ubuntu:22.04` | Distro packages | DEB packages, larger |
|
||||
| `node:20-alpine` | Language packages | NPM packages |
|
||||
| `python:3.12-slim` | Language packages | Pip packages |
|
||||
| `golang:1.22-alpine` | Language packages | Go modules |
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Parity Below Threshold
|
||||
|
||||
If parity drops below 95%:
|
||||
|
||||
1. **Check for feed updates**
|
||||
- Grype may have newer vulnerability data
|
||||
- Update StellaOps feeds
|
||||
|
||||
2. **Review differences**
|
||||
- Run parity analysis: `dotnet test --filter "Category=Parity" --logger "console;verbosity=detailed"`
|
||||
- Categorize differences using `FindingsParityAnalyzer`
|
||||
|
||||
3. **Validate with golden corpus**
|
||||
- Compare against known-good results in `bench/golden-corpus/categories/interop/`
|
||||
|
||||
4. **Update acceptable differences**
|
||||
- Document new acceptable differences in this README
|
||||
- Adjust tolerance if justified
|
||||
|
||||
### Tool Installation Failures
|
||||
|
||||
If Syft/Grype/cosign fail to install:
|
||||
|
||||
```bash
|
||||
# Check versions
|
||||
syft --version
|
||||
grype --version
|
||||
cosign version
|
||||
|
||||
# Reinstall if needed
|
||||
rm /usr/local/bin/{syft,grype,cosign}
|
||||
# Re-run installation commands
|
||||
```
|
||||
|
||||
### SBOM Validation Failures
|
||||
|
||||
If SBOMs fail schema validation:
|
||||
|
||||
1. Verify format version:
|
||||
```bash
|
||||
jq '.specVersion' sbom-cyclonedx.json # Should be "1.6"
|
||||
jq '.spdxVersion' sbom-spdx.json # Should be "SPDX-3.0"
|
||||
```
|
||||
|
||||
2. Validate against official schemas:
|
||||
```bash
|
||||
# CycloneDX
|
||||
npm install -g @cyclonedx/cdx-cli
|
||||
cdx-cli validate --input-file sbom-cyclonedx.json
|
||||
|
||||
# SPDX (TODO: Add SPDX validation tool)
|
||||
```
|
||||
|
||||
## Continuous Improvement
|
||||
|
||||
### Adding New Test Cases
|
||||
|
||||
1. Add new image to test matrix in `*RoundTripTests.cs`
|
||||
2. Update `TestImages` member data
|
||||
3. Run locally to verify
|
||||
4. Submit PR with updated tests
|
||||
|
||||
### Updating Parity Thresholds
|
||||
|
||||
Current threshold: **95%**
|
||||
|
||||
To adjust:
|
||||
1. Document justification in sprint file
|
||||
2. Update `tolerancePercent` parameter in test calls
|
||||
3. Update this README
|
||||
|
||||
### Tool Version Pinning
|
||||
|
||||
Tools are currently installed from `latest`. To pin versions:
|
||||
|
||||
1. Update `.gitea/workflows/interop-e2e.yml`
|
||||
2. Specify version in install commands
|
||||
3. Document version compatibility in this README
|
||||
|
||||
## References
|
||||
|
||||
- [CycloneDX 1.6 Specification](https://cyclonedx.org/docs/1.6/)
|
||||
- [SPDX 3.0.1 Specification](https://spdx.github.io/spdx-spec/v3.0/)
|
||||
- [Syft Documentation](https://github.com/anchore/syft)
|
||||
- [Grype Documentation](https://github.com/anchore/grype)
|
||||
- [cosign Documentation](https://github.com/sigstore/cosign)
|
||||
|
||||
## Contacts
|
||||
|
||||
For questions about interop testing:
|
||||
- **Sprint:** SPRINT_5100_0003_0001
|
||||
- **Owner:** QA Team
|
||||
- **Dependencies:** Sprint 5100.0001.0002 (Evidence Index)
|
||||
Reference in New Issue
Block a user