docs consolidation
This commit is contained in:
StellaOps Bot
202
docs/testing/schema-validation.md
Normal file
202
docs/testing/schema-validation.md
Normal file
@@ -0,0 +1,202 @@
|
||||
# SBOM Schema Validation
|
||||
|
||||
This document describes the schema validation system for SBOM (Software Bill of Materials) fixtures in StellaOps.
|
||||
|
||||
## Overview
|
||||
|
||||
StellaOps validates all SBOM fixtures against official JSON schemas to detect schema drift before runtime. This ensures:
|
||||
|
||||
- CycloneDX 1.6 fixtures are compliant with the official schema
|
||||
- SPDX 3.0.1 fixtures meet specification requirements
|
||||
- OpenVEX fixtures follow the 0.2.0 specification
|
||||
- Invalid fixtures are detected early in the CI pipeline
|
||||
|
||||
## Supported Formats
|
||||
|
||||
| Format | Version | Schema Location | Validator |
|
||||
|--------|---------|-----------------|-----------|
|
||||
| CycloneDX | 1.6 | `docs/schemas/cyclonedx-bom-1.6.schema.json` | sbom-utility |
|
||||
| SPDX | 3.0.1 | `docs/schemas/spdx-jsonld-3.0.1.schema.json` | pyspdxtools / check-jsonschema |
|
||||
| OpenVEX | 0.2.0 | `docs/schemas/openvex-0.2.0.schema.json` | ajv-cli |
|
||||
|
||||
## CI Workflows
|
||||
|
||||
### Schema Validation Workflow
|
||||
|
||||
**File:** `.gitea/workflows/schema-validation.yml`
|
||||
|
||||
Runs on:
|
||||
- Pull requests touching `bench/golden-corpus/**`, `src/Scanner/**`, `docs/schemas/**`, or `scripts/validate-*.sh`
|
||||
- Push to `main` branch
|
||||
|
||||
Jobs:
|
||||
1. **validate-cyclonedx** - Validates all CycloneDX 1.6 fixtures
|
||||
2. **validate-spdx** - Validates all SPDX 3.0.1 fixtures
|
||||
3. **validate-vex** - Validates all OpenVEX 0.2.0 fixtures
|
||||
4. **validate-negative** - Verifies invalid fixtures are correctly rejected
|
||||
5. **summary** - Aggregates results
|
||||
|
||||
### Determinism Gate Integration
|
||||
|
||||
**File:** `.gitea/workflows/determinism-gate.yml`
|
||||
|
||||
The determinism gate includes schema validation as a prerequisite step. If schema validation fails, determinism checks are blocked.
|
||||
|
||||
To skip schema validation (e.g., during debugging):
|
||||
```bash
|
||||
# Via workflow_dispatch
|
||||
skip_schema_validation: true
|
||||
```
|
||||
|
||||
## Fixture Directories
|
||||
|
||||
Validation scans these directories for SBOM fixtures:
|
||||
|
||||
| Directory | Purpose |
|
||||
|-----------|---------|
|
||||
| `bench/golden-corpus/` | Golden reference fixtures for reproducibility testing |
|
||||
| `tests/fixtures/` | Test fixtures for unit and integration tests |
|
||||
| `seed-data/` | Initial seed data for development environments |
|
||||
| `tests/fixtures/invalid/` | **Excluded** - Contains intentionally invalid fixtures for negative testing |
|
||||
|
||||
## Local Validation
|
||||
|
||||
### Using the Validation Scripts
|
||||
|
||||
```bash
|
||||
# Validate a single CycloneDX file
|
||||
./scripts/validate-sbom.sh path/to/sbom.json
|
||||
|
||||
# Validate all CycloneDX files in a directory
|
||||
./scripts/validate-sbom.sh --all path/to/directory
|
||||
|
||||
# Validate SPDX file
|
||||
./scripts/validate-spdx.sh path/to/sbom.spdx.json
|
||||
|
||||
# Validate OpenVEX file
|
||||
./scripts/validate-vex.sh path/to/vex.openvex.json
|
||||
```
|
||||
|
||||
### Using sbom-utility Directly
|
||||
|
||||
```bash
|
||||
# Install sbom-utility
|
||||
curl -sSfL "https://github.com/CycloneDX/sbom-utility/releases/download/v0.16.0/sbom-utility-v0.16.0-linux-amd64.tar.gz" | tar xz
|
||||
sudo mv sbom-utility /usr/local/bin/
|
||||
|
||||
# Validate
|
||||
sbom-utility validate --input-file sbom.json --schema docs/schemas/cyclonedx-bom-1.6.schema.json
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Validation Errors
|
||||
|
||||
#### 1. Invalid specVersion
|
||||
|
||||
**Error:** `enum: must be equal to one of the allowed values`
|
||||
|
||||
**Cause:** The `specVersion` field contains an invalid or unsupported version.
|
||||
|
||||
**Solution:**
|
||||
```json
|
||||
// Invalid
|
||||
"specVersion": "2.0"
|
||||
|
||||
// Valid
|
||||
"specVersion": "1.6"
|
||||
```
|
||||
|
||||
#### 2. Missing Required Fields
|
||||
|
||||
**Error:** `required: must have required property 'name'`
|
||||
|
||||
**Cause:** A component is missing required fields.
|
||||
|
||||
**Solution:** Ensure all components have required fields:
|
||||
```json
|
||||
{
|
||||
"type": "library",
|
||||
"name": "example-package",
|
||||
"version": "1.0.0"
|
||||
}
|
||||
```
|
||||
|
||||
#### 3. Invalid Component Type
|
||||
|
||||
**Error:** `enum: type must be equal to one of the allowed values`
|
||||
|
||||
**Cause:** The component type is not a valid CycloneDX type.
|
||||
|
||||
**Solution:** Use valid types: `application`, `framework`, `library`, `container`, `operating-system`, `device`, `firmware`, `file`, `data`
|
||||
|
||||
#### 4. Invalid PURL Format
|
||||
|
||||
**Error:** `format: must match format "purl"`
|
||||
|
||||
**Cause:** The package URL (purl) is malformed.
|
||||
|
||||
**Solution:** Use correct purl format:
|
||||
```json
|
||||
// Invalid
|
||||
"purl": "npm:example@1.0.0"
|
||||
|
||||
// Valid
|
||||
"purl": "pkg:npm/example@1.0.0"
|
||||
```
|
||||
|
||||
### CI Failure Recovery
|
||||
|
||||
1. **Identify the failing fixture:** Check CI logs for the specific file
|
||||
2. **Download the fixture:** `cat path/to/failing-fixture.json`
|
||||
3. **Run local validation:** `./scripts/validate-sbom.sh path/to/failing-fixture.json`
|
||||
4. **Fix the schema issues:** Use the error messages to guide corrections
|
||||
5. **Verify the fix:** Re-run local validation
|
||||
6. **Push and verify CI passes**
|
||||
|
||||
### Negative Test Failures
|
||||
|
||||
If negative tests fail with "UNEXPECTED PASS":
|
||||
|
||||
1. The invalid fixture in `tests/fixtures/invalid/` somehow passed validation
|
||||
2. Review the fixture to ensure it contains actual schema violations
|
||||
3. Update the fixture to include more obvious violations
|
||||
4. Document the expected error in `tests/fixtures/invalid/README.md`
|
||||
|
||||
## Adding New Fixtures
|
||||
|
||||
### Valid Fixtures
|
||||
|
||||
1. Create fixture in appropriate directory (`bench/golden-corpus/`, `tests/fixtures/`)
|
||||
2. Ensure it contains the format marker:
|
||||
- CycloneDX: `"bomFormat": "CycloneDX"`
|
||||
- SPDX: `"spdxVersion"` or `"@context"` with SPDX
|
||||
- OpenVEX: `"@context"` with openvex
|
||||
3. Run local validation before committing
|
||||
4. CI will automatically validate on PR
|
||||
|
||||
### Invalid Fixtures (Negative Testing)
|
||||
|
||||
1. Create fixture in `tests/fixtures/invalid/`
|
||||
2. Add `$comment` field explaining the defect
|
||||
3. Update `tests/fixtures/invalid/README.md` with expected error
|
||||
4. Ensure the fixture has the correct format marker
|
||||
5. CI will verify it fails validation
|
||||
|
||||
## Schema Updates
|
||||
|
||||
When updating schema versions:
|
||||
|
||||
1. Download new schema to `docs/schemas/`
|
||||
2. Update `SBOM_UTILITY_VERSION` in workflows if needed
|
||||
3. Run full validation to check for new violations
|
||||
4. Update documentation with new version
|
||||
5. Update `docs/reproducibility.md` with schema version changes
|
||||
|
||||
## References
|
||||
|
||||
- [CycloneDX Specification](https://cyclonedx.org/specification/overview/)
|
||||
- [CycloneDX sbom-utility](https://github.com/CycloneDX/sbom-utility)
|
||||
- [SPDX Specification](https://spdx.github.io/spdx-spec/v3.0.1/)
|
||||
- [SPDX Python Tools](https://github.com/spdx/tools-python)
|
||||
- [OpenVEX Specification](https://github.com/openvex/spec)
|
||||
Reference in New Issue
Block a user