stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 2 Releases Wiki Activity

docs consolidation

Browse Source
This commit is contained in:
StellaOps Bot
2025-12-25 12:16:13 +02:00
parent deb82b4f03
commit 223843f1d1
34 changed files with 2141 additions and 106 deletions
Download Patch File Download Diff File Expand all files Collapse all files

354
docs/testing/e2e-reproducibility.md Normal file
View File

@@ -0,0 +1,354 @@
# End-to-End Reproducibility Testing Guide
> **Sprint:** SPRINT_8200_0001_0004_e2e_reproducibility_test
> **Tasks:** E2E-8200-025, E2E-8200-026
> **Last Updated:** 2025-06-15
## Overview
StellaOps implements comprehensive end-to-end (E2E) reproducibility testing to ensure that identical inputs always produce identical outputs across:
- Sequential pipeline runs
- Parallel pipeline runs
- Different execution environments (Ubuntu, Windows, macOS)
- Different points in time (using frozen timestamps)
This document describes the E2E test structure, how to run tests, and how to troubleshoot reproducibility failures.
## Test Architecture
### Pipeline Stages
The E2E reproducibility tests cover the full security scanning pipeline:
```
┌─────────────────────────────────────────────────────────────────────────────┐
│ Full E2E Pipeline │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌──────────┐ ┌───────────┐ ┌──────┐ ┌────────┐ ┌──────────┐ │
│ │ Ingest │───▶│ Normalize │───▶│ Diff │───▶│ Decide │───▶│ Attest │ │
│ │ Advisory │ │ Merge & │ │ SBOM │ │ Policy │ │ DSSE │ │
│ │ Feeds │ │ Dedup │ │ vs │ │ Verdict│ │ Envelope │ │
│ └──────────┘ └───────────┘ │Adviso│ └────────┘ └──────────┘ │
│ │ries │ │ │
│ └──────┘ ▼ │
│ ┌──────────┐ │
│ │ Bundle │ │
│ │ Package │ │
│ └──────────┘ │
└─────────────────────────────────────────────────────────────────────────────┘
```
### Key Components
| Component | File | Purpose |
|-----------|------|---------|
| Test Project | `StellaOps.Integration.E2E.csproj` | MSBuild project for E2E tests |
| Test Fixture | `E2EReproducibilityTestFixture.cs` | Pipeline composition and execution |
| Tests | `E2EReproducibilityTests.cs` | Reproducibility verification tests |
| Comparer | `ManifestComparer.cs` | Byte-for-byte manifest comparison |
| CI Workflow | `.gitea/workflows/e2e-reproducibility.yml` | Cross-platform CI pipeline |
## Running E2E Tests
### Prerequisites
- .NET 10.0 SDK
- Docker (for PostgreSQL container)
- At least 4GB RAM available
### Local Execution
```bash
# Run all E2E reproducibility tests
dotnet test tests/integration/StellaOps.Integration.E2E/ \
--logger "console;verbosity=detailed"
# Run specific test category
dotnet test tests/integration/StellaOps.Integration.E2E/ \
--filter "Category=Integration" \
--logger "console;verbosity=detailed"
# Run with code coverage
dotnet test tests/integration/StellaOps.Integration.E2E/ \
--collect:"XPlat Code Coverage" \
--results-directory ./TestResults
```
### CI Execution
E2E tests run automatically on:
- Pull requests affecting `src/**` or `tests/integration/**`
- Pushes to `main` and `develop` branches
- Nightly at 2:00 AM UTC (full cross-platform suite)
- Manual trigger with optional cross-platform flag
## Test Categories
### 1. Sequential Reproducibility (Tasks 11-14)
Tests that the pipeline produces identical results when run multiple times:
```csharp
[Fact]
public async Task FullPipeline_ProducesIdenticalVerdictHash_AcrossRuns()
{
// Arrange
var inputs = await _fixture.SnapshotInputsAsync();
// Act - Run twice
var result1 = await _fixture.RunFullPipelineAsync(inputs);
var result2 = await _fixture.RunFullPipelineAsync(inputs);
// Assert
result1.VerdictId.Should().Be(result2.VerdictId);
result1.BundleManifestHash.Should().Be(result2.BundleManifestHash);
}
```
### 2. Parallel Reproducibility (Task 14)
Tests that concurrent execution produces identical results:
```csharp
[Fact]
public async Task FullPipeline_ParallelExecution_10Concurrent_AllIdentical()
{
var inputs = await _fixture.SnapshotInputsAsync();
const int concurrentRuns = 10;
var tasks = Enumerable.Range(0, concurrentRuns)
.Select(_ => _fixture.RunFullPipelineAsync(inputs));
var results = await Task.WhenAll(tasks);
var comparison = ManifestComparer.CompareMultiple(results.ToList());
comparison.AllMatch.Should().BeTrue();
}
```
### 3. Cross-Platform Reproducibility (Tasks 15-18)
Tests that identical inputs produce identical outputs on different operating systems:
| Platform | Runner | Status |
|----------|--------|--------|
| Ubuntu | `ubuntu-latest` | Primary (runs on every PR) |
| Windows | `windows-latest` | Nightly / On-demand |
| macOS | `macos-latest` | Nightly / On-demand |
### 4. Golden Baseline Verification (Tasks 19-21)
Tests that current results match a pre-approved baseline:
```json
// bench/determinism/golden-baseline/e2e-hashes.json
{
"verdict_hash": "sha256:abc123...",
"manifest_hash": "sha256:def456...",
"envelope_hash": "sha256:ghi789...",
"updated_at": "2025-06-15T12:00:00Z",
"updated_by": "ci",
"commit": "abc123def456"
}
```
## Troubleshooting Reproducibility Failures
### Common Causes
#### 1. Non-Deterministic Ordering
**Symptom:** Different verdict hashes despite identical inputs.
**Diagnosis:**
```csharp
// Check if collections are being ordered
var comparison = ManifestComparer.Compare(result1, result2);
var report = ManifestComparer.GenerateDiffReport(comparison);
Console.WriteLine(report);
```
**Solution:** Ensure all collections are sorted before hashing:
```csharp
// Bad - non-deterministic
var findings = results.ToList();
// Good - deterministic
var findings = results.OrderBy(f => f.CveId, StringComparer.Ordinal)
.ThenBy(f => f.Purl, StringComparer.Ordinal)
.ToList();
```
#### 2. Timestamp Drift
**Symptom:** Bundle manifests differ in `createdAt` field.
**Diagnosis:**
```csharp
var jsonComparison = ManifestComparer.CompareJson(
result1.BundleManifest,
result2.BundleManifest);
```
**Solution:** Use frozen timestamps in tests:
```csharp
// In test fixture
public DateTimeOffset FrozenTimestamp { get; } =
new DateTimeOffset(2025, 6, 15, 12, 0, 0, TimeSpan.Zero);
```
#### 3. Platform-Specific Behavior
**Symptom:** Tests pass on Ubuntu but fail on Windows/macOS.
**Common causes:**
- Line ending differences (`\n` vs `\r\n`)
- Path separator differences (`/` vs `\`)
- Unicode normalization differences
- Floating-point representation differences
**Diagnosis:**
```bash
# Download artifacts from all platforms
# Compare hex dumps
xxd ubuntu-manifest.bin > ubuntu.hex
xxd windows-manifest.bin > windows.hex
diff ubuntu.hex windows.hex
```
**Solution:** Use platform-agnostic serialization:
```csharp
// Use canonical JSON
var json = CanonJson.Serialize(data);
// Normalize line endings
var normalized = content.Replace("\r\n", "\n");
```
#### 4. Key/Signature Differences
**Symptom:** Envelope hashes differ despite identical payloads.
**Diagnosis:**
```csharp
// Compare envelope structure
var envelope1 = JsonSerializer.Deserialize<DsseEnvelope>(result1.EnvelopeBytes);
var envelope2 = JsonSerializer.Deserialize<DsseEnvelope>(result2.EnvelopeBytes);
// Check if payloads match
envelope1.Payload.SequenceEqual(envelope2.Payload).Should().BeTrue();
```
**Solution:** Use deterministic key generation:
```csharp
// Generate key from fixed seed for reproducibility
private static ECDsa GenerateDeterministicKey(int seed)
{
var rng = new DeterministicRng(seed);
var keyBytes = new byte[32];
rng.GetBytes(keyBytes);
// ... create key from bytes
}
```
### Debugging Tools
#### ManifestComparer
```csharp
// Full comparison
var comparison = ManifestComparer.Compare(expected, actual);
// Multiple results
var multiComparison = ManifestComparer.CompareMultiple(results);
// Detailed report
var report = ManifestComparer.GenerateDiffReport(comparison);
// Hex dump for byte-level debugging
var hexDump = ManifestComparer.GenerateHexDump(expected.BundleManifest, actual.BundleManifest);
```
#### JSON Comparison
```csharp
var jsonComparison = ManifestComparer.CompareJson(
expected.BundleManifest,
actual.BundleManifest);
foreach (var diff in jsonComparison.Differences)
{
Console.WriteLine($"Path: {diff.Path}");
Console.WriteLine($"Expected: {diff.Expected}");
Console.WriteLine($"Actual: {diff.Actual}");
}
```
## Updating the Golden Baseline
When intentional changes affect reproducibility (e.g., new fields, algorithm changes):
### 1. Manual Update
```bash
# Run tests and capture new hashes
dotnet test tests/integration/StellaOps.Integration.E2E/ \
--results-directory ./TestResults
# Update baseline
cp ./TestResults/verdict_hash.txt ./bench/determinism/golden-baseline/
# ... update e2e-hashes.json
```
### 2. CI Update (Recommended)
```bash
# Trigger workflow with update flag
# Via Gitea UI: Actions → E2E Reproducibility → Run workflow
# Set update_baseline = true
```
### 3. Approval Process
1. Create PR with baseline update
2. Explain why the change is intentional
3. Verify all platforms produce consistent results
4. Get approval from Platform Guild lead
5. Merge after CI passes
## CI Workflow Reference
### Jobs
| Job | Runs On | Trigger | Purpose |
|-----|---------|---------|---------|
| `reproducibility-ubuntu` | Every PR | PR/Push | Primary reproducibility check |
| `reproducibility-windows` | Nightly | Schedule/Manual | Cross-platform Windows |
| `reproducibility-macos` | Nightly | Schedule/Manual | Cross-platform macOS |
| `cross-platform-compare` | After platform jobs | Schedule/Manual | Compare hashes |
| `golden-baseline` | After Ubuntu | Always | Baseline verification |
| `reproducibility-gate` | After all | Always | Final status check |
### Artifacts
| Artifact | Retention | Contents |
|----------|-----------|----------|
| `e2e-results-{platform}` | 14 days | Test results (.trx), logs |
| `hashes-{platform}` | 14 days | Hash files for comparison |
| `cross-platform-report` | 30 days | Markdown comparison report |
## Related Documentation
- [Reproducibility Architecture](../reproducibility.md)
- [VerdictId Content-Addressing](../modules/policy/architecture.md#verdictid)
- [DSSE Envelope Format](../modules/attestor/architecture.md#dsse)
- [Determinism Testing](./determinism-verification.md)
## Sprint History
- **8200.0001.0004** - Initial E2E reproducibility test implementation
- **8200.0001.0001** - VerdictId content-addressing (dependency)
- **8200.0001.0002** - DSSE round-trip testing (dependency)

202
docs/testing/schema-validation.md Normal file
View File

@@ -0,0 +1,202 @@
# SBOM Schema Validation
This document describes the schema validation system for SBOM (Software Bill of Materials) fixtures in StellaOps.
## Overview
StellaOps validates all SBOM fixtures against official JSON schemas to detect schema drift before runtime. This ensures:
- CycloneDX 1.6 fixtures are compliant with the official schema
- SPDX 3.0.1 fixtures meet specification requirements
- OpenVEX fixtures follow the 0.2.0 specification
- Invalid fixtures are detected early in the CI pipeline
## Supported Formats
| Format | Version | Schema Location | Validator |
|--------|---------|-----------------|-----------|
| CycloneDX | 1.6 | `docs/schemas/cyclonedx-bom-1.6.schema.json` | sbom-utility |
| SPDX | 3.0.1 | `docs/schemas/spdx-jsonld-3.0.1.schema.json` | pyspdxtools / check-jsonschema |
| OpenVEX | 0.2.0 | `docs/schemas/openvex-0.2.0.schema.json` | ajv-cli |
## CI Workflows
### Schema Validation Workflow
**File:** `.gitea/workflows/schema-validation.yml`
Runs on:
- Pull requests touching `bench/golden-corpus/**`, `src/Scanner/**`, `docs/schemas/**`, or `scripts/validate-*.sh`
- Push to `main` branch
Jobs:
1. **validate-cyclonedx** - Validates all CycloneDX 1.6 fixtures
2. **validate-spdx** - Validates all SPDX 3.0.1 fixtures
3. **validate-vex** - Validates all OpenVEX 0.2.0 fixtures
4. **validate-negative** - Verifies invalid fixtures are correctly rejected
5. **summary** - Aggregates results
### Determinism Gate Integration
**File:** `.gitea/workflows/determinism-gate.yml`
The determinism gate includes schema validation as a prerequisite step. If schema validation fails, determinism checks are blocked.
To skip schema validation (e.g., during debugging):
```bash
# Via workflow_dispatch
skip_schema_validation: true
```
## Fixture Directories
Validation scans these directories for SBOM fixtures:
| Directory | Purpose |
|-----------|---------|
| `bench/golden-corpus/` | Golden reference fixtures for reproducibility testing |
| `tests/fixtures/` | Test fixtures for unit and integration tests |
| `seed-data/` | Initial seed data for development environments |
| `tests/fixtures/invalid/` | **Excluded** - Contains intentionally invalid fixtures for negative testing |
## Local Validation
### Using the Validation Scripts
```bash
# Validate a single CycloneDX file
./scripts/validate-sbom.sh path/to/sbom.json
# Validate all CycloneDX files in a directory
./scripts/validate-sbom.sh --all path/to/directory
# Validate SPDX file
./scripts/validate-spdx.sh path/to/sbom.spdx.json
# Validate OpenVEX file
./scripts/validate-vex.sh path/to/vex.openvex.json
```
### Using sbom-utility Directly
```bash
# Install sbom-utility
curl -sSfL "https://github.com/CycloneDX/sbom-utility/releases/download/v0.16.0/sbom-utility-v0.16.0-linux-amd64.tar.gz" | tar xz
sudo mv sbom-utility /usr/local/bin/
# Validate
sbom-utility validate --input-file sbom.json --schema docs/schemas/cyclonedx-bom-1.6.schema.json
```
## Troubleshooting
### Common Validation Errors
#### 1. Invalid specVersion
**Error:** `enum: must be equal to one of the allowed values`
**Cause:** The `specVersion` field contains an invalid or unsupported version.
**Solution:**
```json
// Invalid
"specVersion": "2.0"
// Valid
"specVersion": "1.6"
```
#### 2. Missing Required Fields
**Error:** `required: must have required property 'name'`
**Cause:** A component is missing required fields.
**Solution:** Ensure all components have required fields:
```json
{
"type": "library",
"name": "example-package",
"version": "1.0.0"
}
```
#### 3. Invalid Component Type
**Error:** `enum: type must be equal to one of the allowed values`
**Cause:** The component type is not a valid CycloneDX type.
**Solution:** Use valid types: `application`, `framework`, `library`, `container`, `operating-system`, `device`, `firmware`, `file`, `data`
#### 4. Invalid PURL Format
**Error:** `format: must match format "purl"`
**Cause:** The package URL (purl) is malformed.
**Solution:** Use correct purl format:
```json
// Invalid
"purl": "npm:example@1.0.0"
// Valid
"purl": "pkg:npm/example@1.0.0"
```
### CI Failure Recovery
1. **Identify the failing fixture:** Check CI logs for the specific file
2. **Download the fixture:** `cat path/to/failing-fixture.json`
3. **Run local validation:** `./scripts/validate-sbom.sh path/to/failing-fixture.json`
4. **Fix the schema issues:** Use the error messages to guide corrections
5. **Verify the fix:** Re-run local validation
6. **Push and verify CI passes**
### Negative Test Failures
If negative tests fail with "UNEXPECTED PASS":
1. The invalid fixture in `tests/fixtures/invalid/` somehow passed validation
2. Review the fixture to ensure it contains actual schema violations
3. Update the fixture to include more obvious violations
4. Document the expected error in `tests/fixtures/invalid/README.md`
## Adding New Fixtures
### Valid Fixtures
1. Create fixture in appropriate directory (`bench/golden-corpus/`, `tests/fixtures/`)
2. Ensure it contains the format marker:
- CycloneDX: `"bomFormat": "CycloneDX"`
- SPDX: `"spdxVersion"` or `"@context"` with SPDX
- OpenVEX: `"@context"` with openvex
3. Run local validation before committing
4. CI will automatically validate on PR
### Invalid Fixtures (Negative Testing)
1. Create fixture in `tests/fixtures/invalid/`
2. Add `$comment` field explaining the defect
3. Update `tests/fixtures/invalid/README.md` with expected error
4. Ensure the fixture has the correct format marker
5. CI will verify it fails validation
## Schema Updates
When updating schema versions:
1. Download new schema to `docs/schemas/`
2. Update `SBOM_UTILITY_VERSION` in workflows if needed
3. Run full validation to check for new violations
4. Update documentation with new version
5. Update `docs/reproducibility.md` with schema version changes
## References
- [CycloneDX Specification](https://cyclonedx.org/specification/overview/)
- [CycloneDX sbom-utility](https://github.com/CycloneDX/sbom-utility)
- [SPDX Specification](https://spdx.github.io/spdx-spec/v3.0.1/)
- [SPDX Python Tools](https://github.com/spdx/tools-python)
- [OpenVEX Specification](https://github.com/openvex/spec)