24 KiB
Here’s a tight, drop-in acceptance-tests pack for Stella Ops that turns common failure modes into concrete guardrails you can ship this sprint.
1) Feed outages & integrity drift (e.g., Grype DB / CDN hiccups)
Lesson: Never couple scans to a single live feed; pin, verify, and cache.
Add to acceptance tests
-
Rollback-safe updaters
- If a feed update fails checksum or signature, the system keeps using the last “good” bundle.
- On restart, the updater falls back to the last verified bundle without network access.
-
Signed offline bundles
- Every feed bundle (SBOM catalogs, CVE DB shards, rules) must be DSSE-signed; verification blocks ingestion on mismatch.
- Bundle manifest lists SHA-256 for each file; any deviation = reject.
Test cases (CI)
- Simulate 404/timeout from feed URL → scanner still produces results from cached bundle.
- Serve a tampered bundle (wrong hash) → updater logs failure; no swap; previous bundle remains active.
- Air-gap mode: no network → scanner loads from
/var/lib/stellaops/offline-bundles/*and passes verification.
2) SBOM quality & schema drift
Lesson: Garbage in = garbage VEX. Gate on schema, completeness, and provenance.
Add to acceptance tests
-
SBOM schema gating
- Reject SBOMs not valid CycloneDX 1.6 / SPDX 2.3 (your chosen set).
- Require: component
bom-ref, supplier, version, hashes, and build provenance (SLSA/in-toto attestation) if provided.
-
Minimum completeness
- Thresholds: ≥95% components with cryptographic hashes; no unknown package ecosystem fields for top 20 deps.
Test cases
- Feed malformed CycloneDX →
400 SBOM_VALIDATION_FAILEDwith pointer to failing JSON path. - SBOM missing hashes for >5% of components → blocked from graph ingestion; actionable error.
- SBOM with unsigned provenance when policy="RequireAttestation" → rejected.
3) DB/data corruption or operator error
Lesson: Snapshots save releases.
Add to acceptance tests
-
DB snapshot cadence
- Postgres: base backup nightly + WAL archiving; RPO ≤ 15 min; automated restore rehearsals.
- Mongo (while still in use): per-collection dumps until conversion completes; checksum each artifact.
-
Deterministic replay
- Any graph view must be reproducible from snapshot + bundle manifest (same revision hash).
Test cases
- Run chaos test that deletes last 24h tables → PITR restore to T-15m succeeds; graph revision IDs match pre-failure.
- Restore rehearsal produces identical VEX verdict counts for a pinned revision.
4) Reachability engines & graph evaluation flakiness
Lesson: When reachability is uncertain, degrade gracefully and be explicit.
Add to acceptance tests
-
Reachability fallbacks
- If call-graph build fails or language analyzer missing, verdict moves to “Potentially Affected (Unproven Reach)” with a reason code.
- Policies must allow “conservative mode” (assume reachable) vs “lenient mode” (assume not-reachable) toggled per environment.
-
Stable graph IDs
- Graph revision ID is a content hash of inputs (SBOM set + rules + feed versions); identical inputs → identical ID.
Test cases
- Remove a language analyzer container at runtime → status flips to fallback code; no 500s; policy evaluation still completes.
- Re-ingest same inputs → same graph revision ID and same verdict distribution.
5) Update pipelines & job routing
Lesson: No single point of truth; isolate, audit, and prove swaps.
Add to acceptance tests
-
Two-phase bundle swaps
- Stage → verify → atomic symlink/label swap; all scanners pick up new label within 1 minute, or roll back.
-
Authority-gated policy changes
- Any policy change (severity threshold, allowlist) is a signed request via Authority; audit trail must include signer and DSSE envelope hash.
Test cases
- Introduce a new CVE ruleset; verification passes → atomic swap; running scans continue; new scans use N+1 bundle.
- Attempt policy change with invalid signature → rejected; audit log entry created; unchanged policy in effect.
How to wire this in Stella Ops (quick pointers)
-
Offline bundle format
bundle.json(manifest: file list + SHA-256 + DSSE signature),/sboms/*.json,/feeds/cve/*.sqlite(or shards),/rules/*.yaml,/provenance/*.intoto.jsonl.- Verification entrypoint in .NET 10:
StellaOps.Bundle.VerifyAsync(manifest, keyring)before any ingestion.
-
Authority integration
- Define
PolicyChangeRequest(subject, diff, reason, expiry, DSSE envelope). - Gate
PUT /policies/*behindAuthority.Verify(envelope) == trueandenvelope.subject == computed_diff_hash.
- Define
-
Graph determinism
GraphRevisionId = SHA256(Sort(JSON([SBOMRefs, RulesetVersion, FeedBundleIds, LatticeConfig, NormalizationVersion]))).
-
Postgres snapshots (until full conversion)
- Use
pg_basebackupnightly +wal-gfor WAL; GitLab job runs restore rehearsal weekly intostellaops-restorenamespace and asserts revision parity against prod.
- Use
Minimal developer checklist (copy to your sprint board)
- Add
BundleVerifierto scanner startup; block if verification fails. - Implement
CacheLastGoodBundle()and atomic label swap (/current -> /v-YYYYMMDDHHmm). - Add
SbomGatewith JSON-Schema validation + completeness thresholds. - Emit reasoned fallbacks:
REACH_FALLBACK_NO_ANALYZER,REACH_FALLBACK_TIMEOUT. - Compute and display
GraphRevisionIdeverywhere (API + UI + logs). - Configure nightly PG backups + weekly restore rehearsal that asserts revision parity.
- Route all policy mutations through Authority DSSE verification + auditable ledger entry.
If you want, I can turn this into ready-to-merge .NET test fixtures (xUnit) and a GitLab CI job that runs the feed-tamper/air-gap simulations automatically. I’ll take the 5 “miss” areas and turn them into concrete, implementable test plans, with suggested projects, fixtures, and key cases your team can start coding.
I’ll keep names aligned to .NET 10/xUnit and your Stella Ops modules.
0. Test layout proposal
Solution structure (tests)
/tests
/StellaOps.Bundle.Tests
BundleVerificationTests.cs
CachedBundleFallbackTests.cs
/StellaOps.SbomGate.Tests
SbomSchemaValidationTests.cs
SbomCompletenessTests.cs
/StellaOps.Scanner.Tests
ScannerOfflineBundleTests.cs
ReachabilityFallbackTests.cs
GraphRevisionDeterminismTests.cs
/StellaOps.DataRecoverability.Tests
PostgresSnapshotRestoreTests.cs
GraphReplayParityTests.cs
/StellaOps.Authority.Tests
PolicyChangeSignatureTests.cs
/StellaOps.System.Acceptance
FeedOutageEndToEndTests.cs
AirGapModeEndToEndTests.cs
BundleSwapEndToEndTests.cs
/testdata
/bundles
/sboms
/graphs
/db
Use xUnit + FluentAssertions, plus Testcontainers for Postgres.
1) Feed outages & integrity drift
Objectives
- Scanner never “goes dark” because the CDN/feed is down.
- Only verified bundles are used; tampered bundles are never ingested.
- Offline/air-gap mode is a first-class, tested behavior.
Components under test
StellaOps.BundleVerifier(core library)StellaOps.Scanner.Webservice(scanner, bundle loader)- Bundle filesystem layout:
/opt/stellaops/bundles/v-<timestamp>/*+/opt/stellaops/bundles/currentsymlink
Test dimensions
- Network: OK / timeout / 404 / TLS failure / DNS failure.
- Remote bundle: correct / tampered (hash mismatch) / wrong signature / truncated.
- Local cache: last-good present / absent / corrupted.
- Mode: online / offline (air-gap).
Detailed test suites
1.1 Bundle verification unit tests
Project: StellaOps.Bundle.Tests
Fixtures:
testdata/bundles/good-bundle/testdata/bundles/hash-mismatch-bundle/testdata/bundles/bad-signature-bundle/testdata/bundles/missing-file-bundle/
Key tests:
-
VerifyAsync_ValidBundle_ReturnsSuccess-
Arrange: Load
good-bundlemanifest + DSSE signature. -
Act:
BundleVerifier.VerifyAsync(manifest, keyring) -
Assert:
result.IsValid == trueresult.Files.All(f => f.Status == Verified)
-
-
VerifyAsync_HashMismatch_FailsFast-
Use
hash-mismatch-bundlewhere one file’s SHA256 differs. -
Assert:
IsValid == falseErrorscontainsBUNDLE_FILE_HASH_MISMATCHand the offending path.
-
-
VerifyAsync_InvalidSignature_RejectsBundle-
DSSE envelope signed with unknown key.
-
Assert:
IsValid == falseErrorscontainsBUNDLE_SIGNATURE_INVALID.
-
-
VerifyAsync_MissingFile_RejectsBundle-
Manifest lists file that does not exist on disk.
-
Assert:
IsValid == falseErrorscontainsBUNDLE_FILE_MISSING.
-
1.2 Cached bundle fallback logic
Class under test: BundleManager
Simplified interface:
public interface IBundleManager {
Task<BundleRef> GetActiveBundleAsync();
Task<BundleRef> UpdateFromRemoteAsync(CancellationToken ct);
}
Key tests:
-
UpdateFromRemoteAsync_RemoteUnavailable_KeepsLastGoodBundle-
Arrange:
lastGoodbundle exists and is marked verified.- Remote HTTP client always throws
TaskCanceledException(simulated timeout).
-
Act:
UpdateFromRemoteAsync. -
Assert:
- Returned bundle ID equals
lastGood.Id. - No changes to
currentsymlink.
- Returned bundle ID equals
-
-
UpdateFromRemoteAsync_RemoteTampered_DoesNotReplaceCurrent-
Remote returns bundle
temp-bundlewhich failsBundleVerifier. -
Assert:
currentstill points tolastGood.- An error metric is emitted (e.g.
stellaops_bundle_update_failures_total++).
-
-
GetActiveBundle_NoVerifiedBundle_ThrowsDomainError- No bundle is verified on disk.
GetActiveBundleAsyncthrows a domain exception with codeNO_VERIFIED_BUNDLE_AVAILABLE.- Consumption pattern in Scanner: scanner fails fast on startup with clear log.
1.3 Scanner behavior with outages (integration)
Project: StellaOps.Scanner.Tests
Use in-memory host (WebApplicationFactory<ScannerProgram>).
Scenarios:
- F1: CDN timeout, last-good present.
- F2: CDN 404, last-good present.
- F3: CDN returns tampered bundle; verification fails.
- F4: Air-gap: network disabled, last-good present.
- F5: Air-gap + no last-good: scanner must refuse to start.
Example test:
[Fact]
public async Task Scanner_UsesLastGoodBundle_WhenCdnTimesOut() {
// Arrange: put good bundle under /bundles/v-1, symlink /bundles/current -> v-1
using var host = TestScannerHost.WithBundle("v-1", good: true, simulateCdnTimeout: true);
// Act: call /api/scan with small fixture image
var response = await host.Client.PostAsJsonAsync("/api/scan", scanRequest);
// Assert:
response.StatusCode.Should().Be(HttpStatusCode.OK);
var content = await response.Content.ReadFromJsonAsync<ScanResult>();
content.BundleId.Should().Be("v-1");
host.Logs.Should().Contain("Falling back to last verified bundle");
}
1.4 System acceptance (GitLab CI)
Job idea: acceptance:feed-resilience
Steps:
-
Spin up
scanner+ stubfeedsercontainer. -
Phase A: feed OK → run baseline scan; capture
bundleIdandgraphRevisionId. -
Phase B: re-run with feed stub configured to:
- timeout,
- 404,
- return tampered bundle.
-
For each phase:
- Assert
bundleIdremains the baseline one. - Assert
graphRevisionIdunchanged.
- Assert
Failure of any assertion should break the pipeline.
2) SBOM quality & schema drift
Objectives
- Only syntactically valid SBOMs are ingested into the graph.
- Enforce minimum completeness (hash coverage, supplier etc.).
- Clear, machine-readable error responses from SBOM ingestion API.
Components
StellaOps.SbomGate(validation service)- SBOM ingestion endpoint in Scanner/Concelier:
POST /api/sboms
Schema validation tests
Project: StellaOps.SbomGate.Tests
Fixtures:
sbom-cdx-1.6-valid.jsonsbom-cdx-1.6-malformed.jsonsbom-spdx-2.3-valid.jsonsbom-unsupported-schema.jsonsbom-missing-hashes-10percent.jsonsbom-no-supplier.json
Key tests:
-
Validate_ValidCycloneDx16_Succeeds- Assert type
SbomValidationResult.Success. - Ensure
DetectedSchema == CycloneDx16.
- Assert type
-
Validate_MalformedJson_FailsWithSyntaxError-
Malformed JSON.
-
Assert:
IsValid == falseErrorscontainsSBOM_JSON_SYNTAX_ERRORwith path info.
-
-
Validate_UnsupportedSchemaVersion_Fails- SPDX 2.1 (if you only allow 2.3).
- Expect
SBOM_SCHEMA_UNSUPPORTEDwithschemaUriecho.
-
Validate_MissingHashesOverThreshold_Fails-
SBOM where >5% components lack hashes.
-
Policy:
MinHashCoverage = 0.95. -
Assert:
IsValid == falseErrorscontainsSBOM_HASH_COVERAGE_BELOW_THRESHOLDwith actual ratio.
-
-
Validate_MissingSupplier_Fails- Critical components missing supplier info.
- Expect
SBOM_REQUIRED_FIELD_MISSINGwithcomponent.supplier.
API-level tests
Project: StellaOps.Scanner.Tests (or StellaOps.Concelier.Tests depending where SBOM ingestion lives).
Key scenarios:
-
POST /api/sbomswith malformed JSON-
Request body:
sbom-cdx-1.6-malformed.json. -
Expected:
- HTTP 400.
- Body:
{ "code": "SBOM_VALIDATION_FAILED", "details": [ ... ], "correlationId": "..." }. - At least one detail contains
SBOM_JSON_SYNTAX_ERROR.
-
-
POST /api/sbomswith missing hashes- Body:
sbom-missing-hashes-10percent.json. - HTTP 400 with
SBOM_HASH_COVERAGE_BELOW_THRESHOLD.
- Body:
-
POST /api/sbomswith unsupported schema- Body:
sbom-unsupported-schema.json. - HTTP 400 with
SBOM_SCHEMA_UNSUPPORTED.
- Body:
-
POST /api/sbomsvalid- Body:
sbom-cdx-1.6-valid.json. - HTTP 202 or 201 (depending on design).
- Response contains SBOM ID; subsequent graph build sees that SBOM.
- Body:
3) DB/data corruption & operator error
Objectives
- You can restore Postgres to a point in time and reproduce previous graph results.
- Graphs are deterministic given bundle + SBOM + rules.
- Obvious corruptions are detected and surfaced, not silently masked.
Components
- Postgres cluster (new canonical store)
StellaOps.Scanner.Webservice(graph builder, persistence)GraphRevisionIdcomputation
3.1 Postgres snapshot / WAL tests
Project: StellaOps.DataRecoverability.Tests
Use Testcontainers to spin up Postgres.
Scenarios:
-
PITR_Restore_ReplaysGraphsWithSameRevisionIds-
Arrange:
- Spin DB container with WAL archiving enabled.
- Apply schema migrations.
- Ingest fixed set of SBOMs + bundle refs + rules.
- Trigger graph build → record
graphRevisionIdsfrom API. - Take base backup snapshot (simulate daily snapshot).
-
Act:
- Destroy container.
- Start new container from base backup + replay WAL up to a specific LSN.
- Start Scanner against restored DB.
- Query graphs again.
-
Assert:
- For each known graph:
revisionId_restored == revisionId_original. - Number of nodes/edges is identical.
- For each known graph:
-
-
PartialDataLoss_DetectedByHealthCheck-
After initial load, deliberately delete some rows (e.g. all edges for a given graph).
-
Run health check endpoint, e.g.
/health/graph. -
Expect:
- HTTP 503.
- Body indicates
GRAPH_INTEGRITY_FAILEDwith details of missing edges.
-
This test forces a discipline to implement a basic graph integrity check (e.g. counts by state vs expected).
3.2 Deterministic replay tests
Project: StellaOps.Scanner.Tests → GraphRevisionDeterminismTests.cs
Precondition: Graph revision ID computed as:
GraphRevisionId = SHA256(
Normalize([
BundleId,
OrderedSbomIds,
RulesetVersion,
FeedBundleIds,
LatticeConfigVersion,
NormalizationVersion
])
);
Scenarios:
-
SameInputs_SameRevisionId- Run graph build twice for same inputs.
- Assert identical
GraphRevisionId.
-
DifferentBundle_DifferentRevisionId- Same SBOMs & rules; change vulnerability bundle ID.
- Assert
GraphRevisionIdchanges.
-
DifferentRuleset_DifferentRevisionId- Same SBOM & bundle; change ruleset version.
- Assert
GraphRevisionIdchanges.
-
OrderingIrrelevant_StableRevision- Provide SBOMs in different order.
- Assert
GraphRevisionIdsame (because of internal sorting).
4) Reachability engine & graph evaluation flakiness
Objectives
- If reachability cannot be computed, you do not break; you downgrade verdicts with explicit reason codes.
- Deterministic reachability for “golden fixtures”.
- Graph evaluation remains stable even when analyzers come and go.
Components
StellaOps.Scanner.Webservice(lattice / reachability engine)- Language analyzers (sidecar or gRPC microservices)
- Verdict representation, e.g.:
public sealed record VulnerabilityVerdict(
string Status, // "NotAffected", "Affected", "PotentiallyAffected"
string ReasonCode, // "REACH_CONFIRMED", "REACH_FALLBACK_NO_ANALYZER", ...
string? AnalyzerId
);
4.1 Golden reachability fixtures
Project: StellaOps.Scanner.Tests → GoldenReachabilityTests.cs
Fixtures directory: /testdata/reachability/fixture-*/
Each fixture:
/testdata/reachability/fixture-01-log4j/
sbom.json
code-snippets/...
expected-vex.json
config.json # language, entrypoints, etc.
Test pattern:
For each fixture:
- Load SBOM + configuration.
- Trigger reachability analysis.
- Collect raw reachability graph + final VEX verdicts.
- Compare to
expected-vex.json(status + reason codes). - Store the
GraphRevisionIdand set it as golden as well.
Key cases:
- R1: simple direct call → reachability confirmed →
Status = "Affected", ReasonCode = "REACH_CONFIRMED". - R2: library present but not called →
Status = "NotAffected", ReasonCode = "REACH_ANALYZED_UNREACHABLE". - R3: language analyzer missing →
Status = "PotentiallyAffected", ReasonCode = "REACH_FALLBACK_NO_ANALYZER". - R4: analysis timeout →
Status = "PotentiallyAffected", ReasonCode = "REACH_FALLBACK_TIMEOUT".
4.2 Analyzer unavailability / fallback behavior
Project: StellaOps.Scanner.Tests → ReachabilityFallbackTests.cs
Scenarios:
-
NoAnalyzerRegistered_ForLanguage_UsesFallback-
Scanner config lists a component in language “go” but no analyzer registered.
-
Expect:
- No 500 error from
/api/graphs/.... - All applicable vulnerabilities for that component have
Status = "PotentiallyAffected"andReasonCode = "REACH_FALLBACK_NO_ANALYZER".
- No 500 error from
-
-
AnalyzerRpcFailure_UsesFallback- Analyzer responds with gRPC error or HTTP 500.
- Scanner logs error and keeps going.
- Same semantics as missing analyzer, but with
AnalyzerIdpopulated and optionalReasonDetails(e.g.RPC_UNAVAILABLE).
-
AnalyzerTimeout_UsesTimeoutFallback- Force analyzer calls to time out.
ReasonCode = "REACH_FALLBACK_TIMEOUT".
4.3 Concurrency & determinism
Add a test that:
-
Triggers N parallel graph builds for the same inputs.
-
Asserts that:
- All builds succeed.
- All
GraphRevisionIdare identical. - All reachability reason codes are identical.
This is important for concurrent scanners and ensures lack of race conditions in graph construction.
5) Update pipelines & job routing
Objectives
- Bundle swaps are atomic: scanners see either old or new, never partially written bundles.
- Policy changes are always signed via Authority; unsigned/invalid changes never apply.
- Job routing changes (if/when you move to direct microservice pools) remain stateless and testable.
5.1 Two-phase bundle swap tests
Bundle layout:
-
/opt/stellaops/bundles/current→ symlink tov-YYYYMMDDHHmmss -
New bundle:
- Download to
/opt/stellaops/bundles/staging/<temp-id> - Verify
- Atomic
ln -s v-new current.tmp && mv -T current.tmp current
- Download to
Project: StellaOps.Bundle.Tests → BundleSwapTests.cs
Scenarios:
-
Swap_Success_IsAtomic-
Simulate swap in a temp directory.
-
During swap, spawn parallel tasks that repeatedly read
currentand openmanifest.json. -
Assert:
- Readers never fail with “file not found” / partial manifest.
- Readers only see either
v-oldorv-new, no mixed state.
-
-
Swap_VerificationFails_NoChangeToCurrent-
Stage bundle which fails
BundleVerifier. -
After attempted swap:
currentstill points tov-old.- No new directory with the name expected for
v-newis referenced bycurrent.
-
-
Swap_CrashBetweenVerifyAndMv_LeavesSystemConsistent-
Simulate crash after creating
current.tmpbut beforemv -T. -
On “restart”:
- Cleanup code must detect
current.tmpand remove it. - Ensure
currentstill points to last good.
- Cleanup code must detect
-
5.2 Authority-gated policy changes
Component: StellaOps.Authority + any service that exposes /policies.
Policy change flow:
- Client sends DSSE-signed
PolicyChangeRequestto/authority/verify. - Authority validates signature, subject hash.
- Service applies change only if Authority approves.
Project: StellaOps.Authority.Tests + StellaOps.Scanner.Tests (or wherever policies live).
Key tests:
-
PolicyChange_WithValidSignature_Applies- Signed request’s
subjecthash matches computed diff of old->new policy. - Authority returns
Approved. - Policy service updates policy; audit log entry recorded.
- Signed request’s
-
PolicyChange_InvalidSignature_Rejected-
Signature verifiable with no trusted key, or corrupted payload.
-
Expect:
- HTTP 403 or 400 from policy endpoint.
- No policy change in DB.
- Audit log entry with reason
SIGNATURE_INVALID.
-
-
PolicyChange_SubjectHashMismatch_Rejected- Attacker changes policy body but not DSSE subject.
- On verification, recomputed diff doesn’t match subject hash.
- Authority rejects with
SUBJECT_MISMATCH.
-
PolicyChange_ExpiredEnvelope_Rejected- Envelope contains
expiryin past. - Authority rejects with
ENVELOPE_EXPIRED.
- Envelope contains
-
PolicyChange_AuditTrail_Complete-
After valid change:
- Audit log contains:
policyName,oldHash,newHash,signerId,envelopeId,timestamp.
- Audit log contains:
-
5.3 Job routing (if/when you use DB-backed routing tables)
You discussed a routing table:
domain text,
instance_id uuid,
last_heartbeat timestamptz,
table_name text
Key tests (once implemented):
-
HeartbeatExpired_DropsRoutingEntry- Insert entry with
last_heartbeatolder than 1 minute. - Routing GC job should remove it.
- API gateway must not route new jobs to that instance.
- Insert entry with
-
RoundRobinAcrossAliveInstances- Multiple routing rows for same domain with fresh heartbeats.
- Issue M requests via gateway.
- Assert approximately round-robin distribution across
instance_id.
-
NoDurabilityRequired_JobsNotReplayedAfterRestart- Confirm that in-memory or temp tables are used appropriately where you do not want durable queues.
If you decide to go with “N gateways x M microservices via Docker load balancer only”, then the main tests here move to health-check based routing in the load balancer and become more infra than app tests.
6) CI wiring summary
To make this actually enforceable:
-
Unit test job (
test:unit)- Runs
StellaOps.Bundle.Tests,StellaOps.SbomGate.Tests,StellaOps.Authority.Tests,StellaOps.Scanner.Tests.
- Runs
-
DB recoverability job (
test:db-recoverability)- Uses Testcontainers to run
StellaOps.DataRecoverability.Tests. - Marked as “required” for
mainbranch merges.
- Uses Testcontainers to run
-
Acceptance job (
test:acceptance-system)-
Spins up a minimal stack via Docker Compose.
-
Executes
StellaOps.System.Acceptancetests:- Feed outages & fallback.
- Air-gap modes.
- Bundle swap.
-
Can be slower; run on main and release branches.
-
-
Nightly chaos job (
test:nightly-chaos)- Optional: run more expensive tests (simulated DB corruption, analyzer outages, etc.).
If you want, next step I can generate skeleton xUnit test classes and a /testdata layout you can paste directly into your repo (with TODOs where real fixtures are needed).