Add comprehensive security tests for OWASP A03 (Injection) and A10 (SSRF)
- Implemented InjectionTests.cs to cover various injection vulnerabilities including SQL, NoSQL, Command, LDAP, and XPath injections. - Created SsrfTests.cs to test for Server-Side Request Forgery (SSRF) vulnerabilities, including internal URL access, cloud metadata access, and URL allowlist bypass attempts. - Introduced MaliciousPayloads.cs to store a collection of malicious payloads for testing various security vulnerabilities. - Added SecurityAssertions.cs for common security-specific assertion helpers. - Established SecurityTestBase.cs as a base class for security tests, providing common infrastructure and mocking utilities. - Configured the test project StellaOps.Security.Tests.csproj with necessary dependencies for testing.
This commit is contained in:
@@ -205,3 +205,51 @@ CREATE INDEX IF NOT EXISTS idx_locks_expires ON scheduler.locks(expires_at);
|
||||
CREATE INDEX IF NOT EXISTS idx_run_summaries_tenant ON scheduler.run_summaries(tenant_id, period_start DESC);
|
||||
CREATE INDEX IF NOT EXISTS idx_audit_tenant_time ON scheduler.audit(tenant_id, occurred_at DESC);
|
||||
CREATE INDEX IF NOT EXISTS idx_audit_entity ON scheduler.audit(entity_type, entity_id);
|
||||
|
||||
-- =============================================================================
|
||||
-- Failure Signatures table for predictive TTFS signal hints
|
||||
-- Tracks common failure patterns by scope, toolchain, and error code
|
||||
-- Added: Sprint 0341
|
||||
-- =============================================================================
|
||||
CREATE TABLE IF NOT EXISTS scheduler.failure_signatures (
|
||||
signature_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
tenant_id UUID NOT NULL,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
|
||||
-- Scope: what artifact/repo/image this signature applies to
|
||||
scope_type TEXT NOT NULL CHECK (scope_type IN ('repo', 'image', 'artifact', 'global')),
|
||||
scope_id TEXT NOT NULL,
|
||||
|
||||
-- Toolchain: build environment fingerprint
|
||||
toolchain_hash TEXT NOT NULL,
|
||||
|
||||
-- Error classification
|
||||
error_code TEXT NULL,
|
||||
error_category TEXT NULL CHECK (error_category IN ('network', 'auth', 'validation', 'resource', 'timeout', 'config', 'unknown')),
|
||||
|
||||
-- Signature statistics
|
||||
occurrence_count INT NOT NULL DEFAULT 1,
|
||||
first_seen_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
last_seen_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
|
||||
-- Resolution status
|
||||
resolution_status TEXT NOT NULL DEFAULT 'unresolved' CHECK (resolution_status IN ('unresolved', 'investigating', 'resolved', 'wont_fix')),
|
||||
resolution_notes TEXT NULL,
|
||||
resolved_at TIMESTAMPTZ NULL,
|
||||
resolved_by TEXT NULL,
|
||||
|
||||
-- Predictive hints
|
||||
predicted_outcome TEXT NULL CHECK (predicted_outcome IN ('pass', 'fail', 'flaky', 'unknown')),
|
||||
confidence_score DECIMAL(5, 4) NULL CHECK (confidence_score >= 0 AND confidence_score <= 1),
|
||||
|
||||
-- Composite unique constraint
|
||||
UNIQUE (tenant_id, scope_type, scope_id, toolchain_hash, error_code)
|
||||
);
|
||||
|
||||
-- Indexes for failure_signatures
|
||||
CREATE INDEX IF NOT EXISTS idx_failure_sig_tenant ON scheduler.failure_signatures(tenant_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_failure_sig_scope ON scheduler.failure_signatures(scope_type, scope_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_failure_sig_error ON scheduler.failure_signatures(error_code) WHERE error_code IS NOT NULL;
|
||||
CREATE INDEX IF NOT EXISTS idx_failure_sig_last_seen ON scheduler.failure_signatures(last_seen_at DESC);
|
||||
CREATE INDEX IF NOT EXISTS idx_failure_sig_unresolved ON scheduler.failure_signatures(tenant_id, resolution_status) WHERE resolution_status = 'unresolved';
|
||||
|
||||
@@ -36,9 +36,9 @@ This sprint delivers enhancements to the TTFS system including predictive failur
|
||||
|
||||
| ID | Task | Owner | Status | Notes |
|
||||
|----|------|-------|--------|-------|
|
||||
| T1 | Create `failure_signatures` table | — | TODO | Database schema |
|
||||
| T2 | Create `IFailureSignatureRepository` | — | TODO | Data access |
|
||||
| T3 | Implement `FailureSignatureIndexer` | — | TODO | Background indexer |
|
||||
| T1 | Create `failure_signatures` table | Agent | DONE | Added to scheduler.sql |
|
||||
| T2 | Create `IFailureSignatureRepository` | Agent | DONE | Interface + Postgres impl |
|
||||
| T3 | Implement `FailureSignatureIndexer` | Agent | DONE | Background indexer service |
|
||||
| T4 | Integrate signatures into FirstSignal | — | TODO | lastKnownOutcome |
|
||||
| T5 | Add "Verify locally" commands to EvidencePanel | — | TODO | Copy affordances |
|
||||
| T6 | Create ProofSpine sub-component | — | TODO | Bundle hashes |
|
||||
|
||||
@@ -63,23 +63,23 @@ Per advisory §5:
|
||||
| T7 | Integrate with `DsseVerifier` for validation | TODO | | |
|
||||
| T8 | Integrate with Rekor offline verifier | TODO | | |
|
||||
| **Step 3: Normalization** | | | | |
|
||||
| T9 | Design normalization rules | TODO | | |
|
||||
| T10 | Implement stable JSON sorting | TODO | | |
|
||||
| T11 | Implement timestamp stripping | TODO | | |
|
||||
| T12 | Implement URI lowercase normalization | TODO | | |
|
||||
| T9 | Design normalization rules | DONE | Agent | `NormalizationOptions` with configurable rules. |
|
||||
| T10 | Implement stable JSON sorting | DONE | Agent | `JsonNormalizer.NormalizeObject()` with ordinal key sorting. |
|
||||
| T11 | Implement timestamp stripping | DONE | Agent | `JsonNormalizer` strips timestamp fields and values. |
|
||||
| T12 | Implement URI lowercase normalization | DONE | Agent | `JsonNormalizer.NormalizeValue()` lowercases URIs. |
|
||||
| T13 | Create canonical SBOM transformer | TODO | | |
|
||||
| **Step 4: Lattice Rules** | | | | |
|
||||
| T14 | Design `SourcePrecedence` lattice | DONE | Agent | `SourcePrecedence` enum (vendor > maintainer > 3rd-party) introduced in reconciliation models. |
|
||||
| T15 | Implement VEX merge with precedence | TODO | | |
|
||||
| T16 | Implement conflict resolution | TODO | | |
|
||||
| T17 | Create lattice configuration loader | TODO | | |
|
||||
| T15 | Implement VEX merge with precedence | DONE | Agent | `SourcePrecedenceLattice.Merge()` implements lattice-based merging. |
|
||||
| T16 | Implement conflict resolution | DONE | Agent | `SourcePrecedenceLattice.ResolveConflict()` with timestamp and status priority fallbacks. |
|
||||
| T17 | Create lattice configuration loader | DONE | Agent | `LatticeConfiguration` record with custom source mappings. |
|
||||
| **Step 5: Graph Emission** | | | | |
|
||||
| T18 | Design `EvidenceGraph` schema | TODO | | JSON Schema |
|
||||
| T19 | Implement deterministic graph serializer | TODO | | |
|
||||
| T20 | Create SHA-256 manifest generator | TODO | | |
|
||||
| T18 | Design `EvidenceGraph` schema | DONE | Agent | `EvidenceGraph`, `EvidenceNode`, `EvidenceEdge` models. |
|
||||
| T19 | Implement deterministic graph serializer | DONE | Agent | `EvidenceGraphSerializer` with stable ordering. |
|
||||
| T20 | Create SHA-256 manifest generator | DONE | Agent | `EvidenceGraphSerializer.ComputeHash()` writes `evidence-graph.sha256`. |
|
||||
| T21 | Integrate DSSE signing for output | TODO | | |
|
||||
| **Integration & Testing** | | | | |
|
||||
| T22 | Create `IEvidenceReconciler` service | TODO | | |
|
||||
| T22 | Create `IEvidenceReconciler` service | DONE | Agent | `IEvidenceReconciler` + `EvidenceReconciler` implementing 5-step algorithm. |
|
||||
| T23 | Wire to CLI `verify offline` command | TODO | | |
|
||||
| T24 | Write golden-file tests | TODO | | Determinism |
|
||||
| T25 | Write property-based tests | TODO | | Lattice properties |
|
||||
|
||||
@@ -40,13 +40,13 @@ Read before implementation:
|
||||
|
||||
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
|
||||
|---|---------|--------|---------------------------|--------|-----------------|
|
||||
| 1 | QGATE-0350-001 | TODO | None | Platform | Create `scripts/ci/compute-reachability-metrics.sh` to compute recall/precision from corpus |
|
||||
| 2 | QGATE-0350-002 | TODO | After #1 | Platform | Create `scripts/ci/reachability-thresholds.yaml` with enforcement thresholds |
|
||||
| 1 | QGATE-0350-001 | DONE | None | Platform | Create `scripts/ci/compute-reachability-metrics.sh` to compute recall/precision from corpus |
|
||||
| 2 | QGATE-0350-002 | DONE | After #1 | Platform | Create `scripts/ci/reachability-thresholds.yaml` with enforcement thresholds |
|
||||
| 3 | QGATE-0350-003 | TODO | After #2 | Platform | Add reachability gate job to `build-test-deploy.yml` |
|
||||
| 4 | QGATE-0350-004 | TODO | None | Platform | Create `scripts/ci/compute-ttfs-metrics.sh` to extract TTFS from test runs |
|
||||
| 5 | QGATE-0350-005 | TODO | After #4 | Platform | Create `bench/baselines/ttfs-baseline.json` with p50/p95 targets |
|
||||
| 4 | QGATE-0350-004 | DONE | None | Platform | Create `scripts/ci/compute-ttfs-metrics.sh` to extract TTFS from test runs |
|
||||
| 5 | QGATE-0350-005 | DONE | After #4 | Platform | Create `bench/baselines/ttfs-baseline.json` with p50/p95 targets |
|
||||
| 6 | QGATE-0350-006 | TODO | After #5 | Platform | Add TTFS regression gate to `build-test-deploy.yml` |
|
||||
| 7 | QGATE-0350-007 | TODO | None | Platform | Create `scripts/ci/enforce-performance-slos.sh` for scan/compute SLOs |
|
||||
| 7 | QGATE-0350-007 | DONE | None | Platform | Create `scripts/ci/enforce-performance-slos.sh` for scan/compute SLOs |
|
||||
| 8 | QGATE-0350-008 | TODO | After #7 | Platform | Add performance SLO gate to `build-test-deploy.yml` |
|
||||
| 9 | QGATE-0350-009 | TODO | After #3, #6, #8 | Platform | Create `docs/testing/ci-quality-gates.md` documentation |
|
||||
| 10 | QGATE-0350-010 | TODO | After #9 | Platform | Add quality gate status badges to repository README |
|
||||
|
||||
@@ -61,15 +61,15 @@ The SCA Failure Catalogue covers real-world scanner failure modes that have occu
|
||||
|
||||
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
|
||||
|---|---------|--------|---------------------------|--------|-----------------|
|
||||
| 1 | SCA-0351-001 | TODO | None | Scanner | Create FC6 fixture: Java Shadow JAR failure case |
|
||||
| 2 | SCA-0351-002 | TODO | None | Scanner | Create FC7 fixture: .NET Transitive Pinning failure case |
|
||||
| 3 | SCA-0351-003 | TODO | None | Scanner | Create FC8 fixture: Docker Multi-Stage Leakage failure case |
|
||||
| 4 | SCA-0351-004 | TODO | None | Scanner | Create FC9 fixture: PURL Namespace Collision failure case |
|
||||
| 5 | SCA-0351-005 | TODO | None | Scanner | Create FC10 fixture: CVE Split/Merge failure case |
|
||||
| 6 | SCA-0351-006 | TODO | After #1-5 | Scanner | Create DSSE manifests for all new fixtures |
|
||||
| 7 | SCA-0351-007 | TODO | After #6 | Scanner | Update `tests/fixtures/sca/catalogue/inputs.lock` |
|
||||
| 1 | SCA-0351-001 | DONE | None | Scanner | Create FC6 fixture: Java Shadow JAR failure case |
|
||||
| 2 | SCA-0351-002 | DONE | None | Scanner | Create FC7 fixture: .NET Transitive Pinning failure case |
|
||||
| 3 | SCA-0351-003 | DONE | None | Scanner | Create FC8 fixture: Docker Multi-Stage Leakage failure case |
|
||||
| 4 | SCA-0351-004 | DONE | None | Scanner | Create FC9 fixture: PURL Namespace Collision failure case |
|
||||
| 5 | SCA-0351-005 | DONE | None | Scanner | Create FC10 fixture: CVE Split/Merge failure case |
|
||||
| 6 | SCA-0351-006 | DONE | After #1-5 | Scanner | Create DSSE manifests for all new fixtures |
|
||||
| 7 | SCA-0351-007 | DONE | After #6 | Scanner | Update `tests/fixtures/sca/catalogue/inputs.lock` |
|
||||
| 8 | SCA-0351-008 | TODO | After #7 | Scanner | Add xUnit tests for FC6-FC10 in Scanner test project |
|
||||
| 9 | SCA-0351-009 | TODO | After #8 | Scanner | Update `tests/fixtures/sca/catalogue/README.md` documentation |
|
||||
| 9 | SCA-0351-009 | DONE | After #8 | Scanner | Update `tests/fixtures/sca/catalogue/README.md` documentation |
|
||||
| 10 | SCA-0351-010 | TODO | After #9 | Scanner | Validate all fixtures pass determinism checks |
|
||||
|
||||
## Wave Coordination
|
||||
|
||||
@@ -53,12 +53,12 @@ Read before implementation:
|
||||
|
||||
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
|
||||
|---|---------|--------|---------------------------|--------|-----------------|
|
||||
| 1 | SEC-0352-001 | TODO | None | Security | Create `tests/security/` directory structure and base classes |
|
||||
| 2 | SEC-0352-002 | TODO | After #1 | Security | Implement A01: Broken Access Control tests for Authority |
|
||||
| 1 | SEC-0352-001 | DONE | None | Security | Create `tests/security/` directory structure and base classes |
|
||||
| 2 | SEC-0352-002 | DONE | After #1 | Security | Implement A01: Broken Access Control tests for Authority |
|
||||
| 3 | SEC-0352-003 | TODO | After #1 | Security | Implement A02: Cryptographic Failures tests for Signer |
|
||||
| 4 | SEC-0352-004 | TODO | After #1 | Security | Implement A03: Injection tests (SQL, Command, ORM) |
|
||||
| 4 | SEC-0352-004 | DONE | After #1 | Security | Implement A03: Injection tests (SQL, Command, ORM) |
|
||||
| 5 | SEC-0352-005 | TODO | After #1 | Security | Implement A07: Authentication Failures tests |
|
||||
| 6 | SEC-0352-006 | TODO | After #1 | Security | Implement A10: SSRF tests for Scanner and Concelier |
|
||||
| 6 | SEC-0352-006 | DONE | After #1 | Security | Implement A10: SSRF tests for Scanner and Concelier |
|
||||
| 7 | SEC-0352-007 | TODO | After #2-6 | Security | Implement A05: Security Misconfiguration tests |
|
||||
| 8 | SEC-0352-008 | TODO | After #2-6 | Security | Implement A08: Software/Data Integrity tests |
|
||||
| 9 | SEC-0352-009 | TODO | After #7-8 | Platform | Add security test job to CI workflow |
|
||||
|
||||
@@ -62,15 +62,15 @@ Read before implementation:
|
||||
|
||||
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
|
||||
|---|---------|--------|---------------------------|--------|-----------------|
|
||||
| 1 | MUT-0353-001 | TODO | None | Platform | Install Stryker.NET tooling and create base configuration |
|
||||
| 2 | MUT-0353-002 | TODO | After #1 | Scanner | Configure Stryker for Scanner.Core module |
|
||||
| 3 | MUT-0353-003 | TODO | After #1 | Policy | Configure Stryker for Policy.Engine module |
|
||||
| 4 | MUT-0353-004 | TODO | After #1 | Authority | Configure Stryker for Authority.Core module |
|
||||
| 1 | MUT-0353-001 | DONE | None | Platform | Install Stryker.NET tooling and create base configuration |
|
||||
| 2 | MUT-0353-002 | DONE | After #1 | Scanner | Configure Stryker for Scanner.Core module |
|
||||
| 3 | MUT-0353-003 | DONE | After #1 | Policy | Configure Stryker for Policy.Engine module |
|
||||
| 4 | MUT-0353-004 | DONE | After #1 | Authority | Configure Stryker for Authority.Core module |
|
||||
| 5 | MUT-0353-005 | TODO | After #2-4 | Platform | Run initial mutation testing, establish baselines |
|
||||
| 6 | MUT-0353-006 | TODO | After #5 | Platform | Create mutation score threshold configuration |
|
||||
| 6 | MUT-0353-006 | DONE | After #5 | Platform | Create mutation score threshold configuration |
|
||||
| 7 | MUT-0353-007 | TODO | After #6 | Platform | Add mutation testing job to CI workflow |
|
||||
| 8 | MUT-0353-008 | TODO | After #2-4 | Platform | Configure Stryker for secondary modules (Signer, Attestor) |
|
||||
| 9 | MUT-0353-009 | TODO | After #7 | Platform | Create `docs/testing/mutation-testing-guide.md` |
|
||||
| 9 | MUT-0353-009 | DONE | After #7 | Platform | Create `docs/testing/mutation-testing-guide.md` |
|
||||
| 10 | MUT-0353-010 | TODO | After #9 | Platform | Add mutation score badges and reporting |
|
||||
|
||||
## Wave Coordination
|
||||
|
||||
@@ -58,15 +58,15 @@ Before starting, read:
|
||||
## Delivery Tracker
|
||||
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| 1 | T1 | DOING | Update `IRekorClient` contract | Attestor Guild | Add `VerifyInclusionAsync` to `IRekorClient` interface |
|
||||
| 2 | T2 | TODO | Implement RFC 6962 verifier | Attestor Guild | Implement `MerkleProofVerifier` utility class |
|
||||
| 1 | T1 | DONE | Update `IRekorClient` contract | Attestor Guild | Add `VerifyInclusionAsync` to `IRekorClient` interface |
|
||||
| 2 | T2 | DONE | Implement RFC 6962 verifier | Attestor Guild | Implement `MerkleProofVerifier` utility class |
|
||||
| 3 | T3 | TODO | Parse and verify checkpoint signatures | Attestor Guild | Implement checkpoint signature verification |
|
||||
| 4 | T4 | TODO | Expose verification settings | Attestor Guild | Add Rekor public key configuration to `AttestorOptions` |
|
||||
| 5 | T5 | TODO | Use verifiers in HTTP client | Attestor Guild | Implement `HttpRekorClient.VerifyInclusionAsync` |
|
||||
| 6 | T6 | TODO | Stub verification behavior | Attestor Guild | Implement `StubRekorClient.VerifyInclusionAsync` |
|
||||
| 5 | T5 | DONE | Use verifiers in HTTP client | Attestor Guild | Implement `HttpRekorClient.VerifyInclusionAsync` |
|
||||
| 6 | T6 | DONE | Stub verification behavior | Attestor Guild | Implement `StubRekorClient.VerifyInclusionAsync` |
|
||||
| 7 | T7 | TODO | Wire verification pipeline | Attestor Guild | Integrate verification into `AttestorVerificationService` |
|
||||
| 8 | T8 | TODO | Add sealed/offline checkpoint mode | Attestor Guild | Add offline verification mode with bundled checkpoint |
|
||||
| 9 | T9 | TODO | Add unit coverage | Attestor Guild | Add unit tests for Merkle proof verification |
|
||||
| 9 | T9 | DONE | Add unit coverage | Attestor Guild | Add unit tests for Merkle proof verification |
|
||||
| 10 | T10 | TODO | Add integration coverage | Attestor Guild | Add integration tests with mock Rekor responses |
|
||||
| 11 | T11 | TODO | Expose verification counters | Attestor Guild | Update `AttestorMetrics` with verification counters |
|
||||
| 12 | T12 | TODO | Sync docs | Attestor Guild | Update module documentation
|
||||
|
||||
@@ -39,9 +39,9 @@ Implement the three-tier fidelity metrics framework for measuring deterministic
|
||||
| 7 | FID-3403-007 | TODO | After #6 | Determinism Team | Integrate fidelity metrics into `DeterminismReport` |
|
||||
| 8 | FID-3403-008 | TODO | After #6 | Telemetry Team | Add Prometheus gauges for BF, SF, PF metrics |
|
||||
| 9 | FID-3403-009 | TODO | After #8 | Telemetry Team | Add SLO alerting for fidelity thresholds |
|
||||
| 10 | FID-3403-010 | TODO | After #3 | Determinism Team | Unit tests for bitwise fidelity calculation |
|
||||
| 11 | FID-3403-011 | TODO | After #4 | Determinism Team | Unit tests for semantic fidelity comparison |
|
||||
| 12 | FID-3403-012 | TODO | After #5 | Determinism Team | Unit tests for policy fidelity comparison |
|
||||
| 10 | FID-3403-010 | DONE | After #3 | Determinism Team | Unit tests for bitwise fidelity calculation |
|
||||
| 11 | FID-3403-011 | DONE | After #4 | Determinism Team | Unit tests for semantic fidelity comparison |
|
||||
| 12 | FID-3403-012 | DONE | After #5 | Determinism Team | Unit tests for policy fidelity comparison |
|
||||
| 13 | FID-3403-013 | TODO | After #7 | QA | Integration test: fidelity metrics in determinism harness |
|
||||
| 14 | FID-3403-014 | TODO | After #9 | Docs Guild | Document fidelity metrics in `docs/benchmarks/fidelity-metrics.md` |
|
||||
|
||||
|
||||
@@ -32,16 +32,16 @@ Implement gate detection and multipliers for reachability scoring, reducing risk
|
||||
|
||||
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
|
||||
|---|---------|--------|---------------------------|--------|-----------------|
|
||||
| 1 | GATE-3405-001 | TODO | None | Reachability Team | Define `GateType` enum and `DetectedGate` record |
|
||||
| 2 | GATE-3405-002 | TODO | None | Reachability Team | Define gate detection patterns for each language analyzer |
|
||||
| 3 | GATE-3405-003 | TODO | After #1 | Reachability Team | Implement `AuthGateDetector` for authentication checks |
|
||||
| 4 | GATE-3405-004 | TODO | After #1 | Reachability Team | Implement `FeatureFlagDetector` for feature flag checks |
|
||||
| 5 | GATE-3405-005 | TODO | After #1 | Reachability Team | Implement `AdminOnlyDetector` for admin/role checks |
|
||||
| 6 | GATE-3405-006 | TODO | After #1 | Reachability Team | Implement `ConfigGateDetector` for non-default config checks |
|
||||
| 1 | GATE-3405-001 | DONE | None | Reachability Team | Define `GateType` enum and `DetectedGate` record |
|
||||
| 2 | GATE-3405-002 | DONE | None | Reachability Team | Define gate detection patterns for each language analyzer |
|
||||
| 3 | GATE-3405-003 | DONE | After #1 | Reachability Team | Implement `AuthGateDetector` for authentication checks |
|
||||
| 4 | GATE-3405-004 | DONE | After #1 | Reachability Team | Implement `FeatureFlagDetector` for feature flag checks |
|
||||
| 5 | GATE-3405-005 | DONE | After #1 | Reachability Team | Implement `AdminOnlyDetector` for admin/role checks |
|
||||
| 6 | GATE-3405-006 | DONE | After #1 | Reachability Team | Implement `ConfigGateDetector` for non-default config checks |
|
||||
| 7 | GATE-3405-007 | TODO | After #3-6 | Reachability Team | Implement `CompositeGateDetector` orchestrating all detectors |
|
||||
| 8 | GATE-3405-008 | TODO | After #7 | Reachability Team | Extend `RichGraphEdge` with `Gates` property |
|
||||
| 8 | GATE-3405-008 | DONE | After #7 | Reachability Team | Extend `RichGraphEdge` with `Gates` property |
|
||||
| 9 | GATE-3405-009 | TODO | After #8 | Reachability Team | Integrate gate detection into RichGraph building pipeline |
|
||||
| 10 | GATE-3405-010 | TODO | After #9 | Signals Team | Implement `GateMultiplierCalculator` applying multipliers |
|
||||
| 10 | GATE-3405-010 | DONE | After #9 | Signals Team | Implement `GateMultiplierCalculator` applying multipliers |
|
||||
| 11 | GATE-3405-011 | TODO | After #10 | Signals Team | Integrate multipliers into `ReachabilityScoringService` |
|
||||
| 12 | GATE-3405-012 | TODO | After #11 | Signals Team | Update `ReachabilityReport` contract with gates array |
|
||||
| 13 | GATE-3405-013 | TODO | After #3 | Reachability Team | Unit tests for AuthGateDetector patterns |
|
||||
|
||||
@@ -951,7 +951,7 @@ public interface ISuppressionOverrideProvider
|
||||
| # | Task ID | Status | Description | Assignee | Notes |
|
||||
|---|---------|--------|-------------|----------|-------|
|
||||
| 1 | SDIFF-FND-001 | DONE | Create `StellaOps.Scanner.SmartDiff` project | | Library created |
|
||||
| 2 | SDIFF-FND-002 | TODO | Add smart-diff JSON Schema to Attestor.Types | | `stellaops-smart-diff.v1.schema.json` |
|
||||
| 2 | SDIFF-FND-002 | DONE | Add smart-diff JSON Schema to Attestor.Types | | `stellaops-smart-diff.v1.schema.json` exists |
|
||||
| 3 | SDIFF-FND-003 | TODO | Register predicate in type generator | | `SmartDiffPredicateDefinition.cs` |
|
||||
| 4 | SDIFF-FND-004 | DONE | Implement `SmartDiffPredicate.cs` models | | All records implemented |
|
||||
| 5 | SDIFF-FND-005 | DONE | Implement `ReachabilityGate` with 3-bit class | | ComputeClass method implemented |
|
||||
@@ -960,11 +960,11 @@ public interface ISuppressionOverrideProvider
|
||||
| 8 | SDIFF-FND-008 | DONE | Create `StellaOps.Policy.Suppression` namespace | | Created |
|
||||
| 9 | SDIFF-FND-009 | DONE | Implement `SuppressionRuleEvaluator` | | Full implementation |
|
||||
| 10 | SDIFF-FND-010 | DONE | Implement `ISuppressionOverrideProvider` | | Interface defined |
|
||||
| 11 | SDIFF-FND-011 | TODO | Add patch churn suppression logic | | `EvaluatePatchChurn` method |
|
||||
| 12 | SDIFF-FND-012 | TODO | Unit tests for `ReachabilityGate.ComputeClass` | | All 8 class values + null cases |
|
||||
| 11 | SDIFF-FND-011 | DONE | Add patch churn suppression logic | | `EvaluatePatchChurn` method exists |
|
||||
| 12 | SDIFF-FND-012 | DONE | Unit tests for `ReachabilityGate.ComputeClass` | | ReachabilityGateTests.cs has full coverage |
|
||||
| 13 | SDIFF-FND-013 | DONE | Unit tests for `SinkRegistry.MatchSink` | | SinkRegistryTests.cs |
|
||||
| 14 | SDIFF-FND-014 | DONE | Unit tests for `SuppressionRuleEvaluator` | | SuppressionRuleEvaluatorTests.cs |
|
||||
| 15 | SDIFF-FND-015 | TODO | Golden fixtures for predicate serialization | | Determinism test |
|
||||
| 15 | SDIFF-FND-015 | DONE | Golden fixtures for predicate serialization | | PredicateGoldenFixtureTests.cs |
|
||||
| 16 | SDIFF-FND-016 | TODO | JSON Schema validation tests | | Via `JsonSchema.Net` |
|
||||
| 17 | SDIFF-FND-017 | TODO | Run type generator to produce TS/Go bindings | | `dotnet run` generator |
|
||||
| 18 | SDIFF-FND-018 | TODO | Update Scanner AGENTS.md | | New contracts |
|
||||
|
||||
210
docs/testing/mutation-testing-guide.md
Normal file
210
docs/testing/mutation-testing-guide.md
Normal file
@@ -0,0 +1,210 @@
|
||||
# Mutation Testing Guide
|
||||
|
||||
This guide documents the integration and usage of Stryker.NET mutation testing in StellaOps.
|
||||
|
||||
## Overview
|
||||
|
||||
Mutation testing measures test suite effectiveness by introducing small code changes (mutants) and verifying that tests detect them. Unlike line coverage, mutation testing answers: **"Would my tests catch this bug?"**
|
||||
|
||||
## Installation
|
||||
|
||||
Stryker.NET is configured as a local dotnet tool:
|
||||
|
||||
```bash
|
||||
# Restore tools (includes Stryker.NET)
|
||||
dotnet tool restore
|
||||
|
||||
# Verify installation
|
||||
dotnet stryker --version
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
### Solution-Level Configuration
|
||||
|
||||
Base configuration is at `stryker-config.json` in the solution root. Module-specific configs override these settings.
|
||||
|
||||
### Module Configurations
|
||||
|
||||
| Module | Config Path | Mutation Break Threshold |
|
||||
|--------|-------------|-------------------------|
|
||||
| Scanner.Core | `src/Scanner/__Libraries/StellaOps.Scanner.Core/stryker-config.json` | 60% |
|
||||
| Policy.Engine | `src/Policy/StellaOps.Policy.Engine/stryker-config.json` | 60% |
|
||||
| Authority | `src/Authority/StellaOps.Authority/stryker-config.json` | 65% |
|
||||
|
||||
## Running Mutation Tests
|
||||
|
||||
### Single Module
|
||||
|
||||
```bash
|
||||
# Navigate to module directory
|
||||
cd src/Scanner/__Libraries/StellaOps.Scanner.Core
|
||||
|
||||
# Run mutation testing
|
||||
dotnet stryker
|
||||
|
||||
# With specific config
|
||||
dotnet stryker --config-file stryker-config.json
|
||||
```
|
||||
|
||||
### All Configured Modules
|
||||
|
||||
```bash
|
||||
# From solution root
|
||||
dotnet stryker --solution StellaOps.Router.slnx
|
||||
```
|
||||
|
||||
### CI Mode (Threshold Enforcement)
|
||||
|
||||
```bash
|
||||
# Fails if mutation score below threshold
|
||||
dotnet stryker --break-at-score 60
|
||||
```
|
||||
|
||||
## Understanding Results
|
||||
|
||||
### Mutation Score
|
||||
|
||||
```
|
||||
Mutation Score = (Killed Mutants / Total Mutants) × 100
|
||||
```
|
||||
|
||||
- **Killed**: Test failed when mutant was introduced (good!)
|
||||
- **Survived**: Test passed with mutant present (test gap!)
|
||||
- **No Coverage**: No test covered the mutated code
|
||||
- **Timeout**: Test timed out (usually treated as killed)
|
||||
|
||||
### Thresholds
|
||||
|
||||
| Level | Score | Meaning |
|
||||
|-------|-------|---------|
|
||||
| High | ≥80% | Excellent test effectiveness |
|
||||
| Low | ≥60% | Acceptable, improvements needed |
|
||||
| Break | <50% | Build fails, critical gaps |
|
||||
|
||||
### Example Output
|
||||
|
||||
```
|
||||
All mutants have been tested, and your mutation score has been calculated
|
||||
╔═══════════════════════════════════════════════════════════════════════╗
|
||||
║ Mutation Testing Report ║
|
||||
╠═══════════════════════════════════════════════════════════════════════╣
|
||||
║ Mutants tested: 156 ║
|
||||
║ Mutants killed: 134 ║
|
||||
║ Mutants survived: 18 ║
|
||||
║ Mutants no coverage: 4 ║
|
||||
║ Mutation score: 85.90% ║
|
||||
╚═══════════════════════════════════════════════════════════════════════╝
|
||||
```
|
||||
|
||||
## Common Mutators
|
||||
|
||||
| Mutator | Original | Mutant |
|
||||
|---------|----------|--------|
|
||||
| Comparison | `>=` | `>` |
|
||||
| Equality | `==` | `!=` |
|
||||
| Boolean | `true` | `false` |
|
||||
| Logical | `&&` | `\|\|` |
|
||||
| Arithmetic | `+` | `-` |
|
||||
| NullCoalescing | `??` | ` ` (remove) |
|
||||
|
||||
## Fixing Survived Mutants
|
||||
|
||||
### 1. Analyze the Report
|
||||
|
||||
Open the HTML report in `.stryker/output/<module>/mutation-report.html`.
|
||||
|
||||
### 2. Identify the Gap
|
||||
|
||||
Look at the survived mutant:
|
||||
|
||||
```csharp
|
||||
// Original
|
||||
if (score >= threshold) { return "PASS"; }
|
||||
|
||||
// Mutant (survived!)
|
||||
if (score > threshold) { return "PASS"; }
|
||||
```
|
||||
|
||||
### 3. Add Missing Test
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public void Should_Pass_When_Score_Equals_Threshold()
|
||||
{
|
||||
var score = 60;
|
||||
var threshold = 60;
|
||||
|
||||
var result = EvaluateScore(score, threshold);
|
||||
|
||||
result.Should().Be("PASS"); // Now kills the >= to > mutant
|
||||
}
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
### 1. Focus on Critical Modules First
|
||||
|
||||
Prioritize mutation testing for:
|
||||
- Security-critical code (Authority, Signer)
|
||||
- Business logic (Policy decisions, Scanner matching)
|
||||
- Boundary conditions
|
||||
|
||||
### 2. Don't Chase 100%
|
||||
|
||||
Some mutants are false positives or equivalent mutants. Aim for 80%+ on critical modules.
|
||||
|
||||
### 3. Use Baseline Mode
|
||||
|
||||
Enable baseline to only test changed files:
|
||||
|
||||
```bash
|
||||
dotnet stryker --with-baseline:main
|
||||
```
|
||||
|
||||
### 4. Exclude Non-Critical Code
|
||||
|
||||
Exclude from mutation testing:
|
||||
- DTOs and models
|
||||
- Generated code
|
||||
- Migrations
|
||||
- UI components
|
||||
|
||||
## CI Integration
|
||||
|
||||
Mutation testing runs in CI:
|
||||
|
||||
```yaml
|
||||
mutation-test:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- name: Run Stryker
|
||||
run: |
|
||||
dotnet tool restore
|
||||
dotnet stryker --break-at-score 60
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Slow Execution
|
||||
|
||||
- Use `--concurrency` to control parallelism
|
||||
- Enable `coverage-analysis: perTest` for smarter mutant selection
|
||||
- Use `--since:main` to only test changed code
|
||||
|
||||
### Out of Memory
|
||||
|
||||
- Reduce `--concurrency` value
|
||||
- Exclude large test projects
|
||||
|
||||
### Timeout Issues
|
||||
|
||||
- Adjust `--timeout` setting
|
||||
- Some infinite loop mutants may timeout (this is expected)
|
||||
|
||||
## References
|
||||
|
||||
- [Stryker.NET Documentation](https://stryker-mutator.io/docs/stryker-net/introduction/)
|
||||
- [Mutation Testing Theory](https://en.wikipedia.org/wiki/Mutation_testing)
|
||||
- StellaOps Test Suite Overview: `docs/19_TEST_SUITE_OVERVIEW.md`
|
||||
Reference in New Issue
Block a user