test fixes and new product advisories work
This commit is contained in:
master
@@ -14,9 +14,9 @@
|
||||
|
||||
## Cadence
|
||||
- Per change: unit tests plus relevant integration tests and determinism checks.
|
||||
- Nightly: full integration and end-to-end suites per module.
|
||||
- Weekly: performance baselines and flakiness triage.
|
||||
- Release gate: full test matrix, security verification, and reproducible build checks.
|
||||
- Nightly: full integration, end-to-end suites, and longevity tests per module.
|
||||
- Weekly: performance baselines, flakiness triage, and cross-version compatibility checks.
|
||||
- Release gate: full test matrix, security verification, reproducible build checks, and interop validation.
|
||||
|
||||
## Evidence and reporting
|
||||
- Record results in sprint Execution Logs with date, scope, and outcomes.
|
||||
@@ -27,3 +27,165 @@
|
||||
- Use UTC timestamps, fixed seeds, and CultureInfo.InvariantCulture where relevant.
|
||||
- Avoid live network calls; rely on fixtures and local emulators only.
|
||||
- Inject time and ID providers (TimeProvider, IGuidGenerator) for testability.
|
||||
|
||||
---
|
||||
|
||||
## Intent tagging (Turn #6)
|
||||
|
||||
Every non-trivial test must declare its intent using the `Intent` trait. Intent clarifies *why* the behavior exists and enables CI to flag changes that violate intent even if tests pass.
|
||||
|
||||
**Intent categories:**
|
||||
- `Regulatory`: compliance, audit requirements, legal obligations.
|
||||
- `Safety`: security invariants, fail-secure behavior, cryptographic correctness.
|
||||
- `Performance`: latency, throughput, resource usage guarantees.
|
||||
- `Competitive`: parity with competitor tools (Syft, Grype, Trivy, Anchore).
|
||||
- `Operational`: observability, diagnosability, operability requirements.
|
||||
|
||||
**Usage:**
|
||||
```csharp
|
||||
[Trait("Intent", "Safety")]
|
||||
[Trait("Category", "Unit")]
|
||||
public void Signer_RejectsExpiredCertificate()
|
||||
{
|
||||
// Test that expired certificates are rejected (safety invariant)
|
||||
}
|
||||
|
||||
[Trait("Intent", "Regulatory")]
|
||||
[Trait("Category", "Integration")]
|
||||
public void EvidenceBundle_IsImmutableAfterSigning()
|
||||
{
|
||||
// Test that signed evidence cannot be modified (audit requirement)
|
||||
}
|
||||
```
|
||||
|
||||
**Enforcement:**
|
||||
- Tests without intent tags in regulatory modules (Policy, Authority, Signer, Attestor, EvidenceLocker) will trigger CI warnings.
|
||||
- Intent coverage metrics are tracked per module in TEST_COVERAGE_MATRIX.md.
|
||||
|
||||
---
|
||||
|
||||
## Observability contract testing (Turn #6)
|
||||
|
||||
Logs, metrics, and traces are APIs. WebService tests (W1 model) must validate observability contracts.
|
||||
|
||||
**OTel trace contracts:**
|
||||
- Required spans must exist for core operations.
|
||||
- Span attributes must include required fields (correlation ID, tenant ID where applicable).
|
||||
- Attribute cardinality must be bounded (no unbounded label explosion).
|
||||
|
||||
**Structured log contracts:**
|
||||
- Required fields must be present (timestamp, level, message, correlation ID).
|
||||
- No PII in logs (validated via pattern matching).
|
||||
- Log levels must be appropriate (no ERROR for expected conditions).
|
||||
|
||||
**Metrics contracts:**
|
||||
- Required metrics must exist for core operations.
|
||||
- Label cardinality must be bounded (< 100 distinct values per label).
|
||||
- Counters must be monotonic.
|
||||
|
||||
**Usage:**
|
||||
```csharp
|
||||
using var otel = new OtelCapture();
|
||||
await sut.ProcessAsync(request);
|
||||
|
||||
OTelContractAssert.HasRequiredSpans(otel, "ProcessRequest", "ValidateInput", "PersistResult");
|
||||
OTelContractAssert.SpanHasAttributes(otel.GetSpan("ProcessRequest"), "corr_id", "tenant_id");
|
||||
OTelContractAssert.NoHighCardinalityAttributes(otel, threshold: 100);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Evidence traceability (Turn #6)
|
||||
|
||||
Every critical behavior must link: requirement -> test -> run -> artifact -> deployed version. This chain enables audit and root cause analysis.
|
||||
|
||||
**Requirement linking:**
|
||||
```csharp
|
||||
[Requirement("REQ-EVIDENCE-001", SprintTaskId = "TEST-ENH6-06")]
|
||||
[Trait("Intent", "Regulatory")]
|
||||
public void EvidenceChain_IsComplete()
|
||||
{
|
||||
// Test that evidence chain is traceable
|
||||
}
|
||||
```
|
||||
|
||||
**Artifact immutability:**
|
||||
- Tests for compliance-critical artifacts must verify hash stability.
|
||||
- Use `EvidenceChainAssert.ArtifactImmutable()` for determinism verification.
|
||||
|
||||
**Traceability reporting:**
|
||||
- CI generates traceability matrix linking requirements to tests to artifacts.
|
||||
- Orphaned tests (no requirement reference) in regulatory modules trigger warnings.
|
||||
|
||||
---
|
||||
|
||||
## Cross-version and environment testing (Turn #6)
|
||||
|
||||
Integration tests must validate interoperability across versions and environments.
|
||||
|
||||
**Cross-version testing (Interop):**
|
||||
- N-1 compatibility: current service must work with previous schema/API version.
|
||||
- N+1 compatibility: previous service must work with current schema/API version.
|
||||
- Run before releases to prevent breaking changes.
|
||||
|
||||
**Environment skew testing:**
|
||||
- Run integration tests across varied infrastructure profiles.
|
||||
- Profiles: standard, high-latency (100ms), low-bandwidth (10 Mbps), packet-loss (1%).
|
||||
- Assert result equivalence across profiles.
|
||||
|
||||
**Usage:**
|
||||
```csharp
|
||||
[Trait("Category", "Interop")]
|
||||
public async Task SchemaV2_CompatibleWithV1Client()
|
||||
{
|
||||
await using var v1Client = await fixture.StartVersion("v1.0.0", "EvidenceLocker");
|
||||
await using var v2Server = await fixture.StartVersion("v2.0.0", "EvidenceLocker");
|
||||
|
||||
var result = await fixture.TestHandshake(v1Client, v2Server);
|
||||
Assert.True(result.IsCompatible);
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Time-extended and post-incident testing (Turn #6)
|
||||
|
||||
Long-running tests surface issues that only emerge over time. Post-incident tests prevent recurrence.
|
||||
|
||||
**Time-extended (longevity) tests:**
|
||||
- Run E2E scenarios continuously for hours to detect memory leaks, counter drift, quota exhaustion.
|
||||
- Verify memory returns to baseline after sustained load.
|
||||
- Verify connection pools do not leak under sustained load.
|
||||
- Run nightly; release-gating for critical modules.
|
||||
|
||||
**Post-incident replay tests:**
|
||||
- Every production incident (P1/P2) produces a permanent E2E regression test.
|
||||
- Test derived from replay manifest capturing exact event sequence.
|
||||
- Test includes incident metadata (ID, root cause, severity).
|
||||
- Tests tagged with `[Trait("Category", "PostIncident")]`.
|
||||
|
||||
**Usage:**
|
||||
```csharp
|
||||
[Trait("Category", "Longevity")]
|
||||
[Trait("Intent", "Operational")]
|
||||
public async Task ScannerWorker_NoMemoryLeakUnderLoad()
|
||||
{
|
||||
var runner = new StabilityTestRunner();
|
||||
await runner.RunExtended(
|
||||
scenario: () => ProcessScanBatch(),
|
||||
duration: TimeSpan.FromHours(1),
|
||||
metrics: new StabilityMetrics(),
|
||||
ct: CancellationToken.None);
|
||||
|
||||
var report = runner.GenerateReport();
|
||||
Assert.True(report.MemoryGrowthRate < 0.01, "Memory growth rate exceeds threshold");
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Related documents
|
||||
- Test strategy models: `docs/technical/testing/testing-strategy-models.md`
|
||||
- CI quality gates: `docs/technical/testing/ci-quality-gates.md`
|
||||
- TestKit usage: `docs/technical/testing/testkit-usage-guide.md`
|
||||
- Test coverage matrix: `docs/technical/testing/TEST_COVERAGE_MATRIX.md`
|
||||
|
||||
Reference in New Issue
Block a user