test fixes and new product advisories work

2026-01-28 02:30:48 +02:00
parent 82caceba56
commit 644887997c
288 changed files with 69101 additions and 375 deletions
--- a/docs/technical/testing/testing-strategy-models.md
+++ b/docs/technical/testing/testing-strategy-models.md
@@ -50,3 +50,115 @@ Supersedes/extends: `docs/product/advisories/archived/2025-12-21-testing-strateg
 - Test suite overview: `docs/technical/testing/TEST_SUITE_OVERVIEW.md`
 - Quality guardrails: `docs/technical/testing/testing-quality-guardrails-implementation.md`
 - Code samples from the advisory: `docs/benchmarks/testing/better-testing-strategy-samples.md`
+
+---
+
+## Turn #6 Enhancements (2026-01-27)
+
+Source advisory: Testing Enhancements (Automation Turn #6)
+Sprint: `docs/implplan/SPRINT_0127_002_DOCS_testing_enhancements_turn6.md`
+
+### New test intent categories
+
+Every non-trivial test must declare intent. Intent clarifies *why* the behavior exists.
+
+```csharp
+public static class TestIntents
+{
+    public const string Regulatory = "Regulatory";   // Compliance, audit, legal
+    public const string Safety = "Safety";           // Security, fail-secure, crypto
+    public const string Performance = "Performance"; // Latency, throughput, resources
+    public const string Competitive = "Competitive"; // Parity with competitor tools
+    public const string Operational = "Operational"; // Observability, operability
+}
+
+// Usage
+[Trait("Intent", TestIntents.Safety)]
+[Trait("Category", "Unit")]
+public void Signer_RejectsExpiredCertificate() { /* ... */ }
+```
+
+### New test trait categories
+
+| Category | Purpose | Example Usage |
+|----------|---------|---------------|
+| `Intent` | Test intent classification | `[Trait("Intent", "Safety")]` |
+| `Evidence` | Evidence chain validation | `[Trait("Category", "Evidence")]` |
+| `Observability` | OTel/log/metrics contracts | `[Trait("Category", "Observability")]` |
+| `Longevity` | Time-extended stability tests | `[Trait("Category", "Longevity")]` |
+| `Interop` | Cross-version/environment skew | `[Trait("Category", "Interop")]` |
+| `PostIncident` | Tests from production incidents | `[Trait("Category", "PostIncident")]` |
+
+### Updated test model requirements
+
+| Model | Turn #6 Additions |
+|-------|-------------------|
+| L0 (Library/Core) | + Intent trait required for non-trivial tests |
+| S1 (Storage/Postgres) | + Interop tests for schema version migrations |
+| W1 (WebService/API) | + Observability contract tests (OTel spans, log fields, metrics) |
+| WK1 (Worker/Indexer) | + Longevity tests for memory/connection stability |
+| CLI1 (Tool/CLI) | + PostIncident regression tests |
+
+### New CI lanes
+
+| Lane | Purpose | Cadence | Gating |
+|------|---------|---------|--------|
+| Evidence | Evidence chain validation, traceability | Per PR | PR-gating for regulatory modules |
+| Longevity | Time-extended stability tests | Nightly | Release-gating |
+| Interop | Cross-version compatibility | Weekly + pre-release | Release-gating |
+
+### Observability contract requirements (W1 model)
+
+WebService tests must validate:
+- **OTel spans**: required spans exist, attributes present, cardinality bounded.
+- **Structured logs**: required fields present, no PII, appropriate log levels.
+- **Metrics**: required metrics exist, label cardinality bounded, counters monotonic.
+
+```csharp
+[Trait("Category", "Observability")]
+[Trait("Intent", "Operational")]
+public async Task Scanner_EmitsRequiredTelemetry()
+{
+    using var otel = new OtelCapture();
+    await sut.ScanAsync(request);
+
+    OTelContractAssert.HasRequiredSpans(otel, "ScanImage", "ExtractLayers", "AnalyzeSBOM");
+    OTelContractAssert.NoHighCardinalityAttributes(otel, threshold: 100);
+}
+```
+
+### Evidence traceability requirements
+
+Regulatory tests must link to requirements:
+```csharp
+[Requirement("REQ-EVIDENCE-001")]
+[Trait("Intent", "Regulatory")]
+public void EvidenceBundle_IsImmutableAfterSigning() { /* ... */ }
+```
+
+CI generates traceability matrix: requirement -> test -> artifact.
+
+### Cross-version testing requirements (Interop)
+
+For modules with schema or API versioning:
+- Test N-1 compatibility (current server, previous client).
+- Test N+1 compatibility (previous server, current client).
+- Document compatibility matrix.
+
+### Time-extended testing requirements (Longevity)
+
+For worker modules (WK1 model):
+- Memory stability: verify no growth under sustained load.
+- Connection pool stability: verify no leaks.
+- Counter drift: verify values remain bounded.
+
+Run duration: 1+ hours for nightly, 4+ hours for release validation.
+
+### Post-incident testing requirements
+
+For P1/P2 production incidents:
+1. Capture event sequence via replay infrastructure.
+2. Generate test scaffold from replay manifest.
+3. Include incident metadata (ID, root cause, severity).
+4. Tag with `[Trait("Category", "PostIncident")]`.
+5. Test failures block releases.