test fixes and new product advisories work

This commit is contained in:
master
2026-01-28 02:30:48 +02:00
parent 82caceba56
commit 644887997c
288 changed files with 69101 additions and 375 deletions

View File

@@ -147,6 +147,158 @@ If baselines become stale:
./scripts/ci/compute-reachability-metrics.sh --update-baseline
```
---
## Turn #6 Quality Gates (2026-01-27)
Source: Testing Enhancements (Automation Turn #6)
Sprint: `docs/implplan/SPRINT_0127_002_DOCS_testing_enhancements_turn6.md`
### Intent Violation Gate
**Purpose:** Detect test changes that violate declared intent categories.
**Script:** `scripts/ci/check-intent-violations.sh`
| Check | Description | Action |
|-------|-------------|--------|
| Intent missing | Non-trivial test without Intent trait | Warning (regulatory modules: Error) |
| Intent contradiction | Test behavior contradicts declared intent | Error |
| Intent coverage drop | Module intent coverage decreased | Warning |
**Enforcement:**
- PR-gating for regulatory modules (Policy, Authority, Signer, Attestor, EvidenceLocker).
- Warning-only for other modules (to allow gradual adoption).
### Observability Contract Gate
**Purpose:** Validate OTel spans, structured logs, and metrics contracts.
**Script:** `scripts/ci/check-observability-contracts.sh`
| Check | Description | Threshold |
|-------|-------------|-----------|
| Required spans missing | Core operation spans not emitted | Error |
| Span attribute missing | Required attributes not present | Error |
| High cardinality attribute | Label cardinality exceeds limit | Warning (> 50), Error (> 100) |
| PII in logs | Sensitive data patterns in log output | Error |
| Missing log fields | Required fields not present | Warning |
**Enforcement:**
- PR-gating for all W1 (WebService) modules.
- Run as part of contract test lane.
### Evidence Chain Gate
**Purpose:** Verify requirement -> test -> artifact traceability.
**Script:** `scripts/ci/check-evidence-chain.sh`
| Check | Description | Action |
|-------|-------------|--------|
| Orphaned test | Regulatory test without Requirement attribute | Warning |
| Artifact hash drift | Artifact hash changed unexpectedly | Error |
| Artifact non-deterministic | Multiple runs produce different artifacts | Error |
| Traceability gap | Requirement without test coverage | Warning |
**Enforcement:**
- PR-gating for regulatory modules.
- Traceability report generated as CI artifact.
### Longevity Gate (Release Gating)
**Purpose:** Detect memory leaks, connection leaks, and counter drift under sustained load.
**Script:** `scripts/ci/run-longevity-gate.sh`
**Cadence:** Nightly + pre-release
| Metric | Description | Threshold |
|--------|-------------|-----------|
| Memory growth rate | Memory increase per hour | ≤ 1% |
| Connection pool leaks | Unreturned connections | 0 |
| Counter drift | Counter value outside expected range | Error |
| GC pressure | Gen2 collections per hour | ≤ 10 |
**Enforcement:**
- Not PR-gating (too slow).
- Release-gating: longevity tests must pass before release.
- Results stored for trend analysis.
### Interop Gate (Release Gating)
**Purpose:** Validate cross-version and environment compatibility.
**Script:** `scripts/ci/run-interop-gate.sh`
**Cadence:** Weekly + pre-release
| Check | Description | Threshold |
|-------|-------------|-----------|
| N-1 compatibility | Current server with previous client | Must pass |
| N+1 compatibility | Previous server with current client | Must pass |
| Environment equivalence | Same results across infra profiles | ≤ 5% deviation |
**Profiles Tested:**
- `standard`: default Testcontainers configuration.
- `high-latency`: +100ms network latency.
- `low-bandwidth`: 10 Mbps limit.
- `packet-loss`: 1% packet loss (Linux only).
**Enforcement:**
- Not PR-gating (requires multi-version infrastructure).
- Release-gating: interop tests must pass before release.
### Post-Incident Gate
**Purpose:** Ensure incident-derived tests are maintained and passing.
**Script:** `scripts/ci/check-post-incident-tests.sh`
| Check | Description | Action |
|-------|-------------|--------|
| Incident test failing | PostIncident test not passing | Error (P1/P2), Warning (P3) |
| Incident test missing metadata | Missing IncidentId or RootCause | Warning |
| Incident coverage | P1/P2 incidents without tests | Error |
**Enforcement:**
- PR-gating: P1/P2 incident tests must pass.
- Release-gating: all incident tests must pass.
---
## Gate Summary by Gating Level
### PR-Gating (Must Pass for Merge)
| Gate | Scope |
|------|-------|
| Reachability Quality | All |
| TTFS Regression | All |
| Intent Violation | Regulatory modules |
| Observability Contract | W1 modules |
| Evidence Chain | Regulatory modules |
| Post-Incident (P1/P2) | All |
### Release-Gating (Must Pass for Release)
| Gate | Scope |
|------|-------|
| All PR gates | All |
| Longevity | Worker modules |
| Interop | Schema/API-dependent modules |
| Post-Incident (all) | All |
| Performance SLO | All |
### Warning-Only (Informational)
| Gate | Scope |
|------|-------|
| Intent missing | Non-regulatory modules |
| Intent coverage drop | All |
| Orphaned test | All |
| Traceability gap | All |
---
## Related Documentation
- [Test Suite Overview](../TEST_SUITE_OVERVIEW.md)
@@ -155,3 +307,4 @@ If baselines become stale:
- [Reachability Corpus Plan](../reachability/corpus-plan.md)
- [Performance Workbook](../PERFORMANCE_WORKBOOK.md)
- [Testing Quality Guardrails](./testing-quality-guardrails-implementation.md)
- [Testing Practices](../../code-of-conduct/TESTING_PRACTICES.md)