test fixes and new product advisories work
This commit is contained in:
@@ -147,6 +147,158 @@ If baselines become stale:
|
||||
./scripts/ci/compute-reachability-metrics.sh --update-baseline
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Turn #6 Quality Gates (2026-01-27)
|
||||
|
||||
Source: Testing Enhancements (Automation Turn #6)
|
||||
Sprint: `docs/implplan/SPRINT_0127_002_DOCS_testing_enhancements_turn6.md`
|
||||
|
||||
### Intent Violation Gate
|
||||
|
||||
**Purpose:** Detect test changes that violate declared intent categories.
|
||||
|
||||
**Script:** `scripts/ci/check-intent-violations.sh`
|
||||
|
||||
| Check | Description | Action |
|
||||
|-------|-------------|--------|
|
||||
| Intent missing | Non-trivial test without Intent trait | Warning (regulatory modules: Error) |
|
||||
| Intent contradiction | Test behavior contradicts declared intent | Error |
|
||||
| Intent coverage drop | Module intent coverage decreased | Warning |
|
||||
|
||||
**Enforcement:**
|
||||
- PR-gating for regulatory modules (Policy, Authority, Signer, Attestor, EvidenceLocker).
|
||||
- Warning-only for other modules (to allow gradual adoption).
|
||||
|
||||
### Observability Contract Gate
|
||||
|
||||
**Purpose:** Validate OTel spans, structured logs, and metrics contracts.
|
||||
|
||||
**Script:** `scripts/ci/check-observability-contracts.sh`
|
||||
|
||||
| Check | Description | Threshold |
|
||||
|-------|-------------|-----------|
|
||||
| Required spans missing | Core operation spans not emitted | Error |
|
||||
| Span attribute missing | Required attributes not present | Error |
|
||||
| High cardinality attribute | Label cardinality exceeds limit | Warning (> 50), Error (> 100) |
|
||||
| PII in logs | Sensitive data patterns in log output | Error |
|
||||
| Missing log fields | Required fields not present | Warning |
|
||||
|
||||
**Enforcement:**
|
||||
- PR-gating for all W1 (WebService) modules.
|
||||
- Run as part of contract test lane.
|
||||
|
||||
### Evidence Chain Gate
|
||||
|
||||
**Purpose:** Verify requirement -> test -> artifact traceability.
|
||||
|
||||
**Script:** `scripts/ci/check-evidence-chain.sh`
|
||||
|
||||
| Check | Description | Action |
|
||||
|-------|-------------|--------|
|
||||
| Orphaned test | Regulatory test without Requirement attribute | Warning |
|
||||
| Artifact hash drift | Artifact hash changed unexpectedly | Error |
|
||||
| Artifact non-deterministic | Multiple runs produce different artifacts | Error |
|
||||
| Traceability gap | Requirement without test coverage | Warning |
|
||||
|
||||
**Enforcement:**
|
||||
- PR-gating for regulatory modules.
|
||||
- Traceability report generated as CI artifact.
|
||||
|
||||
### Longevity Gate (Release Gating)
|
||||
|
||||
**Purpose:** Detect memory leaks, connection leaks, and counter drift under sustained load.
|
||||
|
||||
**Script:** `scripts/ci/run-longevity-gate.sh`
|
||||
**Cadence:** Nightly + pre-release
|
||||
|
||||
| Metric | Description | Threshold |
|
||||
|--------|-------------|-----------|
|
||||
| Memory growth rate | Memory increase per hour | ≤ 1% |
|
||||
| Connection pool leaks | Unreturned connections | 0 |
|
||||
| Counter drift | Counter value outside expected range | Error |
|
||||
| GC pressure | Gen2 collections per hour | ≤ 10 |
|
||||
|
||||
**Enforcement:**
|
||||
- Not PR-gating (too slow).
|
||||
- Release-gating: longevity tests must pass before release.
|
||||
- Results stored for trend analysis.
|
||||
|
||||
### Interop Gate (Release Gating)
|
||||
|
||||
**Purpose:** Validate cross-version and environment compatibility.
|
||||
|
||||
**Script:** `scripts/ci/run-interop-gate.sh`
|
||||
**Cadence:** Weekly + pre-release
|
||||
|
||||
| Check | Description | Threshold |
|
||||
|-------|-------------|-----------|
|
||||
| N-1 compatibility | Current server with previous client | Must pass |
|
||||
| N+1 compatibility | Previous server with current client | Must pass |
|
||||
| Environment equivalence | Same results across infra profiles | ≤ 5% deviation |
|
||||
|
||||
**Profiles Tested:**
|
||||
- `standard`: default Testcontainers configuration.
|
||||
- `high-latency`: +100ms network latency.
|
||||
- `low-bandwidth`: 10 Mbps limit.
|
||||
- `packet-loss`: 1% packet loss (Linux only).
|
||||
|
||||
**Enforcement:**
|
||||
- Not PR-gating (requires multi-version infrastructure).
|
||||
- Release-gating: interop tests must pass before release.
|
||||
|
||||
### Post-Incident Gate
|
||||
|
||||
**Purpose:** Ensure incident-derived tests are maintained and passing.
|
||||
|
||||
**Script:** `scripts/ci/check-post-incident-tests.sh`
|
||||
|
||||
| Check | Description | Action |
|
||||
|-------|-------------|--------|
|
||||
| Incident test failing | PostIncident test not passing | Error (P1/P2), Warning (P3) |
|
||||
| Incident test missing metadata | Missing IncidentId or RootCause | Warning |
|
||||
| Incident coverage | P1/P2 incidents without tests | Error |
|
||||
|
||||
**Enforcement:**
|
||||
- PR-gating: P1/P2 incident tests must pass.
|
||||
- Release-gating: all incident tests must pass.
|
||||
|
||||
---
|
||||
|
||||
## Gate Summary by Gating Level
|
||||
|
||||
### PR-Gating (Must Pass for Merge)
|
||||
|
||||
| Gate | Scope |
|
||||
|------|-------|
|
||||
| Reachability Quality | All |
|
||||
| TTFS Regression | All |
|
||||
| Intent Violation | Regulatory modules |
|
||||
| Observability Contract | W1 modules |
|
||||
| Evidence Chain | Regulatory modules |
|
||||
| Post-Incident (P1/P2) | All |
|
||||
|
||||
### Release-Gating (Must Pass for Release)
|
||||
|
||||
| Gate | Scope |
|
||||
|------|-------|
|
||||
| All PR gates | All |
|
||||
| Longevity | Worker modules |
|
||||
| Interop | Schema/API-dependent modules |
|
||||
| Post-Incident (all) | All |
|
||||
| Performance SLO | All |
|
||||
|
||||
### Warning-Only (Informational)
|
||||
|
||||
| Gate | Scope |
|
||||
|------|-------|
|
||||
| Intent missing | Non-regulatory modules |
|
||||
| Intent coverage drop | All |
|
||||
| Orphaned test | All |
|
||||
| Traceability gap | All |
|
||||
|
||||
---
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Test Suite Overview](../TEST_SUITE_OVERVIEW.md)
|
||||
@@ -155,3 +307,4 @@ If baselines become stale:
|
||||
- [Reachability Corpus Plan](../reachability/corpus-plan.md)
|
||||
- [Performance Workbook](../PERFORMANCE_WORKBOOK.md)
|
||||
- [Testing Quality Guardrails](./testing-quality-guardrails-implementation.md)
|
||||
- [Testing Practices](../../code-of-conduct/TESTING_PRACTICES.md)
|
||||
|
||||
Reference in New Issue
Block a user