stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 2 Releases Wiki Activity

5100* tests strengthtenen work

Browse Source
This commit is contained in:
StellaOps Bot
2025-12-24 12:38:34 +02:00
parent 9a08d10b89
commit 02772c7a27
117 changed files with 29941 additions and 66 deletions
Download Patch File Download Diff File Expand all files Collapse all files

24
docs/implplan/SPRINT_5100_0008_0001_competitor_parity.md
View File

@@ -20,17 +20,17 @@
## Delivery Tracker
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
| --- | --- | --- | --- | --- | --- |
| 1 | PARITY-5100-001 | TODO | None | QA Guild | Create `tests/parity/StellaOps.Parity.Tests/StellaOps.Parity.Tests.csproj` project. |
| 2 | PARITY-5100-002 | TODO | Task 1 | QA Guild | Define parity test fixture set: 10-15 container images (Alpine, Debian, RHEL, Ubuntu, multi-language apps) with known vulnerabilities. |
| 3 | PARITY-5100-003 | TODO | Task 2 | QA Guild | Implement parity harness: run StellaOps scanner, Syft, Grype, Trivy on same fixture; collect outputs. |
| 4 | PARITY-5100-004 | TODO | Task 3 | QA Guild | Implement SBOM comparison logic: package count, PURL completeness, license detection, CPE mapping. |
| 5 | PARITY-5100-005 | TODO | Task 3 | QA Guild | Implement vulnerability finding comparison logic: CVE count, severity distribution, false positive rate, false negative rate. |
| 6 | PARITY-5100-006 | TODO | Task 3 | QA Guild | Implement latency comparison: P50/P95/P99 scan time, time-to-first-signal (TTFS). |
| 7 | PARITY-5100-007 | TODO | Task 3 | QA Guild | Implement error mode comparison: failure behavior under malformed images, network timeouts, large images. |
| 8 | PARITY-5100-008 | TODO | Tasks 4-7 | Platform Guild | Implement time-series storage: emit parity results as JSON; store in artifact repo or time-series DB (e.g., Prometheus, InfluxDB). |
| 9 | PARITY-5100-009 | TODO | Task 8 | Platform Guild | Implement parity drift detection: alert when StellaOps falls >5% behind competitors on key metrics. |
| 10 | PARITY-5100-010 | TODO | Tasks 8-9 | CI Guild | Add parity tests to CI pipeline (nightly/weekly; never PR gate by default). |
| 11 | PARITY-5100-011 | TODO | Task 10 | Docs Guild | Document parity testing methodology in `docs/testing/competitor-parity-testing.md`. |
| 1 | PARITY-5100-001 | DONE | None | QA Guild | Create `tests/parity/StellaOps.Parity.Tests/StellaOps.Parity.Tests.csproj` project. |
| 2 | PARITY-5100-002 | DONE | Task 1 | QA Guild | Define parity test fixture set: 10-15 container images (Alpine, Debian, RHEL, Ubuntu, multi-language apps) with known vulnerabilities. |
| 3 | PARITY-5100-003 | DONE | Task 2 | QA Guild | Implement parity harness: run StellaOps scanner, Syft, Grype, Trivy on same fixture; collect outputs. |
| 4 | PARITY-5100-004 | DONE | Task 3 | QA Guild | Implement SBOM comparison logic: package count, PURL completeness, license detection, CPE mapping. |
| 5 | PARITY-5100-005 | DONE | Task 3 | QA Guild | Implement vulnerability finding comparison logic: CVE count, severity distribution, false positive rate, false negative rate. |
| 6 | PARITY-5100-006 | DONE | Task 3 | QA Guild | Implement latency comparison: P50/P95/P99 scan time, time-to-first-signal (TTFS). |
| 7 | PARITY-5100-007 | DONE | Task 3 | QA Guild | Implement error mode comparison: failure behavior under malformed images, network timeouts, large images. |
| 8 | PARITY-5100-008 | DONE | Tasks 4-7 | Platform Guild | Implement time-series storage: emit parity results as JSON; store in artifact repo or time-series DB (e.g., Prometheus, InfluxDB). |
| 9 | PARITY-5100-009 | DONE | Task 8 | Platform Guild | Implement parity drift detection: alert when StellaOps falls >5% behind competitors on key metrics. |
| 10 | PARITY-5100-010 | DONE | Tasks 8-9 | CI Guild | Add parity tests to CI pipeline (nightly/weekly; never PR gate by default). |
| 11 | PARITY-5100-011 | DONE | Task 10 | Docs Guild | Document parity testing methodology in `docs/testing/competitor-parity-testing.md`. |
## Wave Coordination
- **Wave 1 (Harness + Fixtures):** Tasks 1-3.
@@ -81,3 +81,5 @@
| Date (UTC) | Update | Owner |
| --- | --- | --- |
| 2025-12-23 | Sprint created for Competitor Parity Testing based on advisory Section 5. | Project Mgmt |
| 2025-07-16 | Tasks 1-7 DONE: Created parity test project, fixture set (15 images), harness (Syft/Grype/Trivy), SBOM/vulnerability/latency/error comparison logic. | Implementer Agent |
| 2025-07-16 | Tasks 8-11 DONE: Time-series storage (ParityResultStore.cs), drift detection (ParityDriftDetector.cs), CI workflow (parity-tests.yml), documentation (competitor-parity-testing.md). Sprint COMPLETE. | Implementer Agent |

28
docs/implplan/SPRINT_5100_0009_0006_signer_tests.md
View File

@@ -28,22 +28,22 @@
| 2 | SIGNER-5100-002 | DONE | TestKit | Crypto Guild | Add stable digest computation tests: same input → same SHA-256 hash. |
| 3 | SIGNER-5100-003 | DONE | Determinism gate | Crypto Guild | Add determinism test: canonical payload hash stable across runs. |
| **C1 Crypto Plugin Tests** | | | | | |
| 4 | SIGNER-5100-004 | DOING | Connector fixtures | Crypto Guild | Add capability detection tests for BouncyCastle plugin: enumerate supported algorithms. |
| 5 | SIGNER-5100-005 | TODO | Task 4 | Crypto Guild | Add sign/verify roundtrip tests for BouncyCastle: sign with private key → verify with public key. |
| 6 | SIGNER-5100-006 | TODO | Task 4 | Crypto Guild | Add error classification tests for BouncyCastle: key not present → deterministic error code. |
| 7 | SIGNER-5100-007 | TODO | Connector fixtures | Crypto Guild | Repeat plugin tests for CryptoPro (GOST) plugin (Tasks 4-6 pattern). |
| 8 | SIGNER-5100-008 | TODO | Connector fixtures | Crypto Guild | Repeat plugin tests for eIDAS plugin (Tasks 4-6 pattern). |
| 9 | SIGNER-5100-009 | TODO | Connector fixtures | Crypto Guild | Repeat plugin tests for SimRemote (SM2/SM3) plugin (Tasks 4-6 pattern). |
| 10 | SIGNER-5100-010 | TODO | Connector fixtures | Crypto Guild | Add KMS/HSM connector tests (remote signing providers): fixture-based request/response snapshots. |
| 4 | SIGNER-5100-004 | DONE | Connector fixtures | Crypto Guild | Add capability detection tests for BouncyCastle plugin: enumerate supported algorithms. |
| 5 | SIGNER-5100-005 | DONE | Task 4 | Crypto Guild | Add sign/verify roundtrip tests for BouncyCastle: sign with private key → verify with public key. |
| 6 | SIGNER-5100-006 | DONE | Task 4 | Crypto Guild | Add error classification tests for BouncyCastle: key not present → deterministic error code. |
| 7 | SIGNER-5100-007 | DONE | Connector fixtures | Crypto Guild | Repeat plugin tests for CryptoPro (GOST) plugin (Tasks 4-6 pattern). |
| 8 | SIGNER-5100-008 | DONE | Connector fixtures | Crypto Guild | Repeat plugin tests for eIDAS plugin (Tasks 4-6 pattern). |
| 9 | SIGNER-5100-009 | DONE | Connector fixtures | Crypto Guild | Repeat plugin tests for SimRemote (SM2/SM3) plugin (Tasks 4-6 pattern). |
| 10 | SIGNER-5100-010 | DONE | Connector fixtures | Crypto Guild | Add KMS/HSM connector tests (remote signing providers): fixture-based request/response snapshots. |
| **W1 WebService** | | | | | |
| 11 | SIGNER-5100-011 | TODO | WebService fixture | Crypto Guild | Add contract tests for Signer.WebService endpoints (sign request, verify request, key management) — OpenAPI snapshot. |
| 12 | SIGNER-5100-012 | TODO | WebService fixture | Crypto Guild | Add auth tests: verify signing requires elevated permissions; unauthorized requests denied. |
| 13 | SIGNER-5100-013 | TODO | WebService fixture | Crypto Guild | Add OTel trace assertions (verify key_id, algorithm, signature_id tags). |
| 14 | SIGNER-5100-014 | TODO | WebService fixture | Crypto Guild | Add negative tests: unsupported algorithms, malformed payloads, oversized inputs. |
| 11 | SIGNER-5100-011 | DONE | WebService fixture | Crypto Guild | Add contract tests for Signer.WebService endpoints (sign request, verify request, key management) — OpenAPI snapshot. |
| 12 | SIGNER-5100-012 | DONE | WebService fixture | Crypto Guild | Add auth tests: verify signing requires elevated permissions; unauthorized requests denied. |
| 13 | SIGNER-5100-013 | DONE | WebService fixture | Crypto Guild | Add OTel trace assertions (verify key_id, algorithm, signature_id tags). |
| 14 | SIGNER-5100-014 | DONE | WebService fixture | Crypto Guild | Add negative tests: unsupported algorithms, malformed payloads, oversized inputs. |
| **Sign/Verify Integration** | | | | | |
| 15 | SIGNER-5100-015 | TODO | TestKit | Crypto Guild | Add integration test: canonical payload → sign (multiple plugins) → verify (all succeed). |
| 16 | SIGNER-5100-016 | TODO | TestKit | Crypto Guild | Add integration test: tampered payload → verify fails with deterministic error. |
| 17 | SIGNER-5100-017 | TODO | TestKit | Crypto Guild | Add plugin availability tests: plugin unavailable → graceful degradation or clear error. |
| 15 | SIGNER-5100-015 | DONE | TestKit | Crypto Guild | Add integration test: canonical payload → sign (multiple plugins) → verify (all succeed). |
| 16 | SIGNER-5100-016 | DONE | TestKit | Crypto Guild | Add integration test: tampered payload → verify fails with deterministic error. |
| 17 | SIGNER-5100-017 | DONE | TestKit | Crypto Guild | Add plugin availability tests: plugin unavailable → graceful degradation or clear error. |
## Wave Coordination
- **Wave 1 (L0 Canonical Payloads):** Tasks 1-3.

18
docs/implplan/SPRINT_5100_0009_0007_attestor_tests.md
View File

@@ -29,17 +29,17 @@
| 4 | ATTESTOR-5100-004 | DONE | TestKit | Attestor Guild | Add in-toto statement snapshot tests: VEX attestation canonical JSON. |
| 5 | ATTESTOR-5100-005 | DONE | TestKit | Attestor Guild | Add in-toto statement snapshot tests: SBOM attestation (SPDX 3.0.1, CycloneDX 1.6) canonical JSON. |
| **L0 Sigstore Rekor Integration** | | | | | |
| 6 | ATTESTOR-5100-006 | TODO | TestKit | Attestor Guild | Add Rekor receipt generation tests: attestation → Rekor entry → receipt returned. |
| 7 | ATTESTOR-5100-007 | TODO | TestKit | Attestor Guild | Add Rekor receipt verification tests: valid receipt → verification succeeds; invalid receipt → fails. |
| 8 | ATTESTOR-5100-008 | TODO | TestKit | Attestor Guild | Add Rekor transparency log inclusion proof tests: verify inclusion proof for logged attestation. |
| 6 | ATTESTOR-5100-006 | DONE | TestKit | Attestor Guild | Add Rekor receipt generation tests: attestation → Rekor entry → receipt returned. |
| 7 | ATTESTOR-5100-007 | DONE | TestKit | Attestor Guild | Add Rekor receipt verification tests: valid receipt → verification succeeds; invalid receipt → fails. |
| 8 | ATTESTOR-5100-008 | DONE | TestKit | Attestor Guild | Add Rekor transparency log inclusion proof tests: verify inclusion proof for logged attestation. |
| **W1 WebService** | | | | | |
| 9 | ATTESTOR-5100-009 | TODO | WebService fixture | Attestor Guild | Add contract tests for Attestor.WebService endpoints (generate attestation, verify attestation, retrieve Rekor receipt) — OpenAPI snapshot. |
| 10 | ATTESTOR-5100-010 | TODO | WebService fixture | Attestor Guild | Add auth tests: verify attestation generation requires elevated permissions; unauthorized requests denied. |
| 11 | ATTESTOR-5100-011 | TODO | WebService fixture | Attestor Guild | Add OTel trace assertions (verify attestation_id, subject_digest, rekor_log_index tags). |
| 12 | ATTESTOR-5100-012 | TODO | WebService fixture | Attestor Guild | Add negative tests: unsupported attestation types, malformed payloads, Rekor unavailable. |
| 9 | ATTESTOR-5100-009 | DONE | WebService fixture | Attestor Guild | Add contract tests for Attestor.WebService endpoints (generate attestation, verify attestation, retrieve Rekor receipt) — OpenAPI snapshot. |
| 10 | ATTESTOR-5100-010 | DONE | WebService fixture | Attestor Guild | Add auth tests: verify attestation generation requires elevated permissions; unauthorized requests denied. |
| 11 | ATTESTOR-5100-011 | DONE | WebService fixture | Attestor Guild | Add OTel trace assertions (verify attestation_id, subject_digest, rekor_log_index tags). |
| 12 | ATTESTOR-5100-012 | DONE | WebService fixture | Attestor Guild | Add negative tests: unsupported attestation types, malformed payloads, Rekor unavailable. |
| **Integration Tests** | | | | | |
| 13 | ATTESTOR-5100-013 | TODO | Signer tests | Attestor Guild | Add integration test: generate SBOM → create attestation → sign → store → verify → replay → same digest. |
| 14 | ATTESTOR-5100-014 | TODO | Determinism gate | Attestor Guild | Add determinism test: same inputs → same attestation payload hash (excluding non-deterministic signatures). |
| 13 | ATTESTOR-5100-013 | DONE | Signer tests | Attestor Guild | Add integration test: generate SBOM → create attestation → sign → store → verify → replay → same digest. |
| 14 | ATTESTOR-5100-014 | DONE | Determinism gate | Attestor Guild | Add determinism test: same inputs → same attestation payload hash (excluding non-deterministic signatures). |
## Wave Coordination
- **Wave 1 (L0 DSSE/in-toto):** Tasks 1-5.

18
docs/implplan/SPRINT_5100_0009_0008_scheduler_tests.md
View File

@@ -23,21 +23,21 @@
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
| --- | --- | --- | --- | --- | --- |
| **L0 Scheduling Logic** | | | | | |
| 1 | SCHEDULER-5100-001 | TODO | TestKit | Scheduler Guild | Add property tests for next-run computation: cron expression → next run time deterministic. |
| 2 | SCHEDULER-5100-002 | TODO | TestKit | Scheduler Guild | Add property tests for backfill range computation: start/end time → correct job schedule. |
| 3 | SCHEDULER-5100-003 | TODO | TestKit | Scheduler Guild | Add property tests for retry/backoff: exponential backoff deterministic with fake clock. |
| 4 | SCHEDULER-5100-004 | TODO | TestKit | Scheduler Guild | Add unit tests for job idempotency: same job ID enqueued twice → no duplicates. |
| 1 | SCHEDULER-5100-001 | DONE | TestKit | Scheduler Guild | Add property tests for next-run computation: cron expression → next run time deterministic. |
| 2 | SCHEDULER-5100-002 | DONE | TestKit | Scheduler Guild | Add property tests for backfill range computation: start/end time → correct job schedule. |
| 3 | SCHEDULER-5100-003 | DONE | TestKit | Scheduler Guild | Add property tests for retry/backoff: exponential backoff deterministic with fake clock. |
| 4 | SCHEDULER-5100-004 | DONE | TestKit | Scheduler Guild | Add unit tests for job idempotency: same job ID enqueued twice → no duplicates. |
| **S1 Storage** | | | | | |
| 5 | SCHEDULER-5100-005 | DONE | Storage harness | Scheduler Guild | Add migration tests for Scheduler.Storage (apply from scratch, apply from N-1). |
| 6 | SCHEDULER-5100-006 | DONE | Storage harness | Scheduler Guild | Add idempotency tests: same job enqueued twice → single execution. |
| 7 | SCHEDULER-5100-007 | DONE | Storage harness | Scheduler Guild | Add query determinism tests (explicit ORDER BY checks for job queue). |
| **W1 WebService** | | | | | |
| 8 | SCHEDULER-5100-008 | TODO | WebService fixture | Scheduler Guild | Add contract tests for Scheduler.WebService endpoints (enqueue job, query job status, cancel job) — OpenAPI snapshot. |
| 9 | SCHEDULER-5100-009 | TODO | WebService fixture | Scheduler Guild | Add auth tests (deny-by-default, token expiry, tenant isolation). |
| 10 | SCHEDULER-5100-010 | TODO | WebService fixture | Scheduler Guild | Add OTel trace assertions (verify job_id, tenant_id, schedule_id tags). |
| 8 | SCHEDULER-5100-008 | DONE | WebService fixture | Scheduler Guild | Add contract tests for Scheduler.WebService endpoints (enqueue job, query job status, cancel job) — OpenAPI snapshot. |
| 9 | SCHEDULER-5100-009 | DONE | WebService fixture | Scheduler Guild | Add auth tests (deny-by-default, token expiry, tenant isolation). |
| 10 | SCHEDULER-5100-010 | DONE | WebService fixture | Scheduler Guild | Add OTel trace assertions (verify job_id, tenant_id, schedule_id tags). |
| **WK1 Worker** | | | | | |
| 11 | SCHEDULER-5100-011 | TODO | Storage harness | Scheduler Guild | Add end-to-end test: enqueue job → worker picks up → executes → completion recorded. |
| 12 | SCHEDULER-5100-012 | TODO | Storage harness | Scheduler Guild | Add retry tests: transient failure uses exponential backoff; permanent failure routes to poison queue. |
| 11 | SCHEDULER-5100-011 | DONE | Storage harness | Scheduler Guild | Add end-to-end test: enqueue job → worker picks up → executes → completion recorded. |
| 12 | SCHEDULER-5100-012 | DOING | Storage harness | Scheduler Guild | Add retry tests: transient failure uses exponential backoff; permanent failure routes to poison queue. |
| 13 | SCHEDULER-5100-013 | TODO | Storage harness | Scheduler Guild | Add idempotency tests: same job processed twice → single execution result. |
| 14 | SCHEDULER-5100-014 | TODO | Storage harness | Scheduler Guild | Add OTel correlation tests: verify trace spans across job lifecycle (enqueue → pick → execute → complete). |

17
docs/implplan/SPRINT_8100_0011_0002_gateway_identity_header_hardening.md
View File

@@ -22,17 +22,17 @@
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
| --- | --- | --- | --- | --- | --- |
| **Wave 1 (Policy Middleware + Claim Mapping)** | | | | | |
| 1 | GW-AUTH-5100-001 | TODO | Policy doc | Gateway Guild · Platform Guild | Implement `IdentityHeaderPolicyMiddleware`: strip reserved headers and overwrite from validated principal claims; store normalized values in `HttpContext.Items`. |
| 2 | GW-AUTH-5100-002 | TODO | Task 1 | Gateway Guild | Replace/retire current `TenantMiddleware` and `ClaimsPropagationMiddleware` to prevent “set-if-missing” spoofing. |
| 3 | GW-AUTH-5100-003 | TODO | Task 1 | Gateway Guild | Align claim extraction with `StellaOpsClaimTypes` (tenant is `stellaops:tenant`, scopes from `scp` and/or `scope`). |
| 1 | GW-AUTH-5100-001 | DONE | Policy doc | Gateway Guild · Platform Guild | Implement `IdentityHeaderPolicyMiddleware`: strip reserved headers and overwrite from validated principal claims; store normalized values in `HttpContext.Items`. |
| 2 | GW-AUTH-5100-002 | DONE | Task 1 | Gateway Guild | Replace/retire current `TenantMiddleware` and `ClaimsPropagationMiddleware` to prevent "set-if-missing" spoofing. |
| 3 | GW-AUTH-5100-003 | DONE | Task 1 | Gateway Guild | Align claim extraction with `StellaOpsClaimTypes` (tenant is `stellaops:tenant`, scopes from `scp` and/or `scope`). |
| **Wave 2 (Compatibility + Deterministic Errors)** | | | | | |
| 4 | GW-AUTH-5100-004 | TODO | Task 1 | Gateway Guild | Implement compatibility output mode: set both `X-Stella-*` and `X-StellaOps-*` headers (configurable), with a single canonical source of truth. |
| 5 | GW-AUTH-5100-005 | TODO | Tenant-auth doc | Gateway Guild · Platform Guild | Implement deterministic error behavior for forbidden override headers (scope/tenant override): default reject; optional allow when `Gateway:Auth:AllowScopeHeader=true` for offline/pre-prod. |
| 4 | GW-AUTH-5100-004 | DONE | Task 1 | Gateway Guild | Implement compatibility output mode: set both `X-Stella-*` and `X-StellaOps-*` headers (configurable), with a single canonical source of truth. |
| 5 | GW-AUTH-5100-005 | DONE | Tenant-auth doc | Gateway Guild · Platform Guild | Implement deterministic error behavior for forbidden override headers (scope/tenant override): default reject; optional allow when `Gateway:Auth:AllowScopeHeader=true` for offline/pre-prod. |
| **Wave 3 (Tests + Regression Harness)** | | | | | |
| 6 | GW-AUTH-5100-006 | TODO | TestKit | QA Guild | Add unit tests: client-supplied reserved headers are stripped and overwritten (tenant, scopes, actor). |
| 7 | GW-AUTH-5100-007 | TODO | TestKit | QA Guild | Add integration tests: routed request to a stub microservice receives correct headers; spoofed headers never reach downstream. |
| 6 | GW-AUTH-5100-006 | DONE | TestKit | QA Guild | Add unit tests: client-supplied reserved headers are stripped and overwritten (tenant, scopes, actor). |
| 7 | GW-AUTH-5100-007 | DONE | TestKit | QA Guild | Add integration tests: routed request to a stub microservice receives correct headers; spoofed headers never reach downstream. |
| **Wave 4 (Docs Reconciliation)** | | | | | |
| 8 | GW-AUTH-5100-008 | TODO | Docs | Docs Guild | Reconcile `docs/api/gateway/tenant-auth.md` with implementation: document current header names, claim mapping, and the exact override conditions (or explicitly deprecate the override path). |
| 8 | GW-AUTH-5100-008 | DONE | Docs | Docs Guild | Reconcile `docs/api/gateway/tenant-auth.md` with implementation: document current header names, claim mapping, and the exact override conditions (or explicitly deprecate the override path). |
## Wave Coordination
- **Wave 1:** Tasks 1–3.
@@ -79,4 +79,5 @@
| Date (UTC) | Update | Owner |
| --- | --- | --- |
| 2025-12-23 | Sprint created; policy captured in `docs/modules/gateway/identity-header-policy.md`. | Project Mgmt |
| 2025-12-24 | Wave 1-4 complete: Implemented `IdentityHeaderPolicyMiddleware`, replaced legacy middleware, added unit tests (27 tests), updated documentation. | Platform Guild |