Add tests for SBOM generation determinism across multiple formats
- Created `StellaOps.TestKit.Tests` project for unit tests related to determinism. - Implemented `DeterminismManifestTests` to validate deterministic output for canonical bytes and strings, file read/write operations, and error handling for invalid schema versions. - Added `SbomDeterminismTests` to ensure identical inputs produce consistent SBOMs across SPDX 3.0.1 and CycloneDX 1.6/1.7 formats, including parallel execution tests. - Updated project references in `StellaOps.Integration.Determinism` to include the new determinism testing library.
This commit is contained in:
master
StellaOps Bot
230
docs/testing/webservice-test-rollout-plan.md
Normal file
230
docs/testing/webservice-test-rollout-plan.md
Normal file
@@ -0,0 +1,230 @@
|
||||
# WebService Test Rollout Plan
|
||||
|
||||
This document defines the rollout plan for applying the WebService test discipline to all StellaOps web services.
|
||||
|
||||
## Overview
|
||||
|
||||
Following the pilot implementation on Scanner.WebService (Sprint 5100.0007.0006), this plan defines the order and timeline for rolling out comprehensive WebService tests to all remaining services.
|
||||
|
||||
---
|
||||
|
||||
## Service Inventory
|
||||
|
||||
| Service | Module Path | Priority | Status | Sprint |
|
||||
|---------|-------------|----------|--------|--------|
|
||||
| Scanner.WebService | `src/Scanner/StellaOps.Scanner.WebService` | P0 (Pilot) | ✅ Existing tests | 5100.0007.0006 |
|
||||
| Concelier.WebService | `src/Concelier/StellaOps.Concelier.WebService` | P1 | Pending | TBD |
|
||||
| Excititor.WebService | `src/Excititor/StellaOps.Excititor.WebService` | P1 | Pending | TBD |
|
||||
| Policy.Engine | `src/Policy/StellaOps.Policy.Engine` | P1 | Pending | TBD |
|
||||
| Scheduler.WebService | `src/Scheduler/StellaOps.Scheduler.WebService` | P2 | Pending | TBD |
|
||||
| Notify.WebService | `src/Notify/StellaOps.Notify.WebService` | P2 | Pending | TBD |
|
||||
| Authority | `src/Authority/StellaOps.Authority` | P2 | Pending | TBD |
|
||||
| Signer | `src/Signer/StellaOps.Signer` | P3 | Pending | TBD |
|
||||
| Attestor | `src/Attestor/StellaOps.Attestor` | P3 | Pending | TBD |
|
||||
| ExportCenter.WebService | `src/ExportCenter/StellaOps.ExportCenter.WebService` | P3 | Pending | TBD |
|
||||
| Registry.TokenService | `src/Registry/StellaOps.Registry.TokenService` | P3 | Pending | TBD |
|
||||
| VulnExplorer.Api | `src/VulnExplorer/StellaOps.VulnExplorer.Api` | P3 | Pending | TBD |
|
||||
| Graph.Api | `src/Graph/StellaOps.Graph.Api` | P3 | Pending | TBD |
|
||||
| Orchestrator | `src/Orchestrator/StellaOps.Orchestrator` | P4 | Pending | TBD |
|
||||
|
||||
---
|
||||
|
||||
## Rollout Phases
|
||||
|
||||
### Phase 1: Core Data Flow Services (P1)
|
||||
|
||||
**Timeline**: Sprint 5100.0008.* (Q1 2026)
|
||||
|
||||
**Services**:
|
||||
- **Concelier.WebService** — Primary advisory ingestion service
|
||||
- **Excititor.WebService** — Enrichment and correlation service
|
||||
- **Policy.Engine** — Policy evaluation service
|
||||
|
||||
**Rationale**: These services form the core data flow pipeline. They have high traffic, complex contracts, and critical security boundaries.
|
||||
|
||||
**Test Requirements**:
|
||||
| Test Type | Concelier | Excititor | Policy |
|
||||
|-----------|-----------|-----------|--------|
|
||||
| Contract (OpenAPI) | Required | Required | Required |
|
||||
| OTel traces | Required | Required | Required |
|
||||
| Negative tests | Required | Required | Required |
|
||||
| Auth/AuthZ | Required | Required | Required |
|
||||
| Tenant isolation | Required | Required | Required |
|
||||
|
||||
---
|
||||
|
||||
### Phase 2: Scheduling & Notification Services (P2)
|
||||
|
||||
**Timeline**: Sprint 5100.0009.* (Q2 2026)
|
||||
|
||||
**Services**:
|
||||
- **Scheduler.WebService** — Job scheduling and orchestration
|
||||
- **Notify.WebService** — Notification dispatch
|
||||
- **Authority** — Authentication/authorization service
|
||||
|
||||
**Rationale**: These services support operational workflows. Authority is critical for security testing of all other services.
|
||||
|
||||
**Test Requirements**:
|
||||
| Test Type | Scheduler | Notify | Authority |
|
||||
|-----------|-----------|--------|-----------|
|
||||
| Contract (OpenAPI) | Required | Required | Required |
|
||||
| OTel traces | Required | Required | Required |
|
||||
| Negative tests | Required | Required | Required |
|
||||
| Auth/AuthZ | N/A (system) | Required | N/A (self) |
|
||||
| Token issuance | N/A | N/A | Required |
|
||||
|
||||
---
|
||||
|
||||
### Phase 3: Signing & Attestation Services (P3)
|
||||
|
||||
**Timeline**: Sprint 5100.0010.* (Q2-Q3 2026)
|
||||
|
||||
**Services**:
|
||||
- **Signer** — Cryptographic signing service
|
||||
- **Attestor** — Attestation generation/verification
|
||||
- **ExportCenter.WebService** — Report export service
|
||||
- **Registry.TokenService** — OCI registry token service
|
||||
- **VulnExplorer.Api** — Vulnerability exploration API
|
||||
- **Graph.Api** — Graph query API
|
||||
|
||||
**Rationale**: These services have specialized contracts and lower traffic. They require careful security testing due to cryptographic operations.
|
||||
|
||||
**Test Requirements**:
|
||||
| Test Type | Signer | Attestor | Others |
|
||||
|-----------|--------|----------|--------|
|
||||
| Contract (OpenAPI) | Required | Required | Required |
|
||||
| OTel traces | Required | Required | Required |
|
||||
| Negative tests | Required | Required | Required |
|
||||
| Crypto validation | Required | Required | N/A |
|
||||
|
||||
---
|
||||
|
||||
### Phase 4: Orchestration Services (P4)
|
||||
|
||||
**Timeline**: Sprint 5100.0011.* (Q3 2026)
|
||||
|
||||
**Services**:
|
||||
- **Orchestrator** — Workflow orchestration
|
||||
|
||||
**Rationale**: Orchestrator is a meta-service that coordinates other services. Testing depends on other services being testable first.
|
||||
|
||||
---
|
||||
|
||||
## Test Coverage Targets
|
||||
|
||||
### Minimum Requirements (PR-Gating)
|
||||
|
||||
| Test Category | Min Coverage | Lane |
|
||||
|---------------|-------------|------|
|
||||
| Contract (OpenAPI) | 100% of public endpoints | Contract |
|
||||
| Negative (4xx errors) | 100% of error codes | Unit/Security |
|
||||
| Auth/AuthZ | 100% of protected endpoints | Security |
|
||||
|
||||
### Recommended (Quality Gate)
|
||||
|
||||
| Test Category | Target Coverage | Lane |
|
||||
|---------------|-----------------|------|
|
||||
| OTel traces | 80% of endpoints | Integration |
|
||||
| Tenant isolation | 100% of data endpoints | Security |
|
||||
| Performance baselines | Key endpoints | Performance |
|
||||
|
||||
---
|
||||
|
||||
## Implementation Checklist per Service
|
||||
|
||||
```markdown
|
||||
## <Service Name> WebService Tests
|
||||
|
||||
### Setup
|
||||
- [ ] Create `<Service>ApplicationFactory` (WebApplicationFactory)
|
||||
- [ ] Create `<Service>TestFixture` if custom setup needed
|
||||
- [ ] Add test project: `StellaOps.<Service>.WebService.Tests`
|
||||
- [ ] Add reference to `StellaOps.TestKit`
|
||||
|
||||
### Contract Tests
|
||||
- [ ] Extract OpenAPI schema snapshot (`Snapshots/<service>-v1.json`)
|
||||
- [ ] Add schema stability test
|
||||
- [ ] Add response contract tests for key endpoints
|
||||
|
||||
### OTel Tests
|
||||
- [ ] Add trace assertion tests for key endpoints
|
||||
- [ ] Verify required tags (tenant.id, request.id, http.route)
|
||||
|
||||
### Negative Tests
|
||||
- [ ] Malformed content type → 415
|
||||
- [ ] Oversized payload → 413
|
||||
- [ ] Method mismatch → 405
|
||||
- [ ] Missing required field → 400
|
||||
- [ ] Invalid field value → 400
|
||||
|
||||
### Auth Tests
|
||||
- [ ] Anonymous request → 401
|
||||
- [ ] Expired token → 401
|
||||
- [ ] Missing scope → 403
|
||||
- [ ] Cross-tenant access → 404/403
|
||||
|
||||
### CI Integration
|
||||
- [ ] Verify traits assigned: Contract, Security, Integration, Unit
|
||||
- [ ] PR passes all lanes
|
||||
- [ ] Add to TEST_COVERAGE_MATRIX.md
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Sprint Planning Template
|
||||
|
||||
When creating sprints for new service tests:
|
||||
|
||||
```markdown
|
||||
# Sprint 5100.XXXX.YYYY - <Service> WebService Tests
|
||||
|
||||
## Topic & Scope
|
||||
- Apply WebService test discipline to <Service>.WebService
|
||||
- Contract tests, OTel traces, negative tests, auth tests
|
||||
- **Working directory:** `src/<Module>/__Tests/StellaOps.<Module>.WebService.Tests`
|
||||
|
||||
## Delivery Tracker
|
||||
| # | Task ID | Status | Task Definition |
|
||||
|---|---------|--------|-----------------|
|
||||
| 1 | WEBSVC-XXXX-001 | TODO | Create <Service>ApplicationFactory |
|
||||
| 2 | WEBSVC-XXXX-002 | TODO | Add OpenAPI contract tests |
|
||||
| 3 | WEBSVC-XXXX-003 | TODO | Add OTel trace tests |
|
||||
| 4 | WEBSVC-XXXX-004 | TODO | Add negative tests (4xx) |
|
||||
| 5 | WEBSVC-XXXX-005 | TODO | Add auth/authz tests |
|
||||
| 6 | WEBSVC-XXXX-006 | TODO | Update TEST_COVERAGE_MATRIX.md |
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Success Metrics
|
||||
|
||||
| Metric | Target | Measurement |
|
||||
|--------|--------|-------------|
|
||||
| Services with contract tests | 100% | Count of services with OpenAPI snapshot tests |
|
||||
| Services with auth tests | 100% | Count of services with auth boundary tests |
|
||||
| Contract test failures in production | 0 | Breaking changes detected in staging |
|
||||
| Security test coverage | 100% of auth endpoints | Audit of protected routes vs tests |
|
||||
|
||||
---
|
||||
|
||||
## Risks & Mitigations
|
||||
|
||||
| Risk | Impact | Mitigation |
|
||||
|------|--------|------------|
|
||||
| Services lack OpenAPI spec | Cannot do contract testing | Generate spec via Swashbuckle/NSwag |
|
||||
| OTel not configured in service | Cannot verify traces | Add OTel middleware as prerequisite |
|
||||
| Auth disabled in test mode | False confidence | Test with auth enabled, use test tokens |
|
||||
| Test fixtures are slow | CI timeout | Share fixtures, use in-memory providers |
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- [WebService Test Discipline](./webservice-test-discipline.md)
|
||||
- [Test Coverage Matrix](./TEST_COVERAGE_MATRIX.md)
|
||||
- [CI Lane Filters](./ci-lane-filters.md)
|
||||
- [Testing Strategy Models](./testing-strategy-models.md)
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2025-06-30 · Sprint 5100.0007.0006*
|
||||
Reference in New Issue
Block a user