Add tests for SBOM generation determinism across multiple formats
- Created `StellaOps.TestKit.Tests` project for unit tests related to determinism. - Implemented `DeterminismManifestTests` to validate deterministic output for canonical bytes and strings, file read/write operations, and error handling for invalid schema versions. - Added `SbomDeterminismTests` to ensure identical inputs produce consistent SBOMs across SPDX 3.0.1 and CycloneDX 1.6/1.7 formats, including parallel execution tests. - Updated project references in `StellaOps.Integration.Determinism` to include the new determinism testing library.
This commit is contained in:
master
StellaOps Bot
366
docs/testing/webservice-test-discipline.md
Normal file
366
docs/testing/webservice-test-discipline.md
Normal file
@@ -0,0 +1,366 @@
|
||||
# WebService Test Discipline
|
||||
|
||||
This document defines the testing discipline for StellaOps WebService projects. All web services must follow these patterns to ensure consistent test coverage, contract stability, telemetry verification, and security hardening.
|
||||
|
||||
## Overview
|
||||
|
||||
WebService tests use `WebServiceFixture<TProgram>` from `StellaOps.TestKit` and `WebApplicationFactory<TProgram>` from `Microsoft.AspNetCore.Mvc.Testing`. Tests are organized into four categories:
|
||||
|
||||
1. **Contract Tests** — OpenAPI schema stability
|
||||
2. **OTel Trace Tests** — Telemetry verification
|
||||
3. **Negative Tests** — Error handling validation
|
||||
4. **Auth/AuthZ Tests** — Security boundary enforcement
|
||||
|
||||
---
|
||||
|
||||
## 1. Test Infrastructure
|
||||
|
||||
### WebServiceFixture Pattern
|
||||
|
||||
```csharp
|
||||
using StellaOps.TestKit.Fixtures;
|
||||
|
||||
public class ScannerWebServiceTests : WebServiceTestBase<ScannerProgram>
|
||||
{
|
||||
public ScannerWebServiceTests() : base(new WebServiceFixture<ScannerProgram>())
|
||||
{
|
||||
}
|
||||
|
||||
// Tests inherit shared fixture setup
|
||||
}
|
||||
```
|
||||
|
||||
### Fixture Configuration
|
||||
|
||||
Each web service should have a dedicated fixture class that configures test-specific settings:
|
||||
|
||||
```csharp
|
||||
public sealed class ScannerTestFixture : WebServiceFixture<ScannerProgram>
|
||||
{
|
||||
protected override void ConfigureTestServices(IServiceCollection services)
|
||||
{
|
||||
// Replace external dependencies with test doubles
|
||||
services.AddSingleton<IStorageClient, InMemoryStorageClient>();
|
||||
services.AddSingleton<IQueueClient, InMemoryQueueClient>();
|
||||
}
|
||||
|
||||
protected override void ConfigureTestConfiguration(IDictionary<string, string?> config)
|
||||
{
|
||||
config["scanner:storage:driver"] = "inmemory";
|
||||
config["scanner:events:enabled"] = "false";
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Contract Tests
|
||||
|
||||
Contract tests ensure OpenAPI schema stability and detect breaking changes.
|
||||
|
||||
### Pattern
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
[Trait("Lane", "Contract")]
|
||||
public async Task OpenApi_Schema_MatchesSnapshot()
|
||||
{
|
||||
// Arrange
|
||||
using var client = Fixture.CreateClient();
|
||||
|
||||
// Act
|
||||
var response = await client.GetAsync("/swagger/v1/swagger.json");
|
||||
var schema = await response.Content.ReadAsStringAsync();
|
||||
|
||||
// Assert
|
||||
await ContractTestHelper.AssertSchemaMatchesSnapshot(schema, "scanner-v1");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
[Trait("Lane", "Contract")]
|
||||
public async Task Api_Response_MatchesContract()
|
||||
{
|
||||
// Arrange
|
||||
using var client = Fixture.CreateClient();
|
||||
var request = new ScanRequest { /* test data */ };
|
||||
|
||||
// Act
|
||||
var response = await client.PostAsJsonAsync("/api/v1/scans", request);
|
||||
var result = await response.Content.ReadFromJsonAsync<ScanResponse>();
|
||||
|
||||
// Assert
|
||||
ContractTestHelper.AssertResponseMatchesSchema(result, "ScanResponse");
|
||||
}
|
||||
```
|
||||
|
||||
### Snapshot Management
|
||||
|
||||
- Snapshots stored in `Snapshots/` directory relative to test project
|
||||
- Schema format: `<service>-<version>.json`
|
||||
- Update snapshots intentionally when breaking changes are approved
|
||||
|
||||
---
|
||||
|
||||
## 3. OTel Trace Tests
|
||||
|
||||
OTel tests verify that telemetry spans are emitted correctly with required tags.
|
||||
|
||||
### Pattern
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
[Trait("Lane", "Integration")]
|
||||
public async Task ScanEndpoint_EmitsOtelTrace()
|
||||
{
|
||||
// Arrange
|
||||
using var otelCapture = Fixture.CaptureOtelTraces();
|
||||
using var client = Fixture.CreateClient();
|
||||
var request = new ScanRequest { ImageRef = "nginx:1.25" };
|
||||
|
||||
// Act
|
||||
await client.PostAsJsonAsync("/api/v1/scans", request);
|
||||
|
||||
// Assert
|
||||
otelCapture.AssertHasSpan("scanner.scan");
|
||||
otelCapture.AssertHasTag("scanner.scan", "scan.image_ref", "nginx:1.25");
|
||||
otelCapture.AssertHasTag("scanner.scan", "tenant.id", ExpectedTenantId);
|
||||
}
|
||||
```
|
||||
|
||||
### Required Tags
|
||||
|
||||
All WebService endpoints must emit these tags:
|
||||
|
||||
| Tag | Description | Example |
|
||||
|-----|-------------|---------|
|
||||
| `tenant.id` | Tenant identifier | `tenant-a` |
|
||||
| `request.id` | Correlation ID | `req-abc123` |
|
||||
| `http.route` | Endpoint route | `/api/v1/scans` |
|
||||
| `http.status_code` | Response code | `200` |
|
||||
|
||||
Service-specific tags are documented in each module's architecture doc.
|
||||
|
||||
---
|
||||
|
||||
## 4. Negative Tests
|
||||
|
||||
Negative tests verify proper error handling for invalid inputs.
|
||||
|
||||
### Pattern
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
[Trait("Lane", "Security")]
|
||||
public async Task MalformedContentType_Returns415()
|
||||
{
|
||||
// Arrange
|
||||
using var client = Fixture.CreateClient();
|
||||
var content = new StringContent("{}", Encoding.UTF8, "text/plain");
|
||||
|
||||
// Act
|
||||
var response = await client.PostAsync("/api/v1/scans", content);
|
||||
|
||||
// Assert
|
||||
Assert.Equal(HttpStatusCode.UnsupportedMediaType, response.StatusCode);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
[Trait("Lane", "Security")]
|
||||
public async Task OversizedPayload_Returns413()
|
||||
{
|
||||
// Arrange
|
||||
using var client = Fixture.CreateClient();
|
||||
var payload = new string('x', 10_000_001); // Exceeds 10MB limit
|
||||
var content = new StringContent(payload, Encoding.UTF8, "application/json");
|
||||
|
||||
// Act
|
||||
var response = await client.PostAsync("/api/v1/scans", content);
|
||||
|
||||
// Assert
|
||||
Assert.Equal(HttpStatusCode.RequestEntityTooLarge, response.StatusCode);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
[Trait("Lane", "Unit")]
|
||||
public async Task MethodMismatch_Returns405()
|
||||
{
|
||||
// Arrange
|
||||
using var client = Fixture.CreateClient();
|
||||
|
||||
// Act (POST endpoint, but using GET)
|
||||
var response = await client.GetAsync("/api/v1/scans");
|
||||
|
||||
// Assert
|
||||
Assert.Equal(HttpStatusCode.MethodNotAllowed, response.StatusCode);
|
||||
}
|
||||
```
|
||||
|
||||
### Required Coverage
|
||||
|
||||
| Negative Case | Expected Status | Test Trait |
|
||||
|--------------|-----------------|------------|
|
||||
| Malformed content type | 415 | Security |
|
||||
| Oversized payload | 413 | Security |
|
||||
| Method mismatch | 405 | Unit |
|
||||
| Missing required field | 400 | Unit |
|
||||
| Invalid field value | 400 | Unit |
|
||||
| Unknown route | 404 | Unit |
|
||||
|
||||
---
|
||||
|
||||
## 5. Auth/AuthZ Tests
|
||||
|
||||
Auth tests verify security boundaries and tenant isolation.
|
||||
|
||||
### Pattern
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
[Trait("Lane", "Security")]
|
||||
public async Task AnonymousRequest_Returns401()
|
||||
{
|
||||
// Arrange
|
||||
using var client = Fixture.CreateClient(); // No auth
|
||||
|
||||
// Act
|
||||
var response = await client.GetAsync("/api/v1/scans");
|
||||
|
||||
// Assert
|
||||
Assert.Equal(HttpStatusCode.Unauthorized, response.StatusCode);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
[Trait("Lane", "Security")]
|
||||
public async Task ExpiredToken_Returns401()
|
||||
{
|
||||
// Arrange
|
||||
using var client = Fixture.CreateAuthenticatedClient(tokenExpired: true);
|
||||
|
||||
// Act
|
||||
var response = await client.GetAsync("/api/v1/scans");
|
||||
|
||||
// Assert
|
||||
Assert.Equal(HttpStatusCode.Unauthorized, response.StatusCode);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
[Trait("Lane", "Security")]
|
||||
public async Task TenantIsolation_CannotAccessOtherTenantData()
|
||||
{
|
||||
// Arrange
|
||||
using var tenantAClient = Fixture.CreateTenantClient("tenant-a");
|
||||
using var tenantBClient = Fixture.CreateTenantClient("tenant-b");
|
||||
|
||||
// Create scan as tenant A
|
||||
var scanResponse = await tenantAClient.PostAsJsonAsync("/api/v1/scans", new ScanRequest { /* */ });
|
||||
var scan = await scanResponse.Content.ReadFromJsonAsync<ScanResponse>();
|
||||
|
||||
// Act: Try to access as tenant B
|
||||
var response = await tenantBClient.GetAsync($"/api/v1/scans/{scan!.Id}");
|
||||
|
||||
// Assert
|
||||
Assert.Equal(HttpStatusCode.NotFound, response.StatusCode); // Tenant isolation
|
||||
}
|
||||
```
|
||||
|
||||
### Required Coverage
|
||||
|
||||
| Auth Case | Expected Behavior | Test Trait |
|
||||
|-----------|-------------------|------------|
|
||||
| No token | 401 Unauthorized | Security |
|
||||
| Expired token | 401 Unauthorized | Security |
|
||||
| Invalid signature | 401 Unauthorized | Security |
|
||||
| Wrong audience | 401 Unauthorized | Security |
|
||||
| Missing scope | 403 Forbidden | Security |
|
||||
| Cross-tenant access | 404 Not Found or 403 Forbidden | Security |
|
||||
|
||||
---
|
||||
|
||||
## 6. Test Organization
|
||||
|
||||
### Directory Structure
|
||||
|
||||
```
|
||||
src/<Module>/__Tests/StellaOps.<Module>.WebService.Tests/
|
||||
├── StellaOps.<Module>.WebService.Tests.csproj
|
||||
├── <Module>ApplicationFactory.cs # WebApplicationFactory implementation
|
||||
├── <Module>TestFixture.cs # Shared test fixture
|
||||
├── Contract/
|
||||
│ └── OpenApiSchemaTests.cs
|
||||
├── Telemetry/
|
||||
│ └── OtelTraceTests.cs
|
||||
├── Negative/
|
||||
│ ├── ContentTypeTests.cs
|
||||
│ ├── PayloadLimitTests.cs
|
||||
│ └── MethodMismatchTests.cs
|
||||
├── Auth/
|
||||
│ ├── AuthenticationTests.cs
|
||||
│ ├── AuthorizationTests.cs
|
||||
│ └── TenantIsolationTests.cs
|
||||
└── Snapshots/
|
||||
└── <module>-v1.json # OpenAPI schema snapshot
|
||||
```
|
||||
|
||||
### Test Trait Assignment
|
||||
|
||||
| Category | Trait | CI Lane | PR-Gating |
|
||||
|----------|-------|---------|-----------|
|
||||
| Contract | `[Trait("Lane", "Contract")]` | Contract | Yes |
|
||||
| OTel | `[Trait("Lane", "Integration")]` | Integration | Yes |
|
||||
| Negative (security) | `[Trait("Lane", "Security")]` | Security | Yes |
|
||||
| Negative (validation) | `[Trait("Lane", "Unit")]` | Unit | Yes |
|
||||
| Auth/AuthZ | `[Trait("Lane", "Security")]` | Security | Yes |
|
||||
|
||||
---
|
||||
|
||||
## 7. CI Integration
|
||||
|
||||
WebService tests run in the appropriate CI lanes:
|
||||
|
||||
```yaml
|
||||
# .gitea/workflows/test-lanes.yml
|
||||
jobs:
|
||||
contract-tests:
|
||||
steps:
|
||||
- run: ./scripts/test-lane.sh Contract
|
||||
|
||||
security-tests:
|
||||
steps:
|
||||
- run: ./scripts/test-lane.sh Security
|
||||
|
||||
integration-tests:
|
||||
steps:
|
||||
- run: ./scripts/test-lane.sh Integration
|
||||
```
|
||||
|
||||
All lanes are PR-gating. Failed tests block merge.
|
||||
|
||||
---
|
||||
|
||||
## 8. Rollout Checklist
|
||||
|
||||
When adding WebService tests to a new module:
|
||||
|
||||
- [ ] Create `<Module>ApplicationFactory` extending `WebApplicationFactory<TProgram>`
|
||||
- [ ] Create `<Module>TestFixture` extending `WebServiceFixture<TProgram>` if needed
|
||||
- [ ] Add contract tests with OpenAPI schema snapshot
|
||||
- [ ] Add OTel trace tests for key endpoints
|
||||
- [ ] Add negative tests (content type, payload, method)
|
||||
- [ ] Add auth/authz tests (anonymous, expired, tenant isolation)
|
||||
- [ ] Verify all tests have appropriate `[Trait("Lane", "...")]` attributes
|
||||
- [ ] Run locally: `dotnet test --filter "Lane=Contract|Lane=Security|Lane=Integration"`
|
||||
- [ ] Verify CI passes on PR
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- [WebServiceFixture Implementation](../../src/__Libraries/StellaOps.TestKit/Fixtures/WebServiceFixture.cs)
|
||||
- [ContractTestHelper Implementation](../../src/__Libraries/StellaOps.TestKit/Fixtures/ContractTestHelper.cs)
|
||||
- [WebServiceTestBase Implementation](../../src/__Libraries/StellaOps.TestKit/Templates/WebServiceTestBase.cs)
|
||||
- [Test Lanes CI Workflow](../../.gitea/workflows/test-lanes.yml)
|
||||
- [CI Lane Filters Documentation](./ci-lane-filters.md)
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2025-06-30 · Sprint 5100.0007.0006*
|
||||
Reference in New Issue
Block a user