Restructure solution layout by module
Some checks failed
Docs CI / lint-and-preview (push) Has been cancelled
Some checks failed
Docs CI / lint-and-preview (push) Has been cancelled
This commit is contained in:
root
@@ -1,47 +1,47 @@
|
||||
# Automated Test‑Suite Overview
|
||||
|
||||
This document enumerates **every automated check** executed by the Stella Ops
|
||||
CI pipeline, from unit level to chaos experiments. It is intended for
|
||||
contributors who need to extend coverage or diagnose failures.
|
||||
|
||||
> **Build parameters** – values such as `{{ dotnet }}` (runtime) and
|
||||
> `{{ angular }}` (UI framework) are injected at build time.
|
||||
|
||||
---
|
||||
|
||||
## Layer map
|
||||
|
||||
| Layer | Tooling | Entry‑point | Frequency |
|
||||
|-------|---------|-------------|-----------|
|
||||
| **1. Unit** | `xUnit` (<code>dotnet test</code>) | `*.Tests.csproj` | per PR / push |
|
||||
| **2. Property‑based** | `FsCheck` | `SbomPropertyTests` | per PR |
|
||||
| **3. Integration (API)** | `Testcontainers` suite | `test/Api.Integration` | per PR + nightly |
|
||||
# Automated Test‑Suite Overview
|
||||
|
||||
This document enumerates **every automated check** executed by the Stella Ops
|
||||
CI pipeline, from unit level to chaos experiments. It is intended for
|
||||
contributors who need to extend coverage or diagnose failures.
|
||||
|
||||
> **Build parameters** – values such as `{{ dotnet }}` (runtime) and
|
||||
> `{{ angular }}` (UI framework) are injected at build time.
|
||||
|
||||
---
|
||||
|
||||
## Layer map
|
||||
|
||||
| Layer | Tooling | Entry‑point | Frequency |
|
||||
|-------|---------|-------------|-----------|
|
||||
| **1. Unit** | `xUnit` (<code>dotnet test</code>) | `*.Tests.csproj` | per PR / push |
|
||||
| **2. Property‑based** | `FsCheck` | `SbomPropertyTests` | per PR |
|
||||
| **3. Integration (API)** | `Testcontainers` suite | `test/Api.Integration` | per PR + nightly |
|
||||
| **4. Integration (DB-merge)** | in-memory Mongo + Redis | `Concelier.Integration` (vulnerability ingest/merge/export service) | per PR |
|
||||
| **5. Contract (gRPC)** | `Buf breaking` | `buf.yaml` files | per PR |
|
||||
| **6. Front‑end unit** | `Jest` | `ui/src/**/*.spec.ts` | per PR |
|
||||
| **7. Front‑end E2E** | `Playwright` | `ui/e2e/**` | nightly |
|
||||
| **8. Lighthouse perf / a11y** | `lighthouse-ci` (Chrome headless) | `ui/dist/index.html` | nightly |
|
||||
| **9. Load** | `k6` scripted scenarios | `k6/*.js` | nightly |
|
||||
| **10. Chaos CPU / OOM** | `pumba` | Docker Compose overlay | weekly |
|
||||
| **11. Dependency scanning** | `Trivy fs` + `dotnet list package --vuln` | root | per PR |
|
||||
| **12. License compliance** | `LicenceFinder` | root | per PR |
|
||||
| **13. SBOM reproducibility** | `in‑toto attestation` diff | GitLab job | release tags |
|
||||
|
||||
---
|
||||
|
||||
## Quality gates
|
||||
|
||||
| Metric | Budget | Gate |
|
||||
|--------|--------|------|
|
||||
| API unit coverage | ≥ 85 % lines | PR merge |
|
||||
| API response P95 | ≤ 120 ms | nightly alert |
|
||||
| Δ‑SBOM warm scan P95 (4 vCPU) | ≤ 5 s | nightly alert |
|
||||
| Lighthouse performance score | ≥ 90 | nightly alert |
|
||||
| Lighthouse accessibility score | ≥ 95 | nightly alert |
|
||||
| k6 sustained RPS drop | < 5 % vs baseline | nightly alert |
|
||||
|
||||
---
|
||||
|
||||
| **5. Contract (gRPC)** | `Buf breaking` | `buf.yaml` files | per PR |
|
||||
| **6. Front‑end unit** | `Jest` | `ui/src/**/*.spec.ts` | per PR |
|
||||
| **7. Front‑end E2E** | `Playwright` | `ui/e2e/**` | nightly |
|
||||
| **8. Lighthouse perf / a11y** | `lighthouse-ci` (Chrome headless) | `ui/dist/index.html` | nightly |
|
||||
| **9. Load** | `k6` scripted scenarios | `k6/*.js` | nightly |
|
||||
| **10. Chaos CPU / OOM** | `pumba` | Docker Compose overlay | weekly |
|
||||
| **11. Dependency scanning** | `Trivy fs` + `dotnet list package --vuln` | root | per PR |
|
||||
| **12. License compliance** | `LicenceFinder` | root | per PR |
|
||||
| **13. SBOM reproducibility** | `in‑toto attestation` diff | GitLab job | release tags |
|
||||
|
||||
---
|
||||
|
||||
## Quality gates
|
||||
|
||||
| Metric | Budget | Gate |
|
||||
|--------|--------|------|
|
||||
| API unit coverage | ≥ 85 % lines | PR merge |
|
||||
| API response P95 | ≤ 120 ms | nightly alert |
|
||||
| Δ‑SBOM warm scan P95 (4 vCPU) | ≤ 5 s | nightly alert |
|
||||
| Lighthouse performance score | ≥ 90 | nightly alert |
|
||||
| Lighthouse accessibility score | ≥ 95 | nightly alert |
|
||||
| k6 sustained RPS drop | < 5 % vs baseline | nightly alert |
|
||||
|
||||
---
|
||||
|
||||
## Local runner
|
||||
|
||||
```bash
|
||||
@@ -63,13 +63,13 @@ The script spins up MongoDB/Redis via Testcontainers and requires:
|
||||
|
||||
The Concelier connector suite includes a regression test (`OsvGhsaParityRegressionTests`)
|
||||
that checks a curated set of GHSA identifiers against OSV responses. The fixture
|
||||
snapshots live in `src/StellaOps.Concelier.Connector.Osv.Tests/Fixtures/` and are kept
|
||||
snapshots live in `src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Osv.Tests/Fixtures/` and are kept
|
||||
deterministic so the parity report remains reproducible.
|
||||
|
||||
To refresh the fixtures when GHSA/OSV payloads change:
|
||||
|
||||
1. Ensure outbound HTTPS access to `https://api.osv.dev` and `https://api.github.com`.
|
||||
2. Run `UPDATE_PARITY_FIXTURES=1 dotnet test src/StellaOps.Concelier.Connector.Osv.Tests/StellaOps.Concelier.Connector.Osv.Tests.csproj`.
|
||||
2. Run `UPDATE_PARITY_FIXTURES=1 dotnet test src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Osv.Tests/StellaOps.Concelier.Connector.Osv.Tests.csproj`.
|
||||
3. Commit the regenerated `osv-ghsa.*.json` files that the test emits (raw snapshots and canonical advisories).
|
||||
|
||||
The regen flow logs `[Parity]` messages and normalises `recordedAt` timestamps so the
|
||||
@@ -82,28 +82,28 @@ fixtures stay stable across machines.
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph fast-path
|
||||
U[xUnit] --> P[FsCheck] --> I1[Testcontainer API]
|
||||
end
|
||||
|
||||
I1 --> FE[Jest]
|
||||
FE --> E2E[Playwright]
|
||||
E2E --> Lighthouse
|
||||
U[xUnit] --> P[FsCheck] --> I1[Testcontainer API]
|
||||
end
|
||||
|
||||
I1 --> FE[Jest]
|
||||
FE --> E2E[Playwright]
|
||||
E2E --> Lighthouse
|
||||
Lighthouse --> INTEG2[Concelier]
|
||||
INTEG2 --> LOAD[k6]
|
||||
LOAD --> CHAOS[pumba]
|
||||
CHAOS --> RELEASE[Attestation diff]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Adding a new test layer
|
||||
|
||||
1. Extend `scripts/dev-test.sh` so local contributors get the layer by default.
|
||||
2. Add a dedicated GitLab job in `.gitlab-ci.yml` (stage `test` or `nightly`).
|
||||
3. Register the job in `docs/19_TEST_SUITE_OVERVIEW.md` *and* list its metric
|
||||
in `docs/metrics/README.md`.
|
||||
|
||||
---
|
||||
|
||||
*Last updated {{ "now" | date: "%Y‑%m‑%d" }}*
|
||||
|
||||
INTEG2 --> LOAD[k6]
|
||||
LOAD --> CHAOS[pumba]
|
||||
CHAOS --> RELEASE[Attestation diff]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Adding a new test layer
|
||||
|
||||
1. Extend `scripts/dev-test.sh` so local contributors get the layer by default.
|
||||
2. Add a dedicated GitLab job in `.gitlab-ci.yml` (stage `test` or `nightly`).
|
||||
3. Register the job in `docs/19_TEST_SUITE_OVERVIEW.md` *and* list its metric
|
||||
in `docs/metrics/README.md`.
|
||||
|
||||
---
|
||||
|
||||
*Last updated {{ "now" | date: "%Y‑%m‑%d" }}*
|
||||
|
||||
|
||||
Reference in New Issue
Block a user