3a2100aa78
- Implemented comprehensive unit tests for VexCandidateEmitter to validate candidate emission logic based on various scenarios including absent and present APIs, confidence thresholds, and rate limiting. - Added integration tests for SmartDiff PostgreSQL repositories, covering snapshot storage and retrieval, candidate storage, and material risk change handling. - Ensured tests validate correct behavior for storing, retrieving, and querying snapshots and candidates, including edge cases and expected outcomes.
136 lines
4.9 KiB
Markdown
Executable File
136 lines
4.9 KiB
Markdown
Executable File
# Automated Test‑Suite Overview
|
||
|
||
This document enumerates **every automated check** executed by the Stella Ops
|
||
CI pipeline, from unit level to chaos experiments. It is intended for
|
||
contributors who need to extend coverage or diagnose failures.
|
||
|
||
> **Build parameters** – values such as `{{ dotnet }}` (runtime) and
|
||
> `{{ angular }}` (UI framework) are injected at build time.
|
||
|
||
---
|
||
|
||
## Layer map
|
||
|
||
| Layer | Tooling | Entry‑point | Frequency |
|
||
|-------|---------|-------------|-----------|
|
||
| **1. Unit** | `xUnit` (<code>dotnet test</code>) | `*.Tests.csproj` | per PR / push |
|
||
| **2. Property‑based** | `FsCheck` | `SbomPropertyTests` | per PR |
|
||
| **3. Integration (API)** | `Testcontainers` suite | `test/Api.Integration` | per PR + nightly |
|
||
| **4. Integration (DB-merge)** | Testcontainers PostgreSQL + Redis | `Concelier.Integration` (vulnerability ingest/merge/export service) | per PR |
|
||
| **5. Contract (gRPC)** | `Buf breaking` | `buf.yaml` files | per PR |
|
||
| **6. Front‑end unit** | `Jest` | `ui/src/**/*.spec.ts` | per PR |
|
||
| **7. Front‑end E2E** | `Playwright` | `ui/e2e/**` | nightly |
|
||
| **8. Lighthouse perf / a11y** | `lighthouse-ci` (Chrome headless) | `ui/dist/index.html` | nightly |
|
||
| **9. Load** | `k6` scripted scenarios | `k6/*.js` | nightly |
|
||
| **10. Chaos CPU / OOM** | `pumba` | Docker Compose overlay | weekly |
|
||
| **11. Dependency scanning** | `Trivy fs` + `dotnet list package --vuln` | root | per PR |
|
||
| **12. License compliance** | `LicenceFinder` | root | per PR |
|
||
| **13. SBOM reproducibility** | `in‑toto attestation` diff | GitLab job | release tags |
|
||
|
||
---
|
||
|
||
## Quality gates
|
||
|
||
| Metric | Budget | Gate |
|
||
|--------|--------|------|
|
||
| API unit coverage | ≥ 85 % lines | PR merge |
|
||
| API response P95 | ≤ 120 ms | nightly alert |
|
||
| Δ‑SBOM warm scan P95 (4 vCPU) | ≤ 5 s | nightly alert |
|
||
| Lighthouse performance score | ≥ 90 | nightly alert |
|
||
| Lighthouse accessibility score | ≥ 95 | nightly alert |
|
||
| k6 sustained RPS drop | < 5 % vs baseline | nightly alert |
|
||
|
||
---
|
||
|
||
## Local runner
|
||
|
||
```bash
|
||
# minimal run: unit + property + frontend tests
|
||
./scripts/dev-test.sh
|
||
|
||
# full stack incl. Playwright and lighthouse
|
||
./scripts/dev-test.sh --full
|
||
````
|
||
|
||
The script spins up PostgreSQL/Redis via Testcontainers and requires:
|
||
|
||
* Docker ≥ 25
|
||
* Node 20 (for Jest/Playwright)
|
||
|
||
#### PostgreSQL Testcontainers
|
||
|
||
Multiple suites (Concelier connectors, Excititor worker/WebService, Scheduler)
|
||
use Testcontainers with PostgreSQL for integration tests. If you don't have
|
||
Docker available, tests can also run against a local PostgreSQL instance
|
||
listening on `127.0.0.1:5432`.
|
||
|
||
#### Local PostgreSQL helper
|
||
|
||
Some suites (Concelier WebService/Core, Exporter JSON) need a full
|
||
PostgreSQL instance when you want to debug or inspect data with `psql`.
|
||
A helper script is available under `tools/postgres/local-postgres.sh`:
|
||
|
||
```bash
|
||
# start a local PostgreSQL instance
|
||
tools/postgres/local-postgres.sh start
|
||
|
||
# stop / clean
|
||
tools/postgres/local-postgres.sh stop
|
||
tools/postgres/local-postgres.sh clean
|
||
```
|
||
|
||
By default the script uses Docker to run PostgreSQL 16, binds to
|
||
`127.0.0.1:5432`, and creates a database called `stellaops`. The
|
||
connection string is printed on start and you can export it before
|
||
running `dotnet test` if a suite supports overriding its connection string.
|
||
|
||
---
|
||
|
||
### Concelier OSV↔GHSA parity fixtures
|
||
|
||
The Concelier connector suite includes a regression test (`OsvGhsaParityRegressionTests`)
|
||
that checks a curated set of GHSA identifiers against OSV responses. The fixture
|
||
snapshots live in `src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Osv.Tests/Fixtures/` and are kept
|
||
deterministic so the parity report remains reproducible.
|
||
|
||
To refresh the fixtures when GHSA/OSV payloads change:
|
||
|
||
1. Ensure outbound HTTPS access to `https://api.osv.dev` and `https://api.github.com`.
|
||
2. Run `UPDATE_PARITY_FIXTURES=1 dotnet test src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Osv.Tests/StellaOps.Concelier.Connector.Osv.Tests.csproj`.
|
||
3. Commit the regenerated `osv-ghsa.*.json` files that the test emits (raw snapshots and canonical advisories).
|
||
|
||
The regen flow logs `[Parity]` messages and normalises `recordedAt` timestamps so the
|
||
fixtures stay stable across machines.
|
||
|
||
---
|
||
|
||
## CI job layout
|
||
|
||
```mermaid
|
||
flowchart LR
|
||
subgraph fast-path
|
||
U[xUnit] --> P[FsCheck] --> I1[Testcontainer API]
|
||
end
|
||
|
||
I1 --> FE[Jest]
|
||
FE --> E2E[Playwright]
|
||
E2E --> Lighthouse
|
||
Lighthouse --> INTEG2[Concelier]
|
||
INTEG2 --> LOAD[k6]
|
||
LOAD --> CHAOS[pumba]
|
||
CHAOS --> RELEASE[Attestation diff]
|
||
```
|
||
|
||
---
|
||
|
||
## Adding a new test layer
|
||
|
||
1. Extend `scripts/dev-test.sh` so local contributors get the layer by default.
|
||
2. Add a dedicated GitLab job in `.gitlab-ci.yml` (stage `test` or `nightly`).
|
||
3. Register the job in `docs/19_TEST_SUITE_OVERVIEW.md` *and* list its metric
|
||
in `docs/metrics/README.md`.
|
||
|
||
---
|
||
|
||
*Last updated {{ "now" | date: "%Y‑%m‑%d" }}*
|