# Automated Test‑Suite Overview This document enumerates **every automated check** executed by the Stella Ops CI pipeline, from unit level to chaos experiments. It is intended for contributors who need to extend coverage or diagnose failures. > **Build parameters** – values such as `{{ dotnet }}` (runtime) and > `{{ angular }}` (UI framework) are injected at build time. --- ## Layer map | Layer | Tooling | Entry‑point | Frequency | |-------|---------|-------------|-----------| | **1. Unit** | `xUnit` (dotnet test) | `*.Tests.csproj` | per PR / push | | **2. Property‑based** | `FsCheck` | `SbomPropertyTests` | per PR | | **3. Integration (API)** | `Testcontainers` suite | `test/Api.Integration` | per PR + nightly | | **4. Integration (DB-merge)** | in-memory Mongo + Redis | `Concelier.Integration` (vulnerability ingest/merge/export service) | per PR | | **5. Contract (gRPC)** | `Buf breaking` | `buf.yaml` files | per PR | | **6. Front‑end unit** | `Jest` | `ui/src/**/*.spec.ts` | per PR | | **7. Front‑end E2E** | `Playwright` | `ui/e2e/**` | nightly | | **8. Lighthouse perf / a11y** | `lighthouse-ci` (Chrome headless) | `ui/dist/index.html` | nightly | | **9. Load** | `k6` scripted scenarios | `k6/*.js` | nightly | | **10. Chaos CPU / OOM** | `pumba` | Docker Compose overlay | weekly | | **11. Dependency scanning** | `Trivy fs` + `dotnet list package --vuln` | root | per PR | | **12. License compliance** | `LicenceFinder` | root | per PR | | **13. SBOM reproducibility** | `in‑toto attestation` diff | GitLab job | release tags | --- ## Quality gates | Metric | Budget | Gate | |--------|--------|------| | API unit coverage | ≥ 85 % lines | PR merge | | API response P95 | ≤ 120 ms | nightly alert | | Δ‑SBOM warm scan P95 (4 vCPU) | ≤ 5 s | nightly alert | | Lighthouse performance score | ≥ 90 | nightly alert | | Lighthouse accessibility score | ≥ 95 | nightly alert | | k6 sustained RPS drop | < 5 % vs baseline | nightly alert | --- ## Local runner ```bash # minimal run: unit + property + frontend tests ./scripts/dev-test.sh # full stack incl. Playwright and lighthouse ./scripts/dev-test.sh --full ```` The script spins up MongoDB/Redis via Testcontainers and requires: * Docker ≥ 25 * Node 20 (for Jest/Playwright) #### Mongo2Go / OpenSSL shim Multiple suites (Concelier connectors, Excititor worker/WebService, Scheduler) fall back to [Mongo2Go](https://github.com/Mongo2Go/Mongo2Go) when a developer does not have a local `mongod` listening on `127.0.0.1:27017`. **This is a test-only dependency**: production/dev runtime MongoDB always runs inside the compose/k8s network using the standard StellaOps cryptography stack. Modern distros ship OpenSSL 3 by default, so when Mongo2Go starts its embedded `mongod` you **must** expose the legacy OpenSSL 1.1 libraries that binary expects: 1. From the repo root, export the provided binaries before running any tests: ```bash export LD_LIBRARY_PATH="$(pwd)/tests/native/openssl-1.1/linux-x64:${LD_LIBRARY_PATH:-}" ``` 2. (Optional) If you only need the shim for a single command, prefix it: ```bash LD_LIBRARY_PATH="$(pwd)/tests/native/openssl-1.1/linux-x64" \ dotnet test src/Concelier/StellaOps.Concelier.sln --nologo ``` 3. CI runners or dev containers should either copy `tests/native/openssl-1.1/linux-x64/libcrypto.so.1.1` and `libssl.so.1.1` into a directory that is already on the default library path, or export the `LD_LIBRARY_PATH` value shown above before invoking `dotnet test`. The shim lives under `tests/native/openssl-1.1/README.md` with upstream source and licensing details. When the system already has OpenSSL 1.1 installed you can skip this step. #### Local Mongo helper Some suites (Concelier WebService/Core, Exporter JSON) need a full `mongod` instance when you want to debug outside of Mongo2Go (for example to inspect data with `mongosh` or pin a specific server version). A thin wrapper is available under `tools/mongodb/local-mongo.sh`: ```bash # download (cached under .cache/mongodb-local) and start a local replica set tools/mongodb/local-mongo.sh start # reuse an existing data set tools/mongodb/local-mongo.sh restart # stop / clean tools/mongodb/local-mongo.sh stop tools/mongodb/local-mongo.sh clean ``` By default the script downloads MongoDB 6.0.16 for Ubuntu 22.04, binds to `127.0.0.1:27017`, and initialises a single-node replica set called `rs0`. The current URI is printed on start, e.g. `mongodb://127.0.0.1:27017/?replicaSet=rs0`, and you can export it before running `dotnet test` if a suite supports overriding its connection string. --- ### Concelier OSV↔GHSA parity fixtures The Concelier connector suite includes a regression test (`OsvGhsaParityRegressionTests`) that checks a curated set of GHSA identifiers against OSV responses. The fixture snapshots live in `src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Osv.Tests/Fixtures/` and are kept deterministic so the parity report remains reproducible. To refresh the fixtures when GHSA/OSV payloads change: 1. Ensure outbound HTTPS access to `https://api.osv.dev` and `https://api.github.com`. 2. Run `UPDATE_PARITY_FIXTURES=1 dotnet test src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Osv.Tests/StellaOps.Concelier.Connector.Osv.Tests.csproj`. 3. Commit the regenerated `osv-ghsa.*.json` files that the test emits (raw snapshots and canonical advisories). The regen flow logs `[Parity]` messages and normalises `recordedAt` timestamps so the fixtures stay stable across machines. --- ## CI job layout ```mermaid flowchart LR subgraph fast-path U[xUnit] --> P[FsCheck] --> I1[Testcontainer API] end I1 --> FE[Jest] FE --> E2E[Playwright] E2E --> Lighthouse Lighthouse --> INTEG2[Concelier] INTEG2 --> LOAD[k6] LOAD --> CHAOS[pumba] CHAOS --> RELEASE[Attestation diff] ``` --- ## Adding a new test layer 1. Extend `scripts/dev-test.sh` so local contributors get the layer by default. 2. Add a dedicated GitLab job in `.gitlab-ci.yml` (stage `test` or `nightly`). 3. Register the job in `docs/19_TEST_SUITE_OVERVIEW.md` *and* list its metric in `docs/metrics/README.md`. --- *Last updated {{ "now" | date: "%Y‑%m‑%d" }}*