git.stella-ops.org/docs/19_TEST_SUITE_OVERVIEW.md

Automated Test‑Suite Overview

This document enumerates every automated check executed by the Stella Ops CI pipeline, from unit level to chaos experiments. It is intended for contributors who need to extend coverage or diagnose failures.

Build parameters – values such as {{ dotnet }} (runtime) and {{ angular }} (UI framework) are injected at build time.

---

Layer map

Layer	Tooling	Entry‑point	Frequency
1. Unit	xUnit (dotnet test)	*.Tests.csproj	per PR / push
2. Property‑based	FsCheck	SbomPropertyTests	per PR
3. Integration (API)	Testcontainers suite	test/Api.Integration	per PR + nightly
4. Integration (DB-merge)	Testcontainers PostgreSQL + Redis	Concelier.Integration (vulnerability ingest/merge/export service)	per PR
5. Contract (gRPC)	Buf breaking	buf.yaml files	per PR
6. Front‑end unit	Jest	ui/src/**/*.spec.ts	per PR
7. Front‑end E2E	Playwright	ui/e2e/**	nightly
8. Lighthouse perf / a11y	lighthouse-ci (Chrome headless)	ui/dist/index.html	nightly
9. Load	k6 scripted scenarios	k6/*.js	nightly
10. Chaos CPU / OOM	pumba	Docker Compose overlay	weekly
11. Dependency scanning	Trivy fs + dotnet list package --vuln	root	per PR
12. License compliance	LicenceFinder	root	per PR
13. SBOM reproducibility	in‑toto attestation diff	GitLab job	release tags

---

Quality gates

Metric	Budget	Gate
API unit coverage	≥ 85 % lines	PR merge
API response P95	≤ 120 ms	nightly alert
Δ‑SBOM warm scan P95 (4 vCPU)	≤ 5 s	nightly alert
Lighthouse performance score	≥ 90	nightly alert
Lighthouse accessibility score	≥ 95	nightly alert
k6 sustained RPS drop	< 5 % vs baseline	nightly alert

---

Local runner

# minimal run: unit + property + frontend tests
./scripts/dev-test.sh

# full stack incl. Playwright and lighthouse
./scripts/dev-test.sh --full

The script spins up PostgreSQL/Redis via Testcontainers and requires:

• Docker ≥ 25
• Node 20 (for Jest/Playwright)

PostgreSQL Testcontainers

Multiple suites (Concelier connectors, Excititor worker/WebService, Scheduler) use Testcontainers with PostgreSQL for integration tests. If you don't have Docker available, tests can also run against a local PostgreSQL instance listening on 127.0.0.1:5432.

Local PostgreSQL helper

Some suites (Concelier WebService/Core, Exporter JSON) need a full PostgreSQL instance when you want to debug or inspect data with psql. A helper script is available under tools/postgres/local-postgres.sh:

# start a local PostgreSQL instance
tools/postgres/local-postgres.sh start

# stop / clean
tools/postgres/local-postgres.sh stop
tools/postgres/local-postgres.sh clean

By default the script uses Docker to run PostgreSQL 16, binds to 127.0.0.1:5432, and creates a database called stellaops. The connection string is printed on start and you can export it before running dotnet test if a suite supports overriding its connection string.

---

Concelier OSV↔GHSA parity fixtures

The Concelier connector suite includes a regression test (OsvGhsaParityRegressionTests) that checks a curated set of GHSA identifiers against OSV responses. The fixture snapshots live in src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Osv.Tests/Fixtures/ and are kept deterministic so the parity report remains reproducible.

To refresh the fixtures when GHSA/OSV payloads change:

1. Ensure outbound HTTPS access to https://api.osv.dev and https://api.github.com.
2. Run UPDATE_PARITY_FIXTURES=1 dotnet test src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Osv.Tests/StellaOps.Concelier.Connector.Osv.Tests.csproj.
3. Commit the regenerated osv-ghsa.*.json files that the test emits (raw snapshots and canonical advisories).

The regen flow logs [Parity] messages and normalises recordedAt timestamps so the fixtures stay stable across machines.

---

CI job layout

flowchart LR
  subgraph fast-path
    U[xUnit] --> P[FsCheck] --> I1[Testcontainer API]
  end

  I1 --> FE[Jest]
  FE --> E2E[Playwright]
  E2E --> Lighthouse
  Lighthouse --> INTEG2[Concelier]
  INTEG2 --> LOAD[k6]
  LOAD --> CHAOS[pumba]
  CHAOS --> RELEASE[Attestation diff]

---

Adding a new test layer

1. Extend scripts/dev-test.sh so local contributors get the layer by default.
2. Add a dedicated GitLab job in .gitlab-ci.yml (stage test or nightly).
3. Register the job in docs/19_TEST_SUITE_OVERVIEW.md and list its metric in docs/metrics/README.md.

---

Last updated {{ "now" | date: "%Y‑%m‑%d" }}