docs consolidation
This commit is contained in:
master
557
docs/technical/DATA_SCHEMAS.md
Executable file
557
docs/technical/DATA_SCHEMAS.md
Executable file
@@ -0,0 +1,557 @@
|
||||
# Data Schemas & Persistence Contracts
|
||||
|
||||
*Audience* – backend developers, plug‑in authors, DB admins.
|
||||
*Scope* – describes **Valkey**, **PostgreSQL**, and on‑disk blob shapes that power Stella Ops.
|
||||
|
||||
---
|
||||
|
||||
## 0 Document Conventions
|
||||
|
||||
* **CamelCase** for JSON.
|
||||
* All timestamps are **RFC 3339 / ISO 8601** with `Z` (UTC).
|
||||
* `⭑` = planned but *not* shipped yet (kept on Feature Matrix “To Do”).
|
||||
|
||||
---
|
||||
|
||||
## 1 SBOM Wrapper Envelope
|
||||
|
||||
Every SBOM blob (regardless of format) is stored on disk or in object storage with a *sidecar* JSON file that indexes it for the scanners.
|
||||
|
||||
#### 1.1 JSON Shape
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"id": "sha256:417f…", // digest of the SBOM *file* itself
|
||||
"imageDigest": "sha256:e2b9…", // digest of the original container image
|
||||
"created": "2025-07-14T07:02:13Z",
|
||||
"format": "trivy-json-v2", // NEW enum: trivy-json-v2 | spdx-json | cyclonedx-json
|
||||
"layers": [
|
||||
"sha256:d38b…", // layer digests (ordered)
|
||||
"sha256:af45…"
|
||||
],
|
||||
"partial": false, // true => delta SBOM (only some layers)
|
||||
"provenanceId": "prov_0291" // ⭑ link to SLSA attestation (Q1‑2026)
|
||||
}
|
||||
```
|
||||
|
||||
*`format`* **NEW** – added to support **multiple SBOM formats**.
|
||||
*`partial`* **NEW** – true when generated via the **delta SBOM** flow (§1.3).
|
||||
|
||||
#### 1.2 File‑system Layout
|
||||
|
||||
```
|
||||
blobs/
|
||||
├─ 417f… # digest prefix
|
||||
│ ├─ sbom.json # payload (any format)
|
||||
│ └─ sbom.meta.json # wrapper (shape above)
|
||||
```
|
||||
|
||||
> **Note** – RustFS is the primary object store; S3/MinIO compatibility layer available for legacy deployments; driver plug‑ins support multiple backends.
|
||||
|
||||
#### 1.3 Delta SBOM Extension
|
||||
|
||||
When `partial: true`, *only* the missing layers have been scanned.
|
||||
Merging logic inside `scanning` module stitches new data onto the cached full SBOM in Valkey.
|
||||
|
||||
---
|
||||
|
||||
## 2 Valkey Keyspace
|
||||
|
||||
Valkey (Redis-compatible) provides cache, DPoP nonces, event streams, and queues for real-time messaging and rate limiting.
|
||||
|
||||
| Key pattern | Type | TTL | Purpose |
|
||||
|-------------------------------------|---------|------|--------------------------------------------------|
|
||||
| `scan:<digest>` | string | ∞ | Last scan JSON result (as returned by `/scan`) |
|
||||
| `layers:<digest>` | set | 90d | Layers already possessing SBOMs (delta cache) |
|
||||
| `policy:active` | string | ∞ | YAML **or** Rego ruleset |
|
||||
| `quota:<token>` | string | *until next UTC midnight* | Per‑token scan counter for Free tier ({{ quota_token }} scans). |
|
||||
| `policy:history` | list | ∞ | Change audit IDs (see PostgreSQL) |
|
||||
| `feed:nvd:json` | string | 24h | Normalised feed snapshot |
|
||||
| `locator:<imageDigest>` | string | 30d | Maps image digest → sbomBlobId |
|
||||
| `dpop:<jti>` | string | 5m | DPoP nonce cache (RFC 9449) for sender-constrained tokens |
|
||||
| `events:*` | stream | 7d | Event streams for Scheduler/Notify (Valkey Streams) |
|
||||
| `queue:*` | stream | — | Task queues (Scanner jobs, Notify deliveries) |
|
||||
| `metrics:…` | various | — | Prom / OTLP runtime metrics |
|
||||
|
||||
> **Delta SBOM** uses `layers:*` to skip work in <20 ms.
|
||||
> **Quota enforcement** increments `quota:<token>` atomically; when {{ quota_token }} the API returns **429**.
|
||||
> **DPoP & Events**: Valkey Streams support high-throughput, ordered event delivery for re-evaluation and notification triggers.
|
||||
> **Alternative**: NATS JetStream can replace Valkey for queues (opt-in only; requires explicit configuration).
|
||||
|
||||
---
|
||||
|
||||
## 3 PostgreSQL Tables
|
||||
|
||||
PostgreSQL is the canonical persistent store for long-term audit and history.
|
||||
|
||||
| Table | Shape (summary) | Indexes |
|
||||
|--------------------|------------------------------------------------------------|-------------------------------------|
|
||||
| `sbom_history` | Wrapper JSON + `replace_ts` on overwrite | `(image_digest)` `(created)` |
|
||||
| `policy_versions` | `{id, yaml, rego, author_id, created}` | `(created)` |
|
||||
| `attestations` ⭑ | SLSA provenance doc + Rekor log pointer | `(image_digest)` |
|
||||
| `audit_log` | Fully rendered RFC 5424 entries (UI & CLI actions) | `(user_id)` `(ts)` |
|
||||
|
||||
Schema detail for **policy_versions**:
|
||||
|
||||
Samples live under `samples/api/scheduler/` (e.g., `schedule.json`, `run.json`, `impact-set.json`, `audit.json`) and mirror the canonical serializer output shown below.
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"_id": "6619e90b8c5e1f76",
|
||||
"yaml": "version: 1.0\nrules:\n - …",
|
||||
"rego": null, // filled when Rego uploaded
|
||||
"authorId": "u_1021",
|
||||
"created": "2025-07-14T08:15:04Z",
|
||||
"comment": "Imported via API"
|
||||
}
|
||||
```
|
||||
|
||||
### 3.1 Scheduler Sprints 16 Artifacts
|
||||
|
||||
**Tables.** `schedules`, `runs`, `impact_snapshots`, `audit` (module-local). All rows use the canonical JSON emitted by `StellaOps.Scheduler.Models` so agents and fixtures remain deterministic.
|
||||
|
||||
#### 3.1.1 Schedule (`schedules`)
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"id": "sch_20251018a",
|
||||
"tenantId": "tenant-alpha",
|
||||
"name": "Nightly Prod",
|
||||
"enabled": true,
|
||||
"cronExpression": "0 2 * * *",
|
||||
"timezone": "UTC",
|
||||
"mode": "analysis-only",
|
||||
"selection": {
|
||||
"scope": "by-namespace",
|
||||
"namespaces": ["team-a", "team-b"],
|
||||
"repositories": ["app/service-api"],
|
||||
"includeTags": ["canary", "prod"],
|
||||
"labels": [{"key": "env", "values": ["prod", "staging"]}],
|
||||
"resolvesTags": true
|
||||
},
|
||||
"onlyIf": {"lastReportOlderThanDays": 7, "policyRevision": "policy@42"},
|
||||
"notify": {"onNewFindings": true, "minSeverity": "high", "includeKev": true},
|
||||
"limits": {"maxJobs": 1000, "ratePerSecond": 25, "parallelism": 4},
|
||||
"subscribers": ["notify.ops"],
|
||||
"createdAt": "2025-10-18T22:00:00Z",
|
||||
"createdBy": "svc_scheduler",
|
||||
"updatedAt": "2025-10-18T22:00:00Z",
|
||||
"updatedBy": "svc_scheduler"
|
||||
}
|
||||
```
|
||||
|
||||
*Constraints*: arrays are alphabetically sorted; `selection.tenantId` is optional but when present must match `tenantId`. Cron expressions are validated for newline/length, timezones are validated via `TimeZoneInfo`.
|
||||
|
||||
#### 3.1.2 Run (`runs`)
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"_id": "run_20251018_0001",
|
||||
"tenantId": "tenant-alpha",
|
||||
"scheduleId": "sch_20251018a",
|
||||
"trigger": "conselier",
|
||||
"state": "running",
|
||||
"stats": {
|
||||
"candidates": 1280,
|
||||
"deduped": 910,
|
||||
"queued": 624,
|
||||
"completed": 310,
|
||||
"deltas": 42,
|
||||
"newCriticals": 7,
|
||||
"newHigh": 11,
|
||||
"newMedium": 18,
|
||||
"newLow": 6
|
||||
},
|
||||
"reason": {"conselierExportId": "exp-20251018-03"},
|
||||
"createdAt": "2025-10-18T22:03:14Z",
|
||||
"startedAt": "2025-10-18T22:03:20Z",
|
||||
"finishedAt": null,
|
||||
"error": null,
|
||||
"deltas": [
|
||||
{
|
||||
"imageDigest": "sha256:a1b2c3",
|
||||
"newFindings": 3,
|
||||
"newCriticals": 1,
|
||||
"newHigh": 1,
|
||||
"newMedium": 1,
|
||||
"newLow": 0,
|
||||
"kevHits": ["CVE-2025-0002"],
|
||||
"topFindings": [
|
||||
{
|
||||
"purl": "pkg:rpm/openssl@3.0.12-5.el9",
|
||||
"vulnerabilityId": "CVE-2025-0002",
|
||||
"severity": "critical",
|
||||
"link": "https://ui.internal/scans/sha256:a1b2c3"
|
||||
}
|
||||
],
|
||||
"attestation": {"uuid": "rekor-314", "verified": true},
|
||||
"detectedAt": "2025-10-18T22:03:21Z"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Counters are clamped to ≥0, timestamps are converted to UTC, and delta arrays are sorted (critical → info severity precedence, then vulnerability id). Missing `deltas` implies "no change" snapshots.
|
||||
|
||||
#### 3.1.3 Impact Snapshot (`impact_snapshots`)
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"selector": {
|
||||
"scope": "all-images",
|
||||
"tenantId": "tenant-alpha"
|
||||
},
|
||||
"images": [
|
||||
{
|
||||
"imageDigest": "sha256:f1e2d3",
|
||||
"registry": "registry.internal",
|
||||
"repository": "app/api",
|
||||
"namespaces": ["team-a"],
|
||||
"tags": ["prod"],
|
||||
"usedByEntrypoint": true,
|
||||
"labels": {"env": "prod"}
|
||||
}
|
||||
],
|
||||
"usageOnly": true,
|
||||
"generatedAt": "2025-10-18T22:02:58Z",
|
||||
"total": 412,
|
||||
"snapshotId": "impact-20251018-1"
|
||||
}
|
||||
```
|
||||
|
||||
Images are deduplicated and sorted by digest. Label keys are normalised to lowercase to avoid case‑sensitive duplicates during reconciliation. `snapshotId` enables run planners to compare subsequent snapshots for drift.
|
||||
|
||||
#### 3.1.4 Audit (`audit`)
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"_id": "audit_169754",
|
||||
"tenantId": "tenant-alpha",
|
||||
"category": "scheduler",
|
||||
"action": "pause",
|
||||
"occurredAt": "2025-10-18T22:10:00Z",
|
||||
"actor": {"actorId": "user_admin", "displayName": "Cluster Admin", "kind": "user"},
|
||||
"scheduleId": "sch_20251018a",
|
||||
"correlationId": "corr-123",
|
||||
"metadata": {"details": "schedule paused", "reason": "maintenance"},
|
||||
"message": "Paused via API"
|
||||
}
|
||||
```
|
||||
|
||||
Metadata keys are lowercased, first‑writer wins (duplicates with different casing are ignored), and optional IDs (`scheduleId`, `runId`) are trimmed when empty. Use the canonical serializer when emitting events so audit digests remain reproducible.
|
||||
|
||||
#### 3.1.5 Run Summary (`run_summaries`)
|
||||
|
||||
Materialized view powering the Scheduler UI dashboards. Stores the latest roll-up per schedule/tenant, enabling quick “last run” banners and sparkline counters without scanning the full `runs` collection.
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"tenantId": "tenant-alpha",
|
||||
"scheduleId": "sch_20251018a",
|
||||
"updatedAt": "2025-10-18T22:10:10Z",
|
||||
"lastRun": {
|
||||
"runId": "run_20251018_0001",
|
||||
"trigger": "conselier",
|
||||
"state": "completed",
|
||||
"createdAt": "2025-10-18T22:03:14Z",
|
||||
"startedAt": "2025-10-18T22:03:20Z",
|
||||
"finishedAt": "2025-10-18T22:08:45Z",
|
||||
"stats": {
|
||||
"candidates": 1280,
|
||||
"deduped": 910,
|
||||
"queued": 0,
|
||||
"completed": 910,
|
||||
"deltas": 42,
|
||||
"newCriticals": 7,
|
||||
"newHigh": 11,
|
||||
"newMedium": 18,
|
||||
"newLow": 6
|
||||
},
|
||||
"error": null
|
||||
},
|
||||
"recent": [
|
||||
{
|
||||
"runId": "run_20251018_0001",
|
||||
"trigger": "conselier",
|
||||
"state": "completed",
|
||||
"createdAt": "2025-10-18T22:03:14Z",
|
||||
"startedAt": "2025-10-18T22:03:20Z",
|
||||
"finishedAt": "2025-10-18T22:08:45Z",
|
||||
"stats": {
|
||||
"candidates": 1280,
|
||||
"deduped": 910,
|
||||
"queued": 0,
|
||||
"completed": 910,
|
||||
"deltas": 42,
|
||||
"newCriticals": 7,
|
||||
"newHigh": 11,
|
||||
"newMedium": 18,
|
||||
"newLow": 6
|
||||
},
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"runId": "run_20251017_0003",
|
||||
"trigger": "cron",
|
||||
"state": "error",
|
||||
"createdAt": "2025-10-17T22:01:02Z",
|
||||
"startedAt": "2025-10-17T22:01:08Z",
|
||||
"finishedAt": "2025-10-17T22:04:11Z",
|
||||
"stats": {
|
||||
"candidates": 1040,
|
||||
"deduped": 812,
|
||||
"queued": 0,
|
||||
"completed": 640,
|
||||
"deltas": 18,
|
||||
"newCriticals": 2,
|
||||
"newHigh": 4,
|
||||
"newMedium": 7,
|
||||
"newLow": 3
|
||||
},
|
||||
"error": "scanner timeout"
|
||||
}
|
||||
],
|
||||
"counters": {
|
||||
"total": 3,
|
||||
"planning": 0,
|
||||
"queued": 0,
|
||||
"running": 0,
|
||||
"completed": 1,
|
||||
"error": 1,
|
||||
"cancelled": 1,
|
||||
"totalDeltas": 60,
|
||||
"totalNewCriticals": 9,
|
||||
"totalNewHigh": 15,
|
||||
"totalNewMedium": 25,
|
||||
"totalNewLow": 9
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- `_id` combines `tenantId` and `scheduleId` (`tenant:schedule`).
|
||||
- `recent` contains the 20 most recent runs ordered by `createdAt` (UTC). Updates replace the existing entry for a run to respect state transitions.
|
||||
- `counters` aggregate over the retained window (20 runs) for quick trend indicators. Totals are recomputed after every update.
|
||||
- Schedulers should call the projection service after every run state change so the cache mirrors planner/runner progress.
|
||||
|
||||
Sample file: `samples/api/scheduler/run-summary.json`.
|
||||
|
||||
---
|
||||
|
||||
## 4 Policy Schema (YAML v1.0)
|
||||
|
||||
Minimal viable grammar (subset of OSV‑SCHEMA ideas).
|
||||
|
||||
```yaml
|
||||
version: "1.0"
|
||||
rules:
|
||||
- name: Block Critical
|
||||
severity: [Critical]
|
||||
action: block
|
||||
- name: Ignore Low Dev
|
||||
severity: [Low, None]
|
||||
environments: [dev, staging]
|
||||
action: ignore
|
||||
expires: "2026-01-01"
|
||||
- name: Escalate RegionalFeed High
|
||||
sources: [NVD, CNNVD, CNVD, ENISA, JVN, BDU]
|
||||
severity: [High, Critical]
|
||||
action: escalate
|
||||
```
|
||||
|
||||
Validation is performed by `policy:mapping.yaml` JSON‑Schema embedded in backend.
|
||||
|
||||
Canonical schema source: `src/Policy/__Libraries/StellaOps.Policy/Schemas/policy-schema@1.json` (embedded into `StellaOps.Policy`).
|
||||
`PolicyValidationCli` (see `src/Policy/__Libraries/StellaOps.Policy/PolicyValidationCli.cs`) provides the reusable command handler that the main CLI wires up; in the interim it can be invoked from a short host like:
|
||||
|
||||
```csharp
|
||||
await new PolicyValidationCli().RunAsync(new PolicyValidationCliOptions
|
||||
{
|
||||
Inputs = new[] { "policies/root.yaml" },
|
||||
Strict = true,
|
||||
});
|
||||
```
|
||||
|
||||
### 4.1 Rego Variant (Advanced – TODO)
|
||||
|
||||
*Accepted but stored as‑is in `rego` field.*
|
||||
Evaluated via internal **OPA** side‑car once feature graduates from TODO list.
|
||||
|
||||
### 4.2 Policy Scoring Config (JSON)
|
||||
|
||||
*Schema id.* `https://schemas.stella-ops.org/policy/policy-scoring-schema@1.json`
|
||||
*Source.* `src/Policy/__Libraries/StellaOps.Policy/Schemas/policy-scoring-schema@1.json` (embedded in `StellaOps.Policy`), default fixture at `src/Policy/__Libraries/StellaOps.Policy/Schemas/policy-scoring-default.json`.
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"version": "1.0",
|
||||
"severityWeights": {"Critical": 90, "High": 75, "Unknown": 60, "...": 0},
|
||||
"quietPenalty": 45,
|
||||
"warnPenalty": 15,
|
||||
"ignorePenalty": 35,
|
||||
"trustOverrides": {"vendor": 1.0, "distro": 0.85},
|
||||
"reachabilityBuckets": {"entrypoint": 1.0, "direct": 0.85, "runtime": 0.45, "unknown": 0.5},
|
||||
"unknownConfidence": {
|
||||
"initial": 0.8,
|
||||
"decayPerDay": 0.05,
|
||||
"floor": 0.2,
|
||||
"bands": [
|
||||
{"name": "high", "min": 0.65},
|
||||
{"name": "medium", "min": 0.35},
|
||||
{"name": "low", "min": 0.0}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Validation occurs alongside policy binding (`PolicyScoringConfigBinder`), producing deterministic digests via `PolicyScoringConfigDigest`. Bands are ordered descending by `min` so consumers can resolve confidence tiers deterministically. Reachability buckets are case-insensitive keys (`entrypoint`, `direct`, `indirect`, `runtime`, `unreachable`, `unknown`) with numeric multipliers (default ≤1.0).
|
||||
|
||||
**Runtime usage**
|
||||
- `trustOverrides` are matched against `finding.tags` (`trust:<key>`) first, then `finding.source`/`finding.vendor`; missing keys default to `1.0`.
|
||||
- `reachabilityBuckets` consume `finding.tags` with prefix `reachability:` (fallback `usage:` or `unknown`). Missing buckets fall back to `unknown` weight when present, otherwise `1.0`.
|
||||
- Policy verdicts expose scoring inputs (`severityWeight`, `trustWeight`, `reachabilityWeight`, `baseScore`, penalties) plus unknown-state metadata (`unknownConfidence`, `unknownAgeDays`, `confidenceBand`) for auditability. See `samples/policy/policy-preview-unknown.json` and `samples/policy/policy-report-unknown.json` for offline reference payloads validated against the published schemas below.
|
||||
|
||||
Validate the samples locally with **Ajv** before publishing changes:
|
||||
|
||||
```bash
|
||||
# install once per checkout (offline-safe):
|
||||
npm install --no-save ajv-cli@5 ajv-formats@2
|
||||
|
||||
npx ajv validate --spec=draft2020 -c ajv-formats \
|
||||
-s docs/modules/policy/schemas/policy-preview-sample@1.json \
|
||||
-d samples/policy/policy-preview-unknown.json
|
||||
|
||||
npx ajv validate --spec=draft2020 -c ajv-formats \
|
||||
-s docs/modules/policy/schemas/policy-report-sample@1.json \
|
||||
-d samples/policy/policy-report-unknown.json
|
||||
```
|
||||
- Unknown confidence derives from `unknown-age-days:` (preferred) or `unknown-since:` + `observed-at:` tags; with no hints the engine keeps `initial` confidence. Values decay by `decayPerDay` down to `floor`, then resolve to the first matching `bands[].name`.
|
||||
|
||||
---
|
||||
|
||||
## 5 SLSA Attestation Schema ⭑
|
||||
|
||||
Planned for Q1‑2026 (kept here for early plug‑in authors).
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"id": "prov_0291",
|
||||
"imageDigest": "sha256:e2b9…",
|
||||
"buildType": "https://slsa.dev/container/v1",
|
||||
"builder": {
|
||||
"id": "https://git.stella-ops.ru/ci/stella-runner@sha256:f7b7…"
|
||||
},
|
||||
"metadata": {
|
||||
"invocation": {
|
||||
"parameters": {"GIT_SHA": "f6a1…"},
|
||||
"buildStart": "2025-07-14T06:59:17Z",
|
||||
"buildEnd": "2025-07-14T07:01:22Z"
|
||||
},
|
||||
"completeness": {"parameters": true}
|
||||
},
|
||||
"materials": [
|
||||
{"uri": "git+https://git…", "digest": {"sha1": "f6a1…"}}
|
||||
],
|
||||
"rekorLogIndex": 99817 // entry in local Rekor mirror
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6 Notify Foundations (Rule · Channel · Event)
|
||||
|
||||
*Sprint 15 target* – canonically describe the Notify data shapes that UI, workers, and storage consume. JSON Schemas live under `docs/modules/notify/resources/schemas/` and deterministic fixtures under `docs/modules/notify/resources/samples/`.
|
||||
|
||||
| Artifact | Schema | Sample |
|
||||
|----------|--------|--------|
|
||||
| **Rule** (catalogued routing logic) | `docs/modules/notify/resources/schemas/notify-rule@1.json` | `docs/modules/notify/resources/samples/notify-rule@1.sample.json` |
|
||||
| **Channel** (delivery endpoint definition) | `docs/modules/notify/resources/schemas/notify-channel@1.json` | `docs/modules/notify/resources/samples/notify-channel@1.sample.json` |
|
||||
| **Template** (rendering payload) | `docs/modules/notify/resources/schemas/notify-template@1.json` | `docs/modules/notify/resources/samples/notify-template@1.sample.json` |
|
||||
| **Event envelope** (Notify ingest surface) | `docs/modules/notify/resources/schemas/notify-event@1.json` | `docs/modules/notify/resources/samples/notify-event@1.sample.json` |
|
||||
|
||||
### 6.1 Rule highlights (`notify-rule@1`)
|
||||
|
||||
* Keys are lower‑cased camelCase. `schemaVersion` (`notify.rule@1`), `ruleId`, `tenantId`, `name`, `match`, `actions`, `createdAt`, and `updatedAt` are mandatory.
|
||||
* `match.eventKinds`, `match.verdicts`, and other array selectors are pre‑sorted and case‑normalized (e.g. `scanner.report.ready`).
|
||||
* `actions[].throttle` serialises as ISO 8601 duration (`PT5M`), mirroring worker backoff guardrails.
|
||||
* `vex` gates let operators exclude accepted/not‑affected justifications; omit the block to inherit default behaviour.
|
||||
* Use `StellaOps.Notify.Models.NotifySchemaMigration.UpgradeRule(JsonNode)` when deserialising legacy payloads that might lack `schemaVersion` or retain older revisions.
|
||||
* Soft deletions persist `deletedAt` in PostgreSQL (and disable the rule); repository queries automatically filter them.
|
||||
|
||||
### 6.2 Channel highlights (`notify-channel@1`)
|
||||
|
||||
* `schemaVersion` is pinned to `notify.channel@1` and must accompany persisted documents.
|
||||
* `type` matches plug‑in identifiers (`slack`, `teams`, `email`, `webhook`, `custom`).
|
||||
* `config.secretRef` stores an external secret handle (Authority, Vault, K8s). Notify never persists raw credentials.
|
||||
* Optional `config.limits.timeout` uses ISO 8601 durations identical to rule throttles; concurrency/RPM defaults apply when absent.
|
||||
* `StellaOps.Notify.Models.NotifySchemaMigration.UpgradeChannel(JsonNode)` backfills the schema version when older documents omit it.
|
||||
* Channels share the same soft-delete marker (`deletedAt`) so operators can restore prior configuration without purging history.
|
||||
|
||||
### 6.3 Event envelope (`notify-event@1`)
|
||||
|
||||
* Aligns with the platform event contract—`eventId` UUID, RFC 3339 `ts`, tenant isolation enforced.
|
||||
* Enumerated `kind` covers the initial Notify surface (`scanner.report.ready`, `scheduler.rescan.delta`, `zastava.admission`, etc.).
|
||||
* `scope.labels`/`scope.attributes` and top-level `attributes` mirror the metadata dictionaries workers surface for templating and audits.
|
||||
* Notify workers use the same migration helper to wrap event payloads before template rendering, so schema additions remain additive.
|
||||
|
||||
### 6.4 Template highlights (`notify-template@1`)
|
||||
|
||||
* Carries the presentation key (`channelType`, `key`, `locale`) and the raw template body; `schemaVersion` is fixed to `notify.template@1`.
|
||||
* `renderMode` enumerates supported engines (`markdown`, `html`, `adaptiveCard`, `plainText`, `json`) aligning with `NotifyTemplateRenderMode`.
|
||||
* `format` signals downstream connector expectations (`slack`, `teams`, `email`, `webhook`, `json`).
|
||||
* Upgrade legacy definitions with `NotifySchemaMigration.UpgradeTemplate(JsonNode)` to auto-apply the new schema version and ordering.
|
||||
* Templates also record soft deletes via `deletedAt`; UI/API skip them by default while retaining revision history.
|
||||
|
||||
**Validation loop:**
|
||||
|
||||
```bash
|
||||
# Validate Notify schemas and samples (matches Docs CI)
|
||||
for schema in docs/modules/notify/resources/schemas/*.json; do
|
||||
npx ajv compile -c ajv-formats -s "$schema"
|
||||
done
|
||||
|
||||
for sample in docs/modules/notify/resources/samples/*.sample.json; do
|
||||
schema="docs/modules/notify/resources/schemas/$(basename "${sample%.sample.json}").json"
|
||||
npx ajv validate -c ajv-formats -s "$schema" -d "$sample"
|
||||
done
|
||||
```
|
||||
|
||||
Integration tests can embed the sample fixtures to guarantee deterministic serialisation from the `StellaOps.Notify.Models` DTOs introduced in Sprint 15.
|
||||
|
||||
---
|
||||
|
||||
## 6 Validator Contracts
|
||||
|
||||
* For SBOM wrapper – `ISbomValidator` (DLL plug‑in) must return *typed* error list.
|
||||
* For YAML policies – JSON‑Schema at `/schemas/policy‑v1.json`.
|
||||
* For Rego – OPA `opa eval --fail-defined` under the hood.
|
||||
* For **Free‑tier quotas** – `IQuotaService` integration tests ensure `quota:<token>` resets at UTC midnight and produces correct `Retry‑After` headers.
|
||||
|
||||
---
|
||||
|
||||
## 7 Migration Notes
|
||||
|
||||
1. **Add `format` column** to existing SBOM wrappers; default to `trivy-json-v2`.
|
||||
2. **Populate `layers` & `partial`** via backfill script (ship with `stellopsctl migrate` wizard).
|
||||
3. Policy YAML previously stored in Valkey → copy to PostgreSQL if persistence enabled.
|
||||
4. Prepare `attestations` table (empty) – safe to create in advance.
|
||||
|
||||
---
|
||||
|
||||
## 8 Open Questions / Future Work
|
||||
|
||||
* How to de‑duplicate *identical* Rego policies differing only in whitespace?
|
||||
* Embed *GOST 34.11‑2018* digests when users enable Russian crypto suite?
|
||||
* Should enterprise tiers share the same Valkey quota keys (Redis-compatible) or switch to JWT claim `tier != Free` bypass?
|
||||
* Evaluate sliding‑window quota instead of strict daily reset.
|
||||
* Consider rate‑limit for `/layers/missing` to avoid brute‑force enumeration.
|
||||
|
||||
---
|
||||
|
||||
## 9 Change Log
|
||||
|
||||
| Date | Note |
|
||||
|------------|--------------------------------------------------------------------------------|
|
||||
| 2025‑07‑14 | **Added:** `format`, `partial`, delta cache keys, YAML policy schema v1.0. |
|
||||
| 2025‑07‑12 | **Initial public draft** – SBOM wrapper, Valkey keyspace (Redis-compatible), audit collections. |
|
||||
|
||||
---
|
||||
170
docs/technical/PERFORMANCE_WORKBOOK.md
Normal file
170
docs/technical/PERFORMANCE_WORKBOOK.md
Normal file
@@ -0,0 +1,170 @@
|
||||
# 12 - Performance Workbook
|
||||
|
||||
*Purpose* – define **repeatable, data‑driven** benchmarks that guard Stella Ops’ core pledge:
|
||||
> *“P95 vulnerability feedback in ≤ 5 seconds.”*
|
||||
|
||||
---
|
||||
|
||||
## 0 Benchmark Scope
|
||||
|
||||
| Area | Included | Excluded |
|
||||
|------------------|----------------------------------|---------------------------|
|
||||
| SBOM‑first scan | Trivy engine w/ warmed DB | Full image unpack ≥ 300 MB |
|
||||
| Delta SBOM ⭑ | Missing‑layer lookup & merge | Multi‑arch images |
|
||||
| Policy eval ⭑ | YAML → JSON → rule match | Rego (until GA) |
|
||||
| Feed merge | NVD JSON 2023–2025 | GHSA GraphQL (plugin) |
|
||||
| Quota wait‑path | 5 s soft‑wait, 60 s hard‑wait behaviour | Paid tiers (unlimited) |
|
||||
| API latency | REST `/scan`, `/layers/missing` | UI SPA calls |
|
||||
|
||||
⭑ = new in July 2025.
|
||||
|
||||
---
|
||||
|
||||
## 1 Hardware Baseline (Reference Rig)
|
||||
|
||||
| Element | Spec |
|
||||
|-------------|------------------------------------|
|
||||
| CPU | 8 vCPU (Intel Ice‑Lake equiv.) |
|
||||
| Memory | 16 GiB |
|
||||
| Disk | NVMe SSD, 3 GB/s R/W |
|
||||
| Network | 1 Gbit virt. switch |
|
||||
| Container | Docker 25.0 + overlay2 |
|
||||
| OS | Ubuntu 22.04 LTS (kernel 6.8) |
|
||||
|
||||
*All P95 targets assume a **single‑node** deployment on this rig unless stated.*
|
||||
|
||||
---
|
||||
|
||||
## 2 Phase Targets & Gates
|
||||
|
||||
| Phase (ID) | Target P95 | Gate (CI) | Rationale |
|
||||
|-----------------------|-----------:|-----------|----------------------------------------|
|
||||
| **SBOM_FIRST** | ≤ 5 s | `hard` | Core UX promise. |
|
||||
| **IMAGE_UNPACK** | ≤ 10 s | `soft` | Fallback path for legacy flows. |
|
||||
| **DELTA_SBOM** ⭑ | ≤ 1 s | `hard` | Needed to stay sub‑5 s for big bases. |
|
||||
| **POLICY_EVAL** ⭑ | ≤ 50 ms | `hard` | Keeps gate latency invisible to users. |
|
||||
| **QUOTA_WAIT** ⭑ | *soft* ≤ 5 s<br>*hard* ≤ 60 s | `hard` | Ensures graceful Free‑tier throttling. |
|
||||
| **SCHED_RESCAN** | ≤ 30 s | `soft` | Nightly batch – not user‑facing. |
|
||||
| **FEED_MERGE** | ≤ 60 s | `soft` | Off‑peak cron @ 01:00. |
|
||||
| **API_P95** | ≤ 200 ms | `hard` | UI snappiness. |
|
||||
|
||||
*Gate* legend — `hard`: break CI if regression > 3 × target,
|
||||
`soft`: raise warning & issue ticket.
|
||||
|
||||
---
|
||||
|
||||
## 3 Test Harness
|
||||
|
||||
* **Runner** – `perf/run.sh`, accepts `--phase` and `--samples`.
|
||||
* **Language analyzers microbench** – `dotnet run --project src/Bench/StellaOps.Bench/Scanner.Analyzers/StellaOps.Bench.ScannerAnalyzers/StellaOps.Bench.ScannerAnalyzers.csproj -- --repo-root . --out src/Bench/StellaOps.Bench/Scanner.Analyzers/baseline.csv --json out/bench/scanner-analyzers/latest.json --prom out/bench/scanner-analyzers/latest.prom --commit $(git rev-parse HEAD)` produces CSV + JSON + Prometheus gauges for analyzer scenarios. Runs fail if `max_ms` regresses ≥ 20 % against `baseline.csv` or if thresholds are exceeded.
|
||||
* **Metrics** – Prometheus + `jq` extracts; aggregated via `scripts/aggregate.ts`.
|
||||
* **CI** – GitLab CI job *benchmark* publishes JSON to `bench‑artifacts/`.
|
||||
* **Visualisation** – Grafana dashboard *Stella‑Perf* (provisioned JSON).
|
||||
|
||||
> **Note** – harness mounts `/var/cache/trivy` tmpfs to avoid disk noise.
|
||||
|
||||
---
|
||||
|
||||
## 4 Current Results (July 2025)
|
||||
|
||||
| Phase | Samples | Mean (s) | P95 (s) | Target OK? |
|
||||
|---------------|--------:|---------:|--------:|-----------:|
|
||||
| SBOM_FIRST | 100 | 3.7 | 4.9 | ✅ |
|
||||
| IMAGE_UNPACK | 50 | 6.4 | 9.2 | ✅ |
|
||||
| **DELTA_SBOM**| 100 | 0.46 | 0.83 | ✅ |
|
||||
| **POLICY_EVAL** | 1 000 | 0.021 | 0.041 | ✅ |
|
||||
| **QUOTA_WAIT** | 80 | 4.0* | 4.9* | ✅ |
|
||||
| SCHED_RESCAN | 10 | 18.3 | 24.9 | ✅ |
|
||||
| FEED_MERGE | 3 | 38.1 | 41.0 | ✅ |
|
||||
| API_P95 | 20 000 | 0.087 | 0.143 | ✅ |
|
||||
|
||||
*Data files:* `bench-artifacts/2025‑07‑14/phase‑stats.json`.
|
||||
|
||||
---
|
||||
|
||||
## 5 Δ‑SBOM Micro‑Benchmark Detail
|
||||
|
||||
### 5.1 Scenario
|
||||
|
||||
1. Base image `python:3.12-slim` already scanned (all layers cached).
|
||||
2. Application layer (`COPY . /app`) triggers new digest.
|
||||
3. `Stella CLI` lists **7** layers, backend replies *6 hit*, *1 miss*.
|
||||
4. Builder scans **only 1 layer** (~9 MiB, 217 files) & uploads delta.
|
||||
|
||||
### 5.2 Key Timings
|
||||
|
||||
| Step | Time (ms) |
|
||||
|---------------------|----------:|
|
||||
| `/layers/missing` | 13 |
|
||||
| Trivy single layer | 655 |
|
||||
| Upload delta blob | 88 |
|
||||
| Backend merge + CVE | 74 |
|
||||
| **Total wall‑time** | **830 ms** |
|
||||
|
||||
---
|
||||
|
||||
## 6 Quota Wait‑Path Benchmark Detail
|
||||
|
||||
### 6.1 Scenario
|
||||
|
||||
1. Free‑tier token reaches **scan #200** – dashboard shows yellow banner.
|
||||
|
||||
### 6.2 Key Timings
|
||||
|
||||
| Step | Time (ms) |
|
||||
|------------------------------------|----------:|
|
||||
| `/quota/check` Valkey LUA INCR | 0.8 |
|
||||
| Soft wait sleep (server) | 5 000 |
|
||||
| Hard wait sleep (server) | 60 000 |
|
||||
| End‑to‑end wall‑time (soft‑hit) | 5 003 |
|
||||
| End‑to‑end wall‑time (hard‑hit) | 60 004 |
|
||||
|
||||
---
|
||||
## 7 Policy Eval Bench
|
||||
|
||||
### 7.1 Setup
|
||||
|
||||
* Policy YAML: **28** rules, mix severity & package conditions.
|
||||
* Input: scan result JSON with **1 026** findings.
|
||||
* Evaluator: custom rules engine (Go structs → map look‑ups).
|
||||
|
||||
### 7.2 Latency Histogram
|
||||
|
||||
```
|
||||
0‑10 ms ▇▇▇▇▇▇▇▇▇▇ 38 %
|
||||
10‑20 ms ▇▇▇▇▇▇▇▇▇▇ 42 %
|
||||
20‑40 ms ▇▇▇▇▇▇ 17 %
|
||||
40‑50 ms ▇ 3 %
|
||||
```
|
||||
|
||||
P99 = 48 ms. Meets 50 ms gate.
|
||||
|
||||
---
|
||||
|
||||
## 8 Trend Snapshot
|
||||
|
||||
> _Perf trend spark‑line screenshot pending upload._
|
||||
|
||||
> **Grafana/Alerting** – Import `docs/modules/scanner/operations/analyzers-grafana-dashboard.json` and point it at the Prometheus datasource storing `scanner_analyzer_bench_*` metrics. Configure an alert on `scanner_analyzer_bench_regression_ratio` ≥ 1.20 (default limit); the bundled Stat panel surfaces breached scenarios (non-zero values). On-call runbook: `docs/modules/scanner/operations/analyzers.md`.
|
||||
|
||||
_Plot generated weekly by `scripts/update‑trend.py`; shows last 12 weeks P95 per phase._
|
||||
|
||||
---
|
||||
|
||||
## 9 Action Items
|
||||
|
||||
1. **Image Unpack** – Evaluate zstd for layer decompress; aim to shave 1 s.
|
||||
2. **Feed Merge** – Parallelise regional XML feed parse (plugin) once stable.
|
||||
3. **Rego Support** – Prototype OPA side‑car; target ≤ 100 ms eval.
|
||||
4. **Concurrency** – Stress‑test 100 rps on 4‑node Valkey cluster (Redis-compatible) (Q4‑2025).
|
||||
|
||||
---
|
||||
|
||||
## 10 Change Log
|
||||
|
||||
| Date | Note |
|
||||
|------------|-------------------------------------------------------------------------|
|
||||
| 2025‑07‑14 | Added Δ‑SBOM & Policy Eval phases; updated targets & current results. |
|
||||
| 2025‑07‑12 | First public workbook (SBOM‑first, image‑unpack, feed merge). |
|
||||
|
||||
---
|
||||
204
docs/technical/SYSTEM_REQUIREMENTS_SPEC.md
Executable file
204
docs/technical/SYSTEM_REQUIREMENTS_SPEC.md
Executable file
@@ -0,0 +1,204 @@
|
||||
# SYSTEM REQUIREMENTS SPECIFICATION
|
||||
Stella Ops · self‑hosted supply‑chain‑security platform
|
||||
|
||||
> **Audience** – core maintainers and external contributors who need an
|
||||
> authoritative checklist of *what* the software must do (functional
|
||||
> requirements) and *how well* it must do it (non‑functional
|
||||
> requirements). Implementation details belong in Module Specs
|
||||
> or ADRs—**not here**.
|
||||
|
||||
---
|
||||
|
||||
## 1 · Purpose & Scope
|
||||
|
||||
This SRS defines everything the **v0.1.0‑alpha** release of _Stella Ops_ must do, **including the Free‑tier daily quota of {{ quota_token }} SBOM scans per token**.
|
||||
Scope includes core platform, CLI, UI, quota layer, and plug‑in host; commercial or closed‑source extensions are explicitly out‑of‑scope.
|
||||
|
||||
---
|
||||
|
||||
## 2 · References
|
||||
|
||||
* [overview.md](overview.md) – market gap & problem statement
|
||||
* [VISION.md](VISION.md) – north‑star, KPIs, quarterly themes
|
||||
* [ARCHITECTURE_OVERVIEW.md](ARCHITECTURE_OVERVIEW.md) – context & data flow diagrams
|
||||
* [modules/platform/architecture-overview.md](modules/platform/architecture-overview.md) – component APIs & plug‑in contracts
|
||||
* [API_CLI_REFERENCE.md](API_CLI_REFERENCE.md) – REST & CLI surface
|
||||
|
||||
---
|
||||
|
||||
## 3 · Definitions & Acronyms
|
||||
|
||||
| Term | Meaning |
|
||||
|------|---------|
|
||||
| **SBOM** | Software Bill of Materials |
|
||||
| **Delta SBOM** | Partial SBOM covering only image layers not previously analysed |
|
||||
| **Registry** | Anonymous, read‑only Docker Registry v2 hosted internally |
|
||||
| **OPA** | Open Policy Agent (Rego policy engine) |
|
||||
| **Muting Policy** | Rule that downgrades or ignores specific findings |
|
||||
| **SLSA** | Supply‑chain Levels for Software Artifacts (provenance framework) |
|
||||
| **Rekor** | Sigstore transparency log for signatures |
|
||||
|
||||
---
|
||||
|
||||
## 4 · Overall System Description
|
||||
|
||||
The platform consists of:
|
||||
|
||||
* **Stella Ops Backend** – REST API, queue, policy engine, DB.
|
||||
* **StellaOps.Registry** – internal container registry for agents.
|
||||
* **Stella CLI** – extracts SBOMs; supports multi‑format & delta.
|
||||
* **Zastava Agent** – enforcement hook for admission‑control scenarios.
|
||||
* **Web UI** – Angular 17 SPA consuming backend APIs.
|
||||
* **Plug‑ins** – hot‑load binaries extending scanners, attestations, etc.
|
||||
|
||||
All services run in Docker Compose or Kubernetes with optional Internet
|
||||
access.
|
||||
|
||||
---
|
||||
|
||||
## 5 · Functional Requirements (FR)
|
||||
|
||||
### 5.1 Core Scanning
|
||||
|
||||
| ID | Requirement | Priority | Verification |
|
||||
|----|-------------|----------|--------------|
|
||||
| F‑1 | System SHALL ingest **Trivy‑JSON, SPDX‑JSON, CycloneDX‑JSON** files. | MUST | UT‑SBOM‑001 |
|
||||
| F‑2 | System SHALL **auto‑detect** SBOM type when `sbomType` param omitted. | MUST | UT‑SBOM‑002 |
|
||||
| F‑3 | System SHALL **cache analysed layers** and reuse them in subsequent scans. | MUST | IT‑CACHE‑001 |
|
||||
| F‑4 | System SHALL **enforce a soft limit of {{ quota_token }} scans per token per UTC day**. | MUST | IT‑QUOTA‑001 |
|
||||
| F‑4a | Remaining quota SHALL be **persisted in Valkey** under key `quota:<token>:<yyyy‑mm‑dd>`. | MUST | UT‑QUOTA‑VALKEY |
|
||||
| F‑4b | Exhausted quota SHALL trigger **HTTP 429** with `Retry‑After` header (UTC midnight). | MUST | IT‑QUOTA‑002 |
|
||||
| F‑4c | When quota is ≤ 40 % remaining, **UI banner** MUST turn yellow and show count‑down. | SHOULD | UI‑E2E‑005 |
|
||||
| F‑4d | `/quota` endpoint SHALL return JSON `{"limit":{{ quota_token }} ,"remaining":N,"resetsAt":"<ISO‑8601>"}`. | SHOULD | API‑DOC‑003 |
|
||||
| F‑5 | Policy engine SHALL evaluate **YAML rules** against scan results. | MUST | UT‑POL‑001 |
|
||||
| F‑6 | Hot‑pluggable .NET plug‑ins SHALL be loadable **without service restart**. | MUST | IT‑PLUGIN‑001 |
|
||||
| F‑7 | CLI (`stella scan`) SHOULD exit **non‑zero** when CVSS≥7 vulnerabilities found. | SHOULD | CL‑INT‑003 |
|
||||
| *(… all previously documented F‑8 – F‑12 rows retained unchanged …)* |
|
||||
|
||||
|
||||
### 5.2 Internal Docker Repository
|
||||
|
||||
| Ref | Requirement |
|
||||
|-----|-------------|
|
||||
| **FR‑REPO‑1** | Platform SHALL include **StellaOps.Registry** exposing Docker Registry v2 API (ports 5000/443). |
|
||||
| **FR‑REPO‑2** | Registry SHALL allow anonymous, *read‑only* pulls for at least three images:<br>• `stella/sbom‑builder`<br>• `stella/cli`<br>• `stella/zastava`. |
|
||||
| **FR‑REPO‑3** | Registry MAY enable optional basic‑auth without code changes. |
|
||||
|
||||
### 5.3 SBOM Generation & Handling
|
||||
|
||||
| Ref | Requirement |
|
||||
|-----|-------------|
|
||||
| **FR‑SBOM‑1** | SBOM builder SHALL produce Trivy‑JSON **and** at least one additional format: SPDX‑JSON and CycloneDX‑JSON. |
|
||||
| **FR‑SBOM‑2** | For every generated SBOM, builder SHALL create a side‑car file `<image>.sbom.type` containing the format identifier. |
|
||||
| **FR‑SBOM‑3** | Stella CLI SHALL read the `.sbom.type` file and include `sbomType` parameter when uploading. |
|
||||
| **FR‑SBOM‑4** | Backend SHALL auto‑detect SBOM type when parameter is missing. |
|
||||
| **FR‑SBOM‑5** | UI Settings SHALL expose a dropdown to select default SBOM format (system‑wide fallback). |
|
||||
|
||||
#### 5.3.1 Delta SBOM (layer reuse)
|
||||
|
||||
| Ref | Requirement |
|
||||
|-----|-------------|
|
||||
| **FR‑DELTA‑1** | Builder SHALL compute SHA256 digests of each image layer and POST array to `/layers/missing`; response time ≤ 20 ms (P95). |
|
||||
| **FR‑DELTA‑2** | Builder SHALL generate SBOM **only** for layers returned as “missing”. |
|
||||
| **FR‑DELTA‑3** | End‑to‑end warm scan time (image differing by ≤ 2 layers) SHALL be ≤ 1 s (P95). |
|
||||
|
||||
### 5.4 Policy as Code (Muting & Expiration)
|
||||
|
||||
| Ref | Requirement |
|
||||
|-----|-------------|
|
||||
| **FR‑POLICY‑1** | Backend SHALL store policies as YAML by default, convertible to Rego for advanced use‑cases. |
|
||||
| **FR‑POLICY‑2** | Each policy change SHALL create an immutable history record (timestamp, actor, diff). |
|
||||
| **FR‑POLICY‑3** | REST endpoints `/policy/import`, `/policy/export`, `/policy/validate` SHALL accept YAML or Rego payloads. |
|
||||
| **FR‑POLICY‑4** | Web UI Policies tab SHALL provide Monaco editor with linting for YAML and Rego. |
|
||||
| **FR‑POLICY‑5** | **StellaOps.MutePolicies** module SHALL expose CLI `stella policies apply --file scan‑policy.yaml`. |
|
||||
|
||||
### 5.5 SLSA Attestations & Rekor (TODO > 6 mo)
|
||||
|
||||
| Ref | Requirement |
|
||||
|-----|-------------|
|
||||
| **FR‑SLSA‑1** | **TODO** – Generate provenance in SLSA‑Provenance v0.2 for each SBOM. |
|
||||
| **FR‑REKOR‑1** | **TODO** – Sign SBOM hashes and upload to local Rekor mirror; verify during scan. |
|
||||
|
||||
### 5.6 CLI & API Interface
|
||||
|
||||
| Ref | Requirement |
|
||||
|-----|-------------|
|
||||
| **FR‑CLI‑1** | CLI `stella scan` SHALL accept `--sbom-type {trivy,spdx,cyclonedx,auto}`. |
|
||||
| **FR‑API‑1** | API `/scan` SHALL accept `sbomType` query/body field (optional). |
|
||||
| **FR‑API‑2** | API `/layers/missing` SHALL accept JSON array of digests and return JSON array of missing digests. |
|
||||
|
||||
---
|
||||
|
||||
## 6 · Non‑Functional Requirements (NFR)
|
||||
|
||||
| Ref | Category | Requirement |
|
||||
|-----|----------|-------------|
|
||||
| **NFR‑PERF‑1** | Performance | P95 cold scan ≤ 5 s; warm ≤ 1 s (see **FR‑DELTA‑3**). |
|
||||
| **NFR‑PERF‑2** | Throughput | System shall sustain 60 concurrent scans on 8‑core node without queue depth >10. |
|
||||
| **NFR‑AVAIL‑1** | Availability | All services shall start offline; any Internet call must be optional. |
|
||||
| **NFR-SCAL-1** | Scalability | Horizontal scaling via Kubernetes replicas for backend, Valkey cluster, PostgreSQL cluster. |
|
||||
| **NFR‑SEC‑1** | Security | All inter‑service traffic shall use TLS or localhost sockets. |
|
||||
| **NFR‑COMP‑1** | Compatibility | Platform shall run on x86‑64 Linux kernel ≥ 5.10; Windows agents (TODO > 6 mo) must support Server 2019+. |
|
||||
| **NFR‑I18N‑1** | Internationalisation | UI must support EN and at least one additional locale (Cyrillic). |
|
||||
| **NFR‑OBS‑1** | Observability | Export Prometheus metrics for scan duration, queue length, policy eval duration. |
|
||||
|
||||
---
|
||||
|
||||
## 7 Acceptance Criteria <a id="7-acceptance-criteria"></a>
|
||||
|
||||
1. Issue {{ quota_token }} `/scan` calls; next returns random slow down and `Retry‑After`.
|
||||
2. Valkey failure during test → API returns **0 remaining** & warns in logs.
|
||||
3. UI banner activates at 133 remaining; clears next UTC midnight.
|
||||
|
||||
---
|
||||
## 8 · System Interfaces
|
||||
|
||||
### 8.1 External APIs
|
||||
|
||||
*(This is the complete original table, plus new `/quota` row.)*
|
||||
|
||||
| Path | Method | Auth | Quota | Description |
|
||||
|------|--------|------|-------|-------------|
|
||||
| `/scan` | POST | Bearer | ✅ | Submit SBOM or `imageRef` for scanning. |
|
||||
| `/quota` | GET | Bearer | ❌ | Return remaining quota for current token. |
|
||||
| `/policy/rules` | GET/PUT | Bearer+RBAC | ❌ | CRUD YAML or Rego policies. |
|
||||
| `/plugins` | POST/GET | Bearer+Admin | ❌ | Upload or list plug‑ins. |
|
||||
|
||||
```bash
|
||||
GET /quota
|
||||
Authorization: Bearer <token>
|
||||
|
||||
200 OK
|
||||
{
|
||||
"limit": {{ quota_token }},
|
||||
"remaining": 121,
|
||||
"resetsAt": "2025-07-14T23:59:59Z"
|
||||
}
|
||||
```
|
||||
|
||||
## 9 · Assumptions & Constraints
|
||||
|
||||
* Hardware reference: 8 vCPU, 8 GB RAM, NVMe SSD.
|
||||
* PostgreSQL and Valkey run co-located unless horizontal scaling enabled.
|
||||
* All docker images tagged `latest` are immutable (CI process locks digests).
|
||||
* Policy evaluation uses native `stella-dsl@1` DSL implemented in .NET; OPA/Rego integration available for Enterprise tier via external adapter.
|
||||
|
||||
---
|
||||
|
||||
## 10 · Future Work (Beyond 12 Months)
|
||||
|
||||
* Rekor transparency log cross‑cluster replication.
|
||||
* AI‑assisted false‑positive triage plug‑in.
|
||||
* Cluster‑wide injection for live runtime scanning.
|
||||
|
||||
---
|
||||
|
||||
## 11 · Revision History
|
||||
|
||||
| Version | Date | Notes |
|
||||
|---------|------|-------|
|
||||
| **v1.2** | 11‑Jul‑2025 | Commercial references removed; plug‑in contract (§ 3.3) and new NFR categories added; added User Classes & Traceability. |
|
||||
| v1.1 | 11‑Jul‑2025 | Split out RU‑specific items; OSS scope |
|
||||
| v1.0 | 09‑Jul‑2025 | Original unified SRS |
|
||||
|
||||
*(End of System Requirements Specification v1.2‑core)*
|
||||
@@ -0,0 +1,5 @@
|
||||
# High Level Architecture (Compatibility Alias)
|
||||
|
||||
This file is retained to keep older references working.
|
||||
For the current high-level architecture overview, see `docs/ARCHITECTURE_OVERVIEW.md`.
|
||||
For the detailed reference map, see `docs/ARCHITECTURE_REFERENCE.md`.
|
||||
@@ -122,7 +122,7 @@ This report validates that **StellaOps achieves 90%+ alignment** with the refere
|
||||
|
||||
**Evidence:**
|
||||
- `src/Scanner/__Libraries/StellaOps.Scanner.Storage/Epss/`
|
||||
- `docs/architecture/epss-versioning-clarification.md`
|
||||
- `docs/technical/architecture/epss-versioning-clarification.md`
|
||||
|
||||
---
|
||||
|
||||
@@ -240,7 +240,7 @@ This report validates that **StellaOps achieves 90%+ alignment** with the refere
|
||||
|
||||
**Evidence:**
|
||||
- `src/Scanner/__Libraries/StellaOps.Scanner.Unknowns/`
|
||||
- `docs/architecture/signal-contract-mapping.md` (Signal-14 section)
|
||||
- `docs/technical/architecture/signal-contract-mapping.md` (Signal-14 section)
|
||||
|
||||
---
|
||||
|
||||
@@ -295,5 +295,5 @@ StellaOps demonstrates **100% alignment** with the reference advisory architectu
|
||||
- [in-toto Attestation Framework](https://github.com/in-toto/attestation)
|
||||
- [FIRST.org EPSS](https://www.first.org/epss/)
|
||||
- [OpenVEX Specification](https://github.com/openvex/spec)
|
||||
- `docs/architecture/signal-contract-mapping.md`
|
||||
- `docs/architecture/epss-versioning-clarification.md`
|
||||
- `docs/technical/architecture/signal-contract-mapping.md`
|
||||
- `docs/technical/architecture/epss-versioning-clarification.md`
|
||||
|
||||
@@ -22,7 +22,7 @@ Concise descriptions of every top-level component under `src/`, summarising the
|
||||
|
||||
## Policy & Governance
|
||||
- **Policy** — Policy Engine core libraries and services executing lattice logic across SBOM, advisory, and VEX evidence. Emits explain traces, drives Findings, Notifier, and Export Center (`docs/modules/policy/architecture.md`).
|
||||
- **Policy Studio / TaskRunner / PacksRegistry** — Authoring, automation, and reusable template services that orchestrate policy and operational workflows (`docs/task-packs/`, `docs/modules/cli/`, `docs/modules/ui/`).
|
||||
- **Policy Studio / TaskRunner / PacksRegistry** - Authoring, automation, and reusable template services that orchestrate policy and operational workflows (`docs/modules/packs-registry/guides/`, `docs/modules/cli/`, `docs/modules/ui/`).
|
||||
- **Governance components** (Authority scopes, Policy governance, Console policy UI) are covered in `docs/security/policy-governance.md` and `docs/modules/ui/policies.md`.
|
||||
|
||||
## Identity, Signing & Provenance
|
||||
@@ -35,7 +35,7 @@ Concise descriptions of every top-level component under `src/`, summarising the
|
||||
## Scheduling, Orchestration & Automation
|
||||
- **Scheduler** — Detects advisory/VEX deltas and orchestrates deterministic rescan runs toward Scanner and Policy Engine (`docs/modules/scheduler/architecture.md`).
|
||||
- **Orchestrator** — Central coordination service dispatching jobs (scans, exports, policy runs) to modules, working closely with Scheduler, CLI, and UI (`docs/modules/orchestrator/architecture.md`).
|
||||
- **TaskRunner** — Executes automation packs sourced from PacksRegistry, integrating with Orchestrator, CLI, Notify, and Authority (`docs/task-packs/runbook.md`).
|
||||
- **TaskRunner** - Executes automation packs sourced from PacksRegistry, integrating with Orchestrator, CLI, Notify, and Authority (`docs/modules/packs-registry/guides/runbook.md`).
|
||||
- **Signals** — Ingests runtime posture signals and feeds Policy/Notifier workflows (`docs/modules/zastava/architecture.md`, signals sections).
|
||||
- **TimelineIndexer** — Builds timelines of evidence/events for forensics and audit tooling (`docs/modules/timeline-indexer/guides/timeline.md`).
|
||||
|
||||
|
||||
@@ -264,7 +264,7 @@ current EPSS methodology from FIRST.org. EPSS does not use numbered versions lik
|
||||
Instead, EPSS scores are tracked by daily `model_date`. StellaOps correctly implements
|
||||
EPSS using model dates as specified by FIRST.org.
|
||||
|
||||
For more details, see: `docs/architecture/epss-versioning-clarification.md`
|
||||
For more details, see: `docs/technical/architecture/epss-versioning-clarification.md`
|
||||
```
|
||||
|
||||
---
|
||||
@@ -431,7 +431,7 @@ private double CalculateExploitPressure(UnknownRanking ranking)
|
||||
## Related Documents
|
||||
|
||||
- `docs/implplan/SPRINT_5000_0001_0001_advisory_alignment.md` - Parent sprint
|
||||
- `docs/architecture/signal-contract-mapping.md` - Signal contract mapping
|
||||
- `docs/technical/architecture/signal-contract-mapping.md` - Signal contract mapping
|
||||
- `docs/modules/risk-engine/guides/epss-integration-v4.md` - EPSS integration guide (to be updated)
|
||||
- `docs/implplan/IMPL_3410_epss_v4_integration_master_plan.md` - EPSS implementation plan (to be updated)
|
||||
- `docs/modules/risk-engine/guides/formulas.md` - Scoring formulas including EPSS
|
||||
|
||||
331
docs/technical/reproducibility.md
Normal file
331
docs/technical/reproducibility.md
Normal file
@@ -0,0 +1,331 @@
|
||||
# StellaOps Reproducibility Specification
|
||||
|
||||
This document defines the reproducibility guarantees, verdict identity computation, and replay procedures for StellaOps artifacts.
|
||||
|
||||
## Overview
|
||||
|
||||
StellaOps provides **deterministic, reproducible outputs** for all security artifacts:
|
||||
- SBOM generation (CycloneDX 1.6, SPDX 3.0.1)
|
||||
- VEX statements (OpenVEX)
|
||||
- Policy verdicts
|
||||
- Delta computations
|
||||
- DSSE attestations and Sigstore bundles
|
||||
|
||||
**Core Guarantee:** Given identical inputs (image digest, advisory feeds, policies, tool versions), StellaOps produces byte-for-byte identical outputs with matching content-addressed identifiers.
|
||||
|
||||
## Verdict Identity Formula
|
||||
|
||||
### Content-Addressed Verdict ID
|
||||
|
||||
All policy verdicts use content-addressed identifiers computed as:
|
||||
|
||||
```
|
||||
VerdictId = "verdict:sha256:" + HexLower(SHA256(CanonicalJson(VerdictPayload)))
|
||||
```
|
||||
|
||||
Where `VerdictPayload` is a JSON object with the following structure:
|
||||
|
||||
```json
|
||||
{
|
||||
"_canonVersion": "stella:canon:v1",
|
||||
"deltaId": "<content-addressed delta ID>",
|
||||
"blockingDrivers": [
|
||||
{
|
||||
"cveId": "CVE-...",
|
||||
"description": "...",
|
||||
"purl": "pkg:...",
|
||||
"severity": "Critical|High|Medium|Low",
|
||||
"type": "new-reachable-cve|..."
|
||||
}
|
||||
],
|
||||
"warningDrivers": [...],
|
||||
"appliedExceptions": ["EXCEPTION-001", ...],
|
||||
"gateLevel": "G0|G1|G2|G3|G4"
|
||||
}
|
||||
```
|
||||
|
||||
**Determinism guarantees:**
|
||||
- `blockingDrivers` and `warningDrivers` are sorted by `type`, then `cveId`, then `purl`, then `severity`
|
||||
- `appliedExceptions` are sorted lexicographically
|
||||
- All string comparisons use Ordinal (case-sensitive, lexicographic)
|
||||
- Canonical JSON follows RFC 8785 (JCS) with keys sorted alphabetically
|
||||
- The `_canonVersion` field ensures hash stability across algorithm evolution
|
||||
|
||||
### VerdictIdGenerator Implementation
|
||||
|
||||
The `VerdictIdGenerator` class in `StellaOps.Policy.Deltas` computes deterministic verdict IDs:
|
||||
|
||||
```csharp
|
||||
// Create a verdict with content-addressed ID
|
||||
var verdict = new DeltaVerdictBuilder()
|
||||
.AddBlockingDriver(new DeltaDriver
|
||||
{
|
||||
Type = "new-reachable-cve",
|
||||
CveId = "CVE-2024-001",
|
||||
Severity = DeltaDriverSeverity.Critical,
|
||||
Description = "Critical CVE is now reachable"
|
||||
})
|
||||
.Build("delta:sha256:abc123...");
|
||||
|
||||
// VerdictId is deterministic:
|
||||
// verdict.VerdictId == "verdict:sha256:..."
|
||||
|
||||
// Recompute for verification:
|
||||
var generator = new VerdictIdGenerator();
|
||||
var recomputed = generator.ComputeVerdictId(verdict);
|
||||
Debug.Assert(recomputed == verdict.VerdictId);
|
||||
```
|
||||
|
||||
### Input Stamps
|
||||
|
||||
Every artifact includes `InputStamps` capturing the provenance of all inputs:
|
||||
|
||||
```json
|
||||
{
|
||||
"feedSnapshotHash": "sha256:abc123...",
|
||||
"policyManifestHash": "sha256:def456...",
|
||||
"sourceCodeHash": "sha256:789ghi...",
|
||||
"baseImageDigest": "sha256:jkl012...",
|
||||
"vexDocumentHashes": ["sha256:mno345..."],
|
||||
"toolchainVersion": "1.0.0",
|
||||
"custom": {}
|
||||
}
|
||||
```
|
||||
|
||||
### Determinism Manifest
|
||||
|
||||
The `DeterminismManifest` (schema v1.0) tracks artifact reproducibility:
|
||||
|
||||
```json
|
||||
{
|
||||
"schemaVersion": "1.0",
|
||||
"artifact": {
|
||||
"type": "verdict",
|
||||
"name": "scan-verdict",
|
||||
"version": "2025-12-24T12:00:00Z",
|
||||
"format": "StellaOps.DeltaVerdict@1"
|
||||
},
|
||||
"canonicalHash": {
|
||||
"algorithm": "SHA-256",
|
||||
"value": "abc123def456...",
|
||||
"encoding": "hex"
|
||||
},
|
||||
"inputs": {
|
||||
"feedSnapshotHash": "sha256:...",
|
||||
"policyManifestHash": "sha256:...",
|
||||
"baseImageDigest": "sha256:..."
|
||||
},
|
||||
"toolchain": {
|
||||
"platform": ".NET 10.0",
|
||||
"components": [
|
||||
{"name": "StellaOps.Scanner", "version": "1.0.0"},
|
||||
{"name": "StellaOps.Policy", "version": "1.0.0"}
|
||||
]
|
||||
},
|
||||
"reproducibility": {
|
||||
"clockFixed": true,
|
||||
"orderingGuarantee": "stable-sort",
|
||||
"normalizationRules": ["UTF-8", "LF", "canonical-json"]
|
||||
},
|
||||
"generatedAt": "2025-12-24T12:00:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
## Canonical JSON Serialization
|
||||
|
||||
All JSON outputs follow RFC 8785 (JSON Canonicalization Scheme):
|
||||
1. Keys sorted lexicographically
|
||||
2. No whitespace between tokens
|
||||
3. Unicode escaping for non-ASCII
|
||||
4. Numbers without leading zeros
|
||||
5. UTF-8 encoding
|
||||
|
||||
## DSSE Attestation Format
|
||||
|
||||
### Envelope Structure
|
||||
|
||||
```json
|
||||
{
|
||||
"payloadType": "application/vnd.in-toto+json",
|
||||
"payload": "<base64url-encoded statement>",
|
||||
"signatures": [
|
||||
{
|
||||
"keyid": "sha256:...",
|
||||
"sig": "<base64url-encoded signature>"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### In-toto Statement
|
||||
|
||||
```json
|
||||
{
|
||||
"_type": "https://in-toto.io/Statement/v1",
|
||||
"subject": [
|
||||
{
|
||||
"name": "registry.example.com/image:tag",
|
||||
"digest": {"sha256": "abc123..."}
|
||||
}
|
||||
],
|
||||
"predicateType": "https://stellaops.io/attestation/verdict/v1",
|
||||
"predicate": {
|
||||
"verdictId": "sha256:...",
|
||||
"status": "pass",
|
||||
"gate": "G2",
|
||||
"inputs": {...},
|
||||
"evidence": [...]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Sigstore Bundle Format
|
||||
|
||||
StellaOps produces Sigstore bundles (v0.3) for offline verification:
|
||||
|
||||
```json
|
||||
{
|
||||
"$mediaType": "application/vnd.dev.sigstore.bundle.v0.3+json",
|
||||
"verificationMaterial": {
|
||||
"certificate": {...},
|
||||
"tlogEntries": [{
|
||||
"logIndex": "12345",
|
||||
"logId": {...},
|
||||
"inclusionProof": {...}
|
||||
}]
|
||||
},
|
||||
"dsseEnvelope": {...}
|
||||
}
|
||||
```
|
||||
|
||||
## Replay Procedure
|
||||
|
||||
### Prerequisites
|
||||
1. Offline bundle containing:
|
||||
- Advisory feed snapshot
|
||||
- Policy pack
|
||||
- VEX documents
|
||||
- Tool binaries (pinned versions)
|
||||
|
||||
### Steps
|
||||
|
||||
```bash
|
||||
# 1. Extract offline bundle
|
||||
stella offline extract --bundle offline-kit.tar.gz --output ./replay
|
||||
|
||||
# 2. Set deterministic environment
|
||||
export STELLAOPS_DETERMINISTIC_SEED=42
|
||||
export STELLAOPS_CLOCK_FIXED=2025-12-24T12:00:00Z
|
||||
|
||||
# 3. Run scan with pinned inputs
|
||||
stella scan \
|
||||
--image registry.example.com/image@sha256:abc123 \
|
||||
--feeds ./replay/feeds \
|
||||
--policies ./replay/policies \
|
||||
--output ./replay/output
|
||||
|
||||
# 4. Verify hash matches original
|
||||
stella verify \
|
||||
--manifest ./replay/output/manifest.json \
|
||||
--expected-hash sha256:def456...
|
||||
|
||||
# 5. Verify DSSE attestation
|
||||
stella attest verify \
|
||||
--bundle ./replay/output/bundle.sigstore \
|
||||
--policy verification-policy.yaml
|
||||
```
|
||||
|
||||
### Verification Policy
|
||||
|
||||
```yaml
|
||||
apiVersion: stellaops.io/v1
|
||||
kind: VerificationPolicy
|
||||
metadata:
|
||||
name: audit-verification
|
||||
spec:
|
||||
requiredPredicateTypes:
|
||||
- https://stellaops.io/attestation/verdict/v1
|
||||
trustedIssuers:
|
||||
- https://accounts.stellaops.io
|
||||
maxAge: 90d
|
||||
requireRekorEntry: true
|
||||
unknownBudget:
|
||||
maxTotal: 5
|
||||
action: fail
|
||||
```
|
||||
|
||||
## Unknown Budget Attestation
|
||||
|
||||
Policy thresholds are attested in verdict bundles:
|
||||
|
||||
```json
|
||||
{
|
||||
"predicateType": "https://stellaops.io/attestation/budget-check/v1",
|
||||
"predicate": {
|
||||
"environment": "production",
|
||||
"budgetConfig": {
|
||||
"maxUnknownCount": 5,
|
||||
"maxCumulativeUncertainty": 2.0,
|
||||
"reasonLimits": {
|
||||
"Reachability": 0,
|
||||
"Identity": 2,
|
||||
"Provenance": 2
|
||||
}
|
||||
},
|
||||
"actualCounts": {
|
||||
"total": 3,
|
||||
"byReason": {"Identity": 2, "Provenance": 1}
|
||||
},
|
||||
"result": "pass",
|
||||
"configHash": "sha256:..."
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Schema Versions
|
||||
|
||||
| Format | Version | Schema Location |
|
||||
|--------|---------|-----------------|
|
||||
| CycloneDX | 1.6 | `docs/modules/sbom-service/schemas/cyclonedx-bom-1.6.schema.json` |
|
||||
| SPDX | 3.0.1 | `docs/modules/sbom-service/schemas/spdx-3.0.1.schema.json` |
|
||||
| OpenVEX | 0.2.0 | `docs/modules/excititor/schemas/openvex-0.2.0.schema.json` |
|
||||
| Sigstore Bundle | 0.3 | `docs/modules/attestor/schemas/sigstore-bundle-0.3.schema.json` |
|
||||
| DeterminismManifest | 1.0 | `docs/modules/replay/schemas/determinism-manifest-1.0.schema.json` |
|
||||
|
||||
## CI Integration
|
||||
|
||||
### Schema Validation
|
||||
|
||||
```yaml
|
||||
# .gitea/workflows/schema-validation.yml
|
||||
- name: Validate CycloneDX
|
||||
run: |
|
||||
sbom-utility validate \
|
||||
--input-file ${{ matrix.fixture }} \
|
||||
--schema docs/modules/sbom-service/schemas/cyclonedx-bom-1.6.schema.json
|
||||
```
|
||||
|
||||
### Determinism Gate
|
||||
|
||||
```yaml
|
||||
# .gitea/workflows/determinism-gate.yml
|
||||
- name: Verify Verdict Hash
|
||||
run: |
|
||||
HASH1=$(stella scan --image test:latest --output /tmp/run1 | jq -r '.verdictId')
|
||||
HASH2=$(stella scan --image test:latest --output /tmp/run2 | jq -r '.verdictId')
|
||||
[ "$HASH1" = "$HASH2" ] || exit 1
|
||||
```
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Testing Strategy](testing/testing-strategy-models.md)
|
||||
- [Determinism Verification](testing/determinism-verification.md)
|
||||
- [DSSE Attestation Guide](modules/attestor/README.md)
|
||||
- [Offline Operation](OFFLINE_KIT.md)
|
||||
- [Proof Bundle Spec](modules/triage/proof-bundle-spec.md)
|
||||
|
||||
## Changelog
|
||||
|
||||
| Version | Date | Changes |
|
||||
|---------|------|---------|
|
||||
| 1.0 | 2025-12-24 | Initial specification based on product advisory gap analysis |
|
||||
@@ -366,7 +366,7 @@ histogram_quantile(0.95,
|
||||
|
||||
- **CI/CD Workflow**: `.gitea/workflows/cross-platform-determinism.yml`
|
||||
- **Test README**: `src/__Tests/Determinism/README.md`
|
||||
- **Developer Guide**: `docs/testing/DETERMINISM_DEVELOPER_GUIDE.md`
|
||||
- **Developer Guide**: `docs/technical/testing/DETERMINISM_DEVELOPER_GUIDE.md`
|
||||
- **Batch Summary**: `docs/implplan/archived/2025-12-29-completed-sprints/BATCH_20251229_BE_COMPLETION_SUMMARY.md`
|
||||
|
||||
## Changelog
|
||||
|
||||
@@ -180,7 +180,7 @@ TODO → DOING → BLOCKED/IN_REVIEW → DONE
|
||||
- [ ] Pilot adoption in 2+ modules with S1 model (e.g., Scanner, Policy)
|
||||
|
||||
**Epic D (Connectors):**
|
||||
- [ ] Connector fixture discipline documented in `docs/testing/connector-fixture-discipline.md`
|
||||
- [ ] Connector fixture discipline documented in `docs/technical/testing/connector-fixture-discipline.md`
|
||||
- [ ] FixtureUpdater tool operational (with `UPDATE_CONNECTOR_FIXTURES=1` env var guard)
|
||||
- [ ] Pilot adoption in Concelier.Connector.NVD
|
||||
|
||||
|
||||
@@ -390,11 +390,11 @@ ONGOING: QUALITY GATES (Weeks 3-14+)
|
||||
### Appendix B: Reference Documents
|
||||
|
||||
1. **Advisory:** `docs/product-advisories/22-Dec-2026 - Better testing strategy.md`
|
||||
2. **Test Catalog:** `docs/testing/TEST_CATALOG.yml`
|
||||
3. **Test Models:** `docs/testing/testing-strategy-models.md`
|
||||
4. **Dependency Graph:** `docs/testing/SPRINT_DEPENDENCY_GRAPH.md`
|
||||
5. **Coverage Matrix:** `docs/testing/TEST_COVERAGE_MATRIX.md`
|
||||
6. **Execution Playbook:** `docs/testing/SPRINT_EXECUTION_PLAYBOOK.md`
|
||||
2. **Test Catalog:** `docs/technical/testing/TEST_CATALOG.yml`
|
||||
3. **Test Models:** `docs/technical/testing/testing-strategy-models.md`
|
||||
4. **Dependency Graph:** `docs/technical/testing/SPRINT_DEPENDENCY_GRAPH.md`
|
||||
5. **Coverage Matrix:** `docs/technical/testing/TEST_COVERAGE_MATRIX.md`
|
||||
6. **Execution Playbook:** `docs/technical/testing/SPRINT_EXECUTION_PLAYBOOK.md`
|
||||
|
||||
### Appendix C: Budget Estimate (Preliminary)
|
||||
|
||||
|
||||
@@ -259,4 +259,4 @@ Weekly (Optional):
|
||||
**Prepared by:** Project Management
|
||||
**Date:** 2025-12-23
|
||||
**Next Review:** 2026-01-06 (Week 1 kickoff)
|
||||
**Source:** `docs/testing/TEST_CATALOG.yml`, Sprint files 5100.0009.* and 5100.0010.*
|
||||
**Source:** `docs/technical/testing/TEST_CATALOG.yml`, Sprint files 5100.0009.* and 5100.0010.*
|
||||
|
||||
265
docs/technical/testing/TEST_SUITE_OVERVIEW.md
Executable file
265
docs/technical/testing/TEST_SUITE_OVERVIEW.md
Executable file
@@ -0,0 +1,265 @@
|
||||
# Automated Test-Suite Overview
|
||||
|
||||
This document enumerates **every automated check** executed by the Stella Ops
|
||||
CI pipeline, from unit level to chaos experiments. It is intended for
|
||||
contributors who need to extend coverage or diagnose failures.
|
||||
|
||||
> **Build parameters** – values such as `{{ dotnet }}` (runtime) and
|
||||
> `{{ angular }}` (UI framework) are injected at build time.
|
||||
|
||||
---
|
||||
|
||||
## Test Philosophy
|
||||
|
||||
### Core Principles
|
||||
|
||||
1. **Determinism as Contract**: Scan verdicts must be reproducible. Same inputs → byte-identical outputs.
|
||||
2. **Offline by Default**: Every test (except explicitly tagged "online") runs without network access.
|
||||
3. **Evidence-First Validation**: Assertions verify the complete evidence chain, not just pass/fail.
|
||||
4. **Interop is Required**: Compatibility with ecosystem tools (Syft, Grype, Trivy, cosign) blocks releases.
|
||||
5. **Coverage by Risk**: Prioritize testing high-risk paths over line coverage metrics.
|
||||
|
||||
### Test Boundaries
|
||||
|
||||
- **Lattice/policy merge** algorithms run in `scanner.webservice`
|
||||
- **Concelier/Excitors** preserve prune source (no conflict resolution)
|
||||
- Tests enforce these boundaries explicitly
|
||||
|
||||
### Model taxonomy
|
||||
|
||||
See `docs/technical/testing/testing-strategy-models.md` and `docs/technical/testing/TEST_CATALOG.yml` for
|
||||
the required test types per project model and the module-to-model mapping.
|
||||
|
||||
---
|
||||
|
||||
## Layer Map
|
||||
|
||||
| Layer | Tooling | Entry-point | Frequency |
|
||||
|-------|---------|-------------|-----------|
|
||||
| **1. Unit** | `xUnit` (<code>dotnet test</code>) | `*.Tests.csproj` | per PR / push |
|
||||
| **2. Property-based** | `FsCheck` | `SbomPropertyTests`, `Canonicalization` | per PR |
|
||||
| **3. Integration (API)** | `Testcontainers` suite | `test/Api.Integration` | per PR + nightly |
|
||||
| **4. Integration (DB-merge)** | Testcontainers PostgreSQL + Valkey | `Concelier.Integration` | per PR |
|
||||
| **5. Contract (OpenAPI)** | Schema validation | `docs/api/*.yaml` | per PR |
|
||||
| **6. Front-end unit** | `Jest` | `ui/src/**/*.spec.ts` | per PR |
|
||||
| **7. Front-end E2E** | `Playwright` | `ui/e2e/**` | nightly |
|
||||
| **8. Lighthouse perf / a11y** | `lighthouse-ci` (Chrome headless) | `ui/dist/index.html` | nightly |
|
||||
| **9. Load** | `k6` scripted scenarios | `tests/load/*.js` | nightly |
|
||||
| **10. Chaos** | `pumba`, custom harness | `tests/chaos/` | weekly |
|
||||
| **11. Interop** | Syft/Grype/cosign | `tests/interop/` | nightly |
|
||||
| **12. Offline E2E** | Network-isolated containers | `tests/offline/` | nightly |
|
||||
| **13. Replay Verification** | Golden corpus replay | `bench/golden-corpus/` | per PR |
|
||||
| **14. Dependency scanning** | `Trivy fs` + `dotnet list package --vuln` | root | per PR |
|
||||
| **15. License compliance** | `LicenceFinder` | root | per PR |
|
||||
| **16. SBOM reproducibility** | `in-toto attestation` diff | GitLab job | release tags |
|
||||
|
||||
---
|
||||
|
||||
## Test Categories (xUnit Traits)
|
||||
|
||||
```csharp
|
||||
[Trait("Category", "Unit")] // Fast, isolated unit tests
|
||||
[Trait("Category", "Property")] // Property-based checks (sub-trait)
|
||||
[Trait("Category", "Snapshot")] // Golden/snapshot assertions (sub-trait)
|
||||
[Trait("Category", "Integration")] // Tests requiring infrastructure
|
||||
[Trait("Category", "Contract")] // Schema and API contract checks
|
||||
[Trait("Category", "E2E")] // Full end-to-end workflows
|
||||
[Trait("Category", "AirGap")] // Must work without network
|
||||
[Trait("Category", "Interop")] // Third-party tool compatibility
|
||||
[Trait("Category", "Performance")] // Performance benchmarks
|
||||
[Trait("Category", "Chaos")] // Failure injection tests
|
||||
[Trait("Category", "Security")] // Security-focused tests
|
||||
[Trait("Category", "Live")] // Opt-in upstream connector tests
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Quality Gates
|
||||
|
||||
| Metric | Budget | Gate |
|
||||
|--------|--------|------|
|
||||
| API unit coverage | ≥ 85% lines | PR merge |
|
||||
| API response P95 | ≤ 120 ms | nightly alert |
|
||||
| Δ-SBOM warm scan P95 (4 vCPU) | ≤ 5 s | nightly alert |
|
||||
| Lighthouse performance score | ≥ 90 | nightly alert |
|
||||
| Lighthouse accessibility score | ≥ 95 | nightly alert |
|
||||
| k6 sustained RPS drop | < 5% vs baseline | nightly alert |
|
||||
| **Replay determinism** | 0 byte diff | **Release** |
|
||||
| **Interop findings parity** | ≥ 95% | **Release** |
|
||||
| **Offline E2E** | All pass with no network | **Release** |
|
||||
| **Unknowns budget (prod)** | ≤ configured limit | **Release** |
|
||||
| **Router Retry-After compliance** | 100% | Nightly |
|
||||
|
||||
---
|
||||
|
||||
## Local Runner
|
||||
|
||||
```bash
|
||||
# minimal run: unit + property + frontend tests
|
||||
./scripts/dev-test.sh
|
||||
|
||||
# full stack incl. Playwright and lighthouse
|
||||
./scripts/dev-test.sh --full
|
||||
|
||||
# category-specific
|
||||
dotnet test --filter "Category=Unit"
|
||||
dotnet test --filter "Category=AirGap"
|
||||
dotnet test --filter "Category=Interop"
|
||||
```
|
||||
|
||||
The script spins up PostgreSQL/Valkey via Testcontainers and requires:
|
||||
|
||||
* Docker ≥ 25
|
||||
* Node 20 (for Jest/Playwright)
|
||||
|
||||
### PostgreSQL Testcontainers
|
||||
|
||||
Multiple suites (Concelier connectors, Excititor worker/WebService, Scheduler)
|
||||
use Testcontainers with PostgreSQL for integration tests. If you don't have
|
||||
Docker available, tests can also run against a local PostgreSQL instance
|
||||
listening on `127.0.0.1:5432`.
|
||||
|
||||
### Local PostgreSQL Helper
|
||||
|
||||
Some suites (Concelier WebService/Core, Exporter JSON) need a full
|
||||
PostgreSQL instance when you want to debug or inspect data with `psql`.
|
||||
A helper script is available under `tools/postgres/local-postgres.sh`:
|
||||
|
||||
```bash
|
||||
# start a local PostgreSQL instance
|
||||
tools/postgres/local-postgres.sh start
|
||||
|
||||
# stop / clean
|
||||
tools/postgres/local-postgres.sh stop
|
||||
tools/postgres/local-postgres.sh clean
|
||||
```
|
||||
|
||||
By default the script uses Docker to run PostgreSQL 16, binds to
|
||||
`127.0.0.1:5432`, and creates a database called `stellaops`. The
|
||||
connection string is printed on start and you can export it before
|
||||
running `dotnet test` if a suite supports overriding its connection string.
|
||||
|
||||
---
|
||||
|
||||
## New Test Infrastructure (Epic 5100)
|
||||
|
||||
### Run Manifest & Replay
|
||||
|
||||
Every scan captures a **Run Manifest** containing all inputs (artifact digests, feed versions, policy versions, PRNG seed). This enables deterministic replay:
|
||||
|
||||
```bash
|
||||
# Replay a scan from manifest
|
||||
stella replay --manifest run-manifest.json --output verdict.json
|
||||
|
||||
# Verify determinism
|
||||
stella replay verify --manifest run-manifest.json
|
||||
```
|
||||
|
||||
### Evidence Index
|
||||
|
||||
The **Evidence Index** links verdicts to their supporting evidence chain:
|
||||
- Verdict → SBOM digests → Attestation IDs → Tool versions
|
||||
|
||||
### Golden Corpus
|
||||
|
||||
Located at `bench/golden-corpus/`, contains 50+ test cases:
|
||||
- Severity levels (Critical, High, Medium, Low)
|
||||
- VEX scenarios (Not Affected, Affected, Conflicting)
|
||||
- Reachability cases (Reachable, Not Reachable, Inconclusive)
|
||||
- Unknowns scenarios
|
||||
- Scale tests (200 to 50k+ packages)
|
||||
- Multi-distro (Alpine, Debian, RHEL, SUSE, Ubuntu)
|
||||
- Interop fixtures (Syft-generated, Trivy-generated)
|
||||
- Negative cases (malformed inputs)
|
||||
|
||||
### Offline Testing
|
||||
|
||||
Inherit from `NetworkIsolatedTestBase` for air-gap compliance:
|
||||
|
||||
```csharp
|
||||
[Trait("Category", "AirGap")]
|
||||
public class OfflineTests : NetworkIsolatedTestBase
|
||||
{
|
||||
[Fact]
|
||||
public async Task Test_WorksOffline()
|
||||
{
|
||||
// Test implementation
|
||||
AssertNoNetworkCalls(); // Fails if network accessed
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Concelier OSV↔GHSA Parity Fixtures
|
||||
|
||||
The Concelier connector suite includes a regression test (`OsvGhsaParityRegressionTests`)
|
||||
that checks a curated set of GHSA identifiers against OSV responses. The fixture
|
||||
snapshots live in `src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Osv.Tests/Fixtures/` and are kept
|
||||
deterministic so the parity report remains reproducible.
|
||||
|
||||
To refresh the fixtures when GHSA/OSV payloads change:
|
||||
|
||||
1. Ensure outbound HTTPS access to `https://api.osv.dev` and `https://api.github.com`.
|
||||
2. Run `UPDATE_PARITY_FIXTURES=1 dotnet test src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Osv.Tests/StellaOps.Concelier.Connector.Osv.Tests.csproj`.
|
||||
3. Commit the regenerated `osv-ghsa.*.json` files that the test emits (raw snapshots and canonical advisories).
|
||||
|
||||
The regen flow logs `[Parity]` messages and normalises `recordedAt` timestamps so the
|
||||
fixtures stay stable across machines.
|
||||
|
||||
---
|
||||
|
||||
## CI Job Layout
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph fast-path
|
||||
U[xUnit] --> P[FsCheck] --> I1[Testcontainer API]
|
||||
end
|
||||
|
||||
I1 --> FE[Jest]
|
||||
FE --> E2E[Playwright]
|
||||
E2E --> Lighthouse
|
||||
|
||||
subgraph release-gates
|
||||
REPLAY[Replay Verify]
|
||||
INTEROP[Interop E2E]
|
||||
OFFLINE[Offline E2E]
|
||||
BUDGET[Unknowns Gate]
|
||||
end
|
||||
|
||||
Lighthouse --> INTEG2[Concelier]
|
||||
INTEG2 --> LOAD[k6]
|
||||
LOAD --> CHAOS[Chaos Suite]
|
||||
CHAOS --> RELEASE[Attestation diff]
|
||||
|
||||
RELEASE --> release-gates
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Adding a New Test Layer
|
||||
|
||||
1. Extend `scripts/dev-test.sh` so local contributors get the layer by default.
|
||||
2. Add a dedicated workflow in `.gitea/workflows/` (or GitLab job in `.gitlab-ci.yml`).
|
||||
3. Register the job in `docs/technical/testing/TEST_SUITE_OVERVIEW.md` *and* list its metric
|
||||
in `docs/modules/telemetry/guides/README.md`.
|
||||
4. If the test requires network isolation, inherit from `NetworkIsolatedTestBase`.
|
||||
5. If the test uses golden corpus, add cases to `bench/golden-corpus/`.
|
||||
|
||||
---
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Testing Strategy Models](testing/testing-strategy-models.md)
|
||||
- [Test Catalog](testing/TEST_CATALOG.yml)
|
||||
- [Testing README](testing/README.md) - Complete testing documentation index
|
||||
- [CI/CD Test Strategy](cicd/test-strategy.md) - CI/CD integration details
|
||||
- [tests/AGENTS.md](../tests/AGENTS.md)
|
||||
- [Offline Operation Guide](OFFLINE_KIT.md)
|
||||
- [Module Architecture Dossiers](modules/)
|
||||
|
||||
---
|
||||
|
||||
*Last updated 2025-12-23*
|
||||
|
||||
@@ -4,7 +4,7 @@ This document describes how to categorize tests by lane and test type for CI fil
|
||||
|
||||
## Test Lanes
|
||||
|
||||
StellaOps uses standardized test lanes based on `docs/testing/TEST_CATALOG.yml`:
|
||||
StellaOps uses standardized test lanes based on `docs/technical/testing/TEST_CATALOG.yml`:
|
||||
|
||||
| Lane | Purpose | Characteristics | PR Gating |
|
||||
|------|---------|-----------------|-----------|
|
||||
@@ -240,6 +240,6 @@ If you have existing tests without lane attributes:
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- Test Catalog: `docs/testing/TEST_CATALOG.yml`
|
||||
- Testing Strategy: `docs/testing/testing-strategy-models.md`
|
||||
- Test Catalog: `docs/technical/testing/TEST_CATALOG.yml`
|
||||
- Testing Strategy: `docs/technical/testing/testing-strategy-models.md`
|
||||
- TestKit README: `src/__Libraries/StellaOps.TestKit/README.md`
|
||||
|
||||
@@ -303,8 +303,8 @@ Replace per-module test execution with lane-based execution:
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- Test Lane Filters: `docs/testing/ci-lane-filters.md`
|
||||
- Testing Strategy: `docs/testing/testing-strategy-models.md`
|
||||
- Test Catalog: `docs/testing/TEST_CATALOG.yml`
|
||||
- Test Lane Filters: `docs/technical/testing/ci-lane-filters.md`
|
||||
- Testing Strategy: `docs/technical/testing/testing-strategy-models.md`
|
||||
- Test Catalog: `docs/technical/testing/TEST_CATALOG.yml`
|
||||
- TestKit README: `src/__Libraries/StellaOps.TestKit/README.md`
|
||||
- Example Workflow: `.gitea/workflows/test-lanes.yml`
|
||||
|
||||
@@ -287,5 +287,5 @@ When writing determinism tests, verify:
|
||||
## Related Documentation
|
||||
|
||||
- TestKit README: `src/__Libraries/StellaOps.TestKit/README.md`
|
||||
- Testing Strategy: `docs/testing/testing-strategy-models.md`
|
||||
- Test Catalog: `docs/testing/TEST_CATALOG.yml`
|
||||
- Testing Strategy: `docs/technical/testing/testing-strategy-models.md`
|
||||
- Test Catalog: `docs/technical/testing/TEST_CATALOG.yml`
|
||||
|
||||
@@ -15,9 +15,9 @@ StellaOps validates all SBOM fixtures against official JSON schemas to detect sc
|
||||
|
||||
| Format | Version | Schema Location | Validator |
|
||||
|--------|---------|-----------------|-----------|
|
||||
| CycloneDX | 1.6 | `docs/schemas/cyclonedx-bom-1.6.schema.json` | sbom-utility |
|
||||
| SPDX | 3.0.1 | `docs/schemas/spdx-jsonld-3.0.1.schema.json` | pyspdxtools / check-jsonschema |
|
||||
| OpenVEX | 0.2.0 | `docs/schemas/openvex-0.2.0.schema.json` | ajv-cli |
|
||||
| CycloneDX | 1.6 | `docs/modules/sbom-service/schemas/cyclonedx-bom-1.6.schema.json` | sbom-utility |
|
||||
| SPDX | 3.0.1 | `docs/modules/sbom-service/schemas/spdx-jsonld-3.0.1.schema.json` | pyspdxtools / check-jsonschema |
|
||||
| OpenVEX | 0.2.0 | `docs/modules/excititor/schemas/openvex-0.2.0.schema.json` | ajv-cli |
|
||||
|
||||
## CI Workflows
|
||||
|
||||
@@ -26,7 +26,7 @@ StellaOps validates all SBOM fixtures against official JSON schemas to detect sc
|
||||
**File:** `.gitea/workflows/schema-validation.yml`
|
||||
|
||||
Runs on:
|
||||
- Pull requests touching `bench/golden-corpus/**`, `src/Scanner/**`, `docs/schemas/**`, or `scripts/validate-*.sh`
|
||||
- Pull requests touching `bench/golden-corpus/**`, `src/Scanner/**`, `docs/modules/**/schemas/**`, or `scripts/validate-*.sh`
|
||||
- Push to `main` branch
|
||||
|
||||
Jobs:
|
||||
@@ -85,7 +85,7 @@ curl -sSfL "https://github.com/CycloneDX/sbom-utility/releases/download/v0.16.0/
|
||||
sudo mv sbom-utility /usr/local/bin/
|
||||
|
||||
# Validate
|
||||
sbom-utility validate --input-file sbom.json --schema docs/schemas/cyclonedx-bom-1.6.schema.json
|
||||
sbom-utility validate --input-file sbom.json --schema docs/modules/sbom-service/schemas/cyclonedx-bom-1.6.schema.json
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
@@ -187,7 +187,7 @@ If negative tests fail with "UNEXPECTED PASS":
|
||||
|
||||
When updating schema versions:
|
||||
|
||||
1. Download new schema to `docs/schemas/`
|
||||
1. Download new schema to the appropriate module `schemas/` directory (e.g., `docs/modules/sbom-service/schemas/`)
|
||||
2. Update `SBOM_UTILITY_VERSION` in workflows if needed
|
||||
3. Run full validation to check for new violations
|
||||
4. Update documentation with new version
|
||||
|
||||
@@ -160,7 +160,7 @@ thresholds:
|
||||
- `MaliciousPayloads.cs` - Common attack patterns
|
||||
- `SecurityTestBase.cs` - Test infrastructure
|
||||
- `.gitea/workflows/security-tests.yml` - Dedicated CI workflow
|
||||
- `docs/testing/security-testing-guide.md` - Documentation
|
||||
- `docs/technical/testing/security-testing-guide.md` - Documentation
|
||||
|
||||
---
|
||||
|
||||
@@ -182,7 +182,7 @@ thresholds:
|
||||
- `scripts/ci/mutation-thresholds.yaml` - Threshold configuration
|
||||
- `.gitea/workflows/mutation-testing.yml` - Weekly mutation runs
|
||||
- `bench/baselines/mutation-baselines.json` - Baseline scores
|
||||
- `docs/testing/mutation-testing-guide.md` - Developer guide
|
||||
- `docs/technical/testing/mutation-testing-guide.md` - Developer guide
|
||||
|
||||
---
|
||||
|
||||
@@ -273,7 +273,7 @@ src/Scanner/__Libraries/StellaOps.Scanner.Core/stryker-config.json
|
||||
src/Policy/StellaOps.Policy.Engine/stryker-config.json
|
||||
src/Authority/StellaOps.Authority.Core/stryker-config.json
|
||||
|
||||
docs/testing/
|
||||
docs/technical/testing/
|
||||
├── ci-quality-gates.md
|
||||
├── security-testing-guide.md
|
||||
└── mutation-testing-guide.md
|
||||
|
||||
@@ -10,7 +10,7 @@ Supersedes/extends: `docs/product-advisories/archived/2025-12-21-testing-strateg
|
||||
|
||||
## Strategy in brief
|
||||
- Use test models (L0, S1, C1, W1, WK1, T1, AN1, CLI1, PERF) to encode required test types.
|
||||
- Map every module to one or more models in `docs/testing/TEST_CATALOG.yml`.
|
||||
- Map every module to one or more models in `docs/technical/testing/TEST_CATALOG.yml`.
|
||||
- Run tests through standardized CI lanes (Unit, Contract, Integration, Security, Performance, Live).
|
||||
|
||||
## Test models (requirements)
|
||||
@@ -40,13 +40,13 @@ Supersedes/extends: `docs/product-advisories/archived/2025-12-21-testing-strateg
|
||||
- Live: opt-in upstream connector checks (never PR gating by default).
|
||||
|
||||
## Documentation moments (when to update)
|
||||
- New model or required test type: update `docs/testing/TEST_CATALOG.yml`.
|
||||
- New lane or gate: update `docs/TEST_SUITE_OVERVIEW.md` and `docs/testing/ci-quality-gates.md`.
|
||||
- New model or required test type: update `docs/technical/testing/TEST_CATALOG.yml`.
|
||||
- New lane or gate: update `docs/technical/testing/TEST_SUITE_OVERVIEW.md` and `docs/technical/testing/ci-quality-gates.md`.
|
||||
- Module-specific test policy change: update the module dossier under `docs/modules/<module>/`.
|
||||
- New fixtures or runnable harnesses: place under `docs/benchmarks/**` or `tests/**` and link here.
|
||||
|
||||
## Related artifacts
|
||||
- Test catalog (source of truth): `docs/testing/TEST_CATALOG.yml`
|
||||
- Test suite overview: `docs/TEST_SUITE_OVERVIEW.md`
|
||||
- Quality guardrails: `docs/testing/testing-quality-guardrails-implementation.md`
|
||||
- Test catalog (source of truth): `docs/technical/testing/TEST_CATALOG.yml`
|
||||
- Test suite overview: `docs/technical/testing/TEST_SUITE_OVERVIEW.md`
|
||||
- Quality guardrails: `docs/technical/testing/testing-quality-guardrails-implementation.md`
|
||||
- Code samples from the advisory: `docs/benchmarks/testing/better-testing-strategy-samples.md`
|
||||
|
||||
Reference in New Issue
Block a user