# Data Schemas & Persistence Contracts *Audience* – backend developers, plug‑in authors, DB admins. *Scope* – describes **Redis**, **MongoDB** (optional), and on‑disk blob shapes that power Stella Ops. --- ## 0 Document Conventions * **CamelCase** for JSON. * All timestamps are **RFC 3339 / ISO 8601** with `Z` (UTC). * `⭑` = planned but *not* shipped yet (kept on Feature Matrix “To Do”). --- ## 1 SBOM Wrapper Envelope Every SBOM blob (regardless of format) is stored on disk or in object storage with a *sidecar* JSON file that indexes it for the scanners. #### 1.1 JSON Shape ```jsonc { "id": "sha256:417f…", // digest of the SBOM *file* itself "imageDigest": "sha256:e2b9…", // digest of the original container image "created": "2025-07-14T07:02:13Z", "format": "trivy-json-v2", // NEW enum: trivy-json-v2 | spdx-json | cyclonedx-json "layers": [ "sha256:d38b…", // layer digests (ordered) "sha256:af45…" ], "partial": false, // true => delta SBOM (only some layers) "provenanceId": "prov_0291" // ⭑ link to SLSA attestation (Q1‑2026) } ``` *`format`* **NEW** – added to support **multiple SBOM formats**. *`partial`* **NEW** – true when generated via the **delta SBOM** flow (§1.3). #### 1.2 File‑system Layout ``` blobs/ ├─ 417f… # digest prefix │   ├─ sbom.json # payload (any format) │   └─ sbom.meta.json # wrapper (shape above) ``` > **Note** – blob storage can point at S3, MinIO, or plain disk; driver plug‑ins adapt. #### 1.3 Delta SBOM Extension When `partial: true`, *only* the missing layers have been scanned. Merging logic inside `scanning` module stitches new data onto the cached full SBOM in Redis. --- ## 2 Redis Keyspace | Key pattern | Type | TTL | Purpose | |-------------------------------------|---------|------|--------------------------------------------------| | `scan:<digest>` | string | ∞ | Last scan JSON result (as returned by `/scan`) | | `layers:<digest>` | set | 90d | Layers already possessing SBOMs (delta cache) | | `policy:active` | string | ∞ | YAML **or** Rego ruleset | | `quota:<token>` | string | *until next UTC midnight* | Per‑token scan counter for Free tier (333 scans). | | `policy:history` | list | ∞ | Change audit IDs (see Mongo) | | `feed:nvd:json` | string | 24h | Normalised feed snapshot | | `locator:<imageDigest>` | string | 30d | Maps image digest → sbomBlobId | | `metrics:…` | various | — | Prom / OTLP runtime metrics | > **Delta SBOM** uses `layers:*` to skip work in <20 ms. > **Quota enforcement** increments `quota:` atomically; when value ≥ 333 the API returns **429**. --- ## 3 MongoDB Collections (Optional) Only enabled when `MONGO_URI` is supplied (for long‑term audit). | Collection | Shape (summary) | Indexes | |--------------------|------------------------------------------------------------|-------------------------------------| | `sbom_history` | Wrapper JSON + `replaceTs` on overwrite | `{imageDigest}` `{created}` | | `policy_versions` | `{_id, yaml, rego, authorId, created}` | `{created}` | | `attestations` ⭑ | SLSA provenance doc + Rekor log pointer | `{imageDigest}` | | `audit_log` | Fully rendered RFC 5424 entries (UI & CLI actions) | `{userId}` `{ts}` | Schema detail for **policy_versions**: ```jsonc { "_id": "6619e90b8c5e1f76", "yaml": "version: 1.0\nrules:\n - …", "rego": null, // filled when Rego uploaded "authorId": "u_1021", "created": "2025-07-14T08:15:04Z", "comment": "Imported via API" } ``` --- ## 4 Policy Schema (YAML v1.0) Minimal viable grammar (subset of OSV‑SCHEMA ideas). ```yaml version: "1.0" rules: - name: Block Critical severity: [Critical] action: block - name: Ignore Low Dev severity: [Low, None] environments: [dev, staging] action: ignore expires: "2026-01-01" - name: Escalate BDU High sources: [BDU] severity: [High, Critical] action: escalate ``` Validation is performed by `policy:mapping.yaml` JSON‑Schema embedded in backend. ### 4.1 Rego Variant (Advanced – TODO) *Accepted but stored as‑is in `rego` field.* Evaluated via internal **OPA** side‑car once feature graduates from TODO list. --- ## 5 SLSA Attestation Schema ⭑ Planned for Q1‑2026 (kept here for early plug‑in authors). ```jsonc { "id": "prov_0291", "imageDigest": "sha256:e2b9…", "buildType": "https://slsa.dev/container/v1", "builder": { "id": "https://git.stella-ops.ru/ci/stella-runner@sha256:f7b7…" }, "metadata": { "invocation": { "parameters": {"GIT_SHA": "f6a1…"}, "buildStart": "2025-07-14T06:59:17Z", "buildEnd": "2025-07-14T07:01:22Z" }, "completeness": {"parameters": true} }, "materials": [ {"uri": "git+https://git…", "digest": {"sha1": "f6a1…"}} ], "rekorLogIndex": 99817 // entry in local Rekor mirror } ``` --- ## 6 Validator Contracts * For SBOM wrapper – `ISbomValidator` (DLL plug‑in) must return *typed* error list. * For YAML policies – JSON‑Schema at `/schemas/policy‑v1.json`. * For Rego – OPA `opa eval --fail-defined` under the hood. * For **Free‑tier quotas** – `IQuotaService` integration tests ensure `quota:` resets at UTC midnight and produces correct `Retry‑After` headers. --- ## 7 Migration Notes 1. **Add `format` column** to existing SBOM wrappers; default to `trivy-json-v2`. 2. **Populate `layers` & `partial`** via backfill script (ship with `stellopsctl migrate` wizard). 3. Policy YAML previously stored in Redis → copy to Mongo if persistence enabled. 4. Prepare `attestations` collection (empty) – safe to create in advance. --- ## 8 Open Questions / Future Work * How to de‑duplicate *identical* Rego policies differing only in whitespace? * Embed *GOST 34.11‑2018* digests when users enable Russian crypto suite? * Should enterprise tiers share the same Redis quota keys or switch to JWT claim `tier != Free` bypass? * Evaluate sliding‑window quota instead of strict daily reset. * Consider rate‑limit for `/layers/missing` to avoid brute‑force enumeration. --- ## 9 Change Log | Date | Note | |------------|--------------------------------------------------------------------------------| | 2025‑07‑14 | **Added:** `format`, `partial`, delta cache keys, YAML policy schema v1.0. | | 2025‑07‑12 | **Initial public draft** – SBOM wrapper, Redis keyspace, audit collections. | ---