git.stella-ops.org/docs/11_DATA_SCHEMAS.md

# Data Schemas & Persistence Contracts

Audience – backend developers, plug‑in authors, DB admins.
Scope – describes Redis, MongoDB (optional), and on‑disk blob shapes that power Stella Ops.

---

## 0 Document Conventions

• CamelCase for JSON.
• All timestamps are RFC 3339 / ISO 8601 with Z (UTC).
• ⭑ = planned but not shipped yet (kept on Feature Matrix “To Do”).

---

## 1 SBOM Wrapper Envelope

Every SBOM blob (regardless of format) is stored on disk or in object storage with a sidecar JSON file that indexes it for the scanners.

#### 1.1 JSON Shape

{
  "id": "sha256:417f…",          // digest of the SBOM *file* itself
  "imageDigest": "sha256:e2b9…", // digest of the original container image
  "created": "2025-07-14T07:02:13Z",
  "format": "trivy-json-v2",     // NEW enum: trivy-json-v2 | spdx-json | cyclonedx-json
  "layers": [
    "sha256:d38b…",              // layer digests (ordered)
    "sha256:af45…"
  ],
  "partial": false,              // true => delta SBOM (only some layers)
  "provenanceId": "prov_0291"    // ⭑ link to SLSA attestation (Q1‑2026)
}

format NEW – added to support multiple SBOM formats.
partial NEW – true when generated via the delta SBOM flow (§1.3).

#### 1.2 File‑system Layout

blobs/
 ├─ 417f…                # digest prefix
 │   ├─ sbom.json        # payload (any format)
 │   └─ sbom.meta.json   # wrapper (shape above)

Note

– blob storage can point at S3, MinIO, or plain disk; driver plug‑ins adapt.

#### 1.3 Delta SBOM Extension

When partial: true, only the missing layers have been scanned.
Merging logic inside scanning module stitches new data onto the cached full SBOM in Redis.

---

## 2 Redis Keyspace

Key pattern	Type	TTL	Purpose
scan:<digest>	string	∞	Last scan JSON result (as returned by /scan)
layers:<digest>	set	90d	Layers already possessing SBOMs (delta cache)
policy:active	string	∞	YAML or Rego ruleset
quota:<token>	string	until next UTC midnight	Per‑token scan counter for Free tier (333 scans).
policy:history	list	∞	Change audit IDs (see Mongo)
feed:nvd:json	string	24h	Normalised feed snapshot
locator:<imageDigest>	string	30d	Maps image digest → sbomBlobId
metrics:…	various	—	Prom / OTLP runtime metrics

Delta SBOM uses layers:* to skip work in <20 ms. Quota enforcement increments quota:<token> atomically; when value ≥ 333 the API returns 429.

---

## 3 MongoDB Collections (Optional)

Only enabled when MONGO_URI is supplied (for long‑term audit).

Collection	Shape (summary)	Indexes
sbom_history	Wrapper JSON + replaceTs on overwrite	{imageDigest} {created}
policy_versions	{_id, yaml, rego, authorId, created}	{created}
attestations ⭑	SLSA provenance doc + Rekor log pointer	{imageDigest}
audit_log	Fully rendered RFC 5424 entries (UI & CLI actions)	{userId} {ts}

Schema detail for policy_versions:

{
  "_id": "6619e90b8c5e1f76",
  "yaml": "version: 1.0\nrules:\n  - …",
  "rego": null,                    // filled when Rego uploaded
  "authorId": "u_1021",
  "created": "2025-07-14T08:15:04Z",
  "comment": "Imported via API"
}

---

## 4 Policy Schema (YAML v1.0)

Minimal viable grammar (subset of OSV‑SCHEMA ideas).

version: "1.0"
rules:
  - name: Block Critical
    severity: [Critical]
    action: block
  - name: Ignore Low Dev
    severity: [Low, None]
    environments: [dev, staging]
    action: ignore
    expires: "2026-01-01"
  - name: Escalate BDU High
    sources: [BDU]
    severity: [High, Critical]
    action: escalate

Validation is performed by policy:mapping.yaml JSON‑Schema embedded in backend.

### 4.1 Rego Variant (Advanced – TODO)

Accepted but stored as‑is in rego field.
Evaluated via internal OPA side‑car once feature graduates from TODO list.

---

## 5 SLSA Attestation Schema ⭑

Planned for Q1‑2026 (kept here for early plug‑in authors).

{
  "id": "prov_0291",
  "imageDigest": "sha256:e2b9…",
  "buildType": "https://slsa.dev/container/v1",
  "builder": {
    "id": "https://git.stella-ops.ru/ci/stella-runner@sha256:f7b7…"
  },
  "metadata": {
    "invocation": {
      "parameters": {"GIT_SHA": "f6a1…"},
      "buildStart": "2025-07-14T06:59:17Z",
      "buildEnd": "2025-07-14T07:01:22Z"
    },
    "completeness": {"parameters": true}
  },
  "materials": [
    {"uri": "git+https://git…", "digest": {"sha1": "f6a1…"}}
  ],
  "rekorLogIndex": 99817    // entry in local Rekor mirror
}

---

## 6 Validator Contracts

• For SBOM wrapper – ISbomValidator (DLL plug‑in) must return typed error list.
• For YAML policies – JSON‑Schema at /schemas/policy‑v1.json.
• For Rego – OPA opa eval --fail-defined under the hood.
• For Free‑tier quotas – IQuotaService integration tests ensure quota:<token> resets at UTC midnight and produces correct Retry‑After headers.

---

## 7 Migration Notes

1. Add format column to existing SBOM wrappers; default to trivy-json-v2.
2. Populate layers & partial via backfill script (ship with stellopsctl migrate wizard).
3. Policy YAML previously stored in Redis → copy to Mongo if persistence enabled.
4. Prepare attestations collection (empty) – safe to create in advance.

---

## 8 Open Questions / Future Work

• How to de‑duplicate identical Rego policies differing only in whitespace?
• Embed GOST 34.11‑2018 digests when users enable Russian crypto suite?
• Should enterprise tiers share the same Redis quota keys or switch to JWT claim tier != Free bypass?
• Evaluate sliding‑window quota instead of strict daily reset.
• Consider rate‑limit for /layers/missing to avoid brute‑force enumeration.

---

## 9 Change Log

Date	Note
2025‑07‑14	Added: format, partial, delta cache keys, YAML policy schema v1.0.
2025‑07‑12	Initial public draft – SBOM wrapper, Redis keyspace, audit collections.

---