# Data Schemas & Persistence Contracts

*Audience* – backend developers, plug‑in authors, DB admins.  
*Scope* – describes **Redis**, **MongoDB** (optional), and on‑disk blob shapes that power Stella Ops.

---

## 0 Document Conventions

* **CamelCase** for JSON.  
* All timestamps are **RFC 3339 / ISO 8601** with `Z` (UTC).  
* `⭑` = planned but *not* shipped yet (kept on Feature Matrix “To Do”).

---

## 1 SBOM Wrapper Envelope

Every SBOM blob (regardless of format) is stored on disk or in object storage with a *sidecar* JSON file that indexes it for the scanners.

#### 1.1 JSON Shape

```jsonc
{
  "id": "sha256:417f…",          // digest of the SBOM *file* itself
  "imageDigest": "sha256:e2b9…", // digest of the original container image
  "created": "2025-07-14T07:02:13Z",
  "format": "trivy-json-v2",     // NEW enum: trivy-json-v2 | spdx-json | cyclonedx-json
  "layers": [
    "sha256:d38b…",              // layer digests (ordered)
    "sha256:af45…"
  ],
  "partial": false,              // true => delta SBOM (only some layers)
  "provenanceId": "prov_0291"    // ⭑ link to SLSA attestation (Q1‑2026)
}
```

*`format`* **NEW** – added to support **multiple SBOM formats**.  
*`partial`* **NEW** – true when generated via the **delta SBOM** flow (§1.3).

#### 1.2 File‑system Layout

```
blobs/
 ├─ 417f…                # digest prefix
 │   ├─ sbom.json        # payload (any format)
 │   └─ sbom.meta.json   # wrapper (shape above)
```

> **Note** – blob storage can point at S3, MinIO, or plain disk; driver plug‑ins adapt.

#### 1.3 Delta SBOM Extension

When `partial: true`, *only* the missing layers have been scanned.  
Merging logic inside `scanning` module stitches new data onto the cached full SBOM in Redis.

---

## 2 Redis Keyspace

| Key pattern                         | Type    | TTL  | Purpose                                          |
|-------------------------------------|---------|------|--------------------------------------------------|
| `scan:&lt;digest&gt;`               | string  | ∞    | Last scan JSON result (as returned by `/scan`)   |
| `layers:&lt;digest&gt;`             | set     | 90d  | Layers already possessing SBOMs (delta cache)    |
| `policy:active`                     | string  | ∞    | YAML **or** Rego ruleset                         |
| `quota:&lt;token&gt;`              | string  | *until next UTC midnight* | Per‑token scan counter for Free tier ({{ quota_token }} scans). |
| `policy:history`                    | list    | ∞    | Change audit IDs (see Mongo)                     |
| `feed:nvd:json`                     | string  | 24h  | Normalised feed snapshot                         |
| `locator:&lt;imageDigest&gt;`        | string  | 30d  | Maps image digest → sbomBlobId                   |
| `metrics:…`                         | various | —    | Prom / OTLP runtime metrics                      |

> **Delta SBOM** uses `layers:*` to skip work in <20 ms.
> **Quota enforcement** increments `quota:<token>` atomically; when {{ quota_token }} the API returns **429**.

---

## 3 MongoDB Collections (Optional)

Only enabled when `MONGO_URI` is supplied (for long‑term audit).

| Collection         | Shape (summary)                                            | Indexes                             |
|--------------------|------------------------------------------------------------|-------------------------------------|
| `sbom_history`     | Wrapper JSON + `replaceTs` on overwrite                    | `{imageDigest}` `{created}`         |
| `policy_versions`  | `{_id, yaml, rego, authorId, created}`                    | `{created}`                         |
| `attestations` ⭑  | SLSA provenance doc + Rekor log pointer                    | `{imageDigest}`                     |
| `audit_log`        | Fully rendered RFC 5424 entries (UI & CLI actions)         | `{userId}` `{ts}`                   |

Schema detail for **policy_versions**:

```jsonc
{
  "_id": "6619e90b8c5e1f76",
  "yaml": "version: 1.0\nrules:\n  - …",
  "rego": null,                    // filled when Rego uploaded
  "authorId": "u_1021",
  "created": "2025-07-14T08:15:04Z",
  "comment": "Imported via API"
}
```

---

## 4 Policy Schema (YAML v1.0)

Minimal viable grammar (subset of OSV‑SCHEMA ideas).

```yaml
version: "1.0"
rules:
  - name: Block Critical
    severity: [Critical]
    action: block
  - name: Ignore Low Dev
    severity: [Low, None]
    environments: [dev, staging]
    action: ignore
    expires: "2026-01-01"
  - name: Escalate RegionalFeed High
    sources: [NVD, CNNVD, CNVD, ENISA, JVN, BDU]
    severity: [High, Critical]
    action: escalate
```

Validation is performed by `policy:mapping.yaml` JSON‑Schema embedded in backend.

### 4.1 Rego Variant (Advanced – TODO)

*Accepted but stored as‑is in `rego` field.*  
Evaluated via internal **OPA** side‑car once feature graduates from TODO list.

---

## 5 SLSA Attestation Schema ⭑

Planned for Q1‑2026 (kept here for early plug‑in authors).

```jsonc
{
  "id": "prov_0291",
  "imageDigest": "sha256:e2b9…",
  "buildType": "https://slsa.dev/container/v1",
  "builder": {
    "id": "https://git.stella-ops.ru/ci/stella-runner@sha256:f7b7…"
  },
  "metadata": {
    "invocation": {
      "parameters": {"GIT_SHA": "f6a1…"},
      "buildStart": "2025-07-14T06:59:17Z",
      "buildEnd": "2025-07-14T07:01:22Z"
    },
    "completeness": {"parameters": true}
  },
  "materials": [
    {"uri": "git+https://git…", "digest": {"sha1": "f6a1…"}}
  ],
  "rekorLogIndex": 99817    // entry in local Rekor mirror
}
```

---

## 6 Validator Contracts

* For SBOM wrapper – `ISbomValidator` (DLL plug‑in) must return *typed* error list.  
* For YAML policies – JSON‑Schema at `/schemas/policy‑v1.json`.  
* For Rego – OPA `opa eval --fail-defined` under the hood.
* For **Free‑tier quotas** – `IQuotaService` integration tests ensure `quota:<token>` resets at UTC midnight and produces correct `Retry‑After` headers.

---

## 7 Migration Notes

1. **Add `format` column** to existing SBOM wrappers; default to `trivy-json-v2`.  
2. **Populate `layers` & `partial`** via backfill script (ship with `stellopsctl migrate` wizard).  
3. Policy YAML previously stored in Redis → copy to Mongo if persistence enabled.  
4. Prepare `attestations` collection (empty) – safe to create in advance.

---

## 8 Open Questions / Future Work

* How to de‑duplicate *identical* Rego policies differing only in whitespace?  
* Embed *GOST 34.11‑2018* digests when users enable Russian crypto suite?  
* Should enterprise tiers share the same Redis quota keys or switch to JWT claim `tier != Free` bypass?  
* Evaluate sliding‑window quota instead of strict daily reset.  
* Consider rate‑limit for `/layers/missing` to avoid brute‑force enumeration.

---

## 9 Change Log

| Date       | Note                                                                           |
|------------|--------------------------------------------------------------------------------|
| 2025‑07‑14 | **Added:** `format`, `partial`, delta cache keys, YAML policy schema v1.0.     |
| 2025‑07‑12 | **Initial public draft** – SBOM wrapper, Redis keyspace, audit collections.    |

---
