master
b97fc7685a
Some checks failed
Build Test Deploy / authority-container (push) Has been cancelled
Build Test Deploy / docs (push) Has been cancelled
Build Test Deploy / deploy (push) Has been cancelled
Build Test Deploy / build-test (push) Has been cancelled
Docs CI / lint-and-preview (push) Has been cancelled
197 lines
7.4 KiB
Markdown
Executable File
197 lines
7.4 KiB
Markdown
Executable File
# Data Schemas & Persistence Contracts
|
||
|
||
*Audience* – backend developers, plug‑in authors, DB admins.
|
||
*Scope* – describes **Redis**, **MongoDB** (optional), and on‑disk blob shapes that power Stella Ops.
|
||
|
||
---
|
||
|
||
## 0 Document Conventions
|
||
|
||
* **CamelCase** for JSON.
|
||
* All timestamps are **RFC 3339 / ISO 8601** with `Z` (UTC).
|
||
* `⭑` = planned but *not* shipped yet (kept on Feature Matrix “To Do”).
|
||
|
||
---
|
||
|
||
## 1 SBOM Wrapper Envelope
|
||
|
||
Every SBOM blob (regardless of format) is stored on disk or in object storage with a *sidecar* JSON file that indexes it for the scanners.
|
||
|
||
#### 1.1 JSON Shape
|
||
|
||
```jsonc
|
||
{
|
||
"id": "sha256:417f…", // digest of the SBOM *file* itself
|
||
"imageDigest": "sha256:e2b9…", // digest of the original container image
|
||
"created": "2025-07-14T07:02:13Z",
|
||
"format": "trivy-json-v2", // NEW enum: trivy-json-v2 | spdx-json | cyclonedx-json
|
||
"layers": [
|
||
"sha256:d38b…", // layer digests (ordered)
|
||
"sha256:af45…"
|
||
],
|
||
"partial": false, // true => delta SBOM (only some layers)
|
||
"provenanceId": "prov_0291" // ⭑ link to SLSA attestation (Q1‑2026)
|
||
}
|
||
```
|
||
|
||
*`format`* **NEW** – added to support **multiple SBOM formats**.
|
||
*`partial`* **NEW** – true when generated via the **delta SBOM** flow (§1.3).
|
||
|
||
#### 1.2 File‑system Layout
|
||
|
||
```
|
||
blobs/
|
||
├─ 417f… # digest prefix
|
||
│ ├─ sbom.json # payload (any format)
|
||
│ └─ sbom.meta.json # wrapper (shape above)
|
||
```
|
||
|
||
> **Note** – blob storage can point at S3, MinIO, or plain disk; driver plug‑ins adapt.
|
||
|
||
#### 1.3 Delta SBOM Extension
|
||
|
||
When `partial: true`, *only* the missing layers have been scanned.
|
||
Merging logic inside `scanning` module stitches new data onto the cached full SBOM in Redis.
|
||
|
||
---
|
||
|
||
## 2 Redis Keyspace
|
||
|
||
| Key pattern | Type | TTL | Purpose |
|
||
|-------------------------------------|---------|------|--------------------------------------------------|
|
||
| `scan:<digest>` | string | ∞ | Last scan JSON result (as returned by `/scan`) |
|
||
| `layers:<digest>` | set | 90d | Layers already possessing SBOMs (delta cache) |
|
||
| `policy:active` | string | ∞ | YAML **or** Rego ruleset |
|
||
| `quota:<token>` | string | *until next UTC midnight* | Per‑token scan counter for Free tier ({{ quota_token }} scans). |
|
||
| `policy:history` | list | ∞ | Change audit IDs (see Mongo) |
|
||
| `feed:nvd:json` | string | 24h | Normalised feed snapshot |
|
||
| `locator:<imageDigest>` | string | 30d | Maps image digest → sbomBlobId |
|
||
| `metrics:…` | various | — | Prom / OTLP runtime metrics |
|
||
|
||
> **Delta SBOM** uses `layers:*` to skip work in <20 ms.
|
||
> **Quota enforcement** increments `quota:<token>` atomically; when {{ quota_token }} the API returns **429**.
|
||
|
||
---
|
||
|
||
## 3 MongoDB Collections (Optional)
|
||
|
||
Only enabled when `MONGO_URI` is supplied (for long‑term audit).
|
||
|
||
| Collection | Shape (summary) | Indexes |
|
||
|--------------------|------------------------------------------------------------|-------------------------------------|
|
||
| `sbom_history` | Wrapper JSON + `replaceTs` on overwrite | `{imageDigest}` `{created}` |
|
||
| `policy_versions` | `{_id, yaml, rego, authorId, created}` | `{created}` |
|
||
| `attestations` ⭑ | SLSA provenance doc + Rekor log pointer | `{imageDigest}` |
|
||
| `audit_log` | Fully rendered RFC 5424 entries (UI & CLI actions) | `{userId}` `{ts}` |
|
||
|
||
Schema detail for **policy_versions**:
|
||
|
||
```jsonc
|
||
{
|
||
"_id": "6619e90b8c5e1f76",
|
||
"yaml": "version: 1.0\nrules:\n - …",
|
||
"rego": null, // filled when Rego uploaded
|
||
"authorId": "u_1021",
|
||
"created": "2025-07-14T08:15:04Z",
|
||
"comment": "Imported via API"
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
## 4 Policy Schema (YAML v1.0)
|
||
|
||
Minimal viable grammar (subset of OSV‑SCHEMA ideas).
|
||
|
||
```yaml
|
||
version: "1.0"
|
||
rules:
|
||
- name: Block Critical
|
||
severity: [Critical]
|
||
action: block
|
||
- name: Ignore Low Dev
|
||
severity: [Low, None]
|
||
environments: [dev, staging]
|
||
action: ignore
|
||
expires: "2026-01-01"
|
||
- name: Escalate RegionalFeed High
|
||
sources: [NVD, CNNVD, CNVD, ENISA, JVN, BDU]
|
||
severity: [High, Critical]
|
||
action: escalate
|
||
```
|
||
|
||
Validation is performed by `policy:mapping.yaml` JSON‑Schema embedded in backend.
|
||
|
||
### 4.1 Rego Variant (Advanced – TODO)
|
||
|
||
*Accepted but stored as‑is in `rego` field.*
|
||
Evaluated via internal **OPA** side‑car once feature graduates from TODO list.
|
||
|
||
---
|
||
|
||
## 5 SLSA Attestation Schema ⭑
|
||
|
||
Planned for Q1‑2026 (kept here for early plug‑in authors).
|
||
|
||
```jsonc
|
||
{
|
||
"id": "prov_0291",
|
||
"imageDigest": "sha256:e2b9…",
|
||
"buildType": "https://slsa.dev/container/v1",
|
||
"builder": {
|
||
"id": "https://git.stella-ops.ru/ci/stella-runner@sha256:f7b7…"
|
||
},
|
||
"metadata": {
|
||
"invocation": {
|
||
"parameters": {"GIT_SHA": "f6a1…"},
|
||
"buildStart": "2025-07-14T06:59:17Z",
|
||
"buildEnd": "2025-07-14T07:01:22Z"
|
||
},
|
||
"completeness": {"parameters": true}
|
||
},
|
||
"materials": [
|
||
{"uri": "git+https://git…", "digest": {"sha1": "f6a1…"}}
|
||
],
|
||
"rekorLogIndex": 99817 // entry in local Rekor mirror
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
## 6 Validator Contracts
|
||
|
||
* For SBOM wrapper – `ISbomValidator` (DLL plug‑in) must return *typed* error list.
|
||
* For YAML policies – JSON‑Schema at `/schemas/policy‑v1.json`.
|
||
* For Rego – OPA `opa eval --fail-defined` under the hood.
|
||
* For **Free‑tier quotas** – `IQuotaService` integration tests ensure `quota:<token>` resets at UTC midnight and produces correct `Retry‑After` headers.
|
||
|
||
---
|
||
|
||
## 7 Migration Notes
|
||
|
||
1. **Add `format` column** to existing SBOM wrappers; default to `trivy-json-v2`.
|
||
2. **Populate `layers` & `partial`** via backfill script (ship with `stellopsctl migrate` wizard).
|
||
3. Policy YAML previously stored in Redis → copy to Mongo if persistence enabled.
|
||
4. Prepare `attestations` collection (empty) – safe to create in advance.
|
||
|
||
---
|
||
|
||
## 8 Open Questions / Future Work
|
||
|
||
* How to de‑duplicate *identical* Rego policies differing only in whitespace?
|
||
* Embed *GOST 34.11‑2018* digests when users enable Russian crypto suite?
|
||
* Should enterprise tiers share the same Redis quota keys or switch to JWT claim `tier != Free` bypass?
|
||
* Evaluate sliding‑window quota instead of strict daily reset.
|
||
* Consider rate‑limit for `/layers/missing` to avoid brute‑force enumeration.
|
||
|
||
---
|
||
|
||
## 9 Change Log
|
||
|
||
| Date | Note |
|
||
|------------|--------------------------------------------------------------------------------|
|
||
| 2025‑07‑14 | **Added:** `format`, `partial`, delta cache keys, YAML policy schema v1.0. |
|
||
| 2025‑07‑12 | **Initial public draft** – SBOM wrapper, Redis keyspace, audit collections. |
|
||
|
||
---
|