Restructure solution layout by module
This commit is contained in:
master
@@ -1,91 +1,91 @@
|
||||
# Data Schemas & Persistence Contracts
|
||||
|
||||
*Audience* – backend developers, plug‑in authors, DB admins.
|
||||
*Scope* – describes **Redis**, **MongoDB** (optional), and on‑disk blob shapes that power Stella Ops.
|
||||
|
||||
---
|
||||
|
||||
## 0 Document Conventions
|
||||
|
||||
* **CamelCase** for JSON.
|
||||
* All timestamps are **RFC 3339 / ISO 8601** with `Z` (UTC).
|
||||
* `⭑` = planned but *not* shipped yet (kept on Feature Matrix “To Do”).
|
||||
|
||||
---
|
||||
|
||||
## 1 SBOM Wrapper Envelope
|
||||
|
||||
Every SBOM blob (regardless of format) is stored on disk or in object storage with a *sidecar* JSON file that indexes it for the scanners.
|
||||
|
||||
#### 1.1 JSON Shape
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"id": "sha256:417f…", // digest of the SBOM *file* itself
|
||||
"imageDigest": "sha256:e2b9…", // digest of the original container image
|
||||
"created": "2025-07-14T07:02:13Z",
|
||||
"format": "trivy-json-v2", // NEW enum: trivy-json-v2 | spdx-json | cyclonedx-json
|
||||
"layers": [
|
||||
"sha256:d38b…", // layer digests (ordered)
|
||||
"sha256:af45…"
|
||||
],
|
||||
"partial": false, // true => delta SBOM (only some layers)
|
||||
"provenanceId": "prov_0291" // ⭑ link to SLSA attestation (Q1‑2026)
|
||||
}
|
||||
```
|
||||
|
||||
*`format`* **NEW** – added to support **multiple SBOM formats**.
|
||||
*`partial`* **NEW** – true when generated via the **delta SBOM** flow (§1.3).
|
||||
|
||||
#### 1.2 File‑system Layout
|
||||
|
||||
```
|
||||
blobs/
|
||||
├─ 417f… # digest prefix
|
||||
│ ├─ sbom.json # payload (any format)
|
||||
│ └─ sbom.meta.json # wrapper (shape above)
|
||||
```
|
||||
|
||||
> **Note** – blob storage can point at S3, MinIO, or plain disk; driver plug‑ins adapt.
|
||||
|
||||
#### 1.3 Delta SBOM Extension
|
||||
|
||||
When `partial: true`, *only* the missing layers have been scanned.
|
||||
Merging logic inside `scanning` module stitches new data onto the cached full SBOM in Redis.
|
||||
|
||||
---
|
||||
|
||||
## 2 Redis Keyspace
|
||||
|
||||
| Key pattern | Type | TTL | Purpose |
|
||||
|-------------------------------------|---------|------|--------------------------------------------------|
|
||||
| `scan:<digest>` | string | ∞ | Last scan JSON result (as returned by `/scan`) |
|
||||
| `layers:<digest>` | set | 90d | Layers already possessing SBOMs (delta cache) |
|
||||
| `policy:active` | string | ∞ | YAML **or** Rego ruleset |
|
||||
| `quota:<token>` | string | *until next UTC midnight* | Per‑token scan counter for Free tier ({{ quota_token }} scans). |
|
||||
| `policy:history` | list | ∞ | Change audit IDs (see Mongo) |
|
||||
| `feed:nvd:json` | string | 24h | Normalised feed snapshot |
|
||||
| `locator:<imageDigest>` | string | 30d | Maps image digest → sbomBlobId |
|
||||
| `metrics:…` | various | — | Prom / OTLP runtime metrics |
|
||||
|
||||
> **Delta SBOM** uses `layers:*` to skip work in <20 ms.
|
||||
> **Quota enforcement** increments `quota:<token>` atomically; when {{ quota_token }} the API returns **429**.
|
||||
|
||||
---
|
||||
|
||||
## 3 MongoDB Collections (Optional)
|
||||
|
||||
Only enabled when `MONGO_URI` is supplied (for long‑term audit).
|
||||
|
||||
| Collection | Shape (summary) | Indexes |
|
||||
|--------------------|------------------------------------------------------------|-------------------------------------|
|
||||
| `sbom_history` | Wrapper JSON + `replaceTs` on overwrite | `{imageDigest}` `{created}` |
|
||||
| `policy_versions` | `{_id, yaml, rego, authorId, created}` | `{created}` |
|
||||
| `attestations` ⭑ | SLSA provenance doc + Rekor log pointer | `{imageDigest}` |
|
||||
| `audit_log` | Fully rendered RFC 5424 entries (UI & CLI actions) | `{userId}` `{ts}` |
|
||||
|
||||
Schema detail for **policy_versions**:
|
||||
|
||||
# Data Schemas & Persistence Contracts
|
||||
|
||||
*Audience* – backend developers, plug‑in authors, DB admins.
|
||||
*Scope* – describes **Redis**, **MongoDB** (optional), and on‑disk blob shapes that power Stella Ops.
|
||||
|
||||
---
|
||||
|
||||
## 0 Document Conventions
|
||||
|
||||
* **CamelCase** for JSON.
|
||||
* All timestamps are **RFC 3339 / ISO 8601** with `Z` (UTC).
|
||||
* `⭑` = planned but *not* shipped yet (kept on Feature Matrix “To Do”).
|
||||
|
||||
---
|
||||
|
||||
## 1 SBOM Wrapper Envelope
|
||||
|
||||
Every SBOM blob (regardless of format) is stored on disk or in object storage with a *sidecar* JSON file that indexes it for the scanners.
|
||||
|
||||
#### 1.1 JSON Shape
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"id": "sha256:417f…", // digest of the SBOM *file* itself
|
||||
"imageDigest": "sha256:e2b9…", // digest of the original container image
|
||||
"created": "2025-07-14T07:02:13Z",
|
||||
"format": "trivy-json-v2", // NEW enum: trivy-json-v2 | spdx-json | cyclonedx-json
|
||||
"layers": [
|
||||
"sha256:d38b…", // layer digests (ordered)
|
||||
"sha256:af45…"
|
||||
],
|
||||
"partial": false, // true => delta SBOM (only some layers)
|
||||
"provenanceId": "prov_0291" // ⭑ link to SLSA attestation (Q1‑2026)
|
||||
}
|
||||
```
|
||||
|
||||
*`format`* **NEW** – added to support **multiple SBOM formats**.
|
||||
*`partial`* **NEW** – true when generated via the **delta SBOM** flow (§1.3).
|
||||
|
||||
#### 1.2 File‑system Layout
|
||||
|
||||
```
|
||||
blobs/
|
||||
├─ 417f… # digest prefix
|
||||
│ ├─ sbom.json # payload (any format)
|
||||
│ └─ sbom.meta.json # wrapper (shape above)
|
||||
```
|
||||
|
||||
> **Note** – blob storage can point at S3, MinIO, or plain disk; driver plug‑ins adapt.
|
||||
|
||||
#### 1.3 Delta SBOM Extension
|
||||
|
||||
When `partial: true`, *only* the missing layers have been scanned.
|
||||
Merging logic inside `scanning` module stitches new data onto the cached full SBOM in Redis.
|
||||
|
||||
---
|
||||
|
||||
## 2 Redis Keyspace
|
||||
|
||||
| Key pattern | Type | TTL | Purpose |
|
||||
|-------------------------------------|---------|------|--------------------------------------------------|
|
||||
| `scan:<digest>` | string | ∞ | Last scan JSON result (as returned by `/scan`) |
|
||||
| `layers:<digest>` | set | 90d | Layers already possessing SBOMs (delta cache) |
|
||||
| `policy:active` | string | ∞ | YAML **or** Rego ruleset |
|
||||
| `quota:<token>` | string | *until next UTC midnight* | Per‑token scan counter for Free tier ({{ quota_token }} scans). |
|
||||
| `policy:history` | list | ∞ | Change audit IDs (see Mongo) |
|
||||
| `feed:nvd:json` | string | 24h | Normalised feed snapshot |
|
||||
| `locator:<imageDigest>` | string | 30d | Maps image digest → sbomBlobId |
|
||||
| `metrics:…` | various | — | Prom / OTLP runtime metrics |
|
||||
|
||||
> **Delta SBOM** uses `layers:*` to skip work in <20 ms.
|
||||
> **Quota enforcement** increments `quota:<token>` atomically; when {{ quota_token }} the API returns **429**.
|
||||
|
||||
---
|
||||
|
||||
## 3 MongoDB Collections (Optional)
|
||||
|
||||
Only enabled when `MONGO_URI` is supplied (for long‑term audit).
|
||||
|
||||
| Collection | Shape (summary) | Indexes |
|
||||
|--------------------|------------------------------------------------------------|-------------------------------------|
|
||||
| `sbom_history` | Wrapper JSON + `replaceTs` on overwrite | `{imageDigest}` `{created}` |
|
||||
| `policy_versions` | `{_id, yaml, rego, authorId, created}` | `{created}` |
|
||||
| `attestations` ⭑ | SLSA provenance doc + Rekor log pointer | `{imageDigest}` |
|
||||
| `audit_log` | Fully rendered RFC 5424 entries (UI & CLI actions) | `{userId}` `{ts}` |
|
||||
|
||||
Schema detail for **policy_versions**:
|
||||
|
||||
Samples live under `samples/api/scheduler/` (e.g., `schedule.json`, `run.json`, `impact-set.json`, `audit.json`) and mirror the canonical serializer output shown below.
|
||||
|
||||
```jsonc
|
||||
@@ -327,34 +327,34 @@ Materialized view powering the Scheduler UI dashboards. Stores the latest roll-u
|
||||
- Schedulers should call the projection service after every run state change so the cache mirrors planner/runner progress.
|
||||
|
||||
Sample file: `samples/api/scheduler/run-summary.json`.
|
||||
|
||||
---
|
||||
|
||||
## 4 Policy Schema (YAML v1.0)
|
||||
|
||||
Minimal viable grammar (subset of OSV‑SCHEMA ideas).
|
||||
|
||||
```yaml
|
||||
version: "1.0"
|
||||
rules:
|
||||
- name: Block Critical
|
||||
severity: [Critical]
|
||||
action: block
|
||||
- name: Ignore Low Dev
|
||||
severity: [Low, None]
|
||||
environments: [dev, staging]
|
||||
action: ignore
|
||||
expires: "2026-01-01"
|
||||
- name: Escalate RegionalFeed High
|
||||
sources: [NVD, CNNVD, CNVD, ENISA, JVN, BDU]
|
||||
severity: [High, Critical]
|
||||
action: escalate
|
||||
```
|
||||
|
||||
|
||||
---
|
||||
|
||||
## 4 Policy Schema (YAML v1.0)
|
||||
|
||||
Minimal viable grammar (subset of OSV‑SCHEMA ideas).
|
||||
|
||||
```yaml
|
||||
version: "1.0"
|
||||
rules:
|
||||
- name: Block Critical
|
||||
severity: [Critical]
|
||||
action: block
|
||||
- name: Ignore Low Dev
|
||||
severity: [Low, None]
|
||||
environments: [dev, staging]
|
||||
action: ignore
|
||||
expires: "2026-01-01"
|
||||
- name: Escalate RegionalFeed High
|
||||
sources: [NVD, CNNVD, CNVD, ENISA, JVN, BDU]
|
||||
severity: [High, Critical]
|
||||
action: escalate
|
||||
```
|
||||
|
||||
Validation is performed by `policy:mapping.yaml` JSON‑Schema embedded in backend.
|
||||
|
||||
Canonical schema source: `src/StellaOps.Policy/Schemas/policy-schema@1.json` (embedded into `StellaOps.Policy`).
|
||||
`PolicyValidationCli` (see `src/StellaOps.Policy/PolicyValidationCli.cs`) provides the reusable command handler that the main CLI wires up; in the interim it can be invoked from a short host like:
|
||||
Canonical schema source: `src/Policy/__Libraries/StellaOps.Policy/Schemas/policy-schema@1.json` (embedded into `StellaOps.Policy`).
|
||||
`PolicyValidationCli` (see `src/Policy/__Libraries/StellaOps.Policy/PolicyValidationCli.cs`) provides the reusable command handler that the main CLI wires up; in the interim it can be invoked from a short host like:
|
||||
|
||||
```csharp
|
||||
await new PolicyValidationCli().RunAsync(new PolicyValidationCliOptions
|
||||
@@ -363,7 +363,7 @@ await new PolicyValidationCli().RunAsync(new PolicyValidationCliOptions
|
||||
Strict = true,
|
||||
});
|
||||
```
|
||||
|
||||
|
||||
### 4.1 Rego Variant (Advanced – TODO)
|
||||
|
||||
*Accepted but stored as‑is in `rego` field.*
|
||||
@@ -372,7 +372,7 @@ Evaluated via internal **OPA** side‑car once feature graduates from TODO list.
|
||||
### 4.2 Policy Scoring Config (JSON)
|
||||
|
||||
*Schema id.* `https://schemas.stella-ops.org/policy/policy-scoring-schema@1.json`
|
||||
*Source.* `src/StellaOps.Policy/Schemas/policy-scoring-schema@1.json` (embedded in `StellaOps.Policy`), default fixture at `src/StellaOps.Policy/Schemas/policy-scoring-default.json`.
|
||||
*Source.* `src/Policy/__Libraries/StellaOps.Policy/Schemas/policy-scoring-schema@1.json` (embedded in `StellaOps.Policy`), default fixture at `src/Policy/__Libraries/StellaOps.Policy/Schemas/policy-scoring-default.json`.
|
||||
|
||||
```jsonc
|
||||
{
|
||||
@@ -426,25 +426,25 @@ npx ajv validate --spec=draft2020 -c ajv-formats \
|
||||
Planned for Q1‑2026 (kept here for early plug‑in authors).
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"id": "prov_0291",
|
||||
"imageDigest": "sha256:e2b9…",
|
||||
"buildType": "https://slsa.dev/container/v1",
|
||||
"builder": {
|
||||
"id": "https://git.stella-ops.ru/ci/stella-runner@sha256:f7b7…"
|
||||
},
|
||||
"metadata": {
|
||||
"invocation": {
|
||||
"parameters": {"GIT_SHA": "f6a1…"},
|
||||
"buildStart": "2025-07-14T06:59:17Z",
|
||||
"buildEnd": "2025-07-14T07:01:22Z"
|
||||
},
|
||||
"completeness": {"parameters": true}
|
||||
},
|
||||
"materials": [
|
||||
{"uri": "git+https://git…", "digest": {"sha1": "f6a1…"}}
|
||||
],
|
||||
"rekorLogIndex": 99817 // entry in local Rekor mirror
|
||||
{
|
||||
"id": "prov_0291",
|
||||
"imageDigest": "sha256:e2b9…",
|
||||
"buildType": "https://slsa.dev/container/v1",
|
||||
"builder": {
|
||||
"id": "https://git.stella-ops.ru/ci/stella-runner@sha256:f7b7…"
|
||||
},
|
||||
"metadata": {
|
||||
"invocation": {
|
||||
"parameters": {"GIT_SHA": "f6a1…"},
|
||||
"buildStart": "2025-07-14T06:59:17Z",
|
||||
"buildEnd": "2025-07-14T07:01:22Z"
|
||||
},
|
||||
"completeness": {"parameters": true}
|
||||
},
|
||||
"materials": [
|
||||
{"uri": "git+https://git…", "digest": {"sha1": "f6a1…"}}
|
||||
],
|
||||
"rekorLogIndex": 99817 // entry in local Rekor mirror
|
||||
}
|
||||
```
|
||||
|
||||
@@ -509,42 +509,42 @@ done
|
||||
```
|
||||
|
||||
Integration tests can embed the sample fixtures to guarantee deterministic serialisation from the `StellaOps.Notify.Models` DTOs introduced in Sprint 15.
|
||||
|
||||
---
|
||||
|
||||
## 6 Validator Contracts
|
||||
|
||||
* For SBOM wrapper – `ISbomValidator` (DLL plug‑in) must return *typed* error list.
|
||||
* For YAML policies – JSON‑Schema at `/schemas/policy‑v1.json`.
|
||||
* For Rego – OPA `opa eval --fail-defined` under the hood.
|
||||
* For **Free‑tier quotas** – `IQuotaService` integration tests ensure `quota:<token>` resets at UTC midnight and produces correct `Retry‑After` headers.
|
||||
|
||||
---
|
||||
|
||||
## 7 Migration Notes
|
||||
|
||||
1. **Add `format` column** to existing SBOM wrappers; default to `trivy-json-v2`.
|
||||
2. **Populate `layers` & `partial`** via backfill script (ship with `stellopsctl migrate` wizard).
|
||||
3. Policy YAML previously stored in Redis → copy to Mongo if persistence enabled.
|
||||
4. Prepare `attestations` collection (empty) – safe to create in advance.
|
||||
|
||||
---
|
||||
|
||||
## 8 Open Questions / Future Work
|
||||
|
||||
* How to de‑duplicate *identical* Rego policies differing only in whitespace?
|
||||
* Embed *GOST 34.11‑2018* digests when users enable Russian crypto suite?
|
||||
* Should enterprise tiers share the same Redis quota keys or switch to JWT claim `tier != Free` bypass?
|
||||
* Evaluate sliding‑window quota instead of strict daily reset.
|
||||
* Consider rate‑limit for `/layers/missing` to avoid brute‑force enumeration.
|
||||
|
||||
---
|
||||
|
||||
## 9 Change Log
|
||||
|
||||
| Date | Note |
|
||||
|------------|--------------------------------------------------------------------------------|
|
||||
| 2025‑07‑14 | **Added:** `format`, `partial`, delta cache keys, YAML policy schema v1.0. |
|
||||
| 2025‑07‑12 | **Initial public draft** – SBOM wrapper, Redis keyspace, audit collections. |
|
||||
|
||||
---
|
||||
|
||||
---
|
||||
|
||||
## 6 Validator Contracts
|
||||
|
||||
* For SBOM wrapper – `ISbomValidator` (DLL plug‑in) must return *typed* error list.
|
||||
* For YAML policies – JSON‑Schema at `/schemas/policy‑v1.json`.
|
||||
* For Rego – OPA `opa eval --fail-defined` under the hood.
|
||||
* For **Free‑tier quotas** – `IQuotaService` integration tests ensure `quota:<token>` resets at UTC midnight and produces correct `Retry‑After` headers.
|
||||
|
||||
---
|
||||
|
||||
## 7 Migration Notes
|
||||
|
||||
1. **Add `format` column** to existing SBOM wrappers; default to `trivy-json-v2`.
|
||||
2. **Populate `layers` & `partial`** via backfill script (ship with `stellopsctl migrate` wizard).
|
||||
3. Policy YAML previously stored in Redis → copy to Mongo if persistence enabled.
|
||||
4. Prepare `attestations` collection (empty) – safe to create in advance.
|
||||
|
||||
---
|
||||
|
||||
## 8 Open Questions / Future Work
|
||||
|
||||
* How to de‑duplicate *identical* Rego policies differing only in whitespace?
|
||||
* Embed *GOST 34.11‑2018* digests when users enable Russian crypto suite?
|
||||
* Should enterprise tiers share the same Redis quota keys or switch to JWT claim `tier != Free` bypass?
|
||||
* Evaluate sliding‑window quota instead of strict daily reset.
|
||||
* Consider rate‑limit for `/layers/missing` to avoid brute‑force enumeration.
|
||||
|
||||
---
|
||||
|
||||
## 9 Change Log
|
||||
|
||||
| Date | Note |
|
||||
|------------|--------------------------------------------------------------------------------|
|
||||
| 2025‑07‑14 | **Added:** `format`, `partial`, delta cache keys, YAML policy schema v1.0. |
|
||||
| 2025‑07‑12 | **Initial public draft** – SBOM wrapper, Redis keyspace, audit collections. |
|
||||
|
||||
---
|
||||
|
||||
Reference in New Issue
Block a user