feat: Add RustFS artifact object store and migration tool
- Implemented RustFsArtifactObjectStore for managing artifacts in RustFS. - Added unit tests for RustFsArtifactObjectStore functionality. - Created a RustFS migrator tool to transfer objects from S3 to RustFS. - Introduced policy preview and report models for API integration. - Added fixtures and tests for policy preview and report functionality. - Included necessary metadata and scripts for cache_pkg package.
This commit is contained in:
master
@@ -49,7 +49,7 @@
|
||||
|
||||
* **Fulcio** (Sigstore CA) — issues short‑lived signing certs (keyless).
|
||||
* **Rekor v2** (tile‑backed transparency log).
|
||||
* **MinIO** — S3‑compatible object store with lifecycle & Object Lock.
|
||||
* **RustFS** — offline-first object store with deterministic REST API (S3/MinIO fallback available for legacy installs).
|
||||
* **MongoDB** — catalog, advisories, VEX, scheduler, notify.
|
||||
* **Queue** — Redis Streams / NATS / RabbitMQ (pluggable).
|
||||
* **OCI Registry** — must support **Referrers API** (discover SBOMs/signatures).
|
||||
@@ -81,7 +81,7 @@ flowchart LR
|
||||
ATT[Attestor\n(Rekor v2 submit/verify)]
|
||||
UI[Web UI (Angular)]
|
||||
Z[Zastava\n(Runtime Inspector/Enforcer)]
|
||||
MIN[(MinIO S3)]
|
||||
RFS[(RustFS object store)]
|
||||
MGO[(MongoDB)]
|
||||
QUE[(Queue/Streams)]
|
||||
end
|
||||
@@ -94,7 +94,7 @@ flowchart LR
|
||||
CLI -->|scan/build| SW
|
||||
SW -->|jobs| QUE
|
||||
QUE --> WK
|
||||
WK --> MIN
|
||||
WK --> RFS
|
||||
SW --> MGO
|
||||
CONC --> MGO
|
||||
EXC --> MGO
|
||||
@@ -225,13 +225,13 @@ LS --> IA: PoE (mTLS client cert or JWT with cnf=K_inst), CRL/OCSP/introspect
|
||||
|
||||
---
|
||||
|
||||
## 6) Storage & catalogs (MinIO/Mongo)
|
||||
## 6) Storage & catalogs (RustFS/Mongo)
|
||||
|
||||
**RustFS layout (default)**
|
||||
|
||||
**MinIO layout**
|
||||
|
||||
```
|
||||
s3://stellaops/
|
||||
layers/<sha256>/sbom.cdx.json.zst
|
||||
```
|
||||
rustfs://stellaops/
|
||||
layers/<sha256>/sbom.cdx.json.zst
|
||||
layers/<sha256>/sbom.spdx.json.zst
|
||||
images/<imgDigest>/inventory.cdx.pb
|
||||
images/<imgDigest>/usage.cdx.pb
|
||||
@@ -248,7 +248,7 @@ s3://stellaops/
|
||||
|
||||
**Retention**
|
||||
|
||||
* MinIO **ILM** for coarse TTL; Scanner.WebService GC decrements `refCount` and deletes unreferenced metadata; **Object Lock** for immutable classes (auditable artifacts).
|
||||
* RustFS applies retention via `X-RustFS-Retain-Seconds`; Scanner.WebService GC decrements `refCount` and deletes unreferenced metadata; S3/MinIO fallback retains native Object Lock when enabled.
|
||||
|
||||
---
|
||||
|
||||
@@ -395,7 +395,7 @@ services:
|
||||
ui: { image: stellaops/ui, depends_on: [scanner-web, concelier, excititor, scheduler-web, notify-web] }
|
||||
```
|
||||
|
||||
* **Backups:** Mongo dumps; MinIO versioned buckets & replication; Rekor v2 DB snapshots; JWKS/Fulcio/KMS key rotation.
|
||||
* **Backups:** Mongo dumps; RustFS snapshots (or S3 versioning when fallback driver is used); Rekor v2 DB snapshots; JWKS/Fulcio/KMS key rotation.
|
||||
* **Ops runbooks:** Scheduler catch‑up after Concelier/Excititor recovery; connector key rotation (Slack/Teams/SMTP).
|
||||
* **SLOs & alerts:** lag between Concelier/Excititor export and first rescan verdict; delivery failure rates by channel.
|
||||
|
||||
@@ -408,7 +408,7 @@ services:
|
||||
* **Notify metrics:** `notify.sent_total{channel}`, `notify.dropped_total{reason}`, `notify.digest_coalesced_total`, `notify.latency_ms`.
|
||||
* **Tracing:** per‑stage spans; correlation IDs across Scanner→Signer→Attestor and Concelier/Excititor→Scheduler→Scanner→Notify.
|
||||
* **Audit logs:** every signing records `license_id`, `image_digest`, `policy_digest`, and Rekor UUID; Scheduler records who scheduled what; Notify records where, when, and why messages were sent or deduped.
|
||||
* **Compliance:** MinIO **Object Lock** for immutable artifacts; reproducible outputs via policy digest + SBOM digest in predicate.
|
||||
* **Compliance:** RustFS retention headers (or MinIO Object Lock when operating in S3 mode) keep immutable artifacts tamper‑resistant; reproducible outputs via policy digest + SBOM digest in predicate.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -382,28 +382,65 @@ Request body mirrors policy preview inputs (image digest plus findings). The ser
|
||||
```json
|
||||
{
|
||||
"report": {
|
||||
"reportId": "report-3def5f362aa475ef14b6",
|
||||
"imageDigest": "sha256:deadbeef",
|
||||
"reportId": "report-9f8cde21aab54321",
|
||||
"imageDigest": "sha256:7dbe0c9a5d4f1c8184007e9d94dbe55928f8a2db5ab9c1c2d4a2f7bbcdfe1234",
|
||||
"generatedAt": "2025-10-23T15:32:22Z",
|
||||
"verdict": "blocked",
|
||||
"policy": { "revisionId": "rev-1", "digest": "27d2ec2b34feedc304fc564d252ecee1c8fa14ea581a5ff5c1ea8963313d5c8d" },
|
||||
"summary": { "total": 1, "blocked": 1, "warned": 0, "ignored": 0, "quieted": 0 },
|
||||
"policy": {
|
||||
"revisionId": "rev-42",
|
||||
"digest": "8a0f72f8dc5c51c46991db3bba34e9b3c0c8e944a7a6d0a9c29a9aa6b8439876"
|
||||
},
|
||||
"summary": { "total": 2, "blocked": 1, "warned": 1, "ignored": 0, "quieted": 0 },
|
||||
"verdicts": [
|
||||
{
|
||||
"findingId": "finding-1",
|
||||
"findingId": "library:pkg/openssl@1.1.1w",
|
||||
"status": "Blocked",
|
||||
"ruleName": "Block Critical",
|
||||
"ruleAction": "Block",
|
||||
"score": 40.5,
|
||||
"ruleName": "Block vendor unknowns",
|
||||
"ruleAction": "block",
|
||||
"notes": "Unknown vendor telemetry — medium confidence band.",
|
||||
"score": 19.5,
|
||||
"configVersion": "1.0",
|
||||
"inputs": {
|
||||
"reachabilityWeight": 0.45,
|
||||
"baseScore": 40.5,
|
||||
"severityWeight": 90,
|
||||
"trustWeight": 1,
|
||||
"trustWeight.NVD": 1,
|
||||
"reachability.runtime": 0.45
|
||||
"severityWeight": 50,
|
||||
"trustWeight": 0.65,
|
||||
"reachabilityWeight": 0.6,
|
||||
"baseScore": 19.5,
|
||||
"trustWeight.vendor": 0.65,
|
||||
"reachability.unknown": 0.6,
|
||||
"unknownConfidence": 0.55,
|
||||
"unknownAgeDays": 5
|
||||
},
|
||||
"quietedBy": null,
|
||||
"quiet": false,
|
||||
"unknownConfidence": 0.55,
|
||||
"confidenceBand": "medium",
|
||||
"unknownAgeDays": 5,
|
||||
"sourceTrust": "vendor",
|
||||
"reachability": "unknown"
|
||||
},
|
||||
{
|
||||
"findingId": "library:pkg/zlib@1.3.1",
|
||||
"status": "Warned",
|
||||
"ruleName": "Runtime mitigation required",
|
||||
"ruleAction": "warn",
|
||||
"notes": "Runtime reachable unknown — mitigation window required.",
|
||||
"score": 18.75,
|
||||
"configVersion": "1.0",
|
||||
"inputs": {
|
||||
"severityWeight": 75,
|
||||
"trustWeight": 1,
|
||||
"reachabilityWeight": 0.45,
|
||||
"baseScore": 33.75,
|
||||
"reachability.runtime": 0.45,
|
||||
"warnPenalty": 15,
|
||||
"unknownConfidence": 0.35,
|
||||
"unknownAgeDays": 13
|
||||
},
|
||||
"quietedBy": null,
|
||||
"quiet": false,
|
||||
"unknownConfidence": 0.35,
|
||||
"confidenceBand": "medium",
|
||||
"unknownAgeDays": 13,
|
||||
"sourceTrust": "NVD",
|
||||
"reachability": "runtime"
|
||||
}
|
||||
@@ -412,21 +449,21 @@ Request body mirrors policy preview inputs (image digest plus findings). The ser
|
||||
},
|
||||
"dsse": {
|
||||
"payloadType": "application/vnd.stellaops.report+json",
|
||||
"payload": "<base64 canonical report>",
|
||||
"payload": "eyJyZXBvcnQiOnsicmVwb3J0SWQiOiJyZXBvcnQtOWY4Y2RlMjFhYWI1NDMyMSJ9fQ==",
|
||||
"signatures": [
|
||||
{
|
||||
"keyId": "scanner-report-signing",
|
||||
"algorithm": "hs256",
|
||||
"signature": "<base64 signature>"
|
||||
"signature": "MEQCIGHscnJ2bm9wYXlsb2FkZXIAIjANBgkqhkiG9w0BAQsFAAOCAQEASmFja3Nvbk1ldGE="
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- The `report` object omits null fields and is deterministic (ISO timestamps, sorted keys).
|
||||
- The `report` object omits null fields and is deterministic (ISO timestamps, sorted keys) while surfacing `unknownConfidence`, `confidenceBand`, and `unknownAgeDays` for auditability.
|
||||
- `dsse` follows the DSSE (Dead Simple Signing Envelope) shape; `payload` is the canonical UTF-8 JSON and `signatures[0].signature` is the base64 HMAC/Ed25519 value depending on configuration.
|
||||
- A runnable sample envelope is available at `samples/api/reports/report-sample.dsse.json` for tooling tests or signature verification.
|
||||
- Full offline samples live at `samples/policy/policy-report-unknown.json` (request + response) and `samples/api/reports/report-sample.dsse.json` (envelope fixture) for tooling tests or signature verification.
|
||||
|
||||
**Response 404** – `application/problem+json` payload with type `https://stellaops.org/problems/not-found` when the scan identifier is unknown.
|
||||
|
||||
|
||||
@@ -176,8 +176,9 @@ Authority now understands two flavours of sender-constrained OAuth clients:
|
||||
```
|
||||
Operators can override any field via environment variables (e.g. `STELLAOPS_AUTHORITY__SECURITY__SENDERCONSTRAINTS__DPOP__NONCE__STORE=redis`).
|
||||
- Declare client `audiences` in bootstrap manifests or plug-in provisioning metadata; Authority now defaults the token `aud` claim and `resource` indicator from this list, which is also used to trigger nonce enforcement for audiences such as `signer` and `attestor`.
|
||||
- **Mutual TLS clients** – client registrations may declare an mTLS binding (`senderConstraint: mtls`). When enabled via `security.senderConstraints.mtls`, Authority validates the presented client certificate against stored bindings (`certificateBindings[]`), optional chain verification, and timing windows. Successful requests embed `cnf.x5t#S256` into the access token so resource servers can enforce the certificate thumbprint.
|
||||
- Certificate bindings record the certificate thumbprint, optional SANs, subject/issuer metadata, and activation windows. Operators can enforce subject regexes, SAN type allow-lists (`dns`, `uri`, `ip`), trusted certificate authorities, and rotation grace via `security.senderConstraints.mtls.*`.
|
||||
- **Mutual TLS clients** – client registrations may declare an mTLS binding (`senderConstraint: mtls`). When enabled via `security.senderConstraints.mtls`, Authority validates the presented client certificate against stored bindings (`certificateBindings[]`), optional chain verification, and timing windows. Successful requests embed `cnf.x5t#S256` into the access token (and introspection output) so resource servers can enforce the certificate thumbprint.
|
||||
- `security.senderConstraints.mtls.enforceForAudiences` forces mTLS whenever the requested `aud`/`resource` (or the client's configured audiences) intersect the configured allow-list (default includes `signer`). Clients configured for different sender constraints are rejected early so operator policy remains consistent.
|
||||
- Certificate bindings now act as an allow-list: Authority verifies thumbprint, subject, issuer, serial number, and any declared SAN values against the presented certificate, with rotation grace windows applied to `notBefore/notAfter`. Operators can enforce subject regexes, SAN type allow-lists (`dns`, `uri`, `ip`), trusted certificate authorities, and rotation grace via `security.senderConstraints.mtls.*`.
|
||||
|
||||
Both modes persist additional metadata in `authority_tokens`: `senderConstraint` records the enforced policy, while `senderKeyThumbprint` stores the DPoP JWK thumbprint or mTLS certificate hash captured at issuance. Downstream services can rely on these fields (and the corresponding `cnf` claim) when auditing offline copies of the token store.
|
||||
|
||||
|
||||
@@ -306,7 +306,22 @@ Validation occurs alongside policy binding (`PolicyScoringConfigBinder`), produc
|
||||
**Runtime usage**
|
||||
- `trustOverrides` are matched against `finding.tags` (`trust:<key>`) first, then `finding.source`/`finding.vendor`; missing keys default to `1.0`.
|
||||
- `reachabilityBuckets` consume `finding.tags` with prefix `reachability:` (fallback `usage:` or `unknown`). Missing buckets fall back to `unknown` weight when present, otherwise `1.0`.
|
||||
- Policy verdicts expose scoring inputs (`severityWeight`, `trustWeight`, `reachabilityWeight`, `baseScore`, penalties) plus unknown-state metadata (`unknownConfidence`, `unknownAgeDays`, `confidenceBand`) for auditability. See `samples/policy/policy-preview-unknown.json` for an end-to-end preview payload.
|
||||
- Policy verdicts expose scoring inputs (`severityWeight`, `trustWeight`, `reachabilityWeight`, `baseScore`, penalties) plus unknown-state metadata (`unknownConfidence`, `unknownAgeDays`, `confidenceBand`) for auditability. See `samples/policy/policy-preview-unknown.json` and `samples/policy/policy-report-unknown.json` for offline reference payloads validated against the published schemas below.
|
||||
|
||||
Validate the samples locally with **Ajv** before publishing changes:
|
||||
|
||||
```bash
|
||||
# install once per checkout (offline-safe):
|
||||
npm install --no-save ajv-cli@5 ajv-formats@2
|
||||
|
||||
npx ajv validate --spec=draft2020 -c ajv-formats \
|
||||
-s docs/schemas/policy-preview-sample@1.json \
|
||||
-d samples/policy/policy-preview-unknown.json
|
||||
|
||||
npx ajv validate --spec=draft2020 -c ajv-formats \
|
||||
-s docs/schemas/policy-report-sample@1.json \
|
||||
-d samples/policy/policy-report-unknown.json
|
||||
```
|
||||
- Unknown confidence derives from `unknown-age-days:` (preferred) or `unknown-since:` + `observed-at:` tags; with no hints the engine keeps `initial` confidence. Values decay by `decayPerDay` down to `floor`, then resolve to the first matching `bands[].name`.
|
||||
|
||||
---
|
||||
|
||||
@@ -56,7 +56,7 @@
|
||||
## 3 Test Harness
|
||||
|
||||
* **Runner** – `perf/run.sh`, accepts `--phase` and `--samples`.
|
||||
* **Language analyzers microbench** – `dotnet run --project bench/Scanner.Analyzers/StellaOps.Bench.ScannerAnalyzers/StellaOps.Bench.ScannerAnalyzers.csproj -- --repo-root . --out bench/Scanner.Analyzers/baseline.csv` produces deterministic CSVs for analyzer scenarios (Node today, others as they land).
|
||||
* **Language analyzers microbench** – `dotnet run --project bench/Scanner.Analyzers/StellaOps.Bench.ScannerAnalyzers/StellaOps.Bench.ScannerAnalyzers.csproj -- --repo-root . --out bench/Scanner.Analyzers/baseline.csv --json out/bench/scanner-analyzers/latest.json --prom out/bench/scanner-analyzers/latest.prom --commit $(git rev-parse HEAD)` produces CSV + JSON + Prometheus gauges for analyzer scenarios. Runs fail if `max_ms` regresses ≥ 20 % against `baseline.csv` or if thresholds are exceeded.
|
||||
* **Metrics** – Prometheus + `jq` extracts; aggregated via `scripts/aggregate.ts`.
|
||||
* **CI** – GitLab CI job *benchmark* publishes JSON to `bench‑artifacts/`.
|
||||
* **Visualisation** – Grafana dashboard *Stella‑Perf* (provisioned JSON).
|
||||
@@ -143,7 +143,9 @@ P99 = 48 ms. Meets 50 ms gate.
|
||||
|
||||
## 8 Trend Snapshot
|
||||
|
||||
|
||||
|
||||
|
||||
> **Grafana/Alerting** – Import `docs/ops/scanner-analyzers-grafana-dashboard.json` and point it at the Prometheus datasource storing `scanner_analyzer_bench_*` metrics. Configure an alert on `scanner_analyzer_bench_regression_ratio` ≥ 1.20 (default limit); the bundled Stat panel surfaces breached scenarios (non-zero values). On-call runbook: `docs/ops/scanner-analyzers-operations.md`.
|
||||
|
||||
_Plot generated weekly by `scripts/update‑trend.py`; shows last 12 weeks P95 per phase._
|
||||
|
||||
|
||||
@@ -96,11 +96,29 @@ _No external fonts or JS – true offline guarantee._
|
||||
| **Import / Export** | Buttons map to `/policy/import` and `/policy/export`. Accepts `.yaml`, `.rego`, `.zip` (bundle). |
|
||||
| **History** | Immutable audit log; diff viewer highlights rule changes. |
|
||||
|
||||
#### 3.3.1 YAML → Rego Bridge
|
||||
|
||||
If you paste YAML but enable **Strict Mode** (toggle), backend converts to Rego under the hood, stores both representations, and shows a side‑by‑side diff.
|
||||
|
||||
### 3.4 📌 Settings Enhancements
|
||||
#### 3.3.1 YAML → Rego Bridge
|
||||
|
||||
If you paste YAML but enable **Strict Mode** (toggle), backend converts to Rego under the hood, stores both representations, and shows a side‑by‑side diff.
|
||||
|
||||
#### 3.3.2 Preview / Report Fixtures
|
||||
|
||||
- Use the offline fixtures (`samples/policy/policy-preview-unknown.json` and `samples/policy/policy-report-unknown.json`) to exercise the Policies screens without a live backend; both payloads include confidence bands, unknown-age tags, and scoring inputs that map directly to the UI panels.
|
||||
- Keep them in lock-step with the API by validating any edits with Ajv:
|
||||
|
||||
```bash
|
||||
# install once per checkout (offline-safe):
|
||||
npm install --no-save ajv-cli@5 ajv-formats@2
|
||||
|
||||
npx ajv validate --spec=draft2020 -c ajv-formats \
|
||||
-s docs/schemas/policy-preview-sample@1.json \
|
||||
-d samples/policy/policy-preview-unknown.json
|
||||
|
||||
npx ajv validate --spec=draft2020 -c ajv-formats \
|
||||
-s docs/schemas/policy-report-sample@1.json \
|
||||
-d samples/policy/policy-report-unknown.json
|
||||
```
|
||||
|
||||
### 3.4 📌 Settings Enhancements
|
||||
|
||||
| Setting | Details |
|
||||
| --------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
|
||||
@@ -17,11 +17,11 @@ completely isolated network:
|
||||
| **Provenance** | Cosign signature, SPDX 2.3 SBOM, in‑toto SLSA attestation |
|
||||
| **Attested manifest** | `offline-manifest.json` + detached JWS covering bundle metadata, signed during export. |
|
||||
| **Delta patches** | Daily diff bundles keep size \< 350 MB |
|
||||
| **Scanner plug-ins** | OS analyzers plus the Node.js, Go, and .NET language analyzers packaged under `plugins/scanner/analyzers/**` with manifests so Workers load deterministically offline. |
|
||||
| **Scanner plug-ins** | OS analyzers plus the Node.js, Go, .NET, and Python language analyzers packaged under `plugins/scanner/analyzers/**` with manifests so Workers load deterministically offline. |
|
||||
|
||||
**RU BDU note:** ship the official Russian Trusted Root/Sub CA bundle (`certificates/russian_trusted_bundle.pem`) inside the kit so `concelier:httpClients:source.bdu:trustedRootPaths` can resolve it when the service runs in an air‑gapped network. Drop the most recent `vulxml.zip` alongside the kit if operators need a cold-start cache.
|
||||
|
||||
**Language analyzers:** the kit now carries the restart-only Node.js, Go, and .NET analyzer plug-ins (`plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Node/`, `plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Go/`, `plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.DotNet/`). Drop the directories alongside Worker binaries so the unified plug-in catalog can load them without outbound fetches; upcoming Python/Rust plug-ins will follow the same layout.
|
||||
**Language analyzers:** the kit now carries the restart-only Node.js, Go, .NET, and Python analyzer plug-ins (`plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Node/`, `...Lang.Go/`, `...Lang.DotNet/`, `...Lang.Python/`). Drop the directories alongside Worker binaries so the unified plug-in catalog can load them without outbound fetches; Rust remains on the Wave 4 roadmap.
|
||||
|
||||
*Scanner core:* C# 12 on **.NET {{ dotnet }}**.
|
||||
*Imports are idempotent and atomic — no service downtime.*
|
||||
@@ -99,6 +99,24 @@ Example excerpt (2025-10-23 kit) showing the Go and .NET analyzer plug-in payloa
|
||||
"size": 647,
|
||||
"capturedAt": "2025-10-23T00:00:00Z"
|
||||
}
|
||||
{
|
||||
"name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Python/StellaOps.Scanner.Analyzers.Lang.Python.dll",
|
||||
"sha256": "28b6e06c7cabf3b78f13f801cbb14962093f3d42c4ae9ec01babbcd14cda4644",
|
||||
"size": 53760,
|
||||
"capturedAt": "2025-10-23T00:00:00Z"
|
||||
}
|
||||
{
|
||||
"name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Python/StellaOps.Scanner.Analyzers.Lang.Python.pdb",
|
||||
"sha256": "be4e34b4dc9a790fe1299e84213343b7c8ea90a2d22e5d7d1aa7585b8fedc946",
|
||||
"size": 34516,
|
||||
"capturedAt": "2025-10-23T00:00:00Z"
|
||||
}
|
||||
{
|
||||
"name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Python/manifest.json",
|
||||
"sha256": "bceea1e7542aae860b0ec5ba7b8b3aa960b21edc4d1efe60afc98ce289341ac3",
|
||||
"size": 671,
|
||||
"capturedAt": "2025-10-23T00:00:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
@@ -130,7 +148,7 @@ The CLI validates recorded digests (when `.metadata.json` is present) before str
|
||||
**Quick smoke test:** before import, verify the tarball carries the Go analyzer plug-in:
|
||||
|
||||
```bash
|
||||
tar -tzf stella-ops-offline-kit-<DATE>.tgz 'plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Go/*' 'plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.DotNet/*'
|
||||
tar -tzf stella-ops-offline-kit-<DATE>.tgz 'plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Go/*' 'plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.DotNet/*' 'plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Python/*'
|
||||
```
|
||||
|
||||
The manifest lookup above and this `tar` listing should both surface the Go analyzer DLL, PDB, and manifest entries before the kit is promoted.
|
||||
|
||||
@@ -99,6 +99,8 @@ plan? = <plan name> // optional hint for UIs; not used for e
|
||||
* **Client Credentials** (service→service):
|
||||
|
||||
* **mTLS**: mutual TLS + `client_id` → bound token (`cnf.x5t#S256`)
|
||||
* `security.senderConstraints.mtls.enforceForAudiences` forces the mTLS path when requested `aud`/`resource` values intersect high-value audiences (defaults include `signer`). Authority rejects clients attempting to use DPoP/basic secrets for these audiences.
|
||||
* Stored `certificateBindings` are authoritative: thumbprint, subject, issuer, serial number, and SAN values are matched against the presented certificate, with rotation grace applied to activation windows. Failures surface deterministic error codes (e.g. `certificate_binding_subject_mismatch`).
|
||||
* **private_key_jwt**: JWT‑based client auth + **DPoP** header (preferred for tools and CLI)
|
||||
* **Device Code** (CLI): `POST /oauth/device/code` + `POST /oauth/token` poll
|
||||
* **Authorization Code + PKCE** (UI): standard
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# component_architecture_devops.md — **Stella Ops Release & Operations** (2025Q4)
|
||||
|
||||
> **Scope.** Implementation‑ready blueprint for **how Stella Ops is built, versioned, signed, distributed, upgraded, licensed (PoE)**, and operated in customer environments (online and air‑gapped). Covers reproducible builds, supply‑chain attestations, registries, offline kits, migration/rollback, artifact lifecycle (MinIO/Mongo), monitoring SLOs, and customer activation.
|
||||
> **Scope.** Implementation‑ready blueprint for **how Stella Ops is built, versioned, signed, distributed, upgraded, licensed (PoE)**, and operated in customer environments (online and air‑gapped). Covers reproducible builds, supply‑chain attestations, registries, offline kits, migration/rollback, artifact lifecycle (RustFS default + Mongo, S3 fallback), monitoring SLOs, and customer activation.
|
||||
|
||||
---
|
||||
|
||||
@@ -257,12 +257,12 @@ Signer validates **scanner** image’s cosign identity + calendar tag for **rele
|
||||
|
||||
---
|
||||
|
||||
## 7) Artifact lifecycle & storage (MinIO/Mongo)
|
||||
## 7) Artifact lifecycle & storage (RustFS/Mongo)
|
||||
|
||||
### 7.1 Buckets & prefixes (MinIO)
|
||||
### 7.1 Buckets & prefixes (RustFS)
|
||||
|
||||
```
|
||||
s3://stellaops/
|
||||
rustfs://stellaops/
|
||||
scanner/
|
||||
layers/<sha256>/sbom.cdx.json.zst
|
||||
images/<imgDigest>/inventory.cdx.pb
|
||||
@@ -283,7 +283,7 @@ s3://stellaops/
|
||||
|
||||
* **`short`**: working artifacts (diffs, queues) — TTL 7–14 days.
|
||||
* **`default`**: SBOMs & indexes — TTL 90–180 days (configurable).
|
||||
* **`compliance`**: signed reports & attested exports — **Object Lock** (governance/compliance) 1–7 years.
|
||||
* **`compliance`**: signed reports & attested exports — retention enforced via RustFS hold or S3 Object Lock (governance/compliance) 1–7 years.
|
||||
|
||||
### 7.3 Artifact Lifecycle Controller (ALC)
|
||||
|
||||
@@ -292,6 +292,9 @@ s3://stellaops/
|
||||
* Artifacts referenced by **reports** or **tickets** are pinned.
|
||||
* ILM actions logged; UI shows per‑class usage & upcoming purges.
|
||||
|
||||
> **Migration note.** Follow `docs/ops/scanner-rustfs-migration.md` when transitioning existing
|
||||
> MinIO buckets to RustFS. The provided migrator is idempotent and safe to rerun per prefix.
|
||||
|
||||
### 7.4 Mongo retention
|
||||
|
||||
* **Scanner**: `runtime.events` use TTL (e.g., 30–90 days); **catalog** permanent.
|
||||
@@ -313,7 +316,7 @@ s3://stellaops/
|
||||
* **Golden signals**:
|
||||
|
||||
* **Latency**: token issuance, sign→attest round‑trip, scan enqueue→emit, export build.
|
||||
* **Saturation**: queue depth, Mongo write IOPS, MinIO net throughput.
|
||||
* **Saturation**: queue depth, Mongo write IOPS, RustFS throughput / queue depth (or S3 metrics when in fallback mode).
|
||||
* **Traffic**: scans/min, attestations/min, webhook admits/min.
|
||||
* **Errors**: 5xx rates, cosign verification failures, Rekor timeouts.
|
||||
|
||||
@@ -460,7 +463,7 @@ services:
|
||||
* `attestor.submit_latency_seconds{quantile=0.95}` < 0.3.
|
||||
* `scanner.scan_latency_seconds{quantile=0.95}` < target per image size.
|
||||
* `concelier.export.duration_seconds` stable; `excititor.consensus.conflicts_total` not exploding after policy changes.
|
||||
* MinIO `s3_requests_errors_total` near zero; Mongo `opcounters` hit expected baseline.
|
||||
* RustFS request error rate near zero (or `s3_requests_errors_total` when operating against S3); Mongo `opcounters` hit expected baseline.
|
||||
|
||||
### Appendix B — Upgrade safety checklist
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# component_architecture_scanner.md — **Stella Ops Scanner** (2025Q4)
|
||||
|
||||
> **Scope.** Implementation‑ready architecture for the **Scanner** subsystem: WebService, Workers, analyzers, SBOM assembly (inventory & usage), per‑layer caching, three‑way diffs, artifact catalog (MinIO+Mongo), attestation hand‑off, and scale/security posture. This document is the contract between the scanning plane and everything else (Policy, Excititor, Concelier, UI, CLI).
|
||||
> **Scope.** Implementation‑ready architecture for the **Scanner** subsystem: WebService, Workers, analyzers, SBOM assembly (inventory & usage), per‑layer caching, three‑way diffs, artifact catalog (RustFS default + Mongo, S3-compatible fallback), attestation hand‑off, and scale/security posture. This document is the contract between the scanning plane and everything else (Policy, Excititor, Concelier, UI, CLI).
|
||||
|
||||
---
|
||||
|
||||
@@ -23,7 +23,7 @@ src/
|
||||
├─ StellaOps.Scanner.WebService/ # REST control plane, catalog, diff, exports
|
||||
├─ StellaOps.Scanner.Worker/ # queue consumer; executes analyzers
|
||||
├─ StellaOps.Scanner.Models/ # DTOs, evidence, graph nodes, CDX/SPDX adapters
|
||||
├─ StellaOps.Scanner.Storage/ # Mongo repositories; MinIO object client; ILM/GC
|
||||
├─ StellaOps.Scanner.Storage/ # Mongo repositories; RustFS object client (default) + S3 fallback; ILM/GC
|
||||
├─ StellaOps.Scanner.Queue/ # queue abstraction (Redis/NATS/RabbitMQ)
|
||||
├─ StellaOps.Scanner.Cache/ # layer cache; file CAS; bloom/bitmap indexes
|
||||
├─ StellaOps.Scanner.EntryTrace/ # ENTRYPOINT/CMD → terminal program resolver (shell AST)
|
||||
@@ -81,7 +81,7 @@ The DI extension (`AddScannerQueue`) wires the selected transport, so future add
|
||||
## 2) External dependencies
|
||||
|
||||
* **OCI registry** with **Referrers API** (discover attached SBOMs/signatures).
|
||||
* **MinIO** (S3‑compatible) for SBOM artifacts; **Object Lock** for immutable classes; **ILM** for TTL.
|
||||
* **RustFS** (default, offline-first) for SBOM artifacts; optional S3/MinIO compatibility retained for migration; **Object Lock** semantics emulated via retention headers; **ILM** for TTL.
|
||||
* **MongoDB** for catalog, job state, diffs, ILM rules.
|
||||
* **Queue** (Redis Streams/NATS/RabbitMQ).
|
||||
* **Authority** (on‑prem OIDC) for **OpToks** (DPoP/mTLS).
|
||||
@@ -133,7 +133,7 @@ No confidences. Either a fact is proven with listed mechanisms, or it is not cla
|
||||
* `jobs { _id, kind, args, state, startedAt, heartbeatAt, endedAt, error }`
|
||||
* `lifecycleRules { ruleId, scope, ttlDays, retainIfReferenced, immutable }`
|
||||
|
||||
### 3.3 Object store layout (MinIO)
|
||||
### 3.3 Object store layout (RustFS)
|
||||
|
||||
```
|
||||
layers/<sha256>/sbom.cdx.json.zst
|
||||
@@ -145,6 +145,13 @@ diffs/<old>_<new>/diff.json.zst
|
||||
attest/<artifactSha256>.dsse.json # DSSE bundle (cert chain + Rekor proof)
|
||||
```
|
||||
|
||||
RustFS exposes a deterministic HTTP API (`PUT|GET|DELETE /api/v1/buckets/{bucket}/objects/{key}`).
|
||||
Scanner clients tag immutable uploads with `X-RustFS-Immutable: true` and, when retention applies,
|
||||
`X-RustFS-Retain-Seconds: <ttlSeconds>`. Additional headers can be injected via
|
||||
`scanner.artifactStore.headers` to support custom auth or proxy requirements. Legacy MinIO/S3
|
||||
deployments remain supported by setting `scanner.artifactStore.driver = "s3"` during phased
|
||||
migrations.
|
||||
|
||||
---
|
||||
|
||||
## 4) REST API (Scanner.WebService)
|
||||
@@ -396,7 +403,7 @@ scanner:
|
||||
* **HA**: WebService horizontal scale; Workers autoscale by queue depth & CPU; distributed locks on layers.
|
||||
* **Retention**: ILM rules per artifact class (`short`, `default`, `compliance`); **Object Lock** for compliance artifacts (reports, signed SBOMs).
|
||||
* **Upgrades**: bump **cache schema** when analyzer outputs change; WebService triggers refresh of dependent artifacts.
|
||||
* **Backups**: Mongo (daily dumps); MinIO (versioned buckets, replication); Rekor v2 DB snapshots.
|
||||
* **Backups**: Mongo (daily dumps); RustFS snapshots (filesystem-level rsync/ZFS) or S3 versioning when legacy driver enabled; Rekor v2 DB snapshots.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -80,6 +80,8 @@ Everything here is open‑source and versioned — when you check out a git ta
|
||||
- **29 – [Concelier CISA ICS Connector Operations](ops/concelier-icscisa-operations.md)**
|
||||
- **30 – [Concelier CERT-Bund Connector Operations](ops/concelier-certbund-operations.md)**
|
||||
- **31 – [Concelier MSRC Connector – AAD Onboarding](ops/concelier-msrc-operations.md)**
|
||||
- **32 – [Scanner Analyzer Bench Operations](ops/scanner-analyzers-operations.md)**
|
||||
- **33 – [Scanner Artifact Store Migration](ops/scanner-rustfs-migration.md)**
|
||||
|
||||
### Legal & licence
|
||||
- **32 – [Legal & Quota FAQ](29_LEGAL_FAQ_QUOTA.md)**
|
||||
|
||||
@@ -14,6 +14,8 @@
|
||||
> **Status update (2025-10-19):** `ValidateDpopProofHandler`, `AuthorityClientCertificateValidator`, and the supporting storage/audit plumbing now live in `src/StellaOps.Authority`. DPoP proofs populate `cnf.jkt`, mTLS bindings enforce certificate thumbprints via `cnf.x5t#S256`, and token documents persist the sender constraint metadata. In-memory nonce issuance is wired (Redis implementation to follow). Documentation and configuration references were updated (`docs/11_AUTHORITY.md`). Targeted unit/integration tests were added; running the broader test suite is currently blocked by pre-existing `StellaOps.Concelier.Storage.Mongo` build errors.
|
||||
>
|
||||
> **Status update (2025-10-20):** Redis-backed nonce configuration is exposed through `security.senderConstraints.dpop.nonce` with sample YAML (`etc/authority.yaml.sample`) and architecture docs refreshed. Operator guide now includes concrete Redis/required audiences snippet; nonce challenge regression remains covered by `ValidateDpopProof_IssuesNonceChallenge_WhenNonceMissing`.
|
||||
>
|
||||
> **Status update (2025-10-23):** mTLS enforcement now honours `security.senderConstraints.mtls.enforceForAudiences`, automatically rejecting non-mTLS clients targeting audiences such as `signer`. Certificate bindings validate thumbprint, issuer, subject, serial number, and SAN values, producing deterministic error codes for operators. Introspection responses include `cnf.x5t#S256`, and new unit tests cover audience enforcement, binding mismatches, and bootstrap storage. Docs/sample config updated accordingly.
|
||||
|
||||
## Design Summary
|
||||
- Extract the existing Scanner `DpopProofValidator` stack into a shared `StellaOps.Auth.Security` library used by Authority and resource servers.
|
||||
|
||||
155
docs/ops/scanner-analyzers-grafana-dashboard.json
Normal file
155
docs/ops/scanner-analyzers-grafana-dashboard.json
Normal file
@@ -0,0 +1,155 @@
|
||||
{
|
||||
"title": "StellaOps Scanner Analyzer Benchmarks",
|
||||
"uid": "scanner-analyzer-bench",
|
||||
"schemaVersion": 38,
|
||||
"version": 1,
|
||||
"editable": true,
|
||||
"timezone": "",
|
||||
"graphTooltip": 0,
|
||||
"time": {
|
||||
"from": "now-24h",
|
||||
"to": "now"
|
||||
},
|
||||
"templating": {
|
||||
"list": [
|
||||
{
|
||||
"name": "datasource",
|
||||
"type": "datasource",
|
||||
"query": "prometheus",
|
||||
"refresh": 1,
|
||||
"hide": 0,
|
||||
"current": {}
|
||||
}
|
||||
]
|
||||
},
|
||||
"annotations": {
|
||||
"list": []
|
||||
},
|
||||
"panels": [
|
||||
{
|
||||
"id": 1,
|
||||
"title": "Max Duration (ms)",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${datasource}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "ms",
|
||||
"displayName": "{{scenario}}"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "single",
|
||||
"sort": "none"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "scanner_analyzer_bench_max_ms",
|
||||
"legendFormat": "{{scenario}}",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"expr": "scanner_analyzer_bench_baseline_max_ms",
|
||||
"legendFormat": "{{scenario}} baseline",
|
||||
"refId": "B"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"title": "Regression Ratio vs Limit",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${datasource}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "percentunit",
|
||||
"displayName": "{{scenario}}",
|
||||
"min": 0,
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 20
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "none"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "(scanner_analyzer_bench_regression_ratio - 1) * 100",
|
||||
"legendFormat": "{{scenario}} regression %",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"expr": "(scanner_analyzer_bench_regression_limit - 1) * 100",
|
||||
"legendFormat": "{{scenario}} limit %",
|
||||
"refId": "B"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 3,
|
||||
"title": "Breached Scenarios",
|
||||
"type": "stat",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${datasource}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"displayName": "{{scenario}}",
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"colorMode": "value",
|
||||
"graphMode": "area",
|
||||
"justifyMode": "center",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"last"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "scanner_analyzer_bench_regression_breached",
|
||||
"legendFormat": "{{scenario}}",
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
48
docs/ops/scanner-analyzers-operations.md
Normal file
48
docs/ops/scanner-analyzers-operations.md
Normal file
@@ -0,0 +1,48 @@
|
||||
# Scanner Analyzer Benchmarks – Operations Guide
|
||||
|
||||
## Purpose
|
||||
Keep the language analyzer microbench under the < 5 s SBOM pledge. CI emits Prometheus metrics and JSON fixtures so trend dashboards and alerts stay in lockstep with the repository baseline.
|
||||
|
||||
> **Grafana note:** Import `docs/ops/scanner-analyzers-grafana-dashboard.json` into your Prometheus-backed Grafana stack to monitor `scanner_analyzer_bench_*` metrics and alert on regressions.
|
||||
|
||||
## Publishing workflow
|
||||
1. CI (or engineers running locally) execute:
|
||||
```bash
|
||||
dotnet run \
|
||||
--project bench/Scanner.Analyzers/StellaOps.Bench.ScannerAnalyzers/StellaOps.Bench.ScannerAnalyzers.csproj \
|
||||
-- \
|
||||
--repo-root . \
|
||||
--out bench/Scanner.Analyzers/baseline.csv \
|
||||
--json out/bench/scanner-analyzers/latest.json \
|
||||
--prom out/bench/scanner-analyzers/latest.prom \
|
||||
--commit "$(git rev-parse HEAD)" \
|
||||
--environment "${CI_ENVIRONMENT_NAME:-local}"
|
||||
```
|
||||
2. Publish the artefacts (`baseline.csv`, `latest.json`, `latest.prom`) to `bench-artifacts/<date>/`.
|
||||
3. Promtail (or the CI job) pushes `latest.prom` into Prometheus; JSON lands in long-term storage for workbook snapshots.
|
||||
4. The harness exits non-zero if:
|
||||
- `max_ms` for any scenario breaches its configured threshold; or
|
||||
- `max_ms` regresses ≥ 20 % versus `baseline.csv`.
|
||||
|
||||
## Grafana dashboard
|
||||
- Import `docs/ops/scanner-analyzers-grafana-dashboard.json`.
|
||||
- Point the template variable `datasource` to the Prometheus instance ingesting `scanner_analyzer_bench_*` metrics.
|
||||
- Panels:
|
||||
- **Max Duration (ms)** – compares live runs vs baseline.
|
||||
- **Regression Ratio vs Limit** – plots `(max / baseline_max - 1) * 100`.
|
||||
- **Breached Scenarios** – stat panel sourced from `scanner_analyzer_bench_regression_breached`.
|
||||
|
||||
## Alerting & on-call response
|
||||
- **Primary alert**: fire when `scanner_analyzer_bench_regression_ratio{scenario=~".+"} >= 1.20` for 2 consecutive samples (10 min default). Suggested PromQL:
|
||||
```
|
||||
max_over_time(scanner_analyzer_bench_regression_ratio[10m]) >= 1.20
|
||||
```
|
||||
- Suppress duplicates using the `scenario` label.
|
||||
- Pager payload should include `scenario`, `max_ms`, `baseline_max_ms`, and `commit`.
|
||||
- Immediate triage steps:
|
||||
1. Check `latest.json` artefact for the failing scenario – confirm commit and environment.
|
||||
2. Re-run the harness with `--captured-at` and `--baseline` pointing at the last known good CSV to verify determinism.
|
||||
3. If regression persists, open an incident ticket tagged `scanner-analyzer-perf` and page the owning language guild.
|
||||
4. Roll back the offending change or update the baseline after sign-off from the guild lead and Perf captain.
|
||||
|
||||
Document the outcome in `docs/12_PERFORMANCE_WORKBOOK.md` (section 8) so trendlines reflect any accepted regressions.
|
||||
88
docs/ops/scanner-rustfs-migration.md
Normal file
88
docs/ops/scanner-rustfs-migration.md
Normal file
@@ -0,0 +1,88 @@
|
||||
# Scanner Artifact Store Migration (MinIO → RustFS)
|
||||
|
||||
## Overview
|
||||
|
||||
Sprint 11 introduces **RustFS** as the default artifact store for the Scanner plane. Existing
|
||||
deployments running MinIO (or any S3-compatible backend) must migrate stored SBOM artefacts to RustFS
|
||||
before switching the Scanner hosts to `scanner.artifactStore.driver = "rustfs"`.
|
||||
|
||||
This runbook covers the recommended migration workflow and validation steps.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- RustFS service deployed and reachable from the Scanner control plane (`http(s)://rustfs:8080`).
|
||||
- Existing MinIO/S3 credentials with read access to the current bucket.
|
||||
- CLI environment with the StellaOps source tree (for the migration tool) and `dotnet 10` SDK.
|
||||
- Maintenance window sized to copy all artefacts (migration is read-only on the source bucket).
|
||||
|
||||
## 1. Snapshot source bucket (optional but recommended)
|
||||
|
||||
If the MinIO deployment offers versioning or snapshots, take one before migrating. For non-versioned
|
||||
deployments, capture an external backup (e.g., `mc mirror` to offline storage).
|
||||
|
||||
## 2. Dry-run the migrator
|
||||
|
||||
```
|
||||
dotnet run --project tools/RustFsMigrator -- \
|
||||
--s3-bucket scanner-artifacts \
|
||||
--s3-endpoint http://stellaops-minio:9000 \
|
||||
--s3-access-key stellaops \
|
||||
--s3-secret-key dev-minio-secret \
|
||||
--rustfs-endpoint http://stellaops-rustfs:8080 \
|
||||
--rustfs-bucket scanner-artifacts \
|
||||
--prefix scanner/ \
|
||||
--dry-run
|
||||
```
|
||||
|
||||
The dry-run enumerates keys and reports the object count without writing to RustFS. Use this to
|
||||
estimate migration time.
|
||||
|
||||
## 3. Execute migration
|
||||
|
||||
Remove the `--dry-run` flag to copy data. Optional flags:
|
||||
|
||||
- `--immutable` – mark all migrated objects as immutable (`X-RustFS-Immutable`).
|
||||
- `--retain-days 365` – request retention (in days) via `X-RustFS-Retain-Seconds`.
|
||||
- `--rustfs-api-key-header` / `--rustfs-api-key` – provide auth headers when RustFS is protected.
|
||||
|
||||
The tool streams each object from S3 and performs an idempotent `PUT` to RustFS preserving the key
|
||||
structure (e.g., `scanner/layers/<sha256>/sbom.cdx.json.zst`).
|
||||
|
||||
## 4. Verify sample objects
|
||||
|
||||
Pick a handful of SBOM digests and confirm:
|
||||
|
||||
1. `GET /api/v1/buckets/<bucket>/objects/<key>` returns the expected payload (size + SHA-256).
|
||||
2. Scanner WebService configured with `scanner.artifactStore.driver = "rustfs"` can fetch the same
|
||||
artefacts (Smoke test: `GET /api/v1/scanner/sboms/<digest>?format=cdx-json`).
|
||||
|
||||
## 5. Switch Scanner hosts
|
||||
|
||||
Update configuration (Helm/Compose/environment) to set:
|
||||
|
||||
```
|
||||
scanner:
|
||||
artifactStore:
|
||||
driver: rustfs
|
||||
endpoint: http://stellaops-rustfs:8080
|
||||
bucket: scanner-artifacts
|
||||
timeoutSeconds: 30
|
||||
```
|
||||
|
||||
Redeploy Scanner WebService and Worker. Monitor logs for `RustFS` upload/download messages and
|
||||
Prometheus scrape (`rustfs_requests_total`).
|
||||
|
||||
## 6. Cleanup legacy MinIO (optional)
|
||||
|
||||
After a complete migration and validation period, decommission the MinIO bucket or repurpose it for
|
||||
other components (Concelier still supports S3). Ensure backups reference RustFS snapshots going
|
||||
forward.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- **Uploads fail (HTTP 4xx/5xx):** Check RustFS logs and confirm API key headers. Re-run the migrator
|
||||
for the affected keys.
|
||||
- **Missing objects post-cutover:** Re-run the migrator with the specific `--prefix`. The tool is
|
||||
idempotent and safely overwrites existing objects.
|
||||
- **Performance tuning:** Run multiple instances of the migrator with disjoint prefixes if needed; the
|
||||
RustFS API is stateless and supports parallel PUTs.
|
||||
314
docs/schemas/policy-preview-sample@1.json
Normal file
314
docs/schemas/policy-preview-sample@1.json
Normal file
@@ -0,0 +1,314 @@
|
||||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"$id": "https://schemas.stella-ops.org/policy/policy-preview-sample@1.json",
|
||||
"title": "Policy Preview Sample",
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"previewRequest",
|
||||
"previewResponse"
|
||||
],
|
||||
"properties": {
|
||||
"previewRequest": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"imageDigest",
|
||||
"findings"
|
||||
],
|
||||
"properties": {
|
||||
"imageDigest": {
|
||||
"type": "string",
|
||||
"pattern": "^sha256:[0-9a-f]{64}$"
|
||||
},
|
||||
"findings": {
|
||||
"type": "array",
|
||||
"minItems": 1,
|
||||
"items": {
|
||||
"$ref": "#/$defs/finding"
|
||||
}
|
||||
},
|
||||
"baseline": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/$defs/baselineVerdict"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"previewResponse": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"success",
|
||||
"policyDigest",
|
||||
"revisionId",
|
||||
"changed",
|
||||
"diffs",
|
||||
"issues"
|
||||
],
|
||||
"properties": {
|
||||
"success": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"policyDigest": {
|
||||
"type": "string",
|
||||
"pattern": "^[0-9a-f]{64}$"
|
||||
},
|
||||
"revisionId": {
|
||||
"type": "string"
|
||||
},
|
||||
"changed": {
|
||||
"type": "integer",
|
||||
"minimum": 0
|
||||
},
|
||||
"diffs": {
|
||||
"type": "array",
|
||||
"minItems": 1,
|
||||
"items": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"findingId",
|
||||
"baseline",
|
||||
"projected",
|
||||
"changed"
|
||||
],
|
||||
"properties": {
|
||||
"findingId": {
|
||||
"type": "string"
|
||||
},
|
||||
"baseline": {
|
||||
"$ref": "#/$defs/baselineVerdict"
|
||||
},
|
||||
"projected": {
|
||||
"$ref": "#/$defs/projectedVerdict"
|
||||
},
|
||||
"changed": {
|
||||
"type": "boolean"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"issues": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"code",
|
||||
"message",
|
||||
"severity",
|
||||
"path"
|
||||
],
|
||||
"properties": {
|
||||
"code": {
|
||||
"type": "string"
|
||||
},
|
||||
"message": {
|
||||
"type": "string"
|
||||
},
|
||||
"severity": {
|
||||
"type": "string"
|
||||
},
|
||||
"path": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"$defs": {
|
||||
"finding": {
|
||||
"type": "object",
|
||||
"required": [
|
||||
"id",
|
||||
"severity",
|
||||
"source"
|
||||
],
|
||||
"properties": {
|
||||
"id": {
|
||||
"type": "string"
|
||||
},
|
||||
"severity": {
|
||||
"type": "string"
|
||||
},
|
||||
"source": {
|
||||
"type": "string"
|
||||
},
|
||||
"tags": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
},
|
||||
"additionalProperties": true
|
||||
},
|
||||
"inputs": {
|
||||
"type": "object",
|
||||
"minProperties": 1,
|
||||
"propertyNames": {
|
||||
"type": "string",
|
||||
"maxLength": 64
|
||||
},
|
||||
"additionalProperties": {
|
||||
"type": "number"
|
||||
}
|
||||
},
|
||||
"baselineVerdict": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"findingId",
|
||||
"status",
|
||||
"configVersion",
|
||||
"score"
|
||||
],
|
||||
"properties": {
|
||||
"findingId": {
|
||||
"type": "string"
|
||||
},
|
||||
"status": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"Pass",
|
||||
"Blocked",
|
||||
"Warned",
|
||||
"Ignored",
|
||||
"Deferred",
|
||||
"Escalated",
|
||||
"RequiresVex"
|
||||
]
|
||||
},
|
||||
"ruleName": {
|
||||
"type": [
|
||||
"string",
|
||||
"null"
|
||||
]
|
||||
},
|
||||
"ruleAction": {
|
||||
"type": [
|
||||
"string",
|
||||
"null"
|
||||
]
|
||||
},
|
||||
"notes": {
|
||||
"type": [
|
||||
"string",
|
||||
"null"
|
||||
]
|
||||
},
|
||||
"score": {
|
||||
"type": "number"
|
||||
},
|
||||
"configVersion": {
|
||||
"type": "string"
|
||||
},
|
||||
"inputs": {
|
||||
"$ref": "#/$defs/inputs"
|
||||
},
|
||||
"quietedBy": {
|
||||
"type": [
|
||||
"string",
|
||||
"null"
|
||||
]
|
||||
},
|
||||
"quiet": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"unknownConfidence": {
|
||||
"type": "number",
|
||||
"minimum": 0
|
||||
},
|
||||
"confidenceBand": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"low",
|
||||
"medium",
|
||||
"high",
|
||||
"unspecified"
|
||||
]
|
||||
},
|
||||
"unknownAgeDays": {
|
||||
"type": "number",
|
||||
"minimum": 0
|
||||
},
|
||||
"sourceTrust": {
|
||||
"type": "string"
|
||||
},
|
||||
"reachability": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"unknown",
|
||||
"runtime",
|
||||
"entrypoint",
|
||||
"direct",
|
||||
"indirect",
|
||||
"unreachable"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"projectedVerdict": {
|
||||
"allOf": [
|
||||
{
|
||||
"$ref": "#/$defs/baselineVerdict"
|
||||
},
|
||||
{
|
||||
"type": "object",
|
||||
"required": [
|
||||
"ruleName",
|
||||
"ruleAction",
|
||||
"unknownConfidence",
|
||||
"confidenceBand",
|
||||
"unknownAgeDays",
|
||||
"sourceTrust",
|
||||
"reachability"
|
||||
],
|
||||
"properties": {
|
||||
"ruleName": {
|
||||
"type": "string"
|
||||
},
|
||||
"ruleAction": {
|
||||
"type": "string"
|
||||
},
|
||||
"unknownConfidence": {
|
||||
"type": "number",
|
||||
"minimum": 0
|
||||
},
|
||||
"confidenceBand": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"low",
|
||||
"medium",
|
||||
"high",
|
||||
"unspecified"
|
||||
]
|
||||
},
|
||||
"unknownAgeDays": {
|
||||
"type": "number",
|
||||
"minimum": 0
|
||||
},
|
||||
"sourceTrust": {
|
||||
"type": "string"
|
||||
},
|
||||
"reachability": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"unknown",
|
||||
"runtime",
|
||||
"entrypoint",
|
||||
"direct",
|
||||
"indirect",
|
||||
"unreachable"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
396
docs/schemas/policy-report-sample@1.json
Normal file
396
docs/schemas/policy-report-sample@1.json
Normal file
@@ -0,0 +1,396 @@
|
||||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"$id": "https://schemas.stella-ops.org/policy/policy-report-sample@1.json",
|
||||
"title": "Policy Report Sample",
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"reportRequest",
|
||||
"reportResponse"
|
||||
],
|
||||
"properties": {
|
||||
"reportRequest": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"imageDigest",
|
||||
"findings"
|
||||
],
|
||||
"properties": {
|
||||
"imageDigest": {
|
||||
"type": "string",
|
||||
"pattern": "^sha256:[0-9a-f]{64}$"
|
||||
},
|
||||
"findings": {
|
||||
"type": "array",
|
||||
"minItems": 1,
|
||||
"items": {
|
||||
"$ref": "#/$defs/finding"
|
||||
}
|
||||
},
|
||||
"baseline": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/$defs/baselineVerdict"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"reportResponse": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"report",
|
||||
"dsse"
|
||||
],
|
||||
"properties": {
|
||||
"report": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"reportId",
|
||||
"imageDigest",
|
||||
"generatedAt",
|
||||
"verdict",
|
||||
"policy",
|
||||
"summary",
|
||||
"verdicts",
|
||||
"issues"
|
||||
],
|
||||
"properties": {
|
||||
"reportId": {
|
||||
"type": "string"
|
||||
},
|
||||
"imageDigest": {
|
||||
"type": "string",
|
||||
"pattern": "^sha256:[0-9a-f]{64}$"
|
||||
},
|
||||
"generatedAt": {
|
||||
"type": "string",
|
||||
"format": "date-time"
|
||||
},
|
||||
"verdict": {
|
||||
"type": "string"
|
||||
},
|
||||
"policy": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"revisionId",
|
||||
"digest"
|
||||
],
|
||||
"properties": {
|
||||
"revisionId": {
|
||||
"type": "string"
|
||||
},
|
||||
"digest": {
|
||||
"type": "string",
|
||||
"pattern": "^[0-9a-f]{64}$"
|
||||
}
|
||||
}
|
||||
},
|
||||
"summary": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"total",
|
||||
"blocked",
|
||||
"warned",
|
||||
"ignored",
|
||||
"quieted"
|
||||
],
|
||||
"properties": {
|
||||
"total": {
|
||||
"type": "integer",
|
||||
"minimum": 0
|
||||
},
|
||||
"blocked": {
|
||||
"type": "integer",
|
||||
"minimum": 0
|
||||
},
|
||||
"warned": {
|
||||
"type": "integer",
|
||||
"minimum": 0
|
||||
},
|
||||
"ignored": {
|
||||
"type": "integer",
|
||||
"minimum": 0
|
||||
},
|
||||
"quieted": {
|
||||
"type": "integer",
|
||||
"minimum": 0
|
||||
}
|
||||
}
|
||||
},
|
||||
"verdicts": {
|
||||
"type": "array",
|
||||
"minItems": 1,
|
||||
"items": {
|
||||
"$ref": "#/$defs/projectedVerdict"
|
||||
}
|
||||
},
|
||||
"issues": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"code",
|
||||
"message",
|
||||
"severity",
|
||||
"path"
|
||||
],
|
||||
"properties": {
|
||||
"code": {
|
||||
"type": "string"
|
||||
},
|
||||
"message": {
|
||||
"type": "string"
|
||||
},
|
||||
"severity": {
|
||||
"type": "string"
|
||||
},
|
||||
"path": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"dsse": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"payloadType",
|
||||
"payload",
|
||||
"signatures"
|
||||
],
|
||||
"properties": {
|
||||
"payloadType": {
|
||||
"type": "string"
|
||||
},
|
||||
"payload": {
|
||||
"type": "string"
|
||||
},
|
||||
"signatures": {
|
||||
"type": "array",
|
||||
"minItems": 1,
|
||||
"items": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"keyId",
|
||||
"algorithm",
|
||||
"signature"
|
||||
],
|
||||
"properties": {
|
||||
"keyId": {
|
||||
"type": "string"
|
||||
},
|
||||
"algorithm": {
|
||||
"type": "string"
|
||||
},
|
||||
"signature": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"$defs": {
|
||||
"finding": {
|
||||
"type": "object",
|
||||
"required": [
|
||||
"id",
|
||||
"severity",
|
||||
"source"
|
||||
],
|
||||
"properties": {
|
||||
"id": {
|
||||
"type": "string"
|
||||
},
|
||||
"severity": {
|
||||
"type": "string"
|
||||
},
|
||||
"source": {
|
||||
"type": "string"
|
||||
},
|
||||
"tags": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
},
|
||||
"additionalProperties": true
|
||||
},
|
||||
"inputs": {
|
||||
"type": "object",
|
||||
"minProperties": 1,
|
||||
"propertyNames": {
|
||||
"type": "string",
|
||||
"maxLength": 64
|
||||
},
|
||||
"additionalProperties": {
|
||||
"type": "number"
|
||||
}
|
||||
},
|
||||
"baselineVerdict": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"findingId",
|
||||
"status",
|
||||
"configVersion",
|
||||
"score"
|
||||
],
|
||||
"properties": {
|
||||
"findingId": {
|
||||
"type": "string"
|
||||
},
|
||||
"status": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"Pass",
|
||||
"Blocked",
|
||||
"Warned",
|
||||
"Ignored",
|
||||
"Deferred",
|
||||
"Escalated",
|
||||
"RequiresVex"
|
||||
]
|
||||
},
|
||||
"ruleName": {
|
||||
"type": [
|
||||
"string",
|
||||
"null"
|
||||
]
|
||||
},
|
||||
"ruleAction": {
|
||||
"type": [
|
||||
"string",
|
||||
"null"
|
||||
]
|
||||
},
|
||||
"notes": {
|
||||
"type": [
|
||||
"string",
|
||||
"null"
|
||||
]
|
||||
},
|
||||
"score": {
|
||||
"type": "number"
|
||||
},
|
||||
"configVersion": {
|
||||
"type": "string"
|
||||
},
|
||||
"inputs": {
|
||||
"$ref": "#/$defs/inputs"
|
||||
},
|
||||
"quietedBy": {
|
||||
"type": [
|
||||
"string",
|
||||
"null"
|
||||
]
|
||||
},
|
||||
"quiet": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"unknownConfidence": {
|
||||
"type": "number",
|
||||
"minimum": 0
|
||||
},
|
||||
"confidenceBand": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"low",
|
||||
"medium",
|
||||
"high",
|
||||
"unspecified"
|
||||
]
|
||||
},
|
||||
"unknownAgeDays": {
|
||||
"type": "number",
|
||||
"minimum": 0
|
||||
},
|
||||
"sourceTrust": {
|
||||
"type": "string"
|
||||
},
|
||||
"reachability": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"unknown",
|
||||
"runtime",
|
||||
"entrypoint",
|
||||
"direct",
|
||||
"indirect",
|
||||
"unreachable"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"projectedVerdict": {
|
||||
"allOf": [
|
||||
{
|
||||
"$ref": "#/$defs/baselineVerdict"
|
||||
},
|
||||
{
|
||||
"type": "object",
|
||||
"required": [
|
||||
"ruleName",
|
||||
"ruleAction",
|
||||
"unknownConfidence",
|
||||
"confidenceBand",
|
||||
"unknownAgeDays",
|
||||
"sourceTrust",
|
||||
"reachability"
|
||||
],
|
||||
"properties": {
|
||||
"ruleName": {
|
||||
"type": "string"
|
||||
},
|
||||
"ruleAction": {
|
||||
"type": "string"
|
||||
},
|
||||
"unknownConfidence": {
|
||||
"type": "number",
|
||||
"minimum": 0
|
||||
},
|
||||
"confidenceBand": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"low",
|
||||
"medium",
|
||||
"high",
|
||||
"unspecified"
|
||||
]
|
||||
},
|
||||
"unknownAgeDays": {
|
||||
"type": "number",
|
||||
"minimum": 0
|
||||
},
|
||||
"sourceTrust": {
|
||||
"type": "string"
|
||||
},
|
||||
"reachability": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"unknown",
|
||||
"runtime",
|
||||
"entrypoint",
|
||||
"direct",
|
||||
"indirect",
|
||||
"unreachable"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user