feat: Add RustFS artifact object store and migration tool

- Implemented RustFsArtifactObjectStore for managing artifacts in RustFS.
- Added unit tests for RustFsArtifactObjectStore functionality.
- Created a RustFS migrator tool to transfer objects from S3 to RustFS.
- Introduced policy preview and report models for API integration.
- Added fixtures and tests for policy preview and report functionality.
- Included necessary metadata and scripts for cache_pkg package.
This commit is contained in:
master
2025-10-23 18:53:18 +03:00
parent 5cb3144e5e
commit 70d7fb529e
117 changed files with 4849 additions and 725 deletions

View File

@@ -49,7 +49,7 @@
* **Fulcio** (Sigstore CA) — issues shortlived signing certs (keyless).
* **Rekor v2** (tilebacked transparency log).
* **MinIO** — S3compatible object store with lifecycle & Object Lock.
* **RustFS** — offline-first object store with deterministic REST API (S3/MinIO fallback available for legacy installs).
* **MongoDB** — catalog, advisories, VEX, scheduler, notify.
* **Queue** — Redis Streams / NATS / RabbitMQ (pluggable).
* **OCI Registry** — must support **Referrers API** (discover SBOMs/signatures).
@@ -81,7 +81,7 @@ flowchart LR
ATT[Attestor\n(Rekor v2 submit/verify)]
UI[Web UI (Angular)]
Z[Zastava\n(Runtime Inspector/Enforcer)]
MIN[(MinIO S3)]
RFS[(RustFS object store)]
MGO[(MongoDB)]
QUE[(Queue/Streams)]
end
@@ -94,7 +94,7 @@ flowchart LR
CLI -->|scan/build| SW
SW -->|jobs| QUE
QUE --> WK
WK --> MIN
WK --> RFS
SW --> MGO
CONC --> MGO
EXC --> MGO
@@ -225,13 +225,13 @@ LS --> IA: PoE (mTLS client cert or JWT with cnf=K_inst), CRL/OCSP/introspect
---
## 6) Storage & catalogs (MinIO/Mongo)
## 6) Storage & catalogs (RustFS/Mongo)
**RustFS layout (default)**
**MinIO layout**
```
s3://stellaops/
layers/<sha256>/sbom.cdx.json.zst
```
rustfs://stellaops/
layers/<sha256>/sbom.cdx.json.zst
layers/<sha256>/sbom.spdx.json.zst
images/<imgDigest>/inventory.cdx.pb
images/<imgDigest>/usage.cdx.pb
@@ -248,7 +248,7 @@ s3://stellaops/
**Retention**
* MinIO **ILM** for coarse TTL; Scanner.WebService GC decrements `refCount` and deletes unreferenced metadata; **Object Lock** for immutable classes (auditable artifacts).
* RustFS applies retention via `X-RustFS-Retain-Seconds`; Scanner.WebService GC decrements `refCount` and deletes unreferenced metadata; S3/MinIO fallback retains native Object Lock when enabled.
---
@@ -395,7 +395,7 @@ services:
ui: { image: stellaops/ui, depends_on: [scanner-web, concelier, excititor, scheduler-web, notify-web] }
```
* **Backups:** Mongo dumps; MinIO versioned buckets & replication; Rekor v2 DB snapshots; JWKS/Fulcio/KMS key rotation.
* **Backups:** Mongo dumps; RustFS snapshots (or S3 versioning when fallback driver is used); Rekor v2 DB snapshots; JWKS/Fulcio/KMS key rotation.
* **Ops runbooks:** Scheduler catchup after Concelier/Excititor recovery; connector key rotation (Slack/Teams/SMTP).
* **SLOs & alerts:** lag between Concelier/Excititor export and first rescan verdict; delivery failure rates by channel.
@@ -408,7 +408,7 @@ services:
* **Notify metrics:** `notify.sent_total{channel}`, `notify.dropped_total{reason}`, `notify.digest_coalesced_total`, `notify.latency_ms`.
* **Tracing:** perstage spans; correlation IDs across Scanner→Signer→Attestor and Concelier/Excititor→Scheduler→Scanner→Notify.
* **Audit logs:** every signing records `license_id`, `image_digest`, `policy_digest`, and Rekor UUID; Scheduler records who scheduled what; Notify records where, when, and why messages were sent or deduped.
* **Compliance:** MinIO **Object Lock** for immutable artifacts; reproducible outputs via policy digest + SBOM digest in predicate.
* **Compliance:** RustFS retention headers (or MinIO Object Lock when operating in S3 mode) keep immutable artifacts tamperresistant; reproducible outputs via policy digest + SBOM digest in predicate.
---