feat: Add RustFS artifact object store and migration tool
Some checks failed
Docs CI / lint-and-preview (push) Has been cancelled

- Implemented RustFsArtifactObjectStore for managing artifacts in RustFS.
- Added unit tests for RustFsArtifactObjectStore functionality.
- Created a RustFS migrator tool to transfer objects from S3 to RustFS.
- Introduced policy preview and report models for API integration.
- Added fixtures and tests for policy preview and report functionality.
- Included necessary metadata and scripts for cache_pkg package.
This commit is contained in:
Vladimir Moushkov
2025-10-23 18:53:18 +03:00
parent aaa5fbfb78
commit f4d7a15a00
117 changed files with 4849 additions and 725 deletions

View File

@@ -1,6 +1,6 @@
# component_architecture_devops.md — **StellaOps Release & Operations** (2025Q4)
> **Scope.** Implementationready blueprint for **how StellaOps is built, versioned, signed, distributed, upgraded, licensed (PoE)**, and operated in customer environments (online and airgapped). Covers reproducible builds, supplychain attestations, registries, offline kits, migration/rollback, artifact lifecycle (MinIO/Mongo), monitoring SLOs, and customer activation.
> **Scope.** Implementationready blueprint for **how StellaOps is built, versioned, signed, distributed, upgraded, licensed (PoE)**, and operated in customer environments (online and airgapped). Covers reproducible builds, supplychain attestations, registries, offline kits, migration/rollback, artifact lifecycle (RustFS default + Mongo, S3 fallback), monitoring SLOs, and customer activation.
---
@@ -257,12 +257,12 @@ Signer validates **scanner** images cosign identity + calendar tag for **rele
---
## 7) Artifact lifecycle & storage (MinIO/Mongo)
## 7) Artifact lifecycle & storage (RustFS/Mongo)
### 7.1 Buckets & prefixes (MinIO)
### 7.1 Buckets & prefixes (RustFS)
```
s3://stellaops/
rustfs://stellaops/
scanner/
layers/<sha256>/sbom.cdx.json.zst
images/<imgDigest>/inventory.cdx.pb
@@ -283,7 +283,7 @@ s3://stellaops/
* **`short`**: working artifacts (diffs, queues) — TTL 714 days.
* **`default`**: SBOMs & indexes — TTL 90180 days (configurable).
* **`compliance`**: signed reports & attested exports — **Object Lock** (governance/compliance) 17 years.
* **`compliance`**: signed reports & attested exports — retention enforced via RustFS hold or S3 Object Lock (governance/compliance) 17 years.
### 7.3 Artifact Lifecycle Controller (ALC)
@@ -292,6 +292,9 @@ s3://stellaops/
* Artifacts referenced by **reports** or **tickets** are pinned.
* ILM actions logged; UI shows perclass usage & upcoming purges.
> **Migration note.** Follow `docs/ops/scanner-rustfs-migration.md` when transitioning existing
> MinIO buckets to RustFS. The provided migrator is idempotent and safe to rerun per prefix.
### 7.4 Mongo retention
* **Scanner**: `runtime.events` use TTL (e.g., 3090 days); **catalog** permanent.
@@ -313,7 +316,7 @@ s3://stellaops/
* **Golden signals**:
* **Latency**: token issuance, sign→attest roundtrip, scan enqueue→emit, export build.
* **Saturation**: queue depth, Mongo write IOPS, MinIO net throughput.
* **Saturation**: queue depth, Mongo write IOPS, RustFS throughput / queue depth (or S3 metrics when in fallback mode).
* **Traffic**: scans/min, attestations/min, webhook admits/min.
* **Errors**: 5xx rates, cosign verification failures, Rekor timeouts.
@@ -460,7 +463,7 @@ services:
* `attestor.submit_latency_seconds{quantile=0.95}` < 0.3.
* `scanner.scan_latency_seconds{quantile=0.95}` < target per image size.
* `concelier.export.duration_seconds` stable; `excititor.consensus.conflicts_total` not exploding after policy changes.
* MinIO `s3_requests_errors_total` near zero; Mongo `opcounters` hit expected baseline.
* RustFS request error rate near zero (or `s3_requests_errors_total` when operating against S3); Mongo `opcounters` hit expected baseline.
### Appendix B — Upgrade safety checklist