Files

Vladimir Moushkov f4d7a15a00

Docs CI / lint-and-preview (push) Has been cancelled

Details

feat: Add RustFS artifact object store and migration tool

- Implemented RustFsArtifactObjectStore for managing artifacts in RustFS.
- Added unit tests for RustFsArtifactObjectStore functionality.
- Created a RustFS migrator tool to transfer objects from S3 to RustFS.
- Introduced policy preview and report models for API integration.
- Added fixtures and tests for policy preview and report functionality.
- Included necessary metadata and scripts for cache_pkg package.

2025-10-23 18:53:18 +03:00

3.4 KiB

Raw Blame History

Scanner Artifact Store Migration (MinIO → RustFS)

Overview

Sprint 11 introduces RustFS as the default artifact store for the Scanner plane. Existing deployments running MinIO (or any S3-compatible backend) must migrate stored SBOM artefacts to RustFS before switching the Scanner hosts to scanner.artifactStore.driver = "rustfs".

This runbook covers the recommended migration workflow and validation steps.

Prerequisites

RustFS service deployed and reachable from the Scanner control plane (http(s)://rustfs:8080).
Existing MinIO/S3 credentials with read access to the current bucket.
CLI environment with the StellaOps source tree (for the migration tool) and dotnet 10 SDK.
Maintenance window sized to copy all artefacts (migration is read-only on the source bucket).

1. Snapshot source bucket (optional but recommended)

If the MinIO deployment offers versioning or snapshots, take one before migrating. For non-versioned deployments, capture an external backup (e.g., mc mirror to offline storage).

2. Dry-run the migrator

dotnet run --project tools/RustFsMigrator -- \
  --s3-bucket scanner-artifacts \
  --s3-endpoint http://stellaops-minio:9000 \
  --s3-access-key stellaops \
  --s3-secret-key dev-minio-secret \
  --rustfs-endpoint http://stellaops-rustfs:8080 \
  --rustfs-bucket scanner-artifacts \
  --prefix scanner/ \
  --dry-run

The dry-run enumerates keys and reports the object count without writing to RustFS. Use this to estimate migration time.

3. Execute migration

Remove the --dry-run flag to copy data. Optional flags:

--immutable – mark all migrated objects as immutable (X-RustFS-Immutable).
--retain-days 365 – request retention (in days) via X-RustFS-Retain-Seconds.
--rustfs-api-key-header / --rustfs-api-key – provide auth headers when RustFS is protected.

The tool streams each object from S3 and performs an idempotent PUT to RustFS preserving the key structure (e.g., scanner/layers/<sha256>/sbom.cdx.json.zst).

4. Verify sample objects

Pick a handful of SBOM digests and confirm:

GET /api/v1/buckets/<bucket>/objects/<key> returns the expected payload (size + SHA-256).
Scanner WebService configured with scanner.artifactStore.driver = "rustfs" can fetch the same artefacts (Smoke test: GET /api/v1/scanner/sboms/<digest>?format=cdx-json).

5. Switch Scanner hosts

Update configuration (Helm/Compose/environment) to set:

scanner:
  artifactStore:
    driver: rustfs
    endpoint: http://stellaops-rustfs:8080
    bucket: scanner-artifacts
    timeoutSeconds: 30

Redeploy Scanner WebService and Worker. Monitor logs for RustFS upload/download messages and Prometheus scrape (rustfs_requests_total).

6. Cleanup legacy MinIO (optional)

After a complete migration and validation period, decommission the MinIO bucket or repurpose it for other components (Concelier still supports S3). Ensure backups reference RustFS snapshots going forward.

Troubleshooting

Uploads fail (HTTP 4xx/5xx): Check RustFS logs and confirm API key headers. Re-run the migrator for the affected keys.
Missing objects post-cutover: Re-run the migrator with the specific --prefix. The tool is idempotent and safely overwrites existing objects.
Performance tuning: Run multiple instances of the migrator with disjoint prefixes if needed; the RustFS API is stateless and supports parallel PUTs.

3.4 KiB Raw Blame History Unescape Escape