feat: Add RustFS artifact object store and migration tool
- Implemented RustFsArtifactObjectStore for managing artifacts in RustFS. - Added unit tests for RustFsArtifactObjectStore functionality. - Created a RustFS migrator tool to transfer objects from S3 to RustFS. - Introduced policy preview and report models for API integration. - Added fixtures and tests for policy preview and report functionality. - Included necessary metadata and scripts for cache_pkg package.
This commit is contained in:
		
							
								
								
									
										88
									
								
								docs/ops/scanner-rustfs-migration.md
									
									
									
									
									
										Normal file
									
								
							
							
						
						
									
										88
									
								
								docs/ops/scanner-rustfs-migration.md
									
									
									
									
									
										Normal file
									
								
							@@ -0,0 +1,88 @@
 | 
			
		||||
# Scanner Artifact Store Migration (MinIO → RustFS)
 | 
			
		||||
 | 
			
		||||
## Overview
 | 
			
		||||
 | 
			
		||||
Sprint 11 introduces **RustFS** as the default artifact store for the Scanner plane. Existing
 | 
			
		||||
deployments running MinIO (or any S3-compatible backend) must migrate stored SBOM artefacts to RustFS
 | 
			
		||||
before switching the Scanner hosts to `scanner.artifactStore.driver = "rustfs"`.
 | 
			
		||||
 | 
			
		||||
This runbook covers the recommended migration workflow and validation steps.
 | 
			
		||||
 | 
			
		||||
## Prerequisites
 | 
			
		||||
 | 
			
		||||
- RustFS service deployed and reachable from the Scanner control plane (`http(s)://rustfs:8080`).
 | 
			
		||||
- Existing MinIO/S3 credentials with read access to the current bucket.
 | 
			
		||||
- CLI environment with the StellaOps source tree (for the migration tool) and `dotnet 10` SDK.
 | 
			
		||||
- Maintenance window sized to copy all artefacts (migration is read-only on the source bucket).
 | 
			
		||||
 | 
			
		||||
## 1. Snapshot source bucket (optional but recommended)
 | 
			
		||||
 | 
			
		||||
If the MinIO deployment offers versioning or snapshots, take one before migrating. For non-versioned
 | 
			
		||||
deployments, capture an external backup (e.g., `mc mirror` to offline storage).
 | 
			
		||||
 | 
			
		||||
## 2. Dry-run the migrator
 | 
			
		||||
 | 
			
		||||
```
 | 
			
		||||
dotnet run --project tools/RustFsMigrator -- \
 | 
			
		||||
  --s3-bucket scanner-artifacts \
 | 
			
		||||
  --s3-endpoint http://stellaops-minio:9000 \
 | 
			
		||||
  --s3-access-key stellaops \
 | 
			
		||||
  --s3-secret-key dev-minio-secret \
 | 
			
		||||
  --rustfs-endpoint http://stellaops-rustfs:8080 \
 | 
			
		||||
  --rustfs-bucket scanner-artifacts \
 | 
			
		||||
  --prefix scanner/ \
 | 
			
		||||
  --dry-run
 | 
			
		||||
```
 | 
			
		||||
 | 
			
		||||
The dry-run enumerates keys and reports the object count without writing to RustFS. Use this to
 | 
			
		||||
estimate migration time.
 | 
			
		||||
 | 
			
		||||
## 3. Execute migration
 | 
			
		||||
 | 
			
		||||
Remove the `--dry-run` flag to copy data. Optional flags:
 | 
			
		||||
 | 
			
		||||
- `--immutable` – mark all migrated objects as immutable (`X-RustFS-Immutable`).
 | 
			
		||||
- `--retain-days 365` – request retention (in days) via `X-RustFS-Retain-Seconds`.
 | 
			
		||||
- `--rustfs-api-key-header` / `--rustfs-api-key` – provide auth headers when RustFS is protected.
 | 
			
		||||
 | 
			
		||||
The tool streams each object from S3 and performs an idempotent `PUT` to RustFS preserving the key
 | 
			
		||||
structure (e.g., `scanner/layers/<sha256>/sbom.cdx.json.zst`).
 | 
			
		||||
 | 
			
		||||
## 4. Verify sample objects
 | 
			
		||||
 | 
			
		||||
Pick a handful of SBOM digests and confirm:
 | 
			
		||||
 | 
			
		||||
1. `GET /api/v1/buckets/<bucket>/objects/<key>` returns the expected payload (size + SHA-256).
 | 
			
		||||
2. Scanner WebService configured with `scanner.artifactStore.driver = "rustfs"` can fetch the same
 | 
			
		||||
   artefacts (Smoke test: `GET /api/v1/scanner/sboms/<digest>?format=cdx-json`).
 | 
			
		||||
 | 
			
		||||
## 5. Switch Scanner hosts
 | 
			
		||||
 | 
			
		||||
Update configuration (Helm/Compose/environment) to set:
 | 
			
		||||
 | 
			
		||||
```
 | 
			
		||||
scanner:
 | 
			
		||||
  artifactStore:
 | 
			
		||||
    driver: rustfs
 | 
			
		||||
    endpoint: http://stellaops-rustfs:8080
 | 
			
		||||
    bucket: scanner-artifacts
 | 
			
		||||
    timeoutSeconds: 30
 | 
			
		||||
```
 | 
			
		||||
 | 
			
		||||
Redeploy Scanner WebService and Worker. Monitor logs for `RustFS` upload/download messages and
 | 
			
		||||
Prometheus scrape (`rustfs_requests_total`).
 | 
			
		||||
 | 
			
		||||
## 6. Cleanup legacy MinIO (optional)
 | 
			
		||||
 | 
			
		||||
After a complete migration and validation period, decommission the MinIO bucket or repurpose it for
 | 
			
		||||
other components (Concelier still supports S3). Ensure backups reference RustFS snapshots going
 | 
			
		||||
forward.
 | 
			
		||||
 | 
			
		||||
## Troubleshooting
 | 
			
		||||
 | 
			
		||||
- **Uploads fail (HTTP 4xx/5xx):** Check RustFS logs and confirm API key headers. Re-run the migrator
 | 
			
		||||
  for the affected keys.
 | 
			
		||||
- **Missing objects post-cutover:** Re-run the migrator with the specific `--prefix`. The tool is
 | 
			
		||||
  idempotent and safely overwrites existing objects.
 | 
			
		||||
- **Performance tuning:** Run multiple instances of the migrator with disjoint prefixes if needed; the
 | 
			
		||||
  RustFS API is stateless and supports parallel PUTs.
 | 
			
		||||
		Reference in New Issue
	
	Block a user