doctor enhancements, setup, enhancements, ui functionality and design consolidation and , test projects fixes , product advisory attestation/rekor and delta verfications enhancements

2026-01-19 09:02:59 +02:00
parent 8c4bf54aed
commit 17419ba7c4
809 changed files with 170738 additions and 12244 deletions
--- a/docs/operations/artifact-migration-runbook.md
+++ b/docs/operations/artifact-migration-runbook.md
@@ -0,0 +1,212 @@
+# Artifact Store Migration Runbook
+
+Sprint: SPRINT_20260118_017_Evidence_artifact_store_unification (AS-006)
+
+## Overview
+
+This runbook covers the migration of existing evidence from legacy artifact stores to the unified ArtifactStore.
+
+## Migration Sources
+
+| Source | Legacy Path | Description |
+|--------|-------------|-------------|
+| EvidenceLocker | `tenants/{tenantId}/bundles/{bundleId}/{sha256}-{name}` | Evidence bundles |
+| Attestor | `attest/dsse/{bundleSha256}.json` | DSSE envelopes |
+| Vex | `{prefix}/{format}/{digest}.{ext}` | VEX documents |
+
+## Target Path Convention
+
+All artifacts are migrated to: `/artifacts/{bom-ref-encoded}/{serialNumber}/{artifactId}.json`
+
+## Pre-Migration Checklist
+
+- [ ] Backup existing S3 buckets
+- [ ] Verify PostgreSQL backup is current
+- [ ] Ensure sufficient storage for duplicated data
+- [ ] Review migration in dry-run mode first
+- [ ] Notify stakeholders of potential service impact
+
+## Running the Migration
+
+### Dry Run (Recommended First Step)
+
+```bash
+stella artifacts migrate --source all --dry-run --output migration-preview.json
+```
+
+### Full Migration
+
+```bash
+# Migrate all sources with default settings
+stella artifacts migrate --source all
+
+# Migrate with increased parallelism
+stella artifacts migrate --source all --parallelism 8 --batch-size 200
+
+# Migrate specific source
+stella artifacts migrate --source evidence --output migration-report.json
+
+# Migrate specific tenant
+stella artifacts migrate --source all --tenant <tenant-uuid>
+```
+
+### Resuming Failed Migration
+
+```bash
+# Use checkpoint ID from previous run
+stella artifacts migrate --source all --resume-from <checkpoint-id>
+```
+
+## Progress Monitoring
+
+The CLI displays real-time progress:
+
+```
+  Progress: 1500/10000 (15.0%) - Success: 1495, Failed: 3, Skipped: 2
+```
+
+## Rollback Procedure
+
+### When to Rollback
+
+- Migration corrupted data
+- Performance degradation after migration
+- Business-critical bug discovered
+
+### Rollback Steps
+
+#### 1. Stop New Writes to Unified Store
+
+```bash
+# Disable unified store in configuration
+kubectl set env deployment/evidence-locker ARTIFACT_STORE_UNIFIED_ENABLED=false
+kubectl set env deployment/attestor ARTIFACT_STORE_UNIFIED_ENABLED=false
+```
+
+#### 2. Revert Application Configuration
+
+```yaml
+# etc/appsettings.yaml
+artifactStore:
+  useUnifiedStore: false
+  legacyMode: true
+```
+
+#### 3. Clear Unified Store Index
+
+```sql
+-- Clear PostgreSQL index (preserves S3 data)
+TRUNCATE TABLE artifact_store.artifacts;
+```
+
+#### 4. (Optional) Remove Migrated S3 Objects
+
+```bash
+# Only if disk space is critical and you're certain about rollback
+# WARNING: This is destructive!
+aws s3 rm s3://artifacts-bucket/artifacts/ --recursive
+```
+
+#### 5. Restart Services
+
+```bash
+kubectl rollout restart deployment/evidence-locker
+kubectl rollout restart deployment/attestor
+```
+
+#### 6. Verify Legacy Stores Work
+
+```bash
+# Test evidence retrieval
+stella evidence get --bundle-id <test-bundle>
+
+# Test attestation retrieval  
+stella attestor get --digest <test-digest>
+```
+
+## Post-Migration Validation
+
+### Verify Artifact Counts
+
+```sql
+-- Count migrated artifacts by source
+SELECT 
+  CASE 
+    WHEN storage_key LIKE '%evidence%' THEN 'evidence'
+    WHEN storage_key LIKE '%dsse%' THEN 'attestor'
+    WHEN storage_key LIKE '%vex%' THEN 'vex'
+    ELSE 'unknown'
+  END as source,
+  COUNT(*) as count
+FROM artifact_store.artifacts
+GROUP BY 1;
+```
+
+### Verify bom-ref Extraction
+
+```sql
+-- Check for artifacts with synthetic bom-refs (extraction failed)
+SELECT COUNT(*) as synthetic_count
+FROM artifact_store.artifacts
+WHERE bom_ref LIKE 'sha256:%';
+```
+
+### Test Retrieval
+
+```bash
+# Query by bom-ref
+curl "https://api.example.com/api/v1/artifacts?bom_ref=pkg:docker/acme/api@sha256:abc123"
+
+# Verify content matches original
+stella artifacts compare \
+  --original tenants/xxx/bundles/yyy/sha256-sbom.json \
+  --migrated /artifacts/encoded-ref/serial/artifact.json
+```
+
+## Troubleshooting
+
+### Migration Stuck
+
+```bash
+# Check for stuck workers
+ps aux | grep migrate
+
+# Check migration checkpoints
+cat /var/lib/stella/migration-checkpoint.json
+```
+
+### High Failure Rate
+
+1. Check migration report for common errors
+2. Verify source store connectivity
+3. Check for corrupted source artifacts
+4. Increase batch size for memory issues
+
+### Slow Migration
+
+1. Increase parallelism (up to CPU count)
+2. Run during off-peak hours
+3. Consider migrating by tenant in parallel
+4. Verify network bandwidth to S3
+
+## Representative Dataset Testing
+
+Before production migration, test with representative dataset:
+
+```bash
+# Export sample from each source
+stella evidence list --limit 100 --output sample-evidence.json
+stella attestor list --limit 100 --output sample-attestor.json
+
+# Create test environment with samples
+stella artifacts migrate --source all --tenant test-tenant --output test-report.json
+
+# Verify counts and content
+diff <(cat sample-evidence.json | jq '.total') <(cat test-report.json | jq '.succeeded')
+```
+
+## Related Documentation
+
+- [Artifact Store API](../api/artifact-store-api.yaml)
+- [IArtifactStore Interface](../../src/__Libraries/StellaOps.Artifact.Core/IArtifactStore.cs)
+- [PostgreSQL Index Schema](../../src/__Libraries/StellaOps.Artifact.Infrastructure/Migrations/001_artifact_index_schema.sql)