Add tests for SBOM generation determinism across multiple formats

- Created `StellaOps.TestKit.Tests` project for unit tests related to determinism.
- Implemented `DeterminismManifestTests` to validate deterministic output for canonical bytes and strings, file read/write operations, and error handling for invalid schema versions.
- Added `SbomDeterminismTests` to ensure identical inputs produce consistent SBOMs across SPDX 3.0.1 and CycloneDX 1.6/1.7 formats, including parallel execution tests.
- Updated project references in `StellaOps.Integration.Determinism` to include the new determinism testing library.

docs2/operations/airgap-runbooks.md

@@ -6,6 +6,30 @@ Core runbooks
 - Quarantine: isolate bundles with hash or signature mismatches.
 - Sealed startup diagnostics: confirm egress block and time anchor validity.
 
+Offline kit management
+- Generate full or delta kits in connected environments.
+- Verify kit hash and signature before transfer.
+- Import and install kit, then confirm component freshness.
+
+Feed updates
+- Use delta kits for smaller updates.
+- Roll back to previous snapshot when feeds introduce regressions.
+- Track feed age and kit expiry thresholds.
+
+Scanning in air-gap mode
+- Scan local images or SBOMs without registry pull.
+- Generate SBOMs locally and scan from file.
+- Force offline feeds when required by policy.
+
+Verification in air-gap mode
+- Verify proof bundles offline with local trust roots.
+- Export and import trust bundles for signer and CA rotation.
+- Run score replay with frozen timestamps if needed.
+
+Health checks
+- Monitor kit age, feed freshness, trust store validity, disk usage.
+- Use deterministic health checks and keep results for audit.
+
 Import and verify
 - Validate bundle hash, manifest entries, and schema checks.
 - Record import receipt with operator, time anchor, and manifest hash.

docs2/operations/key-rotation.md

@@ -0,0 +1,49 @@
+# Key rotation
+
+Purpose
+- Rotate signing keys without invalidating historical DSSE proofs.
+
+Principles
+- Do not mutate old DSSE envelopes.
+- Keep key history; revoke instead of delete.
+- Publish key material to trust anchors and mirrors.
+- Audit all key lifecycle events.
+
+Key profiles (examples)
+- default: SHA256-ED25519
+- fips: SHA256-ECDSA-P256
+- gost: GOST-R-34.10-2012
+- sm2: SM2-P256
+- pqc: ML-DSA-65
+
+Rotation workflow
+1. Generate a new key in the configured keystore.
+2. Add the key to the trust anchor without removing old keys.
+3. Run a transition period where both keys verify.
+4. Revoke the old key with an effective date.
+5. Publish updated key material to attestation feeds or mirrors.
+
+Trust anchors
+- Scoped by PURL pattern and allowed predicate types.
+- Store allowedKeyIds, revokedKeys, and keyHistory with timestamps.
+
+Verification with key history
+- Verify signatures using the key valid at the time of signing.
+- Revoked keys remain valid for pre-revocation attestations.
+
+Emergency revocation
+- Revoke compromised keys immediately and publish updated anchors.
+- Re-issue trust bundles and notify downstream verifiers.
+
+Metrics and alerts
+- signer_key_age_days
+- signer_keys_active_total
+- signer_keys_revoked_total
+- signer_rotation_events_total
+- signer_verification_key_lookups_total
+- Alerts when keys near or exceed maximum age.
+
+Related references
+- security/crypto-and-trust.md
+- provenance/attestation-workflow.md
+- docs/operations/key-rotation-runbook.md

docs2/operations/proof-verification.md

@@ -0,0 +1,37 @@
+# Proof verification
+
+Purpose
+- Verify DSSE bundles and transparency proofs for scan and score evidence.
+
+Components
+- DSSE envelope and signature bundle.
+- Certificate chain and trust roots.
+- Rekor inclusion proof and checkpoint when online.
+
+Basic verification
+- Verify DSSE signature against trusted roots.
+- Confirm subject digest matches expected artifact.
+- Validate Merkle inclusion proof when available.
+
+Offline verification
+- Use embedded proofs and local trust bundles.
+- Skip online Rekor queries in sealed mode.
+- Record verification results in timeline events.
+
+Transparency log integration
+- Check Rekor entry status and inclusion proof.
+- When Rekor is unavailable, rely on cached checkpoint and proofs.
+
+Troubleshooting cues
+- DSSE signature invalid: check key rotation or trust anchors.
+- Merkle root mismatch: verify checkpoint and bundle integrity.
+- Certificate chain failure: refresh trust roots.
+
+Monitoring
+- Track verification latency and failure counts.
+- Alert on certificate expiry or rising verification failures.
+
+Related references
+- provenance/attestation-workflow.md
+- release/promotion-attestations.md
+- docs/operations/proof-verification-runbook.md

docs2/operations/reachability.md

@@ -0,0 +1,36 @@
+# Reachability operations
+
+Purpose
+- Operate call graph ingestion, reachability computation, and explain queries.
+
+Reachability statuses
+- unreachable, possibly_reachable, reachable_static, reachable_proven, unknown.
+
+Call graph operations
+- Upload call graphs and validate schema.
+- Inspect entrypoints and merge graphs when required.
+- Enforce size limits and deterministic ordering.
+
+Computation
+- Trigger reachability computation per scan or batch.
+- Monitor jobs for timeouts and memory caps.
+- Persist results with graph_cache_epoch for replay.
+
+Explain queries
+- Explain a single finding or batch.
+- Provide alternate paths and reasons for unreachable results.
+
+Drift handling
+- Track changes due to graph updates or reachability algorithm changes.
+- Use drift reports to compare runs and highlight path changes.
+
+Monitoring
+- Track computation latency, queue depth, and explain request rates.
+- Alert on repeated timeouts or inconsistent results.
+
+Related references
+- architecture/reachability-lattice.md
+- architecture/reachability-evidence.md
+- operations/score-proofs.md
+- docs/operations/reachability-runbook.md
+- docs/operations/reachability-drift-guide.md

docs2/operations/runbooks.md

@@ -12,6 +12,12 @@ Runbook set (current)
 - docs/runbooks/replay_ops.md
 - docs/runbooks/vex-ops.md
 - docs/runbooks/vuln-ops.md
+- operations/score-proofs.md
+- operations/proof-verification.md
+- operations/reachability.md
+- operations/trust-lattice.md
+- operations/unknowns-queue.md
+- operations/key-rotation.md
 
 Common expectations
 - Hash and store any inbound artifacts with SHA256SUMS.

docs2/operations/score-proofs.md

@@ -0,0 +1,46 @@
+# Score proofs and replay
+
+Purpose
+- Provide deterministic score proofs with replayable inputs and attestations.
+
+When to replay
+- Determinism audits and compliance checks.
+- Dispute resolution or vendor verification.
+- Regression investigation after feed or policy changes.
+
+Replay operations
+- Trigger replay via CLI or API with scan or job id.
+- Support batch replay with concurrency limits.
+- Nightly replay jobs validate determinism at scale.
+
+Verification
+- Online verification uses DSSE and Rekor proofs.
+- Offline verification uses embedded proofs and local trust bundles.
+- Verification checks include bundle hash, signature, and input digests.
+
+Bundle contents
+- Manifest with inputs and hashes.
+- SBOM, advisories, VEX snapshots.
+- Deterministic scoring outputs and explain traces.
+- DSSE bundle and transparency proof.
+
+Retention and export
+- Retain bundles per policy; export for audit with manifests.
+- Store in Evidence Locker and Offline Kits.
+
+Monitoring metrics
+- score_replay_duration_seconds
+- proof_verification_success_rate
+- proof_bundle_size_bytes
+- replay_queue_depth
+- proof_generation_failures
+
+Alerting cues
+- Replay latency p95 > 30s.
+- Verification failures or queue backlog spikes.
+
+Related references
+- operations/proof-verification.md
+- operations/replay-and-determinism.md
+- docs/operations/score-proofs-runbook.md
+- docs/operations/score-replay-runbook.md

docs2/operations/trust-lattice.md

@@ -0,0 +1,33 @@
+# Trust lattice operations
+
+Purpose
+- Monitor and operate trust lattice gates for VEX and policy decisions.
+
+Core components
+- Trust vectors and gate configuration.
+- Verdict replay for deterministic validation.
+
+Monitoring
+- Track gate failure rate, verdict replay failures, and trust vector drift.
+- Use dashboards for gate health and override usage.
+
+Common operations
+- View current trust vectors and gate configuration.
+- Inspect a verdict and its trust inputs.
+- Trigger manual calibration when required.
+
+Emergency procedures
+- High gate failure rate: pause dependent workflows and investigate sources.
+- Verdict replay failures: verify inputs, cache epochs, and policy versions.
+- Trust vector drift: run replay with frozen inputs and compare hashes.
+
+Maintenance
+- Daily checks: gate failure rate and queue depth.
+- Weekly checks: trust vector calibration and drift review.
+- Monthly checks: update trust bundles and audit logs.
+
+Related references
+- architecture/reachability-vex.md
+- vex/consensus.md
+- docs/operations/trust-lattice-runbook.md
+- docs/operations/trust-lattice-troubleshooting.md

docs2/operations/unknowns-queue.md

@@ -0,0 +1,32 @@
+# Unknowns queue operations
+
+Purpose
+- Manage unknown components with deterministic triage and SLA tracking.
+
+Queue model
+- Bands: HOT, WARM, COLD based on score and SLA.
+- Reasons include reachability gaps, provenance gaps, VEX conflicts, and ingestion gaps.
+
+Core workflows
+- List and triage unknowns by band and reason.
+- Escalate or resolve with documented justification.
+- Suppress with expiry and audit trail when approved.
+
+Budgets and SLAs
+- Per-environment budgets cap unknowns by reason.
+- SLA timers trigger alerts when breached.
+
+Monitoring
+- unknowns_total, unknowns_hot_count, unknowns_sla_breached
+- unknowns_escalation_failures, unknowns_avg_age_hours
+- KEV-specific unknown counts and age
+
+Alerting cues
+- HOT band spikes or SLA breaches.
+- KEV unknowns older than 24 hours.
+- Rising queue growth rate.
+
+Related references
+- signals/unknowns.md
+- signals/unknowns-ranking.md
+- docs/operations/unknowns-queue-runbook.md