feat(crypto): Complete Phase 2 - Configuration-driven crypto architecture with 100% compliance

## Summary This commit completes Phase 2 of the configuration-driven crypto architecture, achieving 100% crypto compliance by eliminating all hardcoded cryptographic implementations. ## Key Changes ### Phase 1: Plugin Loader Infrastructure - **Plugin Discovery System**: Created StellaOps.Cryptography.PluginLoader with manifest-based loading - **Configuration Model**: Added CryptoPluginConfiguration with regional profiles support - **Dependency Injection**: Extended DI to support plugin-based crypto provider registration - **Regional Configs**: Created appsettings.crypto.{international,russia,eu,china}.yaml - **CI Workflow**: Added .gitea/workflows/crypto-compliance.yml for audit enforcement ### Phase 2: Code Refactoring - **API Extension**: Added ICryptoProvider.CreateEphemeralVerifier for verification-only scenarios - **Plugin Implementation**: Created OfflineVerificationCryptoProvider with ephemeral verifier support - Supports ES256/384/512, RS256/384/512, PS256/384/512 - SubjectPublicKeyInfo (SPKI) public key format - **100% Compliance**: Refactored DsseVerifier to remove all BouncyCastle cryptographic usage - **Unit Tests**: Created OfflineVerificationProviderTests with 39 passing tests - **Documentation**: Created comprehensive security guide at docs/security/offline-verification-crypto-provider.md - **Audit Infrastructure**: Created scripts/audit-crypto-usage.ps1 for static analysis ### Testing Infrastructure (TestKit) - **Determinism Gate**: Created DeterminismGate for reproducibility validation - **Test Fixtures**: Added PostgresFixture and ValkeyFixture using Testcontainers - **Traits System**: Implemented test lane attributes for parallel CI execution - **JSON Assertions**: Added CanonicalJsonAssert for deterministic JSON comparisons - **Test Lanes**: Created test-lanes.yml workflow for parallel test execution ### Documentation - **Architecture**: Created CRYPTO_CONFIGURATION_DRIVEN_ARCHITECTURE.md master plan - **Sprint Tracking**: Created SPRINT_1000_0007_0002_crypto_refactoring.md (COMPLETE) - **API Documentation**: Updated docs2/cli/crypto-plugins.md and crypto.md - **Testing Strategy**: Created testing strategy documents in docs/implplan/SPRINT_5100_0007_* ## Compliance & Testing - ✅ Zero direct System.Security.Cryptography usage in production code - ✅ All crypto operations go through ICryptoProvider abstraction - ✅ 39/39 unit tests passing for OfflineVerificationCryptoProvider - ✅ Build successful (AirGap, Crypto plugin, DI infrastructure) - ✅ Audit script validates crypto boundaries ## Files Modified **Core Crypto Infrastructure:** - src/__Libraries/StellaOps.Cryptography/CryptoProvider.cs (API extension) - src/__Libraries/StellaOps.Cryptography/CryptoSigningKey.cs (verification-only constructor) - src/__Libraries/StellaOps.Cryptography/EcdsaSigner.cs (fixed ephemeral verifier) **Plugin Implementation:** - src/__Libraries/StellaOps.Cryptography.Plugin.OfflineVerification/ (new) - src/__Libraries/StellaOps.Cryptography.PluginLoader/ (new) **Production Code Refactoring:** - src/AirGap/StellaOps.AirGap.Importer/Validation/DsseVerifier.cs (100% compliant) **Tests:** - src/__Libraries/__Tests/StellaOps.Cryptography.Plugin.OfflineVerification.Tests/ (new, 39 tests) - src/__Libraries/__Tests/StellaOps.Cryptography.PluginLoader.Tests/ (new) **Configuration:** - etc/crypto-plugins-manifest.json (plugin registry) - etc/appsettings.crypto.*.yaml (regional profiles) **Documentation:** - docs/security/offline-verification-crypto-provider.md (600+ lines) - docs/implplan/CRYPTO_CONFIGURATION_DRIVEN_ARCHITECTURE.md (master plan) - docs/implplan/SPRINT_1000_0007_0002_crypto_refactoring.md (Phase 2 complete) ## Next Steps Phase 3: Docker & CI/CD Integration - Create multi-stage Dockerfiles with all plugins - Build regional Docker Compose files - Implement runtime configuration selection - Add deployment validation scripts 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2025-12-23 18:20:00 +02:00
parent b444284be5
commit dac8e10e36
241 changed files with 22567 additions and 307 deletions
--- a/docs2/ingestion/aggregation-and-linksets.md
+++ b/docs2/ingestion/aggregation-and-linksets.md
@@ -0,0 +1,104 @@
+# Ingestion, aggregation, and linksets
+
+StellaOps ingestion is governed by the Aggregation-Only Contract (AOC). The
+rules enforce deterministic, policy-neutral collection of advisory and VEX data.
+
+AOC core rules
+- Ingestion writes raw facts only. No derived severity, consensus, or policy hints.
+- No merges. Each upstream document is stored independently.
+- Provenance is mandatory: source metadata, content hashes, signature fields.
+- Idempotent writes keyed by vendor + upstream id + content hash.
+- Append-only revisions via supersedes pointers.
+- Deterministic output: canonical JSON, UTC timestamps, stable ordering.
+
+Ingestion pipeline (high level)
+1) Fetch upstream payload.
+2) Validate signature and schema.
+3) Normalize metadata (timestamps, ids, content hash).
+4) Persist raw document (append-only).
+5) Emit observation (immutable record).
+6) Build linksets (deterministic correlation).
+7) Expose via API and Offline Kit snapshots.
+
+Advisory observations (Concelier)
+- observationId format: {tenant}:{source.vendor}:{upstreamId}:{revision}.
+- Key fields: tenant, source, upstream, content.raw, identifiers, linkset hints.
+- Supersedes pointer links revisions without mutation.
+
+VEX observations (Excititor)
+- observationId format: {tenant}:{providerId}:{upstreamId}:{revision}.
+- Raw VEX payload plus normalized statement tuples.
+- Linkset hints include purls, cpes, aliases, references.
+
+Linksets and conflicts
+- Linksets correlate observations by product identity while preserving sources.
+- Deterministic ids are hashes of sorted identifiers and observation references.
+- Conflicts are recorded, not resolved. Common conflict types:
+  - severity mismatch
+  - affected range divergence
+  - status or justification mismatch
+  - alias inconsistency
+  - metadata gap (missing provenance)
+
+Observation example (short)
+```json
+{
+  "observationId": "tenant-a:redhat:CVE-2025-0001:1",
+  "tenant": "tenant-a",
+  "source": { "vendor": "redhat", "stream": "csaf" },
+  "upstream": {
+    "upstreamId": "CVE-2025-0001",
+    "documentVersion": "2025-01-10",
+    "contentHash": "sha256:1111...",
+    "signature": { "present": true }
+  },
+  "identifiers": { "cve": "CVE-2025-0001", "aliases": ["RHSA-2025:1234"] },
+  "linkset": { "purls": ["pkg:rpm/redhat/openssl@1.1.1w-12"] }
+}
+```
+
+Deterministic linkset id
+- Build a canonical string with sorted identifiers and observation ids.
+- linksetId = sha256(tenant + "|" + join(sorted(purls)) + "|" + join(sorted(observationIds)))
+
+Linkset example (short)
+```json
+{
+  "linksetId": "tenant-a:sha256:2222...",
+  "observations": ["tenant-a:redhat:CVE-2025-0001:1", "tenant-a:nvd:CVE-2025-0001:3"],
+  "purls": ["pkg:rpm/redhat/openssl@1.1.1w-12"],
+  "conflicts": [{ "type": "severity-mismatch" }]
+}
+```
+
+Idempotency and supersedes
+- Same content hash results in a no-op.
+- New content hash creates a new observation with supersedes set.
+- Supersedes chains are append-only and acyclic.
+
+AOC error model
+- ERR_AOC_001: forbidden derived fields detected.
+- ERR_AOC_002: merge attempt detected.
+- ERR_AOC_003: idempotency violation.
+- ERR_AOC_004: missing provenance.
+- ERR_AOC_005: signature or checksum mismatch.
+- ERR_AOC_006: derived findings write attempt.
+- ERR_AOC_007: schema violation.
+
+Downstream consumers
+- Policy Engine applies rules and produces effective findings.
+- Console and CLI render evidence panels and conflicts.
+- Offline Kit bundles observations and linksets for air-gapped parity.
+
+Validation and tests
+- Schema validators and guard libraries enforce AOC rules.
+- Unit and integration tests validate idempotency and linkset hashes.
+- CLI verifier and offline kit checks confirm determinism.
+
+Related references
+- docs/ingestion/aggregation-only-contract.md
+- docs/aoc/aoc-guardrails.md
+- docs2/ingestion/aoc-guardrails.md
+- ingestion/backfill.md
+- docs/advisories/aggregation.md
+- docs/vex/aggregation.md
--- a/docs2/ingestion/aoc-guardrails.md
+++ b/docs2/ingestion/aoc-guardrails.md
@@ -0,0 +1,33 @@
+# AOC guardrails
+
+AOC guardrails enforce deterministic, policy-neutral ingestion in Concelier and
+Excititor. Ingestion writes raw facts only and never computes precedence,
+severity, or policy hints.
+
+Guardrail rules
+- Ingestion writes immutable observations and linksets only.
+- Derived semantics belong to Policy Engine and downstream views.
+- Provenance metadata is mandatory for every ingested record.
+- Outputs must be deterministic for identical inputs.
+- CI and analyzers should fail builds that violate these rules.
+
+Guard library (StellaOps.Aoc)
+- IAocGuard validates payloads and returns structured violations.
+- AocGuardOptions toggles signature and tenant requirements.
+- AocError carries machine-readable error codes for APIs and CLI.
+- AspNetCore filters enforce guardrails on Minimal API endpoints.
+
+Allowed fields and validation
+- Top-level allowlist enforces schema boundaries.
+- Required fields are configurable for staged schema changes.
+- Unknown fields produce ERR_AOC violations.
+
+Usage guidance
+- Register the guard in ingestion services before repositories.
+- Validate payloads before any persistence.
+- Use RFC 7807 problem responses for consistent errors.
+
+Related references
+- docs/aoc/aoc-guardrails.md
+- docs/aoc/guard-library.md
+- docs/ingestion/aggregation-only-contract.md
--- a/docs2/ingestion/backfill.md
+++ b/docs2/ingestion/backfill.md
@@ -0,0 +1,40 @@
+# AOC linkset backfill
+
+Purpose
+- Safely backfill advisory linksets and observations under Aggregation-Only rules.
+- Preserve offline kit integrity and determinism during data migrations.
+
+Inputs
+- Deterministic NDJSON dataset (gzip) for linksets and observations.
+- Target database and collections for advisory linksets and observations.
+- Offline kit bundle mirroring the backfill dataset.
+
+Preparation
+- Run a dry-run import to validate schema and guardrails.
+- Backup target collections before any import.
+- Stage rollback scripts and confirm indexes are reproducible.
+- Set ingestion flags for backfill windows (link-not-merge enabled, aggregation-only disabled if required for rehearsal).
+
+Execution
+- Import the NDJSON dataset with deterministic ordering.
+- Record import metrics and structured logs.
+- Run a determinism probe test that compares golden hashes.
+
+Rollback
+- Restore from backup and reapply deterministic indexes.
+- Re-run determinism probes and confirm guard flags are reset.
+
+Evidence to capture
+- Backup hash or archive checksum.
+- Import logs with counts and zero merge counters.
+- Determinism test results and hashes.
+- Offline kit bundle hash.
+
+Dataset generation
+- Export from staging with a pinned tenant and stable ordering.
+- Verify determinism by hashing the NDJSON output twice; hashes must match.
+- Publish a .sha256 alongside the dataset.
+
+Offline posture
+- Backfill datasets are mirrored into offline kits for air-gap verification.
+- Exports and evidence are stored as content-addressed artifacts.