documentation cleanse, sprints work and planning. remaining non EF DAL migration to EF

docs-archived/modules/evidence/README.md

@@ -0,0 +1,49 @@
+# Evidence
+
+**Status:** Design/Planning
+**Source:** N/A (cross-cutting concept)
+**Owner:** Platform Team
+
+## Purpose
+
+Evidence defines the unified evidence model for vulnerability findings across StellaOps. Provides canonical data structures for evidence capture, aggregation, and scoring used by Signals, Policy Engine, and EvidenceLocker modules.
+
+## Components
+
+**Concept Documentation:**
+- `unified-model.md` - Unified evidence data model specification
+
+**Evidence Types:**
+- Reachability evidence (call graph, data flow)
+- Runtime evidence (eBPF traces, dynamic observations)
+- Binary evidence (backport detection, fix validation)
+- Exploit evidence (EPSS scores, KEV flags, exploit-db entries)
+- VEX evidence (source trust, statement provenance)
+- Mitigation evidence (active mitigations, compensating controls)
+
+## Implementation Locations
+
+Evidence structures are implemented across multiple modules:
+- **Signals** - Evidence aggregation and normalization
+- **Policy Engine** - Reachability analysis and evidence generation
+- **EvidenceLocker** - Evidence storage and sealing
+- **Scanner** - Binary and vulnerability evidence capture
+- **Concelier** - Backport and exploit evidence enrichment
+
+## Dependencies
+
+- All evidence-producing modules (Scanner, Policy, Concelier, etc.)
+- Signals (evidence aggregation)
+- EvidenceLocker (evidence storage)
+
+## Related Documentation
+
+- Unified Model: `./unified-model.md`
+- Signals: `../signals/`
+- Policy: `../policy/`
+- EvidenceLocker: `../evidence-locker/`
+- Data Schemas: `../../11_DATA_SCHEMAS.md`
+
+## Current Status
+
+Evidence model documented in `unified-model.md`. Implementation distributed across Signals (aggregation), Policy (reachability), EvidenceLocker (storage), and Scanner (capture) modules.

docs-archived/modules/evidence/unified-model.md

@@ -0,0 +1,360 @@
+# Unified Evidence Model
+
+> **Module:** `StellaOps.Evidence.Core`  
+> **Status:** Production  
+> **Owner:** Platform Guild
+
+## Overview
+
+The Unified Evidence Model provides a standardized interface (`IEvidence`) and implementation (`EvidenceRecord`) for representing evidence across all StellaOps modules. This enables:
+
+- **Cross-module evidence linking**: Evidence from Scanner, Attestor, Excititor, and Policy modules share a common contract.
+- **Content-addressed verification**: Evidence records are immutable and verifiable via deterministic hashing.
+- **Unified storage**: A single `IEvidenceStore` interface abstracts persistence across modules.
+- **Cryptographic attestation**: Multiple signatures from different signers (internal, vendor, CI, operator) can vouch for evidence.
+
+## Core Types
+
+### IEvidence Interface
+
+```csharp
+public interface IEvidence
+{
+    string SubjectNodeId { get; }           // Content-addressed subject
+    EvidenceType EvidenceType { get; }      // Type discriminator
+    string EvidenceId { get; }              // Computed hash identifier
+    ReadOnlyMemory<byte> Payload { get; }   // Canonical JSON payload
+    IReadOnlyList<EvidenceSignature> Signatures { get; }
+    EvidenceProvenance Provenance { get; }
+    string? ExternalPayloadCid { get; }     // For large payloads
+    string PayloadSchemaVersion { get; }
+}
+```
+
+### EvidenceType Enum
+
+The platform supports these evidence types:
+
+| Type | Value | Description | Example Payload |
+|------|-------|-------------|-----------------|
+| `Reachability` | 1 | Call graph analysis | Paths, confidence, graph digest |
+| `Scan` | 2 | Vulnerability finding | CVE, severity, affected package |
+| `Policy` | 3 | Policy evaluation | Rule ID, verdict, inputs |
+| `Artifact` | 4 | SBOM entry metadata | PURL, digest, build info |
+| `Vex` | 5 | VEX statement | Status, justification, impact |
+| `Epss` | 6 | EPSS score | Score, percentile, model date |
+| `Runtime` | 7 | Runtime observation | eBPF/ETW traces, call frames |
+| `Provenance` | 8 | Build provenance | SLSA attestation, builder info |
+| `Exception` | 9 | Applied exception | Exception ID, reason, expiry |
+| `Guard` | 10 | Guard/gate analysis | Gate type, condition, bypass |
+| `Kev` | 11 | KEV status | In-KEV flag, added date |
+| `License` | 12 | License analysis | SPDX ID, compliance status |
+| `Dependency` | 13 | Dependency metadata | Graph edge, version range |
+| `Custom` | 100 | User-defined | Schema-versioned custom payload |
+
+### EvidenceRecord
+
+The concrete implementation with deterministic identity:
+
+```csharp
+public sealed record EvidenceRecord : IEvidence
+{
+    public static EvidenceRecord Create(
+        string subjectNodeId,
+        EvidenceType evidenceType,
+        ReadOnlyMemory<byte> payload,
+        EvidenceProvenance provenance,
+        string payloadSchemaVersion,
+        IReadOnlyList<EvidenceSignature>? signatures = null,
+        string? externalPayloadCid = null);
+
+    public bool VerifyIntegrity();
+}
+```
+
+**EvidenceId Computation:**
+
+The `EvidenceId` is a SHA-256 hash of the canonicalized fields using versioned prefixing:
+
+```
+EvidenceId = "evidence:" + CanonJson.HashVersionedPrefixed("IEvidence", "v1", {
+    SubjectNodeId,
+    EvidenceType,
+    PayloadHash,
+    Provenance.GeneratorId,
+    Provenance.GeneratorVersion,
+    Provenance.GeneratedAt (ISO 8601)
+})
+```
+
+### EvidenceSignature
+
+Cryptographic attestation by a signer:
+
+```csharp
+public sealed record EvidenceSignature
+{
+    public required string SignerId { get; init; }
+    public required string Algorithm { get; init; }    // ES256, RS256, EdDSA
+    public required string SignatureBase64 { get; init; }
+    public required DateTimeOffset SignedAt { get; init; }
+    public SignerType SignerType { get; init; }
+    public IReadOnlyList<string>? CertificateChain { get; init; }
+}
+```
+
+**SignerType Values:**
+- `Internal` (0): StellaOps service
+- `Vendor` (1): External vendor/supplier
+- `CI` (2): CI/CD pipeline
+- `Operator` (3): Human operator
+- `TransparencyLog` (4): Rekor/transparency log
+- `Scanner` (5): Security scanner
+- `PolicyEngine` (6): Policy engine
+- `Unknown` (255): Unclassified
+
+### EvidenceProvenance
+
+Generation context:
+
+```csharp
+public sealed record EvidenceProvenance
+{
+    public required string GeneratorId { get; init; }
+    public required string GeneratorVersion { get; init; }
+    public required DateTimeOffset GeneratedAt { get; init; }
+    public string? CorrelationId { get; init; }
+    public Guid? TenantId { get; init; }
+    // ... additional fields
+}
+```
+
+## Adapters
+
+Adapters convert module-specific evidence types to the unified `IEvidence` interface:
+
+### Available Adapters
+
+| Adapter | Source Module | Source Type | Target Evidence Types |
+|---------|---------------|-------------|----------------------|
+| `EvidenceBundleAdapter` | Scanner | `EvidenceBundle` | Reachability, Vex, Provenance, Scan |
+| `EvidenceStatementAdapter` | Attestor | `EvidenceStatement` (in-toto) | Scan |
+| `ProofSegmentAdapter` | Scanner | `ProofSegment` | Varies by segment type |
+| `VexObservationAdapter` | Excititor | `VexObservation` | Vex, Provenance |
+| `ExceptionApplicationAdapter` | Policy | `ExceptionApplication` | Exception |
+
+### Adapter Interface
+
+```csharp
+public interface IEvidenceAdapter<TSource>
+{
+    IReadOnlyList<IEvidence> Convert(
+        TSource source,
+        string subjectNodeId,
+        EvidenceProvenance provenance);
+
+    bool CanConvert(TSource source);
+}
+```
+
+### Using Adapters
+
+Adapters use **input DTOs** to avoid circular dependencies:
+
+```csharp
+// Using VexObservationAdapter
+var adapter = new VexObservationAdapter();
+
+var input = new VexObservationInput
+{
+    ObservationId = "obs-001",
+    ProviderId = "nvd",
+    StreamId = "cve-feed",
+    // ... other fields from VexObservation
+};
+
+var provenance = new EvidenceProvenance
+{
+    GeneratorId = "excititor-ingestor",
+    GeneratorVersion = "1.0.0",
+    GeneratedAt = DateTimeOffset.UtcNow
+};
+
+if (adapter.CanConvert(input))
+{
+    IReadOnlyList<IEvidence> records = adapter.Convert(
+        input,
+        subjectNodeId: "sha256:abc123",
+        provenance);
+}
+```
+
+## Evidence Store
+
+### IEvidenceStore Interface
+
+```csharp
+public interface IEvidenceStore
+{
+    Task<EvidenceRecord> StoreAsync(
+        EvidenceRecord record,
+        CancellationToken ct = default);
+
+    Task<IReadOnlyList<EvidenceRecord>> StoreBatchAsync(
+        IEnumerable<EvidenceRecord> records,
+        CancellationToken ct = default);
+
+    Task<EvidenceRecord?> GetByIdAsync(
+        string evidenceId,
+        CancellationToken ct = default);
+
+    Task<IReadOnlyList<EvidenceRecord>> GetBySubjectAsync(
+        string subjectNodeId,
+        EvidenceType? evidenceType = null,
+        CancellationToken ct = default);
+
+    Task<IReadOnlyList<EvidenceRecord>> GetByTypeAsync(
+        EvidenceType evidenceType,
+        int limit = 100,
+        CancellationToken ct = default);
+
+    Task<bool> ExistsAsync(
+        string evidenceId,
+        CancellationToken ct = default);
+
+    Task<bool> DeleteAsync(
+        string evidenceId,
+        CancellationToken ct = default);
+}
+```
+
+### Implementations
+
+- **`InMemoryEvidenceStore`**: Thread-safe in-memory store for testing and development.
+- **`PostgresEvidenceStore`** (planned): Production store with tenant isolation and indexing.
+
+## Usage Examples
+
+### Creating Evidence
+
+```csharp
+var provenance = new EvidenceProvenance
+{
+    GeneratorId = "scanner-service",
+    GeneratorVersion = "2.1.0",
+    GeneratedAt = DateTimeOffset.UtcNow,
+    TenantId = tenantId
+};
+
+// Serialize payload to canonical JSON
+var payloadBytes = CanonJson.Canonicalize(new
+{
+    cveId = "CVE-2024-1234",
+    severity = "HIGH",
+    affectedPackage = "pkg:npm/lodash@4.17.20"
+});
+
+var evidence = EvidenceRecord.Create(
+    subjectNodeId: "sha256:abc123def456...",
+    evidenceType: EvidenceType.Scan,
+    payload: payloadBytes,
+    provenance: provenance,
+    payloadSchemaVersion: "scan/v1");
+```
+
+### Storing and Retrieving
+
+```csharp
+var store = new InMemoryEvidenceStore();
+
+// Store
+await store.StoreAsync(evidence);
+
+// Retrieve by ID
+var retrieved = await store.GetByIdAsync(evidence.EvidenceId);
+
+// Retrieve all evidence for a subject
+var allForSubject = await store.GetBySubjectAsync(
+    "sha256:abc123def456...",
+    evidenceType: EvidenceType.Scan);
+
+// Verify integrity
+bool isValid = retrieved!.VerifyIntegrity();
+```
+
+### Cross-Module Evidence Linking
+
+```csharp
+// Scanner produces evidence bundle
+var bundle = scanner.ProduceEvidenceBundle(target);
+
+// Convert to unified evidence
+var adapter = new EvidenceBundleAdapter();
+var evidenceRecords = adapter.Convert(bundle, subjectNodeId, provenance);
+
+// Store all records
+await store.StoreBatchAsync(evidenceRecords);
+
+// Later, any module can query by subject
+var allEvidence = await store.GetBySubjectAsync(subjectNodeId);
+
+// Filter by type
+var reachabilityEvidence = allEvidence
+    .Where(e => e.EvidenceType == EvidenceType.Reachability);
+var vexEvidence = allEvidence
+    .Where(e => e.EvidenceType == EvidenceType.Vex);
+```
+
+## Schema Versioning
+
+Each evidence type payload has a schema version (`PayloadSchemaVersion`) for forward compatibility:
+
+- `scan/v1`: Initial scan evidence schema
+- `reachability/v1`: Reachability evidence schema
+- `vex-statement/v1`: VEX statement evidence schema
+- `proof-segment/v1`: Proof segment evidence schema
+- `exception-application/v1`: Exception application schema
+
+Consumers should check `PayloadSchemaVersion` before deserializing payloads to handle schema evolution.
+
+## Integration Patterns
+
+### Module Integration
+
+Each module that produces evidence should:
+
+1. Create an adapter if converting from module-specific types
+2. Use `EvidenceRecord.Create()` for new evidence
+3. Store evidence via `IEvidenceStore`
+4. Include provenance with generator identification
+
+### Verification Flow
+
+```
+1. Retrieve evidence by SubjectNodeId
+2. Call VerifyIntegrity() to check EvidenceId
+3. Verify signatures against known trust roots
+4. Deserialize and validate payload against schema
+```
+
+## Testing
+
+The `StellaOps.Evidence.Core.Tests` project includes:
+
+- **111 unit tests** covering:
+  - EvidenceRecord creation and hash computation
+  - InMemoryEvidenceStore CRUD operations
+  - All adapter conversions (VexObservation, ExceptionApplication, ProofSegment)
+  - Edge cases and error handling
+
+Run tests:
+```bash
+dotnet test src/__Libraries/StellaOps.Evidence.Core.Tests/
+```
+
+## Related Documentation
+
+- [Proof Chain Architecture](../attestor/proof-chain.md)
+- [Evidence Bundle Design](../scanner/evidence-bundle.md)
+- [VEX Observation Model](../excititor/vex-observation.md)
+- [Policy Exceptions](../policy/exceptions.md)