stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 2 Releases Wiki Activity

Add reference architecture and testing strategy documentation

Browse Source 
- Created a new document for the Stella Ops Reference Architecture outlining the system's topology, trust boundaries, artifact association, and interfaces.
- Developed a comprehensive Testing Strategy document detailing the importance of offline readiness, interoperability, determinism, and operational guardrails.
- Introduced a README for the Testing Strategy, summarizing processing details and key concepts implemented.
- Added guidance for AI agents and developers in the tests directory, including directory structure, test categories, key patterns, and rules for test development.
This commit is contained in:
master
2025-12-22 07:59:15 +02:00
parent 5d398ec442
commit 53503cb407
96 changed files with 37565 additions and 71 deletions
Download Patch File Download Diff File Expand all files Collapse all files

817
docs/product-advisories/14-Dec-2025 - Determinism and Reproducibility Technical Reference.md
View File

@@ -1,817 +0,0 @@
# Determinism and Reproducibility Technical Reference
**Source Advisories**:
- 07-Dec-2025 - Designing Deterministic Vulnerability Scores
- 12-Dec-2025 - Designing a Deterministic Vulnerability Scoring Matrix
- 12-Dec-2025 - Replay Fidelity as a Proof Metric
- 01-Dec-2025 - Benchmarks for a Testable Security Moat
- 02-Dec-2025 - Benchmarking a Testable Security Moat
**Last Updated**: 2025-12-14
---
## 1. SCORE FORMULA (BASIS POINTS)
**Total Score:**
```
riskScore = (wB*B + wR*R + wE*E + wP*P) / 10000
```
**Default Weights (basis points, sum = 10000):**
- `wB=1000` (10%) - Base Severity
- `wR=4500` (45%) - Reachability
- `wE=3000` (30%) - Evidence
- `wP=1500` (15%) - Provenance
## 2. SUBSCORE DEFINITIONS (0-100 integers)
### 2.1 BaseSeverity (B)
```
B = round(CVSS * 10) // CVSS 0.0-10.0 → 0-100
```
### 2.2 Reachability (R)
Hop Buckets:
```
0-2 hops: 100
3 hops: 85
4 hops: 70
5 hops: 55
6 hops: 45
7 hops: 35
8+ hops: 20
unreachable: 0
```
Gate Multipliers (in basis points):
```
behind feature flag: ×7000
auth required: ×8000
admin only: ×8500
non-default config: ×7500
```
Final R:
```
R = bucketScore * gateMultiplier / 10000
```
### 2.3 Evidence (E)
Points:
```
runtime trace: +60
DAST/integration test: +30
SAST precise sink: +20
SCA presence only: +10
```
Freshness Multiplier (basis points):
```
≤ 7 days: ×10000
≤ 30 days: ×9000
≤ 90 days: ×7500
≤ 180 days: ×6000
≤ 365 days: ×4000
> 365 days: ×2000
```
Final E:
```
E = min(100, sum(points)) * freshness / 10000
```
### 2.4 Provenance (P)
```
unsigned/unknown: 0
signed image: 30
signed + SBOM hash-linked: 60
signed + SBOM + DSSE attestations: 80
above + reproducible build match: 100
```
## 3. SCORE POLICY YAML SCHEMA
```yaml
policyVersion: score.v1
weightsBps:
baseSeverity: 1000
reachability: 4500
evidence: 3000
provenance: 1500
reachability:
hopBuckets:
- { maxHops: 2, score: 100 }
- { maxHops: 3, score: 85 }
- { maxHops: 4, score: 70 }
- { maxHops: 5, score: 55 }
- { maxHops: 6, score: 45 }
- { maxHops: 7, score: 35 }
- { maxHops: 9999, score: 20 }
unreachableScore: 0
gateMultipliersBps:
featureFlag: 7000
authRequired: 8000
adminOnly: 8500
nonDefaultConfig: 7500
evidence:
points:
runtime: 60
dast: 30
sast: 20
sca: 10
freshnessBuckets:
- { maxAgeDays: 7, multiplierBps: 10000 }
- { maxAgeDays: 30, multiplierBps: 9000 }
- { maxAgeDays: 90, multiplierBps: 7500 }
- { maxAgeDays: 180, multiplierBps: 6000 }
- { maxAgeDays: 365, multiplierBps: 4000 }
- { maxAgeDays: 99999, multiplierBps: 2000 }
provenance:
levels:
unsigned: 0
signed: 30
signedWithSbom: 60
signedWithSbomAndAttestations: 80
reproducible: 100
overrides:
- name: knownExploitedAndReachable
when:
flags:
knownExploited: true
minReachability: 70
setScore: 95
- name: unreachableAndOnlySca
when:
maxReachability: 0
maxEvidence: 10
clampMaxScore: 25
```
## 4. SCORE DATA CONTRACTS
### 4.1 ScoreInput
```json
{
"asOf": "2025-12-14T10:20:30Z",
"policyVersion": "score.v1",
"reachabilityDigest": "sha256:...",
"evidenceDigest": "sha256:...",
"provenanceDigest": "sha256:...",
"baseSeverityDigest": "sha256:..."
}
```
### 4.2 ScoreResult
```json
{
"scoreId": "score_...",
"riskScore": 73,
"subscores": {
"baseSeverity": 75,
"reachability": 85,
"evidence": 60,
"provenance": 60
},
"cvss": {
"v": "3.1",
"base": 7.5,
"environmental": 5.3,
"vector": "CVSS:3.1/AV:N/AC:L/..."
},
"inputsRef": ["evidence_sha256:...", "env_sha256:..."],
"policyVersion": "score.v1",
"policyDigest": "sha256:...",
"engineVersion": "stella-scorer@1.8.2",
"computedAt": "2025-12-09T10:20:30Z",
"resultDigest": "sha256:...",
"explain": [
{"factor": "reachability", "value": 85, "reason": "3 hops from HTTP endpoint"},
{"factor": "evidence", "value": 60, "reason": "Runtime trace (60pts), 20 days old (×90%)"}
]
}
```
### 4.3 ReachabilityReport
```json
{
"artifactDigest": "sha256:...",
"graphDigest": "sha256:...",
"vulnId": "CVE-2024-1234",
"vulnerableSymbol": "org.example.VulnClass.vulnMethod",
"entrypoints": ["POST /api/upload"],
"shortestPath": {
"hops": 3,
"nodes": [
{"symbol": "UploadController.handleUpload", "file": "Controller.cs", "line": 42},
{"symbol": "ProcessorService.process", "file": "Service.cs", "line": 18},
{"symbol": "org.example.VulnClass.vulnMethod", "file": null, "line": null}
]
},
"gates": [
{"type": "authRequired", "detail": "Requires JWT token"},
{"type": "featureFlag", "detail": "FEATURE_UPLOAD_V2=true"}
],
"computedAt": "2025-12-14T10:15:30Z",
"toolVersion": "reachability-analyzer@2.1.0"
}
```
### 4.4 EvidenceBundle
```json
{
"evidenceId": "sha256:...",
"artifactDigest": "sha256:...",
"vulnId": "CVE-2024-1234",
"type": "RUNTIME",
"tool": "runtime-tracer@1.0.0",
"timestamp": "2025-12-10T14:30:00Z",
"confidence": 95,
"subject": "org.example.VulnClass.vulnMethod",
"payloadDigest": "sha256:..."
}
```
### 4.5 ProvenanceReport
```json
{
"artifactDigest": "sha256:...",
"signatureChecks": [
{"signer": "CI-KEY-1", "algorithm": "ECDSA-P256", "result": "VALID"}
],
"sbomDigest": "sha256:...",
"sbomType": "cyclonedx-1.6",
"attestations": ["sha256:...", "sha256:..."],
"transparencyLogRefs": ["rekor://..."],
"reproducibleMatch": true,
"computedAt": "2025-12-14T10:15:30Z",
"toolVersion": "provenance-verifier@1.0.0"
}
```
## 5. DETERMINISM CONSTRAINTS
### 5.1 Fixed-Point Math
- Use integer basis points (100% = 10,000 bps)
- No floating point in scoring math
- Round only at final display
### 5.2 Canonical Serialization
- RFC-style canonical JSON (JCS)
- Sort keys and arrays deterministically
- Stable ordering for explanation lists by `(factorId, contributingObjectDigest)`
### 5.3 Time Handling
- No implicit time
- `asOf` is explicit input
- Freshness = `asOf - evidence.timestamp`
- Use monotonic time internally
## 6. FIDELITY METRICS
### 6.1 Bitwise Fidelity (BF)
```
BF = identical_outputs / total_replays
Target: ≥ 0.98
```
### 6.2 Semantic Fidelity (SF)
- Normalized object comparison (same packages, versions, CVEs, severities, verdicts)
- Allows formatting differences
### 6.3 Policy Fidelity (PF)
- Final policy decision (pass/fail + reason codes) matches
## 7. SCAN MANIFEST SCHEMA
```json
{
"manifest_version": "1.0",
"scan_id": "scan_123",
"created_at": "2025-12-12T10:15:30Z",
"input": {
"type": "oci_image",
"image_ref": "registry/app@sha256:...",
"layers": ["sha256:...", "sha256:..."],
"source_provenance": {"repo_sha": "abc123", "build_id": "ci-999"}
},
"scanner": {
"engine": "stella",
"scanner_image_digest": "sha256:...",
"scanner_version": "2025.12.0",
"config_digest": "sha256:...",
"flags": ["--deep", "--vex"]
},
"feeds": {
"vuln_feed_bundle_digest": "sha256:...",
"license_db_digest": "sha256:..."
},
"policy": {
"policy_bundle_digest": "sha256:...",
"policy_set": "prod-default"
},
"environment": {
"arch": "amd64",
"os": "linux",
"tz": "UTC",
"locale": "C",
"network": "disabled",
"clock_mode": "frozen",
"clock_value": "2025-12-12T10:15:30Z"
},
"normalization": {
"canonicalizer_version": "1.2.0",
"sbom_schema": "cyclonedx-1.6",
"vex_schema": "cyclonedx-vex-1.0"
}
}
```
## 8. MISMATCH CLASSIFICATION TAXONOMY
```
- Feed drift
- Policy drift
- Runtime drift
- Scanner drift
- Nondeterminism (ordering, concurrency, RNG, time-based logic)
- External IO
```
## 9. POSTGRESQL SCHEMA
```sql
CREATE TABLE scan_manifest (
manifest_id UUID PRIMARY KEY,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
artifact_digest TEXT NOT NULL,
feeds_merkle_root TEXT NOT NULL,
engine_build_hash TEXT NOT NULL,
policy_lattice_hash TEXT NOT NULL,
ruleset_hash TEXT NOT NULL,
config_flags JSONB NOT NULL,
environment_fingerprint JSONB NOT NULL,
raw_manifest JSONB NOT NULL,
raw_manifest_sha256 TEXT NOT NULL
);
CREATE TABLE scan_execution (
execution_id UUID PRIMARY KEY,
manifest_id UUID NOT NULL REFERENCES scan_manifest(manifest_id) ON DELETE CASCADE,
started_at TIMESTAMPTZ NOT NULL,
finished_at TIMESTAMPTZ NOT NULL,
t_ingest_ms INT NOT NULL,
t_analyze_ms INT NOT NULL,
t_reachability_ms INT NOT NULL,
t_vex_ms INT NOT NULL,
t_sign_ms INT NOT NULL,
t_publish_ms INT NOT NULL,
proof_bundle_sha256 TEXT NOT NULL,
findings_sha256 TEXT NOT NULL,
vex_bundle_sha256 TEXT NOT NULL,
replay_mode BOOLEAN NOT NULL DEFAULT FALSE
);
CREATE TABLE classification_history (
id BIGSERIAL PRIMARY KEY,
artifact_digest TEXT NOT NULL,
manifest_id UUID NOT NULL REFERENCES scan_manifest(manifest_id) ON DELETE CASCADE,
execution_id UUID NOT NULL REFERENCES scan_execution(execution_id) ON DELETE CASCADE,
previous_status TEXT NOT NULL,
new_status TEXT NOT NULL,
cause TEXT NOT NULL,
changed_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE VIEW scan_tte AS
SELECT
execution_id,
manifest_id,
(finished_at - started_at) AS tte_interval
FROM scan_execution;
CREATE MATERIALIZED VIEW fn_drift_stats AS
SELECT
date_trunc('day', changed_at) AS day_bucket,
COUNT(*) FILTER (WHERE new_status = 'affected') AS affected_count,
COUNT(*) AS total_reclassified,
ROUND(
(COUNT(*) FILTER (WHERE new_status = 'affected')::numeric /
NULLIF(COUNT(*), 0)) * 100, 4
) AS drift_percent
FROM classification_history
GROUP BY 1;
```
## 10. C# CANONICAL DATA STRUCTURES
```csharp
public sealed record CanonicalScanManifest
{
public required string ArtifactDigest { get; init; }
public required string FeedsMerkleRoot { get; init; }
public required string EngineBuildHash { get; init; }
public required string PolicyLatticeHash { get; init; }
public required string RulesetHash { get; init; }
public required IReadOnlyDictionary<string, string> ConfigFlags { get; init; }
public required EnvironmentFingerprint Environment { get; init; }
}
public sealed record EnvironmentFingerprint
{
public required string CpuModel { get; init; }
public required string RuntimeVersion { get; init; }
public required string Os { get; init; }
public required IReadOnlyDictionary<string, string> Extra { get; init; }
}
public sealed record ScanExecutionMetrics
{
public required int IngestMs { get; init; }
public required int AnalyzeMs { get; init; }
public required int ReachabilityMs { get; init; }
public required int VexMs { get; init; }
public required int SignMs { get; init; }
public required int PublishMs { get; init; }
}
```
## 11. CANONICALIZATION IMPLEMENTATION
```csharp
internal static class CanonicalJson
{
private static readonly JsonSerializerOptions Options = new()
{
WriteIndented = false,
PropertyNamingPolicy = JsonNamingPolicy.CamelCase
};
public static string Serialize(object obj)
{
using var stream = new MemoryStream();
using (var writer = new Utf8JsonWriter(stream, new JsonWriterOptions
{
Indented = false,
SkipValidation = false
}))
{
JsonSerializer.Serialize(writer, obj, obj.GetType(), Options);
}
var bytes = stream.ToArray();
var canonical = JsonCanonicalizer.Canonicalize(bytes);
return canonical;
}
}
```
### 11.1 Full Canonical JSON with Sorted Keys
> **Added**: 2025-12-17 from "Building a Deeper Moat Beyond Reachability" advisory
```csharp
using System.Security.Cryptography;
using System.Text;
using System.Text.Json;
public static class CanonJson
{
public static byte[] Canonicalize<T>(T obj)
{
var json = JsonSerializer.SerializeToUtf8Bytes(obj, new JsonSerializerOptions
{
WriteIndented = false,
PropertyNamingPolicy = JsonNamingPolicy.CamelCase
});
using var doc = JsonDocument.Parse(json);
using var ms = new MemoryStream();
using var writer = new Utf8JsonWriter(ms, new JsonWriterOptions { Indented = false });
WriteElementSorted(doc.RootElement, writer);
writer.Flush();
return ms.ToArray();
}
private static void WriteElementSorted(JsonElement el, Utf8JsonWriter w)
{
switch (el.ValueKind)
{
case JsonValueKind.Object:
w.WriteStartObject();
foreach (var prop in el.EnumerateObject().OrderBy(p => p.Name, StringComparer.Ordinal))
{
w.WritePropertyName(prop.Name);
WriteElementSorted(prop.Value, w);
}
w.WriteEndObject();
break;
case JsonValueKind.Array:
w.WriteStartArray();
foreach (var item in el.EnumerateArray())
WriteElementSorted(item, w);
w.WriteEndArray();
break;
default:
el.WriteTo(w);
break;
}
}
public static string Sha256Hex(ReadOnlySpan<byte> bytes)
=> Convert.ToHexString(SHA256.HashData(bytes)).ToLowerInvariant();
}
```
## 11.2 SCORE PROOF LEDGER
> **Added**: 2025-12-17 from "Building a Deeper Moat Beyond Reachability" advisory
The Score Proof Ledger provides an append-only trail of scoring decisions with per-node hashing.
### Proof Node Types
```csharp
public enum ProofNodeKind { Input, Transform, Delta, Score }
public sealed record ProofNode(
string Id,
ProofNodeKind Kind,
string RuleId,
string[] ParentIds,
string[] EvidenceRefs, // digests / refs inside bundle
double Delta, // 0 for non-Delta nodes
double Total, // running total at this node
string Actor, // module name
DateTimeOffset TsUtc,
byte[] Seed,
string NodeHash // sha256 over canonical node (excluding NodeHash)
);
```
### Proof Hashing
```csharp
public static class ProofHashing
{
public static ProofNode WithHash(ProofNode n)
{
var canonical = CanonJson.Canonicalize(new
{
n.Id, n.Kind, n.RuleId, n.ParentIds, n.EvidenceRefs, n.Delta, n.Total,
n.Actor, n.TsUtc, Seed = Convert.ToBase64String(n.Seed)
});
return n with { NodeHash = "sha256:" + CanonJson.Sha256Hex(canonical) };
}
public static string ComputeRootHash(IEnumerable<ProofNode> nodesInOrder)
{
// Deterministic: root hash over canonical JSON array of node hashes in order.
var arr = nodesInOrder.Select(n => n.NodeHash).ToArray();
var bytes = CanonJson.Canonicalize(arr);
return "sha256:" + CanonJson.Sha256Hex(bytes);
}
}
```
### Minimal Ledger
```csharp
public sealed class ProofLedger
{
private readonly List<ProofNode> _nodes = new();
public IReadOnlyList<ProofNode> Nodes => _nodes;
public void Append(ProofNode node)
{
_nodes.Add(ProofHashing.WithHash(node));
}
public string RootHash() => ProofHashing.ComputeRootHash(_nodes);
}
```
### Score Replay Invariant
The score replay must produce identical ledger root hashes given:
- Same manifest (artifact, snapshots, policy)
- Same seed
- Same timestamp (or frozen clock)
```csharp
public class DeterminismTests
{
[Fact]
public void Score_Replay_IsBitIdentical()
{
var seed = Enumerable.Repeat((byte)7, 32).ToArray();
var inputs = new ScoreInputs(9.0, 0.50, false, ReachabilityClass.Unknown, new("enforced","ro"));
var (s1, l1) = RiskScoring.Score(inputs, "scanA", seed, DateTimeOffset.Parse("2025-01-01T00:00:00Z"));
var (s2, l2) = RiskScoring.Score(inputs, "scanA", seed, DateTimeOffset.Parse("2025-01-01T00:00:00Z"));
Assert.Equal(s1, s2, 10);
Assert.Equal(l1.RootHash(), l2.RootHash());
Assert.True(l1.Nodes.Zip(l2.Nodes).All(z => z.First.NodeHash == z.Second.NodeHash));
}
}
```
## 12. REPLAY RUNNER
```csharp
public static class ReplayRunner
{
public static ReplayResult Replay(Guid manifestId, IScannerEngine engine)
{
var manifest = ManifestRepository.Load(manifestId);
var canonical = CanonicalJson.Serialize(manifest.RawObject);
var canonicalHash = Sha256(canonical);
if (canonicalHash != manifest.RawManifestSHA256)
throw new InvalidOperationException("Manifest integrity violation.");
using var feeds = FeedSnapshotResolver.Open(manifest.FeedsMerkleRoot);
var exec = engine.Scan(new ScanRequest
{
ArtifactDigest = manifest.ArtifactDigest,
Feeds = feeds,
LatticeHash = manifest.PolicyLatticeHash,
EngineBuildHash = manifest.EngineBuildHash,
CanonicalManifest = canonical
});
return new ReplayResult(
exec.FindingsHash == manifest.FindingsSHA256,
exec.VexBundleHash == manifest.VexBundleSHA256,
exec.ProofBundleHash == manifest.ProofBundleSHA256,
exec
);
}
}
```
## 13. BENCHMARK METRICS
### 13.1 Time-to-Evidence (TTE)
**Definition:**
```
TTE = t(proof_ready) – t(artifact_ingested)
```
**Targets:**
- P50 < 2m for typical containers (≤ 500 MB)
- P95 < 5m including cold-start/offline-bundle mode
**Stage Breakdown:**
- t_ingest_ms
- t_analyze_ms
- t_reachability_ms
- t_vex_ms
- t_sign_ms
- t_publish_ms
### 13.2 False-Negative Drift Rate (FN-DRIFT)
**Definition (rolling 30d window):**
```
FN-Drift = (# artifacts re-classified from {unaffected/unknown} → affected) / (total artifacts re-evaluated)
```
**Stratification:**
- feed delta
- rule delta
- lattice/policy delta
- reachability delta
**Targets:**
- Engine-caused FN-Drift ≈ 0
- Feed-caused FN-Drift: faster is better
### 13.3 Deterministic Reproducibility
**Proof Object:**
```json
{
"artifact_digest": "sha256:...",
"scan_manifest_hash": "sha256:...",
"feeds_merkle_root": "sha256:...",
"engine_build_hash": "sha256:...",
"policy_lattice_hash": "sha256:...",
"findings_sha256": "sha256:...",
"vex_bundle_sha256": "sha256:...",
"proof_bundle_sha256": "sha256:..."
}
```
**Metric:**
```
Repro rate = identical_outputs / total_replays
Target: 100%
```
### 13.4 Detection Metrics
```
true_positive_count (TP)
false_positive_count (FP)
false_negative_count (FN)
precision = TP / (TP + FP)
recall = TP / (TP + FN)
fp_reduction = (baseline_fp_rate - stella_fp_rate) / baseline_fp_rate
```
### 13.5 Proof Coverage
```
proof_coverage_all = findings_with_valid_receipts / total_findings
proof_coverage_vex = vex_items_with_valid_receipts / total_vex_items
proof_coverage_reachable = reachable_findings_with_proofs / total_reachable_findings
```
## 14. SLO THRESHOLDS
**Fidelity:**
- BF ≥ 0.98 (general)
- BF ≥ 0.95 (regulated projects)
- PF ≈ 1.0 (unless policy changed intentionally)
**Alerts:**
- BF drops ≥2% week-over-week → warn
- BF < 0.90 overall → page/block release
- Regulated BF < 0.95 → page/block release
## 15. DETERMINISTIC PACKAGING (BUNDLES)
Determinism applies to *packaging*, not only algorithms.
Rules for proof bundles and offline kits:
- Prefer `tar` with deterministic ordering; avoid formats that inject timestamps by default.
- Canonical file order: lexicographic path sort; include an `index.json` listing files and their digests in the same order.
- Normalize file metadata: fixed uid/gid, fixed mtime, stable permissions; record the chosen policy in the manifest.
- Compression must be reproducible (fixed level/settings; no embedded timestamps).
- Bundle hash is computed over the canonical archive bytes and must be DSSE-signed.
## 16. BENCHMARK HARNESS (MOAT METRICS)
Use the repo benchmark harness as the single place where moat metrics are measured and enforced:
- Harness root: `bench/README.md` (layout, verifiers, comparison tools).
- Evidence contracts: `docs/benchmarks/vex-evidence-playbook.md` and `docs/replay/DETERMINISTIC_REPLAY.md`.
Developer rules:
- No feature touching scans/policy/proofs ships without at least one benchmark scenario or an extension of an existing one.
- If golden outputs change intentionally, record a short “why” note (which metric improved, which contract changed) and keep artifacts deterministic.
- Bench runs must record and validate `graphRevisionId` and per-verdict receipts (see `docs/product-advisories/14-Dec-2025 - Proof and Evidence Chain Technical Reference.md`).
---
**Document Version**: 1.0
**Target Platform**: .NET 10, PostgreSQL ≥16, Angular v17