stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 12 Releases Wiki Activity
Files
9202cd7da8231f7593cb698c0f10992a01531fce
git.stella-ops.org/docs/product-advisories/14-Dec-2025 - Determinism and Reproducibility Technical Reference.md
StellaOps Bot 9202cd7da8 themed the bulk of advisories
2025-12-14 19:58:38 +02:00

14 KiB
Raw Blame History

Determinism and Reproducibility Technical Reference

Source Advisories:

  • 07-Dec-2025 - Designing Deterministic Vulnerability Scores
  • 12-Dec-2025 - Designing a Deterministic Vulnerability Scoring Matrix
  • 12-Dec-2025 - Replay Fidelity as a Proof Metric
  • 01-Dec-2025 - Benchmarks for a Testable Security Moat
  • 02-Dec-2025 - Benchmarking a Testable Security Moat

Last Updated: 2025-12-14

1. SCORE FORMULA (BASIS POINTS)

Total Score:

riskScore = (wB*B + wR*R + wE*E + wP*P) / 10000

Default Weights (basis points, sum = 10000):

  • wB=1000 (10%) - Base Severity
  • wR=4500 (45%) - Reachability
  • wE=3000 (30%) - Evidence
  • wP=1500 (15%) - Provenance

2. SUBSCORE DEFINITIONS (0-100 integers)

2.1 BaseSeverity (B)

B = round(CVSS * 10)  // CVSS 0.0-10.0 → 0-100

2.2 Reachability (R)

Hop Buckets:

0-2 hops: 100
3 hops: 85
4 hops: 70
5 hops: 55
6 hops: 45
7 hops: 35
8+ hops: 20
unreachable: 0

Gate Multipliers (in basis points):

behind feature flag: ×7000
auth required: ×8000
admin only: ×8500
non-default config: ×7500

Final R:

R = bucketScore * gateMultiplier / 10000

2.3 Evidence (E)

Points:

runtime trace: +60
DAST/integration test: +30
SAST precise sink: +20
SCA presence only: +10

Freshness Multiplier (basis points):

≤ 7 days: ×10000
≤ 30 days: ×9000
≤ 90 days: ×7500
≤ 180 days: ×6000
≤ 365 days: ×4000
> 365 days: ×2000

Final E:

E = min(100, sum(points)) * freshness / 10000

2.4 Provenance (P)

unsigned/unknown: 0
signed image: 30
signed + SBOM hash-linked: 60
signed + SBOM + DSSE attestations: 80
above + reproducible build match: 100

3. SCORE POLICY YAML SCHEMA

policyVersion: score.v1
weightsBps:
  baseSeverity: 1000
  reachability: 4500
  evidence: 3000
  provenance: 1500

reachability:
  hopBuckets:
    - { maxHops: 2, score: 100 }
    - { maxHops: 3, score: 85 }
    - { maxHops: 4, score: 70 }
    - { maxHops: 5, score: 55 }
    - { maxHops: 6, score: 45 }
    - { maxHops: 7, score: 35 }
    - { maxHops: 9999, score: 20 }
  unreachableScore: 0
  gateMultipliersBps:
    featureFlag: 7000
    authRequired: 8000
    adminOnly: 8500
    nonDefaultConfig: 7500

evidence:
  points:
    runtime: 60
    dast: 30
    sast: 20
    sca: 10
  freshnessBuckets:
    - { maxAgeDays: 7, multiplierBps: 10000 }
    - { maxAgeDays: 30, multiplierBps: 9000 }
    - { maxAgeDays: 90, multiplierBps: 7500 }
    - { maxAgeDays: 180, multiplierBps: 6000 }
    - { maxAgeDays: 365, multiplierBps: 4000 }
    - { maxAgeDays: 99999, multiplierBps: 2000 }

provenance:
  levels:
    unsigned: 0
    signed: 30
    signedWithSbom: 60
    signedWithSbomAndAttestations: 80
    reproducible: 100

overrides:
  - name: knownExploitedAndReachable
    when:
      flags:
        knownExploited: true
      minReachability: 70
    setScore: 95

  - name: unreachableAndOnlySca
    when:
      maxReachability: 0
      maxEvidence: 10
    clampMaxScore: 25

4. SCORE DATA CONTRACTS

4.1 ScoreInput

{
  "asOf": "2025-12-14T10:20:30Z",
  "policyVersion": "score.v1",
  "reachabilityDigest": "sha256:...",
  "evidenceDigest": "sha256:...",
  "provenanceDigest": "sha256:...",
  "baseSeverityDigest": "sha256:..."
}

4.2 ScoreResult

{
  "scoreId": "score_...",
  "riskScore": 73,
  "subscores": {
    "baseSeverity": 75,
    "reachability": 85,
    "evidence": 60,
    "provenance": 60
  },
  "cvss": {
    "v": "3.1",
    "base": 7.5,
    "environmental": 5.3,
    "vector": "CVSS:3.1/AV:N/AC:L/..."
  },
  "inputsRef": ["evidence_sha256:...", "env_sha256:..."],
  "policyVersion": "score.v1",
  "policyDigest": "sha256:...",
  "engineVersion": "stella-scorer@1.8.2",
  "computedAt": "2025-12-09T10:20:30Z",
  "resultDigest": "sha256:...",
  "explain": [
    {"factor": "reachability", "value": 85, "reason": "3 hops from HTTP endpoint"},
    {"factor": "evidence", "value": 60, "reason": "Runtime trace (60pts), 20 days old (×90%)"}
  ]
}

4.3 ReachabilityReport

{
  "artifactDigest": "sha256:...",
  "graphDigest": "sha256:...",
  "vulnId": "CVE-2024-1234",
  "vulnerableSymbol": "org.example.VulnClass.vulnMethod",
  "entrypoints": ["POST /api/upload"],
  "shortestPath": {
    "hops": 3,
    "nodes": [
      {"symbol": "UploadController.handleUpload", "file": "Controller.cs", "line": 42},
      {"symbol": "ProcessorService.process", "file": "Service.cs", "line": 18},
      {"symbol": "org.example.VulnClass.vulnMethod", "file": null, "line": null}
    ]
  },
  "gates": [
    {"type": "authRequired", "detail": "Requires JWT token"},
    {"type": "featureFlag", "detail": "FEATURE_UPLOAD_V2=true"}
  ],
  "computedAt": "2025-12-14T10:15:30Z",
  "toolVersion": "reachability-analyzer@2.1.0"
}

4.4 EvidenceBundle

{
  "evidenceId": "sha256:...",
  "artifactDigest": "sha256:...",
  "vulnId": "CVE-2024-1234",
  "type": "RUNTIME",
  "tool": "runtime-tracer@1.0.0",
  "timestamp": "2025-12-10T14:30:00Z",
  "confidence": 95,
  "subject": "org.example.VulnClass.vulnMethod",
  "payloadDigest": "sha256:..."
}

4.5 ProvenanceReport

{
  "artifactDigest": "sha256:...",
  "signatureChecks": [
    {"signer": "CI-KEY-1", "algorithm": "ECDSA-P256", "result": "VALID"}
  ],
  "sbomDigest": "sha256:...",
  "sbomType": "cyclonedx-1.6",
  "attestations": ["sha256:...", "sha256:..."],
  "transparencyLogRefs": ["rekor://..."],
  "reproducibleMatch": true,
  "computedAt": "2025-12-14T10:15:30Z",
  "toolVersion": "provenance-verifier@1.0.0"
}

5. DETERMINISM CONSTRAINTS

5.1 Fixed-Point Math

  • Use integer basis points (100% = 10,000 bps)
  • No floating point in scoring math
  • Round only at final display

5.2 Canonical Serialization

  • RFC-style canonical JSON (JCS)
  • Sort keys and arrays deterministically
  • Stable ordering for explanation lists by (factorId, contributingObjectDigest)

5.3 Time Handling

  • No implicit time
  • asOf is explicit input
  • Freshness = asOf - evidence.timestamp
  • Use monotonic time internally

6. FIDELITY METRICS

6.1 Bitwise Fidelity (BF)

BF = identical_outputs / total_replays
Target: ≥ 0.98

6.2 Semantic Fidelity (SF)

  • Normalized object comparison (same packages, versions, CVEs, severities, verdicts)
  • Allows formatting differences

6.3 Policy Fidelity (PF)

  • Final policy decision (pass/fail + reason codes) matches

7. SCAN MANIFEST SCHEMA

{
  "manifest_version": "1.0",
  "scan_id": "scan_123",
  "created_at": "2025-12-12T10:15:30Z",

  "input": {
    "type": "oci_image",
    "image_ref": "registry/app@sha256:...",
    "layers": ["sha256:...", "sha256:..."],
    "source_provenance": {"repo_sha": "abc123", "build_id": "ci-999"}
  },

  "scanner": {
    "engine": "stella",
    "scanner_image_digest": "sha256:...",
    "scanner_version": "2025.12.0",
    "config_digest": "sha256:...",
    "flags": ["--deep", "--vex"]
  },

  "feeds": {
    "vuln_feed_bundle_digest": "sha256:...",
    "license_db_digest": "sha256:..."
  },

  "policy": {
    "policy_bundle_digest": "sha256:...",
    "policy_set": "prod-default"
  },

  "environment": {
    "arch": "amd64",
    "os": "linux",
    "tz": "UTC",
    "locale": "C",
    "network": "disabled",
    "clock_mode": "frozen",
    "clock_value": "2025-12-12T10:15:30Z"
  },

  "normalization": {
    "canonicalizer_version": "1.2.0",
    "sbom_schema": "cyclonedx-1.6",
    "vex_schema": "cyclonedx-vex-1.0"
  }
}

8. MISMATCH CLASSIFICATION TAXONOMY

- Feed drift
- Policy drift
- Runtime drift
- Scanner drift
- Nondeterminism (ordering, concurrency, RNG, time-based logic)
- External IO

9. POSTGRESQL SCHEMA

CREATE TABLE scan_manifest (
    manifest_id UUID PRIMARY KEY,
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),

    artifact_digest TEXT NOT NULL,
    feeds_merkle_root TEXT NOT NULL,
    engine_build_hash TEXT NOT NULL,
    policy_lattice_hash TEXT NOT NULL,

    ruleset_hash TEXT NOT NULL,
    config_flags JSONB NOT NULL,

    environment_fingerprint JSONB NOT NULL,

    raw_manifest JSONB NOT NULL,
    raw_manifest_sha256 TEXT NOT NULL
);

CREATE TABLE scan_execution (
    execution_id UUID PRIMARY KEY,
    manifest_id UUID NOT NULL REFERENCES scan_manifest(manifest_id) ON DELETE CASCADE,

    started_at TIMESTAMPTZ NOT NULL,
    finished_at TIMESTAMPTZ NOT NULL,

    t_ingest_ms INT NOT NULL,
    t_analyze_ms INT NOT NULL,
    t_reachability_ms INT NOT NULL,
    t_vex_ms INT NOT NULL,
    t_sign_ms INT NOT NULL,
    t_publish_ms INT NOT NULL,

    proof_bundle_sha256 TEXT NOT NULL,
    findings_sha256 TEXT NOT NULL,
    vex_bundle_sha256 TEXT NOT NULL,

    replay_mode BOOLEAN NOT NULL DEFAULT FALSE
);

CREATE TABLE classification_history (
    id BIGSERIAL PRIMARY KEY,
    artifact_digest TEXT NOT NULL,
    manifest_id UUID NOT NULL REFERENCES scan_manifest(manifest_id) ON DELETE CASCADE,
    execution_id UUID NOT NULL REFERENCES scan_execution(execution_id) ON DELETE CASCADE,

    previous_status TEXT NOT NULL,
    new_status TEXT NOT NULL,
    cause TEXT NOT NULL,

    changed_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

CREATE VIEW scan_tte AS
SELECT
    execution_id,
    manifest_id,
    (finished_at - started_at) AS tte_interval
FROM scan_execution;

CREATE MATERIALIZED VIEW fn_drift_stats AS
SELECT
    date_trunc('day', changed_at) AS day_bucket,
    COUNT(*) FILTER (WHERE new_status = 'affected') AS affected_count,
    COUNT(*) AS total_reclassified,
    ROUND(
        (COUNT(*) FILTER (WHERE new_status = 'affected')::numeric /
         NULLIF(COUNT(*), 0)) * 100, 4
    ) AS drift_percent
FROM classification_history
GROUP BY 1;

10. C# CANONICAL DATA STRUCTURES

public sealed record CanonicalScanManifest
{
    public required string ArtifactDigest { get; init; }
    public required string FeedsMerkleRoot { get; init; }
    public required string EngineBuildHash { get; init; }
    public required string PolicyLatticeHash { get; init; }
    public required string RulesetHash { get; init; }

    public required IReadOnlyDictionary<string, string> ConfigFlags { get; init; }
    public required EnvironmentFingerprint Environment { get; init; }
}

public sealed record EnvironmentFingerprint
{
    public required string CpuModel { get; init; }
    public required string RuntimeVersion { get; init; }
    public required string Os { get; init; }
    public required IReadOnlyDictionary<string, string> Extra { get; init; }
}

public sealed record ScanExecutionMetrics
{
    public required int IngestMs { get; init; }
    public required int AnalyzeMs { get; init; }
    public required int ReachabilityMs { get; init; }
    public required int VexMs { get; init; }
    public required int SignMs { get; init; }
    public required int PublishMs { get; init; }
}

11. CANONICALIZATION IMPLEMENTATION

internal static class CanonicalJson
{
    private static readonly JsonSerializerOptions Options = new()
    {
        WriteIndented = false,
        PropertyNamingPolicy = JsonNamingPolicy.CamelCase
    };

    public static string Serialize(object obj)
    {
        using var stream = new MemoryStream();
        using (var writer = new Utf8JsonWriter(stream, new JsonWriterOptions
            {
                Indented = false,
                SkipValidation = false
            }))
        {
            JsonSerializer.Serialize(writer, obj, obj.GetType(), Options);
        }

        var bytes = stream.ToArray();
        var canonical = JsonCanonicalizer.Canonicalize(bytes);

        return canonical;
    }
}

12. REPLAY RUNNER

public static class ReplayRunner
{
    public static ReplayResult Replay(Guid manifestId, IScannerEngine engine)
    {
        var manifest = ManifestRepository.Load(manifestId);
        var canonical = CanonicalJson.Serialize(manifest.RawObject);
        var canonicalHash = Sha256(canonical);

        if (canonicalHash != manifest.RawManifestSHA256)
            throw new InvalidOperationException("Manifest integrity violation.");

        using var feeds = FeedSnapshotResolver.Open(manifest.FeedsMerkleRoot);

        var exec = engine.Scan(new ScanRequest
        {
            ArtifactDigest = manifest.ArtifactDigest,
            Feeds = feeds,
            LatticeHash = manifest.PolicyLatticeHash,
            EngineBuildHash = manifest.EngineBuildHash,
            CanonicalManifest = canonical
        });

        return new ReplayResult(
            exec.FindingsHash == manifest.FindingsSHA256,
            exec.VexBundleHash == manifest.VexBundleSHA256,
            exec.ProofBundleHash == manifest.ProofBundleSHA256,
            exec
        );
    }
}

13. BENCHMARK METRICS

13.1 Time-to-Evidence (TTE)

Definition:

TTE = t(proof_ready) – t(artifact_ingested)

Targets:

  • P50 < 2m for typical containers (≤ 500 MB)
  • P95 < 5m including cold-start/offline-bundle mode

Stage Breakdown:

  • t_ingest_ms
  • t_analyze_ms
  • t_reachability_ms
  • t_vex_ms
  • t_sign_ms
  • t_publish_ms

13.2 False-Negative Drift Rate (FN-DRIFT)

Definition (rolling 30d window):

FN-Drift = (# artifacts re-classified from {unaffected/unknown} → affected) / (total artifacts re-evaluated)

Stratification:

  • feed delta
  • rule delta
  • lattice/policy delta
  • reachability delta

Targets:

  • Engine-caused FN-Drift ≈ 0
  • Feed-caused FN-Drift: faster is better

13.3 Deterministic Reproducibility

Proof Object:

{
  "artifact_digest": "sha256:...",
  "scan_manifest_hash": "sha256:...",
  "feeds_merkle_root": "sha256:...",
  "engine_build_hash": "sha256:...",
  "policy_lattice_hash": "sha256:...",
  "findings_sha256": "sha256:...",
  "vex_bundle_sha256": "sha256:...",
  "proof_bundle_sha256": "sha256:..."
}

Metric:

Repro rate = identical_outputs / total_replays
Target: 100%

13.4 Detection Metrics

true_positive_count (TP)
false_positive_count (FP)
false_negative_count (FN)

precision = TP / (TP + FP)
recall = TP / (TP + FN)

fp_reduction = (baseline_fp_rate - stella_fp_rate) / baseline_fp_rate

13.5 Proof Coverage

proof_coverage_all = findings_with_valid_receipts / total_findings

proof_coverage_vex = vex_items_with_valid_receipts / total_vex_items

proof_coverage_reachable = reachable_findings_with_proofs / total_reachable_findings

14. SLO THRESHOLDS

Fidelity:

  • BF ≥ 0.98 (general)
  • BF ≥ 0.95 (regulated projects)
  • PF ≈ 1.0 (unless policy changed intentionally)

Alerts:

  • BF drops ≥2% week-over-week → warn
  • BF < 0.90 overall → page/block release
  • Regulated BF < 0.95 → page/block release

Document Version: 1.0 Target Platform: .NET 10, PostgreSQL ≥16, Angular v17