up
Some checks failed
Concelier Attestation Tests / attestation-tests (push) Has been cancelled
Policy Simulation / policy-simulate (push) Has been cancelled
AOC Guard CI / aoc-guard (push) Has been cancelled
AOC Guard CI / aoc-verify (push) Has been cancelled
Signals CI & Image / signals-ci (push) Has been cancelled
Signals Reachability Scoring & Events / reachability-smoke (push) Has been cancelled
Signals Reachability Scoring & Events / sign-and-upload (push) Has been cancelled
Docs CI / lint-and-preview (push) Has been cancelled
Policy Lint & Smoke / policy-lint (push) Has been cancelled
Scanner Analyzers / Discover Analyzers (push) Has been cancelled
Scanner Analyzers / Build Analyzers (push) Has been cancelled
Scanner Analyzers / Test Language Analyzers (push) Has been cancelled
Scanner Analyzers / Validate Test Fixtures (push) Has been cancelled
Scanner Analyzers / Verify Deterministic Output (push) Has been cancelled
Some checks failed
Concelier Attestation Tests / attestation-tests (push) Has been cancelled
Policy Simulation / policy-simulate (push) Has been cancelled
AOC Guard CI / aoc-guard (push) Has been cancelled
AOC Guard CI / aoc-verify (push) Has been cancelled
Signals CI & Image / signals-ci (push) Has been cancelled
Signals Reachability Scoring & Events / reachability-smoke (push) Has been cancelled
Signals Reachability Scoring & Events / sign-and-upload (push) Has been cancelled
Docs CI / lint-and-preview (push) Has been cancelled
Policy Lint & Smoke / policy-lint (push) Has been cancelled
Scanner Analyzers / Discover Analyzers (push) Has been cancelled
Scanner Analyzers / Build Analyzers (push) Has been cancelled
Scanner Analyzers / Test Language Analyzers (push) Has been cancelled
Scanner Analyzers / Validate Test Fixtures (push) Has been cancelled
Scanner Analyzers / Verify Deterministic Output (push) Has been cancelled
This commit is contained in:
@@ -8,9 +8,16 @@ The bench harness exercises the language analyzers against representative filesy
|
||||
- `baseline.csv` – Reference numbers captured on the 4 vCPU warm rig described in `docs/12_PERFORMANCE_WORKBOOK.md`. CI publishes fresh CSVs so perf trends stay visible.
|
||||
|
||||
## Current scenarios
|
||||
- `node_detection_gaps_fixture` - runs the Node analyzer across `samples/runtime/node-detection-gaps` (workspaces + lock-only + import scan).
|
||||
- `node_monorepo_walk` → runs the Node analyzer across `samples/runtime/npm-monorepo`.
|
||||
- `java_demo_archive` → runs the Java analyzer against `samples/runtime/java-demo/libs/demo.jar`.
|
||||
- `python_site_packages_walk` → temporary metadata walk over `samples/runtime/python-venv` until the Python analyzer lands.
|
||||
- `python_site_packages_scan` → runs the Python analyzer across `samples/runtime/python-venv`.
|
||||
- `python_pip_cache_fixture` → runs the Python analyzer across the RECORD-heavy pip cache fixture.
|
||||
- `python_layered_editable_fixture` → runs the Python analyzer across layered/container-root layouts (`layers/`, `.layers/`, `layer*`).
|
||||
|
||||
- `bun_multi_workspace_fixture` - runs the Bun analyzer across the Bun multi-workspace fixture under the Bun analyzer tests.
|
||||
|
||||
See `config.json` for the authoritative list.
|
||||
|
||||
## Running locally
|
||||
|
||||
|
||||
@@ -0,0 +1,268 @@
|
||||
using System.Collections.Generic;
|
||||
using System.Linq;
|
||||
using StellaOps.Scanner.Analyzers.Lang;
|
||||
|
||||
namespace StellaOps.Bench.ScannerAnalyzers.Scenarios;
|
||||
|
||||
internal static class NodeBenchMetrics
|
||||
{
|
||||
private static readonly HashSet<string> Extensions = new(StringComparer.OrdinalIgnoreCase)
|
||||
{
|
||||
".js",
|
||||
".jsx",
|
||||
".mjs",
|
||||
".cjs",
|
||||
".ts",
|
||||
".tsx",
|
||||
".mts",
|
||||
".cts"
|
||||
};
|
||||
|
||||
private static readonly string[] IgnoredDirectories =
|
||||
{
|
||||
".bin",
|
||||
".cache",
|
||||
".store",
|
||||
"__pycache__"
|
||||
};
|
||||
|
||||
public static IReadOnlyDictionary<string, double> Compute(string rootPath, LanguageAnalyzerResult result)
|
||||
{
|
||||
ArgumentException.ThrowIfNullOrWhiteSpace(rootPath);
|
||||
ArgumentNullException.ThrowIfNull(result);
|
||||
|
||||
var scanRoots = CollectImportScanRoots(rootPath, result);
|
||||
var packagesScanned = 0;
|
||||
var filesScanned = 0;
|
||||
long bytesScanned = 0;
|
||||
var cappedPackages = 0;
|
||||
|
||||
foreach (var scanRoot in scanRoots)
|
||||
{
|
||||
var counters = CountImportScan(scanRoot);
|
||||
packagesScanned++;
|
||||
filesScanned += counters.FilesScanned;
|
||||
bytesScanned += counters.BytesScanned;
|
||||
|
||||
if (counters.Capped)
|
||||
{
|
||||
cappedPackages++;
|
||||
}
|
||||
}
|
||||
|
||||
return new SortedDictionary<string, double>(StringComparer.Ordinal)
|
||||
{
|
||||
["node.importScan.packages"] = packagesScanned,
|
||||
["node.importScan.filesScanned"] = filesScanned,
|
||||
["node.importScan.bytesScanned"] = bytesScanned,
|
||||
["node.importScan.cappedPackages"] = cappedPackages
|
||||
};
|
||||
}
|
||||
|
||||
public static bool AreEqual(IReadOnlyDictionary<string, double> left, IReadOnlyDictionary<string, double> right)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(left);
|
||||
ArgumentNullException.ThrowIfNull(right);
|
||||
|
||||
if (ReferenceEquals(left, right))
|
||||
{
|
||||
return true;
|
||||
}
|
||||
|
||||
if (left.Count != right.Count)
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
foreach (var (key, value) in left)
|
||||
{
|
||||
if (!right.TryGetValue(key, out var other) || other != value)
|
||||
{
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
private static IReadOnlyList<string> CollectImportScanRoots(string rootPath, LanguageAnalyzerResult result)
|
||||
{
|
||||
var comparer = OperatingSystem.IsWindows() ? StringComparer.OrdinalIgnoreCase : StringComparer.Ordinal;
|
||||
var roots = new HashSet<string>(comparer);
|
||||
|
||||
var fullRoot = Path.GetFullPath(rootPath);
|
||||
|
||||
foreach (var record in result.Components)
|
||||
{
|
||||
if (!string.Equals(record.AnalyzerId, "node", StringComparison.Ordinal))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
if (!record.Metadata.TryGetValue("path", out var relativePath) || string.IsNullOrWhiteSpace(relativePath))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
var isRoot = string.Equals(relativePath, ".", StringComparison.Ordinal);
|
||||
var isWorkspaceMember = record.Metadata.TryGetValue("workspaceMember", out var workspaceMember)
|
||||
&& string.Equals(workspaceMember, "true", StringComparison.OrdinalIgnoreCase);
|
||||
|
||||
if (!isRoot && !isWorkspaceMember)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
var absolute = isRoot
|
||||
? fullRoot
|
||||
: Path.GetFullPath(Path.Combine(fullRoot, relativePath.Replace('/', Path.DirectorySeparatorChar)));
|
||||
|
||||
if (Directory.Exists(absolute))
|
||||
{
|
||||
roots.Add(absolute);
|
||||
}
|
||||
}
|
||||
|
||||
return roots.OrderBy(static p => p, StringComparer.Ordinal).ToArray();
|
||||
}
|
||||
|
||||
private static ImportScanCounters CountImportScan(string rootPath)
|
||||
{
|
||||
const int maxFilesPerPackage = 500;
|
||||
const long maxBytesPerPackage = 5L * 1024 * 1024;
|
||||
const long maxFileBytes = 512L * 1024;
|
||||
const int maxDepth = 20;
|
||||
|
||||
var filesScanned = 0;
|
||||
long bytesScanned = 0;
|
||||
var capped = false;
|
||||
|
||||
foreach (var file in EnumerateSourceFiles(rootPath, maxDepth))
|
||||
{
|
||||
if (filesScanned >= maxFilesPerPackage || bytesScanned >= maxBytesPerPackage)
|
||||
{
|
||||
capped = true;
|
||||
break;
|
||||
}
|
||||
|
||||
long length;
|
||||
try
|
||||
{
|
||||
length = new FileInfo(file).Length;
|
||||
}
|
||||
catch
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
if (length <= 0 || length > maxFileBytes)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
if (bytesScanned + length > maxBytesPerPackage)
|
||||
{
|
||||
capped = true;
|
||||
break;
|
||||
}
|
||||
|
||||
bytesScanned += length;
|
||||
filesScanned++;
|
||||
}
|
||||
|
||||
return new ImportScanCounters(filesScanned, bytesScanned, capped);
|
||||
}
|
||||
|
||||
private static IEnumerable<string> EnumerateSourceFiles(string root, int maxDepth)
|
||||
{
|
||||
var pathComparer = OperatingSystem.IsWindows() ? StringComparer.OrdinalIgnoreCase : StringComparer.Ordinal;
|
||||
var stack = new Stack<(string Path, int Depth)>();
|
||||
stack.Push((root, 0));
|
||||
|
||||
while (stack.Count > 0)
|
||||
{
|
||||
var (current, depth) = stack.Pop();
|
||||
|
||||
IEnumerable<string> files;
|
||||
try
|
||||
{
|
||||
files = Directory.EnumerateFiles(current, "*", SearchOption.TopDirectoryOnly);
|
||||
}
|
||||
catch
|
||||
{
|
||||
files = Array.Empty<string>();
|
||||
}
|
||||
|
||||
foreach (var file in files.OrderBy(static f => f, pathComparer))
|
||||
{
|
||||
var ext = Path.GetExtension(file);
|
||||
if (!string.IsNullOrWhiteSpace(ext) && Extensions.Contains(ext))
|
||||
{
|
||||
yield return file;
|
||||
}
|
||||
}
|
||||
|
||||
if (depth >= maxDepth)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
IEnumerable<string> dirs;
|
||||
try
|
||||
{
|
||||
dirs = Directory.EnumerateDirectories(current, "*", SearchOption.TopDirectoryOnly);
|
||||
}
|
||||
catch
|
||||
{
|
||||
dirs = Array.Empty<string>();
|
||||
}
|
||||
|
||||
var ordered = dirs
|
||||
.Where(static d => !ShouldSkipImportDirectory(Path.GetFileName(d)))
|
||||
.OrderBy(static d => d, pathComparer)
|
||||
.ToArray();
|
||||
|
||||
for (var i = ordered.Length - 1; i >= 0; i--)
|
||||
{
|
||||
stack.Push((ordered[i], depth + 1));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private static bool ShouldSkipImportDirectory(string? name)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(name))
|
||||
{
|
||||
return true;
|
||||
}
|
||||
|
||||
if (string.Equals(name, "node_modules", StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
return true;
|
||||
}
|
||||
|
||||
if (string.Equals(name, ".pnpm", StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
return true;
|
||||
}
|
||||
|
||||
return ShouldSkipDirectory(name);
|
||||
}
|
||||
|
||||
private static bool ShouldSkipDirectory(string name)
|
||||
{
|
||||
if (name.Length == 0)
|
||||
{
|
||||
return true;
|
||||
}
|
||||
|
||||
if (name[0] == '.')
|
||||
{
|
||||
return !string.Equals(name, ".pnpm", StringComparison.OrdinalIgnoreCase);
|
||||
}
|
||||
|
||||
return IgnoredDirectories.Any(ignored => string.Equals(name, ignored, StringComparison.OrdinalIgnoreCase));
|
||||
}
|
||||
|
||||
private readonly record struct ImportScanCounters(int FilesScanned, long BytesScanned, bool Capped);
|
||||
}
|
||||
@@ -43,7 +43,10 @@ internal static class Program
|
||||
stats.P95Ms,
|
||||
stats.MaxMs,
|
||||
iterations,
|
||||
scenarioThreshold);
|
||||
scenarioThreshold)
|
||||
{
|
||||
Metrics = execution.Metrics
|
||||
};
|
||||
|
||||
results.Add(result);
|
||||
|
||||
@@ -101,7 +104,7 @@ internal static class Program
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
Console.Error.WriteLine(ex.Message);
|
||||
Console.Error.WriteLine(ex);
|
||||
return 1;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -53,6 +53,7 @@ internal static class BenchmarkJsonWriter
|
||||
report.Result.P95Ms,
|
||||
report.Result.MaxMs,
|
||||
report.Result.ThresholdMs,
|
||||
report.Result.Metrics,
|
||||
baseline is null
|
||||
? null
|
||||
: new BenchmarkJsonScenarioBaseline(
|
||||
@@ -84,6 +85,7 @@ internal static class BenchmarkJsonWriter
|
||||
double P95Ms,
|
||||
double MaxMs,
|
||||
double ThresholdMs,
|
||||
IReadOnlyDictionary<string, double>? Metrics,
|
||||
BenchmarkJsonScenarioBaseline? Baseline,
|
||||
BenchmarkJsonScenarioRegression Regression);
|
||||
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
using System.Globalization;
|
||||
using StellaOps.Bench.ScannerAnalyzers.Baseline;
|
||||
|
||||
namespace StellaOps.Bench.ScannerAnalyzers.Reporting;
|
||||
@@ -35,7 +36,13 @@ internal sealed class BenchmarkScenarioReport
|
||||
}
|
||||
|
||||
var percentage = (MaxRegressionRatio.Value - 1d) * 100d;
|
||||
return $"{Result.Id} exceeded regression budget: max {Result.MaxMs:F2} ms vs baseline {Baseline!.MaxMs:F2} ms (+{percentage:F1}%)";
|
||||
return string.Format(
|
||||
CultureInfo.InvariantCulture,
|
||||
"{0} exceeded regression budget: max {1:F2} ms vs baseline {2:F2} ms (+{3:F1}%)",
|
||||
Result.Id,
|
||||
Result.MaxMs,
|
||||
Baseline!.MaxMs,
|
||||
percentage);
|
||||
}
|
||||
|
||||
private static double? CalculateRatio(double current, double? baseline)
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
using System.Globalization;
|
||||
using System.Linq;
|
||||
using System.Text;
|
||||
|
||||
namespace StellaOps.Bench.ScannerAnalyzers.Reporting;
|
||||
@@ -20,6 +21,10 @@ internal static class PrometheusWriter
|
||||
var builder = new StringBuilder();
|
||||
builder.AppendLine("# HELP scanner_analyzer_bench_duration_ms Analyzer benchmark duration metrics in milliseconds.");
|
||||
builder.AppendLine("# TYPE scanner_analyzer_bench_duration_ms gauge");
|
||||
builder.AppendLine("# HELP scanner_analyzer_bench_sample_count Analyzer benchmark sample counts (component/file counts).");
|
||||
builder.AppendLine("# TYPE scanner_analyzer_bench_sample_count gauge");
|
||||
builder.AppendLine("# HELP scanner_analyzer_bench_metric Additional analyzer benchmark metrics.");
|
||||
builder.AppendLine("# TYPE scanner_analyzer_bench_metric gauge");
|
||||
|
||||
foreach (var report in reports)
|
||||
{
|
||||
@@ -28,6 +33,7 @@ internal static class PrometheusWriter
|
||||
AppendMetric(builder, "scanner_analyzer_bench_p95_ms", scenarioLabel, report.Result.P95Ms);
|
||||
AppendMetric(builder, "scanner_analyzer_bench_max_ms", scenarioLabel, report.Result.MaxMs);
|
||||
AppendMetric(builder, "scanner_analyzer_bench_threshold_ms", scenarioLabel, report.Result.ThresholdMs);
|
||||
AppendMetric(builder, "scanner_analyzer_bench_sample_count", scenarioLabel, report.Result.SampleCount);
|
||||
|
||||
if (report.Baseline is { } baseline)
|
||||
{
|
||||
@@ -41,6 +47,19 @@ internal static class PrometheusWriter
|
||||
AppendMetric(builder, "scanner_analyzer_bench_regression_limit", scenarioLabel, report.RegressionLimit);
|
||||
AppendMetric(builder, "scanner_analyzer_bench_regression_breached", scenarioLabel, report.RegressionBreached ? 1 : 0);
|
||||
}
|
||||
|
||||
if (report.Result.Metrics is { Count: > 0 } metrics)
|
||||
{
|
||||
foreach (var metric in metrics.OrderBy(static item => item.Key, StringComparer.Ordinal))
|
||||
{
|
||||
builder.Append("scanner_analyzer_bench_metric{scenario=\"");
|
||||
builder.Append(scenarioLabel);
|
||||
builder.Append("\",name=\"");
|
||||
builder.Append(Escape(metric.Key));
|
||||
builder.Append("\"} ");
|
||||
builder.AppendLine(metric.Value.ToString("G17", CultureInfo.InvariantCulture));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
File.WriteAllText(resolved, builder.ToString(), Encoding.UTF8);
|
||||
|
||||
@@ -12,6 +12,8 @@ internal sealed record ScenarioResult(
|
||||
int Iterations,
|
||||
double ThresholdMs)
|
||||
{
|
||||
public IReadOnlyDictionary<string, double>? Metrics { get; init; }
|
||||
|
||||
public string IdColumn => Id.Length <= 28 ? Id.PadRight(28) : Id[..28];
|
||||
|
||||
public string SampleCountColumn => SampleCount.ToString(CultureInfo.InvariantCulture).PadLeft(5);
|
||||
|
||||
@@ -4,6 +4,7 @@ using System.Linq;
|
||||
using System.Text.Json;
|
||||
using System.Text.RegularExpressions;
|
||||
using StellaOps.Scanner.Analyzers.Lang;
|
||||
using StellaOps.Scanner.Analyzers.Lang.Bun;
|
||||
using StellaOps.Scanner.Analyzers.Lang.Go;
|
||||
using StellaOps.Scanner.Analyzers.Lang.Java;
|
||||
using StellaOps.Scanner.Analyzers.Lang.Node;
|
||||
@@ -17,7 +18,7 @@ internal interface IScenarioRunner
|
||||
Task<ScenarioExecutionResult> ExecuteAsync(string rootPath, int iterations, CancellationToken cancellationToken);
|
||||
}
|
||||
|
||||
internal sealed record ScenarioExecutionResult(double[] Durations, int SampleCount);
|
||||
internal sealed record ScenarioExecutionResult(double[] Durations, int SampleCount, IReadOnlyDictionary<string, double>? Metrics = null);
|
||||
|
||||
internal static class ScenarioRunnerFactory
|
||||
{
|
||||
@@ -40,6 +41,7 @@ internal static class ScenarioRunnerFactory
|
||||
internal sealed class LanguageAnalyzerScenarioRunner : IScenarioRunner
|
||||
{
|
||||
private readonly IReadOnlyList<Func<ILanguageAnalyzer>> _analyzerFactories;
|
||||
private readonly bool _includesNodeAnalyzer;
|
||||
|
||||
public LanguageAnalyzerScenarioRunner(IEnumerable<string> analyzerIds)
|
||||
{
|
||||
@@ -48,11 +50,15 @@ internal sealed class LanguageAnalyzerScenarioRunner : IScenarioRunner
|
||||
throw new ArgumentNullException(nameof(analyzerIds));
|
||||
}
|
||||
|
||||
_analyzerFactories = analyzerIds
|
||||
var normalizedIds = analyzerIds
|
||||
.Where(static id => !string.IsNullOrWhiteSpace(id))
|
||||
.Select(CreateFactory)
|
||||
.Select(static id => id.Trim().ToLowerInvariant())
|
||||
.ToArray();
|
||||
|
||||
_includesNodeAnalyzer = normalizedIds.Contains("node", StringComparer.Ordinal);
|
||||
|
||||
_analyzerFactories = normalizedIds.Select(CreateFactory).ToArray();
|
||||
|
||||
if (_analyzerFactories.Count == 0)
|
||||
{
|
||||
throw new InvalidOperationException("At least one analyzer id must be provided.");
|
||||
@@ -70,6 +76,7 @@ internal sealed class LanguageAnalyzerScenarioRunner : IScenarioRunner
|
||||
var engine = new LanguageAnalyzerEngine(analyzers);
|
||||
var durations = new double[iterations];
|
||||
var componentCount = -1;
|
||||
IReadOnlyDictionary<string, double>? metrics = null;
|
||||
|
||||
for (var i = 0; i < iterations; i++)
|
||||
{
|
||||
@@ -91,6 +98,19 @@ internal sealed class LanguageAnalyzerScenarioRunner : IScenarioRunner
|
||||
{
|
||||
throw new InvalidOperationException($"Analyzer output count changed between iterations ({componentCount} vs {currentCount}).");
|
||||
}
|
||||
|
||||
if (_includesNodeAnalyzer)
|
||||
{
|
||||
var currentMetrics = NodeBenchMetrics.Compute(rootPath, result);
|
||||
if (metrics is null)
|
||||
{
|
||||
metrics = currentMetrics;
|
||||
}
|
||||
else if (!NodeBenchMetrics.AreEqual(metrics, currentMetrics))
|
||||
{
|
||||
throw new InvalidOperationException($"Analyzer metrics changed between iterations for '{rootPath}'.");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (componentCount < 0)
|
||||
@@ -98,7 +118,7 @@ internal sealed class LanguageAnalyzerScenarioRunner : IScenarioRunner
|
||||
componentCount = 0;
|
||||
}
|
||||
|
||||
return new ScenarioExecutionResult(durations, componentCount);
|
||||
return new ScenarioExecutionResult(durations, componentCount, metrics);
|
||||
}
|
||||
|
||||
private static Func<ILanguageAnalyzer> CreateFactory(string analyzerId)
|
||||
@@ -106,6 +126,7 @@ internal sealed class LanguageAnalyzerScenarioRunner : IScenarioRunner
|
||||
var id = analyzerId.Trim().ToLowerInvariant();
|
||||
return id switch
|
||||
{
|
||||
"bun" => static () => new BunLanguageAnalyzer(),
|
||||
"java" => static () => new JavaLanguageAnalyzer(),
|
||||
"go" => static () => new GoLanguageAnalyzer(),
|
||||
"node" => static () => new NodeLanguageAnalyzer(),
|
||||
|
||||
@@ -11,6 +11,7 @@
|
||||
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="../../../../Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang/StellaOps.Scanner.Analyzers.Lang.csproj" />
|
||||
<ProjectReference Include="../../../../Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Bun/StellaOps.Scanner.Analyzers.Lang.Bun.csproj" />
|
||||
<ProjectReference Include="../../../../Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Go/StellaOps.Scanner.Analyzers.Lang.Go.csproj" />
|
||||
<ProjectReference Include="../../../../Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Node/StellaOps.Scanner.Analyzers.Lang.Node.csproj" />
|
||||
<ProjectReference Include="../../../../Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Java/StellaOps.Scanner.Analyzers.Lang.Java.csproj" />
|
||||
@@ -21,4 +22,4 @@
|
||||
<ItemGroup>
|
||||
<InternalsVisibleTo Include="StellaOps.Bench.ScannerAnalyzers.Tests" />
|
||||
</ItemGroup>
|
||||
</Project>
|
||||
</Project>
|
||||
|
||||
@@ -1,7 +1,11 @@
|
||||
scenario,iterations,sample_count,mean_ms,p95_ms,max_ms
|
||||
node_monorepo_walk,5,4,6.0975,21.7421,26.8537
|
||||
java_demo_archive,5,1,6.2007,23.4837,29.1143
|
||||
go_buildinfo_fixture,5,2,6.1949,22.6851,27.9196
|
||||
dotnet_multirid_fixture,5,2,11.4884,37.7460,46.4850
|
||||
python_site_packages_scan,5,3,5.6420,18.2943,22.3739
|
||||
python_pip_cache_fixture,5,1,5.8598,13.2855,15.6256
|
||||
node_monorepo_walk,5,4,15.5399,50.3210,61.7146
|
||||
node_detection_gaps_fixture,5,5,31.8434,96.4542,117.3238
|
||||
java_demo_archive,5,1,13.6363,49.4627,61.3100
|
||||
java_fat_archive,5,2,3.5181,8.1467,9.4927
|
||||
go_buildinfo_fixture,5,2,6.9861,25.8818,32.1304
|
||||
dotnet_multirid_fixture,5,2,11.8266,38.9340,47.8401
|
||||
python_site_packages_scan,5,3,36.7930,105.6978,128.4211
|
||||
python_pip_cache_fixture,5,1,20.1829,30.9147,34.3257
|
||||
python_layered_editable_fixture,5,3,31.8757,39.7647,41.5656
|
||||
bun_multi_workspace_fixture,5,2,12.4463,45.1913,55.9832
|
||||
|
||||
|
@@ -10,6 +10,15 @@
|
||||
"node"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "node_detection_gaps_fixture",
|
||||
"label": "Node analyzer detection gaps fixture (workspace + lock-only + imports)",
|
||||
"root": "samples/runtime/node-detection-gaps",
|
||||
"analyzers": [
|
||||
"node"
|
||||
],
|
||||
"thresholdMs": 2000
|
||||
},
|
||||
{
|
||||
"id": "java_demo_archive",
|
||||
"label": "Java analyzer on demo jar",
|
||||
@@ -18,6 +27,15 @@
|
||||
"java"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "java_fat_archive",
|
||||
"label": "Java analyzer on fat jar (embedded libs)",
|
||||
"root": "samples/runtime/java-fat-archive",
|
||||
"analyzers": [
|
||||
"java"
|
||||
],
|
||||
"thresholdMs": 1000
|
||||
},
|
||||
{
|
||||
"id": "go_buildinfo_fixture",
|
||||
"label": "Go analyzer on build-info binary",
|
||||
@@ -49,6 +67,24 @@
|
||||
"analyzers": [
|
||||
"python"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "python_layered_editable_fixture",
|
||||
"label": "Python analyzer on layered/container roots fixture",
|
||||
"root": "src/Scanner/__Tests/StellaOps.Scanner.Analyzers.Lang.Python.Tests/Fixtures/lang/python/layered-editable",
|
||||
"analyzers": [
|
||||
"python"
|
||||
],
|
||||
"thresholdMs": 2000
|
||||
},
|
||||
{
|
||||
"id": "bun_multi_workspace_fixture",
|
||||
"label": "Bun analyzer on multi-workspace fixture",
|
||||
"root": "src/Scanner/__Tests/StellaOps.Scanner.Analyzers.Lang.Bun.Tests/Fixtures/lang/bun/multi-workspace",
|
||||
"analyzers": [
|
||||
"bun"
|
||||
],
|
||||
"thresholdMs": 1000
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
@@ -29,3 +29,5 @@ Results should be committed as deterministic CSV/JSON outputs with accompanying
|
||||
- Added two Python scenarios to `config.json`: the virtualenv sample (`python_site_packages_scan`) and the RECORD-heavy pip cache fixture (`python_pip_cache_fixture`).
|
||||
- Baseline run (Release build, 5 iterations) records means of **5.64 ms** (p95 18.29 ms) for the virtualenv and **5.86 ms** (p95 13.29 ms) for the pip cache verifier; raw numbers stored in `python/hash-throughput-20251023.csv`.
|
||||
- The pip cache fixture exercises `PythonRecordVerifier` with 12 RECORD rows (7 hashed) and mismatched layer coverage, giving a repeatable hash-validation throughput reference for regression gating.
|
||||
- 2025-12-13: Added `python_layered_editable_fixture` scenario with `thresholdMs=2000` to guard container-root paths.
|
||||
- 2025-12-13: Refreshed `baseline.csv` after Python analyzer discovery/VFS changes (see `src/Bench/StellaOps.Bench/Scanner.Analyzers/baseline.csv`).
|
||||
|
||||
@@ -9,3 +9,5 @@
|
||||
| BENCH-POLICY-20-002 | DONE (2025-12-11) | SPRINT_0512_0001_0001_bench | Policy delta benchmark (full vs delta) using baseline/delta NDJSON fixtures; outputs hashed. | `src/Bench/StellaOps.Bench/PolicyDelta` |
|
||||
| BENCH-SIG-26-001 | DONE (2025-12-11) | SPRINT_0512_0001_0001_bench | Reachability scoring harness with schema hash, 10k/50k fixtures, cache outputs for downstream benches. | `src/Bench/StellaOps.Bench/Signals` |
|
||||
| BENCH-SIG-26-002 | DONE (2025-12-11) | SPRINT_0512_0001_0001_bench | Policy evaluation cache bench (cold/warm/mixed) consuming reachability caches; outputs hashed. | `src/Bench/StellaOps.Bench/PolicyCache` |
|
||||
| BENCH-SCANNER-ANALYZERS-405-008 | DONE (2025-12-13) | SPRINT_0405_0001_0001_scanner_python_detection_gaps.md | Extend Scanner analyzer microbench coverage for the Python analyzer (fixtures + thresholds + docs alignment). | `src/Bench/StellaOps.Bench/Scanner.Analyzers` |
|
||||
| BENCH-SCANNER-ANALYZERS-407-009 | DONE (2025-12-13) | SPRINT_0407_0001_0001_scanner_bun_detection_gaps.md | Add Bun analyzer scenario to microbench harness (config + baseline + wiring). | `src/Bench/StellaOps.Bench/Scanner.Analyzers` |
|
||||
|
||||
Reference in New Issue
Block a user