90 KiB
90 KiB
Multi-Language Script Engine
Overview
The Multi-Language Script Engine provides a polyglot scripting platform for deployment hooks, health checks, smoke tests, and custom workflow steps. Scripts are stored as versioned files, executed via language-specific Docker runtime images, and mounted into containers at deploy time. The integrated Monaco-based editor provides full IDE features for C# (.NET 10), Python, Java, Go, and Bash.
This is a best-in-class implementation that gives teams flexibility in their language of choice while maintaining security, versioning, and reusability through a centralized script library.
Design Principles
- Polyglot by Design: First-class support for C#, Python, Java, Go, and Bash
- File-Based Execution: Scripts are files mounted into runtime containers
- IDE-Quality Editing: Monaco editor with IntelliSense, linting, and formatting
- Library Management: Versioned dependencies for each language
- Security Sandboxed: Scripts run in isolated containers with resource limits
- Reusable Components: Shared script library with organization-wide access
Architecture
Component Overview
┌────────────────────────────────────────────────────────────────────────┐
│ Multi-Language Script Engine │
├────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌──────────────────┐ ┌───────────────────┐ ┌─────────────────┐ │
│ │ ScriptRegistry │───▶│ MonacoEditorSvc │───▶│ LanguageServer │ │
│ │ │ │ │ │ Pool │ │
│ └──────────────────┘ └───────────────────┘ └─────────────────┘ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌──────────────────┐ ┌───────────────────┐ ┌─────────────────┐ │
│ │ LibraryManager │ │ RuntimeImageMgr │ │ SampleLibrary │ │
│ │ │ │ │ │ │ │
│ └──────────────────┘ └───────────────────┘ └─────────────────┘ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌──────────────────┐ ┌───────────────────┐ ┌─────────────────┐ │
│ │ ScriptExecutor │ │ MountGenerator │ │ OutputCollector │ │
│ │ │ │ │ │ │ │
│ └──────────────────┘ └───────────────────┘ └─────────────────┘ │
│ │
└────────────────────────────────────────────────────────────────────────┘
Execution Flow
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ Script │ │ Runtime │ │ Target │
│ File │─────▶│ Container │─────▶│ Container │
│ │ │ (mount /) │ │ │
└─────────────┘ └─────────────┘ └─────────────┘
Key Components
1. ScriptRegistry
Central repository for all scripts:
public sealed class ScriptRegistry
{
public async Task<Script> CreateScriptAsync(
CreateScriptRequest request,
CancellationToken ct)
{
// Validate script
var validation = await ValidateScriptAsync(request.Language, request.Content, ct);
if (!validation.IsValid)
{
throw new ScriptValidationException(validation.Errors);
}
var script = new Script
{
Id = Guid.NewGuid(),
Name = request.Name,
Description = request.Description,
Language = request.Language,
Content = request.Content,
EntryPoint = request.EntryPoint ?? GetDefaultEntryPoint(request.Language),
Version = "1.0.0",
Dependencies = request.Dependencies ?? ImmutableArray<ScriptDependency>.Empty,
Visibility = request.Visibility,
Tags = request.Tags ?? ImmutableArray<string>.Empty,
CreatedAt = _timeProvider.GetUtcNow(),
CreatedBy = request.RequestedBy
};
// Store script content
await _scriptStore.SaveContentAsync(script.Id, script.Version, script.Content, ct);
// Index for search
await _searchIndex.IndexAsync(script, ct);
await _scriptStore.SaveAsync(script, ct);
return script;
}
public async Task<Script> UpdateScriptAsync(
Guid scriptId,
UpdateScriptRequest request,
CancellationToken ct)
{
var existing = await _scriptStore.GetAsync(scriptId, ct);
// Validate new content
var validation = await ValidateScriptAsync(existing.Language, request.Content, ct);
if (!validation.IsValid)
{
throw new ScriptValidationException(validation.Errors);
}
// Create new version
var newVersion = IncrementVersion(existing.Version, request.VersionBump);
var updated = existing with
{
Content = request.Content,
Version = newVersion,
Dependencies = request.Dependencies ?? existing.Dependencies,
UpdatedAt = _timeProvider.GetUtcNow(),
UpdatedBy = request.RequestedBy
};
// Store versioned content
await _scriptStore.SaveContentAsync(scriptId, newVersion, request.Content, ct);
await _scriptStore.SaveAsync(updated, ct);
return updated;
}
public async Task<IReadOnlyList<Script>> SearchAsync(
ScriptSearchQuery query,
CancellationToken ct)
{
return await _searchIndex.SearchAsync(new SearchRequest
{
Query = query.Text,
Filters = new Dictionary<string, object>
{
["language"] = query.Languages,
["tags"] = query.Tags,
["visibility"] = query.Visibility
},
Limit = query.Limit,
Offset = query.Offset
}, ct);
}
}
public sealed record Script
{
public Guid Id { get; init; }
public string Name { get; init; }
public string Description { get; init; }
public ScriptLanguage Language { get; init; }
public string Content { get; init; }
public string EntryPoint { get; init; }
public string Version { get; init; }
public ImmutableArray<ScriptDependency> Dependencies { get; init; }
public ScriptVisibility Visibility { get; init; }
public ImmutableArray<string> Tags { get; init; }
public DateTimeOffset CreatedAt { get; init; }
public string CreatedBy { get; init; }
public DateTimeOffset? UpdatedAt { get; init; }
public string? UpdatedBy { get; init; }
}
public enum ScriptLanguage
{
CSharp, // .NET 10 CLI scripts
Python, // Python 3.12+
Java, // Java 21+
Go, // Go 1.22+
Bash, // Bash 5+
TypeScript // TypeScript 5+ (transpiled via tsc)
}
public enum ScriptVisibility
{
Private, // Only creator can use
Team, // Team members can use
Organization, // Anyone in org can use
Public // Published to marketplace
}
2. Monaco Editor Service
Provides IDE-quality editing experience:
public sealed class MonacoEditorService
{
private readonly ImmutableDictionary<ScriptLanguage, ILanguageServer> _languageServers;
public async Task<EditorConfiguration> GetConfigurationAsync(
ScriptLanguage language,
CancellationToken ct)
{
return new EditorConfiguration
{
Language = GetMonacoLanguageId(language),
Theme = "stella-dark",
Options = new EditorOptions
{
AutoIndent = true,
FormatOnPaste = true,
FormatOnType = true,
Minimap = new MinimapOptions { Enabled = true },
Suggest = new SuggestOptions
{
ShowKeywords = true,
ShowSnippets = true,
ShowClasses = true,
ShowFunctions = true,
ShowVariables = true
},
QuickSuggestions = true,
ParameterHints = new ParameterHintsOptions { Enabled = true },
Hover = new HoverOptions { Enabled = true },
CodeLens = true,
FoldingStrategy = "auto"
},
CompletionProviders = await GetCompletionProvidersAsync(language, ct),
DiagnosticProviders = await GetDiagnosticProvidersAsync(language, ct),
Snippets = await GetSnippetsAsync(language, ct)
};
}
public async Task<CompletionList> GetCompletionsAsync(
CompletionRequest request,
CancellationToken ct)
{
var languageServer = _languageServers[request.Language];
return await languageServer.GetCompletionsAsync(
request.Content,
request.Position,
request.TriggerKind,
ct);
}
public async Task<IReadOnlyList<Diagnostic>> GetDiagnosticsAsync(
DiagnosticRequest request,
CancellationToken ct)
{
var languageServer = _languageServers[request.Language];
return await languageServer.GetDiagnosticsAsync(request.Content, ct);
}
public async Task<string> FormatDocumentAsync(
FormatRequest request,
CancellationToken ct)
{
var languageServer = _languageServers[request.Language];
return await languageServer.FormatAsync(request.Content, request.Options, ct);
}
public async Task<HoverInfo?> GetHoverInfoAsync(
HoverRequest request,
CancellationToken ct)
{
var languageServer = _languageServers[request.Language];
return await languageServer.GetHoverAsync(
request.Content,
request.Position,
ct);
}
public async Task<IReadOnlyList<SignatureHelp>> GetSignatureHelpAsync(
SignatureHelpRequest request,
CancellationToken ct)
{
var languageServer = _languageServers[request.Language];
return await languageServer.GetSignatureHelpAsync(
request.Content,
request.Position,
ct);
}
private string GetMonacoLanguageId(ScriptLanguage language) => language switch
{
ScriptLanguage.CSharp => "csharp",
ScriptLanguage.Python => "python",
ScriptLanguage.Java => "java",
ScriptLanguage.Go => "go",
ScriptLanguage.Bash => "shell",
_ => "plaintext"
};
}
// Language Server implementations
public sealed class CSharpLanguageServer : ILanguageServer
{
// Uses OmniSharp or Roslyn for .NET 10 script support
private readonly OmniSharpClient _omniSharp;
public async Task<CompletionList> GetCompletionsAsync(
string content,
Position position,
CompletionTriggerKind triggerKind,
CancellationToken ct)
{
// Wrap script content in minimal project structure
var project = CreateScriptProject(content);
return await _omniSharp.GetCompletionsAsync(project, position, ct);
}
}
public sealed class PythonLanguageServer : ILanguageServer
{
// Uses Pyright/Pylance for Python support
private readonly PyrightClient _pyright;
}
public sealed class JavaLanguageServer : ILanguageServer
{
// Uses Eclipse JDT LS for Java support
private readonly JdtlsClient _jdtls;
}
public sealed class GoLanguageServer : ILanguageServer
{
// Uses gopls for Go support
private readonly GoplsClient _gopls;
}
public sealed class BashLanguageServer : ILanguageServer
{
// Uses bash-language-server
private readonly BashLsClient _bashLs;
}
3. Library Manager
Manages script dependencies:
public sealed class LibraryManager
{
public async Task<ResolvedDependencies> ResolveDependenciesAsync(
ScriptLanguage language,
IReadOnlyList<ScriptDependency> dependencies,
CancellationToken ct)
{
var resolver = GetResolver(language);
return await resolver.ResolveAsync(dependencies, ct);
}
private IDependencyResolver GetResolver(ScriptLanguage language) => language switch
{
ScriptLanguage.CSharp => _nugetResolver,
ScriptLanguage.Python => _pipResolver,
ScriptLanguage.Java => _mavenResolver,
ScriptLanguage.Go => _goModResolver,
ScriptLanguage.Bash => _aptResolver,
_ => throw new UnsupportedLanguageException(language)
};
}
// NuGet resolver for C#
public sealed class NuGetDependencyResolver : IDependencyResolver
{
public async Task<ResolvedDependencies> ResolveAsync(
IReadOnlyList<ScriptDependency> dependencies,
CancellationToken ct)
{
var resolved = new ResolvedDependencies();
foreach (var dep in dependencies)
{
// Resolve from NuGet
var package = await _nugetClient.ResolveAsync(
dep.Name,
NuGetVersion.Parse(dep.Version),
ct);
resolved.Packages.Add(new ResolvedPackage
{
Name = package.Id,
Version = package.Version.ToString(),
DownloadUrl = package.DownloadUrl,
Hash = package.Hash,
TransitiveDependencies = package.Dependencies
.Select(d => d.Id)
.ToImmutableArray()
});
}
// Generate restore commands
resolved.RestoreCommand = GenerateDotnetRestoreCommand(resolved.Packages);
return resolved;
}
}
// pip resolver for Python
public sealed class PipDependencyResolver : IDependencyResolver
{
public async Task<ResolvedDependencies> ResolveAsync(
IReadOnlyList<ScriptDependency> dependencies,
CancellationToken ct)
{
var resolved = new ResolvedDependencies();
// Generate requirements.txt
var requirements = dependencies
.Select(d => $"{d.Name}=={d.Version}")
.ToList();
resolved.RequirementsFile = string.Join("\n", requirements);
resolved.RestoreCommand = "pip install -r /scripts/requirements.txt";
return resolved;
}
}
public sealed record ScriptDependency
{
public string Name { get; init; }
public string Version { get; init; }
public DependencyScope Scope { get; init; }
}
public enum DependencyScope
{
Runtime, // Required at execution time
Test, // Only for testing
Build // Only for compilation
}
4. Runtime Image Manager
Manages Docker runtime images for each language:
public sealed class RuntimeImageManager
{
private readonly ImmutableDictionary<ScriptLanguage, RuntimeImageConfig> _images;
public RuntimeImageManager()
{
_images = new Dictionary<ScriptLanguage, RuntimeImageConfig>
{
[ScriptLanguage.CSharp] = new RuntimeImageConfig
{
BaseImage = "mcr.microsoft.com/dotnet/sdk:10.0-alpine",
WorkDir = "/scripts",
EntryPointTemplate = "dotnet script {script}",
DefaultTimeout = TimeSpan.FromMinutes(5),
ResourceLimits = new ResourceLimits
{
CpuLimit = "1.0",
MemoryLimit = "512m"
}
},
[ScriptLanguage.Python] = new RuntimeImageConfig
{
BaseImage = "python:3.12-slim",
WorkDir = "/scripts",
EntryPointTemplate = "python {script}",
DefaultTimeout = TimeSpan.FromMinutes(5),
ResourceLimits = new ResourceLimits
{
CpuLimit = "1.0",
MemoryLimit = "512m"
}
},
[ScriptLanguage.Java] = new RuntimeImageConfig
{
BaseImage = "eclipse-temurin:21-jdk-alpine",
WorkDir = "/scripts",
EntryPointTemplate = "java {script}",
DefaultTimeout = TimeSpan.FromMinutes(5),
ResourceLimits = new ResourceLimits
{
CpuLimit = "1.0",
MemoryLimit = "1g"
}
},
[ScriptLanguage.Go] = new RuntimeImageConfig
{
BaseImage = "golang:1.22-alpine",
WorkDir = "/scripts",
EntryPointTemplate = "go run {script}",
DefaultTimeout = TimeSpan.FromMinutes(5),
ResourceLimits = new ResourceLimits
{
CpuLimit = "1.0",
MemoryLimit = "512m"
}
},
[ScriptLanguage.Bash] = new RuntimeImageConfig
{
BaseImage = "alpine:3.19",
WorkDir = "/scripts",
EntryPointTemplate = "bash {script}",
DefaultTimeout = TimeSpan.FromMinutes(5),
ResourceLimits = new ResourceLimits
{
CpuLimit = "0.5",
MemoryLimit = "256m"
}
},
[ScriptLanguage.TypeScript] = new RuntimeImageConfig
{
BaseImage = "node:22-alpine",
WorkDir = "/scripts",
EntryPointTemplate = "node {script}", // Runs transpiled JS
DefaultTimeout = TimeSpan.FromMinutes(5),
ResourceLimits = new ResourceLimits
{
CpuLimit = "1.0",
MemoryLimit = "512m"
}
}
}.ToImmutableDictionary();
}
public async Task<string> BuildRuntimeImageAsync(
Script script,
ResolvedDependencies dependencies,
CancellationToken ct)
{
var config = _images[script.Language];
var dockerfile = GenerateDockerfile(script, dependencies, config);
var tag = $"stella-script-{script.Id}:{script.Version}";
await _dockerClient.BuildImageAsync(dockerfile, tag, ct);
return tag;
}
private string GenerateDockerfile(
Script script,
ResolvedDependencies dependencies,
RuntimeImageConfig config)
{
var sb = new StringBuilder();
sb.AppendLine($"FROM {config.BaseImage}");
sb.AppendLine($"WORKDIR {config.WorkDir}");
// Add dependency restoration
if (!string.IsNullOrEmpty(dependencies.RestoreCommand))
{
// Copy dependency manifest
switch (script.Language)
{
case ScriptLanguage.CSharp:
sb.AppendLine("COPY global.json .");
sb.AppendLine("COPY *.csproj .");
sb.AppendLine($"RUN {dependencies.RestoreCommand}");
break;
case ScriptLanguage.Python:
sb.AppendLine("COPY requirements.txt .");
sb.AppendLine($"RUN {dependencies.RestoreCommand}");
break;
case ScriptLanguage.Java:
sb.AppendLine("COPY pom.xml .");
sb.AppendLine($"RUN {dependencies.RestoreCommand}");
break;
case ScriptLanguage.Go:
sb.AppendLine("COPY go.mod go.sum ./");
sb.AppendLine($"RUN {dependencies.RestoreCommand}");
break;
}
}
// Script will be mounted at runtime
sb.AppendLine("VOLUME /scripts");
// Entry point
var entryPoint = config.EntryPointTemplate.Replace("{script}", script.EntryPoint);
sb.AppendLine($"ENTRYPOINT [\"{entryPoint.Split(' ')[0]}\", \"{string.Join("\", \"", entryPoint.Split(' ').Skip(1))}\"]");
return sb.ToString();
}
}
5. Script Executor
Executes scripts with mount-based injection:
public sealed class ScriptExecutor
{
public async Task<ScriptExecutionResult> ExecuteAsync(
ScriptExecutionRequest request,
CancellationToken ct)
{
var script = await _scriptRegistry.GetAsync(request.ScriptId, ct);
var dependencies = await _libraryManager.ResolveDependenciesAsync(
script.Language, script.Dependencies, ct);
// Build or get cached runtime image
var runtimeImage = await _runtimeImageManager.GetOrBuildAsync(script, dependencies, ct);
var result = new ScriptExecutionResult
{
ExecutionId = Guid.NewGuid(),
ScriptId = script.Id,
ScriptVersion = script.Version,
StartedAt = _timeProvider.GetUtcNow()
};
try
{
// Create script mount
var scriptMount = await CreateScriptMountAsync(script, request.Arguments, ct);
// Create container
var container = await _dockerClient.CreateContainerAsync(new ContainerConfig
{
Image = runtimeImage,
Mounts = new[]
{
new Mount
{
Type = MountType.Bind,
Source = scriptMount.HostPath,
Target = "/scripts",
ReadOnly = true
}
},
Env = request.Environment?.Select(kv => $"{kv.Key}={kv.Value}").ToArray(),
NetworkMode = request.NetworkMode ?? "none", // Isolated by default
Resources = new ResourcesConfig
{
CpuLimit = _images[script.Language].ResourceLimits.CpuLimit,
MemoryLimit = _images[script.Language].ResourceLimits.MemoryLimit
}
}, ct);
// Start container
await _dockerClient.StartContainerAsync(container.Id, ct);
// Wait for completion with timeout
var timeout = request.Timeout ?? _images[script.Language].DefaultTimeout;
using var timeoutCts = new CancellationTokenSource(timeout);
using var linkedCts = CancellationTokenSource.CreateLinkedTokenSource(ct, timeoutCts.Token);
var exitCode = await _dockerClient.WaitContainerAsync(container.Id, linkedCts.Token);
// Collect output
var stdout = await _dockerClient.GetLogsAsync(container.Id, stdout: true, ct);
var stderr = await _dockerClient.GetLogsAsync(container.Id, stderr: true, ct);
result.ExitCode = exitCode;
result.Stdout = stdout;
result.Stderr = stderr;
result.Status = exitCode == 0
? ScriptExecutionStatus.Succeeded
: ScriptExecutionStatus.Failed;
// Cleanup
await _dockerClient.RemoveContainerAsync(container.Id, ct);
await CleanupMountAsync(scriptMount, ct);
}
catch (OperationCanceledException) when (ct.IsCancellationRequested)
{
result.Status = ScriptExecutionStatus.Cancelled;
}
catch (OperationCanceledException)
{
result.Status = ScriptExecutionStatus.TimedOut;
}
catch (Exception ex)
{
result.Status = ScriptExecutionStatus.Error;
result.Error = ex.Message;
}
result.CompletedAt = _timeProvider.GetUtcNow();
result.Duration = result.CompletedAt - result.StartedAt;
// Store execution record
await _executionStore.SaveAsync(result, ct);
return result;
}
private async Task<ScriptMount> CreateScriptMountAsync(
Script script,
IReadOnlyDictionary<string, string>? arguments,
CancellationToken ct)
{
var mountDir = Path.Combine(_tempPath, Guid.NewGuid().ToString());
Directory.CreateDirectory(mountDir);
// Write script file
var scriptPath = Path.Combine(mountDir, script.EntryPoint);
await File.WriteAllTextAsync(scriptPath, script.Content, ct);
// Write arguments file if provided
if (arguments?.Any() == true)
{
var argsPath = Path.Combine(mountDir, "args.json");
await File.WriteAllTextAsync(argsPath,
JsonSerializer.Serialize(arguments), ct);
}
return new ScriptMount
{
HostPath = mountDir,
ScriptFile = script.EntryPoint
};
}
}
public sealed record ScriptExecutionResult
{
public Guid ExecutionId { get; init; }
public Guid ScriptId { get; init; }
public string ScriptVersion { get; init; }
public ScriptExecutionStatus Status { get; init; }
public int? ExitCode { get; init; }
public string? Stdout { get; init; }
public string? Stderr { get; init; }
public string? Error { get; init; }
public DateTimeOffset StartedAt { get; init; }
public DateTimeOffset? CompletedAt { get; init; }
public TimeSpan? Duration { get; init; }
}
6. Sample Library
Pre-built sample scripts:
public sealed class SampleLibrary
{
public async Task<IReadOnlyList<Script>> GetSamplesAsync(
ScriptLanguage? language,
string? category,
CancellationToken ct)
{
var samples = _samples.AsEnumerable();
if (language.HasValue)
{
samples = samples.Where(s => s.Language == language.Value);
}
if (!string.IsNullOrEmpty(category))
{
samples = samples.Where(s => s.Tags.Contains(category));
}
return samples.ToList();
}
private readonly ImmutableArray<Script> _samples = new[]
{
// C# Samples
CreateSample(
"health-check-http",
"HTTP Health Check",
ScriptLanguage.CSharp,
"Performs HTTP health check with configurable endpoint and expected status",
new[] { "health-check", "http" },
CSharpSamples.HttpHealthCheck,
"health-check.csx",
new[] { new ScriptDependency { Name = "System.Net.Http.Json", Version = "8.0.0" } }
),
CreateSample(
"smoke-test-api",
"API Smoke Test",
ScriptLanguage.CSharp,
"Runs basic API smoke tests with request/response validation",
new[] { "smoke-test", "api" },
CSharpSamples.ApiSmokeTest,
"smoke-test.csx",
new[] { new ScriptDependency { Name = "System.Net.Http.Json", Version = "8.0.0" } }
),
CreateSample(
"database-migration-check",
"Database Migration Validator",
ScriptLanguage.CSharp,
"Verifies database migrations were applied correctly",
new[] { "database", "migration", "validation" },
CSharpSamples.DatabaseMigrationCheck,
"db-check.csx",
new[]
{
new ScriptDependency { Name = "Npgsql", Version = "8.0.0" },
new ScriptDependency { Name = "Dapper", Version = "2.1.0" }
}
),
// Python Samples
CreateSample(
"log-analyzer",
"Log Analyzer",
ScriptLanguage.Python,
"Analyzes application logs for errors and anomalies",
new[] { "monitoring", "logs", "analysis" },
PythonSamples.LogAnalyzer,
"log_analyzer.py",
new[]
{
new ScriptDependency { Name = "requests", Version = "2.31.0" },
new ScriptDependency { Name = "pandas", Version = "2.1.0" }
}
),
CreateSample(
"prometheus-query",
"Prometheus Query",
ScriptLanguage.Python,
"Queries Prometheus metrics and validates thresholds",
new[] { "monitoring", "metrics", "prometheus" },
PythonSamples.PrometheusQuery,
"prom_query.py",
new[] { new ScriptDependency { Name = "prometheus-api-client", Version = "0.5.4" } }
),
CreateSample(
"slack-notification",
"Slack Notification",
ScriptLanguage.Python,
"Sends deployment notifications to Slack",
new[] { "notification", "slack" },
PythonSamples.SlackNotification,
"slack_notify.py",
new[] { new ScriptDependency { Name = "slack-sdk", Version = "3.23.0" } }
),
// Java Samples
CreateSample(
"jdbc-health-check",
"JDBC Health Check",
ScriptLanguage.Java,
"Validates database connectivity via JDBC",
new[] { "health-check", "database", "jdbc" },
JavaSamples.JdbcHealthCheck,
"JdbcHealthCheck.java",
Array.Empty<ScriptDependency>()
),
CreateSample(
"kafka-consumer-check",
"Kafka Consumer Lag Check",
ScriptLanguage.Java,
"Monitors Kafka consumer group lag",
new[] { "kafka", "monitoring" },
JavaSamples.KafkaConsumerCheck,
"KafkaLagCheck.java",
Array.Empty<ScriptDependency>()
),
// Go Samples
CreateSample(
"tcp-port-check",
"TCP Port Check",
ScriptLanguage.Go,
"Checks if TCP ports are listening",
new[] { "health-check", "network" },
GoSamples.TcpPortCheck,
"portcheck.go",
Array.Empty<ScriptDependency>()
),
CreateSample(
"container-inspect",
"Container Inspector",
ScriptLanguage.Go,
"Inspects running containers and validates configuration",
new[] { "docker", "validation" },
GoSamples.ContainerInspect,
"inspect.go",
Array.Empty<ScriptDependency>()
),
// Bash Samples
CreateSample(
"disk-space-check",
"Disk Space Check",
ScriptLanguage.Bash,
"Checks available disk space on target",
new[] { "health-check", "system" },
BashSamples.DiskSpaceCheck,
"diskcheck.sh",
Array.Empty<ScriptDependency>()
),
CreateSample(
"service-restart",
"Service Restart",
ScriptLanguage.Bash,
"Restarts a systemd service with health verification",
new[] { "operations", "systemd" },
BashSamples.ServiceRestart,
"restart.sh",
Array.Empty<ScriptDependency>()
),
CreateSample(
"backup-verify",
"Backup Verification",
ScriptLanguage.Bash,
"Verifies backup files exist and are recent",
new[] { "backup", "validation" },
BashSamples.BackupVerify,
"backup_verify.sh",
Array.Empty<ScriptDependency>()
),
// TypeScript Samples
CreateSample(
"api-integration-test",
"API Integration Test",
ScriptLanguage.TypeScript,
"Runs integration tests against REST API with schema validation",
new[] { "testing", "api", "integration" },
TypeScriptSamples.ApiIntegrationTest,
"api-test.ts",
new[]
{
new ScriptDependency { Name = "axios", Version = "1.6.0" },
new ScriptDependency { Name = "zod", Version = "3.22.0" }
}
),
CreateSample(
"json-schema-validator",
"JSON Schema Validator",
ScriptLanguage.TypeScript,
"Validates JSON payloads against schemas with detailed error reporting",
new[] { "validation", "schema", "json" },
TypeScriptSamples.JsonSchemaValidator,
"schema-validator.ts",
new[] { new ScriptDependency { Name = "ajv", Version = "8.12.0" } }
),
CreateSample(
"webhook-sender",
"Webhook Sender",
ScriptLanguage.TypeScript,
"Sends webhook notifications with retry logic and delivery confirmation",
new[] { "notifications", "webhook", "integration" },
TypeScriptSamples.WebhookSender,
"webhook.ts",
new[] { new ScriptDependency { Name = "axios", Version = "1.6.0" } }
)
}.ToImmutableArray();
}
Sample Scripts
C# Health Check (.NET 10)
// health-check.csx
// .NET 10 script for HTTP health check
#r "nuget: System.Net.Http.Json, 8.0.0"
using System.Net.Http;
using System.Net.Http.Json;
using System.Text.Json;
// Read arguments from mounted file
var argsPath = "/scripts/args.json";
var args = File.Exists(argsPath)
? JsonSerializer.Deserialize<Dictionary<string, string>>(File.ReadAllText(argsPath))
: new Dictionary<string, string>();
var endpoint = args.GetValueOrDefault("endpoint", "http://localhost:8080/health");
var expectedStatus = int.Parse(args.GetValueOrDefault("expected_status", "200"));
var timeout = TimeSpan.FromSeconds(int.Parse(args.GetValueOrDefault("timeout_seconds", "30")));
Console.WriteLine($"Checking health: {endpoint}");
using var client = new HttpClient { Timeout = timeout };
try
{
var response = await client.GetAsync(endpoint);
var statusCode = (int)response.StatusCode;
if (statusCode == expectedStatus)
{
Console.WriteLine($"✓ Health check passed: {statusCode}");
Environment.Exit(0);
}
else
{
Console.WriteLine($"✗ Health check failed: expected {expectedStatus}, got {statusCode}");
Environment.Exit(1);
}
}
catch (Exception ex)
{
Console.WriteLine($"✗ Health check error: {ex.Message}");
Environment.Exit(1);
}
Python Prometheus Query
# prom_query.py
# Python script for querying Prometheus metrics
import json
import sys
from prometheus_api_client import PrometheusConnect
# Read arguments
args_path = "/scripts/args.json"
args = {}
if os.path.exists(args_path):
with open(args_path) as f:
args = json.load(f)
prometheus_url = args.get("prometheus_url", "http://localhost:9090")
query = args.get("query", 'up{job="myapp"}')
threshold = float(args.get("threshold", "0.95"))
comparison = args.get("comparison", "gte") # gte, lte, gt, lt, eq
print(f"Querying Prometheus: {query}")
try:
prom = PrometheusConnect(url=prometheus_url)
result = prom.custom_query(query)
if not result:
print("✗ No data returned from query")
sys.exit(1)
# Extract value
value = float(result[0]["value"][1])
print(f"Current value: {value}")
# Compare against threshold
passed = False
if comparison == "gte":
passed = value >= threshold
elif comparison == "lte":
passed = value <= threshold
elif comparison == "gt":
passed = value > threshold
elif comparison == "lt":
passed = value < threshold
elif comparison == "eq":
passed = abs(value - threshold) < 0.001
if passed:
print(f"✓ Metric check passed: {value} {comparison} {threshold}")
sys.exit(0)
else:
print(f"✗ Metric check failed: {value} not {comparison} {threshold}")
sys.exit(1)
except Exception as e:
print(f"✗ Query error: {e}")
sys.exit(1)
Java JDBC Health Check
// JdbcHealthCheck.java
// Java script for database connectivity check
import java.sql.*;
import java.nio.file.*;
import com.google.gson.*;
public class JdbcHealthCheck {
public static void main(String[] args) throws Exception {
// Read arguments
Path argsPath = Paths.get("/scripts/args.json");
JsonObject config = new JsonObject();
if (Files.exists(argsPath)) {
String content = Files.readString(argsPath);
config = JsonParser.parseString(content).getAsJsonObject();
}
String jdbcUrl = config.has("jdbc_url")
? config.get("jdbc_url").getAsString()
: "jdbc:postgresql://localhost:5432/mydb";
String username = config.has("username")
? config.get("username").getAsString()
: "postgres";
String password = config.has("password")
? config.get("password").getAsString()
: "";
String validationQuery = config.has("validation_query")
? config.get("validation_query").getAsString()
: "SELECT 1";
System.out.println("Checking database: " + jdbcUrl);
try (Connection conn = DriverManager.getConnection(jdbcUrl, username, password);
Statement stmt = conn.createStatement();
ResultSet rs = stmt.executeQuery(validationQuery)) {
if (rs.next()) {
System.out.println("✓ Database check passed");
System.exit(0);
} else {
System.out.println("✗ Database check failed: no result");
System.exit(1);
}
} catch (SQLException e) {
System.out.println("✗ Database check error: " + e.getMessage());
System.exit(1);
}
}
}
Go TCP Port Check
// portcheck.go
// Go script for TCP port connectivity check
package main
import (
"encoding/json"
"fmt"
"net"
"os"
"time"
)
func main() {
// Read arguments
args := make(map[string]string)
if data, err := os.ReadFile("/scripts/args.json"); err == nil {
json.Unmarshal(data, &args)
}
host := getOrDefault(args, "host", "localhost")
port := getOrDefault(args, "port", "8080")
timeoutSec := getOrDefault(args, "timeout_seconds", "5")
address := fmt.Sprintf("%s:%s", host, port)
timeout, _ := time.ParseDuration(timeoutSec + "s")
fmt.Printf("Checking TCP port: %s\n", address)
conn, err := net.DialTimeout("tcp", address, timeout)
if err != nil {
fmt.Printf("✗ Port check failed: %s\n", err.Error())
os.Exit(1)
}
defer conn.Close()
fmt.Println("✓ Port check passed")
os.Exit(0)
}
func getOrDefault(m map[string]string, key, defaultVal string) string {
if val, ok := m[key]; ok {
return val
}
return defaultVal
}
Bash Disk Space Check
#!/bin/bash
# diskcheck.sh
# Bash script for disk space check
# Read arguments
ARGS_FILE="/scripts/args.json"
if [ -f "$ARGS_FILE" ]; then
MOUNT_POINT=$(jq -r '.mount_point // "/"' "$ARGS_FILE")
THRESHOLD=$(jq -r '.threshold_percent // 80' "$ARGS_FILE")
else
MOUNT_POINT="/"
THRESHOLD=80
fi
echo "Checking disk space on: $MOUNT_POINT"
# Get disk usage
USAGE=$(df "$MOUNT_POINT" | tail -1 | awk '{print $5}' | sed 's/%//')
echo "Current usage: ${USAGE}%"
if [ "$USAGE" -lt "$THRESHOLD" ]; then
echo "✓ Disk space check passed: ${USAGE}% < ${THRESHOLD}%"
exit 0
else
echo "✗ Disk space check failed: ${USAGE}% >= ${THRESHOLD}%"
exit 1
fi
TypeScript API Integration Test
// api-test.ts
// TypeScript script for REST API integration testing
import axios, { AxiosError } from 'axios';
import { z } from 'zod';
import * as fs from 'fs';
// Read arguments from mounted file
const argsPath = '/scripts/args.json';
const args = fs.existsSync(argsPath)
? JSON.parse(fs.readFileSync(argsPath, 'utf-8'))
: {};
const baseUrl = args.base_url ?? 'http://localhost:8080';
const endpoints = args.endpoints ?? ['/api/health', '/api/v1/status'];
const timeout = parseInt(args.timeout_seconds ?? '30', 10) * 1000;
// Define response schemas
const HealthSchema = z.object({
status: z.enum(['healthy', 'degraded', 'unhealthy']),
version: z.string().optional(),
timestamp: z.string().datetime().optional(),
});
const StatusSchema = z.object({
service: z.string(),
status: z.string(),
uptime: z.number().optional(),
});
console.log(`Running API integration tests against: ${baseUrl}`);
const client = axios.create({ baseURL: baseUrl, timeout });
async function testEndpoint(endpoint: string): Promise<boolean> {
try {
console.log(`Testing: ${endpoint}`);
const response = await client.get(endpoint);
// Validate response based on endpoint
if (endpoint.includes('health')) {
HealthSchema.parse(response.data);
console.log(` ✓ Health check passed: ${response.data.status}`);
} else if (endpoint.includes('status')) {
StatusSchema.parse(response.data);
console.log(` ✓ Status check passed: ${response.data.service}`);
} else {
console.log(` ✓ Endpoint responded: ${response.status}`);
}
return true;
} catch (error) {
if (error instanceof z.ZodError) {
console.error(` ✗ Schema validation failed:`, error.errors);
} else if (error instanceof AxiosError) {
console.error(` ✗ Request failed: ${error.message}`);
} else {
console.error(` ✗ Unexpected error:`, error);
}
return false;
}
}
async function main() {
const results = await Promise.all(
endpoints.map((ep: string) => testEndpoint(ep))
);
const passed = results.filter(Boolean).length;
const total = results.length;
console.log(`\nResults: ${passed}/${total} tests passed`);
if (passed === total) {
console.log('✓ All integration tests passed');
process.exit(0);
} else {
console.log('✗ Some integration tests failed');
process.exit(1);
}
}
main();
API Design
REST Endpoints
# Scripts
POST /api/v1/scripts # Create script
GET /api/v1/scripts # List scripts
GET /api/v1/scripts/{id} # Get script
PUT /api/v1/scripts/{id} # Update script
DELETE /api/v1/scripts/{id} # Delete script
GET /api/v1/scripts/{id}/versions # List versions
GET /api/v1/scripts/{id}/versions/{ver} # Get version
# Execution
POST /api/v1/scripts/{id}/execute # Execute script
GET /api/v1/scripts/executions # List executions
GET /api/v1/scripts/executions/{id} # Get execution
GET /api/v1/scripts/executions/{id}/logs # Get execution logs
# Library
GET /api/v1/scripts/samples # List samples
GET /api/v1/scripts/samples/{id} # Get sample
POST /api/v1/scripts/samples/{id}/clone # Clone sample
# Editor
GET /api/v1/editor/config/{language} # Get editor config
POST /api/v1/editor/completions # Get completions
POST /api/v1/editor/diagnostics # Get diagnostics
POST /api/v1/editor/format # Format code
POST /api/v1/editor/hover # Get hover info
# Dependencies
POST /api/v1/dependencies/resolve # Resolve dependencies
GET /api/v1/dependencies/search # Search packages
Monaco Editor Configuration
Frontend Integration
// MonacoScriptEditor.tsx
import * as monaco from 'monaco-editor';
import { useEffect, useRef } from 'react';
interface ScriptEditorProps {
language: ScriptLanguage;
value: string;
onChange: (value: string) => void;
onSave: () => void;
}
export function ScriptEditor({ language, value, onChange, onSave }: ScriptEditorProps) {
const editorRef = useRef<monaco.editor.IStandaloneCodeEditor | null>(null);
const containerRef = useRef<HTMLDivElement>(null);
useEffect(() => {
if (!containerRef.current) return;
// Configure language features
configureLanguageFeatures(language);
// Create editor
editorRef.current = monaco.editor.create(containerRef.current, {
value,
language: getMonacoLanguageId(language),
theme: 'stella-dark',
automaticLayout: true,
minimap: { enabled: true },
suggest: {
showKeywords: true,
showSnippets: true,
showClasses: true,
showFunctions: true,
},
quickSuggestions: true,
parameterHints: { enabled: true },
codeLens: true,
folding: true,
formatOnPaste: true,
formatOnType: true,
});
// Register completion provider
monaco.languages.registerCompletionItemProvider(
getMonacoLanguageId(language),
{
provideCompletionItems: async (model, position) => {
const response = await fetch('/api/v1/editor/completions', {
method: 'POST',
body: JSON.stringify({
language,
content: model.getValue(),
position: { line: position.lineNumber, column: position.column },
}),
});
const completions = await response.json();
return { suggestions: completions.items };
},
}
);
// Register hover provider
monaco.languages.registerHoverProvider(
getMonacoLanguageId(language),
{
provideHover: async (model, position) => {
const response = await fetch('/api/v1/editor/hover', {
method: 'POST',
body: JSON.stringify({
language,
content: model.getValue(),
position: { line: position.lineNumber, column: position.column },
}),
});
return await response.json();
},
}
);
// Save handler
editorRef.current.addCommand(
monaco.KeyMod.CtrlCmd | monaco.KeyCode.KeyS,
onSave
);
// Change handler
editorRef.current.onDidChangeModelContent(() => {
onChange(editorRef.current?.getValue() ?? '');
});
return () => {
editorRef.current?.dispose();
};
}, [language]);
return <div ref={containerRef} style={{ height: '100%', width: '100%' }} />;
}
function getMonacoLanguageId(language: ScriptLanguage): string {
const mapping: Record<ScriptLanguage, string> = {
csharp: 'csharp',
python: 'python',
java: 'java',
go: 'go',
bash: 'shell',
typescript: 'typescript',
};
return mapping[language];
}
Metrics & Observability
Prometheus Metrics
# Script Registry
stella_scripts_total{language, visibility}
stella_script_versions_total{script_id}
# Execution
stella_script_executions_total{language, status}
stella_script_execution_duration_seconds{language}
stella_script_execution_exit_code{language}
# Editor
stella_editor_completions_total{language}
stella_editor_diagnostics_total{language, severity}
stella_editor_format_requests_total{language}
# Library
stella_library_dependencies_resolved_total{language}
stella_library_resolution_duration_seconds{language}
Configuration
script_engine:
languages:
csharp:
enabled: true
runtime_image: "mcr.microsoft.com/dotnet/sdk:10.0-alpine"
default_timeout: "5m"
resource_limits:
cpu: "1.0"
memory: "512m"
python:
enabled: true
runtime_image: "python:3.12-slim"
default_timeout: "5m"
resource_limits:
cpu: "1.0"
memory: "512m"
java:
enabled: true
runtime_image: "eclipse-temurin:21-jdk-alpine"
default_timeout: "5m"
resource_limits:
cpu: "1.0"
memory: "1g"
go:
enabled: true
runtime_image: "golang:1.22-alpine"
default_timeout: "5m"
resource_limits:
cpu: "1.0"
memory: "512m"
bash:
enabled: true
runtime_image: "alpine:3.19"
default_timeout: "5m"
resource_limits:
cpu: "0.5"
memory: "256m"
typescript:
enabled: true
runtime_image: "node:22-alpine"
default_timeout: "5m"
resource_limits:
cpu: "1.0"
memory: "512m"
execution:
max_concurrent: 10
default_network_mode: "none"
allow_host_network: false
temp_path: "/tmp/stella-scripts"
editor:
language_servers:
csharp:
type: "omnisharp"
endpoint: "http://localhost:2000"
python:
type: "pyright"
endpoint: "http://localhost:2001"
java:
type: "jdtls"
endpoint: "http://localhost:2002"
go:
type: "gopls"
endpoint: "http://localhost:2003"
bash:
type: "bash-language-server"
endpoint: "http://localhost:2004"
typescript:
type: "typescript-language-server"
endpoint: "http://localhost:2005"
library:
cache_path: "/var/cache/stella-scripts"
nuget_source: "https://api.nuget.org/v3/index.json"
pypi_source: "https://pypi.org/simple"
maven_source: "https://repo.maven.apache.org/maven2"
npm_source: "https://registry.npmjs.org"
Performance & Caching Architecture
The Problem
Without optimization, each script execution would require:
- Docker image build (~30-60s for .NET, ~10-20s for others)
- Container startup (~2-5s)
- Dependency restoration (~10-30s for NuGet, pip, Maven)
- Compilation (~5-15s for .NET/Java/Go)
- Script execution (actual work)
For a workflow with 10 script steps, this could mean 10+ minutes of overhead.
Target Performance
| Metric | Cold Start | Warm Start |
|---|---|---|
| First script execution | < 30s | N/A |
| Subsequent same-script | < 2s | < 500ms |
| Different script, same language | < 5s | < 1s |
| Workflow with 10 scripts | < 60s total | < 15s total |
Multi-Layer Caching Strategy
┌─────────────────────────────────────────────────────────────────────────────┐
│ Script Execution Performance Stack │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ Layer 1: Pre-compiled Script Cache │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ Script Hash → Compiled Assembly/Bytecode (instant execution) │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ ↓ miss │
│ Layer 2: Warm Container Pool │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ Language → Ready-to-use containers with dependencies (< 500ms) │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ ↓ miss │
│ Layer 3: Pre-built Runtime Images │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ Language+Deps Hash → Docker image with restored packages (< 5s) │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ ↓ miss │
│ Layer 4: Dependency Cache │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ NuGet/pip/Maven/Go packages cached locally (< 15s restore) │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ ↓ miss │
│ Layer 5: Cold Build (fallback) │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ Full image build + dependency download (30-60s) │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
Component Details
1. Pre-compiled Script Cache
For compiled languages (.NET, Java, Go), cache the compiled output:
public sealed class ScriptCompilationCache
{
private readonly IDistributedCache _cache;
private readonly ICompilationService _compiler;
public async Task<CompiledScript> GetOrCompileAsync(
Script script,
CancellationToken ct)
{
// Cache key: hash of script content + dependencies + runtime version
var cacheKey = ComputeCacheKey(script);
// Try L1 (memory) cache
if (_memoryCache.TryGetValue(cacheKey, out CompiledScript cached))
{
return cached;
}
// Try L2 (distributed) cache
var cachedBytes = await _cache.GetAsync(cacheKey, ct);
if (cachedBytes != null)
{
var compiled = DeserializeCompiledScript(cachedBytes);
_memoryCache.Set(cacheKey, compiled, TimeSpan.FromHours(1));
return compiled;
}
// Compile and cache
var result = await _compiler.CompileAsync(script, ct);
await _cache.SetAsync(cacheKey, SerializeCompiledScript(result),
new DistributedCacheEntryOptions
{
AbsoluteExpirationRelativeToNow = TimeSpan.FromDays(7)
}, ct);
_memoryCache.Set(cacheKey, result, TimeSpan.FromHours(1));
return result;
}
private string ComputeCacheKey(Script script)
{
using var sha = SHA256.Create();
var content = $"{script.Language}:{script.Content}:{string.Join(",", script.Dependencies)}";
var hash = sha.ComputeHash(Encoding.UTF8.GetBytes(content));
return $"script:compiled:{Convert.ToHexString(hash)}";
}
}
// Language-specific compilers
public sealed class DotNetScriptCompiler : IScriptCompiler
{
public async Task<CompiledScript> CompileAsync(Script script, CancellationToken ct)
{
// Use Roslyn to compile C# script to assembly
var syntaxTree = CSharpSyntaxTree.ParseText(script.Content);
var references = await ResolveReferencesAsync(script.Dependencies, ct);
var compilation = CSharpCompilation.Create(
$"Script_{script.Id}",
new[] { syntaxTree },
references,
new CSharpCompilationOptions(OutputKind.ConsoleApplication));
using var ms = new MemoryStream();
var result = compilation.Emit(ms);
if (!result.Success)
{
throw new CompilationException(result.Diagnostics);
}
return new CompiledScript
{
ScriptId = script.Id,
Language = ScriptLanguage.CSharp,
AssemblyBytes = ms.ToArray(),
CompiledAt = _timeProvider.GetUtcNow(),
EntryPoint = "Program.Main"
};
}
}
public sealed class JavaScriptCompiler : IScriptCompiler
{
public async Task<CompiledScript> CompileAsync(Script script, CancellationToken ct)
{
// Create temporary directory for compilation
var workDir = Path.Combine(_tempPath, Guid.NewGuid().ToString());
Directory.CreateDirectory(workDir);
try
{
// Write Java source file
var className = ExtractClassName(script.Content);
var sourceFile = Path.Combine(workDir, $"{className}.java");
await File.WriteAllTextAsync(sourceFile, script.Content, ct);
// Compile with javac
var classpath = await ResolveClasspathAsync(script.Dependencies, ct);
var process = await RunProcessAsync(
"javac",
$"-d {workDir} -cp \"{classpath}\" {sourceFile}",
workDir,
ct);
if (process.ExitCode != 0)
{
throw new CompilationException($"javac failed: {process.Stderr}");
}
// Read compiled .class files
var classFiles = Directory.GetFiles(workDir, "*.class", SearchOption.AllDirectories);
var compiledClasses = new Dictionary<string, byte[]>();
foreach (var classFile in classFiles)
{
var relativePath = Path.GetRelativePath(workDir, classFile);
compiledClasses[relativePath] = await File.ReadAllBytesAsync(classFile, ct);
}
return new CompiledScript
{
ScriptId = script.Id,
Language = ScriptLanguage.Java,
CompiledClasses = compiledClasses.ToImmutableDictionary(),
CompiledAt = _timeProvider.GetUtcNow(),
EntryPoint = className
};
}
finally
{
Directory.Delete(workDir, recursive: true);
}
}
private string ExtractClassName(string javaSource)
{
// Extract public class name from Java source
var match = Regex.Match(javaSource, @"public\s+class\s+(\w+)");
return match.Success ? match.Groups[1].Value : "Script";
}
}
public sealed class TypeScriptCompiler : IScriptCompiler
{
public async Task<CompiledScript> CompileAsync(Script script, CancellationToken ct)
{
// Create temporary directory for transpilation
var workDir = Path.Combine(_tempPath, Guid.NewGuid().ToString());
Directory.CreateDirectory(workDir);
try
{
// Write TypeScript source file
var sourceFile = Path.Combine(workDir, "script.ts");
await File.WriteAllTextAsync(sourceFile, script.Content, ct);
// Generate tsconfig.json for optimal output
var tsConfig = new
{
compilerOptions = new
{
target = "ES2022",
module = "commonjs",
strict = true,
esModuleInterop = true,
skipLibCheck = true,
outDir = "./dist",
declaration = false,
sourceMap = false
},
include = new[] { "*.ts" }
};
await File.WriteAllTextAsync(
Path.Combine(workDir, "tsconfig.json"),
JsonSerializer.Serialize(tsConfig),
ct);
// Install dependencies if any
if (script.Dependencies.Any())
{
await InstallNpmDependenciesAsync(workDir, script.Dependencies, ct);
}
// Transpile with tsc
var process = await RunProcessAsync(
"npx",
"tsc --project tsconfig.json",
workDir,
ct);
if (process.ExitCode != 0)
{
throw new CompilationException($"tsc failed: {process.Stderr}");
}
// Read transpiled JavaScript
var jsFile = Path.Combine(workDir, "dist", "script.js");
var transpiledJs = await File.ReadAllTextAsync(jsFile, ct);
return new CompiledScript
{
ScriptId = script.Id,
Language = ScriptLanguage.TypeScript,
TranspiledCode = transpiledJs,
CompiledAt = _timeProvider.GetUtcNow(),
EntryPoint = "script.js"
};
}
finally
{
Directory.Delete(workDir, recursive: true);
}
}
}
public sealed class GoScriptCompiler : IScriptCompiler
{
public async Task<CompiledScript> CompileAsync(Script script, CancellationToken ct)
{
var workDir = Path.Combine(_tempPath, Guid.NewGuid().ToString());
Directory.CreateDirectory(workDir);
try
{
// Write Go source file
var sourceFile = Path.Combine(workDir, "main.go");
await File.WriteAllTextAsync(sourceFile, script.Content, ct);
// Generate go.mod if dependencies exist
if (script.Dependencies.Any())
{
await GenerateGoModAsync(workDir, script.Dependencies, ct);
await RunProcessAsync("go", "mod download", workDir, ct);
}
// Compile to binary
var outputBinary = Path.Combine(workDir, "script");
var process = await RunProcessAsync(
"go",
$"build -o {outputBinary} .",
workDir,
ct);
if (process.ExitCode != 0)
{
throw new CompilationException($"go build failed: {process.Stderr}");
}
var binaryBytes = await File.ReadAllBytesAsync(outputBinary, ct);
return new CompiledScript
{
ScriptId = script.Id,
Language = ScriptLanguage.Go,
BinaryBytes = binaryBytes,
CompiledAt = _timeProvider.GetUtcNow(),
EntryPoint = "script"
};
}
finally
{
Directory.Delete(workDir, recursive: true);
}
}
}
2. Smart Container Pool Management
Intelligent pool management with auto-scaling, health monitoring, and graceful shutdown:
public sealed class SmartContainerPoolManager : IHostedService, IAsyncDisposable
{
private readonly ConcurrentDictionary<ScriptLanguage, ManagedContainerPool> _pools = new();
private readonly PoolConfiguration _config;
private readonly IDockerClient _dockerClient;
private readonly ILogger<SmartContainerPoolManager> _logger;
private readonly CancellationTokenSource _shutdownCts = new();
private Task? _maintenanceTask;
public SmartContainerPoolManager(
PoolConfiguration config,
IDockerClient dockerClient,
ILogger<SmartContainerPoolManager> logger)
{
_config = config;
_dockerClient = dockerClient;
_logger = logger;
}
// IHostedService - Start with agent
public async Task StartAsync(CancellationToken ct)
{
_logger.LogInformation("Starting Smart Container Pool Manager");
// Initialize pools for each configured language
foreach (var langConfig in _config.Languages)
{
var pool = new ManagedContainerPool(
language: langConfig.Key,
config: langConfig.Value,
dockerClient: _dockerClient,
logger: _logger);
_pools[langConfig.Key] = pool;
// Warm up to target size
await pool.WarmUpAsync(ct);
}
// Start background maintenance
_maintenanceTask = RunMaintenanceLoopAsync(_shutdownCts.Token);
_logger.LogInformation(
"Container pools initialized: {Pools}",
string.Join(", ", _pools.Select(p => $"{p.Key}={p.Value.CurrentSize}")));
}
// IHostedService - Graceful shutdown with agent
public async Task StopAsync(CancellationToken ct)
{
_logger.LogInformation("Initiating graceful shutdown of container pools");
// Signal maintenance loop to stop
_shutdownCts.Cancel();
// Wait for maintenance to complete
if (_maintenanceTask != null)
{
await _maintenanceTask.WaitAsync(ct);
}
// Graceful shutdown of all pools
var shutdownTasks = _pools.Values.Select(p => p.ShutdownAsync(ct));
await Task.WhenAll(shutdownTasks);
_logger.LogInformation("All container pools shut down gracefully");
}
public async Task<PooledContainer> AcquireAsync(
ScriptLanguage language,
string dependencyHash,
CancellationToken ct)
{
if (!_pools.TryGetValue(language, out var pool))
{
throw new InvalidOperationException($"No pool configured for {language}");
}
return await pool.AcquireAsync(dependencyHash, ct);
}
public async Task ReleaseAsync(PooledContainer container)
{
if (_pools.TryGetValue(container.Language, out var pool))
{
await pool.ReleaseAsync(container);
}
}
private async Task RunMaintenanceLoopAsync(CancellationToken ct)
{
while (!ct.IsCancellationRequested)
{
try
{
await Task.Delay(_config.MaintenanceInterval, ct);
foreach (var pool in _pools.Values)
{
await pool.PerformMaintenanceAsync(ct);
}
}
catch (OperationCanceledException) when (ct.IsCancellationRequested)
{
break;
}
catch (Exception ex)
{
_logger.LogError(ex, "Error during pool maintenance");
}
}
}
public async ValueTask DisposeAsync()
{
_shutdownCts.Cancel();
_shutdownCts.Dispose();
foreach (var pool in _pools.Values)
{
await pool.DisposeAsync();
}
}
}
public sealed class ManagedContainerPool : IAsyncDisposable
{
private readonly ScriptLanguage _language;
private readonly LanguagePoolConfig _config;
private readonly IDockerClient _dockerClient;
private readonly ILogger _logger;
private readonly Channel<PooledContainer> _availableContainers;
private readonly ConcurrentDictionary<string, PooledContainer> _allContainers = new();
private readonly ConcurrentDictionary<string, PooledContainer> _byDependencyHash = new();
private readonly SemaphoreSlim _scaleLock = new(1, 1);
private readonly UsageTracker _usageTracker = new();
public int CurrentSize => _allContainers.Count;
public int AvailableCount => _availableContainers.Reader.Count;
public ManagedContainerPool(
ScriptLanguage language,
LanguagePoolConfig config,
IDockerClient dockerClient,
ILogger logger)
{
_language = language;
_config = config;
_dockerClient = dockerClient;
_logger = logger;
_availableContainers = Channel.CreateBounded<PooledContainer>(config.MaxSize);
}
/// <summary>
/// Warm up pool to target size on startup
/// </summary>
public async Task WarmUpAsync(CancellationToken ct)
{
_logger.LogInformation(
"Warming up {Language} pool to {Target} containers",
_language, _config.TargetSize);
var warmupTasks = Enumerable.Range(0, _config.TargetSize)
.Select(_ => CreateAndAddContainerAsync(ct));
await Task.WhenAll(warmupTasks);
_logger.LogInformation(
"{Language} pool warmed up with {Count} containers",
_language, CurrentSize);
}
/// <summary>
/// Acquire container, with auto-scaling if needed
/// </summary>
public async Task<PooledContainer> AcquireAsync(
string dependencyHash,
CancellationToken ct)
{
_usageTracker.RecordRequest();
// Try exact dependency match first (fastest)
if (_byDependencyHash.TryRemove(dependencyHash, out var exactMatch))
{
_usageTracker.RecordHit();
return exactMatch;
}
// Try any available container
if (_availableContainers.Reader.TryRead(out var available))
{
_byDependencyHash.TryRemove(available.DependencyHash, out _);
_usageTracker.RecordPartialHit();
return available;
}
// No containers available - check if we can scale up
if (CurrentSize < _config.MaxSize)
{
_usageTracker.RecordMiss();
return await CreateNewContainerAsync(dependencyHash, ct);
}
// At max capacity - wait for one to become available
_logger.LogWarning(
"{Language} pool at max capacity ({Max}), waiting for container",
_language, _config.MaxSize);
return await _availableContainers.Reader.ReadAsync(ct);
}
/// <summary>
/// Release container back to pool (or dispose if unhealthy)
/// </summary>
public async Task ReleaseAsync(PooledContainer container)
{
// Health check before returning to pool
if (!await container.HealthCheckAsync())
{
_logger.LogWarning(
"Container {Id} failed health check, disposing",
container.ContainerId);
await DisposeContainerAsync(container);
return;
}
// Reset container state
await container.ResetAsync();
// Return to pool
container.LastUsed = DateTime.UtcNow;
_byDependencyHash[container.DependencyHash] = container;
if (!_availableContainers.Writer.TryWrite(container))
{
// Pool is full (shouldn't happen), dispose container
await DisposeContainerAsync(container);
}
}
/// <summary>
/// Periodic maintenance: health checks, eviction, auto-scaling
/// </summary>
public async Task PerformMaintenanceAsync(CancellationToken ct)
{
await _scaleLock.WaitAsync(ct);
try
{
// 1. Health check all containers
await HealthCheckAllAsync(ct);
// 2. Evict idle containers above target
await EvictIdleContainersAsync(ct);
// 3. Scale up if below target
await ScaleToTargetAsync(ct);
// 4. Auto-scale based on usage patterns
await AutoScaleAsync(ct);
// Log pool status
_logger.LogDebug(
"{Language} pool status: {Current}/{Target}/{Max} (available: {Available})",
_language, CurrentSize, _config.TargetSize, _config.MaxSize, AvailableCount);
}
finally
{
_scaleLock.Release();
}
}
/// <summary>
/// Graceful shutdown - stop all containers
/// </summary>
public async Task ShutdownAsync(CancellationToken ct)
{
_logger.LogInformation("Shutting down {Language} pool ({Count} containers)", _language, CurrentSize);
// Close channel to prevent new containers
_availableContainers.Writer.Complete();
// Stop all containers gracefully
var stopTasks = _allContainers.Values.Select(c => StopContainerGracefullyAsync(c, ct));
await Task.WhenAll(stopTasks);
_allContainers.Clear();
_byDependencyHash.Clear();
}
private async Task HealthCheckAllAsync(CancellationToken ct)
{
var unhealthy = new List<PooledContainer>();
foreach (var container in _allContainers.Values)
{
if (!await container.HealthCheckAsync())
{
unhealthy.Add(container);
}
}
foreach (var container in unhealthy)
{
_logger.LogWarning(
"Replacing unhealthy container {Id} in {Language} pool",
container.ContainerId, _language);
await DisposeContainerAsync(container);
await CreateAndAddContainerAsync(ct);
}
}
private async Task EvictIdleContainersAsync(CancellationToken ct)
{
var now = DateTime.UtcNow;
var toEvict = _allContainers.Values
.Where(c => c.LastUsed.HasValue &&
now - c.LastUsed.Value > _config.IdleTimeout &&
CurrentSize > _config.TargetSize)
.Take(CurrentSize - _config.TargetSize)
.ToList();
foreach (var container in toEvict)
{
_logger.LogDebug(
"Evicting idle container {Id} from {Language} pool",
container.ContainerId, _language);
await DisposeContainerAsync(container);
}
}
private async Task ScaleToTargetAsync(CancellationToken ct)
{
while (CurrentSize < _config.TargetSize)
{
await CreateAndAddContainerAsync(ct);
}
}
private async Task AutoScaleAsync(CancellationToken ct)
{
var stats = _usageTracker.GetStats();
// Scale up if high utilization
if (stats.HitRate < 0.8 && stats.RequestRate > 1.0 && CurrentSize < _config.MaxSize)
{
var scaleUpCount = Math.Min(2, _config.MaxSize - CurrentSize);
_logger.LogInformation(
"Auto-scaling {Language} pool up by {Count} (hit rate: {HitRate:P0})",
_language, scaleUpCount, stats.HitRate);
for (int i = 0; i < scaleUpCount; i++)
{
await CreateAndAddContainerAsync(ct);
}
}
}
private async Task<PooledContainer> CreateNewContainerAsync(
string dependencyHash,
CancellationToken ct)
{
var container = await CreateContainerAsync(dependencyHash, ct);
_allContainers[container.ContainerId] = container;
return container;
}
private async Task CreateAndAddContainerAsync(CancellationToken ct)
{
var container = await CreateContainerAsync(null, ct);
_allContainers[container.ContainerId] = container;
_availableContainers.Writer.TryWrite(container);
}
private async Task<PooledContainer> CreateContainerAsync(
string? dependencyHash,
CancellationToken ct)
{
var containerConfig = new ContainerCreateConfig
{
Image = _config.BaseImage,
Cmd = new[] { "sleep", "infinity" }, // Keep alive
Labels = new Dictionary<string, string>
{
["stella.pool"] = _language.ToString(),
["stella.created"] = DateTime.UtcNow.ToString("O")
},
HostConfig = new HostConfig
{
Memory = _config.MemoryLimit,
NanoCPUs = (long)(_config.CpuLimit * 1_000_000_000),
AutoRemove = false
}
};
var response = await _dockerClient.Containers.CreateContainerAsync(containerConfig, ct);
await _dockerClient.Containers.StartContainerAsync(response.ID, null, ct);
return new PooledContainer
{
ContainerId = response.ID,
Language = _language,
DependencyHash = dependencyHash ?? "",
CreatedAt = DateTime.UtcNow
};
}
private async Task StopContainerGracefullyAsync(PooledContainer container, CancellationToken ct)
{
try
{
await _dockerClient.Containers.StopContainerAsync(
container.ContainerId,
new ContainerStopParameters { WaitBeforeKillSeconds = 10 },
ct);
await _dockerClient.Containers.RemoveContainerAsync(
container.ContainerId,
new ContainerRemoveParameters { Force = true },
ct);
}
catch (Exception ex)
{
_logger.LogWarning(ex, "Error stopping container {Id}", container.ContainerId);
}
}
private async Task DisposeContainerAsync(PooledContainer container)
{
_allContainers.TryRemove(container.ContainerId, out _);
_byDependencyHash.TryRemove(container.DependencyHash, out _);
try
{
await _dockerClient.Containers.RemoveContainerAsync(
container.ContainerId,
new ContainerRemoveParameters { Force = true });
}
catch (Exception ex)
{
_logger.LogWarning(ex, "Error disposing container {Id}", container.ContainerId);
}
}
public async ValueTask DisposeAsync()
{
await ShutdownAsync(CancellationToken.None);
_scaleLock.Dispose();
}
}
/// <summary>
/// Tracks usage patterns for auto-scaling decisions
/// </summary>
public sealed class UsageTracker
{
private readonly ConcurrentQueue<UsageEvent> _events = new();
private readonly TimeSpan _window = TimeSpan.FromMinutes(5);
public void RecordRequest() => _events.Enqueue(new UsageEvent(UsageEventType.Request));
public void RecordHit() => _events.Enqueue(new UsageEvent(UsageEventType.Hit));
public void RecordPartialHit() => _events.Enqueue(new UsageEvent(UsageEventType.PartialHit));
public void RecordMiss() => _events.Enqueue(new UsageEvent(UsageEventType.Miss));
public UsageStats GetStats()
{
var cutoff = DateTime.UtcNow - _window;
// Prune old events
while (_events.TryPeek(out var oldest) && oldest.Timestamp < cutoff)
{
_events.TryDequeue(out _);
}
var events = _events.ToArray();
var requests = events.Count(e => e.Type == UsageEventType.Request);
var hits = events.Count(e => e.Type == UsageEventType.Hit);
var partialHits = events.Count(e => e.Type == UsageEventType.PartialHit);
return new UsageStats
{
TotalRequests = requests,
HitRate = requests > 0 ? (double)(hits + partialHits) / requests : 1.0,
RequestRate = requests / _window.TotalMinutes
};
}
private record UsageEvent(UsageEventType Type)
{
public DateTime Timestamp { get; } = DateTime.UtcNow;
}
private enum UsageEventType { Request, Hit, PartialHit, Miss }
}
public record UsageStats
{
public int TotalRequests { get; init; }
public double HitRate { get; init; }
public double RequestRate { get; init; } // Requests per minute
}
public sealed record PooledContainer
{
public string ContainerId { get; init; }
public ScriptLanguage Language { get; init; }
public string DependencyHash { get; set; }
public DateTime CreatedAt { get; init; }
public DateTime? LastUsed { get; set; }
public async Task<bool> HealthCheckAsync()
{
// Check if container is still running
// Implementation depends on Docker client
return true;
}
public async Task ResetAsync()
{
// Clean up any state from previous execution
// e.g., clear /tmp, reset environment
}
}
3. Pre-built Runtime Images
Build and cache Docker images with dependencies:
public sealed class RuntimeImageCache
{
public async Task<string> GetOrBuildImageAsync(
ScriptLanguage language,
IReadOnlyList<ScriptDependency> dependencies,
CancellationToken ct)
{
var imageTag = ComputeImageTag(language, dependencies);
// Check if image exists locally
if (await _dockerClient.ImageExistsAsync(imageTag, ct))
{
_metrics.RecordImageCacheHit(language);
return imageTag;
}
// Check if image exists in registry
if (await _registryClient.ImageExistsAsync(imageTag, ct))
{
await _dockerClient.PullImageAsync(imageTag, ct);
_metrics.RecordImageRegistryHit(language);
return imageTag;
}
// Build new image
_metrics.RecordImageCacheMiss(language);
var dockerfile = GenerateDockerfile(language, dependencies);
await _dockerClient.BuildImageAsync(dockerfile, imageTag, ct);
// Push to registry for other agents
await _registryClient.PushImageAsync(imageTag, ct);
return imageTag;
}
private string ComputeImageTag(
ScriptLanguage language,
IReadOnlyList<ScriptDependency> dependencies)
{
var depsHash = ComputeDependencyHash(dependencies);
return $"stella-runtime/{language.ToString().ToLower()}:{depsHash[..12]}";
}
private string GenerateDockerfile(
ScriptLanguage language,
IReadOnlyList<ScriptDependency> dependencies)
{
var baseImage = GetBaseImage(language);
var sb = new StringBuilder();
sb.AppendLine($"FROM {baseImage}");
sb.AppendLine("WORKDIR /scripts");
// Language-specific dependency installation
switch (language)
{
case ScriptLanguage.CSharp:
sb.AppendLine("# Pre-restore NuGet packages");
sb.AppendLine("COPY global.json Directory.Build.props ./");
sb.AppendLine("COPY *.csproj ./");
foreach (var dep in dependencies)
{
sb.AppendLine($"RUN dotnet add package {dep.Name} --version {dep.Version}");
}
sb.AppendLine("RUN dotnet restore");
sb.AppendLine("# Pre-compile common assemblies");
sb.AppendLine("RUN dotnet build --no-restore -c Release");
break;
case ScriptLanguage.Python:
sb.AppendLine("# Pre-install pip packages");
var requirements = string.Join("\n", dependencies.Select(d => $"{d.Name}=={d.Version}"));
sb.AppendLine($"RUN echo '{requirements}' > requirements.txt");
sb.AppendLine("RUN pip install --no-cache-dir -r requirements.txt");
break;
case ScriptLanguage.Java:
sb.AppendLine("# Pre-download Maven dependencies");
// Generate pom.xml with dependencies
sb.AppendLine("COPY pom.xml ./");
sb.AppendLine("RUN mvn dependency:go-offline");
break;
case ScriptLanguage.Go:
sb.AppendLine("# Pre-download Go modules");
sb.AppendLine("COPY go.mod go.sum ./");
sb.AppendLine("RUN go mod download");
break;
}
sb.AppendLine("VOLUME /scripts");
return sb.ToString();
}
}
4. Workflow Script Preloader
When a workflow starts, preload all scripts it will use:
public sealed class WorkflowScriptPreloader
{
public async Task PreloadWorkflowScriptsAsync(
Workflow workflow,
CancellationToken ct)
{
// Find all script steps in workflow
var scriptSteps = workflow.Steps
.Where(s => s.Type == StepType.Script)
.Select(s => s.ScriptId)
.Distinct()
.ToList();
if (scriptSteps.Count == 0) return;
_logger.LogInformation(
"Preloading {Count} scripts for workflow {WorkflowId}",
scriptSteps.Count, workflow.Id);
// Load scripts in parallel
var scripts = await Task.WhenAll(
scriptSteps.Select(id => _scriptRegistry.GetAsync(id, ct)));
// Group by language for efficient batching
var byLanguage = scripts.GroupBy(s => s.Language);
var preloadTasks = new List<Task>();
foreach (var group in byLanguage)
{
// Precompile scripts
foreach (var script in group)
{
preloadTasks.Add(_compilationCache.GetOrCompileAsync(script, ct));
}
// Ensure warm containers for this language
preloadTasks.Add(_containerPool.EnsureWarmAsync(
group.Key,
count: Math.Min(group.Count(), 3),
ct));
// Pre-build runtime images for unique dependency sets
var uniqueDepSets = group
.Select(s => s.Dependencies)
.Distinct(new DependencySetComparer())
.ToList();
foreach (var deps in uniqueDepSets)
{
preloadTasks.Add(_imageCache.GetOrBuildImageAsync(group.Key, deps, ct));
}
}
await Task.WhenAll(preloadTasks);
_logger.LogInformation(
"Preloading complete for workflow {WorkflowId}",
workflow.Id);
}
}
5. Agent-Side Caching
Each agent maintains local caches:
public sealed class AgentScriptCache
{
private readonly string _cachePath;
private readonly LruCache<string, CompiledScript> _compiledScripts;
private readonly LruCache<string, string> _runtimeImages;
public AgentScriptCache(AgentConfiguration config)
{
_cachePath = config.ScriptCachePath ?? "/var/cache/stella-scripts";
_compiledScripts = new LruCache<string, CompiledScript>(
maxSize: config.MaxCachedScripts ?? 100);
_runtimeImages = new LruCache<string, string>(
maxSize: config.MaxCachedImages ?? 20);
// Load persisted cache on startup
LoadPersistedCache();
}
public async Task WarmupAsync(CancellationToken ct)
{
// Pre-pull base images for all languages
var pullTasks = Enum.GetValues<ScriptLanguage>()
.Select(lang => _dockerClient.PullImageAsync(GetBaseImage(lang), ct));
await Task.WhenAll(pullTasks);
// Start warm container pool
await _containerPool.InitializeAsync(ct);
_logger.LogInformation("Agent script cache warmup complete");
}
}
Configuration
script_engine:
# Compilation cache
compilation_cache:
enabled: true
memory_cache_size_mb: 256
distributed_cache: redis # Valkey (Redis-compatible)
ttl_days: 7
# Warm container pool
container_pool:
enabled: true
languages:
csharp:
min_containers: 2
max_containers: 10
idle_timeout: 5m
python:
min_containers: 2
max_containers: 10
idle_timeout: 5m
java:
min_containers: 1
max_containers: 5
idle_timeout: 5m
go:
min_containers: 1
max_containers: 5
idle_timeout: 5m
bash:
min_containers: 2
max_containers: 10
idle_timeout: 3m
typescript:
min_containers: 2
max_containers: 8
idle_timeout: 5m
# Runtime image cache
image_cache:
enabled: true
registry: "registry.internal/stella-runtime"
local_cache_size_gb: 10
push_to_registry: true
# Workflow preloading
preloading:
enabled: true
parallel_preload: true
preload_on_workflow_create: true
# Agent-side cache
agent_cache:
path: "/var/cache/stella-scripts"
max_cached_scripts: 100
max_cached_images: 20
warmup_on_start: true
Metrics
# Cache performance
stella_script_compilation_cache_hits_total{language}
stella_script_compilation_cache_misses_total{language}
stella_script_compilation_duration_seconds{language, cached}
# Container pool
stella_container_pool_size{language}
stella_container_pool_hits_total{language}
stella_container_pool_misses_total{language}
stella_container_acquire_duration_seconds{language}
# Image cache
stella_image_cache_hits_total{language}
stella_image_cache_misses_total{language}
stella_image_build_duration_seconds{language}
# Preloading
stella_workflow_preload_duration_seconds
stella_workflow_preload_scripts_total
Test Strategy
Unit Tests
- Script validation logic
- Dependency resolution
- Mount generation
- Language detection
- Cache key computation
- Container pool management
Integration Tests
- Full script execution flow
- Monaco editor integration
- Language server communication
- Sample script execution
Security Tests
- Container isolation
- Resource limit enforcement
- Network isolation
- Path traversal prevention
Migration Path
Phase 1: Foundation (Week 1-2)
- Script registry
- Script model
- Basic CRUD operations
Phase 2: Execution (Week 3-4)
- Runtime image manager
- Script executor
- Mount generator
- Output collection
Phase 3: Monaco Editor (Week 5-6)
- Editor service
- Language server pool
- Completion providers
- Diagnostic providers
Phase 4: Library Manager (Week 7-8)
- Dependency resolvers
- Package caching
- Version management
Phase 5: Samples (Week 9-10)
- Sample library
- Per-language samples
- Sample clone workflow
Phase 6: Polish (Week 11-12)
- Performance optimization
- Security hardening
- Documentation