stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 1 Releases Wiki Activity

Rename Vexer to Excititor

Browse Source
This commit is contained in:
master
2025-10-18 20:00:46 +03:00
parent e2672ba968
commit dd66f58b00
263 changed files with 848 additions and 848 deletions
Download Patch File Download Diff File Expand all files Collapse all files

22
src/StellaOps.Excititor.Connectors.Abstractions/AGENTS.md Normal file
View File

@@ -0,0 +1,22 @@
# AGENTS
## Role
Defines shared connector infrastructure for Excititor, including base contexts, result contracts, configuration binding, and helper utilities reused by all connector plug-ins.
## Scope
- `IVexConnector` context implementation, raw store helpers, verification hooks, and telemetry utilities.
- Configuration primitives (YAML parsing, secrets handling guidelines) and options validation.
- Connector lifecycle helpers for retries, paging, `.well-known` discovery, and resume markers.
- Documentation for connector packaging, plugin manifest metadata, and DI registration (see `docs/dev/30_EXCITITOR_CONNECTOR_GUIDE.md` and `docs/dev/templates/excititor-connector/`).
## Participants
- All Excititor connector projects reference this module to obtain base classes and context services.
- WebService/Worker instantiate connectors via plugin loader leveraging abstractions defined here.
## Interfaces & contracts
- Connector context, result, and telemetry interfaces; `VexConnectorDescriptor`, `VexConnectorBase`, options binder/validators, authentication helpers.
- Utility classes for HTTP clients, throttling, and deterministic logging.
## In/Out of scope
In: shared abstractions, helper utilities, configuration binding, documentation for connector authors.
Out: provider-specific logic (implemented in individual connector modules), storage persistence, HTTP host code.
## Observability & security expectations
- Provide structured logging helpers, correlation IDs, and metrics instrumentation toggles for connectors.
- Enforce redaction of secrets in logs and config dumps.
## Tests
- Abstraction/unit tests will live in `../StellaOps.Excititor.Connectors.Abstractions.Tests`, covering default behaviors and sample harness.

12
src/StellaOps.Excititor.Connectors.Abstractions/IVexConnectorOptionsValidator.cs Normal file
View File

@@ -0,0 +1,12 @@
using System.Collections.Generic;
namespace StellaOps.Excititor.Connectors.Abstractions;
/// <summary>
/// Custom validator hook executed after connector options are bound.
/// </summary>
/// <typeparam name="TOptions">Connector-specific options type.</typeparam>
public interface IVexConnectorOptionsValidator<in TOptions>
{
void Validate(VexConnectorDescriptor descriptor, TOptions options, IList<string> errors);
}

17
src/StellaOps.Excititor.Connectors.Abstractions/StellaOps.Excititor.Connectors.Abstractions.csproj Normal file
View File

@@ -0,0 +1,17 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>net10.0</TargetFramework>
<LangVersion>preview</LangVersion>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
</PropertyGroup>
<ItemGroup>
<ProjectReference Include="..\StellaOps.Excititor.Core\StellaOps.Excititor.Core.csproj" />
</ItemGroup>
<ItemGroup>
<PackageReference Include="Microsoft.Extensions.Configuration" Version="8.0.0" />
<PackageReference Include="Microsoft.Extensions.Configuration.Binder" Version="8.0.1" />
<PackageReference Include="Microsoft.Extensions.Logging.Abstractions" Version="8.0.0" />
</ItemGroup>
</Project>

7
src/StellaOps.Excititor.Connectors.Abstractions/TASKS.md Normal file
View File

@@ -0,0 +1,7 @@
If you are working on this file you need to read docs/ARCHITECTURE_EXCITITOR.md and ./AGENTS.md).
# TASKS
| Task | Owner(s) | Depends on | Notes |
|---|---|---|---|
|EXCITITOR-CONN-ABS-01-001 – Connector context & base classes|Team Excititor Connectors|EXCITITOR-CORE-01-003|**DONE (2025-10-17)** – Added `StellaOps.Excititor.Connectors.Abstractions` project with `VexConnectorBase`, deterministic logging scopes, metadata builder helpers, and connector descriptors; docs updated to highlight the shared abstractions.|
|EXCITITOR-CONN-ABS-01-002 – YAML options & validation|Team Excititor Connectors|EXCITITOR-CONN-ABS-01-001|**DONE (2025-10-17)** – Delivered `VexConnectorOptionsBinder` + binder options/validators, environment-variable expansion, data-annotation checks, and custom validation hooks with documentation updates covering the workflow.|
|EXCITITOR-CONN-ABS-01-003 – Plugin packaging & docs|Team Excititor Connectors|EXCITITOR-CONN-ABS-01-001|**DONE (2025-10-17)** – Authored `docs/dev/30_EXCITITOR_CONNECTOR_GUIDE.md`, added quick-start template under `docs/dev/templates/excititor-connector/`, and updated module docs to reference the packaging workflow.|

99
src/StellaOps.Excititor.Connectors.Abstractions/VexConnectorBase.cs Normal file
View File

@@ -0,0 +1,99 @@
using System.Collections.Immutable;
using System.Security.Cryptography;
using Microsoft.Extensions.Logging;
using StellaOps.Excititor.Core;
namespace StellaOps.Excititor.Connectors.Abstractions;
/// <summary>
/// Convenience base class for implementing <see cref="IVexConnector" />.
/// </summary>
public abstract class VexConnectorBase : IVexConnector
{
protected VexConnectorBase(VexConnectorDescriptor descriptor, ILogger logger, TimeProvider? timeProvider = null)
{
Descriptor = descriptor ?? throw new ArgumentNullException(nameof(descriptor));
Logger = logger ?? throw new ArgumentNullException(nameof(logger));
TimeProvider = timeProvider ?? TimeProvider.System;
}
/// <inheritdoc />
public string Id => Descriptor.Id;
/// <inheritdoc />
public VexProviderKind Kind => Descriptor.Kind;
protected VexConnectorDescriptor Descriptor { get; }
protected ILogger Logger { get; }
protected TimeProvider TimeProvider { get; }
protected DateTimeOffset UtcNow() => TimeProvider.GetUtcNow();
protected VexRawDocument CreateRawDocument(
VexDocumentFormat format,
Uri sourceUri,
ReadOnlyMemory<byte> content,
ImmutableDictionary<string, string>? metadata = null)
{
if (sourceUri is null)
{
throw new ArgumentNullException(nameof(sourceUri));
}
var digest = ComputeSha256(content.Span);
var captured = TimeProvider.GetUtcNow();
return new VexRawDocument(
Descriptor.Id,
format,
sourceUri,
captured,
digest,
content,
metadata ?? ImmutableDictionary<string, string>.Empty);
}
protected IDisposable BeginConnectorScope(string operation, IReadOnlyDictionary<string, object?>? metadata = null)
=> VexConnectorLogScope.Begin(Logger, Descriptor, operation, metadata);
protected void LogConnectorEvent(LogLevel level, string eventName, string message, IReadOnlyDictionary<string, object?>? metadata = null, Exception? exception = null)
{
using var scope = BeginConnectorScope(eventName, metadata);
if (exception is null)
{
Logger.Log(level, "{Message}", message);
}
else
{
Logger.Log(level, exception, "{Message}", message);
}
}
protected ImmutableDictionary<string, string> BuildMetadata(Action<VexConnectorMetadataBuilder> configure)
{
ArgumentNullException.ThrowIfNull(configure);
var builder = new VexConnectorMetadataBuilder();
configure(builder);
return builder.Build();
}
private static string ComputeSha256(ReadOnlySpan<byte> content)
{
Span<byte> buffer = stackalloc byte[32];
if (SHA256.TryHashData(content, buffer, out _))
{
return "sha256:" + Convert.ToHexString(buffer).ToLowerInvariant();
}
using var sha = SHA256.Create();
var hash = sha.ComputeHash(content.ToArray());
return "sha256:" + Convert.ToHexString(hash).ToLowerInvariant();
}
public abstract ValueTask ValidateAsync(VexConnectorSettings settings, CancellationToken cancellationToken);
public abstract IAsyncEnumerable<VexRawDocument> FetchAsync(VexConnectorContext context, CancellationToken cancellationToken);
public abstract ValueTask<VexClaimBatch> NormalizeAsync(VexRawDocument document, CancellationToken cancellationToken);
}

54
src/StellaOps.Excititor.Connectors.Abstractions/VexConnectorDescriptor.cs Normal file
View File

@@ -0,0 +1,54 @@
using System.Collections.Immutable;
using StellaOps.Excititor.Core;
namespace StellaOps.Excititor.Connectors.Abstractions;
/// <summary>
/// Static descriptor for a Excititor connector plug-in.
/// </summary>
public sealed record VexConnectorDescriptor
{
public VexConnectorDescriptor(string id, VexProviderKind kind, string displayName)
{
if (string.IsNullOrWhiteSpace(id))
{
throw new ArgumentException("Connector id must be provided.", nameof(id));
}
Id = id;
Kind = kind;
DisplayName = string.IsNullOrWhiteSpace(displayName) ? id : displayName;
}
/// <summary>
/// Stable connector identifier (matches provider id).
/// </summary>
public string Id { get; }
/// <summary>
/// Provider kind served by the connector.
/// </summary>
public VexProviderKind Kind { get; }
/// <summary>
/// Human friendly name used in logs/diagnostics.
/// </summary>
public string DisplayName { get; }
/// <summary>
/// Optional friendly description.
/// </summary>
public string? Description { get; init; }
/// <summary>
/// Document formats the connector is expected to emit.
/// </summary>
public ImmutableArray<VexDocumentFormat> SupportedFormats { get; init; } = ImmutableArray<VexDocumentFormat>.Empty;
/// <summary>
/// Optional tags surfaced in diagnostics (e.g. "beta", "offline").
/// </summary>
public ImmutableArray<string> Tags { get; init; } = ImmutableArray<string>.Empty;
public override string ToString() => $"{Id} ({Kind})";
}

50
src/StellaOps.Excititor.Connectors.Abstractions/VexConnectorLogScope.cs Normal file
View File

@@ -0,0 +1,50 @@
using System.Linq;
using Microsoft.Extensions.Logging;
using StellaOps.Excititor.Core;
namespace StellaOps.Excititor.Connectors.Abstractions;
/// <summary>
/// Helper to establish deterministic logging scopes for connector operations.
/// </summary>
public static class VexConnectorLogScope
{
public static IDisposable Begin(ILogger logger, VexConnectorDescriptor descriptor, string operation, IReadOnlyDictionary<string, object?>? metadata = null)
{
ArgumentNullException.ThrowIfNull(logger);
ArgumentNullException.ThrowIfNull(descriptor);
ArgumentException.ThrowIfNullOrEmpty(operation);
var scopeValues = new List<KeyValuePair<string, object?>>
{
new("vex.connector.id", descriptor.Id),
new("vex.connector.kind", descriptor.Kind.ToString()),
new("vex.connector.operation", operation),
};
if (!string.Equals(descriptor.DisplayName, descriptor.Id, StringComparison.Ordinal))
{
scopeValues.Add(new KeyValuePair<string, object?>("vex.connector.displayName", descriptor.DisplayName));
}
if (!string.IsNullOrWhiteSpace(descriptor.Description))
{
scopeValues.Add(new KeyValuePair<string, object?>("vex.connector.description", descriptor.Description));
}
if (!descriptor.Tags.IsDefaultOrEmpty)
{
scopeValues.Add(new KeyValuePair<string, object?>("vex.connector.tags", string.Join(",", descriptor.Tags)));
}
if (metadata is not null)
{
foreach (var kvp in metadata.OrderBy(static pair => pair.Key, StringComparer.Ordinal))
{
scopeValues.Add(new KeyValuePair<string, object?>($"vex.{kvp.Key}", kvp.Value));
}
}
return logger.BeginScope(scopeValues)!;
}
}

37
src/StellaOps.Excititor.Connectors.Abstractions/VexConnectorMetadataBuilder.cs Normal file
View File

@@ -0,0 +1,37 @@
using System.Collections.Immutable;
namespace StellaOps.Excititor.Connectors.Abstractions;
/// <summary>
/// Builds deterministic metadata dictionaries for raw documents and logging scopes.
/// </summary>
public sealed class VexConnectorMetadataBuilder
{
private readonly SortedDictionary<string, string> _values = new(StringComparer.Ordinal);
public VexConnectorMetadataBuilder Add(string key, string? value)
{
if (!string.IsNullOrWhiteSpace(key) && !string.IsNullOrWhiteSpace(value))
{
_values[key] = value!;
}
return this;
}
public VexConnectorMetadataBuilder Add(string key, DateTimeOffset value)
=> Add(key, value.ToUniversalTime().ToString("O"));
public VexConnectorMetadataBuilder AddRange(IEnumerable<KeyValuePair<string, string?>> items)
{
foreach (var item in items)
{
Add(item.Key, item.Value);
}
return this;
}
public ImmutableDictionary<string, string> Build()
=> _values.ToImmutableDictionary(static pair => pair.Key, static pair => pair.Value, StringComparer.Ordinal);
}

157
src/StellaOps.Excititor.Connectors.Abstractions/VexConnectorOptionsBinder.cs Normal file
View File

@@ -0,0 +1,157 @@
using System.Collections.Immutable;
using System.ComponentModel.DataAnnotations;
using System.Linq;
using Microsoft.Extensions.Configuration;
using StellaOps.Excititor.Core;
namespace StellaOps.Excititor.Connectors.Abstractions;
/// <summary>
/// Provides strongly typed binding and validation for connector options.
/// </summary>
public static class VexConnectorOptionsBinder
{
public static TOptions Bind<TOptions>(
VexConnectorDescriptor descriptor,
VexConnectorSettings settings,
VexConnectorOptionsBinderOptions? options = null,
IEnumerable<IVexConnectorOptionsValidator<TOptions>>? validators = null)
where TOptions : class, new()
{
ArgumentNullException.ThrowIfNull(descriptor);
ArgumentNullException.ThrowIfNull(settings);
var binderSettings = options ?? new VexConnectorOptionsBinderOptions();
var transformed = TransformValues(settings, binderSettings);
var configuration = BuildConfiguration(transformed);
var result = new TOptions();
var errors = new List<string>();
try
{
configuration.Bind(
result,
binderOptions => binderOptions.ErrorOnUnknownConfiguration = !binderSettings.AllowUnknownKeys);
}
catch (InvalidOperationException ex) when (!binderSettings.AllowUnknownKeys)
{
errors.Add(ex.Message);
}
binderSettings.PostConfigure?.Invoke(result);
if (binderSettings.ValidateDataAnnotations)
{
ValidateDataAnnotations(result, errors);
}
if (validators is not null)
{
foreach (var validator in validators)
{
validator?.Validate(descriptor, result, errors);
}
}
if (errors.Count > 0)
{
throw new VexConnectorOptionsValidationException(descriptor.Id, errors);
}
return result;
}
private static ImmutableDictionary<string, string?> TransformValues(
VexConnectorSettings settings,
VexConnectorOptionsBinderOptions binderOptions)
{
var builder = ImmutableDictionary.CreateBuilder<string, string?>(StringComparer.OrdinalIgnoreCase);
foreach (var kvp in settings.Values)
{
var value = kvp.Value;
if (binderOptions.TrimWhitespace && value is not null)
{
value = value.Trim();
}
if (binderOptions.TreatEmptyAsNull && string.IsNullOrEmpty(value))
{
value = null;
}
if (value is not null && binderOptions.ExpandEnvironmentVariables)
{
value = Environment.ExpandEnvironmentVariables(value);
}
if (binderOptions.ValueTransformer is not null)
{
value = binderOptions.ValueTransformer.Invoke(kvp.Key, value);
}
builder[kvp.Key] = value;
}
return builder.ToImmutable();
}
private static IConfiguration BuildConfiguration(ImmutableDictionary<string, string?> values)
{
var sources = new List<KeyValuePair<string, string?>>();
foreach (var kvp in values)
{
if (kvp.Value is not null)
{
sources.Add(new KeyValuePair<string, string?>(kvp.Key, kvp.Value));
}
}
var configurationBuilder = new ConfigurationBuilder();
configurationBuilder.Add(new DictionaryConfigurationSource(sources));
return configurationBuilder.Build();
}
private static void ValidateDataAnnotations<TOptions>(TOptions options, IList<string> errors)
{
var validationResults = new List<ValidationResult>();
var validationContext = new ValidationContext(options!);
if (!Validator.TryValidateObject(options!, validationContext, validationResults, validateAllProperties: true))
{
foreach (var validationResult in validationResults)
{
if (!string.IsNullOrWhiteSpace(validationResult.ErrorMessage))
{
errors.Add(validationResult.ErrorMessage);
}
}
}
}
private sealed class DictionaryConfigurationSource : IConfigurationSource
{
private readonly IReadOnlyList<KeyValuePair<string, string?>> _data;
public DictionaryConfigurationSource(IEnumerable<KeyValuePair<string, string?>> data)
{
_data = data?.ToList() ?? new List<KeyValuePair<string, string?>>();
}
public IConfigurationProvider Build(IConfigurationBuilder builder) => new DictionaryConfigurationProvider(_data);
}
private sealed class DictionaryConfigurationProvider : ConfigurationProvider
{
public DictionaryConfigurationProvider(IEnumerable<KeyValuePair<string, string?>> data)
{
foreach (var pair in data)
{
if (pair.Value is not null)
{
Data[pair.Key] = pair.Value;
}
}
}
}
}

45
src/StellaOps.Excititor.Connectors.Abstractions/VexConnectorOptionsBinderOptions.cs Normal file
View File

@@ -0,0 +1,45 @@
namespace StellaOps.Excititor.Connectors.Abstractions;
/// <summary>
/// Customisation options for connector options binding.
/// </summary>
public sealed class VexConnectorOptionsBinderOptions
{
/// <summary>
/// Indicates whether environment variables should be expanded in option values.
/// Defaults to <c>true</c>.
/// </summary>
public bool ExpandEnvironmentVariables { get; set; } = true;
/// <summary>
/// When <c>true</c> the binder trims whitespace around option values.
/// </summary>
public bool TrimWhitespace { get; set; } = true;
/// <summary>
/// Converts empty strings to <c>null</c> before binding. Default: <c>true</c>.
/// </summary>
public bool TreatEmptyAsNull { get; set; } = true;
/// <summary>
/// When <c>false</c>, binding fails if unknown configuration keys are provided.
/// Default: <c>true</c> (permitting unknown keys).
/// </summary>
public bool AllowUnknownKeys { get; set; } = true;
/// <summary>
/// Enables <see cref="System.ComponentModel.DataAnnotations"/> validation after binding.
/// Default: <c>true</c>.
/// </summary>
public bool ValidateDataAnnotations { get; set; } = true;
/// <summary>
/// Optional post-configuration callback executed after binding.
/// </summary>
public Action<object>? PostConfigure { get; set; }
/// <summary>
/// Optional hook to transform raw configuration values before binding.
/// </summary>
public Func<string, string?, string?>? ValueTransformer { get; set; }
}

36
src/StellaOps.Excititor.Connectors.Abstractions/VexConnectorOptionsValidationException.cs Normal file
View File

@@ -0,0 +1,36 @@
using System.Collections.Immutable;
namespace StellaOps.Excititor.Connectors.Abstractions;
public sealed class VexConnectorOptionsValidationException : Exception
{
public VexConnectorOptionsValidationException(
string connectorId,
IEnumerable<string> errors)
: base(BuildMessage(connectorId, errors))
{
ConnectorId = connectorId;
Errors = errors?.ToImmutableArray() ?? ImmutableArray<string>.Empty;
}
public string ConnectorId { get; }
public ImmutableArray<string> Errors { get; }
private static string BuildMessage(string connectorId, IEnumerable<string> errors)
{
var builder = new System.Text.StringBuilder();
builder.Append("Connector options validation failed for '");
builder.Append(connectorId);
builder.Append("'.");
var list = errors?.ToImmutableArray() ?? ImmutableArray<string>.Empty;
if (!list.IsDefaultOrEmpty)
{
builder.Append(" Errors: ");
builder.Append(string.Join("; ", list));
}
return builder.ToString();
}
}