feat: Add initial implementation of Vulnerability Resolver Jobs
Some checks failed
Docs CI / lint-and-preview (push) Has been cancelled

- Created project for StellaOps.Scanner.Analyzers.Native.Tests with necessary dependencies.
- Documented roles and guidelines in AGENTS.md for Scheduler module.
- Implemented IResolverJobService interface and InMemoryResolverJobService for handling resolver jobs.
- Added ResolverBacklogNotifier and ResolverBacklogService for monitoring job metrics.
- Developed API endpoints for managing resolver jobs and retrieving metrics.
- Defined models for resolver job requests and responses.
- Integrated dependency injection for resolver job services.
- Implemented ImpactIndexSnapshot for persisting impact index data.
- Introduced SignalsScoringOptions for configurable scoring weights in reachability scoring.
- Added unit tests for ReachabilityScoringService and RuntimeFactsIngestionService.
- Created dotnet-filter.sh script to handle command-line arguments for dotnet.
- Established nuget-prime project for managing package downloads.
This commit is contained in:
master
2025-11-18 07:52:15 +02:00
parent e69b57d467
commit 8355e2ff75
299 changed files with 13293 additions and 2444 deletions

View File

@@ -1,37 +1,46 @@
# AGENTS
## Role
ASP.NET Minimal API surface for Excititor ingest, provider administration, reconciliation, export, and verification flows.
# Excititor WebService Charter
## Mission
Expose Excititor APIs (console VEX views, graph/Vuln Explorer feeds, observation intake/health) while honoring the Aggregation-Only Contract (no consensus/severity logic in this service).
## Scope
- Program bootstrap, DI wiring for connectors/normalizers/export/attestation/policy/storage.
- HTTP endpoints `/excititor/*` with authentication, authorization scopes, request validation, and deterministic responses.
- Job orchestration bridges for Worker hand-off (when co-hosted) and offline-friendly configuration.
- Observability (structured logs, metrics, tracing) aligned with StellaOps conventions.
- Optional/minor DI dependencies on minimal APIs must be declared with `[FromServices] SomeType? service = null` parameters so endpoint tests do not require bespoke service registrations.
## Participants
- StellaOps.Cli sends `excititor` verbs to this service via token-authenticated HTTPS.
- Worker receives scheduled jobs and uses shared infrastructure via common DI extensions.
- Authority service provides tokens; WebService enforces scopes before executing operations.
## Interfaces & contracts
- DTOs for ingest/export requests, run metadata, provider management.
- Background job interfaces for ingest/resume/reconcile triggering.
- Health/status endpoints exposing pull/export history and current policy revision.
## In/Out of scope
In: HTTP hosting, request orchestration, DI composition, auth/authorization, logging.
Out: long-running ingestion loops (Worker), export rendering (Export module), connector implementations.
## Observability & security expectations
- Enforce bearer token scopes, enforce audit logging (request/response correlation IDs, provider IDs).
- Emit structured events for ingest runs, export invocations, attestation references.
- Provide built-in counters/histograms for latency and throughput.
## Tests
- Minimal API contract/unit tests and integration harness will live in `../StellaOps.Excititor.WebService.Tests`.
- Working directory: `src/Excititor/StellaOps.Excititor.WebService`
- HTTP APIs, DTOs, controllers, authz filters, composition root, telemetry hooks.
- Wiring to Core/Storage libraries; no direct policy or consensus logic.
## Required Reading
- `docs/modules/excititor/architecture.md`
- `docs/modules/platform/architecture-overview.md`
- `docs/modules/excititor/README.md#latest-updates`
- `docs/modules/excititor/vex_observations.md`
- `docs/ingestion/aggregation-only-contract.md`
- `docs/modules/excititor/implementation_plan.md`
## Working Agreement
- 1. Update task status to `DOING`/`DONE` in both correspoding sprint file `/docs/implplan/SPRINT_*.md` and the local `TASKS.md` when you start or finish work.
- 2. Review this charter and the Required Reading documents before coding; confirm prerequisites are met.
- 3. Keep changes deterministic (stable ordering, timestamps, hashes) and align with offline/air-gap expectations.
- 4. Coordinate doc updates, tests, and cross-guild communication whenever contracts or workflows change.
- 5. Revert to `TODO` if you pause the task without shipping changes; leave notes in commit/PR descriptions for context.
## Roles
- Backend developer (.NET 10 / C# preview).
- QA automation (integration + API contract tests).
## Working Agreements
1. Update sprint `Delivery Tracker` when tasks move TODO→DOING→DONE/BLOCKED; mirror notes in Execution Log.
2. Keep APIs aggregation-only: persist raw observations, provenance, and precedence pointers; never merge/weight/consensus here.
3. Enforce tenant scoping and RBAC on all endpoints; default-deny for cross-tenant data.
4. Offline-first: no external network calls; rely on cached/mirrored feeds only.
5. Observability: structured logs, counters, optional OTEL traces behind configuration flags.
## Testing
- Prefer deterministic API/integration tests under `__Tests` with seeded Mongo fixtures.
- Verify RBAC/tenant isolation, idempotent ingestion, and stable ordering of VEX aggregates.
- Use ISO-8601 UTC timestamps and stable sorting in responses; assert on content hashes where applicable.
## Determinism & Data
- MongoDB is the canonical store; never apply consensus transformations before persistence.
- Ensure paged/list endpoints use explicit sort keys (e.g., vendor, upstreamId, version, createdUtc).
- Avoid nondeterministic clocks/randomness; inject clocks and GUID providers for tests.
## Boundaries
- Do not modify Policy Engine or Cartographer schemas from here; consume published contracts only.
- Configuration via appsettings/environment; no hard-coded secrets.
## Ready-to-Start Checklist
- Required docs reviewed.
- Test database/fixtures prepared (no external dependencies).
- Feature flags defined for new endpoints before exposing them.

View File

@@ -471,7 +471,8 @@ app.MapGet("/v1/vex/observations/{vulnerabilityId}/{productKey}", async (
var providerFilter = BuildStringFilterSet(context.Request.Query["providerId"]);
var statusFilter = BuildStatusFilter(context.Request.Query["status"]);
var since = ParseSinceTimestamp(context.Request.Query["since"]);
var limit = ResolveLimit(context.Request.Query["limit"], defaultValue: 200, min: 1, max: 500);
// Evidence chunks follow doc limits: default 500, max 2000.
var limit = ResolveLimit(context.Request.Query["limit"], defaultValue: 500, min: 1, max: 2000);
var request = new VexObservationProjectionRequest(
tenant,
@@ -514,6 +515,10 @@ app.MapGet("/v1/vex/observations/{vulnerabilityId}/{productKey}", async (
result.Truncated,
statements);
// Set total/truncated headers for clients (spec: Excititor-Results-*).
context.Response.Headers["Excititor-Results-Total"] = result.TotalCount.ToString(CultureInfo.InvariantCulture);
context.Response.Headers["Excititor-Results-Truncated"] = result.Truncated ? "true" : "false";
return Results.Json(response);
});
@@ -562,11 +567,21 @@ app.MapGet("/v1/vex/evidence/chunks", async (
}
catch (OperationCanceledException)
{
EvidenceTelemetry.RecordChunkOutcome(tenant, "cancelled");
return Results.StatusCode(StatusCodes.Status499ClientClosedRequest);
}
catch
{
EvidenceTelemetry.RecordChunkOutcome(tenant, "error");
throw;
}
context.Response.Headers["X-Total-Count"] = result.TotalCount.ToString(CultureInfo.InvariantCulture);
context.Response.Headers["X-Truncated"] = result.Truncated ? "true" : "false";
EvidenceTelemetry.RecordChunkOutcome(tenant, "success", result.Chunks.Count, result.Truncated);
EvidenceTelemetry.RecordChunkSignatureStatus(tenant, result.Chunks);
// Align headers with published contract.
context.Response.Headers["Excititor-Results-Total"] = result.TotalCount.ToString(CultureInfo.InvariantCulture);
context.Response.Headers["Excititor-Results-Truncated"] = result.Truncated ? "true" : "false";
context.Response.ContentType = "application/x-ndjson";
var options = new JsonSerializerOptions(JsonSerializerDefaults.Web);

View File

@@ -1,7 +1,9 @@
using System;
using System.Collections.Generic;
using System.Diagnostics.Metrics;
using System.Linq;
using StellaOps.Excititor.Core.Aoc;
using StellaOps.Excititor.WebService.Contracts;
using StellaOps.Excititor.WebService.Services;
namespace StellaOps.Excititor.WebService.Telemetry;
@@ -24,6 +26,18 @@ internal static class EvidenceTelemetry
unit: "statements",
description: "Distribution of statements returned per observation projection request.");
private static readonly Counter<long> EvidenceRequestCounter =
Meter.CreateCounter<long>(
"excititor.vex.evidence.requests",
unit: "requests",
description: "Number of evidence chunk requests handled by the evidence APIs.");
private static readonly Histogram<int> EvidenceChunkHistogram =
Meter.CreateHistogram<int>(
"excititor.vex.evidence.chunk_count",
unit: "chunks",
description: "Distribution of evidence chunks streamed per request.");
private static readonly Counter<long> SignatureStatusCounter =
Meter.CreateCounter<long>(
"excititor.vex.signature.status",
@@ -53,13 +67,27 @@ internal static class EvidenceTelemetry
return;
}
ObservationStatementHistogram.Record(
returnedCount,
new[]
{
new KeyValuePair<string, object?>("tenant", normalizedTenant),
new KeyValuePair<string, object?>("outcome", outcome),
});
ObservationStatementHistogram.Record(returnedCount, tags);
}
public static void RecordChunkOutcome(string? tenant, string outcome, int chunkCount = 0, bool truncated = false)
{
var normalizedTenant = NormalizeTenant(tenant);
var tags = new[]
{
new KeyValuePair<string, object?>("tenant", normalizedTenant),
new KeyValuePair<string, object?>("outcome", outcome),
new KeyValuePair<string, object?>("truncated", truncated),
};
EvidenceRequestCounter.Add(1, tags);
if (!string.Equals(outcome, "success", StringComparison.OrdinalIgnoreCase))
{
return;
}
EvidenceChunkHistogram.Record(chunkCount, tags);
}
public static void RecordSignatureStatus(string? tenant, IReadOnlyList<VexObservationStatementProjection> statements)
@@ -72,6 +100,7 @@ internal static class EvidenceTelemetry
var normalizedTenant = NormalizeTenant(tenant);
var missing = 0;
var unverified = 0;
var verified = 0;
foreach (var statement in statements)
{
@@ -86,6 +115,10 @@ internal static class EvidenceTelemetry
{
unverified++;
}
else
{
verified++;
}
}
if (missing > 0)
@@ -103,11 +136,62 @@ internal static class EvidenceTelemetry
{
SignatureStatusCounter.Add(
unverified,
new[]
{
new KeyValuePair<string, object?>("tenant", normalizedTenant),
new KeyValuePair<string, object?>("status", "unverified"),
});
BuildSignatureTags(normalizedTenant, "unverified"));
}
if (verified > 0)
{
SignatureStatusCounter.Add(
verified,
BuildSignatureTags(normalizedTenant, "verified"));
}
}
public static void RecordChunkSignatureStatus(string? tenant, IReadOnlyList<VexEvidenceChunkResponse> chunks)
{
if (chunks is null || chunks.Count == 0)
{
return;
}
var normalizedTenant = NormalizeTenant(tenant);
var unsigned = 0;
var unverified = 0;
var verified = 0;
foreach (var chunk in chunks)
{
var signature = chunk.Signature;
if (signature is null)
{
unsigned++;
continue;
}
if (signature.VerifiedAt is null)
{
unverified++;
}
else
{
verified++;
}
}
if (unsigned > 0)
{
SignatureStatusCounter.Add(unsigned, BuildSignatureTags(normalizedTenant, "unsigned"));
}
if (unverified > 0)
{
SignatureStatusCounter.Add(unverified, BuildSignatureTags(normalizedTenant, "unverified"));
}
if (verified > 0)
{
SignatureStatusCounter.Add(verified, BuildSignatureTags(normalizedTenant, "verified"));
}
}
@@ -151,4 +235,11 @@ internal static class EvidenceTelemetry
private static string NormalizeSurface(string? surface)
=> string.IsNullOrWhiteSpace(surface) ? "unknown" : surface.ToLowerInvariant();
private static KeyValuePair<string, object?>[] BuildSignatureTags(string tenant, string status)
=> new[]
{
new KeyValuePair<string, object?>("tenant", tenant),
new KeyValuePair<string, object?>("status", status),
};
}