feat: Add initial implementation of Vulnerability Resolver Jobs
Some checks failed
Docs CI / lint-and-preview (push) Has been cancelled
Some checks failed
Docs CI / lint-and-preview (push) Has been cancelled
- Created project for StellaOps.Scanner.Analyzers.Native.Tests with necessary dependencies. - Documented roles and guidelines in AGENTS.md for Scheduler module. - Implemented IResolverJobService interface and InMemoryResolverJobService for handling resolver jobs. - Added ResolverBacklogNotifier and ResolverBacklogService for monitoring job metrics. - Developed API endpoints for managing resolver jobs and retrieving metrics. - Defined models for resolver job requests and responses. - Integrated dependency injection for resolver job services. - Implemented ImpactIndexSnapshot for persisting impact index data. - Introduced SignalsScoringOptions for configurable scoring weights in reachability scoring. - Added unit tests for ReachabilityScoringService and RuntimeFactsIngestionService. - Created dotnet-filter.sh script to handle command-line arguments for dotnet. - Established nuget-prime project for managing package downloads.
This commit is contained in:
@@ -1,37 +1,46 @@
|
||||
# AGENTS
|
||||
## Role
|
||||
ASP.NET Minimal API surface for Excititor ingest, provider administration, reconciliation, export, and verification flows.
|
||||
# Excititor WebService Charter
|
||||
|
||||
## Mission
|
||||
Expose Excititor APIs (console VEX views, graph/Vuln Explorer feeds, observation intake/health) while honoring the Aggregation-Only Contract (no consensus/severity logic in this service).
|
||||
|
||||
## Scope
|
||||
- Program bootstrap, DI wiring for connectors/normalizers/export/attestation/policy/storage.
|
||||
- HTTP endpoints `/excititor/*` with authentication, authorization scopes, request validation, and deterministic responses.
|
||||
- Job orchestration bridges for Worker hand-off (when co-hosted) and offline-friendly configuration.
|
||||
- Observability (structured logs, metrics, tracing) aligned with StellaOps conventions.
|
||||
- Optional/minor DI dependencies on minimal APIs must be declared with `[FromServices] SomeType? service = null` parameters so endpoint tests do not require bespoke service registrations.
|
||||
## Participants
|
||||
- StellaOps.Cli sends `excititor` verbs to this service via token-authenticated HTTPS.
|
||||
- Worker receives scheduled jobs and uses shared infrastructure via common DI extensions.
|
||||
- Authority service provides tokens; WebService enforces scopes before executing operations.
|
||||
## Interfaces & contracts
|
||||
- DTOs for ingest/export requests, run metadata, provider management.
|
||||
- Background job interfaces for ingest/resume/reconcile triggering.
|
||||
- Health/status endpoints exposing pull/export history and current policy revision.
|
||||
## In/Out of scope
|
||||
In: HTTP hosting, request orchestration, DI composition, auth/authorization, logging.
|
||||
Out: long-running ingestion loops (Worker), export rendering (Export module), connector implementations.
|
||||
## Observability & security expectations
|
||||
- Enforce bearer token scopes, enforce audit logging (request/response correlation IDs, provider IDs).
|
||||
- Emit structured events for ingest runs, export invocations, attestation references.
|
||||
- Provide built-in counters/histograms for latency and throughput.
|
||||
## Tests
|
||||
- Minimal API contract/unit tests and integration harness will live in `../StellaOps.Excititor.WebService.Tests`.
|
||||
- Working directory: `src/Excititor/StellaOps.Excititor.WebService`
|
||||
- HTTP APIs, DTOs, controllers, authz filters, composition root, telemetry hooks.
|
||||
- Wiring to Core/Storage libraries; no direct policy or consensus logic.
|
||||
|
||||
## Required Reading
|
||||
- `docs/modules/excititor/architecture.md`
|
||||
- `docs/modules/platform/architecture-overview.md`
|
||||
- `docs/modules/excititor/README.md#latest-updates`
|
||||
- `docs/modules/excititor/vex_observations.md`
|
||||
- `docs/ingestion/aggregation-only-contract.md`
|
||||
- `docs/modules/excititor/implementation_plan.md`
|
||||
|
||||
## Working Agreement
|
||||
- 1. Update task status to `DOING`/`DONE` in both correspoding sprint file `/docs/implplan/SPRINT_*.md` and the local `TASKS.md` when you start or finish work.
|
||||
- 2. Review this charter and the Required Reading documents before coding; confirm prerequisites are met.
|
||||
- 3. Keep changes deterministic (stable ordering, timestamps, hashes) and align with offline/air-gap expectations.
|
||||
- 4. Coordinate doc updates, tests, and cross-guild communication whenever contracts or workflows change.
|
||||
- 5. Revert to `TODO` if you pause the task without shipping changes; leave notes in commit/PR descriptions for context.
|
||||
## Roles
|
||||
- Backend developer (.NET 10 / C# preview).
|
||||
- QA automation (integration + API contract tests).
|
||||
|
||||
## Working Agreements
|
||||
1. Update sprint `Delivery Tracker` when tasks move TODO→DOING→DONE/BLOCKED; mirror notes in Execution Log.
|
||||
2. Keep APIs aggregation-only: persist raw observations, provenance, and precedence pointers; never merge/weight/consensus here.
|
||||
3. Enforce tenant scoping and RBAC on all endpoints; default-deny for cross-tenant data.
|
||||
4. Offline-first: no external network calls; rely on cached/mirrored feeds only.
|
||||
5. Observability: structured logs, counters, optional OTEL traces behind configuration flags.
|
||||
|
||||
## Testing
|
||||
- Prefer deterministic API/integration tests under `__Tests` with seeded Mongo fixtures.
|
||||
- Verify RBAC/tenant isolation, idempotent ingestion, and stable ordering of VEX aggregates.
|
||||
- Use ISO-8601 UTC timestamps and stable sorting in responses; assert on content hashes where applicable.
|
||||
|
||||
## Determinism & Data
|
||||
- MongoDB is the canonical store; never apply consensus transformations before persistence.
|
||||
- Ensure paged/list endpoints use explicit sort keys (e.g., vendor, upstreamId, version, createdUtc).
|
||||
- Avoid nondeterministic clocks/randomness; inject clocks and GUID providers for tests.
|
||||
|
||||
## Boundaries
|
||||
- Do not modify Policy Engine or Cartographer schemas from here; consume published contracts only.
|
||||
- Configuration via appsettings/environment; no hard-coded secrets.
|
||||
|
||||
## Ready-to-Start Checklist
|
||||
- Required docs reviewed.
|
||||
- Test database/fixtures prepared (no external dependencies).
|
||||
- Feature flags defined for new endpoints before exposing them.
|
||||
|
||||
@@ -471,7 +471,8 @@ app.MapGet("/v1/vex/observations/{vulnerabilityId}/{productKey}", async (
|
||||
var providerFilter = BuildStringFilterSet(context.Request.Query["providerId"]);
|
||||
var statusFilter = BuildStatusFilter(context.Request.Query["status"]);
|
||||
var since = ParseSinceTimestamp(context.Request.Query["since"]);
|
||||
var limit = ResolveLimit(context.Request.Query["limit"], defaultValue: 200, min: 1, max: 500);
|
||||
// Evidence chunks follow doc limits: default 500, max 2000.
|
||||
var limit = ResolveLimit(context.Request.Query["limit"], defaultValue: 500, min: 1, max: 2000);
|
||||
|
||||
var request = new VexObservationProjectionRequest(
|
||||
tenant,
|
||||
@@ -514,6 +515,10 @@ app.MapGet("/v1/vex/observations/{vulnerabilityId}/{productKey}", async (
|
||||
result.Truncated,
|
||||
statements);
|
||||
|
||||
// Set total/truncated headers for clients (spec: Excititor-Results-*).
|
||||
context.Response.Headers["Excititor-Results-Total"] = result.TotalCount.ToString(CultureInfo.InvariantCulture);
|
||||
context.Response.Headers["Excititor-Results-Truncated"] = result.Truncated ? "true" : "false";
|
||||
|
||||
return Results.Json(response);
|
||||
});
|
||||
|
||||
@@ -562,11 +567,21 @@ app.MapGet("/v1/vex/evidence/chunks", async (
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
EvidenceTelemetry.RecordChunkOutcome(tenant, "cancelled");
|
||||
return Results.StatusCode(StatusCodes.Status499ClientClosedRequest);
|
||||
}
|
||||
catch
|
||||
{
|
||||
EvidenceTelemetry.RecordChunkOutcome(tenant, "error");
|
||||
throw;
|
||||
}
|
||||
|
||||
context.Response.Headers["X-Total-Count"] = result.TotalCount.ToString(CultureInfo.InvariantCulture);
|
||||
context.Response.Headers["X-Truncated"] = result.Truncated ? "true" : "false";
|
||||
EvidenceTelemetry.RecordChunkOutcome(tenant, "success", result.Chunks.Count, result.Truncated);
|
||||
EvidenceTelemetry.RecordChunkSignatureStatus(tenant, result.Chunks);
|
||||
|
||||
// Align headers with published contract.
|
||||
context.Response.Headers["Excititor-Results-Total"] = result.TotalCount.ToString(CultureInfo.InvariantCulture);
|
||||
context.Response.Headers["Excititor-Results-Truncated"] = result.Truncated ? "true" : "false";
|
||||
context.Response.ContentType = "application/x-ndjson";
|
||||
|
||||
var options = new JsonSerializerOptions(JsonSerializerDefaults.Web);
|
||||
|
||||
@@ -1,7 +1,9 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Diagnostics.Metrics;
|
||||
using System.Linq;
|
||||
using StellaOps.Excititor.Core.Aoc;
|
||||
using StellaOps.Excititor.WebService.Contracts;
|
||||
using StellaOps.Excititor.WebService.Services;
|
||||
|
||||
namespace StellaOps.Excititor.WebService.Telemetry;
|
||||
@@ -24,6 +26,18 @@ internal static class EvidenceTelemetry
|
||||
unit: "statements",
|
||||
description: "Distribution of statements returned per observation projection request.");
|
||||
|
||||
private static readonly Counter<long> EvidenceRequestCounter =
|
||||
Meter.CreateCounter<long>(
|
||||
"excititor.vex.evidence.requests",
|
||||
unit: "requests",
|
||||
description: "Number of evidence chunk requests handled by the evidence APIs.");
|
||||
|
||||
private static readonly Histogram<int> EvidenceChunkHistogram =
|
||||
Meter.CreateHistogram<int>(
|
||||
"excititor.vex.evidence.chunk_count",
|
||||
unit: "chunks",
|
||||
description: "Distribution of evidence chunks streamed per request.");
|
||||
|
||||
private static readonly Counter<long> SignatureStatusCounter =
|
||||
Meter.CreateCounter<long>(
|
||||
"excititor.vex.signature.status",
|
||||
@@ -53,13 +67,27 @@ internal static class EvidenceTelemetry
|
||||
return;
|
||||
}
|
||||
|
||||
ObservationStatementHistogram.Record(
|
||||
returnedCount,
|
||||
new[]
|
||||
{
|
||||
new KeyValuePair<string, object?>("tenant", normalizedTenant),
|
||||
new KeyValuePair<string, object?>("outcome", outcome),
|
||||
});
|
||||
ObservationStatementHistogram.Record(returnedCount, tags);
|
||||
}
|
||||
|
||||
public static void RecordChunkOutcome(string? tenant, string outcome, int chunkCount = 0, bool truncated = false)
|
||||
{
|
||||
var normalizedTenant = NormalizeTenant(tenant);
|
||||
var tags = new[]
|
||||
{
|
||||
new KeyValuePair<string, object?>("tenant", normalizedTenant),
|
||||
new KeyValuePair<string, object?>("outcome", outcome),
|
||||
new KeyValuePair<string, object?>("truncated", truncated),
|
||||
};
|
||||
|
||||
EvidenceRequestCounter.Add(1, tags);
|
||||
|
||||
if (!string.Equals(outcome, "success", StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
EvidenceChunkHistogram.Record(chunkCount, tags);
|
||||
}
|
||||
|
||||
public static void RecordSignatureStatus(string? tenant, IReadOnlyList<VexObservationStatementProjection> statements)
|
||||
@@ -72,6 +100,7 @@ internal static class EvidenceTelemetry
|
||||
var normalizedTenant = NormalizeTenant(tenant);
|
||||
var missing = 0;
|
||||
var unverified = 0;
|
||||
var verified = 0;
|
||||
|
||||
foreach (var statement in statements)
|
||||
{
|
||||
@@ -86,6 +115,10 @@ internal static class EvidenceTelemetry
|
||||
{
|
||||
unverified++;
|
||||
}
|
||||
else
|
||||
{
|
||||
verified++;
|
||||
}
|
||||
}
|
||||
|
||||
if (missing > 0)
|
||||
@@ -103,11 +136,62 @@ internal static class EvidenceTelemetry
|
||||
{
|
||||
SignatureStatusCounter.Add(
|
||||
unverified,
|
||||
new[]
|
||||
{
|
||||
new KeyValuePair<string, object?>("tenant", normalizedTenant),
|
||||
new KeyValuePair<string, object?>("status", "unverified"),
|
||||
});
|
||||
BuildSignatureTags(normalizedTenant, "unverified"));
|
||||
}
|
||||
|
||||
if (verified > 0)
|
||||
{
|
||||
SignatureStatusCounter.Add(
|
||||
verified,
|
||||
BuildSignatureTags(normalizedTenant, "verified"));
|
||||
}
|
||||
}
|
||||
|
||||
public static void RecordChunkSignatureStatus(string? tenant, IReadOnlyList<VexEvidenceChunkResponse> chunks)
|
||||
{
|
||||
if (chunks is null || chunks.Count == 0)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
var normalizedTenant = NormalizeTenant(tenant);
|
||||
|
||||
var unsigned = 0;
|
||||
var unverified = 0;
|
||||
var verified = 0;
|
||||
|
||||
foreach (var chunk in chunks)
|
||||
{
|
||||
var signature = chunk.Signature;
|
||||
if (signature is null)
|
||||
{
|
||||
unsigned++;
|
||||
continue;
|
||||
}
|
||||
|
||||
if (signature.VerifiedAt is null)
|
||||
{
|
||||
unverified++;
|
||||
}
|
||||
else
|
||||
{
|
||||
verified++;
|
||||
}
|
||||
}
|
||||
|
||||
if (unsigned > 0)
|
||||
{
|
||||
SignatureStatusCounter.Add(unsigned, BuildSignatureTags(normalizedTenant, "unsigned"));
|
||||
}
|
||||
|
||||
if (unverified > 0)
|
||||
{
|
||||
SignatureStatusCounter.Add(unverified, BuildSignatureTags(normalizedTenant, "unverified"));
|
||||
}
|
||||
|
||||
if (verified > 0)
|
||||
{
|
||||
SignatureStatusCounter.Add(verified, BuildSignatureTags(normalizedTenant, "verified"));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -151,4 +235,11 @@ internal static class EvidenceTelemetry
|
||||
|
||||
private static string NormalizeSurface(string? surface)
|
||||
=> string.IsNullOrWhiteSpace(surface) ? "unknown" : surface.ToLowerInvariant();
|
||||
|
||||
private static KeyValuePair<string, object?>[] BuildSignatureTags(string tenant, string status)
|
||||
=> new[]
|
||||
{
|
||||
new KeyValuePair<string, object?>("tenant", tenant),
|
||||
new KeyValuePair<string, object?>("status", status),
|
||||
};
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user