docs consolidation
This commit is contained in:
master
@@ -8,7 +8,7 @@ Scanner analyses container images layer-by-layer, producing deterministic SBOM f
|
||||
- Python analyzer picks up `requirements*.txt`, `Pipfile.lock`, and `poetry.lock`, tagging installed distributions with lock provenance and generating declared-only components for policy. Use `stella python lock-validate` to run the same checks locally before images are built.
|
||||
- Java analyzer now parses `gradle.lockfile`, `gradle/dependency-locks/**/*.lockfile`, and `pom.xml` dependencies via the new `JavaLockFileCollector`, merging lock metadata onto jar evidence and emitting declared-only components when jars are absent. The new CLI verb `stella java lock-validate` reuses that collector offline (table/JSON output) and records `stellaops.cli.java.lock_validate.count{outcome}` for observability.
|
||||
- Worker/WebService now resolve cache roots and feature flags via `StellaOps.Scanner.Surface.Env`; misconfiguration warnings are documented in `docs/modules/scanner/design/surface-env.md` and surfaced through startup validation.
|
||||
- Platform events rollout (2025-10-19) continues to publish scanner.report.ready@1 and scanner.scan.completed@1 envelopes with embedded DSSE payloads (see docs/updates/2025-10-19-scanner-policy.md and docs/updates/2025-10-19-platform-events.md). Service and consumer tests should round-trip the canonical samples under docs/events/samples/.
|
||||
- Platform events rollout (2025-10-19) continues to publish scanner.report.ready@1 and scanner.scan.completed@1 envelopes with embedded DSSE payloads (see docs/updates/2025-10-19-scanner-policy.md and docs/updates/2025-10-19-platform-events.md). Service and consumer tests should round-trip the canonical samples under docs/modules/signals/events/samples/.
|
||||
- OS/non-language analyzers: evidence is rootfs-relative, warnings are structured/capped, hashing is bounded, and Linux OS analyzers support surface-cache reuse. See `os-analyzers-evidence.md`.
|
||||
|
||||
## Responsibilities
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
Purpose: unblock PREP tasks by freezing analyzer inputs/outputs, resolver traces, and fixtures for Node bundle/source-map coverage, native/WASM detection, and AOC-compliant observation emission.
|
||||
|
||||
## Output artefacts
|
||||
- Sample NDJSON: `docs/samples/scanner/node-phase22/node-phase22-sample.ndjson` (covers 22-006/007/008 in one run).
|
||||
- Sample NDJSON: `docs/modules/scanner/samples/node-phase22/node-phase22-sample.ndjson` (covers 22-006/007/008 in one run).
|
||||
- Resolver trace spec and reason codes (below) are binding for workers and tests.
|
||||
|
||||
## 22-006 · Bundle + source-map reconstruction
|
||||
@@ -37,7 +37,7 @@ Purpose: unblock PREP tasks by freezing analyzer inputs/outputs, resolver traces
|
||||
- Large maps: emit `ERR_NODE_BUNDLE_MAP_TOO_LARGE` and skip map (still report bundle presence with `confidence:0.51`).
|
||||
|
||||
## Fixtures
|
||||
- `docs/samples/scanner/node-phase22/node-phase22-sample.ndjson` contains:
|
||||
- `docs/modules/scanner/samples/node-phase22/node-phase22-sample.ndjson` contains:
|
||||
1) webpack bundle w/ source map mapping to `/src/app.js` (22-006)
|
||||
2) native addon load via `process.dlopen('./native/addon.node')` (22-007)
|
||||
3) WASM module import via `WebAssembly.instantiateStreaming(fetch('./pkg.wasm'))` (22-007)
|
||||
|
||||
@@ -63,7 +63,7 @@ graph LR
|
||||
|
||||
| Artifact | Owner | Location |
|
||||
|----------|-------|----------|
|
||||
| RFC Document | Scanner TL | `docs/rfcs/scanner/` |
|
||||
| RFC Document | Scanner TL | `docs/adr/` |
|
||||
| Mapping CSV | Scanner TL | `docs/modules/scanner/fixtures/adapters/` |
|
||||
| Golden Fixtures | QA | `docs/modules/scanner/fixtures/cdx17-cbom/` |
|
||||
| Hash List | QA | `docs/modules/scanner/fixtures/*/hashes.txt` |
|
||||
@@ -138,7 +138,7 @@ Triggered by:
|
||||
"since": "v2.5.0",
|
||||
"removal": "v3.0.0",
|
||||
"replacement": "ratings[method=CVSSv31]",
|
||||
"migrationGuide": "docs/migrations/cvss-v30-removal.md"
|
||||
"migrationGuide": "docs/technical/migration/cvss-v30-removal.md"
|
||||
}
|
||||
}
|
||||
```
|
||||
@@ -167,7 +167,7 @@ To modify a locked adapter:
|
||||
|
||||
| Record | Location | Retention |
|
||||
|--------|----------|-----------|
|
||||
| RFC decisions | `docs/rfcs/scanner/` | Permanent |
|
||||
| RFC decisions | `docs/adr/` | Permanent |
|
||||
| Hash changes | Git history + `CHANGELOG.md` | Permanent |
|
||||
| Approval records | PR comments | Permanent |
|
||||
| DSSE envelopes | CAS + offline kit | Permanent |
|
||||
|
||||
@@ -9,7 +9,7 @@ This runbook confirms that Scanner.WebService now surfaces the metadata Runtime
|
||||
## 1. Prerequisites
|
||||
|
||||
- Scanner.WebService release includes **SCANNER-POLICY-09-107** (adds quieted provenance and score inputs to `/reports`).
|
||||
- Docs repository at commit containing `docs/events/scanner.report.ready@1.json` with `quietedFindingCount`.
|
||||
- Docs repository at commit containing `docs/modules/signals/events/scanner.report.ready@1.json` with `quietedFindingCount`.
|
||||
- Access to a Scanner environment (staging or sandbox) with an image capable of producing policy verdicts.
|
||||
|
||||
---
|
||||
@@ -26,15 +26,15 @@ This runbook confirms that Scanner.WebService now surfaces the metadata Runtime
|
||||
2. **Check emitted event** – pull the latest `scanner.report.ready` event (from the queue or sample capture). Confirm the payload includes:
|
||||
- `quietedFindingCount` equal to the `summary.quieted` value.
|
||||
- Updated `summary` block with the quieted counter.
|
||||
3. **Schema validation** – optionally validate the payload against `docs/events/scanner.report.ready@1.json` to guarantee downstream compatibility:
|
||||
3. **Schema validation** – optionally validate the payload against `docs/modules/signals/events/scanner.report.ready@1.json` to guarantee downstream compatibility:
|
||||
```bash
|
||||
npx ajv validate -c ajv-formats \
|
||||
-s docs/events/scanner.report.ready@1.json \
|
||||
-s docs/modules/signals/events/scanner.report.ready@1.json \
|
||||
-d <payload.json>
|
||||
```
|
||||
(Use `npm install --no-save ajv ajv-cli ajv-formats` once per clone.)
|
||||
|
||||
> Snapshot fixtures: see `docs/events/samples/scanner.event.report.ready@1.sample.json` for a canonical orchestrator event that already carries `quietedFindingCount`.
|
||||
> Snapshot fixtures: see `docs/modules/signals/events/samples/scanner.event.report.ready@1.sample.json` for a canonical orchestrator event that already carries `quietedFindingCount`.
|
||||
|
||||
---
|
||||
|
||||
|
||||
147
docs/modules/scanner/scanner-core-contracts.md
Normal file
147
docs/modules/scanner/scanner-core-contracts.md
Normal file
@@ -0,0 +1,147 @@
|
||||
# Scanner Core Contracts
|
||||
|
||||
The **Scanner Core** library provides shared contracts, observability helpers, and security utilities consumed by `Scanner.WebService`, `Scanner.Worker`, analyzers, and tooling. These primitives guarantee deterministic identifiers, timestamps, and log context for all scanning flows.
|
||||
|
||||
## Canonical DTOs
|
||||
|
||||
- `ScanJob` & `ScanJobStatus` – canonical job metadata (image reference/digest, tenant, correlation ID, timestamps, failure details). Constructors normalise timestamps to UTC microsecond precision and canonicalise image digests. Round-trips with `JsonSerializerDefaults.Web` using `ScannerJsonOptions`.
|
||||
- `ScanProgressEvent` & `ScanStage`/`ScanProgressEventKind` – stage-level progress surface for queue/stream consumers. Includes deterministic sequence numbers, optional progress percentage, attributes, and attached `ScannerError`.
|
||||
- `ScannerError` & `ScannerErrorCode` – shared error taxonomy spanning queue, analyzers, storage, exporters, and signing. Carries severity, retryability, structured details, and microsecond-precision timestamps.
|
||||
- `ScanJobId` – strongly-typed identifier rendered as `Guid` (lowercase `N` format) with deterministic parsing.
|
||||
|
||||
### Canonical JSON samples
|
||||
|
||||
The golden fixtures consumed by `ScannerCoreContractsTests` document the wire shape shared with downstream services. They live under `src/Scanner/__Tests/StellaOps.Scanner.Core.Tests/Fixtures/` and a representative extract is shown below.
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "8f4cc9c582454b9d9b4f5ae049631b7d",
|
||||
"status": "running",
|
||||
"imageReference": "registry.example.com/stellaops/scanner:1.2.3",
|
||||
"imageDigest": "sha256:abcdef",
|
||||
"createdAt": "2025-10-18T14:30:15.123456+00:00",
|
||||
"updatedAt": "2025-10-18T14:30:20.123456+00:00",
|
||||
"correlationId": "scan-analyzeoperatingsystem-8f4cc9c582454b9d9b4f5ae049631b7d",
|
||||
"tenantId": "tenant-a",
|
||||
"metadata": {
|
||||
"requestId": "req-1234",
|
||||
"source": "ci"
|
||||
},
|
||||
"failure": {
|
||||
"code": "analyzerFailure",
|
||||
"severity": "error",
|
||||
"message": "Analyzer failed to parse layer",
|
||||
"timestamp": "2025-10-18T14:30:15.123456+00:00",
|
||||
"retryable": false,
|
||||
"stage": "AnalyzeOperatingSystem",
|
||||
"component": "os-analyzer",
|
||||
"details": {
|
||||
"layerDigest": "sha256:deadbeef",
|
||||
"attempt": "1"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Progress events follow the same conventions (`jobId`, `stage`, `kind`, `timestamp`, `attributes`, optional embedded `ScannerError`). The fixtures are verified via deterministic JSON comparison in every CI run.
|
||||
|
||||
## Deterministic helpers
|
||||
|
||||
- `ScannerIdentifiers` – derives `ScanJobId`, correlation IDs, and SHA-256 hashes from normalised inputs (image reference/digest, tenant, salt). Ensures case-insensitive stability and reproducible metric keys.
|
||||
- `ScannerTimestamps` – trims to microsecond precision, provides ISO-8601 (`yyyy-MM-ddTHH:mm:ss.ffffffZ`) rendering, and parsing helpers.
|
||||
- `ScannerJsonOptions` – standard JSON options (web defaults, camel-case enums) shared by services/tests.
|
||||
- `ScanAnalysisStore` & `ScanAnalysisKeys` – shared in-memory analysis cache flowing through Worker stages. OS analyzers populate
|
||||
`analysis.os.packages` (raw output), `analysis.os.fragments` (per-analyzer component fragments), and merge into
|
||||
`analysis.layers.fragments` so emit/diff stages can compose SBOMs and diffs without knowledge of individual analyzer
|
||||
implementations.
|
||||
|
||||
## Observability primitives
|
||||
|
||||
- `ScannerDiagnostics` – global `ActivitySource`/`Meter` for scanner components. `StartActivity` seeds deterministic tags (`job_id`, `stage`, `component`, `correlation_id`).
|
||||
- `ScannerMetricNames` – centralises metric prefixes (`stellaops.scanner.*`) and deterministic job/event tag builders.
|
||||
- `ScannerCorrelationContext` & `ScannerCorrelationContextAccessor` – ambient correlation propagation via `AsyncLocal` for log scopes, metrics, and diagnostics.
|
||||
- `ScannerLogExtensions` – `ILogger` scopes for jobs/progress events with automatic correlation context push, minimal allocations, and consistent structured fields.
|
||||
|
||||
### Observability overhead validation
|
||||
|
||||
A micro-benchmark executed on 2025-10-19 (4 vCPU runner, .NET 10.0.100-rc.1) measured the average scope cost across 1 000 000 iterations:
|
||||
|
||||
| Scope | Mean (µs/call) |
|
||||
|-------|----------------|
|
||||
| `BeginScanScope` (logger attached) | 0.80 |
|
||||
| `BeginScanScope` (noop logger) | 0.31 |
|
||||
| `BeginProgressScope` | 0.57 |
|
||||
|
||||
To reproduce, run `dotnet test src/Scanner/__Tests/StellaOps.Scanner.Core.Tests -c Release` (see `ScannerLogExtensionsPerformanceTests`) or copy the snippet below into a throwaway `dotnet run` console project and execute it with `dotnet run -c Release`:
|
||||
|
||||
```csharp
|
||||
using System.Collections.Generic;
|
||||
using System.Diagnostics;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using StellaOps.Scanner.Core.Contracts;
|
||||
using StellaOps.Scanner.Core.Observability;
|
||||
using StellaOps.Scanner.Core.Utility;
|
||||
|
||||
var factory = LoggerFactory.Create(builder => builder.AddFilter(static _ => true));
|
||||
var logger = factory.CreateLogger("bench");
|
||||
|
||||
var jobId = ScannerIdentifiers.CreateJobId("registry.example.com/stellaops/scanner:1.2.3", "sha256:abcdef", "tenant-a", "benchmark");
|
||||
var correlationId = ScannerIdentifiers.CreateCorrelationId(jobId, nameof(ScanStage.AnalyzeOperatingSystem));
|
||||
var now = ScannerTimestamps.Normalize(new DateTimeOffset(2025, 10, 19, 12, 0, 0, TimeSpan.Zero));
|
||||
|
||||
var job = new ScanJob(jobId, ScanJobStatus.Running, "registry.example.com/stellaops/scanner:1.2.3", "sha256:abcdef", now, now, correlationId, "tenant-a", new Dictionary<string, string>(StringComparer.Ordinal) { ["requestId"] = "req-bench" });
|
||||
var progress = new ScanProgressEvent(jobId, ScanStage.AnalyzeOperatingSystem, ScanProgressEventKind.Progress, 42, now, 10.5, "benchmark", new Dictionary<string, string>(StringComparer.Ordinal) { ["sample"] = "true" });
|
||||
|
||||
Console.WriteLine("Scanner Core Observability micro-bench (1,000,000 iterations)");
|
||||
Report("BeginScanScope (logger)", Measure(static ctx => ctx.Logger.BeginScanScope(ctx.Job, ctx.Stage, ctx.Component), new ScopeContext(logger, job, nameof(ScanStage.AnalyzeOperatingSystem), "os-analyzer")));
|
||||
Report("BeginScanScope (no logger)", Measure(static ctx => ScannerLogExtensions.BeginScanScope(null, ctx.Job, ctx.Stage, ctx.Component), new ScopeContext(logger, job, nameof(ScanStage.AnalyzeOperatingSystem), "os-analyzer")));
|
||||
Report("BeginProgressScope", Measure(static ctx => ctx.Logger.BeginProgressScope(ctx.Progress!, ctx.Component), new ScopeContext(logger, job, nameof(ScanStage.AnalyzeOperatingSystem), "os-analyzer", progress)));
|
||||
|
||||
static double Measure(Func<ScopeContext, IDisposable> factory, ScopeContext context)
|
||||
{
|
||||
const int iterations = 1_000_000;
|
||||
for (var i = 0; i < 10_000; i++)
|
||||
{
|
||||
using var scope = factory(context);
|
||||
}
|
||||
|
||||
GC.Collect();
|
||||
GC.WaitForPendingFinalizers();
|
||||
GC.Collect();
|
||||
|
||||
var sw = Stopwatch.StartNew();
|
||||
for (var i = 0; i < iterations; i++)
|
||||
{
|
||||
using var scope = factory(context);
|
||||
}
|
||||
|
||||
sw.Stop();
|
||||
return sw.Elapsed.TotalSeconds * 1_000_000 / iterations;
|
||||
}
|
||||
|
||||
static void Report(string label, double microseconds)
|
||||
=> Console.WriteLine($"{label,-28}: {microseconds:F3} µs");
|
||||
|
||||
readonly record struct ScopeContext(ILogger Logger, ScanJob Job, string? Stage, string? Component, ScanProgressEvent? Progress = null);
|
||||
```
|
||||
|
||||
Both guardrails enforce the ≤ 5 µs acceptance target for SP9-G1.
|
||||
|
||||
## Security utilities
|
||||
|
||||
- `AuthorityTokenSource` – caches short-lived OpToks per audience+scope using deterministic keys and refresh skew (default 30 s). Integrates with `StellaOps.Auth.Client`.
|
||||
- `DpopProofValidator` – validates DPoP proofs (alg allowlist, `htm`/`htu`, nonce, replay window, signature) backed by pluggable `IDpopReplayCache`. Ships with `InMemoryDpopReplayCache` for restart-only deployments.
|
||||
- `RestartOnlyPluginGuard` – enforces restart-time plug-in registration (deterministic path normalisation; throws if new plug-ins added post-seal).
|
||||
- `ServiceCollectionExtensions.AddScannerAuthorityCore` – DI helper wiring Authority client, OpTok source, DPoP validation, replay cache, and plug-in guard.
|
||||
|
||||
## Testing guarantees
|
||||
|
||||
Unit tests (`StellaOps.Scanner.Core.Tests`) assert:
|
||||
|
||||
- DTO JSON round-trips are stable and deterministic (`ScannerCoreContractsTests` + golden fixtures).
|
||||
- Identifier/hash helpers ignore case and emit lowercase hex.
|
||||
- Timestamp normalisation retains UTC semantics.
|
||||
- Log scopes push/pop correlation context predictably while staying under the 5 µs envelope.
|
||||
- Authority token caching honours refresh skew and invalidation.
|
||||
- DPoP validator accepts valid proofs, rejects nonce mismatch/replay, and enforces signature validation.
|
||||
- Restart-only plug-in guard blocks runtime additions post-seal.
|
||||
Reference in New Issue
Block a user