consolidation of some of the modules, localization fixes, product advisories work, qa work
This commit is contained in:
@@ -0,0 +1,24 @@
|
||||
# Using the Chunk API
|
||||
|
||||
Endpoint: `POST /vex/evidence/chunks`
|
||||
- Content-Type: `application/x-ndjson`
|
||||
- See schema: `docs/modules/excititor/schemas/vex-chunk-api.yaml`
|
||||
|
||||
Response: `202 Accepted`
|
||||
```json
|
||||
{ "chunk_digest": "sha256:…", "queue_id": "uuid" }
|
||||
```
|
||||
|
||||
Operational notes
|
||||
- Deterministic hashing: server recomputes `chunk_digest` from canonical JSON; mismatches return 400.
|
||||
- Limits: default 500 items, max 2000 (aligned with Program.cs guard).
|
||||
- Telemetry: metrics under `StellaOps.Excititor.Chunks` (see chunk-telemetry.md).
|
||||
- Headers: correlation/trace headers echoed (`X-Stella-TraceId`, `X-Stella-CorrelationId`).
|
||||
|
||||
Example curl
|
||||
```bash
|
||||
curl -X POST https://excitor.local/vex/evidence/chunks \
|
||||
-H "Authorization: Bearer <token>" \
|
||||
-H "Content-Type: application/x-ndjson" \
|
||||
--data-binary @docs/modules/excititor/samples/chunk-sample.ndjson
|
||||
```
|
||||
@@ -0,0 +1,26 @@
|
||||
# Excititor Chunk Telemetry (Sprint 110)
|
||||
|
||||
## Metrics (Meter: `StellaOps.Excititor.Chunks`)
|
||||
- `vex_chunks_ingested_total` (counter) — tags: `tenant`, `source`, `status` (`accepted|rejected`), `reason` (nullable for accepted). Increments per chunk submitted.
|
||||
- `vex_chunks_item_count` (histogram, unit=items) — records item count per chunk.
|
||||
- `vex_chunks_payload_bytes` (histogram, unit=bytes) — measured from NDJSON payload length.
|
||||
- `vex_chunks_latency_ms` (histogram) — end-to-end ingestion latency per request.
|
||||
|
||||
## Logs
|
||||
- `vex.chunk.ingest.accepted` — includes `chunk_id`, `tenant`, `source`, `item_count`, `chunk_digest`.
|
||||
- `vex.chunk.ingest.rejected` — includes `chunk_id`, `tenant`, `source`, `reason`, validation errors (summarized).
|
||||
|
||||
## Wiring steps
|
||||
1. Register `ChunkTelemetry` as singleton with shared `Meter` instance.
|
||||
2. In `/vex/evidence/chunks` handler, compute `chunk_digest` deterministically from canonical JSON and emit counters/histograms via `ChunkTelemetry`.
|
||||
3. Log using structured templates above; avoid request bodies in logs.
|
||||
4. Expose metrics via default ASP.NET metrics export (Prometheus/OpenTelemetry) already configured in WebService.
|
||||
|
||||
## Determinism & offline posture
|
||||
- Do not include host-specific paths or timestamps in metric dimensions.
|
||||
- Histogram buckets: use standard OTEL defaults; no runtime-generated buckets.
|
||||
- Keep meter name stable; adding new instruments requires version note in sprint Decisions & Risks.
|
||||
|
||||
## Ownership
|
||||
- Implementer: Excititor Observability Guild
|
||||
- Reviewers: Evidence Locker Guild (for parity with attestation metrics)
|
||||
@@ -0,0 +1,31 @@
|
||||
# Excititor Consensus Removal Runbook (AOC-19-004)
|
||||
|
||||
- **Date:** 2025-11-21
|
||||
- **Scope:** EXCITITOR-CORE-AOC-19-004
|
||||
- **Goal:** Eliminate legacy consensus/merged severity fields so Excititor remains aggregation-only.
|
||||
|
||||
## Cutover steps
|
||||
1) **Freeze consensus refresh** — `DisableConsensus=true` (default) forces refresh loop off. Keep this enabled during migration.
|
||||
2) **Schema cleanup** — migrate collections to remove or null legacy fields:
|
||||
- `vex_consensus` / `vex_consensus_holds`: drop/ignore fields `consensusDigest`, `policyVersion`, `policyRevisionId`, `policyDigest`, `summary`, `signals`, `status` (merged) once Policy takes over.
|
||||
- `vex_observations` / materialized exports: ensure no merged severity/status fields are written.
|
||||
- `vex_mirror` exports: stop emitting consensus JSON; retain raw observations only.
|
||||
3) **Telemetry:** emit counter `excititor.ingest.consensus.disabled` (tags `tenant`, `source`, `connectorId`) once per batch to prove cutover.
|
||||
4) **Guards:** AOC guards reject any incoming/derived field in `{mergedSeverity, consensusScore, computedStatus}`.
|
||||
5) **Backfill:** run one-off job to set `consensusDisabled=true` on legacy records and remove merged fields without touching raw observations.
|
||||
6) **Verification:** regression checklist (per tenant):
|
||||
- No writes to `vex_consensus*` collections after cutover.
|
||||
- Ingest + export fixtures show only raw observations/linksets; snapshots deterministic.
|
||||
- Telemetry counter present; absence of consensus refresh logs.
|
||||
|
||||
## Config
|
||||
```
|
||||
Excititor:Worker:
|
||||
DisableConsensus: true # keep true post-cutover
|
||||
```
|
||||
|
||||
## Test plan (after disk space is restored)
|
||||
- Unit: AOC guard rejects merged fields.
|
||||
- Integration (Mongo2Go): ingest batch containing merged fields → rejected; telemetry counter increments.
|
||||
- Worker: start with DisableConsensus=true → consensus refresh loop does not schedule; log once at startup.
|
||||
|
||||
125
docs-archived/modules/excititor/operations/evidence-api.md
Normal file
125
docs-archived/modules/excititor/operations/evidence-api.md
Normal file
@@ -0,0 +1,125 @@
|
||||
# Excititor Advisory-AI evidence APIs (projection + chunks)
|
||||
|
||||
> Covers the read-only evidence surfaces shipped in Sprints 119–120: `/v1/vex/observations/{vulnerabilityId}/{productKey}` and `/v1/vex/evidence/chunks`.
|
||||
|
||||
## Scope and determinism
|
||||
|
||||
- **Aggregation-only**: no consensus, severity merging, or reachability. Responses carry raw statements plus provenance/signature metadata.
|
||||
- **Stable ordering**: both endpoints sort by `lastSeen` DESC; pagination uses a deterministic `limit`.
|
||||
- **Limits**: observation projection default `limit=200`, max `500`; chunk stream default `limit=500`, max `2000`.
|
||||
- **Tenancy**: reads respect `X-Stella-Tenant` when provided; otherwise fall back to `DefaultTenant` configuration.
|
||||
- **Auth**: bearer token with `vex.read` scope required.
|
||||
|
||||
## `/v1/vex/observations/{vulnerabilityId}/{productKey}`
|
||||
|
||||
- **Response**: JSON object with `vulnerabilityId`, `productKey`, `generatedAt`, `totalCount`, `truncated`, `statements[]`.
|
||||
- **Statement fields**: `observationId`, `providerId`, `status`, `justification`, `detail`, `firstSeen`, `lastSeen`, `scope{key,name,version,purl,cpe,componentIdentifiers[]}`, `anchors[]`, `document{digest,format,revision,sourceUri}`, `signature{type,keyId,issuer,verifiedAt}`.
|
||||
- **Filters**:
|
||||
- `providerId` (multi-valued, comma-separated)
|
||||
- `status` (values in `VexClaimStatus`)
|
||||
- `since` (ISO-8601, UTC)
|
||||
- `limit` (ints within bounds)
|
||||
- **Mapping back to storage**:
|
||||
- `observationId` = `{providerId}:{document.digest}`
|
||||
- `document.digest` locates the raw record in `vex_raw`.
|
||||
- `anchors` contain JSON pointers/paragraph locators from source metadata.
|
||||
|
||||
Headers:
|
||||
- `Excititor-Results-Truncated: true|false`
|
||||
- `Excititor-Results-Total: <int>`
|
||||
|
||||
## `/v1/vex/evidence/chunks`
|
||||
|
||||
- **Query params**: `vulnerabilityId` (required), `productKey` (required), optional `providerId`, `status`, `since`, `limit`.
|
||||
- **Limits**: default `limit=500`, max `2000`.
|
||||
- **Response**: **NDJSON** stream; each line is a `VexEvidenceChunkResponse`.
|
||||
- **Chunk fields**: `observationId`, `linksetId`, `vulnerabilityId`, `productKey`, `providerId`, `status`, `justification`, `detail`, `scopeScore` (from confidence or signals), `firstSeen`, `lastSeen`, `scope{...}`, `document{digest,format,sourceUri,revision}`, `signature{type,subject,issuer,keyId,verifiedAt,transparencyRef}`, `metadata` (flattened additionalMetadata).
|
||||
- **Headers**: `Excititor-Results-Total`, `Excititor-Results-Truncated` (mirrors projection API naming).
|
||||
- **Streaming guidance (SDK/clients)**:
|
||||
- Use HTTP client that supports response streaming; read line-by-line and JSON-deserialize per line.
|
||||
- Treat stream as an NDJSON list up to `limit`; no outer array.
|
||||
- Back-off or paginate by adjusting `since` or narrowing providers/statuses.
|
||||
|
||||
OpenAPI (excerpt):
|
||||
|
||||
```yaml
|
||||
paths:
|
||||
/v1/vex/evidence/chunks:
|
||||
get:
|
||||
summary: Stream evidence chunks for a vulnerability/product
|
||||
parameters:
|
||||
- in: query
|
||||
name: vulnerabilityId
|
||||
schema: { type: string }
|
||||
required: true
|
||||
- in: query
|
||||
name: productKey
|
||||
schema: { type: string }
|
||||
required: true
|
||||
- in: query
|
||||
name: providerId
|
||||
schema: { type: string }
|
||||
description: Comma-separated provider ids
|
||||
- in: query
|
||||
name: status
|
||||
schema: { type: string }
|
||||
description: Comma-separated VEX statuses
|
||||
- in: query
|
||||
name: since
|
||||
schema: { type: string, format: date-time }
|
||||
- in: query
|
||||
name: limit
|
||||
schema: { type: integer, minimum: 1, maximum: 2000, default: 500 }
|
||||
responses:
|
||||
"200":
|
||||
description: NDJSON stream of VexEvidenceChunkResponse
|
||||
headers:
|
||||
Excititor-Results-Total: { schema: { type: integer } }
|
||||
Excititor-Results-Truncated: { schema: { type: boolean } }
|
||||
content:
|
||||
application/x-ndjson:
|
||||
schema:
|
||||
type: string
|
||||
description: One JSON object per line (VexEvidenceChunkResponse)
|
||||
```
|
||||
|
||||
Example (curl):
|
||||
|
||||
```bash
|
||||
curl -s -H "Authorization: Bearer <token>" \
|
||||
-H "X-Stella-Tenant: acme" \
|
||||
"https://exc.example.test/v1/vex/evidence/chunks?vulnerabilityId=CVE-2025-0001&productKey=pkg:docker/demo&limit=2" |
|
||||
head -n 2
|
||||
```
|
||||
|
||||
Sample NDJSON line:
|
||||
|
||||
```json
|
||||
{"observationId":"provider-a:4d2f...","linksetId":"CVE-2025-0001:pkg:docker/demo","vulnerabilityId":"CVE-2025-0001","productKey":"pkg:docker/demo","providerId":"provider-a","status":"Affected","justification":"ComponentNotPresent","detail":"demo detail","scopeScore":0.9,"firstSeen":"2025-11-10T12:00:00Z","lastSeen":"2025-11-12T12:00:00Z","scope":{"key":"pkg:docker/demo","name":"demo","version":"1.0.0","purl":"pkg:docker/demo","cpe":null,"componentIdentifiers":["component-a"]},"document":{"digest":"sha256:e7...","format":"sbomcyclonedx","sourceUri":"https://example.test/vex.json","revision":"r1"},"signature":{"type":"cosign","subject":"demo","issuer":"issuer","keyId":"kid","verifiedAt":"2025-11-12T12:00:00Z","transparencyRef":null},"metadata":{}}
|
||||
```
|
||||
|
||||
## `/v1/vex/attestations/{attestationId}`
|
||||
|
||||
- **Purpose**: Lookup attestation provenance (supplier ↔ observation/linkset ↔ product/vulnerability) without touching consensus.
|
||||
- **Response**: `VexAttestationPayload` with fields:
|
||||
- `attestationId`, `supplierId`, `observationId`, `linksetId`, `vulnerabilityId`, `productKey`, `justificationSummary`, `issuedAt`, `metadata{}`.
|
||||
- **Semantics**:
|
||||
- `attestationId` matches the export/attestation ID used when signing (Resolve/Worker flows).
|
||||
- `observationId`/`linksetId` map back to evidence identifiers; clients can stitch provenance for citations.
|
||||
- **Auth**: `vex.read` scope; tenant header optional (payloads are tenant-agnostic).
|
||||
|
||||
## Error model
|
||||
|
||||
- Standard API envelope with `ValidationProblem` for missing required params.
|
||||
- `scope` failures return `403` with problem details.
|
||||
- Tenancy parse failures return `400`.
|
||||
|
||||
## Backwards compatibility
|
||||
|
||||
- No legacy routes are deprecated by these endpoints; they are additive and remain aggregation-only.
|
||||
|
||||
## References
|
||||
|
||||
- Implementation: `src/Excititor/StellaOps.Excititor.WebService/Program.cs` (`/v1/vex/observations/**`, `/v1/vex/evidence/chunks`).
|
||||
- Telemetry: `src/Excititor/StellaOps.Excititor.WebService/Telemetry/EvidenceTelemetry.cs` (`excititor.vex.observation.*`, `excititor.vex.chunks.*`).
|
||||
- Data model: `src/Excititor/StellaOps.Excititor.WebService/Contracts/VexObservationContracts.cs`, `Contracts/VexEvidenceChunkContracts.cs`.
|
||||
@@ -0,0 +1,52 @@
|
||||
# Excititor · Graph Linkouts & Overlays — Implementation Notes (Graph 21-001/002/005/24-101/24-102)
|
||||
|
||||
- **Date:** 2025-11-21
|
||||
- **Scope:** EXCITITOR-GRAPH-21-001, EXCITITOR-GRAPH-21-002, EXCITITOR-GRAPH-21-005
|
||||
- **Status:** Implementation guidance (storage wiring pending).
|
||||
|
||||
## Endpoints
|
||||
1) **Linkouts (21-001)**
|
||||
- `POST /internal/graph/linkouts`
|
||||
- Body: `tenant`, `purls[]` (max 500), `includeJustifications?`, `includeProvenance?`
|
||||
- Response: ordered by input `purls`; each item includes `advisories[]` (`advisoryId`, `source`, `status`, `justification?`, `modifiedAt`, `evidenceHash`, `connectorId`, `dsseEnvelopeHash?`) plus `conflicts[]`; `notFound[]`.
|
||||
|
||||
2) **Overlays (21-002)**
|
||||
- `GET /v1/graph/overlays?purl=<purl>&purl=<purl>&includeJustifications=true|false`
|
||||
- Response per PURL: `summary` counts (`open`, `not_affected`, `under_investigation`, `no_statement`), `latestModifiedAt`, `justifications[]` (unique, sorted), `provenance` (`sources[]`, `lastEvidenceHash`), `cached`, `cacheAgeMs`.
|
||||
|
||||
3) **Status summaries (24-101)**
|
||||
- `GET /v1/graph/status?purl=<purl>&purl=<purl>`
|
||||
- Response mirrors overlay summaries but omits justification payloads; includes `sources[]`, `lastEvidenceHash`, `cached`, `cacheAgeMs`. Intended for Vuln Explorer status colouring.
|
||||
|
||||
4) **Batch observations for tooltips (24-102)**
|
||||
- `GET /v1/graph/observations?purl=<purl>[&purl=...]&includeJustifications=true|false[&limitPerPurl=50][&cursor=<base64>]`
|
||||
- Response per PURL: ordered `observations[]` (`observationId`, `advisoryId`, `status`, `justification?`, `providerId`, `modifiedAt`, `evidenceHash`, `dsseEnvelopeHash?`) plus `truncated`; top-level `nextCursor`, `hasMore` enable paging. Limits enforced per PURL and globally.
|
||||
|
||||
## Storage & Indexes (21-005)
|
||||
- `vex_observations` indexes:
|
||||
- `{ tenant: 1, component.purl: 1, advisoryId: 1, source: 1, modifiedAt: -1 }`
|
||||
- Sparse `{ tenant: 1, component.purl: 1, status: 1 }`
|
||||
- Optional materialized `vex_overlays` cache: unique `{ tenant: 1, purl: 1 }`, TTL on `cachedAt` driven by `excititor:graph:overlayTtlSeconds` (default 300s); payload must validate against `docs/modules/excititor/schemas/vex_overlay.schema.json` (schemaVersion 1.0.0). Bundle sample payload `docs/modules/excititor/samples/vex-overlay-sample.json` in Offline Kits.
|
||||
|
||||
## Determinism
|
||||
- Ordering: input PURL order → `advisoryId` → `source` for linkouts; overlays follow input order.
|
||||
- Truncation: max 200 advisories per PURL; when truncated, include `truncated: true` and `nextCursor` (`advisoryId`, `source`).
|
||||
|
||||
## Config knobs
|
||||
- `excititor:graph:overlayTtlSeconds` (default 300)
|
||||
- `excititor:graph:maxPurls` (default 500)
|
||||
- `excititor:graph:maxAdvisoriesPerPurl` (default 200)
|
||||
- `excititor:graph:maxTooltipItemsPerPurl` (default 50)
|
||||
- `excititor:graph:maxTooltipTotal` (default 1000)
|
||||
|
||||
## Telemetry
|
||||
- Counter `excititor.graph.linkouts.requests` tags: `tenant`, `includeJustifications`, `includeProvenance`.
|
||||
- Counter `excititor.graph.overlays.cache` tags: `tenant`, `hit` (`true|false`).
|
||||
- Histogram `excititor.graph.linkouts.latency.ms` tags: `tenant`.
|
||||
|
||||
## Steps to implement
|
||||
- Bind `GraphOptions` to `Excititor:Graph`.
|
||||
- Add endpoints to WebService with tenant guard; enforce limits.
|
||||
- Implement overlay cache with deterministic sort; respect TTL; surface `cached` + `cacheAgeMs`.
|
||||
- Backfill Mongo indexes above.
|
||||
- Integration tests (WebApplicationFactory + Mongo2Go) for ordering, truncation, cache metadata, tenant isolation.
|
||||
62
docs-archived/modules/excititor/operations/observability.md
Normal file
62
docs-archived/modules/excititor/operations/observability.md
Normal file
@@ -0,0 +1,62 @@
|
||||
# Excititor Observability Guide
|
||||
|
||||
> Added 2025-11-14 alongside Sprint 119 (`EXCITITOR-AIAI-31-003`). Complements the AirGap/mirror runbooks under the same folder.
|
||||
|
||||
Excititor’s evidence APIs now emit first-class OpenTelemetry metrics so Lens, Advisory AI, and Ops can detect misuse or missing provenance without paging through logs. This document lists the counters/histograms shipped by the WebService (`src/Excititor/StellaOps.Excititor.WebService`) and how to hook them into your exporters/dashboards.
|
||||
|
||||
## Telemetry prerequisites
|
||||
|
||||
- Enable `Excititor:Telemetry` in the service configuration (`appsettings.*`), ensuring **metrics** export is on. The WebService automatically adds the evidence meter (`StellaOps.Excititor.WebService.Evidence`) alongside the ingestion meter.
|
||||
- Deploy at least one OTLP or console exporter (see `TelemetryExtensions.ConfigureExcititorTelemetry`). If your region lacks OTLP transport, fall back to scraping the console exporter for smoke tests.
|
||||
- Coordinate with the Ops/Signals guild to provision the span/metric sinks referenced in `docs/modules/platform/architecture-overview.md#observability`.
|
||||
|
||||
## Metrics reference
|
||||
|
||||
| Metric | Type | Description | Key dimensions |
|
||||
| --- | --- | --- | --- |
|
||||
| `excititor.vex.observation.requests` | Counter | Number of `/v1/vex/observations/{vulnerabilityId}/{productKey}` requests handled. | `tenant`, `outcome` (`success`, `error`, `cancelled`), `truncated` (`true/false`) |
|
||||
| `excititor.vex.observation.statement_count` | Histogram | Distribution of statements returned per observation projection request. | `tenant`, `outcome` |
|
||||
| `excititor.vex.signature.status` | Counter | Signature status per statement (missing vs. unverified). | `tenant`, `status` (`missing`, `unverified`) |
|
||||
| `excititor.vex.aoc.guard_violations` | Counter | Aggregated count of Aggregation-Only Contract violations detected by the WebService (ingest + `/v1/vex/aoc/verify`). | `tenant`, `surface` (`ingest`, `aoc_verify`, etc.), `code` (AOC error code) |
|
||||
| `excititor.vex.chunks.requests` | Counter | Requests to `/v1/vex/evidence/chunks` (NDJSON stream). | `tenant`, `outcome` (`success`,`error`,`cancelled`), `truncated` (`true/false`) |
|
||||
| `excititor.vex.chunks.bytes` | Histogram | Size of NDJSON chunk streams served (bytes). | `tenant`, `outcome` |
|
||||
| `excititor.vex.chunks.records` | Histogram | Count of evidence records emitted per chunk stream. | `tenant`, `outcome` |
|
||||
|
||||
> All metrics originate from the `EvidenceTelemetry` helper (`src/Excititor/StellaOps.Excititor.WebService/Telemetry/EvidenceTelemetry.cs`). When disabled (telemetry off), the helper is inert.
|
||||
|
||||
### Dashboard hints
|
||||
|
||||
- **Advisory-AI readiness** – alert when `excititor.vex.signature.status{status="missing"}` spikes for a tenant, indicating connectors aren’t supplying signatures.
|
||||
- **Guardrail monitoring** – graph `excititor.vex.aoc.guard_violations` per `code` to catch upstream feed regressions before they pollute Evidence Locker or Lens caches.
|
||||
- **Capacity planning** – histogram percentiles of `excititor.vex.observation.statement_count` feed API sizing (higher counts mean Advisory AI is requesting broad scopes).
|
||||
|
||||
## Operational steps
|
||||
|
||||
1. **Enable telemetry**: set `Excititor:Telemetry:EnableMetrics=true`, configure OTLP endpoints/headers as described in `TelemetryExtensions`.
|
||||
2. **Add dashboards**: import panels referencing the metrics above (see Grafana JSON snippets in Ops repo once merged).
|
||||
3. **Alerting**: add rules for high guard violation rates, missing signatures, and abnormal chunk bytes/record counts. Tie alerts back to connectors via tenant metadata.
|
||||
4. **Post-deploy checks**: after each release, verify metrics emit by curling `/v1/vex/observations/...` and `/v1/vex/evidence/chunks`, watching the console exporter (dev) or OTLP (prod).
|
||||
|
||||
## SLOs (Sprint 119 – OBS-51-001)
|
||||
|
||||
The following SLOs apply to Excititor evidence read paths when telemetry is enabled. Record them in the shared SLO registry and alert via the platform alertmanager.
|
||||
|
||||
| Surface | SLI | Target | Window | Burn alert | Notes |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| `/v1/vex/observations` | p95 latency | ≤ 450 ms | 7d | 2 % over 1h | Measured on successful responses only; tenant scoped. |
|
||||
| `/v1/vex/observations` | freshness | ≥ 99 % within 5 min of upstream ingest | 7d | 5 % over 4h | Derived from arrival minus `createdAt`; requires ingest clocks in UTC. |
|
||||
| `/v1/vex/observations` | signature presence | ≥ 98 % statements with signature present | 7d | 3 % over 24h | Use `excititor.vex.signature.status{status="missing"}`. |
|
||||
| `/v1/vex/evidence/chunks` | p95 stream duration | ≤ 600 ms | 7d | 2 % over 1h | From request start to last NDJSON write; excludes client disconnects. |
|
||||
| `/v1/vex/evidence/chunks` | truncation rate | ≤ 1 % truncated streams | 7d | 1 % over 1h | `excititor.vex.chunks.records` with `truncated=true`. |
|
||||
| AOC guardrail | zero hard violations | 0 | continuous | immediate | Any `excititor.vex.aoc.guard_violations` with severity `error` pages ops. |
|
||||
|
||||
Implementation notes:
|
||||
- Emit latency/freshness SLOs via OTEL views that pre-aggregate by tenant and route to the platform SLO backend; keep bucket boundaries aligned with 50/100/250/450/650/1000 ms.
|
||||
- Freshness SLI derived from ingest timestamps; ensure clocks are synchronized (NTP) and stored in UTC.
|
||||
- For air-gapped deployments without OTEL sinks, scrape console exporter and push to offline Prometheus; same thresholds apply.
|
||||
|
||||
## Related documents
|
||||
|
||||
- `docs/modules/excititor/architecture.md` – API contract, AOC guardrails, connector responsibilities.
|
||||
- `docs/modules/excititor/mirrors.md` – AirGap/mirror ingestion checklist (feeds into `EXCITITOR-AIRGAP-56/57`).
|
||||
- `docs/modules/platform/architecture-overview.md#observability` – platform-wide telemetry guidance.
|
||||
@@ -0,0 +1,39 @@
|
||||
# Excititor Tenant Authority Client (AOC-19-013)
|
||||
|
||||
- **Date:** 2025-11-21
|
||||
- **Scope:** EXCITITOR-CORE-AOC-19-013
|
||||
- **Files:** `src/Excititor/StellaOps.Excititor.Worker/Auth/TenantAuthorityClientFactory.cs`
|
||||
|
||||
## Contract
|
||||
- Every outbound Authority call must carry `X-Tenant` header and use tenant-specific base URL.
|
||||
- Base URLs and optional client credentials are configured under `Excititor:Authority:` with per-tenant keys.
|
||||
- Factory throws when tenant is missing or not configured to prevent cross-tenant leakage.
|
||||
|
||||
## Configuration shape
|
||||
```json
|
||||
{
|
||||
"Excititor": {
|
||||
"Authority": {
|
||||
"BaseUrls": {
|
||||
"alpha": "https://authority.alpha.local/",
|
||||
"bravo": "https://authority.bravo.local/"
|
||||
},
|
||||
"ClientIds": {
|
||||
"alpha": "alpha-client-id"
|
||||
},
|
||||
"ClientSecrets": {
|
||||
"alpha": "alpha-secret"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Implementation notes
|
||||
- `TenantAuthorityClientFactory` (worker) enforces tenant presence and configured base URL; adds `Accept: application/json` and `X-Tenant` headers.
|
||||
- Registered in DI via `Program.cs` with options binding to `Excititor:Authority`.
|
||||
- Intended to be reused by WebService/Worker components once disk space block is resolved.
|
||||
|
||||
## Next steps
|
||||
- Wire factory into services that call Authority (WebService + Worker jobs), replacing any tenant-agnostic HttpClient usages.
|
||||
- Add integration tests to ensure cross-tenant calls reject when config missing or header mismatched.
|
||||
66
docs-archived/modules/excititor/operations/ubuntu-csaf.md
Normal file
66
docs-archived/modules/excititor/operations/ubuntu-csaf.md
Normal file
@@ -0,0 +1,66 @@
|
||||
# Ubuntu CSAF connector runbook
|
||||
|
||||
> Updated 2025-11-09 alongside sprint 110/120 trust-provenance work.
|
||||
|
||||
## Purpose
|
||||
- Ingest Ubuntu USN/CSAF statements via the restart-only connector (`StellaOps.Excititor.Connectors.Ubuntu.CSAF`).
|
||||
- Preserve Aggregation-Only Contract guarantees while surfacing issuance provenance (`vex.provenance.*`) for VEX Lens and Policy Engine.
|
||||
- Allow operators to tune trust weighting (tiers, fingerprints, cosign issuers) without recompiling the connector.
|
||||
|
||||
## Configuration keys
|
||||
| Key | Default | Notes |
|
||||
| --- | --- | --- |
|
||||
| `Excititor:Connectors:Ubuntu:IndexUri` | `https://ubuntu.com/security/csaf/index.json` | Ubuntu CSAF index. Override only when mirroring the feed. |
|
||||
| `...:Channels` | `["stable"]` | List of channel names to poll. Order preserved for deterministic cursoring. |
|
||||
| `...:MetadataCacheDuration` | `4h` | How long to cache catalog metadata before re-fetching. |
|
||||
| `...:PreferOfflineSnapshot` / `OfflineSnapshotPath` / `PersistOfflineSnapshot` | `false` / `null` / `true` | Enable when running from Offline Kit bundles. Snapshot path must be reachable/read-only under sealed deployments. |
|
||||
| `...:TrustWeight` | `0.75` | Baseline trust weight (0–1). Lens multiplies this by freshness/justification modifiers. |
|
||||
| `...:TrustTier` | `"distro"` | Friendly tier label surfaced via `vex.provenance.trust.tier` (e.g., `distro-trusted`, `community`). |
|
||||
| `...:CosignIssuer` / `CosignIdentityPattern` | `null` | Supply when Ubuntu publishes cosign attestations (issuer URL and identity regex). Required together. |
|
||||
| `...:PgpFingerprints` | `[]` | Ordered list of trusted PGP fingerprints. Emitted verbatim as `vex.provenance.pgp.fingerprints`. |
|
||||
|
||||
## Example `appsettings.json`
|
||||
```jsonc
|
||||
{
|
||||
"Excititor": {
|
||||
"Connectors": {
|
||||
"Ubuntu": {
|
||||
"IndexUri": "https://mirror.example.com/security/csaf/index.json",
|
||||
"Channels": ["stable", "esm-apps"],
|
||||
"TrustWeight": 0.82,
|
||||
"TrustTier": "distro-trusted",
|
||||
"CosignIssuer": "https://issuer.ubuntu.com",
|
||||
"CosignIdentityPattern": "spiffe://ubuntu/vex/*",
|
||||
"PgpFingerprints": [
|
||||
"0123456789ABCDEF0123456789ABCDEF01234567",
|
||||
"89ABCDEF0123456789ABCDEF0123456789ABCDEF"
|
||||
],
|
||||
"PreferOfflineSnapshot": true,
|
||||
"OfflineSnapshotPath": "/opt/stella/offline/ubuntu/index.json"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Environment variable cheatsheet
|
||||
```
|
||||
Excititor__Connectors__Ubuntu__TrustWeight=0.9
|
||||
Excititor__Connectors__Ubuntu__TrustTier=distro-critical
|
||||
Excititor__Connectors__Ubuntu__PgpFingerprints__0=AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
|
||||
Excititor__Connectors__Ubuntu__PgpFingerprints__1=BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB
|
||||
Excititor__Connectors__Ubuntu__CosignIssuer=https://issuer.ubuntu.com
|
||||
Excititor__Connectors__Ubuntu__CosignIdentityPattern=spiffe://ubuntu/vex/*
|
||||
```
|
||||
|
||||
## Operational checklist
|
||||
1. **Before enabling** – import the Ubuntu PGP bundle (Offline Kit provides `certificates/ubuntu-vex.gpg`) and set the fingerprints so provenance metadata stays deterministic.
|
||||
2. **Validate provenance output** – run `dotnet test src/Excititor/__Tests/StellaOps.Excititor.Connectors.Ubuntu.CSAF.Tests --filter FetchAsync_IngestsNewDocument` to ensure the connector emits the `vex.provenance.*` fields expected by VEX Lens.
|
||||
3. **Monitor Lens weights** – Grafana panels `VEX Lens / Trust Inputs` show the weight/tier captured per provider. Ubuntu rows should reflect the configured `TrustWeight` and fingerprints.
|
||||
4. **Rotate fingerprints** – update `PgpFingerprints` when Canonical rotates signing keys. Apply the change, restart Excititor workers, verify the provenance metadata, then trigger a targeted Lens recompute for Ubuntu issuers.
|
||||
5. **Offline mode** – populate `OfflineSnapshotPath` via Offline Kit bundles before toggling `PreferOfflineSnapshot`. Keep snapshots in the sealed `/opt/stella/offline` hierarchy for auditability.
|
||||
|
||||
## Troubleshooting
|
||||
- **Connector refuses to start** – check logs for `InvalidOperationException` referencing `CosignIssuer`/`CosignIdentityPattern` or missing snapshot path; the validator enforces complete pairs and on-disk paths.
|
||||
- **Lens still sees default weights** – confirm the Excititor deployment picked up the new settings (view `/excititor/health` JSON → `connectors.providers[].options`). Lens only overrides when the provenance payload includes `vex.provenance.trust.*` fields.
|
||||
- **PGP mismatch alerts** – if Lens reports fingerprint mismatches, ensure the list ordering matches Canonical’s published order; duplicates are trimmed, so provide each fingerprint once.
|
||||
Reference in New Issue
Block a user