Add tests and implement timeline ingestion options with NATS and Redis subscribers
- Introduced `BinaryReachabilityLifterTests` to validate binary lifting functionality. - Created `PackRunWorkerOptions` for configuring worker paths and execution persistence. - Added `TimelineIngestionOptions` for configuring NATS and Redis ingestion transports. - Implemented `NatsTimelineEventSubscriber` for subscribing to NATS events. - Developed `RedisTimelineEventSubscriber` for reading from Redis Streams. - Added `TimelineEnvelopeParser` to normalize incoming event envelopes. - Created unit tests for `TimelineEnvelopeParser` to ensure correct field mapping. - Implemented `TimelineAuthorizationAuditSink` for logging authorization outcomes.
This commit is contained in:
StellaOps Bot
@@ -1,18 +1,42 @@
|
||||
# Sprint 123 - Ingestion & Evidence · 110.C) Excititor.V
|
||||
# Sprint 0123 · Excititor Ingestion & Evidence (Phase V)
|
||||
|
||||
Active items only. Completed/historic work now resides in docs/implplan/archived/tasks.md (updated 2025-11-08).
|
||||
## Topic & Scope
|
||||
- Feed VEX Lens and Vuln Explorer with enriched, canonical evidence while keeping Excititor aggregation-only.
|
||||
- Lock raw storage validation/idempotency and ship portable evidence bundles plus mirror registration APIs for air-gapped parity.
|
||||
- **Working directory:** `src/Excititor` (WebService, Core, Storage.Mongo) and docs/airgap.
|
||||
|
||||
[Ingestion & Evidence] 110.C) Excititor.V
|
||||
Depends on: Sprint 110.C - Excititor.IV
|
||||
Summary: Ingestion & Evidence focus on Excititor (phase V).
|
||||
> **Prep:** Read `docs/modules/excititor/architecture.md` and the Excititor component `AGENTS.md` files before touching this sprint’s tasks.
|
||||
Task ID | State | Task description | Owners (Source)
|
||||
--- | --- | --- | ---
|
||||
EXCITITOR-VEXLENS-30-001 `VEX evidence enrichers` | DONE | Ensure every observation exported to VEX Lens carries issuer hints, signature blobs, product tree snippets, and staleness metadata so the lens can compute consensus without calling back into Excititor. **Completed:** Enhanced `OpenVexSourceEntry` with enrichment fields (issuerHint, signatureType, keyId, transparencyLogRef, trustWeight, trustTier, stalenessSeconds, productTreeSnippet). Updated `OpenVexStatementMerger.BuildSources()` to extract from VexClaim. Enhanced `OpenVexExportSource` JSON serialization. | Excititor WebService Guild, VEX Lens Guild (src/Excititor/StellaOps.Excititor.WebService)
|
||||
EXCITITOR-VULN-29-001 `VEX key canonicalization` | DONE | Canonicalize advisory/product keys (map to `advisory_key`, capture scope metadata) while preserving original identifiers in `links[]`; run backfill + regression tests. **Completed:** Created `VexAdvisoryKeyCanonicalizer` (CVE/GHSA/RHSA/DSA/USN) and `VexProductKeyCanonicalizer` (PURL/CPE/RPM/DEB/OCI) in `Core/Canonicalization/`. All 47 tests passing. Supports extracting PURLs/CPEs from component identifiers. | Excititor WebService Guild (src/Excititor/StellaOps.Excititor.WebService)
|
||||
EXCITITOR-VULN-29-002 `Evidence retrieval APIs` | DONE | Provide `/vuln/evidence/vex/{advisory_key}` returning tenant-scoped raw statements, provenance, and attestation references for Vuln Explorer evidence tabs. Depends on EXCITITOR-VULN-29-001. **Completed:** Created endpoint at `/vuln/evidence/vex/{advisory_key}` in `EvidenceEndpoints.cs`. Uses `VexAdvisoryKeyCanonicalizer` for key normalization (CVE, GHSA, RHSA, DSA, USN). Returns canonical key, scope, aliases, and statements with provenance (documentDigest, format, sourceUri, revision) and attestation metadata (signatureType, issuer, subject, keyId, verifiedAt, transparencyLogRef, trustWeight, trustTier). Supports cursor-based pagination. | Excititor WebService Guild (src/Excititor/StellaOps.Excititor.WebService)
|
||||
EXCITITOR-VULN-29-004 `Observability` | DONE | Add metrics/logs for normalization errors, suppression scopes, withdrawn statements, and feed them to Vuln Explorer + Advisory AI dashboards. Depends on EXCITITOR-VULN-29-002. **Completed:** Created `NormalizationTelemetry.cs` with comprehensive metrics: advisory/product key canonicalization (success/error counters, scope distribution), evidence retrieval (request counts, statement count histogram, latency histogram), normalization errors by provider/type, suppression scope tracking, withdrawn statement detection/replacements. Registered meter in `TelemetryExtensions.cs`. Added telemetry calls to evidence endpoint. | Excititor WebService Guild, Observability Guild (src/Excititor/StellaOps.Excititor.WebService)
|
||||
EXCITITOR-STORE-AOC-19-001 `vex_raw schema validator` | DONE | Ship Mongo JSON Schema + validator tooling (including Offline Kit instructions) so operators can prove Excititor stores only immutable evidence. **Completed:** Created `VexRawSchemaValidator` in `Storage.Mongo/Validation/` with `Validate()`, `ValidateBatch()`, `GetJsonSchema()` methods. Added Offline Kit docs at `docs/airgap/vex-raw-schema-validation.md`. | Excititor Storage Guild (src/Excititor/__Libraries/StellaOps.Excititor.Storage.Mongo)
|
||||
EXCITITOR-STORE-AOC-19-002 `Idempotency index & migration` | DONE | Create unique indexes, run migrations/backfills, and document rollback steps for the new schema validator. Depends on EXCITITOR-STORE-AOC-19-001. **Completed:** Created `VexRawIdempotencyIndexMigration` with unique indexes (provider+source+digest), query indexes (digest+provider), and time-based index. Added rollback docs at `docs/airgap/vex-raw-migration-rollback.md`. Registered migration in ServiceCollectionExtensions. | Excititor Storage Guild, DevOps Guild (src/Excititor/__Libraries/StellaOps.Excititor.Storage.Mongo)
|
||||
EXCITITOR-AIRGAP-56-001 `Mirror registration APIs` | DONE | Support mirror bundle registration + provenance exposure, including sealed-mode error mapping and staleness metrics surfaced via API responses. **Completed:** Added query methods to `IAirgapImportStore` (ListAsync, FindByBundleIdAsync, CountAsync). Created `AirgapMirrorContracts.cs` with response types for bundle listing/detail/timeline and `AirgapErrorMapping` for structured sealed-mode errors with categories (validation, sealed_mode, trust, duplicate, not_found). Created `MirrorRegistrationEndpoints.cs` with `/airgap/v1/mirror/bundles` endpoints for listing, detail with provenance, and timeline queries. Added `StalenessCalculator` for computing staleness metrics with age categories (fresh, recent, stale, old, very_old). | Excititor WebService Guild (src/Excititor/StellaOps.Excititor.WebService)
|
||||
EXCITITOR-AIRGAP-58-001 `Portable evidence bundles` | DONE | Produce portable evidence bundles linked to timeline + attestation metadata for sealed deployments, and document verifier steps for Advisory AI teams. Depends on EXCITITOR-AIRGAP-56-001. **Completed:** Created `PortableEvidenceBundleBuilder` in `Core/Evidence/` with ZIP bundle creation including: manifest.json (VexLockerManifest with Merkle root), attestation.json (DSSE envelope), evidence items by provider, timeline.json (audit trail), bundle-manifest.json (content index), VERIFY.md (inline verification guide). Added comprehensive verification docs at `docs/airgap/portable-evidence-bundle-verification.md` with Python/Bash scripts for Merkle root verification, attestation checking, timeline validation, and Advisory AI integration patterns. | Excititor Core Guild, Evidence Locker Guild (src/Excititor/__Libraries/StellaOps.Excititor.Core)
|
||||
## Dependencies & Concurrency
|
||||
- Depends on Phase IV outputs (timeline/locker/attestation) and mirror registration contract.
|
||||
- Concurrency: storage validators/indexes first; VEX Lens/Vuln endpoints rely on canonicalization; portable bundles depend on mirror registration endpoints.
|
||||
|
||||
## Documentation Prerequisites
|
||||
- docs/modules/excititor/architecture.md
|
||||
- docs/modules/excititor/implementation_plan.md
|
||||
- docs/airgap/portable-evidence-bundle-verification.md
|
||||
- Excititor AGENTS.md files (WebService, Core, Storage)
|
||||
|
||||
## Delivery Tracker
|
||||
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| 1 | EXCITITOR-VEXLENS-30-001 | DONE | None | Excititor WebService Guild · VEX Lens Guild | Export observations to VEX Lens with issuer hints, signature blobs, product tree snippets, staleness metadata; no consensus logic. |
|
||||
| 2 | EXCITITOR-VULN-29-001 | DONE | None | Excititor WebService Guild | Canonicalize advisory/product keys (`advisory_key`), preserve originals in `links[]`; backfill + tests. |
|
||||
| 3 | EXCITITOR-VULN-29-002 | DONE | Depends on 29-001 | Excititor WebService Guild | `/vuln/evidence/vex/{advisory_key}` returning tenant-scoped raw statements + provenance + attestation references; cursor pagination. |
|
||||
| 4 | EXCITITOR-VULN-29-004 | DONE | Depends on 29-002 | Excititor WebService Guild · Observability Guild | Metrics/logs for normalization errors, suppression scopes, withdrawn statements for Vuln Explorer + Advisory AI dashboards. |
|
||||
| 5 | EXCITITOR-STORE-AOC-19-001 | DONE | None | Excititor Storage Guild | Mongo JSON Schema validator for `vex_raw`; offline kit instructions. |
|
||||
| 6 | EXCITITOR-STORE-AOC-19-002 | DONE | Depends on 19-001 | Excititor Storage Guild · DevOps Guild | Unique indexes/migrations/rollback steps for new validator. |
|
||||
| 7 | EXCITITOR-AIRGAP-56-001 | DONE | None | Excititor WebService Guild · AirGap Importer Guild | Mirror bundle registration + provenance exposure, sealed-mode error mapping, staleness metrics. |
|
||||
| 8 | EXCITITOR-AIRGAP-58-001 | DONE | Depends on 56-001 | Excititor Core Guild · Evidence Locker Guild | Portable evidence bundles linked to timeline + attestation metadata; verifier docs for Advisory AI. |
|
||||
|
||||
## Execution Log
|
||||
| Date (UTC) | Update | Owner |
|
||||
| --- | --- | --- |
|
||||
| 2025-12-03 | Normalised sprint to standard template; working directory set; preserved statuses. | Planning |
|
||||
|
||||
## Decisions & Risks
|
||||
- Aggregation-only posture retained: exports and APIs do not compute verdicts.
|
||||
- Validator rollout could impact ingestion; staged with rollback docs. Ensure unique indexes deployed before enabling enforcement.
|
||||
- Portable bundle contents and mirror registration must stay aligned with Evidence Locker schemas; refresh docs/tests if schema evolves.
|
||||
|
||||
## Next Checkpoints
|
||||
- Re-run bundle verification scripts after any Evidence Locker manifest changes.
|
||||
- Validate VEX Lens/Vuln Explorer still ingest canonicalized keys after downstream schema tweaks.
|
||||
|
||||
Reference in New Issue
Block a user