feat: Implement Filesystem and MongoDB provenance writers for PackRun execution context
- Added `FilesystemPackRunProvenanceWriter` to write provenance manifests to the filesystem. - Introduced `MongoPackRunArtifactReader` to read artifacts from MongoDB. - Created `MongoPackRunProvenanceWriter` to store provenance manifests in MongoDB. - Developed unit tests for filesystem and MongoDB provenance writers. - Established `ITimelineEventStore` and `ITimelineIngestionService` interfaces for timeline event handling. - Implemented `TimelineIngestionService` to validate and persist timeline events with hashing. - Created PostgreSQL schema and migration scripts for timeline indexing. - Added dependency injection support for timeline indexer services. - Developed tests for timeline ingestion and schema validation.
This commit is contained in:
StellaOps Bot
@@ -8,6 +8,7 @@ VEX Lens computes deterministic consensus across conflicting VEX statements whil
|
||||
- [Architecture](./architecture.md)
|
||||
- [Implementation plan](./implementation_plan.md)
|
||||
- [Task board](./TASKS.md)
|
||||
- [Observability runbook](./runbooks/observability.md) (offline import friendly)
|
||||
|
||||
## How to get started
|
||||
1. Review ./architecture.md for consensus algorithm, trust model, and export contracts.
|
||||
|
||||
@@ -27,6 +27,10 @@ VEX Lens produces a deterministic, provenance-rich consensus view of VEX stateme
|
||||
- `VEXLENS-30-005..007` — expose consensus APIs and export flows, aligning docs with future `/docs/vex/consensus-*.md` deliverables.
|
||||
- `DOCS-VEX-30-001..004` — broader documentation set (overview, algorithm, API) tracked in `docs/TASKS.md`; update this README again once those artefacts merge.
|
||||
|
||||
## Latest updates (2025-11-30)
|
||||
- Docs refresh per `docs/implplan/SPRINT_0332_0001_0001_docs_modules_vex_lens.md`; added observability runbook stub and TASKS mirror.
|
||||
- Observability assets: `runbooks/observability.md` and `runbooks/dashboards/vex-lens-observability.json` (offline import).
|
||||
|
||||
## Integrations & dependencies
|
||||
- **Excititor** supplies signature-verified VEX observations and issuer hints.
|
||||
- **Policy Engine** consumes consensus verdicts for suppression/waiver logic and exposes trust controls to operators.
|
||||
@@ -44,7 +48,7 @@ VEX Lens produces a deterministic, provenance-rich consensus view of VEX stateme
|
||||
- [`architecture.md`](architecture.md) — implementation-ready blueprint covering inputs, algorithm, APIs, storage, observability, and exports.
|
||||
- [`implementation_plan.md`](implementation_plan.md) — phased delivery roadmap and acceptance criteria.
|
||||
- [`../../vex/aggregation.md`](../../vex/aggregation.md) — Aggregation-Only Contract boundaries for VEX ingestion and downstream consumers.
|
||||
- Sprint tracking in `docs/implplan/SPRINT_200_documentation_process.md` and module engineering tasks in `src/VexLens/StellaOps.VexLens/TASKS.md`.
|
||||
- Sprint tracking in `docs/implplan/SPRINT_0332_0001_0001_docs_modules_vex_lens.md`; module engineering tasks in `src/VexLens/StellaOps.VexLens/TASKS.md`; doc TASKS mirror in `docs/modules/vex-lens/TASKS.md`.
|
||||
|
||||
## Epic alignment
|
||||
- Epic 7 — VEX Consensus Lens: deterministic VEX adjudication, issuer directory, consensus exports.
|
||||
|
||||
9
docs/modules/vex-lens/TASKS.md
Normal file
9
docs/modules/vex-lens/TASKS.md
Normal file
@@ -0,0 +1,9 @@
|
||||
# VEX Lens · TASKS (status mirror)
|
||||
|
||||
| Task ID | Status | Owner(s) | Notes / Evidence |
|
||||
| --- | --- | --- | --- |
|
||||
| VEX-CONSENSUS-LENS-DOCS-0001 | DONE (2025-11-30) | Docs Guild | README/architecture refresh with consensus workflow and release links; sprint references added. |
|
||||
| VEX-LENS-OPS-0001 | DONE (2025-11-30) | Ops Guild | Observability runbook stub + Grafana JSON placeholder added under `runbooks/`. |
|
||||
| VEX-LENS-ENG-0001 | DONE (2025-11-30) | Module Team | TASKS board created; statuses mirrored with `docs/implplan/SPRINT_0332_0001_0001_docs_modules_vex_lens.md`. |
|
||||
|
||||
> Keep this table in lockstep with the sprint Delivery Tracker (TODO/DOING/DONE/BLOCKED updates go to both places).
|
||||
@@ -71,13 +71,14 @@ All responses include provenance fields (`consensus_digest`, `derived_from`, DSS
|
||||
- Recompute jobs run via Orchestrator; deterministic ordering ensures identical results for the same input set.
|
||||
- Jobs produce SRM-style manifests for recomputation verification.
|
||||
|
||||
## 7) Observability
|
||||
|
||||
- Metrics: `vex_consensus_conflicts_total`, `vex_consensus_latency_seconds`, `vex_consensus_recompute_seconds{reason}`.
|
||||
- Logs: include `artifactId`, `advisoryKey`, `issuer`, `status`, `trustTier`.
|
||||
- Traces: `consensus.group`, `consensus.join`, `consensus.persist` spans.
|
||||
|
||||
## 8) Offline & export
|
||||
|
||||
- Bundle format: `consensus.jsonl`, `conflicts.jsonl`, `manifest.json`, `signatures/`. Each record references raw statement digests and trust metadata.
|
||||
- Export Center uses the bundle for mirror profiles; CLI supports `stella vex consensus export` mirroring the API.
|
||||
## 7) Observability
|
||||
|
||||
- Metrics: `vex_consensus_conflicts_total`, `vex_consensus_latency_seconds`, `vex_consensus_recompute_seconds{reason}`.
|
||||
- Logs: include `artifactId`, `advisoryKey`, `issuer`, `status`, `trustTier`.
|
||||
- Traces: `consensus.group`, `consensus.join`, `consensus.persist` spans.
|
||||
- Runbook + dashboard stub (offline import): `runbooks/observability.md`, `runbooks/dashboards/vex-lens-observability.json`.
|
||||
|
||||
## 8) Offline & export
|
||||
|
||||
- Bundle format: `consensus.jsonl`, `conflicts.jsonl`, `manifest.json`, `signatures/`. Each record references raw statement digests and trust metadata.
|
||||
- Export Center uses the bundle for mirror profiles; CLI supports `stella vex consensus export` mirroring the API.
|
||||
|
||||
@@ -61,3 +61,8 @@
|
||||
- Lens service, issuer directory, API/CLI/Console components deployed with telemetry and runbooks.
|
||||
- Documentation set (overview, algorithm, issuer directory, API, console, policy trust) updated with imposed rule statements.
|
||||
- ./TASKS.md and ../../TASKS.md reflect current status; Offline Kit parity confirmed.
|
||||
|
||||
## Sprint alignment (2025-11-30)
|
||||
- Docs refresh tracked in `docs/implplan/SPRINT_0332_0001_0001_docs_modules_vex_lens.md`; statuses mirrored in `docs/modules/vex-lens/TASKS.md`.
|
||||
- Observability evidence lives in `runbooks/observability.md` with Grafana JSON stub under `runbooks/dashboards/`.
|
||||
- Keep future doc/ops updates mirrored across sprint, TASKS, and module front doors to avoid drift.
|
||||
|
||||
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"_note": "Placeholder Grafana dashboard stub for VEX Lens. Replace panels when metrics endpoints are available; keep offline-import friendly.",
|
||||
"schemaVersion": 39,
|
||||
"title": "VEX Lens Observability (stub)",
|
||||
"panels": []
|
||||
}
|
||||
38
docs/modules/vex-lens/runbooks/observability.md
Normal file
38
docs/modules/vex-lens/runbooks/observability.md
Normal file
@@ -0,0 +1,38 @@
|
||||
# VEX Lens observability runbook (stub · 2025-11-29 demo)
|
||||
|
||||
## Dashboards (offline import)
|
||||
- Grafana JSON: `docs/modules/vex-lens/runbooks/dashboards/vex-lens-observability.json` (import locally; no external data sources assumed).
|
||||
- Planned panels: consensus latency, conflict backlog, recompute duration, issuer trust changes, export job success rate, and DSSE verification failures.
|
||||
|
||||
## Key metrics
|
||||
- `vex_consensus_latency_seconds_bucket` — latency from observation intake to consensus write.
|
||||
- `vex_conflict_queue_depth` — size of unresolved conflict queue.
|
||||
- `vex_recompute_duration_seconds_bucket{reason}` — recompute times by trigger (issuer update, policy knob, ingestion delta).
|
||||
- `vex_export_duration_seconds_bucket` — export job runtime.
|
||||
- `vex_dsse_verification_failures_total` — failed attestations during export/ingest.
|
||||
- `vex_consensus_conflicts_total{reason}` — conflict counts by reason (status disagreement, scope mismatch, missing provenance).
|
||||
|
||||
## Logs & traces
|
||||
- Correlate by `correlationId`, `artifactKey`, `advisoryKey`, and `issuer`. Include `trustTier`, `weightBefore`, `weightAfter`, and `justification` fields for audits.
|
||||
- Traces disabled by default for air-gap; enable by setting `Telemetry:ExportEnabled=true` and pointing OTLP endpoint to on-prem collector.
|
||||
|
||||
## Health/diagnostics
|
||||
- `/health/liveness` and `/health/readiness` (service) must return 200; readiness checks Mongo + cache + event bus reachability.
|
||||
- `/status` exposes build version, commit, feature flags; verify it matches offline bundle manifest.
|
||||
- Export self-check: run `stella vex export --format json --manifest out/manifest.json` and validate hashes against manifest entries.
|
||||
|
||||
## Alert hints
|
||||
- Consensus latency p99 > 1.5s over 5m.
|
||||
- Conflict queue depth > 500 for any tenant.
|
||||
- DSSE verification failures > 0 in a 10m window.
|
||||
- Export failure rate > 2% over 10m.
|
||||
|
||||
## Offline verification steps
|
||||
1) Import Grafana JSON locally; point to Prometheus scrape labeled `vex-lens`.
|
||||
2) Run export CLI above and verify `manifest.json` hashes via `jq -r '.files[].sha256'`.
|
||||
3) Fetch `/status` and confirm commit/version match the exported manifest and offline kit bundle metadata.
|
||||
|
||||
## Evidence locations
|
||||
- Sprint tracker: `docs/implplan/SPRINT_0332_0001_0001_docs_modules_vex_lens.md`.
|
||||
- Module docs: `README.md`, `architecture.md`, `implementation_plan.md`.
|
||||
- Dashboard stub: `runbooks/dashboards/vex-lens-observability.json`.
|
||||
Reference in New Issue
Block a user