Implement MongoDB-based storage for Pack Run approval, artifact, log, and state management
Some checks failed
Docs CI / lint-and-preview (push) Has been cancelled

- Added MongoPackRunApprovalStore for managing approval states with MongoDB.
- Introduced MongoPackRunArtifactUploader for uploading and storing artifacts.
- Created MongoPackRunLogStore to handle logging of pack run events.
- Developed MongoPackRunStateStore for persisting and retrieving pack run states.
- Implemented unit tests for MongoDB stores to ensure correct functionality.
- Added MongoTaskRunnerTestContext for setting up MongoDB test environment.
- Enhanced PackRunStateFactory to correctly initialize state with gate reasons.
This commit is contained in:
master
2025-11-07 10:01:35 +02:00
parent e5ffcd6535
commit a1ce3f74fa
122 changed files with 8730 additions and 914 deletions

View File

@@ -14,7 +14,7 @@
- `CONCELIER-GRAPH-21-001`, `CONCELIER-GRAPH-21-002`, and `CONCELIER-GRAPH-21-005` remain BLOCKED awaiting `CONCELIER-POLICY-20-002` outputs and Cartographer schema (`CARTO-GRAPH-21-002`), keeping downstream Excititor graph consumers on hold.
- `EXCITITOR-GRAPH-21-001`, `EXCITITOR-GRAPH-21-002`, and `EXCITITOR-GRAPH-21-005` stay BLOCKED until the same Cartographer/Link-Not-Merge prerequisites are delivered.
- Connector provenance updates `FEEDCONN-ICSCISA-02-012` (due 2025-10-23) and `FEEDCONN-KISA-02-008` (due 2025-10-24) plus coordination items `FEEDMERGE-COORD-02-901`/`FEEDMERGE-COORD-02-902`/`FEEDMERGE-COORD-02-903` (due 2025-10-21 through 2025-10-24) are past due and need scheduling.
- Connector provenance updates `FEEDCONN-ICSCISA-02-012` (due 2025-10-23) and `FEEDCONN-KISA-02-008` (due 2025-10-24) remain past due and need scheduling. FeedMerge coordination tasks have been dropped (no AOC policy/governance backing yet), so capacity shifts to schema/guard deliverables.
- Mirror evidence work remains blocked until `MIRROR-CRT-56-001` ships; align Export Center (`EXPORT-OBS-51-001`) and AirGap time anchor (`AIRGAP-TIME-57-001`) owners for kickoff.
[Ingestion & Evidence] 110.A) AdvisoryAI
@@ -29,15 +29,15 @@ AIAI-31-004 | DONE (2025-11-04) | Build orchestration pipeline for Summary/Confl
AIAI-31-004A | DONE (2025-11-04) | Wire orchestrator into WebService/Worker, expose API + queue contract, emit metrics, stub cache. Dependencies: AIAI-31-004, AIAI-31-002. | Advisory AI Guild, Platform Guild (src/AdvisoryAI/StellaOps.AdvisoryAI/TASKS.md)
> 2025-11-03: WebService/Worker scaffolds created with in-memory cache/queue, minimal APIs (`/api/v1/advisory/plan`, `/api/v1/advisory/queue`), metrics counters, and plan cache instrumentation; worker processes queue using orchestrator.
> 2025-11-04: SBOM base address now flows via `SbomContextClientOptions.BaseAddress`, worker emits queue/plan metrics, and orchestrator cache keys expanded to cover SBOM hash inputs.
AIAI-31-004B | TODO | Implement prompt assembler, guardrails, cache persistence, DSSE provenance, golden outputs. Dependencies: AIAI-31-004A, DOCS-AIAI-31-003, AUTH-AIAI-31-004. | Advisory AI Guild, Security Guild (src/AdvisoryAI/StellaOps.AdvisoryAI/TASKS.md)
AIAI-31-004C | TODO | Deliver CLI `stella advise run` command, renderer, docs, CLI golden tests. Dependencies: AIAI-31-004B, CLI-AIAI-31-003. | Advisory AI Guild, CLI Guild (src/AdvisoryAI/StellaOps.AdvisoryAI/TASKS.md)
AIAI-31-004B | DONE (2025-11-06) | Implement prompt assembler, guardrails, cache persistence, DSSE provenance, golden outputs. Dependencies: AIAI-31-004A, DOCS-AIAI-31-003, AUTH-AIAI-31-004. | Advisory AI Guild, Security Guild (src/AdvisoryAI/StellaOps.AdvisoryAI/TASKS.md)
AIAI-31-004C | DONE (2025-11-06) | Deliver CLI `stella advise run` command, renderer, docs, CLI golden tests. Dependencies: AIAI-31-004B, CLI-AIAI-31-003. | Advisory AI Guild, CLI Guild (src/AdvisoryAI/StellaOps.AdvisoryAI/TASKS.md)
DOCS-AIAI-31-002 | DONE (2025-11-03) | Author `/docs/advisory-ai/architecture.md` detailing RAG pipeline, deterministic tooling, caching, model profiles. Dependencies: AIAI-31-004. | Docs Guild, Advisory AI Guild (docs/TASKS.md)
DOCS-AIAI-31-001 | DONE (2025-11-03) | Publish `/docs/advisory-ai/overview.md` covering capabilities, guardrails, RBAC personas, and offline posture. | Docs Guild, Advisory AI Guild (docs/TASKS.md)
DOCS-AIAI-31-003 | DONE (2025-11-03) | Write `/docs/advisory-ai/api.md` covering endpoints, schemas, errors, rate limits, and imposed-rule banner. Dependencies: DOCS-AIAI-31-002. | Docs Guild, Advisory AI Guild (docs/TASKS.md)
DOCS-AIAI-31-004 | BLOCKED (2025-11-03) | Create `/docs/advisory-ai/console.md` with screenshots, a11y notes, copy-as-ticket instructions. Dependencies: CONSOLE-VULN-29-001, CONSOLE-VEX-30-001, EXCITITOR-CONSOLE-23-001. | Docs Guild, Console Guild (docs/TASKS.md)
DOCS-AIAI-31-005 | BLOCKED (2025-11-03) | Publish `/docs/advisory-ai/cli.md` covering commands, exit codes, scripting patterns. Dependencies: CLI-VULN-29-001, CLI-VEX-30-001, AIAI-31-004C. | Docs Guild, DevEx/CLI Guild (docs/TASKS.md)
DOCS-AIAI-31-006 | BLOCKED (2025-11-03) | Update `/docs/policy/assistant-parameters.md` covering temperature, token limits, ranking weights, TTLs. Dependencies: POLICY-ENGINE-31-001. | Docs Guild, Policy Guild (docs/TASKS.md)
DOCS-AIAI-31-007 | BLOCKED (2025-11-03) | Write `/docs/security/assistant-guardrails.md` detailing redaction, injection defense, logging. Dependencies: AIAI-31-005. | Docs Guild, Security Guild (docs/TASKS.md)
DOCS-AIAI-31-007 | DONE (2025-11-07) | Write `/docs/security/assistant-guardrails.md` detailing redaction, injection defense, logging. Dependencies: AIAI-31-005. | Docs Guild, Security Guild (docs/TASKS.md)
DOCS-AIAI-31-008 | BLOCKED (2025-11-03) | Publish `/docs/sbom/remediation-heuristics.md` (feasibility scoring, blast radius). Dependencies: SBOM-AIAI-31-001. | Docs Guild, SBOM Service Guild (docs/TASKS.md)
DOCS-AIAI-31-009 | BLOCKED (2025-11-03) | Create `/docs/runbooks/assistant-ops.md` for warmup, cache priming, model outages, scaling. Dependencies: DEVOPS-AIAI-31-001. | Docs Guild, DevOps Guild (docs/TASKS.md)
> 2025-11-03: DOCS-AIAI-31-003 moved to DOING drafting Advisory AI API reference (endpoints, rate limits, error model) for sprint 110.
@@ -48,13 +48,14 @@ DOCS-AIAI-31-009 | BLOCKED (2025-11-03) | Create `/docs/runbooks/assistant-ops.m
> 2025-11-03: DOCS-AIAI-31-004 marked BLOCKED Console widgets/endpoints (CONSOLE-VULN-29-001, CONSOLE-VEX-30-001, EXCITITOR-CONSOLE-23-001) still pending; cannot document UI flows yet.
> 2025-11-03: DOCS-AIAI-31-005 marked BLOCKED CLI implementation (`stella advise run`, CLI-VULN-29-001, CLI-VEX-30-001) plus AIAI-31-004C not shipped; doc blocked until commands exist.
> 2025-11-03: DOCS-AIAI-31-006 marked BLOCKED Advisory AI parameter knobs (POLICY-ENGINE-31-001) absent; doc deferred.
> 2025-11-03: DOCS-AIAI-31-007 marked BLOCKED Guardrail implementation (AIAI-31-005) incomplete.
> 2025-11-07: DOCS-AIAI-31-007 marked DONE `/docs/security/assistant-guardrails.md` now documents redaction rules, blocked phrases, telemetry, and alert procedures.
> 2025-11-03: DOCS-AIAI-31-008 marked BLOCKED Waiting on SBOM heuristics delivery (SBOM-AIAI-31-001).
> 2025-11-03: DOCS-AIAI-31-009 marked BLOCKED DevOps runbook inputs (DEVOPS-AIAI-31-001) outstanding.
AIAI-31-005 | DONE (2025-11-04) | Implement guardrails (redaction, injection defense, output validation, citation enforcement) and fail-safe handling. Dependencies: AIAI-31-004. | Advisory AI Guild, Security Guild (src/AdvisoryAI/StellaOps.AdvisoryAI/TASKS.md)
AIAI-31-006 | DONE (2025-11-04) | Expose REST API endpoints (`/advisory/ai/*`) with RBAC, rate limits, OpenAPI schemas, and batching support. Dependencies: AIAI-31-004..005. | Advisory AI Guild (src/AdvisoryAI/StellaOps.AdvisoryAI/TASKS.md)
> 2025-11-03: Shipped `/api/v1/advisory/{task}` execution and `/api/v1/advisory/outputs/{cacheKey}` retrieval endpoints with guardrail integration, provenance hashes, and metrics (RBAC & rate limiting still pending Authority scope delivery).
AIAI-31-007 | TODO | Instrument metrics (`advisory_ai_latency`, `guardrail_blocks`, `validation_failures`, `citation_coverage`), logs, and traces; publish dashboards/alerts. Dependencies: AIAI-31-004..006. | Advisory AI Guild, Observability Guild (src/AdvisoryAI/StellaOps.AdvisoryAI/TASKS.md)
AIAI-31-007 | DONE (2025-11-06) | Instrument metrics (`advisory_ai_latency`, `guardrail_blocks`, `validation_failures`, `citation_coverage`), logs, and traces; publish dashboards/alerts. Dependencies: AIAI-31-004..006. | Advisory AI Guild, Observability Guild (src/AdvisoryAI/StellaOps.AdvisoryAI/TASKS.md)
> 2025-11-06: AIAI-31-007 completed Advisory AI WebService/Worker emit latency histograms, guardrail/validation counters, citation coverage ratios, and OTEL spans; Grafana dashboard + burn-rate alerts refreshed.
AIAI-31-008 | TODO | Package inference on-prem container, remote inference toggle, Helm/Compose manifests, scaling guidance, offline kit instructions. Dependencies: AIAI-31-006..007. | Advisory AI Guild, DevOps Guild (src/AdvisoryAI/StellaOps.AdvisoryAI/TASKS.md)
AIAI-31-010 | DONE (2025-11-02) | Implement Concelier advisory raw document provider mapping CSAF/OSV payloads into structured chunks for retrieval. Dependencies: CONCELIER-VULN-29-001, EXCITITOR-VULN-29-001. | Advisory AI Guild (src/AdvisoryAI/StellaOps.AdvisoryAI/TASKS.md)
AIAI-31-011 | DONE (2025-11-02) | Implement Excititor VEX document provider to surface structured VEX statements for retrieval. Dependencies: EXCITITOR-LNM-21-201, EXCITITOR-CORE-AOC-19-002. | Advisory AI Guild (src/AdvisoryAI/StellaOps.AdvisoryAI/TASKS.md)
@@ -78,7 +79,7 @@ Depends on: Sprint 100.A - Attestor
Summary: Ingestion & Evidence focus on Concelier (phase I).
Task ID | State | Task description | Owners (Source)
--- | --- | --- | ---
CONCELIER-AIAI-31-001 `Paragraph anchors` | TODO | Expose advisory chunk API returning paragraph anchors, section metadata, and token-safe text for Advisory AI retrieval. | Concelier WebService Guild (src/Concelier/StellaOps.Concelier.WebService/TASKS.md)
CONCELIER-AIAI-31-001 `Paragraph anchors` | DONE | Expose advisory chunk API returning paragraph anchors, section metadata, and token-safe text for Advisory AI retrieval. | Concelier WebService Guild (src/Concelier/StellaOps.Concelier.WebService/TASKS.md)
CONCELIER-AIAI-31-002 `Structured fields` | TODO | Ensure observation APIs expose upstream workaround/fix/CVSS fields with provenance; add caching for summary queries. Dependencies: CONCELIER-AIAI-31-001. | Concelier WebService Guild (src/Concelier/StellaOps.Concelier.WebService/TASKS.md)
CONCELIER-AIAI-31-003 `Advisory AI telemetry` | TODO | Emit metrics/logs for chunk requests, cache hits, and guardrail blocks triggered by advisory payloads. Dependencies: CONCELIER-AIAI-31-001. | Concelier WebService Guild, Observability Guild (src/Concelier/StellaOps.Concelier.WebService/TASKS.md)
CONCELIER-AIRGAP-56-001 `Mirror ingestion adapters` | TODO | Add mirror source adapters reading advisories from imported bundles, preserving source metadata and bundle IDs. Ensure ingestion remains append-only. Dependencies: AIRGAP-IMP-57-002, MIRROR-CRT-56-001. | Concelier Core Guild (src/Concelier/__Libraries/StellaOps.Concelier.Core/TASKS.md)
@@ -197,9 +198,9 @@ FEEDCONN-CISCO-02-009 SemVer range provenance | BE-Conn-Cisco | **TODO (due 2025
FEEDCONN-ICSCISA-02-012 Version range provenance | BE-Conn-ICS-CISA | **DONE (2025-11-03)** Promote existing firmware/semver data into `advisory_observations.affected.versions[]` entries with deterministic comparison keys and provenance identifiers (`ics-cisa:{advisoryId}:{product}`). Add regression coverage for mixed firmware strings and raise a Models ticket only when observation schema needs a new comparison helper.<br>2025-10-29: Follow `docs/dev/normalized-rule-recipes.md` §2 to build observation version entries and log failures without invoking the retired merge helpers.<br>2025-11-03: Completed connector now normalizes semver ranges with provenance notes, RSS fallback content clears the AOC guard, and end-to-end Fetch/Parse/Map integration tests pass. | CONCELIER-LNM-21-001 (src/Concelier/__Libraries/StellaOps.Concelier.Connector.Ics.Cisa/TASKS.md)
FEEDCONN-KISA-02-008 Firmware range provenance | BE-Conn-KISA, Models | **DONE (2025-11-04)** Define comparison helpers for Hangul-labelled firmware ranges (`XFU 1.0.1.0084 ~ 2.0.1.0034`) and map them into `advisory_observations.affected.versions[]` with provenance tags. Coordinate with Models only if a new comparison scheme is required, then update localisation notes and fixtures for the Link-Not-Merge schema.<br>2025-11-03: Analysis in progress auditing existing mapper output/fixtures ahead of implementing firmware range normalization and provenance wiring.<br>2025-11-03: SemVer normalization helper wired through `KisaMapper` with provenance slugs + vendor extensions; integration tests updated and green, follow-up capture for additional Hangul exclusivity markers queued before completion.<br>2025-11-03: Extended connector tests to cover single-ended (`이상`, `초과`, `이하`, `미만`) and non-numeric phrases, verifying normalized rule types (`gt`, `gte`, `lt`, `lte`) and fallback behaviour; broader corpus review remains before transitioning to DONE.<br>2025-11-03: Captured the top 10 `detailDos.do?IDX=` pages into `seed-data/kisa/html/` via `scripts/kisa_capture_html.py`; JSON endpoint (`rssDetailData.do?IDX=…`) now returns error pages, so connector updates must parse the embedded HTML or secure authenticated API access before closing.<br>2025-11-04: Fetch + parse pipeline now consumes the HTML detail pages end to end (metadata persisted, DOM parser extracts vendor/product ranges); fixtures/tests operate on the HTML snapshots to guard normalized SemVer + vendor extension expectations and severity extraction. | CONCELIER-LNM-21-001 (src/Concelier/__Libraries/StellaOps.Concelier.Connector.Kisa/TASKS.md)
FEEDCONN-SHARED-STATE-003 Source state seeding helper | Tools Guild, BE-Conn-MSRC | **DONE (2025-11-04)** Delivered `SourceStateSeeder` CLI + processor APIs, Mongo fixtures, and MSRC runbook updates. Seeds raw docs + cursor state deterministically; tests cover happy/path/idempotent flows (`dotnet test src/Concelier/__Tests/StellaOps.Concelier.Connector.Common.Tests/...` note: requires `libcrypto.so.1.1` when running Mongo2Go locally). | Tools (src/Concelier/__Libraries/StellaOps.Concelier.Connector.Common/TASKS.md)
FEEDMERGE-COORD-02-901 Connector deadline check-ins | BE-Merge | **TODO (due 2025-10-21)** Confirm Cccs/Cisco version-provenance updates land, capture `LinksetVersionCoverage` dashboard snapshots (expect zero missing-range warnings), and update coordination docs with the results.<br>2025-10-29: Observation metrics now surface `version_entries_total`/`missing_version_entries_total`; include screenshots for both when closing this task. | FEEDMERGE-COORD-02-900 (src/Concelier/__Libraries/StellaOps.Concelier.Merge/TASKS.md)
FEEDMERGE-COORD-02-902 ICS-CISA version comparison support | BE-Merge, Models | **TODO (due 2025-10-23)** Review ICS-CISA sample advisories, validate reuse of existing comparison helpers, and pre-stage Models ticket template only if a new firmware comparator is required. Document the outcome and observation coverage logs in coordination docs + tracker files.<br>2025-10-29: `docs/dev/normalized-rule-recipes.md` (§2§3) now covers observation entries; attach decision summary + log sample when handing off to Models. Dependencies: FEEDMERGE-COORD-02-901. | FEEDMERGE-COORD-02-900 (src/Concelier/__Libraries/StellaOps.Concelier.Merge/TASKS.md)
FEEDMERGE-COORD-02-903 KISA firmware scheme review | BE-Merge, Models | **TODO (due 2025-10-24)** Pair with KISA team on proposed firmware comparison helper (`kisa.build` or variant), ensure observation mapper alignment, and open Models ticket only if a new comparator is required. Log the final helper signature and observation coverage metrics in coordination docs + tracker files. Dependencies: FEEDMERGE-COORD-02-902. | FEEDMERGE-COORD-02-900 (src/Concelier/__Libraries/StellaOps.Concelier.Merge/TASKS.md)
FEEDMERGE-COORD-02-901 Connector deadline check-ins | DROPPED (2025-11-07) | Scope removed: FeedMerge coordination requires an AOC policy that does not exist yet. Re-open once governance/ownership is defined. | —
FEEDMERGE-COORD-02-902 ICS-CISA version comparison support | DROPPED (2025-11-07) | Blocked on FEEDMERGE policy/ownership; dropped alongside 02-901. | —
FEEDMERGE-COORD-02-903 KISA firmware scheme review | DROPPED (2025-11-07) | Blocked on FEEDMERGE policy/ownership; dropped alongside 02-901. |
Fixture validation sweep | QA | **DONE (2025-11-04)** Regenerated RHSA CSAF goldens via `scripts/update-redhat-fixtures.sh` (sets `UPDATE_GOLDENS=1`) and re-ran connector tests `dotnet test src/Concelier/__Tests/StellaOps.Concelier.Connector.Distro.RedHat.Tests/StellaOps.Concelier.Connector.Distro.RedHat.Tests.csproj --no-restore` to confirm snapshot parity. | None (src/Concelier/__Libraries/StellaOps.Concelier.Connector.Distro.RedHat/TASKS.md)
Link-Not-Merge version provenance coordination | BE-Merge | **DONE (2025-11-04)** Published connector status tracker + follow-up IDs in `docs/dev/normalized-rule-recipes.md`, enabled `Normalized version rules missing` diagnostics in Merge, and aligned dashboards on `LinksetVersionCoverage`. Remaining gaps (ACSC/CCCS/CERTBUND/Cisco/RU-BDU) documented as upstream data deficiencies awaiting feed updates. Dependencies: CONCELIER-LNM-21-203. | CONCELIER-LNM-21-001 (src/Concelier/__Libraries/StellaOps.Concelier.Merge/TASKS.md)
MERGE-LNM-21-001 | DONE (2025-11-03) | Draft `no-merge` migration playbook, documenting backfill strategy, feature flag rollout, and rollback steps for legacy merge pipeline deprecation.<br>2025-11-03: Authored `docs/migration/no-merge.md` covering rollout phases, backfill/validation checklists, and rollback guidance; shared artefact owners. | BE-Merge, Architecture Guild (src/Concelier/__Libraries/StellaOps.Concelier.Merge/TASKS.md)
@@ -210,8 +211,9 @@ Depends on: Sprint 110.B - Concelier.VI
Summary: Ingestion & Evidence focus on Concelier (phase VII).
Task ID | State | Task description | Owners (Source)
--- | --- | --- | ---
MERGE-LNM-21-002 | DOING (2025-11-06) | Refactor or retire `AdvisoryMergeService` and related pipelines, ensuring callers transition to observation/linkset APIs; add compile-time analyzer preventing merge service usage.<br>2025-11-03: Began dependency audit and call-site inventory ahead of deprecation plan; cataloging service registrations/tests referencing merge APIs.<br>2025-11-05 14:42Z: Drafted `concelier:features:noMergeEnabled` gating, merge job allowlist handling, and deprecation/telemetry changes prior to analyzer rollout.<br>2025-11-06 16:10Z: Landed analyzer project (`CONCELIER0002`), wired into Concelier WebService/tests, and updated docs to direct suppressions through explicit migration notes.<br>2025-11-07 03:25Z: Default-on toggle + job gating break existing Concelier WebService tests; guard/migration adjustments pending before closing the task. | BE-Merge (src/Concelier/__Libraries/StellaOps.Concelier.Merge/TASKS.md)
MERGE-LNM-21-002 | DOING (2025-11-07) | Refactor or retire `AdvisoryMergeService` and related pipelines, ensuring callers transition to observation/linkset APIs; add compile-time analyzer preventing merge service usage.<br>2025-11-03: Began dependency audit and call-site inventory ahead of deprecation plan; cataloging service registrations/tests referencing merge APIs.<br>2025-11-05 14:42Z: Drafted `concelier:features:noMergeEnabled` gating, merge job allowlist handling, and deprecation/telemetry changes prior to analyzer rollout.<br>2025-11-06 16:10Z: Landed analyzer project (`CONCELIER0002`), wired into Concelier WebService/tests, and updated docs to direct suppressions through explicit migration notes.<br>2025-11-07 03:25Z: Default-on toggle + job gating break existing Concelier WebService tests; guard/migration adjustments pending before closing the task.<br>2025-11-07 07:05Z: Added ingest-path diagnostics (hash logging + test log dumping) to trace why HTTP binding loses `upstream.contentHash` with `noMergeEnabled=true`; need to adapt seeding/tests once the binding issue is fixed. | BE-Merge (src/Concelier/__Libraries/StellaOps.Concelier.Merge/TASKS.md)
MERGE-LNM-21-003 Determinism/test updates | QA Guild, BE-Merge | Replace merge determinism suites with observation/linkset regression tests verifying no data mutation and conflicts remain visible. Dependencies: MERGE-LNM-21-002. | MERGE-LNM-21-002 (src/Concelier/__Libraries/StellaOps.Concelier.Merge/TASKS.md)
WEB-AOC-19-001 (dependency) | DONE (2025-11-07) | Shared guard primitives now enforce the top-level allowlist (`_id`, tenant, source, upstream, content, identifiers, linkset, supersedes, created/ingested timestamps, attributes) and emit the reusable `AocError` payload consumed by HTTP/CLI tooling. Extend `AocGuardOptions.AllowedTopLevelFields` when staging new schema fields to avoid false-positive `ERR_AOC_007` violations. | BE-Base Platform Guild (docs/aoc/guard-library.md, src/Web/StellaOps.Web/TASKS.md)
[Ingestion & Evidence] 110.C) Excititor.I