Implement Advisory Canonicalization and Backfill Migration
Some checks failed
Docs CI / lint-and-preview (push) Has been cancelled
Some checks failed
Docs CI / lint-and-preview (push) Has been cancelled
- Added AdvisoryCanonicalizer for canonicalizing advisory identifiers. - Created EnsureAdvisoryCanonicalKeyBackfillMigration to populate advisory_key and links in advisory_raw documents. - Introduced FileSurfaceManifestStore for managing surface manifests with file system backing. - Developed ISurfaceManifestReader and ISurfaceManifestWriter interfaces for reading and writing manifests. - Implemented SurfaceManifestPathBuilder for constructing paths and URIs for surface manifests. - Added tests for FileSurfaceManifestStore to ensure correct functionality and deterministic behavior. - Updated documentation for new features and migration steps.
This commit is contained in:
master
@@ -78,6 +78,10 @@ Follow the sprint files below in order. Update task status in both `SPRINTS` and
|
||||
> 2025-11-02: SURFACE-VAL-01 moved to DOING (Surface Validation Guild) – aligning design document with implementation plan.
|
||||
> 2025-11-02: SURFACE-FS-01 moved to DOING (Surface FS Guild) – finalising cache layout and manifest spec.
|
||||
> 2025-11-02: SURFACE-FS-02 moved to DOING (Surface FS Guild) – building core abstractions and deterministic serializers.
|
||||
> 2025-11-07: SURFACE-FS-01 marked DONE – updated `surface-fs.md` with pointer layout, offline kit flow, and architecture cross-link.
|
||||
> 2025-11-07: SURFACE-FS-02 marked DONE – landed file-backed manifest store (`FileSurfaceManifestStore`), deterministic serialization, and unit coverage.
|
||||
> 2025-11-07: SCHED-SURFACE-02 added (Scheduler Worker Guild) – prefetch Surface manifests before scheduling reruns.
|
||||
> 2025-11-07: ZASTAVA-SURFACE-02 added (Zastava Observer Guild) – adopt Surface manifest reader for drift diagnostics.
|
||||
> 2025-11-02: SURFACE-SECRETS-01 moved to DOING (Surface Secrets Guild) – updating secrets design for provider matrix.
|
||||
> 2025-11-02: SURFACE-SECRETS-02 moved to DOING (Surface Secrets Guild) – implementing base providers + tests.
|
||||
> 2025-11-02: AUTH-POLICY-27-002 marked DONE (Authority Core & Security Guild) – interactive-only policy publish/promote scopes delivered with metadata, fresh-auth enforcement, and audit/docs updates.
|
||||
|
||||
@@ -158,8 +158,8 @@ CONCELIER-SIG-26-001 `Vulnerable symbol exposure` | TODO | Expose advisory metad
|
||||
CONCELIER-STORE-AOC-19-005 `Raw linkset backfill` | TODO (2025-11-04) | Plan and execute advisory_observations `rawLinkset` backfill (online + Offline Kit bundles), supply migration scripts + rehearse rollback. Follow the coordination plan in `docs/dev/raw-linkset-backfill-plan.md`. Dependencies: CONCELIER-CORE-AOC-19-004. | Concelier Storage Guild, DevOps Guild (src/Concelier/__Libraries/StellaOps.Concelier.Storage.Mongo/TASKS.md)
|
||||
CONCELIER-TEN-48-001 `Tenant-aware linking` | TODO | Ensure advisory normalization/linking runs per tenant with RLS enforcing isolation; emit capability endpoint reporting `merge=false`; update events with tenant context. Dependencies: AUTH-TEN-47-001. | Concelier Core Guild (src/Concelier/__Libraries/StellaOps.Concelier.Core/TASKS.md)
|
||||
CONCELIER-VEXLENS-30-001 `Advisory rationale bridges` | TODO | Guarantee advisory key consistency and cross-links for consensus rationale; Label: VEX-Lens. Dependencies: CONCELIER-VULN-29-001, VEXLENS-30-005. | Concelier WebService Guild, VEX Lens Guild (src/Concelier/StellaOps.Concelier.WebService/TASKS.md)
|
||||
CONCELIER-VULN-29-001 `Advisory key canonicalization` | TODO | Canonicalize (lossless) advisory identifiers (CVE/GHSA/vendor) into `advisory_key`, persist `links[]`, expose raw payload snapshots for Explorer evidence tabs; AOC-compliant: no merge, no derived fields, no suppression. Include migration/backfill scripts. Dependencies: CONCELIER-LNM-21-001. | Concelier WebService Guild, Data Integrity Guild (src/Concelier/StellaOps.Concelier.WebService/TASKS.md)
|
||||
CONCELIER-VULN-29-002 `Evidence retrieval API` | TODO | Provide `/vuln/evidence/advisories/{advisory_key}` returning raw advisory docs with provenance, filtering by tenant and source. Dependencies: CONCELIER-VULN-29-001, VULN-API-29-003. | Concelier WebService Guild (src/Concelier/StellaOps.Concelier.WebService/TASKS.md)
|
||||
CONCELIER-VULN-29-001 `Advisory key canonicalization` | DONE (2025-11-07) | Canonicalize (lossless) advisory identifiers (CVE/GHSA/vendor) into `advisory_key`, persist `links[]`, expose raw payload snapshots for Explorer evidence tabs; AOC-compliant: no merge, no derived fields, no suppression. Include migration/backfill scripts. Dependencies: CONCELIER-LNM-21-001. | Concelier WebService Guild, Data Integrity Guild (src/Concelier/StellaOps.Concelier.WebService/TASKS.md)
|
||||
CONCELIER-VULN-29-002 `Evidence retrieval API` | DONE (2025-11-07) | Provide `/vuln/evidence/advisories/{advisory_key}` returning raw advisory docs with provenance, filtering by tenant and source. Dependencies: CONCELIER-VULN-29-001, VULN-API-29-003. | Concelier WebService Guild (src/Concelier/StellaOps.Concelier.WebService/TASKS.md)
|
||||
|
||||
|
||||
[Ingestion & Evidence] 110.B) Concelier.V
|
||||
@@ -211,8 +211,8 @@ Depends on: Sprint 110.B - Concelier.VI
|
||||
Summary: Ingestion & Evidence focus on Concelier (phase VII).
|
||||
Task ID | State | Task description | Owners (Source)
|
||||
--- | --- | --- | ---
|
||||
MERGE-LNM-21-002 | DOING (2025-11-07) | Refactor or retire `AdvisoryMergeService` and related pipelines, ensuring callers transition to observation/linkset APIs; add compile-time analyzer preventing merge service usage.<br>2025-11-03: Began dependency audit and call-site inventory ahead of deprecation plan; cataloging service registrations/tests referencing merge APIs.<br>2025-11-05 14:42Z: Drafted `concelier:features:noMergeEnabled` gating, merge job allowlist handling, and deprecation/telemetry changes prior to analyzer rollout.<br>2025-11-06 16:10Z: Landed analyzer project (`CONCELIER0002`), wired into Concelier WebService/tests, and updated docs to direct suppressions through explicit migration notes.<br>2025-11-07 03:25Z: Default-on toggle + job gating break existing Concelier WebService tests; guard/migration adjustments pending before closing the task.<br>2025-11-07 07:05Z: Added ingest-path diagnostics (hash logging + test log dumping) to trace why HTTP binding loses `upstream.contentHash` with `noMergeEnabled=true`; need to adapt seeding/tests once the binding issue is fixed. | BE-Merge (src/Concelier/__Libraries/StellaOps.Concelier.Merge/TASKS.md)
|
||||
MERGE-LNM-21-003 Determinism/test updates | QA Guild, BE-Merge | Replace merge determinism suites with observation/linkset regression tests verifying no data mutation and conflicts remain visible. Dependencies: MERGE-LNM-21-002. | MERGE-LNM-21-002 (src/Concelier/__Libraries/StellaOps.Concelier.Merge/TASKS.md)
|
||||
MERGE-LNM-21-002 | DONE (2025-11-07) | Refactor or retire `AdvisoryMergeService` and related pipelines, ensuring callers transition to observation/linkset APIs; add compile-time analyzer preventing merge service usage.<br>2025-11-03: Began dependency audit and call-site inventory ahead of deprecation plan; cataloging service registrations/tests referencing merge APIs.<br>2025-11-05 14:42Z: Drafted `concelier:features:noMergeEnabled` gating, merge job allowlist handling, and deprecation/telemetry changes prior to analyzer rollout.<br>2025-11-06 16:10Z: Landed analyzer project (`CONCELIER0002`), wired into Concelier WebService/tests, and updated docs to direct suppressions through explicit migration notes.<br>2025-11-07 03:25Z: Default-on toggle + job gating surfaced ingestion test brittleness; guard/migration diagnostics capture requests missing `upstream.contentHash`.<br>2025-11-07 19:45Z: Set `ConcelierOptions.Features.NoMergeEnabled` default to `true`, added regression coverage (`Features_NoMergeEnabled_DefaultsToTrue`), and rechecked ingest helpers to carry canonical links. Remote .NET 10 CLI run remains queued for validation. | BE-Merge (src/Concelier/__Libraries/StellaOps.Concelier.Merge/TASKS.md)
|
||||
MERGE-LNM-21-003 Determinism/test updates | DOING (2025-11-07) | QA Guild, BE-Merge | Replace merge determinism suites with observation/linkset regression tests verifying no data mutation and conflicts remain visible. Dependencies: MERGE-LNM-21-002.<br>2025-11-07: Drafting test migration plan (`docs/dev/lnm-determinism-tests.md`) to map legacy merge fixtures onto observation/linkset pipelines; identifying coverage gaps (conflict surfacing, raw vs canonical parity, hash stability).<br>2025-11-07 20:05Z: Landed `AdvisoryObservationFactoryTests.Create_IsDeterministicAcrossRuns` to cover canonical JSON stability and pruned the old merge determinism integration test. | MERGE-LNM-21-002 (src/Concelier/__Libraries/StellaOps.Concelier.Merge/TASKS.md)
|
||||
WEB-AOC-19-001 (dependency) | DONE (2025-11-07) | Shared guard primitives now enforce the top-level allowlist (`_id`, tenant, source, upstream, content, identifiers, linkset, supersedes, created/ingested timestamps, attributes) and emit the reusable `AocError` payload consumed by HTTP/CLI tooling. Extend `AocGuardOptions.AllowedTopLevelFields` when staging new schema fields to avoid false-positive `ERR_AOC_007` violations. | BE-Base Platform Guild (docs/aoc/guard-library.md, src/Web/StellaOps.Web/TASKS.md)
|
||||
|
||||
|
||||
|
||||
@@ -156,6 +156,8 @@ SCANNER-ENG-0027 | TODO | Deliver Windows policy/offline integration per `design
|
||||
SCANNER-SURFACE-01 | DONE (2025-11-06) | Persist Surface.FS manifests after analyzer stages, including layer CAS metadata and EntryTrace fragments.<br>2025-11-02: Worker pipeline emitting draft Surface.FS manifests for sample scans; determinism checks running.<br>2025-11-06: Continuing with manifest writer abstraction + telemetry wiring for Surface.FS persistence.<br>2025-11-06 18:45Z: Resumed work; targeting manifest writer abstraction, CAS persistence hooks, and telemetry/test coverage updates.<br>2025-11-06 20:20Z: Published Surface worker Grafana dashboard + updated design doc; WebService pointer integration test now covers manifest/payload artefacts. | Scanner Worker Guild (src/Scanner/StellaOps.Scanner.Worker/TASKS.md)
|
||||
SCANNER-SURFACE-02 | DONE (2025-11-05) | Publish Surface.FS pointers (CAS URIs, manifests) via scan/report APIs and update attestation metadata. Dependencies: SCANNER-SURFACE-01.<br>2025-11-05: Surface pointer projection wired through WebService endpoints, orchestrator samples & DSSE fixtures refreshed with `surface` manifest block, and regression suite (platform events, report sample, ready check) updated. | Scanner WebService Guild (src/Scanner/StellaOps.Scanner.WebService/TASKS.md)
|
||||
SCANNER-SURFACE-03 | DONE (2025-11-07) | Push layer manifests and entry fragments into Surface.FS during build-time SBOM generation. Dependencies: SCANNER-SURFACE-02.<br>2025-11-06: Starting BuildX manifest upload implementation with Surface.FS client abstraction and integration tests.<br>2025-11-07 15:30Z: Resumed BuildX plugin Surface wiring; analyzing Surface.FS models, CAS flow, and upcoming tests before coding.<br>2025-11-07 22:10Z: Added Surface manifest writer + CLI flags to the BuildX plug-in, persisted artefacts into CAS, regenerated docs/fixtures, and shipped new tests covering the writer + descriptor flow. | BuildX Plugin Guild (src/Scanner/StellaOps.Scanner.Sbomer.BuildXPlugin/TASKS.md)
|
||||
SCHED-SURFACE-02 | TODO | Integrate Scheduler worker prefetch using Surface manifest reader and persist manifest pointers with rerun plans. Dependencies: SURFACE-FS-02, SCHED-SURFACE-01. Reference `docs/modules/scanner/design/surface-fs-consumers.md` §3 for implementation checklist. | Scheduler Worker Guild (src/Scheduler/__Libraries/StellaOps.Scheduler.Worker/TASKS.md)
|
||||
ZASTAVA-SURFACE-02 | TODO | Use Surface manifest reader helpers to resolve `cas://` pointers and enrich drift diagnostics with manifest provenance. Dependencies: SURFACE-FS-02, ZASTAVA-SURFACE-01. Reference `docs/modules/scanner/design/surface-fs-consumers.md` §4 for integration steps. | Zastava Observer Guild (src/Zastava/StellaOps.Zastava.Observer/TASKS.md)
|
||||
|
||||
[Scanner & Surface] 130.A) Scanner.VIII
|
||||
Depends on: Sprint 130.A - Scanner.VII
|
||||
|
||||
Reference in New Issue
Block a user