Files
git.stella-ops.org/docs/implplan/SPRINT_112_concelier_i.md
master 61f963fd52
Some checks failed
Docs CI / lint-and-preview (push) Has been cancelled
Implement ledger metrics for observability and add tests for Ruby packages endpoints
- Added `LedgerMetrics` class to record write latency and total events for ledger operations.
- Created comprehensive tests for Ruby packages endpoints, covering scenarios for missing inventory, successful retrieval, and identifier handling.
- Introduced `TestSurfaceSecretsScope` for managing environment variables during tests.
- Developed `ProvenanceMongoExtensions` for attaching DSSE provenance and trust information to event documents.
- Implemented `EventProvenanceWriter` and `EventWriter` classes for managing event provenance in MongoDB.
- Established MongoDB indexes for efficient querying of events based on provenance and trust.
- Added models and JSON parsing logic for DSSE provenance and trust information.
2025-11-13 09:29:09 +02:00

11 KiB
Raw Blame History

Sprint 112 · Concelier.I — Canonical Evidence & Provenance (Rebaseline 2025-11-13)

Phase 110.B keeps Concelier focused on ingestion fidelity and evidence APIs. All active work here assumes Advisory AI consumes canonical advisory documents (no merge transforms) and that every field we emit carries exact provenance anchors.

Canonical Model Commitments

  • Single source of truth: /advisories/{key}/chunks must render from the canonical Advisory aggregate (document id + latest observation set), never from derived cache copies.
  • Provenance anchors: Each structured field cites both the Mongo _id of the backing observation document and the JSON Pointer into that observation (observationPath). This mirrors how GHSAs GraphQL securityAdvisory.references and Cisco PSIRTs openVuln feeds expose source handles, so downstream tooling can reconcile fields deterministically.
  • Deterministic ordering: Sort structured entries by (fieldType, observationPath, sourceId) to keep cache keys and telemetry stable across nodes. We are keeping this policy “as-is” for now to avoid churn in Advisory AI prompts.
  • External parity: Continue mapping fields named in competitor docs (GitHub Security Advisory GraphQL, Red Hat CVE data API, Cisco PSIRT openVuln) so migrations remain predictable.

Workstream A — Advisory AI Structured Fields (AIAI-31)

Task ID State Exit criteria Owners
CONCELIER-AIAI-31-002 Structured fields DOING 1) Program.cs endpoint fully rewritten to resolve the canonical advisory (via IAdvisoryStore/IAliasStore) and issue structured field entries. 2) Cache key = tenant + AdvisoryFingerprint. 3) Responses contain {chunkId, fingerprint, entries[], provenance.documentId, provenance.observationPath} with deterministic ordering. 4) Tests updated (StatementProvenanceEndpointAttachesMetadata, new structured chunk fixture) and Mongo2Go coverage passes. Concelier WebService Guild (src/Concelier/StellaOps.Concelier.WebService)
CONCELIER-AIAI-31-003 Advisory AI telemetry DONE (2025-11-12) OTEL counters (advisory_ai_chunk_requests_total, advisory_ai_chunk_cache_hits_total, advisory_ai_guardrail_blocks_total) tagged with tenant/result/cache. Nothing further planned unless guardrail policy changes. Concelier WebService Guild · Observability Guild

Implementation checklist (kept inline until CONCELIER-AIAI-31-002 ships)

  1. Add ResolveAdvisoryAsync helper with alias fallback + tenant guard.
  2. Update AdvisoryChunkCacheKey to include AdvisoryFingerprint.
  3. Rewrite /advisories/{key}/chunks handler to call the structured builder and emit provenance anchors.
  4. Refresh telemetry tests to assert Response.Entries.Count.
  5. Extend docs (docs/provenance/inline-dsse.md + Advisory AI API reference) with the structured schema mirroring GHSA / Cisco references.

Workstream B — Mirror & Offline Provenance (AIRGAP-56/57/58)

Task ID State Exit criteria / notes Owners
CONCELIER-AIRGAP-56-001 Mirror ingestion adapters TODO Implement read paths for Offline Kit bundles, persist bundleId, merkleRoot, and maintain append-only ledger comparisons. Concelier Core Guild (src/Concelier/__Libraries/StellaOps.Concelier.Core)
CONCELIER-AIRGAP-56-002 Bundle catalog linking TODO Every observation/linkset stores {bundleId, merkleRoot, observationPath} so exported evidence can cite provenance exactly once; depends on 56-001. Concelier Core Guild · AirGap Importer Guild
CONCELIER-AIRGAP-57-001 Sealed-mode source restrictions TODO Feature flag + policy that rejects non-mirror connectors with actionable diagnostics; depends on 56-001. Concelier Core Guild · AirGap Policy Guild
CONCELIER-AIRGAP-57-002 Staleness annotations TODO Compute fetchedAt/publishedAt/clockSource deltas per bundle and expose via observation APIs without mutating evidence; depends on 56-002. Concelier Core Guild · AirGap Time Guild
CONCELIER-AIRGAP-58-001 Portable advisory evidence TODO Package advisory observations/linksets + provenance notes (document id + observationPath) into timeline-bound portable bundles with verifier instructions; depends on 57-002. Concelier Core Guild · Evidence Locker Guild

Workstream C — Transparency & Attestor (ATTEST-73)

Task ID State Exit criteria / notes Owners
CONCELIER-ATTEST-73-001 ScanResults attestation inputs TODO Emit {observationDigest, linksetDigest, documentId} pairs required by Attestor so DSSE bundles include the same provenance anchors Advisory AI emits. Concelier Core Guild · Attestor Service Guild
CONCELIER-ATTEST-73-002 Transparency metadata TODO Read APIs expose bundleId, Rekor references, and observation paths for external transparency explorers; depends on 73-001. Concelier Core Guild

Workstream D — Console & Search Surfaces (CONSOLE-23)

Task ID State Exit criteria / notes Owners
CONCELIER-CONSOLE-23-001 Advisory aggregation views TODO /console/advisories returns grouped linksets with per-source severity/status chips plus {documentId, observationPath} provenance references (matching GHSA + Red Hat CVE browser expectations); depends on CONCELIER-LNM-21-201/202. Concelier WebService Guild · BE-Base Platform Guild
CONCELIER-CONSOLE-23-002 Dashboard deltas API TODO Deterministic “new/modified/conflicting” sets referencing linkset IDs and field paths rather than computed verdicts; depends on 23-001. Concelier WebService Guild
CONCELIER-CONSOLE-23-003 Search fan-out helpers TODO CVE/GHSA/PURL lookups return observation excerpts, provenance anchors, and cache hints so tenants can preview evidence safely; reuse structured field taxonomy from Workstream A. Concelier WebService Guild

Workstream E — Tenant Scope & AOC Guardrails

Task ID State Exit criteria / notes Owners
CONCELIER-CORE-AOC-19-013 Authority tenant scope smoke coverage TODO Expand smoke/e2e suites so Authority tokens + tenant headers are mandatory for ingest/read paths (including the new provenance endpoint). Must assert no merge-side effects and that provenance anchors always round-trip. Concelier Core Guild (src/Concelier/__Libraries/StellaOps.Concelier.Core)

Recent Updates

  • 2025-11-12: CONCELIER-AIAI-31-003 shipped OTEL counters for Advisory AI chunk traffic; dashboards now display cache hit ratios and guardrail blocks per tenant.
  • 2025-11-13: Sprint rebaseline complete; structured field scope locked to canonical model + provenance anchors, matching competitor schemas for short-term parity.

Current status (2025-11-13)

Workstream State Notes
A Advisory AI structured fields 🔶 DOING CONCELIER-AIAI-31-002 code work in progress; schema locked, telemetry landed, release blocked on Link-Not-Merge + CARTO schemas.
B Mirror & offline provenance 🔴 BLOCKED No work can start until MIRROR-CRT-56-001 staffing and Offline Kit bundle contracts finalize.
C Transparency & Attestor 🔴 BLOCKED Waiting on Workstream A output plus attestation backlog sequencing (Sprint 110/Excititor).
D Console & search surfaces 🔶 WATCHING Scoped but dependencies on Link-Not-Merge + Console backlog; preparing schema docs in parallel.
E Tenant scope & AOC guardrails 🔶 WATCHING Requires Authority smoke coverage; no active engineering yet but tests ready to clone once structured endpoint stabilizes.

Blockers & dependencies

Dependency Impacted work Owner(s) Status
Link-Not-Merge schema (CONCELIER-LNM-21-*, CARTO-GRAPH-21-002) Workstream A release, Workstream D APIs Concelier Core · Cartographer Guild · Platform Events Guild Review scheduled 2025-11-14; approval required before shipping structured fields/console APIs.
MIRROR-CRT-56-001 staffing Workstream B (AIRGAP-56/57/58) Mirror Creator Guild · Exporter Guild · AirGap Time Guild Owner not assigned (per Sprint 110); kickoff on 2025-11-15 must resolve.
Evidence Locker attestation contract Workstream C (ATTEST-73) Evidence Locker Guild · Concelier Core Needs alignment with Excititor attestation plan on 2025-11-15.
Authority scope smoke coverage (CONCELIER-CORE-AOC-19-013) Workstream E Concelier Core · Authority Guild Waiting on structured endpoint readiness + AUTH-SIG-26-001 validation.

Next actions (target: 2025-11-16)

Workstream Owner(s) Action Status
A Concelier WebService Guild Finish ResolveAdvisoryAsync, cache key update, and structured response builder; prep PR for review once schema approved. In progress
A Docs Guild Draft structured field schema appendix referencing provenance anchors for Advisory AI docs. Pending
B Concelier Core + Mirror leadership Join 2025-11-15 kickoff, capture MIRROR-CRT-56-001 owner, and align bundle metadata contract. Pending
C Concelier Core + Evidence Locker Produce attestation payload outline so ATTEST-73-001 can start immediately after sequencing meeting. Pending
D Concelier WebService Guild Prepare /console/advisories API spec (field list, provenance references) so implementation can begin once Link-Not-Merge clears. Drafting
E Concelier Core Clone Authority smoke suites to cover new structured endpoint once Workstream A enters review. Pending

Standup prompts

  1. Has Link-Not-Merge schema review resolved all blocking comments? If not, what fields remain at risk?
  2. Who will own MIRROR-CRT-56-001 after the 2025-11-15 kickoff, and do we have staffing for follow-on AIRGAP tasks?
  3. Did Evidence Locker accept the attestation contract draft, enabling ATTEST-73-001 to move forward?
  4. Are Authority/AOC smoke tests ready to clone once structured fields release, or do we need additional scope from AUTH-SIG-26-001?

Risks (snapshot 2025-11-13)

Risk Impact Mitigation / owner
Link-Not-Merge schema slips past 2025-11-14 Structured fields + console APIs stay unreleased, blocking Advisory AI and Console surfaces. Push for schema sign-off during 2025-11-14 review; prep fallback adapter if necessary.
Mirror staffing unresolved AirGap provenance work (AIRGAP-56/57/58) cannot start, delaying Offline Kit parity. Escalate at 2025-11-15 kickoff; consider borrowing engineers from Evidence Locker or Export guilds.
Evidence Locker contract delay ATTEST-73 work cannot begin, leaving Advisory AI without attested provenance. Align with Excititor/Evidence Locker owners during 2025-11-15 sequencing session; draft interim spec.
Authority smoke coverage gap AOC guardrails may regress when structured endpoint ships. Schedule paired testing with Authority guild once Workstream A PR is ready.