git.stella-ops.org/docs/modules/advisory-ai/orchestration-pipeline.md

Advisory AI Orchestration Pipeline (Planning Notes)

Status: Draft – prerequisite design for AIAI-31-004 integration work.
Audience: Advisory AI guild, WebService/Worker guilds, CLI guild, Docs/QA support teams.

1. Goal

Wire the deterministic pipeline (Summary / Conflict / Remediation flows) into the Advisory AI service, workers, and CLI with deterministic caching, prompt preparation, and guardrail fallback. This document captures the pre-integration checklist and task breakdown so each guild understands their responsibilities before coding begins.

2. Prerequisites

Area	Requirement	Owner	Status
Toolset	Deterministic comparators, dependency analyzer (IDeterministicToolset, AdvisoryPipelineOrchestrator)	Advisory AI	✅ landed (AIAI-31-003)
SBOM context	Real SBOM context client delivering timelines + dependency paths	SBOM Service Guild	⏳ pending (AIAI-31-002)
Prompt artifacts	Liquid/Handlebars prompt templates for summary/conflict/remediation	Advisory AI Docs Guild	⏳ authoring needed
Cache strategy	Decision on DSSE or hash-only cache entries, TTLs, and eviction policy	Advisory AI + Platform	🔲 define
Auth scopes	Confirm service account scopes for new API endpoints/worker-to-service calls	Authority Guild	🔲 define

Blocking risk: SBOM client and prompt templates must exist (even stubbed) before the orchestrator can produce stable plans.

3. Integration plan (high-level)

1. Service layer (WebService / Worker)
   • Inject IAdvisoryPipelineOrchestrator via AddAdvisoryPipeline.
   • Define REST endpoint POST /v1/advisories/{key}/pipeline/{task} (task ∈ summary/conflict/remediation).
   • Worker consumes queue messages (advisory.pipeline.execute) -> fetches plan -> executes prompt -> persists output & provenance.
   • Add metrics: advisory_pipeline_requests_total, advisory_pipeline_plan_cache_hits_total, advisory_pipeline_latency_seconds.
2. CLI
   • New command stella advise run <task> with flags for artifact id, profile, policy version, --force-refresh.
   • Render JSON/Markdown outputs; handle caching hints (print cache key, status).
3. Caching / storage
   • Choose storage (Mongo collection vs existing DSSE output store).
   • Persist AdvisoryTaskPlan metadata + generated output keyed by cache key + policy version.
   • Expose TTL/force-refresh semantics.
4. Docs & QA
   • Publish API spec (docs/advisory-ai/api.md) + CLI docs.
   • Add golden outputs for deterministic runs; property tests for cache key stability.

4. Task Breakdown

AIAI-31-004A (Service orchestration wiring)

• Scope: WebService/Worker injection, REST/queue plumbing, metrics counters, basic cache stub.
• Dependencies: AddAdvisoryPipeline, SBOM client stub.
• Exit: API responds with plan metadata + queue message; worker logs execution attempt; metrics emitted.

AIAI-31-004B (Prompt assembly & cache persistence)

• Scope: Implement prompt assembler, connect to guardrails, persist cache entries w/ DSSE metadata.
• Dependencies: Prompt templates, cache storage decision, guardrail interface.
• Exit: Deterministic outputs stored; force-refresh honoured; tests cover prompt assembly + caching.

2025-11-03: Prompt assembler now emits deterministic JSON payloads, guardrail pipeline wiring is stubbed for upcoming security hardening, and outputs persist with DSSE-ready provenance metadata plus golden test coverage.

AIAI-31-004C (CLI integration & docs)

• Scope: CLI command + output renderer, docs updates, CLI tests (golden outputs).
• Dependencies: Service endpoints stable, caching semantics documented.
• Exit: CLI command produces deterministic output, docs updated, smoke tests recorded.

AIAI-31-006 (Service API surface)

• Scope: Expose REST endpoints for summary/conflict/remediation execution plus cached output retrieval (POST /api/v1/advisory/{task}, GET /api/v1/advisory/outputs/{cacheKey}). Include guardrail execution, provenance hashing, metrics, and stubs for RBAC/rate limits.
• Dependencies: Guardrail enforcement (AIAI-31-005), Authority scope wiring (advisory-ai:view / advisory-ai:operate), Offline kit docs.
• Exit: Endpoints return sanitized prompts with citations, guardrail metadata, DSSE hash, and plan cache indicators; OpenAPI description updated; rate-limit hooks ready for Authority integration.

2025-11-03: Initial REST surface shipped – direct execution runs through guardrail pipeline, outputs persist with DSSE-ready provenance, metrics advisory_outputs_stored/advisory_guardrail_blocks emit, and cache retrieval endpoint exposes stored artefacts (RBAC/header enforcement pending scope delivery).

Supporting tasks (other guilds)

• AUTH-AIAI-31-004 – Update scopes and DSSE policy (Authority guild).
• DOCS-AIAI-31-003 – Publish API documentation, CLI guide updates (Docs guild).
• QA-AIAI-31-004 – Golden/properties/perf suite for pipeline (QA guild).

5. Acceptance checklist (per task)

Item	Notes
Cache key stability	AdvisoryPipelineOrchestrator hash must remain stable under re-run of identical inputs.
Metrics & logging	Request id, cache key, task type, profile, latency; guardrail results logged without sensitive prompt data.
Offline readiness	All prompt templates bundled with Offline Kit; CLI works in air-gapped mode with cached data.
Policy awareness	Plans encode policy version used; outputs reference policy digest for audit.
Testing	Unit tests (plan generation, cache keys, DI), integration (service endpoint, worker, CLI), deterministic golden outputs.

6. Next steps

1. Finalize SBOM context client (AIAI-31-002) and prompt templates.
2. Create queue schema spec (docs/modules/advisory-ai/queue-contracts.md) if not already available.
3. Schedule cross-guild kickoff to agree on cache store & DSSE policy.

Last updated: 2025-11-02