stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 2 Releases Wiki Activity

feat(metrics): Add new histograms for chunk latency, results, and sources in AdvisoryAiMetrics
Some checks failed
Docs CI / lint-and-preview (push) Has been cancelled
Details

Browse Source 
feat(telemetry): Record chunk latency, result count, and source count in AdvisoryAiTelemetry

fix(endpoint): Include telemetry source count in advisory chunks endpoint response

test(metrics): Enhance WebServiceEndpointsTests to validate new metrics for chunk latency, results, and sources

refactor(tests): Update test utilities for Deno language analyzer tests

chore(tests): Add performance tests for AdvisoryGuardrail with scenarios and blocked phrases

docs: Archive Sprint 137 design document for scanner and surface enhancements
This commit is contained in:
master
2025-11-10 22:26:43 +02:00
parent 56c687253f
commit b059bc7675
22 changed files with 427 additions and 37 deletions
Download Patch File Download Diff File Expand all files Collapse all files

20
docs/implplan/SPRINT_111_advisoryai.md
View File

@@ -5,8 +5,18 @@ Active items only. Completed/historic work now resides in docs/implplan/archived
[Ingestion & Evidence] 110.A) AdvisoryAI
Depends on: Sprint 100.A - Attestor
Summary: Ingestion & Evidence focus on AdvisoryAI.
Task ID | State | Task description | Owners (Source)
--- | --- | --- | ---
DOCS-AIAI-31-006 | BLOCKED (2025-11-03) | Update `/docs/policy/assistant-parameters.md` covering temperature, token limits, ranking weights, TTLs. Dependencies: POLICY-ENGINE-31-001. | Docs Guild, Policy Guild (docs)
DOCS-AIAI-31-008 | BLOCKED (2025-11-03) | Publish `/docs/sbom/remediation-heuristics.md` (feasibility scoring, blast radius). Dependencies: SBOM-AIAI-31-001. | Docs Guild, SBOM Service Guild (docs)
DOCS-AIAI-31-009 | BLOCKED (2025-11-03) | Create `/docs/runbooks/assistant-ops.md` for warmup, cache priming, model outages, scaling. Dependencies: DEVOPS-AIAI-31-001. | Docs Guild, DevOps Guild (docs)
SBOM-AIAI-31-003 | TODO (2025-11-03) | Publish the Advisory AI hand-off kit for `/v1/sbom/context`, share base URL/API key + tenant header contract, and run a joint end-to-end retrieval smoke test with Advisory AI. Dependencies: SBOM-AIAI-31-001. | SBOM Service Guild, Advisory AI Guild (src/SbomService/StellaOps.SbomService)
AIAI-31-008 | TODO | Package inference on-prem container, remote inference toggle, Helm/Compose manifests, scaling guidance, offline kit instructions. Dependencies: AIAI-31-006..007. | Advisory AI Guild, DevOps Guild (src/AdvisoryAI/StellaOps.AdvisoryAI)
AIAI-31-009 | DOING (2025-11-09) | Develop unit/golden/property/perf tests, injection harness, and regression suite; ensure determinism with seeded caches. Dependencies: AIAI-31-001..006. | Advisory AI Guild, QA Guild (src/AdvisoryAI/StellaOps.AdvisoryAI) |
> 2025-11-03: WebService/Worker scaffolds created with in-memory cache/queue, minimal APIs (`/api/v1/advisory/plan`, `/api/v1/advisory/queue`), metrics counters, and plan cache instrumentation; worker processes queue using orchestrator.
> 2025-11-04: SBOM base address now flows via `SbomContextClientOptions.BaseAddress`, worker emits queue/plan metrics, and orchestrator cache keys expanded to cover SBOM hash inputs.
DOCS-AIAI-31-004 | DOING (2025-11-07) | Create `/docs/advisory-ai/console.md` with screenshots, a11y notes, copy-as-ticket instructions. Dependencies: CONSOLE-VULN-29-001, CONSOLE-VEX-30-001, EXCITITOR-CONSOLE-23-001. | Docs Guild, Console Guild (docs)
@@ -14,10 +24,6 @@ DOCS-AIAI-31-004 | DOING (2025-11-07) | Create `/docs/advisory-ai/console.md` wi
> 2025-11-08: Console endpoints are staffed (CONSOLE-VULN-29-001 / CONSOLE-VEX-30-001 DOING); still waiting on EXCITITOR-CONSOLE-23-001 feeds before capturing screenshots/tests.
> 2025-11-09: Guardrail/inference sections and offline playbooks documented; screenshot placeholders remain open.
DOCS-AIAI-31-005 | BLOCKED (2025-11-03) | Publish `/docs/advisory-ai/cli.md` covering commands, exit codes, scripting patterns. Dependencies: CLI-VULN-29-001, CLI-VEX-30-001, AIAI-31-004C. | Docs Guild, DevEx/CLI Guild (docs)
DOCS-AIAI-31-006 | BLOCKED (2025-11-03) | Update `/docs/policy/assistant-parameters.md` covering temperature, token limits, ranking weights, TTLs. Dependencies: POLICY-ENGINE-31-001. | Docs Guild, Policy Guild (docs)
DOCS-AIAI-31-008 | BLOCKED (2025-11-03) | Publish `/docs/sbom/remediation-heuristics.md` (feasibility scoring, blast radius). Dependencies: SBOM-AIAI-31-001. | Docs Guild, SBOM Service Guild (docs)
DOCS-AIAI-31-009 | BLOCKED (2025-11-03) | Create `/docs/runbooks/assistant-ops.md` for warmup, cache priming, model outages, scaling. Dependencies: DEVOPS-AIAI-31-001. | Docs Guild, DevOps Guild (docs)
SBOM-AIAI-31-003 | TODO (2025-11-03) | Publish the Advisory AI hand-off kit for `/v1/sbom/context`, share base URL/API key + tenant header contract, and run a joint end-to-end retrieval smoke test with Advisory AI. Dependencies: SBOM-AIAI-31-001. | SBOM Service Guild, Advisory AI Guild (src/SbomService/StellaOps.SbomService)
> 2025-11-03: DOCS-AIAI-31-003 moved to DOING – drafting Advisory AI API reference (endpoints, rate limits, error model) for sprint 110.
> 2025-11-04: AIAI-31-005 DONE – guardrail pipeline redacts secrets, enforces citation/injection policies, emits block counters, and tests (`AdvisoryGuardrailPipelineTests`) cover redaction + citation validation.
> 2025-11-03: DOCS-AIAI-31-003 marked DONE – `docs/advisory-ai/api.md` published with scopes, request/response schemas, rate limits, and error catalogue (Docs Guild).
@@ -31,12 +37,8 @@ SBOM-AIAI-31-003 | TODO (2025-11-03) | Publish the Advisory AI hand-off kit for
> 2025-11-03: DOCS-AIAI-31-009 marked BLOCKED – DevOps runbook inputs (DEVOPS-AIAI-31-001) outstanding.
> 2025-11-03: Shipped `/api/v1/advisory/{task}` execution and `/api/v1/advisory/outputs/{cacheKey}` retrieval endpoints with guardrail integration, provenance hashes, and metrics (RBAC & rate limiting still pending Authority scope delivery).
> 2025-11-06: AIAI-31-007 completed – Advisory AI WebService/Worker emit latency histograms, guardrail/validation counters, citation coverage ratios, and OTEL spans; Grafana dashboard + burn-rate alerts refreshed.
AIAI-31-008 | TODO | Package inference on-prem container, remote inference toggle, Helm/Compose manifests, scaling guidance, offline kit instructions. Dependencies: AIAI-31-006..007. | Advisory AI Guild, DevOps Guild (src/AdvisoryAI/StellaOps.AdvisoryAI)
AIAI-31-009 | DOING (2025-11-09) | Develop unit/golden/property/perf tests, injection harness, and regression suite; ensure determinism with seeded caches. Dependencies: AIAI-31-001..006. | Advisory AI Guild, QA Guild (src/AdvisoryAI/StellaOps.AdvisoryAI)
> 2025-11-09: Guardrail harness converted to JSON fixtures + legacy payloads, property-style plan cache load tests added, and file-system cache/output suites cover seeded/offline scenarios.
> 2025-11-02: AIAI-31-004 kicked off orchestration pipeline design – establishing deterministic task sequence (summary/conflict/remediation) and cache key strategy.
> 2025-11-02: AIAI-31-004 orchestration prerequisites documented in docs/modules/advisory-ai/orchestration-pipeline.md (tasks 004A/004B/004C).
> 2025-11-02: AIAI-31-003 moved to DOING – beginning deterministic tooling (comparators, dependency analysis) while awaiting SBOM context client. Semantic & EVR comparators shipped; toolset interface published for orchestrator adoption.