Refactor code structure for improved readability and maintainability; removed redundant code blocks and optimized function calls.

2025-11-20 07:50:52 +02:00
parent 616ec73133
commit 10212d67c0
473 changed files with 316758 additions and 388 deletions
--- a/docs/advisory-ai/console.md
+++ b/docs/advisory-ai/console.md
@@ -82,7 +82,7 @@ This guide documents the forthcoming Advisory AI console experience so that cons
 1. **Volume readiness** – confirm the RWX volume (`/var/lib/advisory-ai/{queue,plans,outputs}`) is mounted; the console should poll `/api/v1/advisory-ai/health` and surface “Queue not available” if the worker is offline.
 2. **Cached responses** – when running air-gapped, highlight that only cached plans/responses are available by showing the `planFromCache` badge plus the `generatedAtUtc` timestamp.
 3. **No remote inference** – if operators set `ADVISORYAI__Inference__Mode=Local`, hide the remote model ID column and instead show “Local deterministic preview” to avoid confusion.
-4. **Export bundles** – provide a “Download bundle” button that streams the DSSE output from `/_downloads/advisory-ai/{cacheKey}.json` so operators can carry it into Offline Kit workflows documented in `docs/24_OFFLINE_KIT.md`.
+4. **Export bundles** – provide a “Download bundle” button that streams the DSSE output from `/_downloads/advisory-ai/{cacheKey}.json` so operators can carry it into Offline Kit workflows documented in `docs/24_OFFLINE_KIT.md`. While staging endpoints are pending, reuse the Evidence Bundle v1 sample at `docs/samples/evidence-bundle/evidence-bundle-v1.tar.gz` (hash in `evidence-bundle-v1.tar.gz.sha256`) to validate wiring and screenshots.

 ## 6. Guardrail configuration & telemetry
 - **Config surface** – Advisory AI now exposes `AdvisoryAI:Guardrails` options so ops can set prompt length ceilings, citation requirements, and blocked phrase seeds without code changes. Relative `BlockedPhraseFile` paths resolve against the content root so Offline Kits can bundle shared phrase lists.
@@ -106,12 +106,23 @@ This guide documents the forthcoming Advisory AI console experience so that cons
 - **Console wiring** – the guardrail ribbon pulls `guardrail.blocked`, `guardrail.violations`, and `guardrail.metadata.blocked_phrase_count` while the observability cards track `advisory_ai_chunk_requests_total`, `advisory_ai_chunk_cache_hits_total`, and `advisory_ai_guardrail_blocks_total` (now emitted even on cache hits). Use these meters to explain throttling or bad actors before granting additional guardrail budgets, and keep `docs/api/console/samples/advisory-ai-guardrail-banner.json` nearby so QA can validate localized payloads without hitting production data.

 ## 5. Open items before publication
- [ ] Replace placeholder API responses with captures from the first merged build of CONSOLE-VULN-29-001 / CONSOLE-VEX-30-001.
+- [ ] Replace placeholder API responses with captures from the first merged build of CONSOLE-VULN-29-001 / CONSOLE-VEX-30-001 (blocked on SBOM-AIAI-31-003).
 - [ ] Capture at least two screenshots (list view + evidence drawer) using the fixture-backed workspace; commit both `*-payload.json` and `*-screenshot.png` with deterministic filenames.
 - [ ] Verify copy-as-ticket instructions with Support to ensure the payload fields align with existing SOC runbooks.
 - [ ] Add latency tooltip + remote/local badge screenshots after Grafana wiring is stable.
- [ ] Attach SBOM/VEX bundle example (sealed DSSE) to the doc and link it from Section 2.3 for auditors.
+- [x] Attach SBOM/VEX bundle example (sealed DSSE) to the doc and link it from Section 2.3 for auditors (using Evidence Bundle v1 sample).
+
+### Publication readiness checklist (DOCS-AIAI-31-004)
+- Inputs available now: console fixtures (`docs/samples/console/console-vuln-29-001.json`, `console-vex-30-001.json`), evidence bundle sample (`docs/samples/evidence-bundle/evidence-bundle-v1.tar.gz`), guardrail ribbon contract.
+- Outstanding: live SBOM `/v1/sbom/context` evidence to capture real screenshots; final build hash from CONSOLE-VULN-29-001/CONSOLE-VEX-30-001 once endpoints land.
+- Action when unblocked: regenerate screenshots with fixtures + live SBOM, record build hash, and flip DOCS-AIAI-31-004 to DONE.

 > Tracking: DOCS-AIAI-31-004 (Docs Guild, Console Guild)

+### Guardrail console fixtures (unchecked-integration)
+
+- Vulnerability search sample: `docs/samples/console/console-vuln-29-001.json` (maps to CONSOLE-VULN-29-001).
+- VEX search sample: `docs/samples/console/console-vex-30-001.json` (maps to CONSOLE-VEX-30-001).
+- Use these until live endpoints are exposed; replace with real captures when staging is available.
+
 **Reference**: API contracts and sample payloads live in `docs/api/console/workspaces.md` (see `/console/vuln/*` and `/console/vex/*` sections) plus the JSON fixtures under `docs/api/console/samples/`.
--- a/docs/advisory-ai/evidence-payloads.md
+++ b/docs/advisory-ai/evidence-payloads.md
@@ -0,0 +1,104 @@
+# Advisory AI Evidence Payloads (LNM-Aligned)
+
+_Updated: 2025-11-18 · Owner: Advisory AI Docs Guild · Sprint: 0111 (AIAI-RAG-31-003)_
+
+This document defines how Advisory AI consumes Link-Not-Merge (LNM) observations and linksets for Retrieval-Augmented Generation (RAG). It aligns payloads with the frozen LNM v1 schema (`docs/modules/concelier/link-not-merge-schema.md`, 2025-11-17) and replaces prior draft payloads.
+
+## 1) Input envelope (per task)
+
+```json
+{
+  "advisoryKey": "csaf:redhat:RHSA-2025:1001",
+  "profile": "fips-local",
+  "policyVersion": "2025.10.1",
+  "lnm": {
+    "observationIds": ["6561e41b3e3f4a6e9d3b91c1", "6561e41b3e3f4a6e9d3b91c2"],
+    "linksetId": "6561e41b3e3f4a6e9d3b91d0",
+    "provenanceHash": "sha256:0f7c...9ad3"
+  },
+  "sbom": {
+    "artifactId": "registry.stella-ops.internal/runtime/api",
+    "purl": "pkg:oci/runtime-api@sha256:d2c3...",
+    "timelineClamp": 500,
+    "dependencyPathClamp": 200
+  }
+}
+```
+
+Rules:
+- `lnm.linksetId` and `lnm.observationIds` are **required**. Missing values → `409 advisory.contextUnavailable`.
+- `provenanceHash` must match the hash list embedded in the LNM linkset; Advisory AI refuses linksets whose hashes mismatch.
+- SBOM fields optional; if absent, remediation tasks skip SBOM deltas and still return deterministic outputs.
+
+## 2) Canonical chunk mapping
+
+| LNM source | Advisory AI chunk | Transformation |
+| --- | --- | --- |
+| `advisory_observations._id` | `source_id` | Stored verbatim; used for citations. |
+| `advisory_observations.advisoryId` | `advisory_key` | Also populates `content_hash` seed. |
+| `advisory_observations.summary` | `text` | Trimmed, Markdown-safe. |
+| `advisory_observations.affected[].purl` | `purl` | Lowercased, deduped; no range merging. |
+| `advisory_observations.severities[]` | `severity` | Passed through; multiple severities allowed. |
+| `advisory_observations.references[]` | `references` | Sorted for determinism. |
+| `advisory_observations.relationships[]` | `relationships` | Surface upstream `type/source/target/provenance`; no merge. |
+| `advisory_observations.provenance.sourceArtifactSha` | `content_hash` | Drives dedup + cache key. |
+| `advisory_linksets.conflicts[]` | `conflicts` | Serialized verbatim for conflict tasks. |
+| `advisory_linksets.normalized.purls|versions|ranges|severities` | `normalized` | Used as hints only; never overwrite observation fields. |
+
+Chunk ordering: observations sorted by `(source, advisoryId, provenance.fetchedAt)` as per LNM invariant; chunks are emitted in the same order to keep cache keys stable.
+
+## 3) Output citation rules
+
+- `citations[n].sourceId` points to the LNM `source_id`; `citations[n].uri` must remain the upstream reference URI when present.
+- If SBOM deltas are included, they appear as separate citations with `kind: "sbom"` and `sourceId` built from SBOM context digest (`sbom:{artifactId}:{digest}`).
+- Conflict outputs must echo `linkset.conflicts[].reason` in the Markdown body with matching citation indexes; guardrails block outputs where a conflict reason lacks a citation.
+
+## 4) Error conditions (aligned to LNM)
+
+| Condition | Code | Notes |
+| --- | --- | --- |
+| Missing `lnm.linksetId` or `lnm.observationIds` | `409 advisory.contextUnavailable` | Caller should pass LNM IDs; retry once upstream emits them. |
+| Hash mismatch between `provenanceHash` and linkset | `409 advisory.contextHashMismatch` | Indicates tampering or stale payload; retry after refreshing linkset. |
+| Observation count exceeds clamp (defaults: 200 obs, 600 chunks) | `413 advisory.contextTooLarge` | Caller may request narrower `preferredSections` or reduce obs set. |
+| Conflicts array empty for conflict task | `422 advisory.conflict.noConflicts` | Signals upstream data gap; reported to Concelier. |
+
+## 5) Sample normalized RAG bundle
+
+```json
+{
+  "taskType": "Summary",
+  "advisoryKey": "csaf:redhat:RHSA-2025:1001",
+  "lnmBundle": {
+    "linksetId": "6561e41b3e3f4a6e9d3b91d0",
+    "provenanceHash": "sha256:0f7c...9ad3",
+    "chunks": [
+      {
+        "source_id": "concelier:ghsa:GHSA-xxxx:obs:6561e41b3e3f4a6e9d3b91c1",
+        "content_hash": "sha256:1234...",
+        "advisory_key": "csaf:redhat:RHSA-2025:1001",
+        "purl": "pkg:maven/org.example/foo@1.2.3",
+        "severity": [{"system":"cvssv3","score":7.8,"vector":"AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H"}],
+        "references": ["https://access.redhat.com/errata/RHSA-2025:1001"],
+        "relationships": [{"type":"affects","source":"nvd","target":"cpe:/o:redhat:enterprise_linux:9"}]
+      }
+    ],
+    "conflicts": [
+      {"field":"affected.versions","reason":"vendor_range_differs","values":["<1.2.0","<=1.2.3"]}
+    ]
+  },
+  "sbomSummary": {
+    "artifactId": "registry.stella-ops.internal/runtime/api",
+    "versionTimeline": 8,
+    "dependencyPaths": 5
+  }
+}
+```
+
+Operators can store this bundle alongside plan cache entries; the `lnmBundle.provenanceHash` proves the evidence set matches the frozen Concelier linkset.
+
+## 6) Operator validation steps
+
+- Verify LNM collections at schema v1 (2025-11-17 freeze) before enabling Advisory AI tasks.
+- Ensure `lnm.provenanceHash` matches linkset `observationHashes` before calling Advisory AI.
+- Keep clamps deterministic: observations ≤200, chunks ≤600, timeline entries ≤500, dependency paths ≤200 (defaults; override only if documented).
+- When running offline, include LNM linkset exports in the Offline Kit to preserve citation replay.
--- a/docs/advisory-ai/guardrails-and-evidence.md
+++ b/docs/advisory-ai/guardrails-and-evidence.md
@@ -0,0 +1,62 @@
+# Advisory AI Guardrails & Evidence Intake
+
+_Updated: 2025-11-18 · Owner: Advisory AI Docs Guild · Status: Draft (Sprint 0111)_
+
+This note captures the guardrail behaviors and evidence intake boundaries required by Sprint 0111 tasks (`AIAI-DOCS-31-001`, `AIAI-RAG-31-003`). It binds Advisory AI guardrails to upstream evidence sources and clarifies how Link-Not-Merge (LNM) documents flow into Retrieval-Augmented Generation (RAG) payloads.
+
+## 1) Evidence sources and contracts
+
+- **Advisory observations (LNM)** — Consume immutable `advisory_observations` and `advisory_linksets` produced per `docs/modules/concelier/link-not-merge-schema.md` (frozen v1, 2025-11-17).
+- **VEX statements** — Excititor + VEX Lens linksets with trust weights; treated as structured chunks with `source_id` and `confidence`.
+- **SBOM context** — `SBOM-AIAI-31-001` contract: timelines and dependency paths retrieved via `ISbomContextRetriever` (`AddSbomContextHttpClient`), default clamps 500 timeline entries / 200 paths.
+- **Policy explain traces** — Policy Engine digests referenced by `policyVersion`; cache keys include policy hash to keep outputs replayable.
+- **Runtime posture (optional)** — Zastava signals (`exposure`, `admissionStatus`) when provided by Link-Not-Merge-enabled tenants; optional chunks tagged `runtime`.
+
+All evidence items must carry `content_hash` + `source_id`; Advisory AI never mutates or merges upstream facts (Aggregation-Only Contract).
+
+## 2) Guardrail stages
+
+1. **Pre-flight sanitization**
+   - Redact secrets (AWS-style keys, PEM blobs, generic tokens).
+   - Strip prompt-injection phrases; enforce max input payload 16 kB (configurable, default).
+   - Reject requests missing `advisoryKey` or linkset-backed evidence (LNM guard).
+2. **Prompt assembly**
+   - Deterministic section order: advisory excerpts → VEX statements → SBOM deltas → policy traces → runtime hints.
+   - Vector previews capped at 600 chars + ellipsis; section budgets fixed per profile (`default`, `gost-local`, `cloud-openai`).
+3. **LLM invocation (local/remote)**
+   - Profiles selected via `profile` field; remote profiles require Authority tenant consent and `advisory-ai:operate` + `aoc:verify`.
+4. **Validation & citation enforcement**
+   - Every emitted fact must map to an input chunk (`source_id` + `content_hash`); citations serialized as `[n]` in Markdown.
+   - Block outputs lacking citations, exceeding section budgets, or including unredacted PII.
+5. **Output sealing**
+   - Store `outputHash`, `inputDigest`, `provenanceHash`; wrap in DSSE when configured.
+   - Cache TTL defaults to 24h; regenerate only when inputs change or `forceRefresh=true`.
+
+Metrics: `advisory_ai_guardrail_blocks_total`, `advisory_ai_outputs_stored_total`, `advisory_ai_citation_coverage_ratio`. Logs carry `output_hash`, `profile`, and block reason; no secrets or raw prompt bodies are logged.
+
+## 3) RAG payload mapping to LNM (summary)
+
+| LNM field | RAG chunk field | Notes |
+| --- | --- | --- |
+| `observation._id` | `source_id` | Used for citations and conflict surfacing. |
+| `observation.advisoryId` | `advisory_key` | Keyed alongside task type in cache. |
+| `observation.affected[].purl` | `purl` | Included for remediation + SBOM joins. |
+| `observation.severities[]` | `severity` | Passed through unmerged; multiple severities allowed. |
+| `linkset.conflicts[]` | `conflicts` | Rendered verbatim for conflict tasks; no inference merges. |
+| `provenance.sourceArtifactSha` | `content_hash` | Drives determinism and replay. |
+
+See `docs/advisory-ai/evidence-payloads.md` for full JSON examples and alignment rules.
+
+## 4) Compliance with upstream artefacts
+
+- References: `CONSOLE-VULN-29-001`, `CONSOLE-VEX-30-001`, `CLI-VULN-29-001`, `CLI-VEX-30-001`, `EXCITITOR-CONSOLE-23-001`, `DEVOPS-AIAI-31-001`.
+- Guardrails must remain compatible with `docs/policy/assistant-parameters.md`; configuration knobs documented there are authoritative for env vars and defaults.
+- Packaging tasks (AIAI-PACKAGING-31-002) must include this guardrail summary in DSSE metadata to keep Offline Kit parity.
+
+## 5) Operator checklist
+
+- [ ] LNM feed enabled and Concelier schemas at v1 (2025-11-17).
+- [ ] SBOM retriever configured or `NullSbomContextClient` left as safe default.
+- [ ] Policy hash pinned via `policyVersion` when reproducibility is required.
+- [ ] Remote profiles only after Authority consent and profile allowlist are set.
+- [ ] Cache directories shared between web + worker hosts for DSSE sealing.
--- a/docs/advisory-ai/packaging.md
+++ b/docs/advisory-ai/packaging.md
@@ -0,0 +1,65 @@
+# Advisory AI Packaging & SBOM Bundle (AIAI-PACKAGING-31-002)
+
+_Updated: 2025-11-18 · Owner: Advisory AI Release · Status: Draft_
+
+Defines the artefacts and provenance required to ship Advisory AI in Sprint 0111, covering offline kits and on-prem deployments.
+
+## 1) Bundle contents
+
+| Artefact | Purpose | Provenance |
+| --- | --- | --- |
+| `advisory-ai-web` image | API surface + plan cache | SBOM: `SBOM-AIAI-31-001:web`; DSSE attestation signed by Release key |
+| `advisory-ai-worker` image | Queue + inference executor | SBOM: `SBOM-AIAI-31-001:worker`; DSSE attestation |
+| Prompt + guardrail pack | Deterministic prompts, redaction lists, validation rules | DSSE sealed; hash recorded in `prompts.manifest` |
+| Profile catalog | `default`, `fips-local`, `gost-local`, `cloud-openai` (disabled) | Versioned JSON, hashed; tenant consent flags captured |
+| Policy bundle | `policyVersion` digest for baseline evaluation; Authority importable | DSSE + provenance to Policy Engine digests |
+| LNM evidence export (optional) | Concelier `advisory_linksets` + `advisory_observations` for air-gap replay | Hash list aligned to `provenanceHash` in RAG bundles |
+| SBOM context client config | Example `AddSbomContextHttpClient` settings (`BaseAddress`, `Endpoint`, `ApiKey`) | Signed `sbom-context.example.json` |
+
+## 2) Directory layout (Offline Kit)
+
+```
+/offline-kit/advisory-ai/
+  images/
+    advisory-ai-web.tar.zst
+    advisory-ai-worker.tar.zst
+  sboms/
+    SBOM-AIAI-31-001-web.json
+    SBOM-AIAI-31-001-worker.json
+  provenance/
+    advisory-ai-web.intoto.jsonl
+    advisory-ai-worker.intoto.jsonl
+    prompts.manifest.dsse
+    profiles.catalog.json
+    policy-bundle.intoto.jsonl
+  config/
+    advisoryai.appsettings.example.json
+    sbom-context.example.json
+  evidence/
+    lnm-linksets.ndjson      # optional; aligns to linkset hashes in RAG bundles
+    lnm-observations.ndjson  # optional; immutable raw docs
+```
+
+- All files hashed into `SHA256SUMS` with DSSE signature (`SHA256SUMS.dsse`).
+- Profiles catalog and prompt pack hashes must be propagated into `AdvisoryAI:Provenance` settings for runtime verification.
+
+## 3) SBOM & provenance rules
+
+- SBOMs must follow SPDX 3.0.1; embed image digest (`sha256:<...>`) and build args.
+- Attestations use DSSE + SPDX predicate; signer key matches Release guild key referenced in `DEVOPS-AIAI-31-001`.
+- For air-gapped installs, operators verify: `slsa-verifier verify-attestation --source=stellaops/advisory-ai-web --bundle advisory-ai-web.intoto.jsonl --digest <image-digest>`.
+
+## 4) Deployment checklist
+
+- [ ] Import `advisory-ai-web` and `advisory-ai-worker` images to registry.
+- [ ] Apply `profiles.catalog.json`; ensure remote profiles disabled unless Authority consent granted.
+- [ ] Load prompt pack and set `AdvisoryAI:Prompts:ManifestHash` to `prompts.manifest`.
+- [ ] Configure SBOM client (or keep `NullSbomContextClient` default).
+- [ ] If shipping LNM evidence, seed `advisory_linksets` and `advisory_observations` collections before enabling inference.
+- [ ] Record hashes in deployment log; surface in Authority audit via `advisory_ai.output.generated` events.
+
+## 5) Update obligations
+
+- Any change to prompts, guardrails, or profiles → bump manifest hash and regenerate DSSE.
+- SBOM updates follow the same `SBOM-AIAI-31-001` idempotent contract; replace files, update `SHA256SUMS`, resign.
+- Link all changes into the sprint Execution Log and Decisions & Risks sections.