Add support for ГОСТ Р 34.10 digital signatures
- Implemented the GostKeyValue class for handling public key parameters in ГОСТ Р 34.10 digital signatures. - Created the GostSignedXml class to manage XML signatures using ГОСТ 34.10, including methods for computing and checking signatures. - Developed the GostSignedXmlImpl class to encapsulate the signature computation logic and public key retrieval. - Added specific key value classes for ГОСТ Р 34.10-2001, ГОСТ Р 34.10-2012/256, and ГОСТ Р 34.10-2012/512 to support different signature algorithms. - Ensured compatibility with existing XML signature standards while integrating ГОСТ cryptography.
This commit is contained in:
master
@@ -1,136 +1,136 @@
|
||||
# Advisory AI architecture
|
||||
|
||||
> Captures the retrieval, guardrail, and inference packaging requirements defined in the Advisory AI implementation plan and related module guides.
|
||||
|
||||
## 1) Goals
|
||||
|
||||
- Summarise advisories/VEX evidence into operator-ready briefs with citations.
|
||||
- Explain conflicting statements with provenance and trust weights (using VEX Lens & Excititor data).
|
||||
- Suggest remediation plans aligned with Offline Kit deployment models and scheduler follow-ups.
|
||||
- Operate deterministically where possible; cache generated artefacts with digests for audit.
|
||||
|
||||
## 2) Pipeline overview
|
||||
|
||||
```
|
||||
+---------------------+
|
||||
Concelier/VEX Lens | Evidence Retriever |
|
||||
Policy Engine ----> | (vector + keyword) | ---> Context Pack (JSON)
|
||||
Zastava runtime +---------------------+
|
||||
|
|
||||
v
|
||||
+-------------+
|
||||
| Prompt |
|
||||
| Assembler |
|
||||
+-------------+
|
||||
|
|
||||
v
|
||||
+-------------+
|
||||
| Guarded LLM |
|
||||
| (local/host)|
|
||||
+-------------+
|
||||
|
|
||||
v
|
||||
+-----------------+
|
||||
| Citation & |
|
||||
| Validation |
|
||||
+-----------------+
|
||||
|
|
||||
v
|
||||
+----------------+
|
||||
| Output cache |
|
||||
| (hash, bundle) |
|
||||
+----------------+
|
||||
```
|
||||
|
||||
## 3) Retrieval & context
|
||||
|
||||
- Hybrid search: vector embeddings (SBERT-compatible) + keyword filters for advisory IDs, PURLs, CVEs.
|
||||
- Context packs include:
|
||||
- Advisory raw excerpts with highlighted sections and source URLs.
|
||||
- VEX statements (normalized tuples + trust metadata).
|
||||
- Policy explain traces for the affected finding.
|
||||
- Runtime/impact hints from Zastava (exposure, entrypoints).
|
||||
- Export-ready remediation data (fixed versions, patches).
|
||||
- **SBOM context retriever** (AIAI-31-002) hydrates:
|
||||
- Version timelines (first/last observed, status, fix availability).
|
||||
- Dependency paths (runtime vs build/test, deduped by coordinate chain).
|
||||
- Tenant environment flags (prod/stage toggles) with optional blast radius summary.
|
||||
- Service-side clamps: max 500 timeline entries, 200 dependency paths, with client-provided toggles for env/blast data.
|
||||
- `AddSbomContextHttpClient(...)` registers the typed HTTP client that calls `/v1/sbom/context`, while `NullSbomContextClient` remains the safe default for environments that have not yet exposed the SBOM service.
|
||||
|
||||
**Sample configuration** (wire real SBOM base URL + API key):
|
||||
|
||||
```csharp
|
||||
services.AddSbomContextHttpClient(options =>
|
||||
{
|
||||
options.BaseAddress = new Uri("https://sbom-service.internal");
|
||||
options.Endpoint = "/v1/sbom/context";
|
||||
options.ApiKey = configuration["SBOM_SERVICE_API_KEY"];
|
||||
options.UserAgent = "stellaops-advisoryai/1.0";
|
||||
options.Tenant = configuration["TENANT_ID"];
|
||||
});
|
||||
|
||||
services.AddAdvisoryPipeline();
|
||||
```
|
||||
|
||||
After configuration, issue a smoke request (e.g., `ISbomContextRetriever.RetrieveAsync`) during deployment validation to confirm end-to-end connectivity and credentials before enabling Advisory AI endpoints.
|
||||
|
||||
Retriever requests and results are trimmed/normalized before hashing; metadata (counts, provenance keys) is returned for downstream guardrails. Unit coverage ensures deterministic ordering and flag handling.
|
||||
|
||||
All context references include `content_hash` and `source_id` enabling verifiable citations.
|
||||
|
||||
## 4) Guardrails
|
||||
|
||||
- Prompt templates enforce structure: summary, conflicts, remediation, references.
|
||||
- Response validator ensures:
|
||||
- No hallucinated advisories (every fact must map to input context).
|
||||
- Citations follow `[n]` indexing referencing actual sources.
|
||||
- Remediation suggestions only cite policy-approved sources (fixed versions, vendor hotfixes).
|
||||
- Moderation/PII filters prevent leaking secrets; responses failing validation are rejected and logged.
|
||||
- Pre-flight guardrails redact secrets (AWS keys, generic API tokens, PEM blobs), block "ignore previous instructions"-style prompt injection attempts, enforce citation presence, and cap prompt payload length (default 16 kB). Guardrail outcomes and redaction counts surface via `advisory_guardrail_blocks` / `advisory_outputs_stored` metrics.
|
||||
|
||||
## 5) Deterministic tooling
|
||||
|
||||
- **Version comparators** — offline semantic version + RPM EVR parsers with range evaluators. Supports chained constraints (`>=`, `<=`, `!=`) used by remediation advice and blast radius calcs.
|
||||
- Registered via `AddAdvisoryDeterministicToolset` for reuse across orchestrator, CLI, and services.
|
||||
- **Orchestration pipeline** — see `orchestration-pipeline.md` for prerequisites, task breakdown, and cross-guild responsibilities before wiring the execution flows.
|
||||
- **Planned extensions** — NEVRA/EVR comparators, ecosystem-specific normalisers, dependency chain scorers (AIAI-31-003 scope).
|
||||
- Exposed via internal interfaces to allow orchestrator/toolchain reuse; all helpers stay side-effect free and deterministic for golden testing.
|
||||
|
||||
## 6) Output persistence
|
||||
|
||||
- Cached artefacts stored in `advisory_ai_outputs` with fields:
|
||||
- `output_hash` (sha256 of JSON response).
|
||||
- `input_digest` (hash of context pack).
|
||||
- `summary`, `conflicts`, `remediation`, `citations`.
|
||||
- `generated_at`, `model_id`, `profile` (Sovereign/FIPS etc.).
|
||||
- `signatures` (optional DSSE if run in deterministic mode).
|
||||
- Offline bundle format contains `summary.md`, `citations.json`, `context_manifest.json`, `signatures/`.
|
||||
|
||||
## 7) Profiles & sovereignty
|
||||
|
||||
- **Profiles:** `default`, `fips-local` (FIPS-compliant local model), `gost-local`, `cloud-openai` (optional, disabled by default). Each profile defines allowed models, key management, and telemetry endpoints.
|
||||
- **CryptoProfile/RootPack integration:** generated artefacts can be signed using configured CryptoProfile to satisfy procurement/trust requirements.
|
||||
|
||||
## 8) APIs
|
||||
|
||||
- `POST /api/v1/advisory/{task}` — executes Summary/Conflict/Remediation pipeline (`task` ∈ `summary|conflict|remediation`). Requests accept `{advisoryKey, artifactId?, policyVersion?, profile, preferredSections?, forceRefresh}` and return sanitized prompt payloads, citations, guardrail metadata, provenance hash, and cache hints.
|
||||
- `GET /api/v1/advisory/outputs/{cacheKey}?taskType=SUMMARY&profile=default` — retrieves cached artefacts for downstream consumers (Console, CLI, Export Center). Guardrail state and provenance hash accompany results.
|
||||
|
||||
All endpoints accept `profile` parameter (default `fips-local`) and return `output_hash`, `input_digest`, and `citations` for verification.
|
||||
|
||||
## 9) Observability
|
||||
|
||||
- Metrics: `advisory_ai_requests_total{profile,type}`, `advisory_ai_latency_seconds`, `advisory_ai_validation_failures_total`.
|
||||
- Logs: include `output_hash`, `input_digest`, `profile`, `model_id`, `tenant`, `artifacts`. Sensitive context is not logged.
|
||||
- Traces: spans for retrieval, prompt assembly, model inference, validation, cache write.
|
||||
|
||||
## 10) Operational controls
|
||||
|
||||
- Feature flags per tenant (`ai.summary.enabled`, `ai.remediation.enabled`).
|
||||
- Rate limits (per tenant, per profile) enforced by Orchestrator to prevent runaway usage.
|
||||
- Offline/air-gapped deployments run local models packaged with Offline Kit; model weights validated via manifest digests.
|
||||
|
||||
# Advisory AI architecture
|
||||
|
||||
> Captures the retrieval, guardrail, and inference packaging requirements defined in the Advisory AI implementation plan and related module guides.
|
||||
|
||||
## 1) Goals
|
||||
|
||||
- Summarise advisories/VEX evidence into operator-ready briefs with citations.
|
||||
- Explain conflicting statements with provenance and trust weights (using VEX Lens & Excititor data).
|
||||
- Suggest remediation plans aligned with Offline Kit deployment models and scheduler follow-ups.
|
||||
- Operate deterministically where possible; cache generated artefacts with digests for audit.
|
||||
|
||||
## 2) Pipeline overview
|
||||
|
||||
```
|
||||
+---------------------+
|
||||
Concelier/VEX Lens | Evidence Retriever |
|
||||
Policy Engine ----> | (vector + keyword) | ---> Context Pack (JSON)
|
||||
Zastava runtime +---------------------+
|
||||
|
|
||||
v
|
||||
+-------------+
|
||||
| Prompt |
|
||||
| Assembler |
|
||||
+-------------+
|
||||
|
|
||||
v
|
||||
+-------------+
|
||||
| Guarded LLM |
|
||||
| (local/host)|
|
||||
+-------------+
|
||||
|
|
||||
v
|
||||
+-----------------+
|
||||
| Citation & |
|
||||
| Validation |
|
||||
+-----------------+
|
||||
|
|
||||
v
|
||||
+----------------+
|
||||
| Output cache |
|
||||
| (hash, bundle) |
|
||||
+----------------+
|
||||
```
|
||||
|
||||
## 3) Retrieval & context
|
||||
|
||||
- Hybrid search: vector embeddings (SBERT-compatible) + keyword filters for advisory IDs, PURLs, CVEs.
|
||||
- Context packs include:
|
||||
- Advisory raw excerpts with highlighted sections and source URLs.
|
||||
- VEX statements (normalized tuples + trust metadata).
|
||||
- Policy explain traces for the affected finding.
|
||||
- Runtime/impact hints from Zastava (exposure, entrypoints).
|
||||
- Export-ready remediation data (fixed versions, patches).
|
||||
- **SBOM context retriever** (AIAI-31-002) hydrates:
|
||||
- Version timelines (first/last observed, status, fix availability).
|
||||
- Dependency paths (runtime vs build/test, deduped by coordinate chain).
|
||||
- Tenant environment flags (prod/stage toggles) with optional blast radius summary.
|
||||
- Service-side clamps: max 500 timeline entries, 200 dependency paths, with client-provided toggles for env/blast data.
|
||||
- `AddSbomContextHttpClient(...)` registers the typed HTTP client that calls `/v1/sbom/context`, while `NullSbomContextClient` remains the safe default for environments that have not yet exposed the SBOM service.
|
||||
|
||||
**Sample configuration** (wire real SBOM base URL + API key):
|
||||
|
||||
```csharp
|
||||
services.AddSbomContextHttpClient(options =>
|
||||
{
|
||||
options.BaseAddress = new Uri("https://sbom-service.internal");
|
||||
options.Endpoint = "/v1/sbom/context";
|
||||
options.ApiKey = configuration["SBOM_SERVICE_API_KEY"];
|
||||
options.UserAgent = "stellaops-advisoryai/1.0";
|
||||
options.Tenant = configuration["TENANT_ID"];
|
||||
});
|
||||
|
||||
services.AddAdvisoryPipeline();
|
||||
```
|
||||
|
||||
After configuration, issue a smoke request (e.g., `ISbomContextRetriever.RetrieveAsync`) during deployment validation to confirm end-to-end connectivity and credentials before enabling Advisory AI endpoints.
|
||||
|
||||
Retriever requests and results are trimmed/normalized before hashing; metadata (counts, provenance keys) is returned for downstream guardrails. Unit coverage ensures deterministic ordering and flag handling.
|
||||
|
||||
All context references include `content_hash` and `source_id` enabling verifiable citations.
|
||||
|
||||
## 4) Guardrails
|
||||
|
||||
- Prompt templates enforce structure: summary, conflicts, remediation, references.
|
||||
- Response validator ensures:
|
||||
- No hallucinated advisories (every fact must map to input context).
|
||||
- Citations follow `[n]` indexing referencing actual sources.
|
||||
- Remediation suggestions only cite policy-approved sources (fixed versions, vendor hotfixes).
|
||||
- Moderation/PII filters prevent leaking secrets; responses failing validation are rejected and logged.
|
||||
- Pre-flight guardrails redact secrets (AWS keys, generic API tokens, PEM blobs), block "ignore previous instructions"-style prompt injection attempts, enforce citation presence, and cap prompt payload length (default 16 kB). Guardrail outcomes and redaction counts surface via `advisory_guardrail_blocks` / `advisory_outputs_stored` metrics.
|
||||
|
||||
## 5) Deterministic tooling
|
||||
|
||||
- **Version comparators** — offline semantic version + RPM EVR parsers with range evaluators. Supports chained constraints (`>=`, `<=`, `!=`) used by remediation advice and blast radius calcs.
|
||||
- Registered via `AddAdvisoryDeterministicToolset` for reuse across orchestrator, CLI, and services.
|
||||
- **Orchestration pipeline** — see `orchestration-pipeline.md` for prerequisites, task breakdown, and cross-guild responsibilities before wiring the execution flows.
|
||||
- **Planned extensions** — NEVRA/EVR comparators, ecosystem-specific normalisers, dependency chain scorers (AIAI-31-003 scope).
|
||||
- Exposed via internal interfaces to allow orchestrator/toolchain reuse; all helpers stay side-effect free and deterministic for golden testing.
|
||||
|
||||
## 6) Output persistence
|
||||
|
||||
- Cached artefacts stored in `advisory_ai_outputs` with fields:
|
||||
- `output_hash` (sha256 of JSON response).
|
||||
- `input_digest` (hash of context pack).
|
||||
- `summary`, `conflicts`, `remediation`, `citations`.
|
||||
- `generated_at`, `model_id`, `profile` (Sovereign/FIPS etc.).
|
||||
- `signatures` (optional DSSE if run in deterministic mode).
|
||||
- Offline bundle format contains `summary.md`, `citations.json`, `context_manifest.json`, `signatures/`.
|
||||
|
||||
## 7) Profiles & sovereignty
|
||||
|
||||
- **Profiles:** `default`, `fips-local` (FIPS-compliant local model), `gost-local`, `cloud-openai` (optional, disabled by default). Each profile defines allowed models, key management, and telemetry endpoints.
|
||||
- **CryptoProfile/RootPack integration:** generated artefacts can be signed using configured CryptoProfile to satisfy procurement/trust requirements.
|
||||
|
||||
## 8) APIs
|
||||
|
||||
- `POST /api/v1/advisory/{task}` — executes Summary/Conflict/Remediation pipeline (`task` ∈ `summary|conflict|remediation`). Requests accept `{advisoryKey, artifactId?, policyVersion?, profile, preferredSections?, forceRefresh}` and return sanitized prompt payloads, citations, guardrail metadata, provenance hash, and cache hints.
|
||||
- `GET /api/v1/advisory/outputs/{cacheKey}?taskType=SUMMARY&profile=default` — retrieves cached artefacts for downstream consumers (Console, CLI, Export Center). Guardrail state and provenance hash accompany results.
|
||||
|
||||
All endpoints accept `profile` parameter (default `fips-local`) and return `output_hash`, `input_digest`, and `citations` for verification.
|
||||
|
||||
## 9) Observability
|
||||
|
||||
- Metrics: `advisory_ai_requests_total{profile,type}`, `advisory_ai_latency_seconds`, `advisory_ai_validation_failures_total`.
|
||||
- Logs: include `output_hash`, `input_digest`, `profile`, `model_id`, `tenant`, `artifacts`. Sensitive context is not logged.
|
||||
- Traces: spans for retrieval, prompt assembly, model inference, validation, cache write.
|
||||
|
||||
## 10) Operational controls
|
||||
|
||||
- Feature flags per tenant (`ai.summary.enabled`, `ai.remediation.enabled`).
|
||||
- Rate limits (per tenant, per profile) enforced by Orchestrator to prevent runaway usage.
|
||||
- Offline/air-gapped deployments run local models packaged with Offline Kit; model weights validated via manifest digests.
|
||||
|
||||
## 11) Hosting surfaces
|
||||
|
||||
- **WebService** — exposes `/v1/advisory-ai/pipeline/{task}` to materialise plans and enqueue execution messages.
|
||||
@@ -140,7 +140,7 @@ All endpoints accept `profile` parameter (default `fips-local`) and return `outp
|
||||
|
||||
## 12) QA harness & determinism (Sprint 110 refresh)
|
||||
|
||||
- **Injection fixtures:** `src/AdvisoryAI/__Tests/StellaOps.AdvisoryAI.Tests/TestData/prompt-injection-fixtures.txt` drives `AdvisoryGuardrailInjectionTests`, ensuring blocked phrases (`ignore previous instructions`, `override the system prompt`, etc.) are rejected with redaction counters, preventing prompt-injection regressions.
|
||||
- **Injection fixtures:** `src/AdvisoryAI/__Tests/StellaOps.AdvisoryAI.Tests/TestData/guardrail-injection-cases.json` now enumerates both blocked and allow-listed prompts (redactions, citation checks, prompt-length clamps) while the legacy `prompt-injection-fixtures.txt` file continues to supply quick block-only payloads. `AdvisoryGuardrailInjectionTests` consumes both datasets so guardrail regressions surface with metadata (blocked phrase counts, redaction counters, citation enforcement) instead of single-signal failures.
|
||||
- **Golden prompts:** `summary-prompt.json` now pairs with `conflict-prompt.json`; `AdvisoryPromptAssemblerTests` load both to enforce deterministic JSON payloads across task types and verify vector preview truncation (600 characters + ellipsis) keeps prompts under the documented perf ceiling.
|
||||
- **Plan determinism:** `AdvisoryPipelineOrchestratorTests` shuffle structured/vector/SBOM inputs and assert cache keys + metadata remain stable, proving that seeded plan caches stay deterministic even when retrievers emit out-of-order results.
|
||||
- **Execution telemetry:** `AdvisoryPipelineExecutorTests` exercise partial citation coverage (target ≥0.5 when only half the structured chunks are cited) so `advisory_ai_citation_coverage_ratio` reflects real guardrail quality.
|
||||
|
||||
@@ -321,6 +321,7 @@ Every Stella Ops service that consumes Authority tokens **must**:
|
||||
* `authority.jwks_rotations_total`
|
||||
* `authority.errors_total{type}`
|
||||
* **Audit log** (immutable sink): token issuance (`sub`, `aud`, `scopes`, `tid`, `inst`, `cnf thumbprint`, `jti`), revocations, admin changes.
|
||||
* **Plugin telemetry**: password-capable plug-ins (Standard, LDAP) emit `authority.plugin.<name>.password_verification` events via `IAuthEventSink`, inheriting correlation/client/tenant/network metadata from `AuthorityCredentialAuditContext`. Each event includes `plugin.failed_attempts`, `plugin.lockout_until`, `plugin.retry_after_seconds`, `plugin.failure_code`, and any plug-in specific signals so SOC tooling can trace lockouts and rate-limit responses even in air-gapped deployments. Offline Kits ship the plug-in binaries plus the curated manifests (`etc/authority.plugins/*.yaml`) so these audit flows exist out of the box.
|
||||
* **Tracing**: token flows, DB reads, JWKS cache.
|
||||
|
||||
---
|
||||
|
||||
@@ -52,3 +52,5 @@ StellaOps:
|
||||
3. Export `STELLA_PROFILE=<name>` before running `stella auth login` or individual pack commands.
|
||||
|
||||
The CLI reads the profile, applies the Authority configuration, and requests the listed scopes so the resulting tokens satisfy Task Runner and Packs Registry expectations.
|
||||
|
||||
> **Pack approval tip** – `stella pack approve` now relays `--pack-run-id`, `--pack-gate-id`, and `--pack-plan-hash` to Authority whenever it asks for `packs.approve`. Profiles don’t store these values (they change per run), but keeping the approver profile loaded ensures the CLI can prompt for the metadata, validate it against the plan hash, and satisfy the Authority procedure documented in `docs/task-packs/runbook.md#4-approvals-workflow`.
|
||||
|
||||
@@ -33,6 +33,7 @@ Excititor converts heterogeneous VEX feeds into raw observations and linksets th
|
||||
- MongoDB for observation storage and job metadata.
|
||||
- Offline kit packaging aligned with Concelier merges.
|
||||
- Connector-specific runbooks (see `docs/modules/concelier/operations/connectors`).
|
||||
- Ubuntu CSAF provenance knobs: [`operations/ubuntu-csaf.md`](operations/ubuntu-csaf.md) captures TrustWeight/Tier, cosign, and fingerprint configuration for the sprint 120 enrichment.
|
||||
|
||||
## Backlog references
|
||||
- DOCS-LNM-22-006 / DOCS-LNM-22-007 (shared with Concelier).
|
||||
|
||||
66
docs/modules/excititor/operations/ubuntu-csaf.md
Normal file
66
docs/modules/excititor/operations/ubuntu-csaf.md
Normal file
@@ -0,0 +1,66 @@
|
||||
# Ubuntu CSAF connector runbook
|
||||
|
||||
> Updated 2025-11-09 alongside sprint 110/120 trust-provenance work.
|
||||
|
||||
## Purpose
|
||||
- Ingest Ubuntu USN/CSAF statements via the restart-only connector (`StellaOps.Excititor.Connectors.Ubuntu.CSAF`).
|
||||
- Preserve Aggregation-Only Contract guarantees while surfacing issuance provenance (`vex.provenance.*`) for VEX Lens and Policy Engine.
|
||||
- Allow operators to tune trust weighting (tiers, fingerprints, cosign issuers) without recompiling the connector.
|
||||
|
||||
## Configuration keys
|
||||
| Key | Default | Notes |
|
||||
| --- | --- | --- |
|
||||
| `Excititor:Connectors:Ubuntu:IndexUri` | `https://ubuntu.com/security/csaf/index.json` | Ubuntu CSAF index. Override only when mirroring the feed. |
|
||||
| `...:Channels` | `["stable"]` | List of channel names to poll. Order preserved for deterministic cursoring. |
|
||||
| `...:MetadataCacheDuration` | `4h` | How long to cache catalog metadata before re-fetching. |
|
||||
| `...:PreferOfflineSnapshot` / `OfflineSnapshotPath` / `PersistOfflineSnapshot` | `false` / `null` / `true` | Enable when running from Offline Kit bundles. Snapshot path must be reachable/read-only under sealed deployments. |
|
||||
| `...:TrustWeight` | `0.75` | Baseline trust weight (0–1). Lens multiplies this by freshness/justification modifiers. |
|
||||
| `...:TrustTier` | `"distro"` | Friendly tier label surfaced via `vex.provenance.trust.tier` (e.g., `distro-trusted`, `community`). |
|
||||
| `...:CosignIssuer` / `CosignIdentityPattern` | `null` | Supply when Ubuntu publishes cosign attestations (issuer URL and identity regex). Required together. |
|
||||
| `...:PgpFingerprints` | `[]` | Ordered list of trusted PGP fingerprints. Emitted verbatim as `vex.provenance.pgp.fingerprints`. |
|
||||
|
||||
## Example `appsettings.json`
|
||||
```jsonc
|
||||
{
|
||||
"Excititor": {
|
||||
"Connectors": {
|
||||
"Ubuntu": {
|
||||
"IndexUri": "https://mirror.example.com/security/csaf/index.json",
|
||||
"Channels": ["stable", "esm-apps"],
|
||||
"TrustWeight": 0.82,
|
||||
"TrustTier": "distro-trusted",
|
||||
"CosignIssuer": "https://issuer.ubuntu.com",
|
||||
"CosignIdentityPattern": "spiffe://ubuntu/vex/*",
|
||||
"PgpFingerprints": [
|
||||
"0123456789ABCDEF0123456789ABCDEF01234567",
|
||||
"89ABCDEF0123456789ABCDEF0123456789ABCDEF"
|
||||
],
|
||||
"PreferOfflineSnapshot": true,
|
||||
"OfflineSnapshotPath": "/opt/stella/offline/ubuntu/index.json"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Environment variable cheatsheet
|
||||
```
|
||||
Excititor__Connectors__Ubuntu__TrustWeight=0.9
|
||||
Excititor__Connectors__Ubuntu__TrustTier=distro-critical
|
||||
Excititor__Connectors__Ubuntu__PgpFingerprints__0=AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
|
||||
Excititor__Connectors__Ubuntu__PgpFingerprints__1=BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB
|
||||
Excititor__Connectors__Ubuntu__CosignIssuer=https://issuer.ubuntu.com
|
||||
Excititor__Connectors__Ubuntu__CosignIdentityPattern=spiffe://ubuntu/vex/*
|
||||
```
|
||||
|
||||
## Operational checklist
|
||||
1. **Before enabling** – import the Ubuntu PGP bundle (Offline Kit provides `certificates/ubuntu-vex.gpg`) and set the fingerprints so provenance metadata stays deterministic.
|
||||
2. **Validate provenance output** – run `dotnet test src/Excititor/__Tests/StellaOps.Excititor.Connectors.Ubuntu.CSAF.Tests --filter FetchAsync_IngestsNewDocument` to ensure the connector emits the `vex.provenance.*` fields expected by VEX Lens.
|
||||
3. **Monitor Lens weights** – Grafana panels `VEX Lens / Trust Inputs` show the weight/tier captured per provider. Ubuntu rows should reflect the configured `TrustWeight` and fingerprints.
|
||||
4. **Rotate fingerprints** – update `PgpFingerprints` when Canonical rotates signing keys. Apply the change, restart Excititor workers, verify the provenance metadata, then trigger a targeted Lens recompute for Ubuntu issuers.
|
||||
5. **Offline mode** – populate `OfflineSnapshotPath` via Offline Kit bundles before toggling `PreferOfflineSnapshot`. Keep snapshots in the sealed `/opt/stella/offline` hierarchy for auditability.
|
||||
|
||||
## Troubleshooting
|
||||
- **Connector refuses to start** – check logs for `InvalidOperationException` referencing `CosignIssuer`/`CosignIdentityPattern` or missing snapshot path; the validator enforces complete pairs and on-disk paths.
|
||||
- **Lens still sees default weights** – confirm the Excititor deployment picked up the new settings (view `/excititor/health` JSON → `connectors.providers[].options`). Lens only overrides when the provenance payload includes `vex.provenance.trust.*` fields.
|
||||
- **PGP mismatch alerts** – if Lens reports fingerprint mismatches, ensure the list ordering matches Canonical’s published order; duplicates are trimmed, so provide each fingerprint once.
|
||||
66
docs/modules/scanner/deterministic-sbom-compose.md
Normal file
66
docs/modules/scanner/deterministic-sbom-compose.md
Normal file
@@ -0,0 +1,66 @@
|
||||
# Deterministic SBOM Composition (Spec Draft)
|
||||
|
||||
> **Status:** Draft v0.1 (Sprint 136 / 203 / 209 linkage)
|
||||
> **Owners:** Scanner Guild · DevEx/CLI Guild · UI Guild · Docs Guild · Security Guild
|
||||
> **Related Tasks:** `SCANNER-SURFACE-04`, `SURFACE-FS-07`, `SCANNER-EMIT-15-001`, `SCANNER-SORT-02`, `CLI-SBOM-60-001`, `CLI-SBOM-60-002`, `UI-SBOM-DET-01`, `UI-POLICY-DET-01`, `DOCS-SCANNER-DET-01`, `DOCS-POLICY-DET-01`, `DOCS-CLI-DET-01`, `SCANNER-CRYPTO-90-002`, `SCANNER-CRYPTO-90-003`
|
||||
|
||||
## 1. Purpose
|
||||
|
||||
Guarantee that every container scan yields **provably deterministic** SBOM artifacts that can be verified offline. Each layer fragment is DSSE-signed before merge, `_composition.json` captures the canonical merge recipe, and the final CycloneDX inventory/usage SBOMs expose Merkle roots and `stella.contentHash` properties. CLI/UI/policy layers consume those signals to block non-deterministic releases and provide human-friendly diagnostics.
|
||||
|
||||
## 2. Scope
|
||||
|
||||
### 2.1 Fragment attestation
|
||||
|
||||
- Scanner Worker emits `layer.fragments` payloads as canonical JSON (lexicographic keys, compact whitespace, normalized timestamps, ordered component arrays).
|
||||
- Each fragment is signed via DSSE (default Ed25519; PQ/Dilithium toggle routed through `ICryptoProviderRegistry`).
|
||||
- `_composition.json` records `{layerDigest, fragmentSha256, dsseEnvelopeSha256}` per fragment alongside the overall Merkle root.
|
||||
- Surface manifests append links to fragment DSSE envelopes so offline kits can fetch them without re-scan.
|
||||
|
||||
### 2.2 Canonical merge
|
||||
|
||||
- Merge order strictly follows `layerDigest` (ascending hex) and then `component.identity.purl` (fallback to `identity.key`).
|
||||
- CycloneDX metadata gains:
|
||||
- `properties["stellaops:stella.contentHash"]` for each fragment and composition root.
|
||||
- `properties["stellaops:composition.manifest"]` referencing `_composition.json` CAS URI.
|
||||
- `properties["stellaops:merkle.root"]` for the composed BOM.
|
||||
- `ScannerTimestamps.Normalize` continues to zero fractional microseconds; SBOM timestamps should default to `"0001-01-01T00:00:00Z"` when no semantic timestamp is required.
|
||||
|
||||
### 2.3 Surface manifest extensions
|
||||
|
||||
- `SurfaceManifestArtifact` gains optional `attestations[]` with `{kind, mediaType, digest, uri}` for DSSE envelopes.
|
||||
- `_composition.json` is published as an additional artifact kind `composition.recipe`.
|
||||
- Surface reader/writer validate Merkle roots before caching manifest entries.
|
||||
|
||||
### 2.4 Tooling impacts
|
||||
|
||||
- **CLI (`stella sbomer ...`)**: adds `layer` and `compose` verbs, deterministic diff reporting, and offline verification per `_composition.json`.
|
||||
- **UI/Policy**: determinism badge, drift diffs, and a policy gate that blocks releases when fragment DSSE/verifications fail.
|
||||
- **Docs**: new guides under `docs/scanner` & `docs/cli` plus policy references detailing how to interpret determinism metadata.
|
||||
- **Crypto**: PQ-friendly DSSE toggle delivered via `SCANNER-CRYPTO-90-002/003` so sovereign bundles can select Dilithium/Falcon.
|
||||
|
||||
## 3. Verification Flow (offline kit)
|
||||
|
||||
1. Verify DSSE on each fragment (using `verifiers.json`).
|
||||
2. Recompute `sha256(c14n(fragment))` and compare with `_composition.json`.
|
||||
3. Re-run composition locally (using canonical ordering) and compare `sha256(c14n(composed))` against `manifest.properties["stellaops:merkle.root"]`.
|
||||
4. Optionally validate provided Merkle proofs (leaf → root) and attest that the UI/Policy gate marked the scan as deterministic.
|
||||
|
||||
## 4. Deliverables Checklist
|
||||
|
||||
| Area | Deliverable |
|
||||
| --- | --- |
|
||||
| Scanner Worker | DSSE per fragment, `_composition.json`, canonical fragment serializer, Surface manifest updates |
|
||||
| Emit pipeline | Layer-sorted composition, `stella.contentHash`, Merkle metadata, PQ-aware signing hooks |
|
||||
| CLI | `stella sbomer layer/compose/drift`, verification commands, documentation |
|
||||
| UI | Determinism badge, drift diagnostics, policy gate wiring |
|
||||
| Docs | Updated scanner/cli/policy guides, offline kit instructions |
|
||||
| Tests | Regression suites covering canonicalization, DSSE verification, PQ keypaths, Merkle roots |
|
||||
|
||||
## 5. References
|
||||
|
||||
- `docs/modules/scanner/architecture.md`
|
||||
- `docs/modules/scanner/design/surface-fs.md`
|
||||
- `docs/replay/DETERMINISTIC_REPLAY.md`
|
||||
- `docs/modules/cli/architecture.md`
|
||||
- `docs/modules/policy/architecture.md`
|
||||
@@ -15,6 +15,13 @@ VEX Lens produces a deterministic, provenance-rich consensus view of VEX stateme
|
||||
- **Explainability traces** — capture derived-from chains, conflicting issuers, and trust deltas to power UI drilldowns and CLI audits.
|
||||
- **Recompute orchestration** — Orchestrator jobs trigger recompute on Excititor deltas, issuer updates, or policy knob changes with deterministic ordering and SRM manifests.
|
||||
|
||||
### Provenance-aware trust weighting (new)
|
||||
|
||||
- **Connector metadata contract.** Excititor connectors now emit `vex.provenance.*` fields (provider id/name/kind, `trust.weight`, `trust.tier`, human-readable `trust.note`, `cosign.*`, and ordered `pgp.fingerprints`). VEX Lens must ingest these keys verbatim so the trust engine can reason about issuer pedigree without hitting external registries for every statement.
|
||||
- **Weight calculation.** Lens uses the supplied `trust.weight` as the baseline score, then multiplies by freshness decay and justification scope multipliers. Missing weights default to the Issuer Directory profile, but connector-provided values take precedence so Ubuntu/SUSE mirror feeds can tune their relative influence.
|
||||
- **Integrity hints.** Presence of `vex.provenance.cosign.*` or `pgp.fingerprints` toggles signature-policy shortcuts: if Lens sees a statement whose provenance indicates cosign keyless mode plus Rekor URI, it can skip redundant issuer lookups and apply the “cryptographically verified” confidence tier immediately.
|
||||
- **Policy exposure.** Consensus APIs surface the original provenance payload inside each `sources[]` entry so Policy Engine, Advisory AI, and Console can explain why a lower-tier issuer lost a conflict (e.g., different `trust.tier` or missing fingerprints). See the updated payload reference in `docs/vex/consensus-json.md`.
|
||||
|
||||
## Current workstreams (Q4 2025)
|
||||
- `VEXLENS-30-001..004` — build normalisation pipeline, product mapping library, and trust weighting engine (in progress; dependencies captured in src/VexLens/StellaOps.VexLens/TASKS.md).
|
||||
- `VEXLENS-30-005..007` — expose consensus APIs and export flows, aligning docs with future `/docs/vex/consensus-*.md` deliverables.
|
||||
|
||||
@@ -8,9 +8,23 @@ Compute a deterministic, reproducible consensus view over multiple VEX statement
|
||||
|
||||
## 2) Inputs
|
||||
|
||||
- `vex_normalized` tuples emitted by Excititor (status, justification, scope, timestamp, content hash).
|
||||
- Issuer trust registry (`vex_issuer_registry`) providing trust tier, confidence, authority scope.
|
||||
- Optional runtime context (Zastava exposure) and policy precedence rules.
|
||||
- `vex_normalized` tuples emitted by Excititor (status, justification, scope, timestamp, content hash).
|
||||
- Issuer trust registry (`vex_issuer_registry`) providing trust tier, confidence, authority scope.
|
||||
- Optional runtime context (Zastava exposure) and policy precedence rules.
|
||||
|
||||
### Provenance field mapping (new input contract)
|
||||
|
||||
Excititor connectors now stamp every raw VEX document with `vex.provenance.*` metadata. Lens ingests these keys alongside the normalized tuples:
|
||||
|
||||
| Field | Description | Lens usage |
|
||||
| --- | --- | --- |
|
||||
| `vex.provenance.provider` / `providerName` / `providerKind` | Logical issuer identity and type supplied by the connector (e.g., `excititor:ubuntu`, `distro`). | Seed issuer lookup, short-circuit Issuer Directory calls when we already trust the connector’s profile. |
|
||||
| `vex.provenance.trust.weight` | Connector-provided base weight (0–1). | Multiplied by freshness decay & justification multipliers; overrides registry default. |
|
||||
| `vex.provenance.trust.tier` & `trust.note` | Human/ops tier labels (`vendor`, `distro-trusted`, etc.) plus descriptive note. | Drives secondary sort (after timestamp) and Console labels; conflicts report per-tier deltas. |
|
||||
| `vex.provenance.cosign.*` | Cosign issuer/identity pattern (+ optional Fulcio/Rekor URIs). | When present, Lens marks the statement as “cryptographically attested” and applies the higher confidence bucket immediately. |
|
||||
| `vex.provenance.pgp.fingerprints` | Ordered list of PGP fingerprints used by the feed. | Enables Lens to validate deterministic fingerprint sets against Issuer Directory entries and flag mismatches in conflict summaries. |
|
||||
|
||||
The trust engine preserves the raw metadata so downstream components can audit decisions or remap tiers without replaying ingestion.
|
||||
|
||||
## 3) Core algorithm
|
||||
|
||||
|
||||
@@ -15,11 +15,11 @@
|
||||
## Work breakdown
|
||||
- **VEX Lens service**
|
||||
- Normalise VEX payloads, maintain scope scores, compute consensus digest.
|
||||
- Trust weighting functions (issuer tier, freshness decay, scope quality).
|
||||
- Trust weighting functions (issuer tier, freshness decay, scope quality) ingest the new `vex.provenance.*` contract emitted by Excititor connectors (provider weight/tier, cosign metadata, fingerprints) so connector-tuned trust flows all the way to consensus.
|
||||
- Idempotent workers for consensus projection and history tracking.
|
||||
- Conflict handling queue for manual review and notifications.
|
||||
- **Integrations**
|
||||
- Excitor: enrich VEX events with issuer hints, signatures, product trees.
|
||||
- Excitor: enrich VEX events with issuer hints, signatures, product trees, and now connector-supplied trust weights/tiers that Lens consumes directly.
|
||||
- Policy Engine: trust knobs, simulation endpoints, policy-driven recompute.
|
||||
- Vuln Explorer & Advisory AI: consensus badges, conflict surfacing.
|
||||
- **Issuer Directory**
|
||||
|
||||
Reference in New Issue
Block a user