Add Authority Advisory AI and API Lifecycle Configuration

- Introduced AuthorityAdvisoryAiOptions and related classes for managing advisory AI configurations, including remote inference options and tenant-specific settings. - Added AuthorityApiLifecycleOptions to control API lifecycle settings, including legacy OAuth endpoint configurations. - Implemented validation and normalization methods for both advisory AI and API lifecycle options to ensure proper configuration. - Created AuthorityNotificationsOptions and its related classes for managing notification settings, including ack tokens, webhooks, and escalation options. - Developed IssuerDirectoryClient and related models for interacting with the issuer directory service, including caching mechanisms and HTTP client configurations. - Added support for dependency injection through ServiceCollectionExtensions for the Issuer Directory Client. - Updated project file to include necessary package references for the new Issuer Directory Client library.
2025-11-02 13:40:38 +02:00
parent 66cb6c4b8a
commit f98cea3bcf
516 changed files with 68157 additions and 24754 deletions
--- a/docs/modules/attestor/architecture.md
+++ b/docs/modules/attestor/architecture.md
@@ -45,17 +45,23 @@ Trust boundary: **Only the Signer** is allowed to call submission endpoints; enf
 - `StellaOps.BuildProvenance@1`
 - `StellaOps.SBOMAttestation@1`
 - `StellaOps.ScanResults@1`
- `StellaOps.PolicyEvaluation@1`
- `StellaOps.VEXAttestation@1`
- `StellaOps.RiskProfileEvidence@1`
+- `StellaOps.PolicyEvaluation@1`
+- `StellaOps.VEXAttestation@1`
+- `StellaOps.RiskProfileEvidence@1`
+
+Each predicate embeds subject digests, issuer metadata, policy context, materials, and optional transparency hints. Unsupported predicates return `422 predicate_unsupported`.
+
+> **Golden fixtures:** Deterministic JSON statements for each predicate live in `src/Attestor/StellaOps.Attestor.Types/samples`. They are kept stable by the `StellaOps.Attestor.Types.Tests` project so downstream docs and contracts can rely on them without drifting.

-Each predicate embeds subject digests, issuer metadata, policy context, materials, and optional transparency hints. Unsupported predicates return `422 predicate_unsupported`.
-
-### Envelope & signature model
- DSSE envelopes canonicalised (stable JSON ordering) prior to hashing.
- Signature modes: keyless (Fulcio cert chain), keyful (KMS/HSM), hardware (FIDO2/WebAuthn). Multiple signatures allowed.
- Rekor entry stores bundle hash, certificate chain, and optional witness endorsements.
- Archive CAS retains original envelope plus metadata for offline verification.
+### Envelope & signature model
+- DSSE envelopes canonicalised (stable JSON ordering) prior to hashing.
+- Signature modes: keyless (Fulcio cert chain), keyful (KMS/HSM), hardware (FIDO2/WebAuthn). Multiple signatures allowed.
+- Rekor entry stores bundle hash, certificate chain, and optional witness endorsements.
+- Archive CAS retains original envelope plus metadata for offline verification.
+- Envelope serializer emits **compact** (canonical, minified) and **expanded** (annotated, indented) JSON variants off the same canonical byte stream so hashing stays deterministic while humans get context.
+- Payload handling supports **optional compression** (`gzip`, `brotli`) with compression metadata recorded in the expanded view and digesting always performed over the uncompressed bytes.
+- Expanded envelopes surface **detached payload references** (URI, digest, media type, size) so large artifacts can live in CAS/object storage while the canonical payload remains embedded for verification.
+- Payload previews auto-render JSON or UTF-8 text in the expanded output to simplify triage in air-gapped and offline review flows.

 ### Verification pipeline overview
 1. Fetch envelope (from request, cache, or storage) and validate DSSE structure.
@@ -151,11 +157,53 @@ Indexes:

 ## 4) APIs

-### 4.1 Submission
-
-`POST /api/v1/rekor/entries`  *(mTLS + OpTok required)*
-
-* **Body**: as above.
+### 4.1 Signing
+
+`POST /api/v1/attestations:sign` *(mTLS + OpTok required)*
+
+* **Purpose**: Deterministically wrap Stella Ops payloads in DSSE envelopes before Rekor submission. Reuses the submission rate limiter and honours caller tenancy/audience scopes.
+* **Body**:
+
+  ```json
+  {
+    "keyId": "signing-key-id",
+    "payloadType": "application/vnd.in-toto+json",
+    "payload": "<base64 payload>",
+    "mode": "keyless|keyful|kms",
+    "certificateChain": ["-----BEGIN CERTIFICATE-----..."],
+    "artifact": {
+      "sha256": "<subject sha256>",
+      "kind": "sbom|report|vex-export",
+      "imageDigest": "sha256:...",
+      "subjectUri": "oci://..."
+    },
+    "logPreference": "primary|mirror|both",
+    "archive": true
+  }
+  ```
+
+* **Behaviour**:
+  * Resolve the signing key from `attestor.signing.keys[]` (includes algorithm, provider, and optional KMS version).
+  * Compute DSSE pre‑authentication encoding, sign with the resolved provider (default EC, BouncyCastle Ed25519, or File‑KMS ES256), and add static + request certificate chains.
+  * Canonicalise the resulting bundle, derive `bundleSha256`, and mirror the request meta shape used by `/api/v1/rekor/entries`.
+  * Emit `attestor.sign_total{result,algorithm,provider}` and `attestor.sign_latency_seconds{algorithm,provider}` metrics and append an audit row (`action=sign`).
+* **Response 200**:
+
+  ```json
+  {
+    "bundle": { "dsse": { "payloadType": "...", "payload": "...", "signatures": [{ "keyid": "signing-key-id", "sig": "..." }] }, "certificateChain": ["..."], "mode": "kms" },
+    "meta": { "artifact": { "sha256": "...", "kind": "sbom" }, "bundleSha256": "...", "logPreference": "primary", "archive": true },
+    "key": { "keyId": "signing-key-id", "algorithm": "ES256", "mode": "kms", "provider": "kms", "signedAt": "2025-11-01T12:34:56Z" }
+  }
+  ```
+
+* **Errors**: `400 key_not_found`, `400 payload_missing|payload_invalid_base64|artifact_sha_missing`, `400 mode_not_allowed`, `403 client_certificate_required`, `401 invalid_token`, `500 signing_failed`.
+
+### 4.2 Submission
+
+`POST /api/v1/rekor/entries`  *(mTLS + OpTok required)*
+
+* **Body**: as above.
 * **Behavior**:

  * Verify caller (mTLS + OpTok).
@@ -178,16 +226,16 @@ Indexes:
    "status": "included"
  }
  ```
-* **Errors**: `401 invalid_token`, `403 not_signer|chain_untrusted`, `409 duplicate_bundle` (with existing `uuid`), `502 rekor_unavailable`, `504 proof_timeout`.
-
-### 4.2 Proof retrieval
-
-`GET /api/v1/rekor/entries/{uuid}`
+* **Errors**: `401 invalid_token`, `403 not_signer|chain_untrusted`, `409 duplicate_bundle` (with existing `uuid`), `502 rekor_unavailable`, `504 proof_timeout`.
+
+### 4.3 Proof retrieval
+
+`GET /api/v1/rekor/entries/{uuid}`

 * Returns `entries` row (refreshes proof from Rekor if stale/missing).
 * Accepts `?refresh=true` to force backend query.

-### 4.3 Verification (third‑party or internal)
+### 4.4 Verification (third‑party or internal)

 `POST /api/v1/rekor/verify`

@@ -202,17 +250,28 @@ Indexes:
  1. **Bundle signature** → cert chain to Fulcio/KMS roots configured.
  2. **Inclusion proof** → recompute leaf hash; verify Merkle path against checkpoint root.
  3. Optionally verify **checkpoint** against local trust anchors (if Rekor signs checkpoints).
-  4. Confirm **subject.digest** matches caller‑provided hash (when given).
+  4. Confirm **subject.digest** matches caller‑provided hash (when given).
+  5. Fetch **transparency witness** statement when enabled; cache results and downgrade status to WARN when endorsements are missing or mismatched.

-* **Response**:
-
-  ```json
-  { "ok": true, "uuid": "…", "index": 123, "logURL": "…", "checkedAt": "…" }
-  ```
-
-### 4.4 Batch submission (optional)
-
-`POST /api/v1/rekor/batch` accepts an array of submission objects; processes with per‑item results.
+* **Response**:
+
+  ```json
+  { "ok": true, "uuid": "…", "index": 123, "logURL": "…", "checkedAt": "…" }
+  ```
+
+### 4.5 Bulk verification
+
+`POST /api/v1/rekor/verify:bulk` enqueues a verification job containing up to `quotas.bulk.maxItemsPerJob` items. Each item mirrors the single verification payload (uuid | artifactSha256 | subject+envelopeId, optional policyVersion/refreshProof). The handler persists a MongoDB job document (`bulk_jobs` collection) and returns `202 Accepted` with a job descriptor and polling URL.
+
+`GET /api/v1/rekor/verify:bulk/{jobId}` returns progress and per-item results (subject/uuid, status, issues, cached verification report if available). Jobs are tenant- and subject-scoped; only the initiating principal can read their progress.
+
+**Worker path:** `BulkVerificationWorker` claims queued jobs (`status=queued → running`), executes items sequentially through the cached verification service, updates progress counters, and records metrics:
+
+- `attestor.bulk_jobs_total{status}` – completed/failed jobs
+- `attestor.bulk_job_duration_seconds{status}` – job runtime
+- `attestor.bulk_items_total{status}` – per-item outcomes (`succeeded`, `verification_failed`, `exception`)
+
+The worker honours `bulkVerification.itemDelayMilliseconds` for throttling and reschedules persistence conflicts with optimistic version checks. Results hydrate the verification cache; failed items record the error reason without aborting the overall job.

 ---

@@ -244,8 +303,10 @@ Indexes:
  * `subject.digest.sha256` values must be present and well‑formed (hex).
 * **No public submission** path. **Never** accept bundles from untrusted clients.
 * **Client certificate allowlists**: optional `security.mtls.allowedSubjects` / `allowedThumbprints` tighten peer identity checks beyond CA pinning.
-* **Rate limits**: token-bucket per caller derived from `quotas.perCaller` (QPS/burst) returns `429` + `Retry-After` when exceeded.
-* **Redaction**: Attestor never logs secret material; DSSE payloads **should** be public by design (SBOMs/reports). If customers require redaction, enforce policy at Signer (predicate minimization) **before** Attestor.
+* **Rate limits**: token-bucket per caller derived from `quotas.perCaller` (QPS/burst) returns `429` + `Retry-After` when exceeded.
+* **Scope enforcement**: API separates `attestor.write`, `attestor.verify`, and `attestor.read` policies; verification/list endpoints accept read or verify scopes while submission endpoints remain write-only.
+* **Request hygiene**: JSON content-type is mandatory (415 returned otherwise); DSSE payloads are capped (default 2 MiB), certificate chains limited to six entries, and signatures to six per envelope to mitigate parsing abuse.
+* **Redaction**: Attestor never logs secret material; DSSE payloads **should** be public by design (SBOMs/reports). If customers require redaction, enforce policy at Signer (predicate minimization) **before** Attestor.

 ---

@@ -268,24 +329,32 @@ Indexes:

 ## 8) Observability & audit

-**Metrics** (Prometheus):
-
-* `attestor.submit_total{result,backend}`
-* `attestor.submit_latency_seconds{backend}`
-* `attestor.proof_fetch_total{result}`
-* `attestor.verify_total{result}`
-* `attestor.dedupe_hits_total`
-* `attestor.errors_total{type}`
-
-**Correlation**:
-
-* HTTP callers may supply `X-Correlation-Id`; Attestor will echo the header and push `CorrelationId` into the log scope for cross-service tracing.
-
-**Tracing**:
-
-* Spans: `validate`, `rekor.submit`, `rekor.poll`, `persist`, `archive`, `verify`.
-
-**Audit**:
+**Metrics** (Prometheus):
+
+* `attestor.sign_total{result,algorithm,provider}`
+* `attestor.sign_latency_seconds{algorithm,provider}`
+* `attestor.submit_total{result,backend}`
+* `attestor.submit_latency_seconds{backend}`
+* `attestor.proof_fetch_total{subject,issuer,policy,result,attestor.log.backend}`
+* `attestor.verify_total{subject,issuer,policy,result}`
+* `attestor.verify_latency_seconds{subject,issuer,policy,result}`
+* `attestor.dedupe_hits_total`
+* `attestor.errors_total{type}`
+
+SLO guardrails:
+
+* `attestor.verify_latency_seconds` P95 ≤ 2 s per policy.
+* `attestor.verify_total{result="failed"}` ≤ 1 % of `attestor.verify_total` over 30 min rolling windows.
+
+**Correlation**:
+
+* HTTP callers may supply `X-Correlation-Id`; Attestor will echo the header and push `CorrelationId` into the log scope for cross-service tracing.
+
+**Tracing**:
+
+* Spans: `attestor.sign`, `validate`, `rekor.submit`, `rekor.poll`, `persist`, `archive`, `attestor.verify`, `attestor.verify.refresh_proof`.
+
+**Audit**:

 * Immutable `audit` rows (ts, caller, action, hashes, uuid, index, backend, result, latency).

@@ -296,20 +365,45 @@ Indexes:
 ```yaml
 attestor:
  listen: "https://0.0.0.0:8444"
-  security:
-    mtls:
-      caBundle: /etc/ssl/signer-ca.pem
-      requireClientCert: true
-    authority:
-      issuer: "https://authority.internal"
-      jwksUrl: "https://authority.internal/jwks"
-      requireSenderConstraint: "dpop"   # or "mtls"
-    signerIdentity:
-      mode: ["keyless","kms"]
-      fulcioRoots: ["/etc/fulcio/root.pem"]
-      allowedSANs: ["urn:stellaops:signer"]
-      kmsKeys: ["kms://cluster-kms/stellaops-signer"]
-  rekor:
+  security:
+    mtls:
+      caBundle: /etc/ssl/signer-ca.pem
+      requireClientCert: true
+    authority:
+      issuer: "https://authority.internal"
+      jwksUrl: "https://authority.internal/jwks"
+      requireSenderConstraint: "dpop"   # or "mtls"
+    signerIdentity:
+      mode: ["keyless","kms"]
+      fulcioRoots: ["/etc/fulcio/root.pem"]
+      allowedSANs: ["urn:stellaops:signer"]
+      kmsKeys: ["kms://cluster-kms/stellaops-signer"]
+    submissionLimits:
+      maxPayloadBytes: 2097152
+      maxCertificateChainEntries: 6
+      maxSignatures: 6
+  signing:
+    preferredProviders: ["kms","bouncycastle.ed25519","default"]
+    kms:
+      enabled: true
+      rootPath: "/var/lib/stellaops/kms"
+      password: "${ATTESTOR_KMS_PASSWORD}"
+    keys:
+      - keyId: "kms-primary"
+        algorithm: ES256
+        mode: kms
+        provider: "kms"
+        providerKeyId: "kms-primary"
+        kmsVersionId: "v1"
+      - keyId: "ed25519-offline"
+        algorithm: Ed25519
+        mode: keyful
+        provider: "bouncycastle.ed25519"
+        materialFormat: base64
+        materialPath: "/etc/stellaops/keys/ed25519.key"
+        certificateChain:
+          - "-----BEGIN CERTIFICATE-----...-----END CERTIFICATE-----"
+  rekor:
    primary:
      url: "https://rekor-v2.internal"
      proofTimeoutMs: 15000
@@ -328,13 +422,20 @@ attestor:
    objectLock: "governance"
  redis:
    url: "redis://redis:6379/2"
-  quotas:
-    perCaller:
-      qps: 50
-      burst: 100
-```
-
---
+  quotas:
+    perCaller:
+      qps: 50
+      burst: 100
+```
+
+**Notes:**
+
+* `signing.preferredProviders` defines the resolution order when multiple providers support the requested algorithm. Omit to fall back to registration order.
+* File-backed KMS (`signing.kms`) is required when at least one key uses `mode: kms`; the password should be injected via secret store or environment.
+* For keyful providers, supply inline `material` or `materialPath` plus `materialFormat` (`pem` (default), `base64`, or `hex`). KMS keys ignore these fields and require `kmsVersionId`.
+* `certificateChain` entries are appended to returned bundles so offline verifiers do not need to dereference external stores.
+
+---

 ## 10) End‑to‑end sequences