old sprints work, new sprints for exposing functionality via cli, improve code_of_conduct and other agents instructions

This commit is contained in:
master
2026-01-15 18:37:59 +02:00
parent c631bacee2
commit 88a85cdd92
208 changed files with 32271 additions and 2287 deletions

View File

@@ -1,20 +1,20 @@
# component_architecture_attestor.md **StellaOps Attestor** (2025Q4)
# component_architecture_attestor.md — **Stella Ops Attestor** (2025Q4)
> Derived from Epic19 Attestor Console with provenance hooks aligned to the Export Center bundle workflows scoped in Epic10.
> Derived from Epic 19 – Attestor Console with provenance hooks aligned to the Export Center bundle workflows scoped in Epic 10.
> **Scope.** Implementationready architecture for the **Attestor**: the service that **submits** DSSE envelopes to **Rekor v2**, retrieves/validates inclusion proofs, caches results, and exposes verification APIs. It accepts DSSE **only** from the **Signer** over mTLS, enforces chainoftrust to StellaOps roots, and returns `{uuid, index, proof, logURL}` to calling services (Scanner.WebService for SBOMs; backend for final reports; Excititor exports when configured).
> **Scope.** Implementation‑ready architecture for the **Attestor**: the service that **submits** DSSE envelopes to **Rekor v2**, retrieves/validates inclusion proofs, caches results, and exposes verification APIs. It accepts DSSE **only** from the **Signer** over mTLS, enforces chainâ€ofâ€trust to Stella Ops roots, and returns `{uuid, index, proof, logURL}` to calling services (Scanner.WebService for SBOMs; backend for final reports; Excititor exports when configured).
---
## 0) Mission & boundaries
**Mission.** Turn a signed DSSE envelope from the Signer into a **transparencylogged, verifiable fact** with a durable, replayable proof (Merkle inclusion + (optional) checkpoint anchoring). Provide **fast verification** for downstream consumers and a stable retrieval interface for UI/CLI.
**Mission.** Turn a signed DSSE envelope from the Signer into a **transparency‑logged, verifiable fact** with a durable, replayable proof (Merkle inclusion + (optional) checkpoint anchoring). Provide **fast verification** for downstream consumers and a stable retrieval interface for UI/CLI.
**Boundaries.**
* Attestor **does not sign**; it **must not** accept unsigned or thirdpartysigned bundles.
* Attestor **does not sign**; it **must not** accept unsigned or third‑party‑signed bundles.
* Attestor **does not decide PASS/FAIL**; it logs attestations for SBOMs, reports, and export artifacts.
* Rekor v2 backends may be **local** (selfhosted) or **remote**; Attestor handles both with retries, backoff, and idempotency.
* Rekor v2 backends may be **local** (self‑hosted) or **remote**; Attestor handles both with retries, backoff, and idempotency.
---
@@ -24,22 +24,22 @@
**Dependencies:**
* **Signer** (caller) authenticated via **mTLS** and **Authority** OpToks.
* **Rekor v2** tilebacked transparency log endpoint(s).
* **RustFS (S3-compatible)** optional archive store for DSSE envelopes & verification bundles.
* **PostgreSQL** local cache of `{uuid, index, proof, artifactSha256, bundleSha256}`; job state; audit.
* **Valkey** dedupe/idempotency keys and shortlived ratelimit buckets.
* **Licensing Service (optional)** — “endorse call for crosslog publishing when customer optsin.
* **Signer** (caller) — authenticated via **mTLS** and **Authority** OpToks.
* **Rekor v2** — tile‑backed transparency log endpoint(s).
* **RustFS (S3-compatible)** — optional archive store for DSSE envelopes & verification bundles.
* **PostgreSQL** — local cache of `{uuid, index, proof, artifactSha256, bundleSha256}`; job state; audit.
* **Valkey** — dedupe/idempotency keys and short‑lived rate‑limit buckets.
* **Licensing Service (optional)** — “endorse” call for cross‑log publishing when customer opts‑in.
Trust boundary: **Only the Signer** is allowed to call submission endpoints; enforced by **mTLS peer cert allowlist** + `aud=attestor` OpTok.
---
### Roles, identities & scopes
- **Subjects** immutable digests for artifacts (container images, SBOMs, reports) referenced in DSSE envelopes.
- **Issuers** authenticated builders/scanners/policy engines signing evidence; tracked with mode (`keyless`, `kms`, `hsm`, `fido2`) and tenant scope.
- **Consumers** Scanner, Export Center, CLI, Console, Policy Engine that verify proofs using Attestor APIs.
- **Authority scopes** `attestor.write`, `attestor.verify`, `attestor.read`, and administrative scopes for key management; all calls mTLS/DPoP-bound.
- **Subjects** — immutable digests for artifacts (container images, SBOMs, reports) referenced in DSSE envelopes.
- **Issuers** — authenticated builders/scanners/policy engines signing evidence; tracked with mode (`keyless`, `kms`, `hsm`, `fido2`) and tenant scope.
- **Consumers** — Scanner, Export Center, CLI, Console, Policy Engine that verify proofs using Attestor APIs.
- **Authority scopes** — `attestor.write`, `attestor.verify`, `attestor.read`, and administrative scopes for key management; all calls mTLS/DPoP-bound.
### Supported predicate types
- `StellaOps.BuildProvenance@1`
@@ -75,9 +75,9 @@ Each predicate embeds subject digests, issuer metadata, policy context, material
The Attestor implements RFC 6962-compliant Merkle inclusion proof verification for Rekor transparency log entries:
**Components:**
- `MerkleProofVerifier` Verifies Merkle audit paths per RFC 6962 Section 2.1.1
- `CheckpointSignatureVerifier` Parses and verifies Rekor checkpoint signatures (ECDSA/Ed25519)
- `RekorVerificationOptions` Configuration for public keys, offline mode, and checkpoint caching
- `MerkleProofVerifier` — Verifies Merkle audit paths per RFC 6962 Section 2.1.1
- `CheckpointSignatureVerifier` — Parses and verifies Rekor checkpoint signatures (ECDSA/Ed25519)
- `RekorVerificationOptions` — Configuration for public keys, offline mode, and checkpoint caching
**Verification Flow:**
1. Parse checkpoint body (origin, tree size, root hash)
@@ -92,10 +92,10 @@ The Attestor implements RFC 6962-compliant Merkle inclusion proof verification f
- `AllowOfflineWithoutSignature` for fully disconnected scenarios (reduced security)
**Metrics:**
- `attestor.rekor_inclusion_verify_total` Verification attempts by result
- `attestor.rekor_checkpoint_verify_total` Checkpoint signature verifications
- `attestor.rekor_offline_verify_total` Offline mode verifications
- `attestor.rekor_checkpoint_cache_hits/misses` Checkpoint cache performance
- `attestor.rekor_inclusion_verify_total` — Verification attempts by result
- `attestor.rekor_checkpoint_verify_total` — Checkpoint signature verifications
- `attestor.rekor_offline_verify_total` — Offline mode verifications
- `attestor.rekor_checkpoint_cache_hits/misses` — Checkpoint cache performance
### UI & CLI touchpoints
- Console: Evidence browser, verification report, chain-of-custody graph, issuer/key management, attestation workbench, bulk verification views.
@@ -103,9 +103,9 @@ The Attestor implements RFC 6962-compliant Merkle inclusion proof verification f
- SDKs expose sign/verify primitives for build pipelines.
### Performance & observability targets
- Throughput goal: ≥1000 envelopes/minute per worker with cached verification.
- Throughput goal: ≥1 000 envelopes/minute per worker with cached verification.
- Metrics: `attestor_submission_total`, `attestor_verify_seconds`, `attestor_rekor_latency_seconds`, `attestor_cache_hit_ratio`.
- Logs include `tenant`, `issuer`, `subjectDigest`, `rekorUuid`, `proofStatus`; traces cover submission Rekor cache response path.
- Logs include `tenant`, `issuer`, `subjectDigest`, `rekorUuid`, `proofStatus`; traces cover submission → Rekor → cache → response path.
---
@@ -171,8 +171,8 @@ Database: `attestor`
Indexes:
* `entries`: indexes on `artifact_sha256`, `bundle_sha256`, `created_at`, and composite `(status, created_at DESC)`.
* `dedupe`: unique index on `key`; scheduled job cleans rows where `ttl_at < NOW()` (2448h retention).
* `audit`: index on `ts` for timerange queries.
* `dedupe`: unique index on `key`; scheduled job cleans rows where `ttl_at < NOW()` (24–48h retention).
* `audit`: index on `ts` for time‑range queries.
---
@@ -330,10 +330,10 @@ SBOM-to-component linkage metadata.
**Attestor accepts only** DSSE envelopes that satisfy all of:
1. **mTLS** peer certificate maps to `signer` service (CApinned).
1. **mTLS** peer certificate maps to `signer` service (CA‑pinned).
2. **Authority** OpTok with `aud=attestor`, `scope=attestor.write`, DPoP or mTLS bound.
3. DSSE envelope is **signed by the Signers key** (or includes a **Fulcioissued** cert chain) and **chains to configured roots** (Fulcio/KMS).
4. **Predicate type** is one of StellaOps types (sbom/report/vexexport) with valid schema.
3. DSSE envelope is **signed by the Signer’s key** (or includes a **Fulcio‑issued** cert chain) and **chains to configured roots** (Fulcio/KMS).
4. **Predicate type** is one of Stella Ops types (sbom/report/vex‑export) with valid schema.
5. `subject[*].digest.sha256` is present and canonicalized.
**Wire shape (JSON):**
@@ -360,7 +360,7 @@ SBOM-to-component linkage metadata.
`POST /api/v1/attestations:sign` *(mTLS + OpTok required)*
* **Purpose**: Deterministically wrap StellaOps payloads in DSSE envelopes before Rekor submission. Reuses the submission rate limiter and honours caller tenancy/audience scopes.
* **Purpose**: Deterministically wrap Stella Ops payloads in DSSE envelopes before Rekor submission. Reuses the submission rate limiter and honours caller tenancy/audience scopes.
* **Body**:
```json
@@ -383,7 +383,7 @@ SBOM-to-component linkage metadata.
* **Behaviour**:
* Resolve the signing key from `attestor.signing.keys[]` (includes algorithm, provider, and optional KMS version).
* Compute DSSE preauthentication encoding, sign with the resolved provider (default EC, BouncyCastle Ed25519, or FileKMS ES256), and add static + request certificate chains.
* Compute DSSE pre‑authentication encoding, sign with the resolved provider (default EC, BouncyCastle Ed25519, or File‑KMS ES256), and add static + request certificate chains.
* Canonicalise the resulting bundle, derive `bundleSha256`, and mirror the request meta shape used by `/api/v1/rekor/entries`.
* Emit `attestor.sign_total{result,algorithm,provider}` and `attestor.sign_latency_seconds{algorithm,provider}` metrics and append an audit row (`action=sign`).
* **Response 200**:
@@ -415,13 +415,13 @@ SBOM-to-component linkage metadata.
```json
{
"uuid": "",
"uuid": "…",
"index": 123456,
"proof": {
"checkpoint": { "origin": "rekor@site", "size": 987654, "rootHash": "", "timestamp": "" },
"inclusion": { "leafHash": "", "path": ["…","…"] }
"checkpoint": { "origin": "rekor@site", "size": 987654, "rootHash": "…", "timestamp": "…" },
"inclusion": { "leafHash": "…", "path": ["…","…"] }
},
"logURL": "https://rekor/api/v2/log//entries/",
"logURL": "https://rekor…/api/v2/log/…/entries/…",
"status": "included"
}
```
@@ -434,28 +434,28 @@ SBOM-to-component linkage metadata.
* Returns `entries` row (refreshes proof from Rekor if stale/missing).
* Accepts `?refresh=true` to force backend query.
### 4.4 Verification (thirdparty or internal)
### 4.4 Verification (third‑party or internal)
`POST /api/v1/rekor/verify`
* **Body** (one of):
* `{ "uuid": "" }`
* `{ "bundle": { DSSE } }`
* `{ "artifactSha256": "" }` *(looks up most recent entry)*
* `{ "uuid": "…" }`
* `{ "bundle": { …DSSE… } }`
* `{ "artifactSha256": "…" }` *(looks up most recent entry)*
* **Checks**:
1. **Bundle signature** cert chain to Fulcio/KMS roots configured.
2. **Inclusion proof** recompute leaf hash; verify Merkle path against checkpoint root.
1. **Bundle signature** → cert chain to Fulcio/KMS roots configured.
2. **Inclusion proof** → recompute leaf hash; verify Merkle path against checkpoint root.
3. Optionally verify **checkpoint** against local trust anchors (if Rekor signs checkpoints).
4. Confirm **subject.digest** matches callerprovided hash (when given).
4. Confirm **subject.digest** matches caller‑provided hash (when given).
5. Fetch **transparency witness** statement when enabled; cache results and downgrade status to WARN when endorsements are missing or mismatched.
* **Response**:
```json
{ "ok": true, "uuid": "", "index": 123, "logURL": "", "checkedAt": "" }
{ "ok": true, "uuid": "…", "index": 123, "logURL": "…", "checkedAt": "…" }
```
### 4.5 Bulk verification
@@ -464,11 +464,11 @@ SBOM-to-component linkage metadata.
`GET /api/v1/rekor/verify:bulk/{jobId}` returns progress and per-item results (subject/uuid, status, issues, cached verification report if available). Jobs are tenant- and subject-scoped; only the initiating principal can read their progress.
**Worker path:** `BulkVerificationWorker` claims queued jobs (`status=queued running`), executes items sequentially through the cached verification service, updates progress counters, and records metrics:
**Worker path:** `BulkVerificationWorker` claims queued jobs (`status=queued → running`), executes items sequentially through the cached verification service, updates progress counters, and records metrics:
- `attestor.bulk_jobs_total{status}` completed/failed jobs
- `attestor.bulk_job_duration_seconds{status}` job runtime
- `attestor.bulk_items_total{status}` per-item outcomes (`succeeded`, `verification_failed`, `exception`)
- `attestor.bulk_jobs_total{status}` – completed/failed jobs
- `attestor.bulk_job_duration_seconds{status}` – job runtime
- `attestor.bulk_items_total{status}` – per-item outcomes (`succeeded`, `verification_failed`, `exception`)
The worker honours `bulkVerification.itemDelayMilliseconds` for throttling and reschedules persistence conflicts with optimistic version checks. Results hydrate the verification cache; failed items record the error reason without aborting the overall job.
@@ -478,7 +478,7 @@ The worker honours `bulkVerification.itemDelayMilliseconds` for throttling and r
* **Canonicalization**: DSSE envelopes are **normalized** (stable JSON ordering, no insignificant whitespace) before hashing and submission.
* **Transport**: HTTP/2 with retries (exponential backoff, jitter), budgeted timeouts.
* **Idempotency**: if backend returns already exists, map to existing `uuid`.
* **Idempotency**: if backend returns “already exists,” map to existing `uuid`.
* **Proof acquisition**:
* In synchronous mode, poll the log for inclusion up to `proofTimeoutMs`.
@@ -486,25 +486,25 @@ The worker honours `bulkVerification.itemDelayMilliseconds` for throttling and r
* **Mirrors/dual logs**:
* When `logPreference="both"`, submit to primary and mirror; store **both** UUIDs (primary canonical).
* Optional **cloud endorsement**: POST to the StellaOps cloud `/attest/endorse` with `{uuid, artifactSha256}`; store returned endorsement id.
* Optional **cloud endorsement**: POST to the Stella Ops cloud `/attest/endorse` with `{uuid, artifactSha256}`; store returned endorsement id.
---
## 6) Security model
* **mTLS required** for submission from **Signer** (CApinned).
* **mTLS required** for submission from **Signer** (CA‑pinned).
* **Authority token** with `aud=attestor` and DPoP/mTLS binding must be presented; Attestor verifies both.
* **Bundle acceptance policy**:
* DSSE signature must chain to the configured **Fulcio** (keyless) or **KMS/HSM** roots.
* SAN (Subject Alternative Name) must match **Signer identity** policy (e.g., `urn:stellaops:signer` or pinned OIDC issuer).
* Predicate `predicateType` must be on allowlist (sbom/report/vex-export).
* `subject.digest.sha256` values must be present and wellformed (hex).
* `subject.digest.sha256` values must be present and well‑formed (hex).
* **No public submission** path. **Never** accept bundles from untrusted clients.
* **Client certificate allowlists**: optional `security.mtls.allowedSubjects` / `allowedThumbprints` tighten peer identity checks beyond CA pinning.
* **Rate limits**: token-bucket per caller derived from `quotas.perCaller` (QPS/burst) returns `429` + `Retry-After` when exceeded.
* **Scope enforcement**: API separates `attestor.write`, `attestor.verify`, and `attestor.read` policies; verification/list endpoints accept read or verify scopes while submission endpoints remain write-only.
* **Request hygiene**: JSON content-type is mandatory (415 returned otherwise); DSSE payloads are capped (default 2MiB), certificate chains limited to six entries, and signatures to six per envelope to mitigate parsing abuse.
* **Request hygiene**: JSON content-type is mandatory (415 returned otherwise); DSSE payloads are capped (default 2 MiB), certificate chains limited to six entries, and signatures to six per envelope to mitigate parsing abuse.
* **Redaction**: Attestor never logs secret material; DSSE payloads **should** be public by design (SBOMs/reports). If customers require redaction, enforce policy at Signer (predicate minimization) **before** Attestor.
---
@@ -542,8 +542,8 @@ The worker honours `bulkVerification.itemDelayMilliseconds` for throttling and r
SLO guardrails:
* `attestor.verify_latency_seconds` P95 2s per policy.
* `attestor.verify_total{result="failed"}` 1% of `attestor.verify_total` over 30min rolling windows.
* `attestor.verify_latency_seconds` P95 ≤ 2 s per policy.
* `attestor.verify_total{result="failed"}` ≤ 1 % of `attestor.verify_total` over 30 min rolling windows.
**Correlation**:
@@ -636,7 +636,7 @@ attestor:
---
## 10) Endtoend sequences
## 10) Endâ€toâ€end sequences
**A) Submit & include (happy path)**
@@ -695,19 +695,19 @@ sequenceDiagram
* Stateless; scale horizontally.
* **Targets**:
* Submit+proof P95 **300ms** (warm log; local Rekor).
* Verify P95 **30ms** from cache; **120ms** with live proof fetch.
* Submit+proof P95 ≤ **300 ms** (warm log; local Rekor).
* Verify P95 ≤ **30 ms** from cache; ≤ **120 ms** with live proof fetch.
* 1k submissions/minute per replica sustained.
* **Hot caches**: `dedupe` (bundle hash uuid), recent `entries` by artifact sha256.
* **Hot caches**: `dedupe` (bundle hash → uuid), recent `entries` by artifact sha256.
---
## 13) Testing matrix
* **Happy path**: valid DSSE, inclusion within timeout.
* **Idempotency**: resubmit same `bundleSha256` same `uuid`.
* **Security**: reject nonSigner mTLS, wrong `aud`, DPoP replay, untrusted cert chain, forbidden predicateType.
* **Rekor variants**: promisethenproof, proof delayed, mirror dualsubmit, mirror failure.
* **Idempotency**: resubmit same `bundleSha256` → same `uuid`.
* **Security**: reject non‑Signer mTLS, wrong `aud`, DPoP replay, untrusted cert chain, forbidden predicateType.
* **Rekor variants**: promise‑then‑proof, proof delayed, mirror dual‑submit, mirror failure.
* **Verification**: corrupt leaf path, wrong root, tampered bundle.
* **Throughput**: soak test with 10k submissions; latency SLOs, zero drops.
@@ -718,16 +718,16 @@ sequenceDiagram
* Language: **.NET 10** minimal API; `HttpClient` with **sockets handler** tuned for HTTP/2.
* JSON: **canonical writer** for DSSE payload hashing.
* Crypto: use **BouncyCastle**/**System.Security.Cryptography**; PEM parsing for cert chains.
* Rekor client: pluggable driver; treat backend errors as retryable/nonretryable with granular mapping.
* Safety: size caps on bundles; decompress bombs guarded; strict UTF8.
* Rekor client: pluggable driver; treat backend errors as retryable/non‑retryable with granular mapping.
* Safety: size caps on bundles; decompress bombs guarded; strict UTF‑8.
* CLI integration: `stellaops verify attestation <uuid|bundle|artifact>` calls `/rekor/verify`.
---
## 15) Optional features
* **Duallog** write (primary + mirror) and **crosslog proof** packaging.
* **Cloud endorsement**: send `{uuid, artifactSha256}` to StellaOps cloud; store returned endorsement id for marketing/chainofcustody.
* **Dual‑log** write (primary + mirror) and **cross‑log proof** packaging.
* **Cloud endorsement**: send `{uuid, artifactSha256}` to Stella Ops cloud; store returned endorsement id for marketing/chainâ€ofâ€custody.
* **Checkpoint pinning**: periodically pin latest Rekor checkpoints to an external audit store for independent monitoring.
---
@@ -739,3 +739,54 @@ sequenceDiagram
- Health endpoints: `/health/liveness`, `/health/readiness`, `/status`; verification probe `/api/attestations/verify` once demo bundle is available (see runbook).
- Alert hints: signing latency > 1s p99, verification failure spikes, tlog submission lag >10s, key rotation age over policy threshold, backlog above configured threshold.
---
## 17) Rekor Entry Events
> Sprint: SPRINT_20260112_007_ATTESTOR_rekor_entry_events
Attestor emits deterministic events when DSSE bundles are logged to Rekor and inclusion proofs become available. These events drive policy reanalysis.
### Event Types
| Event Type | Constant | Description |
|------------|----------|-------------|
| `rekor.entry.logged` | `RekorEventTypes.EntryLogged` | Bundle successfully logged with inclusion proof |
| `rekor.entry.queued` | `RekorEventTypes.EntryQueued` | Bundle queued for logging (async mode) |
| `rekor.entry.inclusion_verified` | `RekorEventTypes.InclusionVerified` | Inclusion proof independently verified |
| `rekor.entry.failed` | `RekorEventTypes.EntryFailed` | Logging or verification failed |
### RekorEntryEvent Schema
```jsonc
{
"eventId": "rekor-evt-sha256:...",
"eventType": "rekor.entry.logged",
"tenant": "default",
"bundleDigest": "sha256:abc123...",
"artifactDigest": "sha256:def456...",
"predicateType": "StellaOps.ScanResults@1",
"rekorEntry": {
"uuid": "24296fb24b8ad77a...",
"logIndex": 123456789,
"logUrl": "https://rekor.sigstore.dev",
"integratedTime": "2026-01-15T10:30:02Z"
},
"reanalysisHints": {
"cveIds": ["CVE-2026-1234"],
"productKeys": ["pkg:npm/lodash@4.17.21"],
"mayAffectDecision": true,
"reanalysisScope": "immediate"
},
"occurredAtUtc": "2026-01-15T10:30:05Z"
}
```
### Offline Mode Behavior
When operating in offline/air-gapped mode:
1. Events are not emitted when Rekor is unreachable
2. Bundles are queued locally for later submission
3. Verification uses bundled checkpoints
4. Events are generated when connectivity is restored