diff --git a/README.md b/README.md index c673de47..08be7469 100755 --- a/README.md +++ b/README.md @@ -25,4 +25,10 @@ Pipeline note: deployment workflows should template `etc/feedser.yaml` during CI injecting environment-specific Mongo credentials and telemetry endpoints. Upcoming releases will add Microsoft OAuth (Entra ID) authentication support—track the quickstart for integration steps once available. + +## Documentation + +- `docs/README.md` now consolidates the platform index and points to the updated high-level architecture. +- Module architecture dossiers live under `docs/ARCHITECTURE_*.md`; the most relevant here are `docs/ARCHITECTURE_FEEDSER.md` (service layout, merge engine, exports) and `docs/ARCHITECTURE_CLI.md` (command surface, AOT packaging, auth flows). Related services such as the Signer, Attestor, Authority, Scanner, UI, Vexer, Zastava, and DevOps pipeline each have their own dossier. +- Offline operation guidance moved to `docs/24_OFFLINE_KIT.md`, which details bundle composition, verification, and delta workflows. Feedser-specific connector operations stay in `docs/ops/feedser-certbund-operations.md` and companion runbooks under `docs/ops/`. diff --git a/docs/07_HIGH_LEVEL_ARCHITECTURE.md b/docs/07_HIGH_LEVEL_ARCHITECTURE.md index fd8d5250..e27f49ee 100755 --- a/docs/07_HIGH_LEVEL_ARCHITECTURE.md +++ b/docs/07_HIGH_LEVEL_ARCHITECTURE.md @@ -1,388 +1,430 @@ -# 7 · High‑Level Architecture — **Stella Ops** +Below is the **revised, consolidated** `high_level_architecture.md`. +It **absorbs** all content from `components.md` so you have a single, authoritative file. No separate components doc is required. --- -## 0 Purpose & Scope +# High‑Level Architecture — **Stella Ops** (Consolidated • 2025Q4) -Give contributors, DevOps engineers and auditors a **complete yet readable map** of the Core: - -* Major runtime components and message paths. -* Where plug‑ins, CLI helpers and runtime agents attach. -* Technology choices that enable the sub‑5 second SBOM goal. -* Typical operational scenarios (pipeline scan, mute, nightly re‑scan, etc.). - -Anything enterprise‑only (signed PDF, custom/regulated TLS, LDAP, enforcement) **must arrive as a plug‑in**; the Core never hard‑codes those concerns. ---- -## 1 Component Overview - -| # | Component | Responsibility | -|---|-----------|---------------| -| 1 | **API Gateway** | REST endpoints (`/scan`, `/quota`, **`/token/offline`**); token auth; quota enforcement | -| 2 | **Scan Service** | SBOM parsing, Delta‑SBOM cache, vulnerability lookup | -| 3 | **Policy Engine** | YAML / (optional) Rego rule evaluation; verdict assembly | -| 4 | **Quota Service** | Per‑token counters; **333 scans/day**; waits & HTTP 429 | -| 5 | **Client‑JWT Issuer** | Issues 30‑day offline tokens; bundles them into OUK | -| 5 | **Registry** | Anonymous internal Docker registry for agents, SBOM uploads | -| 6 | **Web UI** | React/Blazor SPA; dashboards, policy editor, quota banner | -| 7 | **Data Stores** | **Redis** (cache, quota) & **MongoDB** (SBOMs, findings, audit) | -| 8 | **Plugin Host** | Hot‑load .NET DLLs; isolates community plug‑ins | -| 9 | **Agents** | `sbom‑builder`, `Stella CLI` scanner CLI, future `StellaOpsAttestor` | - - -```mermaid -flowchart TD - subgraph "External Actors" - DEV["Developer / DevSecOps / Manager"] - CI["CI/CD Pipeline (e.g., Stella CLI)"] - K8S["Kubernetes Cluster (e.g., Zastava Agent)"] - end - - subgraph "Stella Ops Runtime" - subgraph "Core Services" - CORE["Stella Core
(REST + gRPC APIs, Orchestration)"] - REDIS[("Redis
(Cache, Queues, Trivy DB Mirror)")] - MONGO[("MongoDB
(Optional: Long-term Storage)")] - POL["Mute Policies
(OPA & YAML Evaluator)"] - REG["StellaOps Registry
(Docker Registry v2)"] - ATT["StellaOps Attestor
(SLSA + Rekor)"] - end - - subgraph "Agents & Builders" - SB["SBOM Builder
(Go Binary: Extracts Layers, Generates SBOMs)"] - SA["Stella CLI
(Pipeline Helper: Invokes Builder, Triggers Scans)"] - ZA["Zastava Agent
(K8s Webhook: Enforces Policies, Inventories Containers)"] - end - - subgraph "Scanners & UI" - TRIVY["Trivy Scanner
(Plugin Container: Vulnerability Scanning)"] - UI["Web UI
(Vue3 + Tailwind: Dashboards, Policy Editor)"] - CLI["Stella CLI
(CLI Helper: Triggers Scans, Mutes)"] - end - end - - DEV -->|Browses Findings, Mutes CVEs| UI - DEV -->|Triggers Scans| CLI - CI -->|Generates SBOM, Calls /scan| SA - K8S -->|Inventories Containers, Enforces Gates| ZA - - UI -- "REST" --> CORE - CLI -- "REST/gRPC" --> CORE - SA -->|Scan Requests| CORE - SB -->|Uploads SBOMs| CORE - ZA -->|Policy Gates| CORE - - CORE -- "Queues, Caches" --> REDIS - CORE -- "Persists Data" --> MONGO - CORE -->|Evaluates Policies| POL - CORE -->|Attests Provenance| ATT - CORE -->|Scans Vulnerabilities| TRIVY - - SB -- "Pulls Images" --> REG - SA -- "Pulls Images" --> REG - ZA -- "Pulls Images" --> REG - - style DEV fill:#f9f,stroke:#333 - style CI fill:#f9f,stroke:#333 - style K8S fill:#f9f,stroke:#333 - style CORE fill:#ddf,stroke:#333 - style REDIS fill:#fdd,stroke:#333 - style MONGO fill:#fdd,stroke:#333 - style POL fill:#dfd,stroke:#333 - style REG fill:#dfd,stroke:#333 - style ATT fill:#dfd,stroke:#333 - style SB fill:#fdf,stroke:#333 - style SA fill:#fdf,stroke:#333 - style ZA fill:#fdf,stroke:#333 - style TRIVY fill:#ffd,stroke:#333 - style UI fill:#ffd,stroke:#333 - style CLI fill:#ffd,stroke:#333 -``` - -* **Developer / DevSecOps / Manager** – browses findings, mutes CVEs, triggers scans. -* **Stella CLI** – generates SBOMs and calls `/scan` during CI. -* **Zastava Agent** – inventories live containers; Core ships it in *passive* mode only (no kill). - -### 1.1 Client‑JWT Lifecycle (offline aware) - -1. **Online instance** – user signs in → `/connect/token` issues JWT valid 12 h. -2. **Offline instance** – JWT with `exp ≈ 30 days` ships in OUK; backend - **re‑signs** and stores it during import. -3. Tokens embed a `tier` claim (“Free”) and `maxScansPerDay: 333`. -4. On expiry the UI surfaces a red toast **7 days** in advance. +> **Purpose.** A complete, implementation‑ready map of Stella Ops: product vision, all runtime components, trust boundaries, tokens/licensing, control/data flows, storage, APIs, security, scale, DevOps, and verification logic. +> **Scope.** This file **replaces** the separate `components.md`; all component details now live here. --- -## 2 · Component Responsibilities (runtime view) +## 0) Product vision & principles -| Component | Core Responsibility | Implementation Highlights | -| -------------------------- | ---------------------------------------------------------------------------------------------------------- | --------------------------------------------------------- | -| **Stella Core** | Orchestrates scans, persists SBOM blobs, serves REST/gRPC APIs, fans out jobs to scanners & policy engine. | .NET {{ dotnet }}, CQRS, Redis Streams; pluggable runner interfaces. | -| **SBOM Builder** | Extracts image layers, queries Core for *missing* layers, generates SBOMs (multi‑format), uploads blobs. | Go binary; wraps Trivy & Syft libs. | -| **Stella CLI** | Pipeline‑side helper; invokes Builder, triggers scan, streams progress back to CI/CD. | Static musl build. | -| **Zastava Agent** | K8s admission webhook enforcing policy verdicts before Pod creation. | Rust for sub‑10 ms latencies. | -| **UI** | Angular 17 SPA for dashboards, settings, policy editor. | Tailwind CSS; Webpack module federation (future). | -| **Redis** | Cache, queue, Trivy‑DB mirror, layer diffing. | Single instance or Sentinel. | -| **MongoDB** (opt.) | Long‑term SBOM & policy audit storage (> 180 days). | Optional; enabled via flag. | -| **StellaOps.Registry** | Anonymous read‑only Docker v2 registry with optional Cosign verification. | `registry :2` behind nginx reverse proxy. | -| **StellaOps.MutePolicies** | YAML/Rego evaluator, policy version store, `/policy/*` API. | Embeds OPA‑WASM; falls back to `opa exec`. | -| **StellaOpsAttestor** | Generate SLSA provenance & Rekor signatures; verify on demand. | Side‑car container; DSSE + Rekor CLI. | +**Vision.** Stella Ops is a **deterministic SBOM + VEX platform** for CI/CD and runtime, tuned for **speed** (per‑layer deltas), **quiet output** (usage‑scoped views), and **verifiability** (DSSE + Rekor v2). It is **self‑hostable**, **air‑gap capable**, and **commercially enforceable**: only licensed installations can produce **Stella Ops‑verified** attestations. -All cross‑component calls use dependency‑injected interfaces—no -intra‑component reach‑ins. +**Operating principles.** + +* **Scanner‑owned SBOMs.** We generate our own BOMs; we do not warehouse third‑party SBOM content (we can **link** to attested SBOMs). +* **Deterministic evidence.** Facts come from package DBs, installed metadata, linkers, and verified attestations; no fuzzy guessing in the core. +* **Per‑layer caching.** Cache fragments by **layer digest** and compose image SBOMs via **CycloneDX BOM‑Link** / **SPDX ExternalRef**. +* **Inventory vs Usage.** Always record the full **inventory** of what exists; separately present **usage** (entrypoint closure + loaded libs). +* **Backend decides.** PASS/FAIL is produced by **Policy** + **VEX** + **Advisories**. The scanner reports facts. +* **Attest or it didn’t happen.** Every export is signed as **in‑toto/DSSE** and logged in **Rekor v2**. +* **Sovereign‑ready.** Cloud is used only for licensing and optional endorsement; everything else is first‑party and self‑hostable. --- -## 3 · Principal Backend Modules & Plug‑in Hooks +## 1) Service topology & trust boundaries -| Namespace | Responsibility | Built‑in Tech / Default | Plug‑in Contract | -| --------------- | -------------------------------------------------- | ----------------------- | ------------------------------------------------- | -| `configuration` | Parse env/JSON, health‑check endpoint | .NET {{ dotnet }} Options | `IConfigValidator` | -| `identity` | Embedded OAuth2/OIDC (OpenIddict 6) | MIT OpenIddict | `IIdentityProvider` for LDAP/SAML/JWT gateway | -| `pluginloader` | Discover DLLs, SemVer gate, optional Cosign verify | Reflection + Cosign | `IPluginLifecycleHook` for telemetry | -| `scanning` | SBOM‑ & image‑flow orchestration; runner pool | Trivy CLI (default) | `IScannerRunner` – e.g., Grype, Copacetic, Clair | -| `feedser` (vulnerability ingest/merge/export service) | Nightly NVD merge & feed enrichment | Hangfire job | drop-in `*.Schedule.dll` for OSV, GHSA, NVD 2.0, CNNVD, CNVD, ENISA, JVN and BDU feeds | -| `tls` | TLS provider abstraction | OpenSSL | `ITlsProvider` for custom suites (incl. **SM2**, where law or security requires it) | -| `reporting` | Render HTML/PDF reports | RazorLight | `IReportRenderer` | -| `ui` | Angular SPA & i18n | Angular {{ angular }} | new locales via `/locales/{lang}.json` | -| `scheduling` | Cron + retries | Hangfire | any recurrent job via `*.Schedule.dll` | +### 1.1 Runtime inventory (first‑party) -```mermaid -classDiagram - class configuration - class identity - class pluginloader - class scanning - class feedser - class tls - class reporting - class ui - class scheduling +| Service / Tool | Container image | Core role | Scale pattern | +| ------------------------------- | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------- | +| **Scanner.WebService** | `stellaops/scanner-web` | Control plane for scans; catalog; SBOM composition (inventory & usage); diff; exports. | Stateless; N replicas behind LB. | +| **Scanner.Worker** | `stellaops/scanner-worker` | Runs analyzers (OS, Lang: Java/Node/Python/Go/.NET/Rust, Native ELF/PE/Mach‑O, EntryTrace); emits per‑layer SBOMs and composes image SBOMs. | Horizontal; queue‑driven; sharded by layer digest. | +| **Scanner.Sbomer.BuildXPlugin** | `stellaops/sbom-indexer` | BuildKit **generator** for build‑time SBOMs as OCI **referrers**. | CI‑side; ephemeral. | +| **Scanner.Sbomer.DockerImage** | `stellaops/scanner-cli` | CLI‑orchestrated scanner container for post‑build scans. | Local/CI; ephemeral. | +| **Feedser.WebService** | `stellaops/feedser-web` | Vulnerability ingest/normalize/merge/export (JSON + Trivy DB). | HA via Mongo locks. | +| **Vexer.WebService** | `stellaops/vexer-web` | VEX ingest/normalize/consensus; conflict retention; exports. | HA via Mongo locks. | +| **Policy Engine** | (in `scanner-web`) | YAML DSL evaluator (waivers, vendor preferences, KEV/EPSS, license, usage‑gating); produces **policy digest**. | In‑process; cache per digest. | +| **Signer** | `stellaops/signer` | **Hard gate:** validates entitlement + release integrity; mints signing cert (Fulcio keyless) or uses KMS; signs DSSE. | Stateless; HPA by QPS. | +| **Attestor** | `stellaops/attestor` | Posts DSSE bundles to **Rekor v2**; verification endpoints. | Stateless; HPA by QPS. | +| **Authority** | `stellaops/authority` | On‑prem OIDC issuing **short‑lived OpToks** with DPoP/mTLS sender constraint. | HA behind LB. | +| **Zastava** (Runtime) | `stellaops/zastava` | Runtime inspector/enforcer (observer + optional Admission Webhook). | DaemonSet + Webhook. | +| **Web UI** | `stellaops/ui` | Angular app for scans, diffs, policy, VEX, runtime, reports. | Stateless. | +| **StellaOps.Cli** | `stellaops/cli` | CLI for init/scan/export/diff/policy/report/verify; Buildx helper. | Local/CI. | - class AllModules +### 1.2 Third‑party (self‑hosted) - configuration ..> identity : Uses - identity ..> pluginloader : Authenticates Plugins - pluginloader ..> scanning : Loads Scanner Runners - scanning ..> feedser : Triggers Feed Merges - tls ..> AllModules : Provides TLS Abstraction - reporting ..> ui : Renders Reports for UI - scheduling ..> feedser : Schedules Nightly Jobs +* **Fulcio** (Sigstore CA) — issues short‑lived signing certs (keyless). +* **Rekor v2** (tile‑backed transparency log). +* **MinIO** — S3‑compatible object store with lifecycle & Object Lock. +* **MongoDB** — catalog, advisories, VEX. +* **Queue** — Redis Streams / NATS / RabbitMQ (pluggable). +* **OCI Registry** — must support **Referrers API** (discover SBOMs/signatures). - note for scanning "Pluggable: ISScannerRunner
e.g., Trivy, Grype" - note for feedser "Pluggable: *.Schedule.dll
e.g., OSV, GHSA Feeds" - note for identity "Pluggable: IIdentityProvider
e.g., LDAP, SAML" - note for reporting "Pluggable: IReportRenderer
e.g., Custom PDF" -``` +### 1.3 Cloud licensing (Stella Ops) -**When remaining = 0:** -API returns `429 Too Many Requests`, `Retry‑After: ` (sequence omitted for brevity). +* **Licensing Service** (`www.stella-ops.org`) — issues long‑lived **License Tokens (LT)**; exchanges LT → **Proof‑of‑Entitlement (PoE)** bound to an installation key; revoke/introspect PoE; optional cross‑log **endorsement**. ---- - -## 4 · Data Flows - -### 4.1 SBOM‑First (≤ 5 s P95) - -Builder produces SBOM locally, so Core never touches the Docker -socket. -Trivy path hits ≤ 5 s on alpine:3.19 with warmed DB. -Image‑unpack fallback stays ≤ 10 s for 200 MB images. - -```mermaid -sequenceDiagram - participant CI as CI/CD Pipeline (Stella CLI) - participant SB as SBOM Builder - participant CORE as Stella Core - participant REDIS as Redis Queue - participant RUN as Scanner Runner (e.g., Trivy) - participant POL as Policy Evaluator - - CI->>SB: Invoke SBOM Generation - SB->>CORE: Check Missing Layers (/layers/missing) - CORE->>REDIS: Query Layer Diff (SDIFF) - REDIS-->>CORE: Missing Layers List - CORE-->>SB: Return Missing Layers - SB->>SB: Generate Delta SBOM - SB->>CORE: Upload SBOM Blob (POST /scan(sbom)) - CORE->>REDIS: Enqueue Scan Job - REDIS->>RUN: Fan Out to Runner - RUN->>RUN: Perform Vulnerability Scan - RUN-->>CORE: Return Scan Results - CORE->>POL: Evaluate Mute Policies - POL-->>CORE: Policy Verdict - CORE-->>CI: JSON Verdict & Progress Stream - Note over CORE,CI: Achieves ≤5s P95 with Warmed DB -``` - -### 4.2 Delta SBOM - -Builder collects layer digests. -`POST /layers/missing` → Redis SDIFF → missing layer list (< 20 ms). -SBOM generated only for those layers and uploaded. - -### 4.3 Feedser Harvest & Export - -```mermaid -sequenceDiagram - participant SCHED as Feedser Scheduler - participant CONN as Source Connector Plug-in - participant FEEDSER as Feedser Core - participant MONGO as MongoDB (Canonical Advisories) - participant EXPORT as Exporter (JSON / Trivy DB) - participant ART as Artifact Store / Offline Kit - - SCHED->>CONN: Trigger window (init/resume) - CONN->>CONN: Fetch source documents + metadata - CONN->>FEEDSER: Submit raw document for parsing - FEEDSER->>FEEDSER: Parse & normalize to DTO - FEEDSER->>FEEDSER: Merge & deduplicate canonical advisory - FEEDSER->>MONGO: Write advisory, provenance, merge_event - FEEDSER->>EXPORT: Queue export delta request - EXPORT->>MONGO: Read canonical snapshot/deltas - EXPORT->>EXPORT: Build deterministic JSON & Trivy DB artifacts - EXPORT->>ART: Publish artifacts / Offline Kit bundle - ART-->>FEEDSER: Record export state + digests -``` - -### 4.4 Identity & Auth Flow - -OpenIddict issues JWTs via client‑credentials or password grant. -An IIdentityProvider plug‑in can delegate to LDAP, SAML or external OIDC -without Core changes. ---- -## 5 · Runtime Helpers - -| Helper | Form | Purpose | Extensible Bits | -|-----------|---------------------------------------|--------------------------------------------------------------------|-------------------------------------------| -| **Stella CLI** | Distroless CLI | Generates SBOM, calls `/scan`, honours threshold flag | `--engine`, `--pdf-out` piped to plug‑ins | -| **Zastava** | Static Go binary / DaemonSet | Watches Docker/CRI‑O events; uploads SBOMs; can enforce gate | Policy plug‑in could alter thresholds | - ---- - -## 6 · Persistence & Cache Strategy - -| Store | Primary Use | Why chosen | -|----------------|-----------------------------------------------|--------------------------------| -| **MongoDB** | Feedser canonical advisories, merge events, export state | Deterministic canonical store with flexible schema | -| **Redis 7** | CLI quotas, short-lived job scheduling, layer diff cache | Sub-1 ms P99 latency for hot-path coordination | -| **Local tmpfs**| Trivy layer cache (`/var/cache/trivy`) | Keeps disk I/O off hot path | +### 1.4 Diagram (control/data planes & trust) ```mermaid flowchart LR - subgraph "Persistence Layers" - REDIS[(Redis: Quotas & Short-lived Queues
Sub-1ms P99)] - MONGO[(MongoDB: Canonical Advisories
Merge Events & Export State)] - TMPFS[(Local tmpfs: Trivy Layer Cache
Low I/O Overhead)] - end + subgraph Cloud["www.stella-ops.org (Cloud)"] + LS[Licensing Service
LT→PoE / revoke / introspect] + end - CORE["Stella Core"] -- Queues & SBOM Cache --> REDIS - CORE -- Long-term Storage --> MONGO - TRIVY["Trivy Scanner"] -- Layer Unpack Cache --> TMPFS + subgraph OnPrem["Customer Site (Self-hosted)"] + Auth[Authority (OIDC)\nOpTok (DPoP/mTLS)] + SW[Scanner.WebService] + WK[Scanner.Worker xN] + FEED[Feedser] + VEX[Vexer] + POL[Policy Engine (in Scanner.Web)] + SGN[Signer\n(entitlement + signing)] + ATT[Attestor\n(Rekor v2 submit/verify)] + UI[Web UI (Angular)] + Z[Zastava\n(Runtime Inspector/Enforcer)] + MIN[(MinIO S3)] + MGO[(MongoDB)] + QUE[(Queue/Streams)] + end - style REDIS fill:#fdd,stroke:#333 - style MONGO fill:#dfd,stroke:#333 - style TMPFS fill:#ffd,stroke:#333 + CLI[StellaOps.Cli / Buildx Plugin] + REG[(OCI Registry with Referrers)] + FUL[ Fulcio ] + REK[ Rekor v2 (tiles) ] + + CLI -->|scan/build| SW + SW -->|jobs| QUE + QUE --> WK + WK --> MIN + SW --> MGO + FEED --> MGO + VEX --> MGO + UI --> SW + Z --> SW + + SGN <--> Auth + SGN --> FUL + SGN -->|mTLS| ATT + ATT --> REK + + SGN <-->|verify referrers| REG +``` + +**Trust boundaries.** Only **Signer** can sign; only **Attestor** can write to **Rekor v2**. Scanner/UI never sign. + +--- + +## 2) Licensing & tokens (installation‑ready, theft‑resistant) + +**Two‑token model.** + +* **License Token (LT)** — long‑lived JWT from **Licensing Service**; used **once** to enroll the installation; never used in hot path. +* **Proof‑of‑Entitlement (PoE)** — bound to the installation key (mTLS client cert **or** DPoP‑bound JWT with `cnf`); medium‑lived; renewable; revocable. +* **Operational token (OpTok)** — 2–5 min OIDC token from **Authority**, **sender‑constrained** (DPoP or mTLS). Used to authenticate to **Signer**/**Scanner.WebService**. + +**Signer enforces both:** PoE proves entitlement; OpTok proves “who is calling now”. It also **independently verifies** the **scanner image digest** is **Stella Ops‑signed** via **Referrers + cosign** before signing anything. + +**Enrollment sequence (LT → PoE).** + +```plantuml +@startuml +actor Operator +participant "Install Agent" as IA +participant "Licensing Service" as LS +Operator -> IA: Provide LT +IA -> IA: Generate K_inst +IA -> LS: /license/enroll {LT, pub(K_inst)} +LS --> IA: PoE (mTLS client cert or JWT with cnf=K_inst), CRL/OCSP/introspect +@enduml ``` --- -## 7 · Typical Scenarios +## 3) Scanner subsystem (facts engine) -| # | Flow | Steps | -|---------|----------------------------|-------------------------------------------------------------------------------------------------| -| **S‑1** | Pipeline Scan & Alert | Stella CLI → SBOM → `/scan` → policy verdict → CI exit code & link to *Scan Detail* | -| **S‑2** | Mute Noisy CVE | Dev toggles **Mute** in UI → rule stored in Redis → next build passes | -| **S‑3** | Nightly Re‑scan | `SbomNightly.Schedule` re‑queues SBOMs (mask‑filter) → dashboard highlights new Criticals | -| **S‑4** | Feed Update Cycle | `Feedser (vulnerability ingest/merge/export service)` refreshes feeds → UI *Feed Age* tile turns green | -| **S‑5** | Custom Report Generation | Plug‑in registers `IReportRenderer` → `/report/custom/{digest}` → CI downloads artifact | +### 3.1 Analyzers (deterministic only) + +* **OS packages:** apk/dpkg/rpm (Linux); Windows MSI/SxS/GAC (M2). +* **Language (installed state):** + + * Java (pom.properties / MANIFEST) → `pkg:maven/...` + * Node (`node_modules/*/package.json`) → `pkg:npm/...` + * Python (`*.dist-info/METADATA`) → `pkg:pypi/...` + * Go (buildinfo) → `pkg:golang/...` + * .NET (`*.deps.json`) → `pkg:nuget/...` + * **Rust:** deterministic **language markers** (symbol mangling) and crates only when present; otherwise `bin:{sha256}`. +* **Native:** ELF/PE/Mach‑O imports, DT_NEEDED, RPATH/RUNPATH, symbol versions, PE version info. +* **EntryTrace:** parse `ENTRYPOINT`/`CMD`; shell AST; resolve launchers (Java/Node/Python) to terminal program; record file:line chain. + +### 3.2 Caching & composition + +* **Layer cache:** `{layerDigest → SBOM fragment + analyzer meta}`. +* **File CAS:** `{sha256(file) → parse result (ELF/JAR metadata/etc.)}`. +* **Composition:** build **image SBOMs** from fragments via **BOM‑Link/ExternalRef**; emit **two views**: + + * **Inventory** (complete filesystem inventory). + * **Usage** (entrypoint closure + linked libs). +* **Transport:** JSON **and** **CycloneDX Protobuf** (compact, fast to parse). +* **Index:** BOM‑Index sidecar with purl table + roaring bitmap + `usedByEntrypoint` flag for fast joins. + +### 3.3 Diff (image → layer → package) + +* Added / Removed / Version‑changed changes, **attributed** to the layer that caused them. +* Raw diffs preserved; backend view applies **VEX + Policy**. + +### 3.4 Build‑time SBOMs (fast CI path) + +* Buildx **generator** runs analyzers during `docker buildx build --attest=type=sbom,generator=stellaops/sbom-indexer`, attaches SBOMs as **OCI referrers**. +* Scanner.WebService can trust these (policy‑configurable) and **skip** re‑scan; DSSE + Rekor v2 can be done either at build time or post‑push via Signer/Attestor. + +--- + +## 4) Backend evaluation (decider) + +### 4.1 Feedser (advisories) + +* Ingests vendor, distro, OSS feeds; normalizes & merges; persists canonical advisories in Mongo; exports **deterministic JSON** and **Trivy DB**. +* Offline kit bundles for air‑gapped sites. + +### 4.2 Vexer (VEX) + +* Ingests **OpenVEX / CSAF VEX / CycloneDX VEX**; normalizes claims; retains conflicts; computes **consensus** with provider trust weights and justification gates. + +### 4.3 Policy Engine (YAML DSL) + +* Matchers: `image/repo/env/purl/cve/vendor/source/path/layerDigest/usedByEntrypoint` +* Actions: `ignore(until, justification)`, `fail`, `warn`, `defer`, `requireVEX{vendors, justifications}`, `escalate {sev, KEV, EPSS}`, license constraints. +* Produces a **policy digest** (SHA‑256 of canonicalized policy). + +### 4.4 PASS/FAIL flow + +1. SBOM (Inventory / Usage) → join with **Feedser** advisories. +2. Apply **Vexer** consensus (statuses & justifications). +3. Apply **Policy**; compute PASS/FAIL with waiver TTLs. +4. Sign the **final report** (DSSE via **Signer**) and log to **Rekor v2** via **Attestor**. + +--- + +## 5) Runtime enforcement (Zastava) + +* **Observer:** inventories running containers, checks image signatures, SBOM presence (referrers), detects drift (entrypoint chain divergence), flags unapproved images. +* **Admission Webhook (optional):** blocks policy‑fail pods (dry‑run first). +* **Integration:** posts runtime events to Scanner.WebService; can request **delta scans** on changed layers. + +--- + +## 6) Storage & catalogs (MinIO/Mongo) + +**MinIO layout** + +``` +s3://stellaops/ + layers//sbom.cdx.json.zst + layers//sbom.spdx.json.zst + images//inventory.cdx.pb + images//usage.cdx.pb + indexes//bom-index.bin + attest/.dsse.json +``` + +**Catalog (Mongo)** + +* `artifacts` (type/format/sha/size/rekor/ttl/immutable/refCount/createdAt) +* `images`, `layers`, `links`, `lifecycleRules` + +**Retention** + +* MinIO **ILM** for coarse TTL; Scanner.WebService GC decrements `refCount` and deletes unreferenced metadata; **Object Lock** for immutable classes (auditable artifacts). + +--- + +## 7) APIs (consolidated surface) + +### 7.1 Scanner.WebService + +``` +POST /api/scans { imageRef|digest, force? } → { scanId } +GET /api/scans/{id} → { status, digests, artifacts[] } +GET /api/sboms/{imageDigest} ?format=cdx-json|cdx-pb|spdx-json&view=inventory|usage +GET /api/diff?old=&new= → { added[], removed[], changed[], byLayer[] } +POST /api/exports { imageDigest, format, view } → { artifactId, rekorUrl } +POST /api/reports { imageDigest, policyRevision? } → { reportId, rekorUrl } +GET /api/catalog/artifacts/{id} → { size, ttl, immutable, rekor, refs } +GET /healthz | /readyz | /metrics +``` + +### 7.2 Signer (mTLS; hard gate) + +``` +POST /sign/dsse # body: {subjectHash, imageDigest, predicate}; headers: OpTok (DPoP/mTLS) + PoE +GET /verify/referrers?imageDigest=sha256:... # is this image StellaOps-signed? +``` + +### 7.3 Attestor (mTLS) + +``` +POST /rekor/entries # DSSE bundle → {uuid, index, proof, logURL} +GET /rekor/entries/{uuid} +``` + +### 7.4 Authority (OIDC) + +* `/.well-known/openid-configuration`, `/oauth/token` (DPoP/mTLS), `/oauth/introspect`, `/jwks` + +### 7.5 Licensing (cloud) + +``` +POST /license/enroll { LT, pubKey } → PoE + introspection endpoints +POST /license/revoke { license_id } → ok +POST /license/introspect { poe } → { active, claims, exp } +POST /attest/endorse { bundle } → endorsement bundle (optional) +``` + +--- + +## 8) Security & verifiability + +* **Sender‑constrained tokens.** All operational calls use **DPoP** (RFC 9449) or **mTLS‑bound** tokens (RFC 8705). +* **Entitlement.** **PoE** is mandatory; revocation honored online. +* **Release integrity.** **Signer** independently verifies **scanner image digest** via **Referrers + cosign** before signing. +* **Separation of duties.** Scanner/UI cannot sign; only **Signer** can sign; only **Attestor** can write to **Rekor v2**. +* **Verifiers.** Anyone can verify: DSSE signature → certificate chain to **Stella Ops Fulcio/KMS root** → **Rekor v2** inclusion. +* **Community vs Authorized.** Free/community runs throttled with no official attestations; authorized runs full speed and produce **Stella Ops‑verified** bundles. + +**DSSE predicate (SBOM/report)** + +```json +{ + "predicateType": "https://stella-ops.org/attestations/sbom/1", + "subject": [{ "name": "s3://stellaops/images//inventory.cdx.pb", "digest": { "sha256": "" } }], + "predicate": { + "image_digest": "", + "stellaops_version": "2.3.1 (2027.04)", + "license_id": "LIC-9F2A...", + "customer_id": "CUST-ACME", + "plan": "pro", + "policy_digest": "sha256:...", + "views": ["inventory","usage"], + "created": "2025-10-17T12:34:56Z" + } +} +``` + +**BOM‑Index sidecar** +Binary header + purl table + roaring bitmaps; optional `usedByEntrypoint` flags for fast policy joins. + +--- + +## 9) Scale, performance & quotas + +* **Workers:** horizontal; **distributed lock per layer digest**; global CAS in MinIO. +* **Queues:** Redis Streams / NATS / RabbitMQ. HPA by queue depth, CPU, memory. +* **Registry throttling:** per‑registry concurrency budgets. +* **Targets:** + + * Build‑time path P95 ≤ 3–5 s on warmed bases. + * Post‑build delta scan P95 ≤ 10 s for 200 MB images. + * Policy + VEX evaluation ≤ 500 ms for 5k components using BOM‑Index. +* **Quotas:** license plan enforces QPS/concurrency/size; **Signer** throttles and can deny DSSE. + +--- + +## 10) DevOps & distribution + +* **Releases:** all first‑party images **cosign‑signed**; labels embed `org.stellaops.version` and `org.stellaops.release_date`. +* **Channels:** + + * **Community** (public registry): throttled, non‑attesting. + * **Authorized** (private registry): full speed, DSSE enabled. +* **Client update flow:** containers self‑verify signatures at boot; report version; **Signer** enforces `valid_release_year` / `max_version` from PoE before signing. +* **Compose skeleton:** + +```yaml +services: + authority: { image: stellaops/authority } + fulcio: { image: sigstore/fulcio } + rekor: { image: sigstore/rekor-v2 } + minio: { image: minio/minio, command: server /data --console-address ":9001" } + mongo: { image: mongo:7 } + signer: { image: stellaops/signer, depends_on: [authority, fulcio] } + attestor: { image: stellaops/attestor, depends_on: [rekor, signer] } + scanner-web:{ image: stellaops/scanner-web, depends_on: [mongo, minio, signer, attestor] } + scanner-worker: + image: stellaops/scanner-worker + deploy: { replicas: 4 } + depends_on: [scanner-web] + feedser: { image: stellaops/feedser-web, depends_on: [mongo] } + vexer: { image: stellaops/vexer-web, depends_on: [mongo] } + ui: { image: stellaops/ui, depends_on: [scanner-web, feedser, vexer] } +``` + +* **Backups:** Mongo dumps; MinIO versioned buckets & replication; Rekor v2 DB snapshots; JWKS/Fulcio/KMS key rotation. + +--- + +## 11) Observability & audit + +* **Metrics:** scan latency, layer cache hit %, artifact bytes, DSSE/Rekor latency, policy evaluation time, queue depth, admission decisions (Zastava). +* **Tracing:** per‑stage spans; correlation IDs across Scanner→Signer→Attestor. +* **Audit logs:** every signing records `license_id`, `image_digest`, `policy_digest`, and Rekor UUID. +* **Compliance:** MinIO **Object Lock** for immutable artifacts; reproducible outputs via policy digest + SBOM digest in predicate. + +--- + +## 12) Roadmap (anchored to this architecture) + +* M2: Windows MSI/SxS/GAC analyzers; deeper Rust (DWARF enrichers). +* M2: Buildx generator certified flows; cross‑registry trust policies. +* M3: Patch‑Presence plugin (signature‑based backport detection), opt‑in. +* M3: Zastava Admission control GA with policy presets and dry‑run→enforce stages. +* Continuous: Policy UX (waiver TTLs, vendor rules), Vexer connectors expansion. + +--- + +## 13) Canonical sequences (verification & signing) + +**Sign & log (OpTok + PoE, image verify, DSSE, Rekor).** ```mermaid sequenceDiagram - participant DEV as Developer - participant UI as Web UI - participant CORE as Stella Core - participant REDIS as Redis - participant RUN as Scanner Runner + autonumber + participant Scan as Scanner.WebService + participant Auth as Authority (OIDC) + participant Sign as Signer + participant Reg as OCI Registry + participant Ful as Fulcio/KMS + participant Att as Attestor + participant Rek as Rekor v2 - DEV->>UI: Toggle Mute for CVE - UI->>CORE: Update Mute Rule (POST /policy/mute) - CORE->>REDIS: Store Mute Policy - Note over CORE,REDIS: YAML/Rego Evaluator Updates - - alt Next Pipeline Build - CI->>CORE: Trigger Scan (POST /scan) - CORE->>RUN: Enqueue & Scan - RUN-->>CORE: Raw Findings - CORE->>REDIS: Apply Mute Policies - REDIS-->>CORE: Filtered Verdict (Passes) - CORE-->>CI: Success Exit Code - end + Scan->>Auth: Get OpTok (DPoP/mTLS) + Scan->>Sign: sign(request) + OpTok + PoE + DPoP proof + Sign->>Auth: Validate OpTok & sender-constraint + Sign->>Sign: Validate PoE (introspect/revocation) + Sign->>Reg: Verify scanner image is StellaOps-signed (Referrers + cosign) + alt OK + Sign->>Ful: Get signing cert (keyless) or use KMS key + Sign-->>Scan: DSSE bundle (cert chain) + Scan->>Att: Submit bundle + Att-->>Rek: Create entry + Rek-->>Att: {uuid,index,proof} + Att-->>Scan: Rekor URL + else Deny + Sign-->>Scan: 403 (no attestation) + end ``` -```mermaid -sequenceDiagram - participant CRON as SbomNightly.Schedule - participant CORE as Stella Core - participant REDIS as Redis Queue - participant RUN as Scanner Runner - participant UI as Dashboard +**Verification (third party).** - CRON->>CORE: Re-queue SBOMs (Mask-Filter) - CORE->>REDIS: Enqueue Filtered Jobs - REDIS->>RUN: Fan Out to Runners - RUN-->>CORE: New Scan Results - CORE->>UI: Highlight New Criticals - Note over CORE,UI: Focus on Changes Since Last Scan +```plantuml +@startuml +actor Verifier +participant "stellaops verify" as Tool +database "Fulcio/KMS root" as Root +participant "Rekor v2" as R2 +Verifier -> Tool: bundle (URL/file) +Tool -> Tool: Verify DSSE signature +Tool -> Root: Verify cert chain to StellaOps root +Tool -> R2: Verify inclusion proof / query by UUID +Tool -> Verifier: OK + claims (license_id, policy_digest, version) +@enduml ``` ---- - -## 8 · UI Fast Facts - -* **Stack** – Angular 17 + Vite dev server; Tailwind CSS. -* **State** – Signals + RxJS for live scan progress. -* **i18n / l10n** – JSON bundles served from `/locales/{lang}.json`. -* **Module Structure** – Lazy‑loaded feature modules (`dashboard`, `scans`, `settings`); runtime route injection by UI plug‑ins (road‑map Q2‑2026). --- -## 9 · Cross‑Cutting Concerns - -* **Security** – containers run non‑root, `CAP_DROP:ALL`, read‑only FS, hardened seccomp profiles. -* **Observability** – Serilog JSON, OpenTelemetry OTLP exporter, Prometheus `/metrics`. -* **Upgrade Policy** – `/api/v1` endpoints & CLI flags stable across a minor; breaking changes bump major. - ---- - -## 10 · Performance & Scalability - -| Scenario | P95 target | Bottleneck | Mitigation | -|-----------------|-----------:|-----------------|-------------------------------------------------| -| SBOM‑first | ≤ 5 s | Redis queue | More CPU, increase `ScannerPool.Workers` | -| Image‑unpack | ≤ 10 s | Layer unpack | Prefer SBOM path, warm Docker cache | -| High concurrency| 40 rps | Runner CPU | Scale Core replicas + side‑car scanner services | - ---- - -## 11 · Future Architectural Anchors - -* **ScanService micro‑split (gRPC)** – isolate heavy runners for large clusters. -* **UI route plug‑ins** – dynamic Angular module loader (road‑map Q2‑2026). -* **Redis Cluster** – transparently sharded cache once sustained > 100 rps. - ---- - -## 12 · Assumptions & Trade‑offs - -Requires Docker/CRI‑O runtime; .NET 9 available on hosts; Windows containers are out‑of‑scope this cycle. -Embedded auth simplifies deployment but may need plug‑ins for enterprise IdPs. -Speed is prioritised over exhaustive feature parity with heavyweight commercial scanners. - ---- - -## 13 · References & Further Reading - -* **C4 Model** – -* **.NET Architecture Guides** – -* **OSS Examples** – Kubernetes Architecture docs, Prometheus design papers, Backstage. - -*(End of High‑Level Architecture v2.2)* +**End of `high_level_architecture.md` (Consolidated).** diff --git a/docs/08_MODULE_SPECIFICATIONS.md b/docs/08_MODULE_SPECIFICATIONS.md deleted file mode 100755 index c9c0d236..00000000 --- a/docs/08_MODULE_SPECIFICATIONS.md +++ /dev/null @@ -1,208 +0,0 @@ -# 8 · Detailed Module Specifications — **Stella Ops Feedser** -_This document describes the Feedser service, its supporting libraries, connectors, exporters, and test assets that live in the OSS repository._ - ---- - -## 0 Scope - -Feedser is the vulnerability ingest/merge/export subsystem of Stella Ops. It -fetches primary advisories, normalizes and deduplicates them into MongoDB, and -produces deterministic JSON and Trivy DB exports. This document lists the -projects that make up that workflow, the extension points they expose, and the -artefacts they ship. - ---- - -## 1 Repository layout (current) - -```text -src/ - ├─ Directory.Build.props / Directory.Build.targets - ├─ StellaOps.Plugin/ - ├─ StellaOps.Feedser.Core/ - ├─ StellaOps.Feedser.Core.Tests/ - ├─ StellaOps.Feedser.Models/ (+ .Tests/) - ├─ StellaOps.Feedser.Normalization/ (+ .Tests/) - ├─ StellaOps.Feedser.Merge/ (+ .Tests/) - ├─ StellaOps.Feedser.Storage.Mongo/ (+ .Tests/) - ├─ StellaOps.Feedser.Exporter.Json/ (+ .Tests/) - ├─ StellaOps.Feedser.Exporter.TrivyDb/ (+ .Tests/) - ├─ StellaOps.Feedser.Source.* / StellaOps.Feedser.Source.*.Tests/ - ├─ StellaOps.Feedser.Testing/ - ├─ StellaOps.Feedser.Tests.Shared/ - ├─ StellaOps.Feedser.WebService/ (+ .Tests/) - ├─ PluginBinaries/ - └─ StellaOps.Feedser.sln -``` - -Each folder is a .NET project (or set of projects) referenced by -`StellaOps.Feedser.sln`. Build assets are shared through the root -`Directory.Build.props/targets` so conventions stay consistent. - ---- - -## 2 Shared libraries - -| Project | Purpose | Key extension points | -|---------|---------|----------------------| -| `StellaOps.Plugin` | Base contracts for connectors, exporters, and DI routines plus Cosign validation helpers. | `IFeedConnector`, `IExporterPlugin`, `IDependencyInjectionRoutine` | -| `StellaOps.DependencyInjection` | Composable service registrations for Feedser and plug-ins. | `IDependencyInjectionRoutine` discovery | -| `StellaOps.Feedser.Testing` | Common fixtures, builders, and harnesses for integration/unit tests. | `FeedserMongoFixture`, test builders | -| `StellaOps.Feedser.Tests.Shared` | Shared assembly metadata and fixtures wired in via `Directory.Build.props`. | Test assembly references | - ---- - -## 3 Core projects - -| Project | Responsibility | Extensibility | -|---------|----------------|---------------| -| `StellaOps.Feedser.WebService` | ASP.NET Core minimal API hosting Feedser jobs, status endpoints, and scheduler. | DI-based plug-in discovery; configuration binding | -| `StellaOps.Feedser.Core` | Job orchestration, connector pipelines, merge workflows, export coordination. | `IFeedConnector`, `IExportJob`, deterministic merge policies | -| `StellaOps.Feedser.Models` | Canonical advisory DTOs and enums persisted in MongoDB and exported artefacts. | Partial classes for source-specific metadata | -| `StellaOps.Feedser.Normalization` | Version comparison, CVSS normalization, text utilities for canonicalization. | Helpers consumed by connectors/merge | -| `StellaOps.Feedser.Merge` | Precedence evaluation, alias graph maintenance, merge-event hashing. | Policy extensions via DI | -| `StellaOps.Feedser.Storage.Mongo` | Repository layer for documents, DTOs, advisories, merge events, export state. | Connection string/config via options | -| `StellaOps.Feedser.Exporter.Json` | Deterministic vuln-list JSON export pipeline. | Dependency injection for storage + plugin to host | -| `StellaOps.Feedser.Exporter.TrivyDb` | Builds Trivy DB artefacts from canonical advisories. | Optional ORAS push routines | - -### 3.1 StellaOps.Feedser.WebService - -* Hosts minimal API endpoints (`/health`, `/status`, `/jobs`). -* Runs the scheduler that triggers connectors and exporters according to - configured windows. -* Applies dependency-injection routines from `PluginBinaries/` at startup only - (restart-time plug-ins). - -### 3.2 StellaOps.Feedser.Core - -* Defines job primitives (fetch, parse, map, merge, export) used by connectors. -* Coordinates deterministic merge flows and writes `merge_event` documents. -* Provides telemetry/log scopes consumed by WebService and exporters. - -### 3.3 StellaOps.Feedser.Storage.Mongo - -* Persists raw documents, DTO records, canonical advisories, aliases, affected - packages, references, merge events, export state, and job leases. -* Exposes repository helpers for exporters to stream full/delta snapshots. - -### 3.4 StellaOps.Feedser.Exporter.* - -* `Exporter.Json` mirrors the Aqua vuln-list tree with canonical ordering. -* `Exporter.TrivyDb` builds Trivy DB Bolt archives and optional OCI bundles. -* Both exporters honour deterministic hashing and respect export cursors. - ---- - -## 4 Source connectors - -Connectors live under `StellaOps.Feedser.Source.*` and conform to the interfaces -in `StellaOps.Plugin`. - -| Family | Project(s) | Notes | -|--------|------------|-------| -| Distro PSIRTs | `StellaOps.Feedser.Source.Distro.*` | Debian, Red Hat, SUSE, Ubuntu connectors with NEVRA/EVR helpers. | -| Vendor PSIRTs | `StellaOps.Feedser.Source.Vndr.*` | Adobe, Apple, Cisco, Chromium, Microsoft, Oracle, VMware. | -| Regional CERTs | `StellaOps.Feedser.Source.Cert*`, `Source.Ru.*`, `Source.Ics.*`, `Source.Kisa` | Provide enrichment metadata while preserving vendor precedence. | -| OSS ecosystems | `StellaOps.Feedser.Source.Ghsa`, `Source.Osv`, `Source.Cve`, `Source.Kev`, `Source.Acsc`, `Source.Cccs`, `Source.Jvn` | Emit SemVer/alias-rich advisories. | - -Each connector ships fixtures/tests under the matching `*.Tests` project. - ---- - -## 5 · Module Details - -> _Focus on the Feedser-specific services that replace the legacy FeedMerge cron._ - -### 5.1 Feedser.Core - -* Owns the fetch → parse → merge → export job pipeline and enforces deterministic - merge hashes (`merge_event`). -* Provides `JobSchedulerBuilder`, job coordinator, and telemetry scopes consumed - by the WebService and exporters. - -### 5.2 Feedser.Storage.Mongo - -* Bootstrapper creates collections/indexes (documents, dto, advisory, alias, - affected, merge_event, export_state, jobs, locks). -* Repository APIs surface full/delta advisory reads for exporters, plus - SourceState and job lease persistence. - -### 5.3 Feedser.Exporter.Json / Feedser.Exporter.TrivyDb - -* JSON exporter mirrors vuln-list layout with per-file digests and manifest. -* Trivy DB exporter shells or native-builds Bolt archives, optionally pushes OCI - layers, and records export cursors. Delta runs reuse unchanged blobs from the - previous full baseline, annotating `metadata.json` with `mode`, `baseExportId`, - `baseManifestDigest`, `resetBaseline`, and `delta.changedFiles[]`/`delta.removedPaths[]`. - ORAS pushes honour `publishFull` / `publishDelta`, and offline bundles respect - `includeFull` / `includeDelta` for air-gapped syncs. - -### 5.4 Feedser.WebService - -* Minimal API host exposing `/health`, `/ready`, `/jobs` and wiring telemetry. -* Loads restart-time plug-ins from `PluginBinaries/`, executes Mongo bootstrap, - and registers built-in connectors/exporters with the scheduler. - -### 5.5 Plugin host & DI bridge - -* `StellaOps.Plugin` + `StellaOps.DependencyInjection` provide the contracts and - helper routines for connectors/exporters to integrate with the WebService. - ---- - -## 6 · Plug-ins & Agents - -* **Plug-in discovery** – restart-only; the WebService enumerates - `PluginBinaries/` (or configured directories) and executes the contained - `IDependencyInjectionRoutine` implementations. -* **Connector/exporter packages** – each source/exporter can ship as a plug-in - assembly with its own options and HttpClient configuration, keeping the core - image minimal. -* **StellaOps CLI (agent)** – new `StellaOps.Cli` module that exposes - `scanner`, `scan`, and `db` verbs (via System.CommandLine 2.0) to download - scanner container bundles, install them locally, execute scans against target - directories, automatically upload results, and trigger Feedser jobs (`db - fetch/merge/export`) aligned with the SBOM-first workflow described in - `AGENTS.md`. -* **Offline Kit** – bundles Feedser plug-ins, JSON tree, Trivy DB, and export - manifests so air-gapped sites can load the latest vulnerability data without - outbound connectivity. - ---- - -## 7 · Docker & Distribution Artefacts - -| Artefact | Path / Identifier | Notes | -|----------|-------------------|-------| -| Feedser WebService image | `containers/feedser/Dockerfile` (built via CI) | Self-contained ASP.NET runtime hosting scheduler/endpoints. | -| Plugin bundle | `PluginBinaries/` | Mounted or baked-in assemblies for connectors/exporters. | -| Offline Kit tarball | Produced by CI release pipeline | Contains JSON tree, Trivy DB OCI layout, export manifest, and plug-ins. | -| Local dev compose | `scripts/` + future compose overlays | Developers can run MongoDB, Redis (optional), and WebService locally. | - ---- - -## 8 · Performance Budget - -| Scenario | Budget | Source | -|----------|--------|--------| -| Advisory upsert (large advisory) | ≤ 500 ms/advisory | `AdvisoryStorePerformanceTests` (Mongo) | -| Advisory fetch (`GetRecent`) | ≤ 200 ms/advisory | Same performance test harness | -| Advisory point lookup (`Find`) | ≤ 200 ms/advisory | Same performance test harness | -| Bulk upsert/fetch cycle | ≤ 28 s total for 30 large advisories | Same performance test harness | -| Feedser job scheduling | Deterministic cron execution via `JobSchedulerHostedService` | `StellaOps.Feedser.Core` tests | -| Trivy DB export | Deterministic digests across runs (ongoing TODO for end-to-end test) | `Exporter.TrivyDb` backlog | - -Budgets are enforced in automated tests where available; outstanding TODO/DOING -items (see task boards) continue tracking gaps such as exporter determinism. - ---- - -## 9 Testing - -* Unit and integration tests live alongside each component (`*.Tests`). -* Shared fixtures come from `StellaOps.Feedser.Testing` and - `StellaOps.Feedser.Tests.Shared` (linked via `Directory.Build.props`). -* Integration suites use ephemeral MongoDB and Redis via Testcontainers to - validate end-to-end flow without external dependencies. - ---- diff --git a/docs/24_OFFLINE_KIT.md b/docs/24_OFFLINE_KIT.md index dbfbe283..df6427da 100755 --- a/docs/24_OFFLINE_KIT.md +++ b/docs/24_OFFLINE_KIT.md @@ -107,6 +107,7 @@ See the detailed rules in ## 6 · Related documentation -* **Install guide:** `/install/#air-gapped` -* **Sovereign mode rationale:** `/sovereign/` -* **Security policy:** `/security/#reporting-a-vulnerability` +* **Install guide:** `/install/#air-gapped` +* **Sovereign mode rationale:** `/sovereign/` +* **Security policy:** `/security/#reporting-a-vulnerability` +* **CERT-Bund snapshots:** `python tools/certbund_offline_snapshot.py --help` (see `docs/ops/feedser-certbund-operations.md`) diff --git a/docs/AGENTS.md b/docs/AGENTS.md index 56aa1d62..a313840b 100644 --- a/docs/AGENTS.md +++ b/docs/AGENTS.md @@ -11,6 +11,7 @@ Produce and maintain offline-friendly documentation for StellaOps modules, cover ## Operating Principles - Keep guides deterministic and in sync with shipped configuration samples. - Prefer tables/checklists for operator steps; flag security-sensitive actions. +- When work involves a specific `StellaOps.` project, consult both `docs/07_HIGH_LEVEL_ARCHITECTURE.md` and the matching dossier `docs/ARCHITECTURE_.md` before drafting or editing content. - Update `docs/TASKS.md` whenever work items change status (TODO/DOING/REVIEW/DONE/BLOCKED). ## Coordination diff --git a/docs/ARCHITECTURE_ATTESTOR.md b/docs/ARCHITECTURE_ATTESTOR.md new file mode 100644 index 00000000..f2057ede --- /dev/null +++ b/docs/ARCHITECTURE_ATTESTOR.md @@ -0,0 +1,384 @@ +# component_architecture_attestor.md — **Stella Ops Attestor** (2025Q4) + +> **Scope.** Implementation‑ready architecture for the **Attestor**: the service that **submits** DSSE envelopes to **Rekor v2**, retrieves/validates inclusion proofs, caches results, and exposes verification APIs. It accepts DSSE **only** from the **Signer** over mTLS, enforces chain‑of‑trust to Stella Ops roots, and returns `{uuid, index, proof, logURL}` to calling services (Scanner.WebService for SBOMs; backend for final reports; Vexer exports when configured). + +--- + +## 0) Mission & boundaries + +**Mission.** Turn a signed DSSE envelope from the Signer into a **transparency‑logged, verifiable fact** with a durable, replayable proof (Merkle inclusion + (optional) checkpoint anchoring). Provide **fast verification** for downstream consumers and a stable retrieval interface for UI/CLI. + +**Boundaries.** + +* Attestor **does not sign**; it **must not** accept unsigned or third‑party‑signed bundles. +* Attestor **does not decide PASS/FAIL**; it logs attestations for SBOMs, reports, and export artifacts. +* Rekor v2 backends may be **local** (self‑hosted) or **remote**; Attestor handles both with retries, backoff, and idempotency. + +--- + +## 1) Topology & dependencies + +**Process shape:** single stateless service `stellaops/attestor` behind mTLS. + +**Dependencies:** + +* **Signer** (caller) — authenticated via **mTLS** and **Authority** OpToks. +* **Rekor v2** — tile‑backed transparency log endpoint(s). +* **MinIO (S3)** — optional archive store for DSSE envelopes & verification bundles. +* **MongoDB** — local cache of `{uuid, index, proof, artifactSha256, bundleSha256}`; job state; audit. +* **Redis** — dedupe/idempotency keys and short‑lived rate‑limit buckets. +* **Licensing Service (optional)** — “endorse” call for cross‑log publishing when customer opts‑in. + +Trust boundary: **Only the Signer** is allowed to call submission endpoints; enforced by **mTLS peer cert allowlist** + `aud=attestor` OpTok. + +--- + +## 2) Data model (Mongo) + +Database: `attestor` + +**Collections & schemas** + +* `entries` + + ``` + { _id: "", + artifact: { sha256: "", kind: "sbom|report|vex-export", imageDigest?, subjectUri? }, + bundleSha256: "", // canonicalized DSSE + index: , // log index/sequence if provided by backend + proof: { // inclusion proof + checkpoint: { origin, size, rootHash, timestamp }, + inclusion: { leafHash, path[] } // Merkle path (tiles) + }, + log: { url, logId? }, + createdAt, status: "included|pending|failed", + signerIdentity: { mode: "keyless|kms", issuer, san?, kid? } + } + ``` + +* `dedupe` + + ``` + { key: "bundle:", rekorUuid, createdAt, ttlAt } // idempotency key + ``` + +* `audit` + + ``` + { _id, ts, caller: { cn, mTLSThumbprint, sub, aud }, // from mTLS + OpTok + action: "submit|verify|fetch", + artifactSha256, bundleSha256, rekorUuid?, index?, result, latencyMs, backend } + ``` + +Indexes: + +* `entries` on `artifact.sha256`, `bundleSha256`, `createdAt`, and `{status:1, createdAt:-1}`. +* `dedupe.key` unique (TTL 24–48h). +* `audit.ts` for time‑range queries. + +--- + +## 3) Input contract (from Signer) + +**Attestor accepts only** DSSE envelopes that satisfy all of: + +1. **mTLS** peer certificate maps to `signer` service (CA‑pinned). +2. **Authority** OpTok with `aud=attestor`, `scope=attestor.write`, DPoP or mTLS bound. +3. DSSE envelope is **signed by the Signer’s key** (or includes a **Fulcio‑issued** cert chain) and **chains to configured roots** (Fulcio/KMS). +4. **Predicate type** is one of Stella Ops types (sbom/report/vex‑export) with valid schema. +5. `subject[*].digest.sha256` is present and canonicalized. + +**Wire shape (JSON):** + +```json +{ + "bundle": { "dsse": { "payloadType": "application/vnd.in-toto+json", "payload": "", "signatures": [ ... ] }, + "certificateChain": [ "-----BEGIN CERTIFICATE-----..." ], + "mode": "keyless" }, + "meta": { + "artifact": { "sha256": "", "kind": "sbom|report|vex-export", "imageDigest": "sha256:..." }, + "bundleSha256": "", + "logPreference": "primary", // "primary" | "mirror" | "both" + "archive": true // whether Attestor should archive bundle to S3 + } +} +``` + +--- + +## 4) APIs + +### 4.1 Submission + +`POST /api/v1/rekor/entries` *(mTLS + OpTok required)* + +* **Body**: as above. +* **Behavior**: + + * Verify caller (mTLS + OpTok). + * Validate DSSE bundle (signature, cert chain to Fulcio/KMS; DSSE structure; payloadType allowed). + * Idempotency: compute `bundleSha256`; check `dedupe`. If present, return existing `rekorUuid`. + * Submit canonicalized bundle to Rekor v2 (primary or mirror according to `logPreference`). + * Retrieve **inclusion proof** (blocking until inclusion or up to `proofTimeoutMs`); if backend returns promise only, return `status=pending` and retry asynchronously. + * Persist `entries` record; archive DSSE to S3 if `archive=true`. +* **Response 200**: + + ```json + { + "uuid": "…", + "index": 123456, + "proof": { + "checkpoint": { "origin": "rekor@site", "size": 987654, "rootHash": "…", "timestamp": "…" }, + "inclusion": { "leafHash": "…", "path": ["…","…"] } + }, + "logURL": "https://rekor…/api/v2/log/…/entries/…", + "status": "included" + } + ``` +* **Errors**: `401 invalid_token`, `403 not_signer|chain_untrusted`, `409 duplicate_bundle` (with existing `uuid`), `502 rekor_unavailable`, `504 proof_timeout`. + +### 4.2 Proof retrieval + +`GET /api/v1/rekor/entries/{uuid}` + +* Returns `entries` row (refreshes proof from Rekor if stale/missing). +* Accepts `?refresh=true` to force backend query. + +### 4.3 Verification (third‑party or internal) + +`POST /api/v1/rekor/verify` + +* **Body** (one of): + + * `{ "uuid": "…" }` + * `{ "bundle": { …DSSE… } }` + * `{ "artifactSha256": "…" }` *(looks up most recent entry)* + +* **Checks**: + + 1. **Bundle signature** → cert chain to Fulcio/KMS roots configured. + 2. **Inclusion proof** → recompute leaf hash; verify Merkle path against checkpoint root. + 3. Optionally verify **checkpoint** against local trust anchors (if Rekor signs checkpoints). + 4. Confirm **subject.digest** matches caller‑provided hash (when given). + +* **Response**: + + ```json + { "ok": true, "uuid": "…", "index": 123, "logURL": "…", "checkedAt": "…" } + ``` + +### 4.4 Batch submission (optional) + +`POST /api/v1/rekor/batch` accepts an array of submission objects; processes with per‑item results. + +--- + +## 5) Rekor v2 driver (backend) + +* **Canonicalization**: DSSE envelopes are **normalized** (stable JSON ordering, no insignificant whitespace) before hashing and submission. +* **Transport**: HTTP/2 with retries (exponential backoff, jitter), budgeted timeouts. +* **Idempotency**: if backend returns “already exists,” map to existing `uuid`. +* **Proof acquisition**: + + * In synchronous mode, poll the log for inclusion up to `proofTimeoutMs`. + * In asynchronous mode, return `pending` and schedule a **proof fetcher** job (Mongo job doc + backoff). +* **Mirrors/dual logs**: + + * When `logPreference="both"`, submit to primary and mirror; store **both** UUIDs (primary canonical). + * Optional **cloud endorsement**: POST to the Stella Ops cloud `/attest/endorse` with `{uuid, artifactSha256}`; store returned endorsement id. + +--- + +## 6) Security model + +* **mTLS required** for submission from **Signer** (CA‑pinned). +* **Authority token** with `aud=attestor` and DPoP/mTLS binding must be presented; Attestor verifies both. +* **Bundle acceptance policy**: + + * DSSE signature must chain to the configured **Fulcio** (keyless) or **KMS/HSM** roots. + * SAN (Subject Alternative Name) must match **Signer identity** policy (e.g., `urn:stellaops:signer` or pinned OIDC issuer). + * Predicate `predicateType` must be on allowlist (sbom/report/vex-export). + * `subject.digest.sha256` values must be present and well‑formed (hex). +* **No public submission** path. **Never** accept bundles from untrusted clients. +* **Rate limits**: per mTLS thumbprint/license (from Signer‑forwarded claims) to avoid flooding the log. +* **Redaction**: Attestor never logs secret material; DSSE payloads **should** be public by design (SBOMs/reports). If customers require redaction, enforce policy at Signer (predicate minimization) **before** Attestor. + +--- + +## 7) Storage & archival + +* **Entries** in Mongo provide a local ledger keyed by `rekorUuid` and **artifact sha256** for quick reverse lookups. +* **S3 archival** (if enabled): + + ``` + s3://stellaops/attest/ + dsse/.json + proof/.json + bundle/.zip # optional verification bundle + ``` +* **Verification bundles** (zip): + + * DSSE (`*.dsse.json`), proof (`*.proof.json`), `chain.pem` (certs), `README.txt` with verification steps & hashes. + +--- + +## 8) Observability & audit + +**Metrics** (Prometheus): + +* `attestor.submit_total{result,backend}` +* `attestor.submit_latency_seconds{backend}` +* `attestor.proof_fetch_total{result}` +* `attestor.verify_total{result}` +* `attestor.dedupe_hits_total` +* `attestor.errors_total{type}` + +**Tracing**: + +* Spans: `validate`, `rekor.submit`, `rekor.poll`, `persist`, `archive`, `verify`. + +**Audit**: + +* Immutable `audit` rows (ts, caller, action, hashes, uuid, index, backend, result, latency). + +--- + +## 9) Configuration (YAML) + +```yaml +attestor: + listen: "https://0.0.0.0:8444" + security: + mtls: + caBundle: /etc/ssl/signer-ca.pem + requireClientCert: true + authority: + issuer: "https://authority.internal" + jwksUrl: "https://authority.internal/jwks" + requireSenderConstraint: "dpop" # or "mtls" + signerIdentity: + mode: ["keyless","kms"] + fulcioRoots: ["/etc/fulcio/root.pem"] + allowedSANs: ["urn:stellaops:signer"] + kmsKeys: ["kms://cluster-kms/stellaops-signer"] + rekor: + primary: + url: "https://rekor-v2.internal" + proofTimeoutMs: 15000 + pollIntervalMs: 250 + maxAttempts: 60 + mirror: + enabled: false + url: "https://rekor-v2.mirror" + mongo: + uri: "mongodb://mongo/attestor" + s3: + enabled: true + endpoint: "http://minio:9000" + bucket: "stellaops" + prefix: "attest/" + objectLock: "governance" + redis: + url: "redis://redis:6379/2" + quotas: + perCaller: + qps: 50 + burst: 100 +``` + +--- + +## 10) End‑to‑end sequences + +**A) Submit & include (happy path)** + +```mermaid +sequenceDiagram + autonumber + participant SW as Scanner.WebService + participant SG as Signer + participant AT as Attestor + participant RK as Rekor v2 + + SW->>SG: POST /sign/dsse (OpTok+PoE) + SG-->>SW: DSSE bundle (+certs) + SW->>AT: POST /rekor/entries (mTLS + OpTok) + AT->>AT: Validate DSSE (chain to Fulcio/KMS; signer identity) + AT->>RK: submit(bundle) + RK-->>AT: {uuid, index?} + AT->>RK: poll inclusion until proof or timeout + RK-->>AT: inclusion proof (checkpoint + path) + AT-->>SW: {uuid, index, proof, logURL} +``` + +**B) Verify by artifact digest (CLI)** + +```mermaid +sequenceDiagram + autonumber + participant CLI as stellaops verify + participant SW as Scanner.WebService + participant AT as Attestor + + CLI->>SW: GET /catalog/artifacts/{id} + SW-->>CLI: {artifactSha256, rekor: {uuid}} + CLI->>AT: POST /rekor/verify { uuid } + AT-->>CLI: { ok: true, index, logURL } +``` + +--- + +## 11) Failure modes & responses + +| Condition | Return | Details | | | +| ------------------------------------- | ----------------------- | --------------------------------------------------------- | -------- | ------------ | +| mTLS/OpTok invalid | `401 invalid_token` | Include `WWW-Authenticate` DPoP challenge when applicable | | | +| Bundle not signed by trusted identity | `403 chain_untrusted` | DSSE accepted only from Signer identities | | | +| Duplicate bundle | `409 duplicate_bundle` | Return existing `uuid` (idempotent) | | | +| Rekor unreachable/timeout | `502 rekor_unavailable` | Retry with backoff; surface `Retry-After` | | | +| Inclusion proof timeout | `202 accepted` | `status=pending`, background job continues to fetch proof | | | +| Archive failure | `207 multi-status` | Entry recorded; archive will retry asynchronously | | | +| Verification mismatch | `400 verify_failed` | Include reason: chain | leafHash | rootMismatch | + +--- + +## 12) Performance & scale + +* Stateless; scale horizontally. +* **Targets**: + + * Submit+proof P95 ≤ **300 ms** (warm log; local Rekor). + * Verify P95 ≤ **30 ms** from cache; ≤ **120 ms** with live proof fetch. + * 1k submissions/minute per replica sustained. +* **Hot caches**: `dedupe` (bundle hash → uuid), recent `entries` by artifact sha256. + +--- + +## 13) Testing matrix + +* **Happy path**: valid DSSE, inclusion within timeout. +* **Idempotency**: resubmit same `bundleSha256` → same `uuid`. +* **Security**: reject non‑Signer mTLS, wrong `aud`, DPoP replay, untrusted cert chain, forbidden predicateType. +* **Rekor variants**: promise‑then‑proof, proof delayed, mirror dual‑submit, mirror failure. +* **Verification**: corrupt leaf path, wrong root, tampered bundle. +* **Throughput**: soak test with 10k submissions; latency SLOs, zero drops. + +--- + +## 14) Implementation notes + +* Language: **.NET 10** minimal API; `HttpClient` with **sockets handler** tuned for HTTP/2. +* JSON: **canonical writer** for DSSE payload hashing. +* Crypto: use **BouncyCastle**/**System.Security.Cryptography**; PEM parsing for cert chains. +* Rekor client: pluggable driver; treat backend errors as retryable/non‑retryable with granular mapping. +* Safety: size caps on bundles; decompress bombs guarded; strict UTF‑8. +* CLI integration: `stellaops verify attestation ` calls `/rekor/verify`. + +--- + +## 15) Optional features + +* **Dual‑log** write (primary + mirror) and **cross‑log proof** packaging. +* **Cloud endorsement**: send `{uuid, artifactSha256}` to Stella Ops cloud; store returned endorsement id for marketing/chain‑of‑custody. +* **Checkpoint pinning**: periodically pin latest Rekor checkpoints to an external audit store for independent monitoring. + diff --git a/docs/ARCHITECTURE_AUTHORITY.md b/docs/ARCHITECTURE_AUTHORITY.md new file mode 100644 index 00000000..956f52c6 --- /dev/null +++ b/docs/ARCHITECTURE_AUTHORITY.md @@ -0,0 +1,394 @@ +# component_architecture_authority.md — **Stella Ops Authority** (2025Q4) + +> **Scope.** Implementation‑ready architecture for **Stella Ops Authority**: the on‑prem **OIDC/OAuth2** service that issues **short‑lived, sender‑constrained operational tokens (OpToks)** to first‑party services and tools. Covers protocols (DPoP & mTLS binding), token shapes, endpoints, storage, rotation, HA, RBAC, audit, and testing. This component is the trust anchor for *who* is calling inside a Stella Ops installation. (Entitlement is proven separately by **PoE** from the cloud Licensing Service; Authority does not issue PoE.) + +--- + +## 0) Mission & boundaries + +**Mission.** Provide **fast, local, verifiable** authentication for Stella Ops microservices and tools by minting **very short‑lived** OAuth2/OIDC tokens that are **sender‑constrained** (DPoP or mTLS‑bound). Support RBAC scopes, multi‑tenant claims, and deterministic validation for APIs (Scanner, Signer, Attestor, Vexer, Feedser, UI, CLI, Zastava). + +**Boundaries.** + +* Authority **does not** validate entitlements/licensing. That’s enforced by **Signer** using **PoE** with the cloud Licensing Service. +* Authority tokens are **operational only** (2–5 min TTL) and must not be embedded in long‑lived artifacts or stored in SBOMs. +* Authority is **stateless for validation** (JWT) and **optional introspection** for services that prefer online checks. + +--- + +## 1) Protocols & cryptography + +* **OIDC Discovery**: `/.well-known/openid-configuration` +* **OAuth2** grant types: + + * **Client Credentials** (service↔service, with mTLS or private_key_jwt) + * **Device Code** (CLI login on headless agents; optional) + * **Authorization Code + PKCE** (browser login for UI; optional) +* **Sender constraint options** (choose per caller or per audience): + + * **DPoP** (Demonstration of Proof‑of‑Possession): proof JWT on each HTTP request, bound to the access token via `cnf.jkt`. + * **OAuth 2.0 mTLS** (certificate‑bound tokens): token bound to client certificate thumbprint via `cnf.x5t#S256`. +* **Signing algorithms**: **EdDSA (Ed25519)** preferred; fallback **ES256 (P‑256)**. Rotation is supported via **kid** in JWKS. +* **Token format**: **JWT** access tokens (compact), optionally opaque reference tokens for services that insist on introspection. +* **Clock skew tolerance**: ±60 s; issue `nbf`, `iat`, `exp` accordingly. + +--- + +## 2) Token model + +### 2.1 Access token (OpTok) — short‑lived (120–300 s) + +**Registered claims** + +``` +iss = https://authority. +sub = +aud = +exp = (<= 300 s from iat) +iat = +nbf = iat - 30 +jti = +scope = "scanner.scan scanner.export signer.sign ..." +``` + +**Sender‑constraint (`cnf`)** + +* **DPoP**: + + ```json + "cnf": { "jkt": "" } + ``` +* **mTLS**: + + ```json + "cnf": { "x5t#S256": "" } + ``` + +**Install/tenant context (custom claims)** + +``` +tid = // multi-tenant +inst = // unique installation +roles = [ "svc.scanner", "svc.signer", "ui.admin", ... ] +plan? = // optional hint for UIs; not used for enforcement +``` + +> **Note**: Do **not** copy PoE claims into OpTok; OpTok ≠ entitlement. Only **Signer** checks PoE. + +### 2.2 Refresh tokens (optional) + +* Default **disabled**. If enabled (for UI interactive logins), pair with **DPoP‑bound** refresh tokens or **mTLS** client sessions; short TTL (≤ 8 h), rotating on use (replay‑safe). + +### 2.3 ID tokens (optional) + +* Issued for UI/browser OIDC flows (Authorization Code + PKCE); not used for service auth. + +--- + +## 3) Endpoints & flows + +### 3.1 OIDC discovery & keys + +* `GET /.well-known/openid-configuration` → endpoints, algs, jwks_uri +* `GET /jwks` → JSON Web Key Set (rotating, at least 2 active keys during transition) + +### 3.2 Token issuance + +* `POST /oauth/token` + + * **Client Credentials** (service→service): + + * **mTLS**: mutual TLS + `client_id` → bound token (`cnf.x5t#S256`) + * **private_key_jwt**: JWT‑based client auth + **DPoP** header (preferred for tools and CLI) + * **Device Code** (CLI): `POST /oauth/device/code` + `POST /oauth/token` poll + * **Authorization Code + PKCE** (UI): standard + +**DPoP handshake (example)** + +1. Client prepares **JWK** (ephemeral keypair). +2. Client sends **DPoP proof** header with fields: + + ``` + htm=POST + htu=https://authority.../oauth/token + iat= + jti= + ``` + + signed with the DPoP private key; header carries JWK. +3. Authority validates proof; issues access token with `cnf.jkt=`. +4. Client uses the same DPoP key to sign **every subsequent API request** to services (Signer, Scanner, …). + +**mTLS flow** + +* Mutual TLS at the connection; Authority extracts client cert, validates chain; token carries `cnf.x5t#S256`. + +### 3.3 Introspection & revocation (optional) + +* `POST /oauth/introspect` → `{ active, sub, scope, aud, exp, cnf, ... }` +* `POST /oauth/revoke` → revokes refresh tokens or opaque access tokens. +* **Replay prevention**: maintain **DPoP `jti` cache** (TTL ≤ 10 min) to reject duplicate proofs when services supply DPoP nonces (Signer requires nonce for high‑value operations). + +### 3.4 UserInfo (optional for UI) + +* `GET /userinfo` (ID token context). + +--- + +## 4) Audiences, scopes & RBAC + +### 4.1 Audiences + +* `signer` — only the **Signer** service should accept tokens with `aud=signer`. +* `attestor`, `scanner`, `feedser`, `vexer`, `ui`, `zastava` similarly. + +Services **must** verify `aud` and **sender constraint** (DPoP/mTLS) per their policy. + +### 4.2 Core scopes + +| Scope | Service | Operation | +| ---------------------------------- | ------------------ | -------------------------- | +| `signer.sign` | Signer | Request DSSE signing | +| `attestor.write` | Attestor | Submit Rekor entries | +| `scanner.scan` | Scanner.WebService | Submit scan jobs | +| `scanner.export` | Scanner.WebService | Export SBOMs | +| `scanner.read` | Scanner.WebService | Read catalog/SBOMs | +| `vex.read` / `vex.admin` | Vexer | Query/operate | +| `feedser.read` / `feedser.export` | Feedser | Query/exports | +| `ui.read` / `ui.admin` | UI | View/admin | +| `zastava.emit` / `zastava.enforce` | Scanner/Zastava | Runtime events / admission | + +**Roles → scopes mapping** is configured centrally (Authority policy) and pushed during token issuance. + +--- + +## 5) Storage & state + +* **Configuration DB** (PostgreSQL/MySQL): clients, audiences, role→scope maps, tenant/installation registry, device code grants, persistent consents (if any). +* **Cache** (Redis): + + * DPoP **jti** replay cache (short TTL) + * **Nonce** store (per resource server, if they demand nonce) + * Device code pollers, rate limiting buckets +* **JWKS**: key material in HSM/KMS or encrypted at rest; JWKS served from memory. + +--- + +## 6) Key management & rotation + +* Maintain **at least 2 signing keys** active during rotation; tokens carry `kid`. +* Prefer **Ed25519** for compact tokens; maintain **ES256** fallback for FIPS contexts. +* Rotation cadence: 30–90 days; emergency rotation supported. +* Publish new JWKS **before** issuing tokens with the new `kid` to avoid cold‑start validation misses. +* Keep **old keys** available **at least** for max token TTL + 5 minutes. + +--- + +## 7) HA & performance + +* **Stateless issuance** (except device codes/refresh) → scale horizontally behind a load‑balancer. +* **DB** only for client metadata and optional flows; token checks are JWT‑local; introspection endpoints hit cache/DB minimally. +* **Targets**: + + * Token issuance P95 ≤ **20 ms** under warm cache. + * DPoP proof validation ≤ **1 ms** extra per request at resource servers (Signer/Scanner). + * 99.9% uptime; HPA on CPU/latency. + +--- + +## 8) Security posture + +* **Strict TLS** (1.3 preferred); HSTS; modern cipher suites. +* **mTLS** enabled where required (Signer/Attestor paths). +* **Replay protection**: DPoP `jti` cache, nonce support for **Signer** (add `DPoP-Nonce` header on 401; clients re‑sign). +* **Rate limits** per client & per IP; exponential backoff on failures. +* **Secrets**: clients use **private_key_jwt** or **mTLS**; never basic secrets over the wire. +* **CSP/CSRF** hardening on UI flows; `SameSite=Lax` cookies; PKCE enforced. +* **Logs** redact `Authorization` and DPoP proofs; store `sub`, `aud`, `scopes`, `inst`, `tid`, `cnf` thumbprints, not full keys. + +--- + +## 9) Multi‑tenancy & installations + +* **Tenant (`tid`)** and **Installation (`inst`)** registries define which audiences/scopes a client can request. +* Cross‑tenant isolation enforced at issuance (disallow rogue `aud`), and resource servers **must** check that `tid` matches their configured tenant. + +--- + +## 10) Admin & operations APIs + +All under `/admin` (mTLS + `authority.admin` scope). + +``` +POST /admin/clients # create/update client (confidential/public) +POST /admin/audiences # register audience resource URIs +POST /admin/roles # define role→scope mappings +POST /admin/tenants # create tenant/install entries +POST /admin/keys/rotate # rotate signing key (zero-downtime) +GET /admin/metrics # Prometheus exposition (token issue rates, errors) +GET /admin/healthz|readyz # health/readiness +``` + +--- + +## 11) Integration hard lines (what resource servers must enforce) + +Every Stella Ops service that consumes Authority tokens **must**: + +1. Verify JWT signature (`kid` in JWKS), `iss`, `aud`, `exp`, `nbf`. +2. Enforce **sender‑constraint**: + + * **DPoP**: validate DPoP proof (`htu`, `htm`, `iat`, `jti`) and match `cnf.jkt`; cache `jti` for replay defense; honor nonce challenges. + * **mTLS**: match presented client cert thumbprint to token `cnf.x5t#S256`. +3. Check **scopes**; optionally map to internal roles. +4. Check **tenant** (`tid`) and **installation** (`inst`) as appropriate. +5. For **Signer** only: require **both** OpTok and **PoE** in the request (enforced by Signer, not Authority). + +--- + +## 12) Error surfaces & UX + +* Token endpoint errors follow OAuth2 (`invalid_client`, `invalid_grant`, `invalid_scope`, `unauthorized_client`). +* Resource servers use RFC 6750 style (`WWW-Authenticate: DPoP error="invalid_token", error_description="…", dpop_nonce="…" `). +* For DPoP nonce challenges, clients retry with the server‑supplied nonce once. + +--- + +## 13) Observability & audit + +* **Metrics**: + + * `authority.tokens_issued_total{grant,aud}` + * `authority.dpop_validations_total{result}` + * `authority.mtls_bindings_total{result}` + * `authority.jwks_rotations_total` + * `authority.errors_total{type}` +* **Audit log** (immutable sink): token issuance (`sub`, `aud`, `scopes`, `tid`, `inst`, `cnf thumbprint`, `jti`), revocations, admin changes. +* **Tracing**: token flows, DB reads, JWKS cache. + +--- + +## 14) Configuration (YAML) + +```yaml +authority: + issuer: "https://authority.internal" + keys: + algs: [ "EdDSA", "ES256" ] + rotationDays: 60 + storage: kms://cluster-kms/authority-signing + tokens: + accessTtlSeconds: 180 + enableRefreshTokens: false + clockSkewSeconds: 60 + dpop: + enable: true + nonce: + enable: true + ttlSeconds: 600 + mtls: + enable: true + caBundleFile: /etc/ssl/mtls/clients-ca.pem + clients: + - clientId: scanner-web + grantTypes: [ "client_credentials" ] + audiences: [ "scanner" ] + auth: { type: "private_key_jwt", jwkFile: "/secrets/scanner-web.jwk" } + senderConstraint: "dpop" + scopes: [ "scanner.scan", "scanner.export", "scanner.read" ] + - clientId: signer + grantTypes: [ "client_credentials" ] + audiences: [ "signer" ] + auth: { type: "mtls" } + senderConstraint: "mtls" + scopes: [ "signer.sign" ] +``` + +--- + +## 15) Testing matrix + +* **JWT validation**: wrong `aud`, expired `exp`, skewed `nbf`, stale `kid`. +* **DPoP**: invalid `htu`/`htm`, replayed `jti`, stale `iat`, wrong `jkt`, nonce dance. +* **mTLS**: wrong client cert, wrong CA, thumbprint mismatch. +* **RBAC**: scope enforcement per audience; over‑privileged client denied. +* **Rotation**: JWKS rotation while load‑testing; zero‑downtime verification. +* **HA**: kill one Authority instance; verify issuance continues; JWKS served by peers. +* **Performance**: 1k token issuance/sec on 2 cores with Redis enabled for jti caching. + +--- + +## 16) Threat model & mitigations (summary) + +| Threat | Vector | Mitigation | +| ------------------- | ---------------- | ------------------------------------------------------------------------------------------ | +| Token theft | Copy of JWT | **Short TTL**, **sender‑constraint** (DPoP/mTLS); replay blocked by `jti` cache and nonces | +| Replay across hosts | Reuse DPoP proof | Enforce `htu`/`htm`, `iat` freshness, `jti` uniqueness; services may require **nonce** | +| Impersonation | Fake client | mTLS or `private_key_jwt` with pinned JWK; client registration & rotation | +| Key compromise | Signing key leak | HSM/KMS storage, key rotation, audit; emergency key revoke path; narrow token TTL | +| Cross‑tenant abuse | Scope elevation | Enforce `aud`, `tid`, `inst` at issuance and resource servers | +| Downgrade to bearer | Strip DPoP | Resource servers require DPoP/mTLS based on `aud`; reject bearer without `cnf` | + +--- + +## 17) Deployment & HA + +* **Stateless** microservice, containerized; run ≥ 2 replicas behind LB. +* **DB**: HA Postgres (or MySQL) for clients/roles; **Redis** for device codes, DPoP nonces/jtis. +* **Secrets**: mount client JWKs via K8s Secrets/HashiCorp Vault; signing keys via KMS. +* **Backups**: DB daily; Redis not critical (ephemeral). +* **Disaster recovery**: export/import of client registry; JWKS rehydrate from KMS. +* **Compliance**: TLS audit; penetration testing for OIDC flows. + +--- + +## 18) Implementation notes + +* Reference stack: **.NET 10** + **OpenIddict 6** (or IdentityServer if licensed) with custom DPoP validator and mTLS binding middleware. +* Keep the DPoP/JTI cache pluggable; allow Redis/Memcached. +* Provide **client SDKs** for C# and Go: DPoP key mgmt, proof generation, nonce handling, token refresh helper. + +--- + +## 19) Quick reference — wire examples + +**Access token (payload excerpt)** + +```json +{ + "iss": "https://authority.internal", + "sub": "scanner-web", + "aud": "signer", + "exp": 1760668800, + "iat": 1760668620, + "nbf": 1760668620, + "jti": "9d9c3f01-6e1a-49f1-8f77-9b7e6f7e3c50", + "scope": "signer.sign", + "tid": "tenant-01", + "inst": "install-7A2B", + "cnf": { "jkt": "KcVb2V...base64url..." } +} +``` + +**DPoP proof header fields (for POST /sign/dsse)** + +```json +{ + "htu": "https://signer.internal/sign/dsse", + "htm": "POST", + "iat": 1760668620, + "jti": "4b1c9b3c-8a95-4c58-8a92-9c6cfb4a6a0b" +} +``` + +Signer validates that `hash(JWK)` in the proof matches `cnf.jkt` in the token. + +--- + +## 20) Rollout plan + +1. **MVP**: Client Credentials (private_key_jwt + DPoP), JWKS, short OpToks, per‑audience scopes. +2. **Add**: mTLS‑bound tokens for Signer/Attestor; device code for CLI; optional introspection. +3. **Hardening**: DPoP nonce support; full audit pipeline; HA tuning. +4. **UX**: Tenant/installation admin UI; role→scope editors; client bootstrap wizards. + diff --git a/docs/ARCHITECTURE_CLI.md b/docs/ARCHITECTURE_CLI.md new file mode 100644 index 00000000..e0ac962a --- /dev/null +++ b/docs/ARCHITECTURE_CLI.md @@ -0,0 +1,389 @@ +# component_architecture_cli.md — **Stella Ops CLI** (2025Q4) + +> **Scope.** Implementation‑ready architecture for **Stella Ops CLI**: command surface, process model, auth (Authority/DPoP), integration with Scanner/Vexer/Feedser/Signer/Attestor, Buildx plug‑in management, offline kit behavior, packaging, observability, security posture, and CI ergonomics. + +--- + +## 0) Mission & boundaries + +**Mission.** Provide a **fast, deterministic, CI‑friendly** command‑line interface to drive Stella Ops workflows: + +* Build‑time SBOM generation via **Buildx generator** orchestration. +* Post‑build **scan/compose/diff/export** against **Scanner.WebService**. +* **Policy** operations and **VEX/Vuln** data pulls (operator tasks). +* **Verification** (attestation, referrers, signatures) for audits. +* Air‑gapped/offline **kit** administration. + +**Boundaries.** + +* CLI **never** signs; it only calls **Signer**/**Attestor** via backend APIs when needed (e.g., `report --attest`). +* CLI **does not** store long‑lived credentials beyond OS keychain; tokens are **short** (Authority OpToks). +* Heavy work (scanning, merging, policy) is executed **server‑side** (Scanner/Vexer/Feedser). + +--- + +## 1) Solution layout & runtime form + +``` +src/ + ├─ StellaOps.Cli/ # net10.0 (Native AOT) single binary + ├─ StellaOps.Cli.Core/ # verb plumbing, config, HTTP, auth + ├─ StellaOps.Cli.Plugins/ # optional verbs packaged as plugins + ├─ StellaOps.Cli.Tests/ # unit + golden-output tests + └─ packaging/ + ├─ msix / msi / deb / rpm / brew formula + └─ scoop manifest / winget manifest +``` + +**Language/runtime**: .NET 10 **Native AOT** for speed/startup; Linux builds use **musl** static when possible. + +**OS targets**: linux‑x64/arm64, windows‑x64/arm64, macOS‑x64/arm64. + +--- + +## 2) Command surface (verbs) + +> All verbs default to **JSON** output when `--json` is set (CI mode). Human output is concise, deterministic. + +### 2.1 Auth & profile + +* `auth login` + + * Modes: **device‑code** (default), **client‑credentials** (service principal). + * Produces **Authority** access token (OpTok) + stores **DPoP** keypair in OS keychain. +* `auth status` — show current issuer, subject, audiences, expiry. +* `auth logout` — wipe cached tokens/keys. + +### 2.2 Build‑time SBOM (Buildx) + +* `buildx install` — install/update the **StellaOps.Scanner.Sbomer.BuildXPlugin** on the host. +* `buildx verify` — ensure generator is usable. +* `buildx build` — thin wrapper around `docker buildx build --attest=type=sbom,generator=stellaops/sbom-indexer` with convenience flags: + + * `--attest` (request Signer/Attestor via backend post‑push) + * `--provenance` pass‑through (optional) + +### 2.3 Scanning & artifacts + +* `scan image ` + + * Options: `--force`, `--wait`, `--view=inventory|usage|both`, `--format=cdx-json|cdx-pb|spdx-json`, `--attest` (ask backend to sign/log). + * Streams progress; exits early unless `--wait`. +* `diff image --old --new [--view ...]` — show layer‑attributed changes. +* `export sbom [--view ... --format ... --out file]` — download artifact. +* `report final [--policy-revision ... --attest]` — request PASS/FAIL report from backend (policy+vex) and optional attestation. + +### 2.4 Policy & data + +* `policy get/set/apply` — fetch active policy, apply staged policy, compute digest. +* `feedser export` — trigger/export canonical JSON or Trivy DB (admin). +* `vexer export` — trigger/export consensus/raw claims (admin). + +### 2.5 Verification + +* `verify attestation --uuid | --artifact | --bundle ` — call **Attestor /verify** and print proof summary. +* `verify referrers ` — ask **Signer /verify/referrers** (is image Stella‑signed?). +* `verify image-signature ` — standalone cosign verification (optional, local). + +### 2.6 Runtime (Zastava helper) + +* `runtime policy test --images [--ns --labels k=v,...]` — ask backend `/policy/runtime` like the webhook would. + +### 2.7 Offline kit + +* `offline kit pull` — fetch latest **Feedser JSON + Trivy DB + Vexer exports** as a tarball from a mirror. +* `offline kit import ` — upload the kit to on‑prem services (Feedser/Vexer). +* `offline kit status` — list current seed versions. + +### 2.8 Utilities + +* `config set/get` — endpoint & defaults. +* `whoami` — short auth display. +* `version` — CLI + protocol versions; release channel. + +--- + +## 3) AuthN: Authority + DPoP + +### 3.1 Token acquisition + +* **Device‑code**: the CLI opens an OIDC device code flow against **Authority**; the browser login is optional for service principals. +* **Client‑credentials**: service principals use **private_key_jwt** or **mTLS** to get tokens. + +### 3.2 DPoP key management + +* On first login, the CLI generates an **ephemeral JWK** (Ed25519) and stores it in the **OS keychain** (Keychain/DPAPI/KWallet/Gnome Keyring). +* Every request to backend services includes a **DPoP proof**; CLI refreshes tokens as needed. + +### 3.3 Multi‑audience & scopes + +* CLI requests **audiences** as needed per verb: + + * `scanner` for scan/export/report/diff + * `signer` (indirect; usually backend calls Signer) + * `attestor` for verify + * `feedser`/`vexer` for admin verbs + +CLI rejects verbs if required scopes are missing. + +--- + +## 4) Process model & reliability + +### 4.1 HTTP client + +* Single **http2** client with connection pooling, DNS pinning, retry/backoff (idempotent GET/POST marked safe). +* **DPoP nonce** handling: on `401` with nonce challenge, CLI replays once. + +### 4.2 Streaming + +* `scan` and `report` support **server‑sent JSON lines** (progress events). +* `--json` prints machine events; human mode shows compact spinners and crucial updates only. + +### 4.3 Exit codes (CI‑safe) + +| Code | Meaning | +| ---- | ------------------------------------------- | +| 0 | Success | +| 2 | Policy fail (final report verdict=fail) | +| 3 | Verification failed (attestation/signature) | +| 4 | Auth error (invalid/missing token/DPoP) | +| 5 | Resource not found (image/SBOM) | +| 6 | Rate limited / quota exceeded | +| 7 | Backend unavailable (retryable) | +| 9 | Invalid arguments | + +--- + +## 5) Configuration model + +**Precedence:** CLI flags → env vars → config file → defaults. + +**Config file**: `${XDG_CONFIG_HOME}/stellaops/config.yaml` (Windows: `%APPDATA%\StellaOps\config.yaml`) + +```yaml +cli: + authority: "https://authority.internal" + backend: + scanner: "https://scanner-web.internal" + attestor: "https://attestor.internal" + feedser: "https://feedser-web.internal" + vexer: "https://vexer-web.internal" + auth: + audienceDefault: "scanner" + deviceCode: true + output: + json: false + color: auto + tls: + caBundle: "/etc/ssl/certs/ca-bundle.crt" + offline: + kitMirror: "s3://mirror/stellaops-kit" +``` + +Environment variables: `STELLAOPS_AUTHORITY`, `STELLAOPS_SCANNER_URL`, etc. + +--- + +## 6) Buildx generator orchestration + +* `buildx install` locates the Docker root directory, writes the **generator** plugin manifest, and pulls `stellaops/sbom-indexer` image (pinned digest). +* `buildx build` wrapper injects: + + * `--attest=type=sbom,generator=stellaops/sbom-indexer` + * `--label org.stellaops.request=sbom` +* Post‑build: CLI optionally calls **Scanner.WebService** to **verify referrers**, **compose** image SBOMs, and **attest** via Signer/Attestor. + +**Detection**: If Buildx or generator unavailable, CLI falls back to **post‑build scan** with a warning. + +--- + +## 7) Artifact handling + +* **Downloads** (`export sbom`, `report final`): stream to file; compute sha256 on the fly; write sidecar `.sha256` and optional **verification bundle** (if `--bundle`). +* **Uploads** (`offline kit import`): chunked upload; retry on transient errors; show progress bar (unless `--json`). + +--- + +## 8) Security posture + +* **DPoP private keys** stored in **OS keychain**; metadata cached in config. +* **No plaintext tokens** on disk; short‑lived **OpToks** held in memory. +* **TLS**: verify backend certificates; allow custom CA bundle for on‑prem. +* **Redaction**: CLI logs remove `Authorization`, DPoP headers, PoE tokens. +* **Supply chain**: CLI distribution binaries are **cosign‑signed**; `stellaops version --verify` checks its own signature. + +--- + +## 9) Observability + +* `--verbose` adds request IDs, timings, and retry traces. +* **Metrics** (optional, disabled by default): Prometheus text file exporter for local monitoring in long‑running agents. +* **Structured logs** (`--json`): per‑event JSON lines with `ts`, `verb`, `status`, `latencyMs`. + +--- + +## 10) Performance targets + +* Startup ≤ **20 ms** (AOT). +* `scan image` request/response overhead ≤ **5 ms** (excluding server work). +* Buildx wrapper overhead negligible (<1 ms). +* Large artifact download (100 MB) sustained ≥ **80 MB/s** on local networks. + +--- + +## 11) Tests & golden outputs + +* **Unit tests**: argument parsing, config precedence, URL resolution, DPoP proof creation. +* **Integration tests** (Testcontainers): mock Authority/Scanner/Attestor; CI pipeline with fake registry. +* **Golden outputs**: verb snapshots for `--json` across OSes; kept in `tests/golden/…`. +* **Contract tests**: ensure API shapes match service OpenAPI; fail build if incompatible. + +--- + +## 12) Error envelopes (human + JSON) + +**Human:** + +``` +✖ Policy FAIL: 3 high, 1 critical (VEX suppressed 12) + - pkg:rpm/openssl (CVE-2025-12345) — affected (vendor) — fixed in 3.0.14 + - pkg:npm/lodash (GHSA-xxxx) — affected — no fix + See: https://ui.internal/scans/sha256:... +Exit code: 2 +``` + +**JSON (`--json`):** + +```json +{ "event":"report", "status":"fail", "critical":1, "high":3, "url":"https://ui..." } +``` + +--- + +## 13) Admin & advanced flags + +* `--authority`, `--scanner`, `--attestor`, `--feedser`, `--vexer` override config URLs. +* `--no-color`, `--quiet`, `--json`. +* `--timeout`, `--retries`, `--retry-backoff-ms`. +* `--ca-bundle`, `--insecure` (dev only; prints warning). +* `--trace` (dump HTTP traces to file; scrubbed). + +--- + +## 14) Interop with other tools + +* Emits **CycloneDX Protobuf** directly to stdout when `export sbom --format cdx-pb --out -`. +* Pipes to `jq`/`yq` cleanly in JSON mode. +* Can act as a **credential helper** for scripts: `stellaops auth token --aud scanner` prints a one‑shot token for curl. + +--- + +## 15) Packaging & distribution + +* **Installers**: deb/rpm (postinst registers completions), Homebrew, Scoop, Winget, MSI/MSIX. +* **Shell completions**: bash/zsh/fish/pwsh. +* **Update channel**: `stellaops self-update` (optional) fetches cosign‑signed release manifest; corporate environments can disable. + +--- + +## 16) Security hard lines + +* Refuse to print token values; redact Authorization headers in verbose output. +* Disallow `--insecure` unless `STELLAOPS_CLI_ALLOW_INSECURE=1` set (double opt‑in). +* Enforce **short token TTL**; refresh proactively when <30 s left. +* Device‑code cache binding to **machine** and **user** (protect against copy to other machines). + +--- + +## 17) Wire sequences + +**A) Scan & wait with attestation** + +```mermaid +sequenceDiagram + autonumber + participant CLI + participant Auth as Authority + participant SW as Scanner.WebService + participant SG as Signer + participant AT as Attestor + + CLI->>Auth: device code flow (DPoP) + Auth-->>CLI: OpTok (aud=scanner) + + CLI->>SW: POST /scans { imageRef, attest:true } + SW-->>CLI: { scanId } + CLI->>SW: GET /scans/{id} (poll) + SW-->>CLI: { status: completed, artifacts, rekor? } # if attested + + alt attestation pending + SW->>SG: POST /sign/dsse (server-side) + SG-->>SW: DSSE + SW->>AT: POST /rekor/entries + AT-->>SW: { uuid, proof } + end + + CLI->>SW: GET /sboms/?format=cdx-pb&view=usage + SW-->>CLI: bytes +``` + +**B) Verify attestation by artifact** + +```mermaid +sequenceDiagram + autonumber + participant CLI + participant AT as Attestor + + CLI->>AT: POST /rekor/verify { artifactSha256 } + AT-->>CLI: { ok:true, uuid, index, logURL } +``` + +--- + +## 18) Roadmap (CLI) + +* `scan fs ` (local filesystem tree) → upload to backend for analysis. +* `policy test --sbom ` (simulate policy results offline using local policy bundle). +* `runtime capture` (developer mode) — capture small `/proc//maps` for troubleshooting. +* Pluggable output renderers for SARIF/HTML (admin‑controlled). + +--- + +## 19) Example CI snippets + +**GitHub Actions (post‑build)** + +```yaml +- name: Login (device code w/ OIDC broker) + run: stellaops auth login --json --authority ${{ secrets.AUTHORITY_URL }} + +- name: Scan + run: stellaops scan image ${{ steps.build.outputs.digest }} --wait --json + +- name: Export (usage view, protobuf) + run: stellaops export sbom ${{ steps.build.outputs.digest }} --view usage --format cdx-pb --out sbom.pb + +- name: Verify attestation + run: stellaops verify attestation --artifact $(sha256sum sbom.pb | cut -d' ' -f1) --json +``` + +**GitLab (buildx generator)** + +```yaml +script: + - stellaops buildx install + - docker buildx build --attest=type=sbom,generator=stellaops/sbom-indexer -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA . + - stellaops scan image $CI_REGISTRY_IMAGE@$IMAGE_DIGEST --wait --json +``` + +--- + +## 20) Test matrix (OS/arch) + +* Linux: ubuntu‑20.04/22.04/24.04 (x64, arm64), alpine (musl). +* macOS: 13–15 (x64, arm64). +* Windows: 10/11, Server 2019/2022 (x64, arm64). +* Docker engines: Docker Desktop, containerd‑based runners. + diff --git a/docs/ARCHITECTURE_DEVOPS.md b/docs/ARCHITECTURE_DEVOPS.md new file mode 100644 index 00000000..901a6f6c --- /dev/null +++ b/docs/ARCHITECTURE_DEVOPS.md @@ -0,0 +1,462 @@ +# component_architecture_devops.md — **Stella Ops Release & Operations** (2025Q4) + +> **Scope.** Implementation‑ready blueprint for **how Stella Ops is built, versioned, signed, distributed, upgraded, licensed (PoE)**, and operated in customer environments (online and air‑gapped). Covers reproducible builds, supply‑chain attestations, registries, offline kits, migration/rollback, artifact lifecycle (MinIO/Mongo), monitoring SLOs, and customer activation. + +--- + +## 0) Product vision (operations lens) + +Stella Ops must be **trustable at a glance** and **boringly operable**: + +* Every release ships with **first‑party SBOMs, provenance, and signatures**; services verify **each other’s** integrity at runtime. +* Customers can deploy by **digest** and stay aligned with **LTS/stable/edge** channels. +* Paid customers receive **attestation authority** (Signer accepts their PoE) while the core platform remains **free to run**. +* Air‑gapped customers receive **offline kits** with verifiable digests and deterministic import. +* Artifacts expire predictably; operators know what’s kept, for how long, and why. + +--- + +## 1) Release trains & versioning + +### 1.1 Channels + +* **LTS** (12‑month support window): quarterly cadence (Q1/Q2/Q3/Q4). +* **Stable** (default): monthly rollup (bug fixes + compatible features). +* **Edge**: weekly; for early adopters, no guarantees. + +### 1.2 Version strings + +Semantic core + calendar tag: + +``` +.. (.) e.g., 2.4.1 (2027.06) +``` + +* **MAJOR**: breaking API/DB changes (rare). +* **MINOR**: new features, compatible schema migrations (expand/contract pattern). +* **PATCH**: bug fixes, perf and security updates. +* **Calendar tag** exposes **release year** used by Signer for **PoE window checks**. + +### 1.3 Component alignment + +A release is a **bundle** of image digests + charts + manifests. All services in a bundle are **wire‑compatible**. Mixed minor versions are allowed within a bounded skew: + +* **Web UI ↔ backend**: `±1 minor`. +* **Scanner ↔ Policy/Vexer/Feedser**: `±1 minor`. +* **Authority/Signer/Attestor triangle**: **must** be same minor (crypto and DPoP/mTLS binding rules). + +At startup, services **self‑advertise** their semver & channel; the UI surfaces **mismatch warnings**. + +--- + +## 2) Supply‑chain pipeline (how a release is built) + +### 2.1 Deterministic builds + +* **Builders**: isolated **BuildKit** workers with pinned base images (digest only). +* **Pinning**: lock files or `go.mod`, `package-lock.json`, `global.json`, `Directory.Packages.props` are **frozen** at tag. +* **Reproducibility**: timestamps normalized; source date epoch; deterministic zips/tars. +* **Multi‑arch**: linux/amd64 + linux/arm64 (Windows images track M2 roadmap). + +### 2.2 First‑party SBOMs & provenance + +* Each image gets **CycloneDX (JSON+Protobuf) SBOM** and **SLSA‑style provenance** attached as **OCI referrers**. +* Scanner’s **Buildx generator** is used to produce SBOMs *during* build; a separate post‑build scan verifies parity (red flag if drift). +* **Release manifest** (see §6.1) lists all digests and SBOM/attestation refs. + +### 2.3 Signing & transparency + +* Images are **cosign‑signed** (keyless) with a Stella Ops release identity; inclusion in a **transparency log** (Rekor) is required. +* SBOM and provenance attestations are **DSSE** and also transparency‑logged. +* Release keys (Fulcio roots or public keys) are embedded in **Signer** policy (for **scanner‑release validation** at customer side). + +### 2.4 Gates & tests + +* **Static**: linters, codegen checks, protobuf API freeze (backward‑compat tests). +* **Unit/integration**: per‑component, plus **end‑to‑end** flows (scan→vex→policy→sign→attest). +* **Perf SLOs**: hot paths (SBOM compose, diff, export) measured against budgets. +* **Security**: dependency audit vs Feedser export; container hardening tests; minimal caps. +* **Canary cohort**: internal staging + selected customers; one week on **edge** before **stable** tag. + +--- + +## 3) Distribution & activation + +### 3.1 Registries + +* **Primary**: `registry.stella-ops.org` (OCI v2, supports Referrers API). +* **Mirrors**: GHCR (read‑only), regional mirrors for latency. +* **Pull by digest only** in Kubernetes/Compose manifests. + +**Gating policy**: + +* **Core images** (Authority, Scanner, Feedser, Vexer, Attestor, UI): public **read**. +* **Enterprise add‑ons** (if any) and **pre‑release**: private repos via OAuth2 token service. + +> Monetization lever is **signing** (PoE gate), not image pulls, so the core remains simple to consume. + +### 3.2 OAuth2 token service (for private repos) + +* Docker Registry’s token flow backed by **Authority**: + + 1. Client hits registry (`401` with `WWW-Authenticate: Bearer realm=…`). + 2. Client gets an **access token** from the token service (validated by Authority) with `scope=repository:…:pull`. + 3. Registry allows pull for the requested repo. +* Tokens are **short‑lived** (60–300 s) and **DPoP‑bound**. + +### 3.3 Offline kits (air‑gapped) + +* Tarball per release channel: + + ``` + stellaops-kit--.tar.zst + /images/ OCI layout with all first-party images (multi-arch) + /sboms/ CycloneDX JSON+PB for each image + /attest/ DSSE bundles + Rekor proofs + /charts/ Helm charts + values templates + /compose/ docker-compose.yml + .env template + /plugins/ Feedser/Vexer connectors (restart-time) + /policy/ example policies + /manifest/ release.yaml (see §6.1) + ``` +* Import via CLI `offline kit import`; checks digests and signatures before load. + +--- + +## 4) Licensing (PoE) & monetization + +**Principle**: **Only paid Stella Ops issues valid signed attestations.** Running the stack is free; signing requires PoE. + +### 4.1 PoE issuance + +* Customers purchase a plan and obtain a **PoE artifact** from `www.stella-ops.org`: + + * **PoE‑JWT** (DPoP/mTLS‑bound) **or** **PoE mTLS client certificate**. + * Contains: `license_id`, `plan`, `valid_release_year`, `max_version`, `exp`, optional `tenant/customer` IDs. + +### 4.2 Online enforcement + +* **Signer** calls **Licensing /license/introspect** on every signing request (see signer doc). +* If **revoked/expired/out‑of‑window** → deny with machine‑readable reason. +* All **valid** bundles are DSSE‑signed and **Attestor** logs them; Rekor UUID returned. +* UI badges: “**Verified by Stella Ops**” with link to the public log. + +### 4.3 Air‑gapped / offline + +* Customers obtain a **time‑boxed PoE lease** (signed JSON, 7–30 days). +* Signer accepts the lease and emits **provisional** attestations (clearly labeled). +* When connectivity returns, a background job **endorses** the provisional entries with the cloud service, updating their status to **verified**. +* Operators can export a **verification bundle** for auditors even before endorsement (contains DSSE + local Rekor proof + lease snapshot). + +### 4.4 Stolen/abused PoE + +* Customers report theft; **Licensing** flags `license_id` as **revoked**. +* Subsequent Signer requests **deny**; previous attestations remain but can be marked **contested** (UI shows badge, optional re‑sign path upon new PoE). + +--- + +## 5) Deployment path (customer side) + +### 5.1 First install + +* **Helm** (Kubernetes) or **Compose** (VMs). Example (K8s): + +```bash +helm repo add stellaops https://charts.stella-ops.org +helm install stella stellaops/platform \ + --version 2.4.0 \ + --set global.channel=stable \ + --set authority.issuer=https://authority.stella.local \ + --set scanner.minio.endpoint=http://minio.stella.local:9000 \ + --set scanner.mongo.uri=mongodb://mongo/scanner \ + --set feedser.mongo.uri=mongodb://mongo/feedser \ + --set vexer.mongo.uri=mongodb://mongo/vexer +``` + +* Post‑install job registers **Authority clients** (Scanner, Signer, Attestor, UI) and prints **bootstrap** URLs and client credentials (sealed secrets). +* UI banner shows **release bundle** and verification state (cosign OK? Rekor OK?). + +### 5.2 Updates + +* **Blue/green**: pull new bundle by **digest**; deploy side‑by‑side; cut traffic. + +* **Rolling**: upgrade stateful components in safe order: + + 1. Authority (stateless, dual‑key rotation ready) + 2. Signer/Attestor (same minor) + 3. Scanner WebService & Workers + 4. Feedser, then Vexer (schema migrations are expand/contract) + 5. UI last + +* **DB migrations** are **expand/contract**: + + * Phase A (release N): **add** new fields/indexes, write old+new. + * Phase B (N+1): **read** new fields; **drop** old. + * Rollback is a matter of redeploying previous images and keeping both schemas valid. + +### 5.3 Rollback + +* Images referenced by **digest**; keep previous release manifest `K` versions back. +* `helm rollback` or compose `docker compose -f release-K.yml up -d`. +* Mongo migrations are additive; **no destructive changes** within a single minor. + +--- + +## 6) Release payloads & manifests + +### 6.1 Release manifest (`release.yaml`) + +```yaml +release: + version: "2.4.1" + channel: "stable" + date: "2027-06-20T12:00:00Z" + calendar: "2027.06" + components: + - name: scanner-webservice + image: registry.stella-ops.org/stellaops/scanner-web@sha256:aa..bb + sbom: oci://.../referrers/cdx-json@sha256:11..22 + provenance: oci://.../attest/provenance@sha256:33..44 + signature: { rekorUUID: "…" } + - name: signer + image: registry.stella-ops.org/stellaops/signer@sha256:cc..dd + signature: { rekorUUID: "…" } + charts: + - name: platform + version: "2.4.1" + digest: "sha256:ee..ff" + compose: + file: "docker-compose.yml" + digest: "sha256:77..88" + checksums: + sha256: "… digest of this release.yaml …" +``` + +The manifest is **cosign‑signed**; UI/CLI can verify a bundle without talking to registries. + +### 6.2 Image labels (release metadata) + +Each image sets OCI labels: + +``` +org.opencontainers.image.version = "2.4.1" +org.opencontainers.image.revision = "" +org.opencontainers.image.created = "2027-06-20T12:00:00Z" +org.stellaops.release.calendar = "2027.06" +org.stellaops.release.channel = "stable" +org.stellaops.build.slsaProvenance = "oci://…" +``` + +Signer validates **scanner** image’s cosign identity + calendar tag for **release window** checks. + +--- + +## 7) Artifact lifecycle & storage (MinIO/Mongo) + +### 7.1 Buckets & prefixes (MinIO) + +``` +s3://stellaops/ + scanner/ + layers//sbom.cdx.json.zst + images//inventory.cdx.pb + images//usage.cdx.pb + diffs/_/diff.json.zst + attest/.dsse.json + feedser/ + json//... + trivy//... + vexer/ + exports//... + attestor/ + dsse/.json + proof/.json +``` + +### 7.2 ILM classes + +* **`short`**: working artifacts (diffs, queues) — TTL 7–14 days. +* **`default`**: SBOMs & indexes — TTL 90–180 days (configurable). +* **`compliance`**: signed reports & attested exports — **Object Lock** (governance/compliance) 1–7 years. + +### 7.3 Artifact Lifecycle Controller (ALC) + +* A background worker (part of Scanner.WebService) enforces **TTL** and **reference counting**: + + * Artifacts referenced by **reports** or **tickets** are pinned. + * ILM actions logged; UI shows per‑class usage & upcoming purges. + +### 7.4 Mongo retention + +* **Scanner**: `runtime.events` use TTL (e.g., 30–90 days); **catalog** permanent. +* **Feedser/Vexer**: raw docs keep **last N windows**; canonical stores permanent. +* **Attestor**: `entries` permanent; `dedupe` TTL 24–48h. + +--- + +## 8) Observability & SLOs (operations) + +* **Uptime SLO**: 99.9% for Signer/Authority/Attestor; 99.5% for Scanner WebService; Vexer/Feedser 99.0%. +* **Error budgets**: tracked per month; dashboards show burn rates. +* **Golden signals**: + + * **Latency**: token issuance, sign→attest round‑trip, scan enqueue→emit, export build. + * **Saturation**: queue depth, Mongo write IOPS, MinIO net throughput. + * **Traffic**: scans/min, attestations/min, webhook admits/min. + * **Errors**: 5xx rates, cosign verification failures, Rekor timeouts. + +Prometheus + OTLP; Grafana dashboards ship in the charts. + +--- + +## 9) Security & compliance operations + +* **Key rotation**: + + * Authority JWKS: 60‑day cadence, dual‑key overlap. + * Release signing identities: rotate per minor or quarterly. + * Sigstore roots mirrored and pinned; alarms on drift. + +* **FIPS mode** (Gov build): + + * Enforce `ES256` + KMS/HSM; disable Ed25519; MLS ciphers only. + * Local **Rekor v2** and **Fulcio** alternatives; **air‑gapped** CA. + +* **Vulnerability response**: + + * Feedser red‑flag advisories trigger accelerated **stable** patch rollout; UI/CLI “security patch available” notice. + +* **Backups/DR**: + + * Mongo nightly snapshots; MinIO versioning + replication (if configured). + * Restore runbooks tested quarterly with synthetic data. + +--- + +## 10) Customer update flow (how versions are fetched & activated) + +### 10.1 Online clusters + +* **UI** surfaces update banner with **release manifest** diff and risk notes. +* Operator approves → **Controller** pulls new images by digest; health‑checks; moves traffic; deprecates old revision. +* Post‑switch, **schema Phase B** migrations (if any) run automatically. + +### 10.2 Air‑gapped clusters + +* Operator downloads **offline kit** from a mirror → `stellaops offline kit import`. +* Controller validates bundle checksums and **cosign signatures**; applies charts/compose by digest. +* After install, **verify** page shows green checks: image sigs, SBOMs attached, provenance logged. + +### 10.3 CLI self‑update (optional) + +* `stellaops self-update` pulls a **signed release manifest** and verifies the **CLI binary** with cosign before swapping (admin can disable). + +--- + +## 11) Compatibility & deprecation policy + +* **APIs** are stable within a **major**; breaking changes imply **MAJOR++** and deprecation period of one minor. +* **Storage**: expand/contract; “drop old fields” only after one minor grace. +* **Config**: feature flags (default off) for risky features (e.g., eBPF). + +--- + +## 12) Runbooks (selected) + +### 12.1 Lost PoE + +1. Suspend **automatic attestation** jobs. +2. Use CLI `stellaops signer status` to confirm `entitlement_denied`. +3. Obtain new PoE from portal; verify on Signer `/poe/verify`. +4. Re‑enable; optionally **re‑sign** last N reports (UI button → batch). + +### 12.2 Rekor outage (self‑hosted) + +* Attestor returns `202 (pending)` with queued proof fetch. +* Keep DSSE bundles locally; re‑submit on schedule; UI badge shows **Pending**. +* If outage > SLA, you can switch to a **mirror** log in config; Attestor writes to both when restored. + +### 12.3 Emergency downgrade + +* Identify prior release manifest (UI → Admin → Releases). +* `helm rollback stella ` (or compose apply previous file). +* Services tolerate skew per §1.3; ensure **Signer/Authority/Attestor** are rolled together. + +--- + +## 13) Example: cluster bootstrap (Compose) + +```yaml +version: "3.9" +services: + authority: + image: registry.stella-ops.org/stellaops/authority@sha256:... + env_file: ./env/authority.env + ports: ["8440:8440"] + signer: + image: registry.stella-ops.org/stellaops/signer@sha256:... + depends_on: [authority] + environment: + - SIGNER__POE__LICENSING__INTROSPECTURL=https://www.stella-ops.org/api/v1/license/introspect + attestor: + image: registry.stella-ops.org/stellaops/attestor@sha256:... + depends_on: [signer] + scanner-web: + image: registry.stella-ops.org/stellaops/scanner-web@sha256:... + environment: + - SCANNER__S3__ENDPOINT=http://minio:9000 + scanner-worker: + image: registry.stella-ops.org/stellaops/scanner-worker@sha256:... + deploy: { replicas: 4 } + feedser: + image: registry.stella-ops.org/stellaops/feedser@sha256:... + vexer: + image: registry.stella-ops.org/stellaops/vexer@sha256:... + web-ui: + image: registry.stella-ops.org/stellaops/web-ui@sha256:... + mongo: + image: mongo:7 + minio: + image: minio/minio:RELEASE.2025-07-10T00-00-00Z +``` + +--- + +## 14) Governance & keys (who owns the trust root) + +* **Release key policy**: only the Release Engineering group can push signed releases; 4‑eyes approval; TUF‑style manifest possible in future. +* **Signer acceptance policy**: embedded release identities are updated **only** via minor upgrade; emergency CRL supported. +* **Customer keys**: none needed for core use; enterprise add‑ons may require per‑customer registries and keys. + +--- + +## 15) Roadmap (Ops) + +* **Windows containers GA** (Scanner + Zastava). +* **Key Transparency** for Signer certs. +* **Delta‑kit** (offline) for incremental updates. +* **Operator CRDs** (K8s) to manage policy and ILM declaratively. +* **SBOM **protobuf** as default transport at rest (smaller, faster). + +--- + +### Appendix A — Minimal SLO monitors + +* `authority.tokens_issued_total` slope ≈ normal. +* `signer.requests_total{result="success"}/minute` > 0 (when scans occur). +* `attestor.submit_latency_seconds{quantile=0.95}` < 0.3. +* `scanner.scan_latency_seconds{quantile=0.95}` < target per image size. +* `feedser.export.duration_seconds` stable; `vexer.consensus.conflicts_total` not exploding after policy changes. +* MinIO `s3_requests_errors_total` near zero; Mongo `opcounters` hit expected baseline. + +### Appendix B — Upgrade safety checklist + +* Verify **release manifest** signature. +* Ensure **Signer/Authority/Attestor** are same minor. +* Verify **DB backups** < 24h old. +* Confirm **ILM** won’t purge compliance artifacts during upgrade window. +* Roll **one component** at a time; watch SLOs; abort on regression. + +--- + +**End — component_architecture_devops.md** diff --git a/docs/ARCHITECTURE_FEEDSER.md b/docs/ARCHITECTURE_FEEDSER.md index 9c11b8a6..d394d635 100644 --- a/docs/ARCHITECTURE_FEEDSER.md +++ b/docs/ARCHITECTURE_FEEDSER.md @@ -1,190 +1,433 @@ -# ARCHITECTURE.md — **StellaOps.Feedser** +# component_architecture_feedser.md — **Stella Ops Feedser** (2025Q4) -> **Goal**: Build a sovereign-ready, self-hostable **feed-merge service** that ingests authoritative vulnerability sources, normalizes and de-duplicates them into **MongoDB**, and exports **JSON** and **Trivy-compatible DB** artifacts. -> **Form factor**: Long-running **Web Service** with **REST APIs** (health, status, control) and an embedded **internal cron scheduler**. Controllable by StellaOps.Cli (# stella db ...) -> **No signing inside Feedser** (signing is a separate pipeline step). -> **Runtime SDK baseline**: .NET 10 Preview 7 (SDK 10.0.100-preview.7.25380.108) targeting `net10.0`, aligned with the deployed api.stella-ops.org service. -> **Four explicit stages**: -> -> 1. **Source Download** → raw documents. -> 2. **Parse & Normalize** → schema-validated DTOs enriched with canonical identifiers. -> 3. **Merge & Deduplicate** → precedence-aware canonical records persisted to MongoDB. -> 4. **Export** → JSON or TrivyDB (full or delta), then (externally) sign/publish. +> **Scope.** Implementation‑ready architecture for **Feedser**: the vulnerability ingest/normalize/merge/export subsystem that produces deterministic advisory data for the Scanner + Policy + Vexer pipeline. Covers domain model, connectors, merge rules, storage schema, exports, APIs, performance, security, and test matrices. --- -## 1) Naming & Solution Layout +## 0) Mission & boundaries -**Source connectors** namespace prefix: `StellaOps.Feedser.Source.*` -**Exporters**: +**Mission.** Acquire authoritative **vulnerability advisories** (vendor PSIRTs, distros, OSS ecosystems, CERTs), normalize them into a **canonical model**, reconcile aliases and version ranges, and export **deterministic artifacts** (JSON, Trivy DB) for fast backend joins. -* `StellaOps.Feedser.Exporter.Json` -* `StellaOps.Feedser.Exporter.TrivyDb` +**Boundaries.** -**Projects** (`/src`): +* Feedser **does not** sign with private keys. When attestation is required, the export artifact is handed to the **Signer**/**Attestor** pipeline (out‑of‑process). +* Feedser **does not** decide PASS/FAIL; it provides data to the **Policy** engine. +* Online operation is **allowlist‑only**; air‑gapped deployments use the **Offline Kit**. + +--- + +## 1) Topology & processes + +**Process shape:** single ASP.NET Core service `StellaOps.Feedser.WebService` hosting: + +* **Scheduler** with distributed locks (Mongo backed). +* **Connectors** (fetch/parse/map). +* **Merger** (canonical record assembly + precedence). +* **Exporters** (JSON, Trivy DB). +* **Minimal REST** for health/status/trigger/export. + +**Scale:** HA by running N replicas; **locks** prevent overlapping jobs per source/exporter. + +--- + +## 2) Canonical domain model + +> Stored in MongoDB (database `feedser`), serialized with a **canonical JSON** writer (stable order, camelCase, normalized timestamps). + +### 2.1 Core entities + +**Advisory** ``` -StellaOps.Feedser.WebService/ # ASP.NET Core (Minimal API, net10.0 preview) WebService + embedded scheduler -StellaOps.Feedser.Core/ # Domain models, pipelines, merge/dedupe engine, jobs orchestration -StellaOps.Feedser.Models/ # Canonical POCOs, JSON Schemas, enums -StellaOps.Feedser.Storage.Mongo/ # Mongo repositories, GridFS access, indexes, resume "flags" -StellaOps.Feedser.Source.Common/ # HTTP clients, rate-limiters, schema validators, parsers utils -StellaOps.Feedser.Source.Cve/ -StellaOps.Feedser.Source.Nvd/ -StellaOps.Feedser.Source.Ghsa/ -StellaOps.Feedser.Source.Osv/ -StellaOps.Feedser.Source.Jvn/ -StellaOps.Feedser.Source.CertCc/ -StellaOps.Feedser.Source.Kev/ -StellaOps.Feedser.Source.Kisa/ -StellaOps.Feedser.Source.CertIn/ -StellaOps.Feedser.Source.CertFr/ -StellaOps.Feedser.Source.CertBund/ -StellaOps.Feedser.Source.Acsc/ -StellaOps.Feedser.Source.Cccs/ -StellaOps.Feedser.Source.Ru.Bdu/ # HTML→schema with LLM fallback (gated) -StellaOps.Feedser.Source.Ru.Nkcki/ # PDF/HTML bulletins → structured -StellaOps.Feedser.Source.Vndr.Msrc/ -StellaOps.Feedser.Source.Vndr.Cisco/ -StellaOps.Feedser.Source.Vndr.Oracle/ -StellaOps.Feedser.Source.Vndr.Adobe/ # APSB ingest; emits vendor RangePrimitives with adobe.track/platform/priority telemetry + fixed-status provenance. -StellaOps.Feedser.Source.Vndr.Apple/ -StellaOps.Feedser.Source.Vndr.Chromium/ -StellaOps.Feedser.Source.Vndr.Vmware/ -StellaOps.Feedser.Source.Distro.RedHat/ -StellaOps.Feedser.Source.Distro.Debian/ # Fetches DSA list + detail HTML, emits EVR RangePrimitives with per-release provenance and telemetry. -StellaOps.Feedser.Source.Distro.Ubuntu/ # Ubuntu Security Notices connector (JSON index → EVR ranges with ubuntu.pocket telemetry). -StellaOps.Feedser.Source.Distro.Suse/ # CSAF fetch pipeline emitting NEVRA RangePrimitives with suse.status vendor telemetry. -StellaOps.Feedser.Source.Ics.Cisa/ -StellaOps.Feedser.Source.Ics.Kaspersky/ -StellaOps.Feedser.Normalization/ # Canonical mappers, validators, version-range normalization -StellaOps.Feedser.Merge/ # Identity graph, precedence, deterministic merge -StellaOps.Feedser.Exporter.Json/ -StellaOps.Feedser.Exporter.TrivyDb/ -StellaOps.Feedser..Tests/ # Component-scoped unit/integration suites (Core, Storage.Mongo, Source.*, Exporter.*, WebService, etc.) +advisoryId // internal GUID +advisoryKey // stable string key (e.g., CVE-2025-12345 or vendor ID) +title // short title (best-of from sources) +summary // normalized summary (English; i18n optional) +published // earliest source timestamp +modified // latest source timestamp +severity // normalized {none, low, medium, high, critical} +cvss // {v2?, v3?, v4?} objects (vector, baseScore, severity, source) +exploitKnown // bool (e.g., KEV/active exploitation flags) +references[] // typed links (advisory, kb, patch, vendor, exploit, blog) +sources[] // provenance for traceability (doc digests, URIs) ``` ---- +**Alias** -## 2) Runtime Shape +``` +advisoryId +scheme // CVE, GHSA, RHSA, DSA, USN, MSRC, etc. +value // e.g., "CVE-2025-12345" +``` -**Process**: single service (`StellaOps.Feedser.WebService`) +**Affected** -* `Program.cs`: top-level entry using **Generic Host**, **DI**, **Options** binding from `appsettings.json` + environment + optional `feedser.yaml`. -* Built-in **scheduler** (cron-like) + **job manager** with **distributed locks** in Mongo to prevent overlaps, enforce timeouts, allow cancel/kill. -* **REST APIs** for health/readiness/progress/trigger/kill/status. +``` +advisoryId +productKey // canonical product identity (see 2.2) +rangeKind // semver | evr | nvra | apk | rpm | deb | generic | exact +introduced? // string (format depends on rangeKind) +fixed? // string (format depends on rangeKind) +lastKnownSafe? // optional explicit safe floor +arch? // arch or platform qualifier if source declares (x86_64, aarch64) +distro? // distro qualifier when applicable (rhel:9, debian:12, alpine:3.19) +ecosystem? // npm|pypi|maven|nuget|golang|… +notes? // normalized notes per source +``` -**Key NuGet concepts** (indicative): `MongoDB.Driver`, `Polly` (retry/backoff), `System.Threading.Channels`, `Microsoft.Extensions.Http`, `Microsoft.Extensions.Hosting`, `Serilog`, `OpenTelemetry`. +**Reference** + +``` +advisoryId +url +kind // advisory | patch | kb | exploit | mitigation | blog | cvrf | csaf +sourceTag // e.g., vendor/redhat, distro/debian, oss/ghsa +``` + +**MergeEvent** + +``` +advisoryKey +beforeHash // canonical JSON hash before merge +afterHash // canonical JSON hash after merge +mergedAt +inputs[] // source doc digests that contributed +``` + +**ExportState** + +``` +exportKind // json | trivydb +baseExportId? // last full baseline +baseDigest? // digest of last full baseline +lastFullDigest? // digest of last full export +lastDeltaDigest? // digest of last delta export +cursor // per-kind incremental cursor +files[] // last manifest snapshot (path → sha256) +``` + +### 2.2 Product identity (`productKey`) + +* **Primary:** `purl` (Package URL). +* **OS packages:** RPM (NEVRA→purl:rpm), DEB (dpkg→purl:deb), APK (apk→purl:alpine), with **EVR/NVRA** preserved. +* **Secondary:** `cpe` retained for compatibility; advisory records may carry both. +* **Image/platform:** `oci:/@` for image‑level advisories (rare). +* **Unmappable:** if a source is non‑deterministic, keep native string under `productKey="native::"` and mark **non‑joinable**. --- -## 3) Data Storage — **MongoDB** (single source of truth) +## 3) Source families & precedence -**Database**: `feedser` -**Write concern**: `majority` for merge/export state, `acknowledged` for raw docs. -**Collections** (with “flags”/resume points): +### 3.1 Families -* `source` - * `_id`, `name`, `type`, `baseUrl`, `auth`, `notes`. -* `source_state` - * Keys: `sourceName` (unique), `enabled`, `cursor`, `lastSuccess`, `failCount`, `backoffUntil`, `paceOverrides`, `paused`. - * Drives incremental fetch/parse/map resume and operator pause/pace controls. -* `document` - * `_id`, `sourceName`, `uri`, `fetchedAt`, `sha256`, `contentType`, `status`, `metadata`, `gridFsId`, `etag`, `lastModified`. - * Index `{sourceName:1, uri:1}` unique; optional TTL for superseded versions. -* `dto` - * `_id`, `sourceName`, `documentId`, `schemaVer`, `payload` (BSON), `validatedAt`. - * Index `{sourceName:1, documentId:1}`. -* `advisory` - * `_id`, `advisoryKey`, `title`, `summary`, `lang`, `published`, `modified`, `severity`, `exploitKnown`. - * Unique `{advisoryKey:1}` plus indexes on `modified` and `published`. -* `alias` - * `advisoryId`, `scheme`, `value` with index `{scheme:1, value:1}`. -* `affected` - * `advisoryId`, `platform`, `name`, `versionRange`, `cpe`, `purl`, `fixedBy`, `introducedVersion`. - * Index `{platform:1, name:1}`, `{advisoryId:1}`. -* `reference` - * `advisoryId`, `url`, `kind`, `sourceTag` (e.g., advisory/patch/kb). -* Flags collections: `kev_flag`, `ru_flags`, `jp_flags`, `psirt_flags` keyed by `advisoryId`. -* `merge_event` - * `_id`, `advisoryKey`, `beforeHash`, `afterHash`, `mergedAt`, `inputs` (document ids). -* `export_state` - * `_id` (`json`/`trivydb`), `baseExportId`, `baseDigest`, `lastFullDigest`, `lastDeltaDigest`, `exportCursor`, `targetRepo`, `exporterVersion`. -* `locks` - * `_id` (`jobKey`), `holder`, `acquiredAt`, `heartbeatAt`, `leaseMs`, `ttlAt` (TTL index cleans dead locks). -* `jobs` - * `_id`, `type`, `args`, `state`, `startedAt`, `endedAt`, `error`, `owner`, `heartbeatAt`, `timeoutMs`. +* **Vendor PSIRTs**: Microsoft, Oracle, Cisco, Adobe, Apple, VMware, Chromium… +* **Linux distros**: Red Hat, SUSE, Ubuntu, Debian, Alpine… +* **OSS ecosystems**: OSV, GHSA (GitHub Security Advisories), PyPI, npm, Maven, NuGet, Go. +* **CERTs / national CSIRTs**: CISA (KEV, ICS), JVN, ACSC, CCCS, KISA, CERT‑FR/BUND, etc. -**GridFS buckets**: `fs.documents` for raw large payloads; referenced by `document.gridFsId`. +### 3.2 Precedence (when claims conflict) + +1. **Vendor PSIRT** (authoritative for their product). +2. **Distro** (authoritative for packages they ship, including backports). +3. **Ecosystem** (OSV/GHSA) for library semantics. +4. **CERTs/aggregators** for enrichment (KEV/known exploited). + +> Precedence affects **Affected** ranges and **fixed** info; **severity** is normalized to the **maximum** credible severity unless policy overrides. Conflicts are retained with **source provenance**. --- -## 4) Job & Scheduler Model +## 4) Connectors & normalization -* Scheduler stores cron expressions per source/exporter in config; persists next-run pointers in Mongo. -* Jobs acquire locks (`locks` collection) to ensure singleton execution per source/exporter. -* Supports manual triggers via API endpoints (`POST /jobs/{type}`) and pause/resume toggles per source. - ---- - -## 5) Connector Contracts - -Connectors implement: +### 4.1 Connector contract ```csharp public interface IFeedConnector { - string SourceName { get; } - Task FetchAsync(IServiceProvider sp, CancellationToken ct); - Task ParseAsync(IServiceProvider sp, CancellationToken ct); - Task MapAsync(IServiceProvider sp, CancellationToken ct); + string SourceName { get; } + Task FetchAsync(IServiceProvider sp, CancellationToken ct); // -> document collection + Task ParseAsync(IServiceProvider sp, CancellationToken ct); // -> dto collection (validated) + Task MapAsync(IServiceProvider sp, CancellationToken ct); // -> advisory/alias/affected/reference } ``` -* Fetch populates `document` rows respecting rate limits, conditional GET, and `source_state.cursor`. -* Parse validates schema (JSON Schema, XSD) and writes sanitized DTO payloads. -* Map produces canonical advisory rows + provenance entries; must be idempotent. -* Base helpers in `StellaOps.Feedser.Source.Common` provide HTTP clients, retry policies, and watermark utilities. +* **Fetch**: windowed (cursor), conditional GET (ETag/Last‑Modified), retry/backoff, rate limiting. +* **Parse**: schema validation (JSON Schema, XSD/CSAF), content type checks; write **DTO** with normalized casing. +* **Map**: build canonical records; all outputs carry **provenance** (doc digest, URI, anchors). + +### 4.2 Version range normalization + +* **SemVer** ecosystems (npm, pypi, maven, nuget, golang): normalize to `introduced`/`fixed` semver ranges (use `~`, `^`, `<`, `>=` canonicalized to intervals). +* **RPM EVR**: `epoch:version-release` with `rpmvercmp` semantics; store raw EVR strings and also **computed order keys** for query. +* **DEB**: dpkg version comparison semantics mirrored; store computed keys. +* **APK**: Alpine version semantics; compute order keys. +* **Generic**: if provider uses text, retain raw; do **not** invent ranges. + +### 4.3 Severity & CVSS + +* Normalize **CVSS v2/v3/v4** where available (vector, baseScore, severity). +* If multiple CVSS sources exist, track them all; **effective severity** defaults to **max** by policy (configurable). +* **ExploitKnown** toggled by KEV and equivalent sources; store **evidence** (source, date). --- -## 6) Merge & Normalization +## 5) Merge engine -* Canonical model stored in `StellaOps.Feedser.Models` with serialization contracts used by storage/export layers. -* `StellaOps.Feedser.Normalization` handles NEVRA/EVR/PURL range parsing, CVSS normalization, localization. -* `StellaOps.Feedser.Merge` builds alias graphs keyed by CVE first, then falls back to vendor/regional IDs. -* Precedence rules: PSIRT/OVAL overrides generic ranges; KEV only toggles exploitation; regional feeds enrich severity but don’t override vendor truth. -* Determinism enforced via canonical JSON hashing logged in `merge_event`. +### 5.1 Keying & identity + +* Identity graph: **CVE** is primary node; vendor/distro IDs resolved via **Alias** edges (from connectors and Feedser’s alias tables). +* `advisoryKey` is the canonical primary key (CVE if present, else vendor/distro key). + +### 5.2 Merge algorithm (deterministic) + +1. **Gather** all rows for `advisoryKey` (across sources). +2. **Select title/summary** by precedence source (vendor>distro>ecosystem>cert). +3. **Union aliases** (dedupe by scheme+value). +4. **Merge `Affected`** with rules: + + * Prefer **vendor** ranges for vendor products; prefer **distro** for **distro‑shipped** packages. + * If both exist for same `productKey`, keep **both**; mark `sourceTag` and `precedence` so **Policy** can decide. + * Never collapse range semantics across different families (e.g., rpm EVR vs semver). +5. **CVSS/severity**: record all CVSS sets; compute **effectiveSeverity** = max (unless policy override). +6. **References**: union with type precedence (advisory > patch > kb > exploit > blog); dedupe by URL; preserve `sourceTag`. +7. Produce **canonical JSON**; compute **afterHash**; store **MergeEvent** with inputs and hashes. + +> The merge is **pure** given inputs. Any change in inputs or precedence matrices changes the **hash** predictably. + +--- + +## 6) Storage schema (MongoDB) + +**Collections & indexes** + +* `source` `{_id, type, baseUrl, enabled, notes}` +* `source_state` `{sourceName(unique), enabled, cursor, lastSuccess, backoffUntil, paceOverrides}` +* `document` `{_id, sourceName, uri, fetchedAt, sha256, contentType, status, metadata, gridFsId?, etag?, lastModified?}` + + * Index: `{sourceName:1, uri:1}` unique, `{fetchedAt:-1}` +* `dto` `{_id, sourceName, documentId, schemaVer, payload, validatedAt}` + + * Index: `{sourceName:1, documentId:1}` +* `advisory` `{_id, advisoryKey, title, summary, published, modified, severity, cvss, exploitKnown, sources[]}` + + * Index: `{advisoryKey:1}` unique, `{modified:-1}`, `{severity:1}`, text index (title, summary) +* `alias` `{advisoryId, scheme, value}` + + * Index: `{scheme:1,value:1}`, `{advisoryId:1}` +* `affected` `{advisoryId, productKey, rangeKind, introduced?, fixed?, arch?, distro?, ecosystem?}` + + * Index: `{productKey:1}`, `{advisoryId:1}`, `{productKey:1, rangeKind:1}` +* `reference` `{advisoryId, url, kind, sourceTag}` + + * Index: `{advisoryId:1}`, `{kind:1}` +* `merge_event` `{advisoryKey, beforeHash, afterHash, mergedAt, inputs[]}` + + * Index: `{advisoryKey:1, mergedAt:-1}` +* `export_state` `{_id(exportKind), baseExportId?, baseDigest?, lastFullDigest?, lastDeltaDigest?, cursor, files[]}` +* `locks` `{_id(jobKey), holder, acquiredAt, heartbeatAt, leaseMs, ttlAt}` (TTL cleans dead locks) +* `jobs` `{_id, type, args, state, startedAt, heartbeatAt, endedAt, error}` + +**GridFS buckets**: `fs.documents` for raw payloads. --- ## 7) Exporters -* JSON exporter mirrors `aquasecurity/vuln-list` layout with deterministic ordering and reproducible timestamps. -* Trivy DB exporter shells out to `trivy-db build`, produces Bolt archives, and reuses unchanged blobs from the last full baseline when running in delta mode. The exporter annotates `metadata.json` with `mode`, `baseExportId`, `baseManifestDigest`, `resetBaseline`, and `delta.changedFiles[]`/`delta.removedPaths[]`, and honours `publishFull` / `publishDelta` (ORAS) plus `includeFull` / `includeDelta` (offline bundle) toggles. -* `StellaOps.Feedser.Storage.Mongo` provides cursors for delta exports based on `export_state.exportCursor` and the persisted per-file manifest (`export_state.files`). -* Export jobs produce OCI tarballs (layer media type `application/vnd.aquasec.trivy.db.layer.v1.tar+gzip`) and optionally push via ORAS; `metadata.json` accompanies each layout so mirrors can decide between full refreshes and deltas. +### 7.1 Deterministic JSON (vuln‑list style) + +* Folder structure mirroring `////…` with one JSON per advisory; deterministic ordering, stable timestamps, normalized whitespace. +* `manifest.json` lists all files with SHA‑256 and a top‑level **export digest**. + +### 7.2 Trivy DB exporter + +* Builds Bolt DB archives compatible with Trivy; supports **full** and **delta** modes. +* In delta, unchanged blobs are reused from the base; metadata captures: + + ``` + { + "mode": "delta|full", + "baseExportId": "...", + "baseManifestDigest": "sha256:...", + "changed": ["path1", "path2"], + "removed": ["path3"] + } + ``` +* Optional ORAS push (OCI layout) for registries. +* Offline kit bundles include Trivy DB + JSON tree + export manifest. + +### 7.3 Hand‑off to Signer/Attestor (optional) + +* On export completion, if `attest: true` is set in job args, Feedser **posts** the artifact metadata to **Signer**/**Attestor**; Feedser itself **does not** hold signing keys. +* Export record stores returned `{ uuid, index, url }` from **Rekor v2**. --- -## 8) Observability +## 8) REST APIs -* Serilog structured logging with enrichment fields (`source`, `uri`, `stage`, `durationMs`). -* OpenTelemetry traces around fetch/parse/map/export; metrics for rate limit hits, schema failures, dedupe ratios, package size. Connector HTTP metrics are emitted via the shared `feedser.source.http.*` instruments tagged with `feedser.source=` so per-source dashboards slice on that label instead of bespoke metric names. -* Prometheus scraping endpoint served by WebService. +All under `/api/v1/feedser`. + +**Health & status** + +``` +GET /healthz | /readyz +GET /status → sources, last runs, export cursors +``` + +**Sources & jobs** + +``` +GET /sources → list of configured sources +POST /sources/{name}/trigger → { jobId } +POST /sources/{name}/pause | /resume → toggle +GET /jobs/{id} → job status +``` + +**Exports** + +``` +POST /exports/json { full?:bool, force?:bool, attest?:bool } → { exportId, digest, rekor? } +POST /exports/trivy { full?:bool, force?:bool, publish?:bool, attest?:bool } → { exportId, digest, rekor? } +GET /exports/{id} → export metadata (kind, digest, createdAt, rekor?) +``` + +**Search (operator debugging)** + +``` +GET /advisories/{key} +GET /advisories?scheme=CVE&value=CVE-2025-12345 +GET /affected?productKey=pkg:rpm/openssl&limit=100 +``` + +**AuthN/Z:** Authority tokens (OpTok) with roles: `feedser.read`, `feedser.admin`, `feedser.export`. --- -## 9) Security Considerations +## 9) Configuration (YAML) -* Offline-first: connectors only reach allowlisted hosts. -* BDU LLM fallback gated by config flag; logs audit trail with confidence score. -* No secrets written to logs; secrets loaded via environment or mounted files. -* Signing handled outside Feedser pipeline. +```yaml +feedser: + mongo: { uri: "mongodb://mongo/feedser" } + s3: + endpoint: "http://minio:9000" + bucket: "stellaops-feedser" + scheduler: + windowSeconds: 30 + maxParallelSources: 4 + sources: + - name: redhat + kind: csaf + baseUrl: https://access.redhat.com/security/data/csaf/v2/ + signature: { type: pgp, keys: [ "…redhat PGP…" ] } + enabled: true + windowDays: 7 + - name: suse + kind: csaf + baseUrl: https://ftp.suse.com/pub/projects/security/csaf/ + signature: { type: pgp, keys: [ "…suse PGP…" ] } + - name: ubuntu + kind: usn-json + baseUrl: https://ubuntu.com/security/notices.json + signature: { type: none } + - name: osv + kind: osv + baseUrl: https://api.osv.dev/v1/ + signature: { type: none } + - name: ghsa + kind: ghsa + baseUrl: https://api.github.com/graphql + auth: { tokenRef: "env:GITHUB_TOKEN" } + exporters: + json: + enabled: true + output: s3://stellaops-feedser/json/ + trivy: + enabled: true + mode: full + output: s3://stellaops-feedser/trivy/ + oras: + enabled: false + repo: ghcr.io/org/feedser + precedence: + vendorWinsOverDistro: true + distroWinsOverOsv: true + severity: + policy: max # or 'vendorPreferred' / 'distroPreferred' +``` --- -## 10) Deployment Notes +## 10) Security & compliance + +* **Outbound allowlist** per connector (domains, protocols); proxy support; TLS pinning where possible. +* **Signature verification** for raw docs (PGP/cosign/x509) with results stored in `document.metadata.sig`. Docs failing verification may still be ingested but flagged; **merge** can down‑weight or ignore them by config. +* **No secrets in logs**; auth material via `env:` or mounted files; HTTP redaction of `Authorization` headers. +* **Multi‑tenant**: per‑tenant DBs or prefixes; per‑tenant S3 prefixes; tenant‑scoped API tokens. +* **Determinism**: canonical JSON writer; export digests stable across runs given same inputs. + +--- + +## 11) Performance targets & scale + +* **Ingest**: ≥ 5k documents/min on 4 cores (CSAF/OpenVEX/JSON). +* **Normalize/map**: ≥ 50k `Affected` rows/min on 4 cores. +* **Merge**: ≤ 10 ms P95 per advisory at steady‑state updates. +* **Export**: 1M advisories JSON in ≤ 90 s (streamed, zstd), Trivy DB in ≤ 60 s on 8 cores. +* **Memory**: hard cap per job; chunked streaming writers; backpressure to avoid GC spikes. + +**Scale pattern**: add Feedser replicas; Mongo scaling via indices and read/write concerns; GridFS only for oversized docs. + +--- + +## 12) Observability + +* **Metrics** + + * `feedser.fetch.docs_total{source}` + * `feedser.fetch.bytes_total{source}` + * `feedser.parse.failures_total{source}` + * `feedser.map.affected_total{source}` + * `feedser.merge.changed_total` + * `feedser.export.bytes{kind}` + * `feedser.export.duration_seconds{kind}` +* **Tracing** around fetch/parse/map/merge/export. +* **Logs**: structured with `source`, `uri`, `docDigest`, `advisoryKey`, `exportId`. + +--- + +## 13) Testing matrix + +* **Connectors:** fixture suites for each provider/format (happy path; malformed; signature fail). +* **Version semantics:** EVR vs dpkg vs semver edge cases (epoch bumps, tilde versions, pre‑releases). +* **Merge:** conflicting sources (vendor vs distro vs OSV); verify precedence & dual retention. +* **Export determinism:** byte‑for‑byte stable outputs across runs; digest equality. +* **Performance:** soak tests with 1M advisories; cap memory; verify backpressure. +* **API:** pagination, filters, RBAC, error envelopes (RFC 7807). +* **Offline kit:** bundle build & import correctness. + +--- + +## 14) Failure modes & recovery + +* **Source outages:** scheduler backs off with exponential delay; `source_state.backoffUntil`; alerts on staleness. +* **Schema drifts:** parse stage marks DTO invalid; job fails with clear diagnostics; connector version flags track supported schema ranges. +* **Partial exports:** exporters write to temp prefix; **manifest commit** is atomic; only then move to final prefix and update `export_state`. +* **Resume:** all stages idempotent; `source_state.cursor` supports window resume. + +--- + +## 15) Operator runbook (quick) + +* **Trigger all sources:** `POST /api/v1/feedser/sources/*/trigger` +* **Force full export JSON:** `POST /api/v1/feedser/exports/json { "full": true, "force": true }` +* **Force Trivy DB delta publish:** `POST /api/v1/feedser/exports/trivy { "full": false, "publish": true }` +* **Inspect advisory:** `GET /api/v1/feedser/advisories?scheme=CVE&value=CVE-2025-12345` +* **Pause noisy source:** `POST /api/v1/feedser/sources/osv/pause` + +--- + +## 16) Rollout plan + +1. **MVP**: Red Hat (CSAF), SUSE (CSAF), Ubuntu (USN JSON), OSV; JSON export. +2. **Add**: GHSA GraphQL, Debian (DSA HTML/JSON), Alpine secdb; Trivy DB export. +3. **Attestation hand‑off**: integrate with **Signer/Attestor** (optional). +4. **Scale & diagnostics**: provider dashboards, staleness alerts, export cache reuse. +5. **Offline kit**: end‑to‑end verified bundles for air‑gap. -* Default storage MongoDB; for air-gapped, bundle Mongo image + seeded data backup. -* Horizontal scale achieved via multiple web service instances sharing Mongo locks. -* Provide `feedser.yaml` template describing sources, rate limits, and export settings. diff --git a/docs/ARCHITECTURE_SCANNER.md b/docs/ARCHITECTURE_SCANNER.md new file mode 100644 index 00000000..abee0e0e --- /dev/null +++ b/docs/ARCHITECTURE_SCANNER.md @@ -0,0 +1,413 @@ +# component_architecture_scanner.md — **Stella Ops Scanner** (2025Q4) + +> **Scope.** Implementation‑ready architecture for the **Scanner** subsystem: WebService, Workers, analyzers, SBOM assembly (inventory & usage), per‑layer caching, three‑way diffs, artifact catalog (MinIO+Mongo), attestation hand‑off, and scale/security posture. This document is the contract between the scanning plane and everything else (Policy, Vexer, Feedser, UI, CLI). + +--- + +## 0) Mission & boundaries + +**Mission.** Produce **deterministic**, **explainable** SBOMs and diffs for container images and filesystems, quickly and repeatedly, without guessing. Emit two views: **Inventory** (everything present) and **Usage** (entrypoint closure + actually linked libs). Attach attestations through **Signer→Attestor→Rekor v2**. + +**Boundaries.** + +* Scanner **does not** produce PASS/FAIL. The backend (Policy + Vexer + Feedser) decides presentation and verdicts. +* Scanner **does not** keep third‑party SBOM warehouses. It may **bind** to existing attestations for exact hashes. +* Core analyzers are **deterministic** (no fuzzy identity). Optional heuristic plug‑ins (e.g., patch‑presence) run under explicit flags and never contaminate the core SBOM. + +--- + +## 1) Solution & project layout + +``` +src/ + ├─ StellaOps.Scanner.WebService/ # REST control plane, catalog, diff, exports + ├─ StellaOps.Scanner.Worker/ # queue consumer; executes analyzers + ├─ StellaOps.Scanner.Models/ # DTOs, evidence, graph nodes, CDX/SPDX adapters + ├─ StellaOps.Scanner.Storage/ # Mongo repositories; MinIO object client; ILM/GC + ├─ StellaOps.Scanner.Queue/ # queue abstraction (Redis/NATS/RabbitMQ) + ├─ StellaOps.Scanner.Cache/ # layer cache; file CAS; bloom/bitmap indexes + ├─ StellaOps.Scanner.EntryTrace/ # ENTRYPOINT/CMD → terminal program resolver (shell AST) + ├─ StellaOps.Scanner.Analyzers.OS.[Apk|Dpkg|Rpm]/ + ├─ StellaOps.Scanner.Analyzers.Lang.[Java|Node|Python|Go|DotNet|Rust]/ + ├─ StellaOps.Scanner.Analyzers.Native.[ELF|PE|MachO]/ # PE/Mach-O planned (M2) + ├─ StellaOps.Scanner.Emit.CDX/ # CycloneDX (JSON + Protobuf) + ├─ StellaOps.Scanner.Emit.SPDX/ # SPDX 3.0.1 JSON + ├─ StellaOps.Scanner.Diff/ # image→layer→component three‑way diff + ├─ StellaOps.Scanner.Index/ # BOM‑Index sidecar (purls + roaring bitmaps) + ├─ StellaOps.Scanner.Tests.* # unit/integration/e2e fixtures + └─ tools/ + ├─ StellaOps.Scanner.Sbomer.BuildXPlugin/ # BuildKit generator (image referrer SBOMs) + └─ StellaOps.Scanner.Sbomer.DockerImage/ # CLI‑driven scanner container +``` + +**Runtime form‑factor:** two deployables + +* **Scanner.WebService** (stateless REST) +* **Scanner.Worker** (N replicas; queue‑driven) + +--- + +## 2) External dependencies + +* **OCI registry** with **Referrers API** (discover attached SBOMs/signatures). +* **MinIO** (S3‑compatible) for SBOM artifacts; **Object Lock** for immutable classes; **ILM** for TTL. +* **MongoDB** for catalog, job state, diffs, ILM rules. +* **Queue** (Redis Streams/NATS/RabbitMQ). +* **Authority** (on‑prem OIDC) for **OpToks** (DPoP/mTLS). +* **Signer** + **Attestor** (+ **Fulcio/KMS** + **Rekor v2**) for DSSE + transparency. + +--- + +## 3) Contracts & data model + +### 3.1 Evidence‑first component model + +**Nodes** + +* `Image`, `Layer`, `File` +* `Component` (`purl?`, `name`, `version?`, `type`, `id` — may be `bin:{sha256}`) +* `Executable` (ELF/PE/Mach‑O), `Library` (native or managed), `EntryScript` (shell/launcher) + +**Edges** (all carry **Evidence**) + +* `contains(Image|Layer → File)` +* `installs(PackageDB → Component)` (OS database row) +* `declares(InstalledMetadata → Component)` (dist‑info, pom.properties, deps.json…) +* `links_to(Executable → Library)` (ELF `DT_NEEDED`, PE imports) +* `calls(EntryScript → Program)` (file:line from shell AST) +* `attests(Rekor → Component|Image)` (SBOM/predicate binding) +* `bound_from_attestation(Component_attested → Component_observed)` (hash equality proof) + +**Evidence** + +``` +{ source: enum, locator: (path|offset|line), sha256?, method: enum, timestamp } +``` + +No confidences. Either a fact is proven with listed mechanisms, or it is not claimed. + +### 3.2 Catalog schema (Mongo) + +* `artifacts` + + ``` + { _id, type: layer-bom|image-bom|diff|index, + format: cdx-json|cdx-pb|spdx-json, + bytesSha256, size, rekor: { uuid,index,url }?, + ttlClass, immutable, refCount, createdAt } + ``` +* `images { imageDigest, repo, tag?, arch, createdAt, lastSeen }` +* `layers { layerDigest, mediaType, size, createdAt, lastSeen }` +* `links { fromType, fromDigest, artifactId }` // image/layer -> artifact +* `jobs { _id, kind, args, state, startedAt, heartbeatAt, endedAt, error }` +* `lifecycleRules { ruleId, scope, ttlDays, retainIfReferenced, immutable }` + +### 3.3 Object store layout (MinIO) + +``` +layers//sbom.cdx.json.zst +layers//sbom.spdx.json.zst +images//inventory.cdx.pb # CycloneDX Protobuf +images//usage.cdx.pb +indexes//bom-index.bin # purls + roaring bitmaps +diffs/_/diff.json.zst +attest/.dsse.json # DSSE bundle (cert chain + Rekor proof) +``` + +--- + +## 4) REST API (Scanner.WebService) + +All under `/api/v1/scanner`. Auth: **OpTok** (DPoP/mTLS); RBAC scopes. + +``` +POST /scans { imageRef|digest, force?:bool } → { scanId } +GET /scans/{id} → { status, imageDigest, artifacts[], rekor? } +GET /sboms/{imageDigest} ?format=cdx-json|cdx-pb|spdx-json&view=inventory|usage → bytes +GET /diff?old=&new=&view=inventory|usage → diff.json +POST /exports { imageDigest, format, view, attest?:bool } → { artifactId, rekor? } +POST /reports { imageDigest, policyRevision? } → { reportId, rekor? } # delegates to backend policy+vex +GET /catalog/artifacts/{id} → { meta } +GET /healthz | /readyz | /metrics +``` + +--- + +## 5) Execution flow (Worker) + +### 5.1 Acquire & verify + +1. **Resolve image** (prefer `repo@sha256:…`). +2. **(Optional) verify image signature** per policy (cosign). +3. **Pull blobs**, compute layer digests; record metadata. + +### 5.2 Layer union FS + +* Apply whiteouts; materialize final filesystem; map **file → first introducing layer**. +* Windows layers (MSI/SxS/GAC) planned in **M2**. + +### 5.3 Evidence harvest (parallel analyzers; deterministic only) + +**A) OS packages** + +* **apk**: `/lib/apk/db/installed` +* **dpkg**: `/var/lib/dpkg/status`, `/var/lib/dpkg/info/*.list` +* **rpm**: `/var/lib/rpm/Packages` (via librpm or parser) +* Record `name`, `version` (epoch/revision), `arch`, source package where present, and **declared file lists**. + +**B) Language ecosystems (installed state only)** + +* **Java**: `META-INF/maven/*/pom.properties`, MANIFEST → `pkg:maven/...` +* **Node**: `node_modules/**/package.json` → `pkg:npm/...` +* **Python**: `*.dist-info/{METADATA,RECORD}` → `pkg:pypi/...` +* **Go**: Go **buildinfo** in binaries → `pkg:golang/...` +* **.NET**: `*.deps.json` + assembly metadata → `pkg:nuget/...` +* **Rust**: crates only when **explicitly present** (embedded metadata or cargo/registry traces); otherwise binaries reported as `bin:{sha256}`. + +> **Rule:** We only report components proven **on disk** with authoritative metadata. Lockfiles are evidence only. + +**C) Native link graph** + +* **ELF**: parse `PT_INTERP`, `DT_NEEDED`, RPATH/RUNPATH, **GNU symbol versions**; map **SONAMEs** to file paths; link executables → libs. +* **PE/Mach‑O** (planned M2): import table, delay‑imports; version resources; code signatures. +* Map libs back to **OS packages** if possible (via file lists); else emit `bin:{sha256}` components. + +**D) EntryTrace (ENTRYPOINT/CMD → terminal program)** + +* Read image config; parse shell (POSIX/Bash subset) with AST: `source`/`.` includes; `case/if`; `exec`/`command`; `run‑parts`. +* Resolve commands via **PATH** within the **built rootfs**; follow language launchers (Java/Node/Python) to identify the terminal program (ELF/JAR/venv script). +* Record **file:line** and choices for each hop; output chain graph. +* Unresolvable dynamic constructs are recorded as **unknown** edges with reasons (e.g., `$FOO` unresolved). + +**E) Attestation & SBOM bind (optional)** + +* For each **file hash** or **binary hash**, query local cache of **Rekor v2** indices; if an SBOM attestation is found for **exact hash**, bind it to the component (origin=`attested`). +* For the **image** digest, likewise bind SBOM attestations (build‑time referrers). + +### 5.4 Component normalization (exact only) + +* Create `Component` nodes only with deterministic identities: purl, or **`bin:{sha256}`** for unlabeled binaries. +* Record **origin** (OS DB, installed metadata, linker, attestation). + +### 5.5 SBOM assembly & emit + +* **Per‑layer SBOM fragments**: components introduced by the layer (+ relationships). +* **Image SBOMs**: merge fragments; refer back to them via **CycloneDX BOM‑Link** (or SPDX ExternalRef). +* Emit both **Inventory** & **Usage** views. +* Serialize **CycloneDX JSON** and **CycloneDX Protobuf**; optionally **SPDX 3.0.1 JSON**. +* Build **BOM‑Index** sidecar: purl table + roaring bitmap; flag `usedByEntrypoint` components for fast backend joins. + +### 5.6 DSSE attestation (via Signer/Attestor) + +* WebService constructs **predicate** with `image_digest`, `stellaops_version`, `license_id`, `policy_digest?` (when emitting **final reports**), timestamps. +* Calls **Signer** (requires **OpTok + PoE**); Signer verifies **entitlement + scanner image integrity** and returns **DSSE bundle**. +* **Attestor** logs to **Rekor v2**; returns `{uuid,index,proof}` → stored in `artifacts.rekor`. + +--- + +## 6) Three‑way diff (image → layer → component) + +### 6.1 Keys & classification + +* Component key: **purl** when present; else `bin:{sha256}`. +* Diff classes: `added`, `removed`, `version_changed` (`upgraded|downgraded`), `metadata_changed` (e.g., origin from attestation vs observed). +* Layer attribution: for each change, resolve the **introducing/removing layer**. + +### 6.2 Algorithm (outline) + +``` +A = components(imageOld, key) +B = components(imageNew, key) + +added = B \ A +removed = A \ B +changed = { k in A∩B : version(A[k]) != version(B[k]) || origin changed } + +for each item in added/removed/changed: + layer = attribute_to_layer(item, imageOld|imageNew) + usageFlag = usedByEntrypoint(item, imageNew) +emit diff.json (grouped by layer with badges) +``` + +Diffs are stored as artifacts and feed **UI** and **CLI**. + +--- + +## 7) Build‑time SBOMs (fast CI path) + +**Scanner.Sbomer.BuildXPlugin** can act as a BuildKit **generator**: + +* During `docker buildx build --attest=type=sbom,generator=stellaops/sbom-indexer`, run analyzers on the build context/output; attach SBOMs as OCI **referrers** to the built image. +* Optionally request **Signer/Attestor** to produce **Stella Ops‑verified** attestation immediately; else, Scanner.WebService can verify and re‑attest post‑push. +* Scanner.WebService trusts build‑time SBOMs per policy, enabling **no‑rescan** for unchanged bases. + +--- + +## 8) Configuration (YAML) + +```yaml +scanner: + queue: + kind: redis + url: "redis://queue:6379/0" + mongo: + uri: "mongodb://mongo/scanner" + s3: + endpoint: "http://minio:9000" + bucket: "stellaops" + objectLock: "governance" # or 'compliance' + analyzers: + os: { apk: true, dpkg: true, rpm: true } + lang: { java: true, node: true, python: true, go: true, dotnet: true, rust: true } + native: { elf: true, pe: false, macho: false } # PE/Mach-O in M2 + entryTrace: { enabled: true, shellMaxDepth: 64, followRunParts: true } + emit: + cdx: { json: true, protobuf: true } + spdx: { json: true } + compress: "zstd" + rekor: + url: "https://rekor-v2.internal" + signer: + url: "https://signer.internal" + limits: + maxParallel: 8 + perRegistryConcurrency: 2 + policyHints: + verifyImageSignature: false + trustBuildTimeSboms: true +``` + +--- + +## 9) Scale & performance + +* **Parallelism**: per‑analyzer concurrency; bounded directory walkers; file CAS dedupe by sha256. +* **Distributed locks** per **layer digest** to prevent duplicate work across Workers. +* **Registry throttles**: per‑host concurrency budgets; exponential backoff on 429/5xx. +* **Targets**: + + * **Build‑time**: P95 ≤ 3–5 s on warmed bases (CI generator). + * **Post‑build delta**: P95 ≤ 10 s for 200 MB images with cache hit. + * **Emit**: CycloneDX Protobuf ≤ 150 ms for 5k components; JSON ≤ 500 ms. + * **Diff**: ≤ 200 ms for 5k vs 5k components. + +--- + +## 10) Security posture + +* **AuthN**: Authority‑issued short OpToks (DPoP/mTLS). +* **AuthZ**: scopes (`scanner.scan`, `scanner.export`, `scanner.catalog.read`). +* **mTLS** to **Signer**/**Attestor**; only **Signer** can sign. +* **No network fetches** during analysis (except registry pulls and optional Rekor index reads). +* **Sandboxing**: non‑root containers; read‑only FS; seccomp profiles; disable execution of scanned content. +* **Release integrity**: all first‑party images are **cosign‑signed**; Workers/WebService self‑verify at startup. + +--- + +## 11) Observability & audit + +* **Metrics**: + + * `scanner.jobs_inflight`, `scanner.scan_latency_seconds` + * `scanner.layer_cache_hits_total`, `scanner.file_cas_hits_total` + * `scanner.artifact_bytes_total{format}` + * `scanner.attestation_latency_seconds`, `scanner.rekor_failures_total` +* **Tracing**: spans for acquire→union→analyzers→compose→emit→sign→log. +* **Audit logs**: DSSE requests log `license_id`, `image_digest`, `artifactSha256`, `policy_digest?`, Rekor UUID on success. + +--- + +## 12) Testing matrix + +* **Determinism:** given same image + analyzers → byte‑identical **CDX Protobuf**; JSON normalized. +* **OS packages:** ground‑truth images per distro; compare to package DB. +* **Lang ecosystems:** sample images per ecosystem (Java/Node/Python/Go/.NET/Rust) with installed metadata; negative tests w/ lockfile‑only. +* **Native & EntryTrace:** ELF graph correctness; shell AST cases (includes, run‑parts, exec, case/if). +* **Diff:** layer attribution against synthetic two‑image sequences. +* **Performance:** cold vs warm cache; large `node_modules` and `site‑packages`. +* **Security:** ensure no code execution from image; fuzz parser inputs; path traversal resistance on layer extract. + +--- + +## 13) Failure modes & degradations + +* **Missing OS DB** (files exist, DB removed): record **files**; do **not** fabricate package components; emit `bin:{sha256}` where unavoidable; flag in evidence. +* **Unreadable metadata** (corrupt dist‑info): record file evidence; skip component creation; annotate. +* **Dynamic shell constructs**: mark unresolved edges with reasons (env var unknown) and continue; **Usage** view may be partial. +* **Registry rate limits**: honor backoff; queue job retries with jitter. +* **Signer refusal** (license/plan/version): scan completes; artifact produced; **no attestation**; WebService marks result as **unverified**. + +--- + +## 14) Optional plug‑ins (off by default) + +* **Patch‑presence detector** (signature‑based backport checks). Reads curated function‑level signatures from advisories; inspects binaries for patched code snippets to lower false‑positives for backported fixes. Runs as a sidecar analyzer that **annotates** components; never overrides core identities. +* **Runtime probes** (with Zastava): when allowed, compare **/proc//maps** (DSOs actually loaded) with static **Usage** view for precision. + +--- + +## 15) DevOps & operations + +* **HA**: WebService horizontal scale; Workers autoscale by queue depth & CPU; distributed locks on layers. +* **Retention**: ILM rules per artifact class (`short`, `default`, `compliance`); **Object Lock** for compliance artifacts (reports, signed SBOMs). +* **Upgrades**: bump **cache schema** when analyzer outputs change; WebService triggers refresh of dependent artifacts. +* **Backups**: Mongo (daily dumps); MinIO (versioned buckets, replication); Rekor v2 DB snapshots. + +--- + +## 16) CLI & UI touch points + +* **CLI**: `stellaops scan `, `stellaops diff --old --new`, `stellaops export`, `stellaops verify attestation `. +* **UI**: Scan detail shows **Inventory/Usage** toggles, **Diff by Layer**, **Attestation badge** (verified/unverified), Rekor link, and **EntryTrace** chain with file:line breadcrumbs. + +--- + +## 17) Roadmap (Scanner) + +* **M2**: Windows containers (MSI/SxS/GAC analyzers), PE/Mach‑O native analyzer, deeper Rust metadata. +* **M2**: Buildx generator GA (certified external registries), cross‑registry trust policies. +* **M3**: Patch‑presence plug‑in GA (opt‑in), cross‑image corpus clustering (evidence‑only; not identity). +* **M3**: Advanced EntryTrace (POSIX shell features breadth, busybox detection). + +--- + +### Appendix A — EntryTrace resolution (pseudo) + +```csharp +ResolveEntrypoint(ImageConfig cfg, RootFs fs): + cmd = Normalize(cfg.ENTRYPOINT, cfg.CMD) + stack = [ Script(cmd, path=FindOnPath(cmd[0], fs)) ] + visited = set() + + while stack not empty and depth < MAX: + cur = stack.pop() + if cur in visited: continue + visited.add(cur) + + if IsShellScript(cur.path): + ast = ParseShell(cur.path) + foreach directive in ast: + if directive is Source include: + p = ResolveInclude(include.path, cur.env, fs) + stack.push(Script(p)) + if directive is Exec call: + p = ResolveExec(call.argv[0], cur.env, fs) + stack.push(Program(p, argv=call.argv)) + if directive is Interpreter (python -m / node / java -jar): + term = ResolveInterpreterTarget(call, fs) + stack.push(Program(term)) + else: + return Terminal(cur.path) + + return Unknown(reason) +``` + +### Appendix B — BOM‑Index sidecar + +``` +struct Header { magic, version, imageDigest, createdAt } +vector purls +map components +optional map usedByEntrypoint +``` + diff --git a/docs/ARCHITECTURE_SIGNER.md b/docs/ARCHITECTURE_SIGNER.md new file mode 100644 index 00000000..fda371ea --- /dev/null +++ b/docs/ARCHITECTURE_SIGNER.md @@ -0,0 +1,418 @@ +# component_architecture_signer.md — **Stella Ops Signer** (2025Q4) + +> **Scope.** Implementation‑ready architecture for the **Signer**: the *only* service allowed to produce **Stella Ops‑verified** signatures over SBOMs and reports. It enforces **entitlement** (PoE), **release integrity** (scanner provenance), **sender‑constrained auth** (DPoP/mTLS), and emits **in‑toto/DSSE** bundles suitable for **Rekor v2** logging by the Attestor. Includes APIs, data flow, storage, quotas, security, and test matrices. + +--- + +## 0) Mission & boundaries + +**Mission.** Convert authenticated signing requests from trusted Stella Ops services into **verifiable** DSSE bundles while enforcing **license policy** and **supply‑chain integrity**. + +**Boundaries.** + +* **Signer does not push to Rekor** — it returns DSSE to the caller; **Attestor** logs to **Rekor v2**. +* **Signer does not compute PASS/FAIL** — it signs SBOMs/reports produced by Scanner/WebService after backend evaluation. +* **Signer is stateless for hot path** — long‑term storage is limited to audit events; all secrets/keys live in KMS/HSM or are ephemeral (keyless). + +--- + +## 1) Responsibilities (contract) + +1. **Authenticate** caller with **OpTok** (Authority OIDC, DPoP or mTLS‑bound). +2. **Authorize** scopes (`signer.sign`) + audience (`aud=signer`) + tenant/installation. +3. **Validate entitlement** via **PoE** (Proof‑of‑Entitlement) against Cloud Licensing `/license/introspect`. +4. **Verify release integrity** of the **scanner** image digest presented in the request: must be **cosign‑signed** by Stella Ops release key, discoverable via **OCI Referrers API**. +5. **Enforce plan & quotas** (concurrency/QPS/artifact size/rate caps). +6. **Mint signing identity**: + + * **Keyless** (default): get a short‑lived X.509 cert from **Fulcio** using the Signer’s OIDC identity and sign the DSSE. + * **Keyful** (optional): sign with an HSM/KMS key. +7. **Return DSSE bundle** (subject digests + predicate + cert chain or KMS key id). +8. **Audit** every decision; expose metrics. + +--- + +## 2) External dependencies + +* **Authority** (on‑prem OIDC): validates OpToks (JWKS/introspection) and DPoP/mTLS. +* **Licensing Service (cloud)**: `/license/introspect` to verify PoE (active, claims, expiry, revocation). +* **Fulcio** (Sigstore) *or* **KMS/HSM**: to obtain certs or perform signatures. +* **OCI Registry (Referrers API)**: to verify **scanner** image release signature. +* **Attestor**: downstream service that writes DSSE bundles to **Rekor v2**. +* **Config/state stores**: Redis (caches, rate buckets), Mongo/Postgres (audit log). + +--- + +## 3) API surface (mTLS; DPoP supported) + +Base path: `/api/v1/signer`. **All endpoints require**: + +* Access token (JWT) from **Authority** with `aud=signer`, `scope=signer.sign`. +* **Sender constraint**: DPoP proof per request or mTLS client cert. +* **PoE** presented as either: + + * **Client TLS cert** (if PoE is mTLS‑style) chained to Licensing CA, *or* + * **PoE JWT** (DPoP/mTLS‑bound) in `X-PoE` header or request body. + +### 3.1 `POST /sign/dsse` + +Request (JSON): + +```json +{ + "subject": [ + { "name": "s3://stellaops/images/sha256:.../inventory.cdx.pb", + "digest": { "sha256": "..." } } + ], + "predicateType": "https://stella-ops.org/attestations/sbom/1", + "predicate": { + "image_digest": "sha256:...", + "stellaops_version": "2.3.1 (2027.04)", + "license_id": "LIC-9F2A...", + "customer_id": "CUST-ACME", + "plan": "pro", + "policy_digest": "sha256:...", // optional for final reports + "views": ["inventory", "usage"], + "created": "2025-10-17T12:34:56Z" + }, + "scannerImageDigest": "sha256:sc-web-or-worker-digest", + "poe": { + "format": "jwt", // or "mtls" + "value": "eyJhbGciOi..." // PoE JWT when not using mTLS PoE + }, + "options": { + "signingMode": "keyless", // "keyless" | "kms" + "expirySeconds": 600, // cert lifetime hint (keyless) + "returnBundle": "dsse+cert" // dsse (default) | dsse+cert + } +} +``` + +Response 200: + +```json +{ + "bundle": { + "dsse": { "payloadType": "application/vnd.in-toto+json", "payload": "", "signatures": [ ... ] }, + "certificateChain": [ "-----BEGIN CERTIFICATE-----...", "... root ..." ], + "mode": "keyless", + "signingIdentity": { "issuer": "https://fulcio.internal", "san": "urn:stellaops:signer", "certExpiry": "2025-10-17T12:44:56Z" } + }, + "policy": { "plan": "pro", "maxArtifactBytes": 104857600, "qpsRemaining": 97 }, + "auditId": "a7c9e3f2-1b7a-4e87-8c3a-90d7d2c3ad12" +} +``` + +Errors (RFC 7807): + +* `401 invalid_token` (JWT/DPoP/mTLS failure) +* `403 entitlement_denied` (PoE invalid/revoked/expired; release year mismatch) +* `403 release_untrusted` (scanner image not Stella‑signed) +* `429 plan_throttled` (license plan caps) +* `413 artifact_too_large` (size cap) +* `400 invalid_request` (schema/predicate/type invalid) +* `500 signing_unavailable` (Fulcio/KMS outage) + +### 3.2 `GET /verify/referrers?imageDigest=` + +Checks whether the **image** at digest is signed by **Stella Ops release key**. + +Response: + +```json +{ "trusted": true, "signatures": [ { "type": "cosign", "digest": "sha256:...", "signedBy": "StellaOps Release 2027 Q2" } ] } +``` + +> **Note:** This endpoint is also used internally by Signer before issuing signatures. + +--- + +## 4) Validation pipeline (hot path) + +```mermaid +sequenceDiagram + autonumber + participant Client as Scanner.WebService + participant Auth as Authority (OIDC) + participant Sign as Signer + participant Lic as Licensing Service (cloud) + participant Reg as OCI Registry (Referrers) + participant Ful as Fulcio/KMS + + Client->>Sign: POST /sign/dsse (OpTok + DPoP/mTLS, PoE, request) + Note over Sign: 1) Validate OpTok, audience, scope, DPoP/mTLS binding + Sign->>Lic: /license/introspect(PoE) + Lic-->>Sign: { active, claims: {license_id, plan, valid_release_year, max_version}, exp } + Note over Sign: 2) Enforce plan/version window and revocation + + Sign->>Reg: Verify scannerImageDigest signed (Referrers + cosign) + Reg-->>Sign: OK with signer identity + Note over Sign: 3) Enforce release integrity + + Note over Sign: 4) Enforce quotas (QPS/concurrency/size) + Sign->>Ful: Mint cert (keyless) or sign via KMS + Ful-->>Sign: Cert or signature + + Sign-->>Client: DSSE bundle (+cert chain), policy counters, auditId +``` + +**DPoP nonce dance (when enabled for high‑value ops):** + +* If DPoP proof lacks a valid nonce, Signer replies `401` with `WWW-Authenticate: DPoP error="use_dpop_nonce", dpop_nonce=""`. +* Client retries with new proof including the nonce; Signer validates nonce and `jti` uniqueness (Redis TTL cache). + +--- + +## 5) Entitlement enforcement (PoE) + +* **Accepted forms**: + + * **mTLS PoE**: client presents a **PoE client cert** at TLS handshake; Signer validates chain to **Licensing CA** (CA bundle configured) and calls `/license/introspect` with cert thumbprint + serial. + * **JWT PoE**: `X-PoE` bearer token (DPoP/mTLS‑bound) is validated (sig + `cnf`) locally (Licensing JWKS) and then **introspected** for status and claims. + +* **Claims required**: + + * `license_id`, `plan` (free|pro|enterprise|gov), `valid_release_year`, `max_version`, `exp`. + * Optional: `tenant_id`, `customer_id`, `entitlements[]`. + +* **Enforcements**: + + * Reject if **revoked**, **expired**, **plan mismatch** or **release outside window** (`stellaops_version` in predicate exceeds `max_version` or release date beyond `valid_release_year`). + * Apply plan **throttles** (QPS/concurrency/artifact bytes) via token‑bucket in Redis keyed by `license_id`. + +--- + +## 6) Release integrity (scanner provenance) + +* **Input**: `scannerImageDigest` representing the actual Scanner component that produced the artifact. + +* **Check**: + + 1. Use **OCI Referrers API** to enumerate signatures of that digest. + 2. Verify **cosign** signatures against the configured **Stella Ops Release** keyring (keyless Fulcio roots *or* keyful public keys). + 3. Optionally require Rekor inclusion for those signatures. + +* **Policy**: + + * If not signed by an authorized **Stella Ops Release** identity → **deny**. + * If signed but **release year** > PoE `valid_release_year` → **deny**. + +* **Cache**: LRU of digest → verification result (TTL 10–30 min) to avoid registry thrash. + +--- + +## 7) Signing modes + +### 7.1 Keyless (default; Sigstore Fulcio) + +* Signer authenticates to **Fulcio** using its on‑prem OIDC identity (client credentials) and requests a **short‑lived cert** (5–10 min). +* Generates **ephemeral keypair**, gets cert for the public key, signs DSSE with the **private key**. +* DSSE **bundle** includes **certificate chain**; verifiers validate to Fulcio root. + +### 7.2 Keyful (optional; KMS/HSM) + +* Signer uses a configured **KMS** key (AWS KMS, GCP KMS, Azure Key Vault, Vault Transit, or HSM). +* DSSE bundle includes **key metadata** (kid, cert chain if x509). +* Recommended for FIPS/sovereign environments. + +--- + +## 8) Predicates & schema + +Supported **predicate types** (extensible): + +* `https://stella-ops.org/attestations/sbom/1` (SBOM emissions) +* `https://stella-ops.org/attestations/report/1` (final PASS/FAIL reports) +* `https://stella-ops.org/attestations/vex-export/1` (Vexer exports; optional) + +**Validation**: + +* JSON‑Schema per predicate type; **canonical property order**. +* `subject[*].digest` must include `sha256`. +* `predicate.stellaops_version` must parse and match policy windows. + +--- + +## 9) Quotas & throttling + +Per `license_id` (from PoE): + +* **QPS** (token bucket), **concurrency** (semaphore), **artifact bytes** (sliding window). +* On exceed → `429 plan_throttled` with `Retry-After`. +* Free/community plan may also receive **randomized delay** to disincentivize farmed signing. + +--- + +## 10) Storage & caches + +* **Redis**: + + * DPoP nonce & `jti` replay cache (TTL ≤ 10 min). + * PoE introspection cache (short TTL, e.g., 60–120 s). + * Release‑verify cache (`scannerImageDigest` → { trusted, ts }). + +* **Audit store** (Mongo or Postgres): `signer.audit_events` + +``` +{ _id, ts, tenantId, installationId, licenseId, customerId, + plan, actor{sub,cnf}, request{predicateType, subjectSha256[], imageDigest}, + poe{type, thumbprint|jwtKid, exp, introspectSnapshot}, + release{digest, signerId, policy}, + mode: "keyless"|"kms", + result: "success"|"deny:"|"error:", + bundleSha256? } +``` + +* **Config**: Stella Ops release signing keyring, Fulcio roots, Licensing CA bundle. + +--- + +## 11) Security & privacy + +* **mTLS** on all Signer endpoints. +* **No bearer fallbacks** — DPoP/mTLS enforced for `aud=signer`. +* **PoE** is never persisted beyond audit snapshots (minimized fields). +* **Secrets**: no long‑lived private keys on disk (keyless) or handled via KMS APIs. +* **Input hardening**: schema‑validate predicates; cap payload sizes; zstd/gzip decompression bombs guarded. +* **Logging**: redact PoE JWTs, access tokens, DPoP proofs; log only hashes and identifiers. + +--- + +## 12) Metrics & observability + +* `signer.requests_total{result}` +* `signer.latency_seconds{stage=auth|introspect|release_verify|sign}` +* `signer.poe_failures_total{reason}` +* `signer.release_verify_failures_total{reason}` +* `signer.plan_throttle_total{license_id}` +* `signer.bundle_bytes_total` +* `signer.keyless_certs_issued_total` / `signer.kms_sign_total` +* OTEL traces across stages; correlation id (`auditId`) returned to client. + +--- + +## 13) Configuration (YAML) + +```yaml +signer: + listen: "https://0.0.0.0:8443" + authority: + issuer: "https://authority.internal" + jwksUrl: "https://authority.internal/jwks" + require: "dpop" # "dpop" | "mtls" + poe: + mode: "both" # "jwt" | "mtls" | "both" + licensing: + introspectUrl: "https://www.stella-ops.org/api/v1/license/introspect" + caBundle: "/etc/ssl/licensing-ca.pem" + cacheTtlSeconds: 90 + release: + referrers: + allowRekorVerified: true + keyrings: + - type: "cosign-keyless" + fulcioRoots: ["/etc/fulcio/root.pem"] + identities: + - san: "mailto:release@stella-ops.org" + - san: "https://sigstore.dev/oidc/stellaops" + signing: + mode: "keyless" # "keyless" | "kms" + fulcio: + issuer: "https://fulcio.internal" + oidcClientId: "signer" + oidcClientSecretRef: "env:FULCIO_CLIENT_SECRET" + certTtlSeconds: 600 + kms: + provider: "aws-kms" + keyId: "arn:aws:kms:...:key/..." + quotas: + default: + qps: 100 + concurrency: 20 + maxArtifactBytes: 104857600 + free: + qps: 5 + concurrency: 1 + maxArtifactBytes: 1048576 +``` + +--- + +## 14) Testing matrix + +* **Auth & DPoP**: bad `aud`, wrong `jkt`, replayed `jti`, missing nonce, mTLS mismatch. +* **PoE**: expired, revoked, plan mismatch, release year gate, max_version gate. +* **Release verify**: unsigned digest, wrong signer, Rekor‑absent (when required), referrers unreachable. +* **Signing**: Fulcio outage; KMS timeouts; bundle correctness (verifier harness). +* **Quotas**: burst above QPS, artifact over size, concurrency overflow. +* **Schema**: invalid predicate types/required fields. +* **Determinism**: same request → identical DSSE (aside from cert validity period). +* **Perf**: P95 end‑to‑end under 120 ms with caches warm (excluding network to Fulcio). + +--- + +## 15) Failure modes & responses + +| Failure | HTTP | Problem type | Notes | +| ----------------------- | ---- | --------------------- | -------------------------------------------- | +| Invalid OpTok / DPoP | 401 | `invalid_token` | `WWW-Authenticate` with DPoP nonce if needed | +| PoE invalid/revoked | 403 | `entitlement_denied` | Include `license_id` (hashed) and reason | +| Scanner image untrusted | 403 | `release_untrusted` | Include digest and required identity | +| Plan throttle | 429 | `plan_throttled` | Include limits and `Retry-After` | +| Artifact too large | 413 | `artifact_too_large` | Include cap | +| Fulcio/KMS down | 503 | `signing_unavailable` | Retry‑After with jitter | + +--- + +## 16) Deployment & HA + +* Run ≥ 2 replicas; front with L7 LB; **sticky** not required. +* Redis for replay/quota caches (HA). +* Audit sink (Mongo/Postgres) in primary region; asynchronous write with local fallback buffer. +* Fulcio/KMS clients configured with retries/backoff; circuit breakers. + +--- + +## 17) Implementation notes + +* **.NET 10** minimal API + Kestrel mTLS; custom DPoP middleware; JWT/JWKS cache. +* **Cosign verification** via sigstore libraries; Referrers queries over registry API with retries. +* **DSSE** via in‑toto libs; canonical JSON writer for predicates. +* **Backpressure** paths: refuse at auth/quota stages before any expensive network calls. + +--- + +## 18) Examples (wire) + +**Request (free plan; expect throttle if burst):** + +```http +POST /api/v1/signer/sign/dsse HTTP/1.1 +Authorization: DPoP +DPoP: +Content-Type: application/json + +{ ...body as above... } +``` + +**Error (release untrusted):** + +```json +{ + "type": "https://stella-ops.org/problems/release_untrusted", + "title": "Scanner image not signed by StellaOps", + "status": 403, + "detail": "sha256:abcd... not in trusted keyring", + "instance": "urn:audit:a7c9e3f2-..." +} +``` + +--- + +## 19) Roadmap + +* **Key Transparency**: optional publication of Signer’s *own* certs to a KT log. +* **Attested Build**: SLSA‑style provenance for Signer container itself, checked at startup. +* **FIPS mode**: enforce `ES256` + KMS/HSM only; disallow Ed25519. +* **Dual attestation**: optional immediate push to **Attestor** (sync mode) with timeout budget, returning Rekor UUID inline. + diff --git a/docs/ARCHITECTURE_UI.md b/docs/ARCHITECTURE_UI.md new file mode 100644 index 00000000..19d5d156 --- /dev/null +++ b/docs/ARCHITECTURE_UI.md @@ -0,0 +1,342 @@ +# component_architecture_web_ui.md — **Stella Ops Web UI** (2025Q4) + +> **Scope.** Implementation‑ready architecture for the **Angular SPA** that operators and developers use to drive Stella Ops. This document defines UX surfaces, module boundaries, data flows, auth, RBAC, real‑time updates, performance targets, i18n/a11y, security headers, testing and deployment. The UI is a *consumer* of backend APIs (Scanner, Policy, Vexer, Feedser, Attestor, Authority) and never performs scanning, merging, or signing on its own. + +--- + +## 0) Mission & non‑goals + +**Mission.** Provide a **fast, explainable** console for: + +* Scans (status, SBOMs, diffs, EntryTrace, attestation). +* Policy management (rules, exemptions, VEX consumption view). +* Vulnerability intel (Feedser status), VEX consensus exploration (Vexer). +* Runtime posture (Zastava observer + admission). +* Admin operations (tenants, tokens, quotas, licensing posture). + +**Non‑goals.** No client‑side crypto signing; no Docker/CRI access; no direct registry access beyond fetching static assets or OCI referrer summaries exposed by backend. + +--- + +## 1) Technology baseline + +* **Framework**: Angular 17+ (Stand‑alone APIs / Signals), TypeScript 5. +* **Styling**: Tailwind CSS + headless component patterns; CSS variables for theming. +* **Charts**: Lightweight SVG (uPlot or Apache ECharts via on‑demand import). +* **State**: Angular **Signals** + `@ngrx/signals` store for cross‑page slices. +* **Transport**: `fetch` + RxJS interop; **SSE** (EventSource) for progress streams. +* **Build**: Angular CLI + Vite builder. +* **Testing**: Jest + Testing Library, Playwright for e2e. +* **Packaging**: Containerized NGINX (immutable assets, ETag + content hashing). + +--- + +## 2) High‑level module map + +``` +/app + ├─ core/ # bootstrap, config, auth, http, error boundary, i18n + ├─ shared/ # UI kit (tables, code-viewers, badges), pipes + ├─ dashboard/ # live tiles, fleet KPIs, feed/vex age, queue depth + ├─ scans/ # scan list, detail, SBOM viewer, diff-by-layer, EntryTrace + ├─ runtime/ # Zastava posture, drift events, admission decisions + ├─ policy/ # rules editor (YAML/Rego), exemptions, previews + ├─ vex/ # VEX explorer (claims, consensus, conflicts) + ├─ feedser/ # source health, export cursors, rebuild/export triggers + ├─ attest/ # attestation proofs, verification bundles, Rekor links + ├─ admin/ # tenants, roles, clients, quotas, licensing posture + └─ plugins/ # route plug-ins (lazy remote modules, governed) +``` + +Each feature folder builds as a **standalone route** (lazy loaded). All HTTP shapes live in `core/api/` clients with shared DTOs. + +--- + +## 3) Navigation & key views + +### 3.1 Dashboard + +* **Tiles**: “New criticals (24h)”, “VEX suppressions applied”, “Attested SBOMs (7d)”, “Feed age per provider”, “Scanner queue depth”, “Admission events”. +* **Trends**: sparkline for vulns/day, pass/fail rates, attestation throughput. + +### 3.2 Scans + +* **Scan list** with status, image digest, repo, time, artifacts, attestation badge. +* **Scan detail**: + + * **SBOM viewer**: Inventory/Usage toggle; component table (virtualized), filters by package type, severity, source. + * **Diff by layer**: A→B change grid (added/removed/upgraded), grouped by introducing/removing layer; tooltips show provenance and links to layer SBOM fragment. + * **EntryTrace**: shell chain with file:line breadcrumbs; jump‑to source viewer (read‑only, hexdump fallback). + * **Attestation**: Rekor UUID, index, inclusion proof; **Verify** button calls Attestor `/verify`. + * **Export**: download buttons (CycloneDX JSON, Protobuf, SPDX JSON); size shown; SHA‑256 inline. + +### 3.3 Runtime (Zastava) + +* **Observer timeline**: container start/stop, drift, policy violations; faceted by namespace/owner. +* **Live process view**: top N processes, loaded libs summary vs Usage SBOM. +* **Admission decisions**: per‑namespace rules, allow/deny events, cache TTL, reasons. + +### 3.4 Policy + +* **Policy bundles**: active vs staged; diff viewer with change summary. +* **Editors**: + + * YAML rules (ignore lists, thresholds, vendor precedence overrides). + * Rego blocks (advanced gates) with **WASM** preview evaluator (client‑side sandbox) for “preview” against sample SBOMs. +* **VEX inclusion controls**: weight sliders (visualization only), provider allow/deny toggles. +* **Preview**: select SBOM (or image digest) → show verdict under staged policy. + +### 3.5 Vexer + +* **Claims explorer**: search by vulnId/productKey/provider; show raw claim (status, justification, evidence). +* **Consensus view**: rollup per (vuln, product) with accepted/rejected sources, weights, timestamps. +* **Conflicts**: grid of top conflicts; filters for justification gates failed. + +### 3.6 Feedser + +* **Sources** table: staleness, last run, errors. +* **Advisory search**: by CVE/alias; show normalized affected ranges. +* **Exports**: trigger full/delta JSON/Trivy DB; show manifest digests and Rekor link if attested. + +### 3.7 Attest + +* **Proofs list**: last 7 days Rekor entries; filter by kind (sbom/report/vex). +* **Verification**: paste UUID or upload bundle → verify; result with explanations (chain, Merkle path). + +### 3.8 Admin + +* **Tenants/Installations**: view/edit, isolation hints. +* **Clients & roles**: Authority clients, role→scope mapping, rotation hints. +* **Quotas**: per license plan, counters, throttle events. +* **Licensing posture**: last PoE introspection snapshot (redacted), release window. + +--- + +## 4) Auth, sessions & RBAC + +### 4.1 OIDC flow + +* **Authorization Code + PKCE** to **Authority**. +* **ID Token** for UX identity; **Access Token** (OpTok) for APIs (2–5 min TTL). +* **DPoP (browser)**: generate ephemeral **WebCrypto** keypair; store public JWK in memory, private key in **IndexedDB** (non‑exportable if platform allows). Access token includes `cnf.jkt`; each API call adds `DPoP` proof; handle nonce challenges automatically. +* **Refresh**: optional DPoP‑bound refresh tokens with rotation; otherwise silent renew. + +### 4.2 RBAC + +* Roles (`ui.read`, `ui.admin`, plus service roles) are embedded in ID token or fetched via `/me` endpoint. +* **Route guards** enforce access; **feature flags** hide admin pages for non‑admins. + +### 4.3 Session storage + +* Access tokens & refresh metadata in memory; persist **only** minimal session (subject, expiries) in `sessionStorage`. Never persist raw JWTs to `localStorage`. Use **SameSite=Lax** cookies for anti‑CSRF if cookies are required (prefer pure bearer headers + DPoP). + +--- + +## 5) HTTP layer & API clients + +* **`core/http/api-client.ts`** centralizes: + + * Base URLs (Scanner, Vexer, Feedser, Attestor). + * **Retry** policies on idempotent GETs (backoff + jitter). + * **Problem+JSON** parser → uniform error toasts with correlation ID. + * **SSE** helper (EventSource) with auto‑reconnect & backpressure. + * **DPoP** injector & nonce handling. + +* Typed API clients (DTOs in `core/api/models.ts`): + + * `ScannerApi`, `PolicyApi`, `VexerApi`, `FeedserApi`, `AttestorApi`, `AuthorityApi`. + +**DTO examples (abbrev):** + +```ts +export type ImageDigest = `sha256:${string}`; +export interface ScanSummary { + imageDigest: ImageDigest; createdAt: string; + artifacts: { view: 'inventory'|'usage'; format: 'cdx-json'|'cdx-pb'|'spdx-json'; sha256: string; size: number }[]; + status: 'queued'|'running'|'completed'|'error'; + rekor?: { uuid: string; index?: number; url?: string }; +} +export interface DiffEntry { + key: string; change: 'added'|'removed'|'upgraded'|'downgraded'; + fromVersion?: string; toVersion?: string; layer: string; usedByEntrypoint?: boolean; +} +export interface VexConsensus { + vulnId: string; productKey: string; rollupStatus: 'affected'|'not_affected'|'fixed'|'under_investigation'; + sources: { providerId: string; status: string; weight: number; accepted: boolean; reason: string }[]; +} +``` + +--- + +## 6) State, caching & real‑time + +* **Per‑page stores** (Signals) for list filters, pagination, and selected entities. +* **Normalized caches** keyed by `(imageDigest, view, format)`; artifacts are downloaded via pre‑signed URLs from Scanner and streamed; SHA‑256 verified client‑side before exposing “verified” badge. +* **SSE channels**: + + * `/scans/{id}/events` → progress log. + * `/runtime/events/stream` (optional) → live drift/admission feed (rate‑limited). +* **Cache invalidation** on job completion or explicit “refresh”. + +--- + +## 7) SBOM viewing & diff UX + +* **Huge tables** rendered with **virtual scrolling** (CDK Virtual Scroll); sort/filter performed client‑side for ≤ 20k rows; beyond that, server‑side queries via BOM‑Index endpoints. +* **Component row** shows purl, version, origin (OS pkg / metadata / linker / attested), licenses, and **used** badge (Usage view). +* **Diff**: compact heatmap per layer; clicking opens a right‑pane with evidence: introducing paths, file hashes, VEX notes (from Vexer consensus) and links to advisories (Feedser). + +--- + +## 8) Policy editor & VEX integration + +* **YAML editor**: Monaco‑based with schema hints; previews show which rules matched. +* **Rego editor**: Monaco with WASM‑OPA sandbox for read‑only evaluation on sample SBOMs. +* **VEX toggles**: per‑provider enable/disable; “explain” drawer shows why a claim was accepted/rejected (justification gate, signature state, weight). +* **Staged → Active** promotion is a two‑step flow with confirmation and automatic policy digest computation. + +--- + +## 9) Accessibility, i18n & theming + +* **A11y**: WCAG 2.2 AA; keyboard navigation, focus management, ARIA roles; color‑contrast tokens verified by unit tests. +* **I18n**: Angular i18n + runtime translation loader (`/locales/{lang}.json`); dates/numbers localized via `Intl`. +* **Languages**: English default; Bulgarian, German, Japanese as initial additions. +* **Theming**: dark/light via CSS variables; persisted in `prefers-color-scheme` aware store. + +--- + +## 10) Performance budgets + +* **TTI** ≤ 1.5 s on 4G/slow CPU (first visit), ≤ 0.6 s repeat (HTTP/2, cached). +* **JS** initial < 300 KB gz (lazy routes). +* **SBOM list**: render 10k rows in < 70 ms with virtualization; filter in < 150 ms. +* **Diff view**: compute client‑side grouping for 5k changes in < 120 ms. + +Techniques: route‑level code splitting; `ChangeDetectionStrategy.OnPush`; Signals; server compression (zstd/gzip), immutable assets with long max‑age (cache busting via hashes). + +--- + +## 11) Security headers & CSP + +* **CSP**: `default-src 'self'; connect-src 'self' https://*.internal; img-src 'self' data:; script-src 'self'; style-src 'self' 'unsafe-inline'; frame-ancestors 'none';` +* **HSTS** enabled at gateway. +* **Referrer Policy**: `no-referrer`. +* **X‑Frame‑Options**: `DENY`. +* **COOP/COEP** to enable faster cross‑origin isolation (for WASM OPA). +* **Subresource Integrity (SRI)** for third‑party fonts (minimize third‑party). + +--- + +## 12) Error handling & UX hygiene + +* **Global error boundary** surfaces Problem+JSON `title/detail/instance` with correlation ID. +* **Retry toast** for 429 (quota throttles) with backoff timer. +* **Auth expiry**: pre‑emptive refresh; unobtrusive banner when < 60 s TTL; re‑login modal if refresh fails. +* **Network down**: offline banner with queued actions (idempotent resubmits). +* **File verify**: show SHA‑256 mismatch warnings if artifact altered in transit. + +--- + +## 13) Observability + +* **Front‑end telemetry** (OpenTelemetry Web): route timings, API latency by service, error counts; sampled to 1–5% and shipped to backend OTLP endpoint. +* **User actions** logged anonymously (no PII): “policy promote”, “scan export”, “attest verify”. +* **Metrics dash** in admin shows SLOs and recent front‑end errors. + +--- + +## 14) Testing strategy + +* **Unit**: pure component logic via Jest + Testing Library (no TestBed when possible). +* **Component harness** for table, code viewer, diff heatmap. +* **Contract tests**: OpenAPI schemas pulled at build time; DTOs validated; breaking changes fail CI. +* **e2e**: Playwright scenarios (login, scan detail, diff, policy edit, admit deny). +* **A11y**: axe-core CI checks; color‑contrast lints. +* **i18n**: key coverage tests (no missing translations in supported locales). + +--- + +## 15) Deployment & ops + +* **Container**: `stellaops/web-ui:-`; NGINX with `gzip_static` + brotli; immutable assets under `/static//…`. +* **Config**: `/config.json` served by gateway (injected at runtime): API base URLs, authority issuer, telemetry sampling. +* **Version banner**: footer shows UI & backend versions; warns on major mismatches. +* **CDN** (optional): cache static bundle; APIs stay behind internal gateway. +* **Feature flags**: environment gates (staged policies, eBPF runtime) readable from config. + +--- + +## 16) Plugin system (route plug‑ins) + +* **Manifest**: Backend provides a signed plug‑in manifest with remote module URLs and **cosign signature** per JS bundle. +* **Loader**: dynamic import with **SRI** and signature verification (WebCrypto). +* **Sandbox**: plug‑ins are routed modules receiving a limited **UI SDK** (navigation, theme, API gateway). No direct token access; API calls proxied through the UI SDK which enforces RBAC. +* **Examples**: custom reports, vendor dashboards, regulated TLS config UIs. + +--- + +## 17) Wire sequences (representative) + +**A) View scan progress** + +```mermaid +sequenceDiagram + autonumber + participant UI + participant Auth as Authority + participant SW as Scanner.WebService + + UI->>Auth: /authorize (PKCE) + Auth-->>UI: code → token (DPoP-bound) + UI->>SW: GET /scans/{id} (Authorization+DPoP) + SW-->>UI: { status: running } + UI->>SW: (SSE) GET /scans/{id}/events + SW-->>UI: progress events … + SW-->>UI: terminal event { status: completed, artifacts[] } +``` + +**B) Verify attestation** + +```mermaid +sequenceDiagram + autonumber + participant UI + participant AT as Attestor + UI->>AT: POST /rekor/verify { uuid } + AT-->>UI: { ok:true, index, logURL } +``` + +**C) Promote policy & preview** + +```mermaid +sequenceDiagram + autonumber + participant UI + participant BE as Scanner.WebService (Policy endpoint) + UI->>BE: POST /policy/stage { yaml, rego } + BE-->>UI: { policyRevision, diagnostics } + UI->>BE: POST /policy/preview { imageDigest, policyRevision } + BE-->>UI: { verdict: pass|fail, reasons[] } + UI->>BE: POST /policy/promote { policyRevision } + BE-->>UI: { ok:true } +``` + +--- + +## 18) Security hard lines + +* Never store JWTs in `localStorage`. +* Enforce DPoP for API calls; if DPoP unsupported for a service, require **SameSite=Lax** cookies with CSRF token header. +* Block mixed‑content; only HTTPS origins allowed. +* Validate and render only **escaped** user content; code viewer uses safe highlighter. +* Downloaded artifacts are treated as **opaque binaries**; no HTML rendering. + +--- + +## 19) Roadmap + +* **PWA** offline shell (read‑only) for dashboards and cached scan details. +* **SBOM graph** visualization (force‑directed) for small components sets. +* **Runtime session replay** (privacy‑safe) to debug operator workflows (opt‑in). +* **Assistive wizards** for policy creation with guided templates. diff --git a/docs/ARCHITECTURE_VEXER.md b/docs/ARCHITECTURE_VEXER.md index c591388d..1f6920f9 100644 --- a/docs/ARCHITECTURE_VEXER.md +++ b/docs/ARCHITECTURE_VEXER.md @@ -1,85 +1,463 @@ -# StellaOps Vexer Architecture +# component_architecture_vexer.md — **Stella Ops Vexer** (2025Q4) -Vexer is StellaOps' vulnerability-exploitability (VEX) platform. It ingests VEX statements from multiple providers, normalizes them into canonical claims, projects trust-weighted consensus, and delivers deterministic export artifacts with signed attestations. This document summarizes the target architecture and how the current implementation maps to those goals. +> **Scope.** This document specifies the **Vexer** service: its purpose, trust model, data structures, APIs, plug‑in contracts, storage schema, normalization/consensus algorithms, performance budgets, testing matrix, and how it integrates with Scanner, Policy, Feedser, and the attestation chain. It is implementation‑ready. -## 1. Solution topology +--- -| Module | Purpose | Key contracts | -| --- | --- | --- | -| `StellaOps.Vexer.Core` | Domain models (`VexClaim`, `VexConsensus`, `VexExportManifest`), deterministic JSON helpers, shared abstractions (connectors, exporters, attestations). | `IVexConnector`, `IVexExporter`, `IVexAttestationClient`, `VexCanonicalJsonSerializer` | -| `StellaOps.Vexer.Policy` | Loads operator policy (weights, overrides, justification gates) and exposes snapshots for consensus. | `IVexPolicyProvider`, `IVexPolicyEvaluator`, `VexPolicyOptions` | -| `StellaOps.Vexer.Storage.Mongo` | Persistence layer for providers, raw docs, claims, consensus, exports, cache. | `IVexRawStore`, `IVexExportStore`, Mongo class maps | -| `StellaOps.Vexer.Export` | Orchestrates export pipeline (query signature → cache lookup → snapshot build → attestation handoff). | `IExportEngine`, `IVexExportDataSource` | -| `StellaOps.Vexer.Attestation` *(planned)* | Builds in-toto/DSSE envelopes and communicates with Sigstore/Rekor. | `IVexAttestationClient` | -| `StellaOps.Vexer.WebService` *(planned)* | Minimal API host for ingest/export endpoints. | `AddVexerWebService()` | -| `StellaOps.Vexer.Worker` *(planned)* | Background executor for scheduled pulls, verification, reconciliation, cache GC. | Hosted services | +## 0) Mission & role in the platform -All modules target .NET 10 preview and follow the same deterministic logging and serialization conventions as Feedser. +**Mission.** Convert heterogeneous **VEX** statements (OpenVEX, CSAF VEX, CycloneDX VEX; vendor/distro/platform sources) into **canonical, queryable claims**; compute **deterministic consensus** per *(vuln, product)*; preserve **conflicts with provenance**; publish **stable, attestable exports** that the backend uses to suppress non‑exploitable findings, prioritize remaining risk, and explain decisions. -## 2. Data model +**Boundaries.** -MongoDB acts as the canonical store; collections (with logical responsibilities) are: +* Vexer **does not** decide PASS/FAIL. It supplies **evidence** (statuses + justifications + provenance weights). +* Vexer preserves **conflicting claims** unchanged; consensus encodes how we would pick, but the raw set is always exportable. +* VEX consumption is **backend‑only**: Scanner never applies VEX. The backend’s **Policy Engine** asks Vexer for status evidence and then decides what to show. -- `vex.providers` – provider metadata, trust tiers, discovery endpoints, and cosign/PGP details. -- `vex.raw` – immutable raw documents (CSAF, CycloneDX VEX, OpenVEX, OCI attestations) with digests, retrieval metadata, and signature state. -- `vex.claims` – normalized `VexClaim` rows; deduped on `(providerId, vulnId, productKey, docDigest)`. -- `vex.consensus` – consensus projections per `(vulnId, productKey)` capturing rollup status, source weights, conflicts, and policy revision. -- `vex.exports` – export manifests containing artifact digests, cache metadata, and attestation pointers. -- `vex.cache` – index from `querySignature`/`format` to export digest for fast reuse. -- `vex.migrations` – tracks applied storage migrations (index bootstrap, future schema updates). +--- -GridFS is used for large raw payloads when necessary, and artifact stores (S3/MinIO/file) hold serialized exports referenced by `vex.exports`. +## 1) Inputs, outputs & canonical domain -## 3. Ingestion and reconciliation flow +### 1.1 Accepted input formats (ingest) -1. **Discovery & configuration** – connectors load YAML/JSON settings via `StellaOps.Vexer.Policy` (provider enablement, trust overrides). -2. **Fetch** – each `IVexConnector` pulls source windows, writing raw documents through `IVexRawDocumentSink` (Mongo-backed) with dedupe on digest. -3. **Verification** – signatures/attestations validated through `IVexSignatureVerifier`; metadata stored alongside raw records. -4. **Normalization** – format-specific `IVexNormalizer` instances translate raw payloads to canonical `VexClaim` batches. -5. **Consensus** – `VexConsensusResolver` (Core) consumes claims with policy weights supplied by `IVexPolicyEvaluator`, producing deterministic consensus entries and conflict annotations. -6. **Export** – query requests pass through `VexExportEngine`, generating `VexExportManifest` instances, caching by `VexQuerySignature`, and emitting artifacts for attestation/signature. -7. **Attestation & transparency** *(planned)* – `IVexAttestationClient` signs exports (in-toto/DSSE) and records bundles in Rekor v2. +* **OpenVEX** JSON documents (attested or raw). +* **CSAF VEX** 2.x (vendor PSIRTs and distros commonly publish CSAF). +* **CycloneDX VEX** 1.4+ (standalone VEX or embedded VEX blocks). +* **OCI‑attached attestations** (VEX statements shipped as OCI referrers) — optional connectors. -The Worker coordinates the long-running steps (fetch/verify/normalize/export), while the WebService exposes synchronous APIs for on-demand operations and status lookups. +All connectors register **source metadata**: provider identity, trust tier, signature expectations (PGP/cosign/PKI), fetch windows, rate limits, and time anchors. -## 4. Policy semantics +### 1.2 Canonical model (normalized) -- **Weights** – default tiers (`vendor=1.0`, `distro=0.9`, `platform=0.7`, `hub=0.5`, `attestation=0.6`) loaded via `VexPolicyOptions.Weights`, with per-provider overrides. -- **Justification gates** – policy enforces that `not_affected` claims must provide a recognized justification; rejected claims are preserved as conflicts with reason metadata. -- **Diagnostics** – policy snapshots carry structured issues for misconfigurations (out-of-range weights, empty overrides) surfaced to operators via logs and future CLI/Web endpoints. +Every incoming statement becomes a set of **VexClaim** records: -Policy snapshots are immutable and versioned so consensus records capture the policy revision used during evaluation. +``` +VexClaim +- providerId // 'redhat', 'suse', 'ubuntu', 'github', 'vendorX' +- vulnId // 'CVE-2025-12345', 'GHSA-xxxx', canonicalized +- productKey // canonical product identity (see §2.2) +- status // affected | not_affected | fixed | under_investigation +- justification? // for 'not_affected'/'affected' where provided +- introducedVersion? // semantics per provider (range or exact) +- fixedVersion? // where provided (range or exact) +- lastObserved // timestamp from source or fetch time +- provenance // doc digest, signature status, fetch URI, line/offset anchors +- evidence[] // raw source snippets for explainability +- supersedes? // optional cross-doc chain (docDigest → docDigest) +``` -## 5. Determinism & caching +### 1.3 Exports (consumption) -- JSON serialization uses `VexCanonicalJsonSerializer`, enforcing property ordering and camelCase naming for reproducible snapshots and test fixtures. -- `VexQuerySignature` produces canonical filter/order strings and SHA-256 digests, enabling cache keys shared across services. -- Export manifests reuse cached artifacts when the same signature/format is requested unless `ForceRefresh` is explicitly set. -- For scorring multiple sources on same VEX topic use - `VEXER_SCORRING.md` +* **VexConsensus** per `(vulnId, productKey)` with: -## 6. Observability & offline posture + * `rollupStatus` (after policy weights/justification gates), + * `sources[]` (winning + losing claims with weights & reasons), + * `policyRevisionId` (identifier of the Vexer policy used), + * `consensusDigest` (stable SHA‑256 over canonical JSON). +* **Raw claims** export for auditing (unchanged, with provenance). +* **Provider snapshots** (per source, last N days) for operator debugging. +* **Index** optimized for backend joins: `(productKey, vulnId) → (status, confidence, sourceSet)`. -- Structured logs (`ILogger`) capture correlation IDs, query signatures, provider IDs, and policy revisions. Metrics/OTel instrumentation will mirror Feedser once tracing hooks are added. -- Offline-first: connectors, policy bundles, and export caches can be bundled inside the Offline Kit; no mandatory outbound calls beyond configured provider allowlists. -- Operator tooling (CLI/WebService) will expose diagnostics (policy issues, verification failures, cache status) so air-gapped deployments maintain visibility without external telemetry. +All exports are **deterministic**, and (optionally) **attested** via DSSE and logged to Rekor v2. -## 7. Roadmap highlights +--- -- Complete storage mappings for providers/consensus/cache and add migrations/indices per collection. -- Implement Rekor/in-toto attestation clients and wire export engine to produce signed bundles. -- Build WebService endpoints (`/vexer/status`, `/vexer/claims`, `/vexer/exports`) plus CLI verbs mirroring Feedser patterns. -- Provide CSAF, CycloneDX VEX, and OpenVEX normalizers along with vendor-specific connectors (Red Hat, Cisco, SUSE, MSRC, Oracle, Ubuntu, OCI attestation). -- Extend policy diagnostics with schema validation, change tracking, and operator-facing diff reports. -- Mongo bootstrapper runs ordered migrations (`vex.migrations`) to ensure indexes for raw documents, providers, consensus snapshots, exports, and cache entries. +## 2) Identity model — products & joins -## Appendix A – Policy diagnostics workflow +### 2.1 Vuln identity -- `StellaOps.Vexer.Policy` now exposes `IVexPolicyDiagnostics`, producing deterministic diagnostics reports with timestamp, severity counts, active provider overrides, and the full issue list surfaced by `IVexPolicyProvider`. -- CLI/WebService layers should call `IVexPolicyDiagnostics.GetDiagnostics()` to display operator-friendly summaries (`vexer policy diagnostics` and `/vexer/policy/diagnostics` are the planned entry points). -- Recommendations in the report guide operators to resolve blocking errors, review warnings, and audit override usage before consensus runs—embed them directly in UX copy instead of re-deriving logic. -- Export/consensus telemetry should log the diagnostic `Version` alongside `policyRevisionId` so dashboards can correlate policy changes with consensus decisions. -- Offline installations can persist the diagnostics report (JSON) in the Offline Kit to document policy headroom during audits; the output is deterministic and diff-friendly. -- Use `VexPolicyBinder` when ingesting operator-supplied YAML/JSON bundles; it normalizes weight/override values, reports deterministic issues, and returns the consensus-ready `VexConsensusPolicyOptions` used by `VexPolicyProvider`. -- Reload telemetry emits `vex.policy.reloads` (tags: `revision`, `version`, `issues`) whenever a new digest is observed—feed this into dashboards to correlate policy changes with consensus outcomes. +* Accepts **CVE**, **GHSA**, vendor IDs (MSRC, RHSA…), distro IDs (DSA/USN/RHSA…) — normalized to `vulnId` with alias sets. +* **Alias graph** maintained (from Feedser) to map vendor/distro IDs → CVE (primary) and to **GHSA** where applicable. + +### 2.2 Product identity (`productKey`) + +* **Primary:** `purl` (Package URL). +* **Secondary links:** `cpe`, **OS package NVRA/EVR**, NuGet/Maven/Golang identity, and **OS package name** when purl unavailable. +* **Fallback:** `oci:/@` for image‑level VEX. +* **Special cases:** kernel modules, firmware, platforms → provider‑specific mapping helpers (connector captures provider’s product taxonomy → canonical `productKey`). + +> Vexer does not invent identities. If a provider cannot be mapped to purl/CPE/NVRA deterministically, we keep the native **product string** and mark the claim as **non‑joinable**; the backend will ignore it unless a policy explicitly whitelists that provider mapping. + +--- + +## 3) Storage schema (MongoDB) + +Database: `vexer` + +### 3.1 Collections + +**`vex.providers`** + +``` +_id: providerId +name, homepage, contact +trustTier: enum {vendor, distro, platform, hub, attestation} +signaturePolicy: { type: pgp|cosign|x509|none, keys[], certs[], cosignKeylessRoots[] } +fetch: { baseUrl, kind: http|oci|file, rateLimit, etagSupport, windowDays } +enabled: bool +createdAt, modifiedAt +``` + +**`vex.raw`** (immutable raw documents) + +``` +_id: sha256(doc bytes) +providerId +uri +ingestedAt +contentType +sig: { verified: bool, method: pgp|cosign|x509|none, keyId|certSubject, bundle? } +payload: GridFS pointer (if large) +disposition: kept|replaced|superseded +correlation: { replaces?: sha256, replacedBy?: sha256 } +``` + +**`vex.claims`** (normalized rows; dedupe on providerId+vulnId+productKey+docDigest) + +``` +_id +providerId +vulnId +productKey +status +justification? +introducedVersion? +fixedVersion? +lastObserved +docDigest +provenance { uri, line?, pointer?, signatureState } +evidence[] { key, value, locator } +indices: + - {vulnId:1, productKey:1} + - {providerId:1, lastObserved:-1} + - {status:1} + - text index (optional) on evidence.value for debugging +``` + +**`vex.consensus`** (rollups) + +``` +_id: sha256(canonical(vulnId, productKey, policyRevision)) +vulnId +productKey +rollupStatus +sources[]: [ + { providerId, status, justification?, weight, lastObserved, accepted:bool, reason } +] +policyRevisionId +evaluatedAt +consensusDigest // same as _id +indices: + - {vulnId:1, productKey:1} + - {policyRevisionId:1, evaluatedAt:-1} +``` + +**`vex.exports`** (manifest of emitted artifacts) + +``` +_id +querySignature +format: raw|consensus|index +artifactSha256 +rekor { uuid, index, url }? +createdAt +policyRevisionId +cacheable: bool +``` + +**`vex.cache`** + +``` +querySignature -> exportId (for fast reuse) +ttl, hits +``` + +**`vex.migrations`** + +* ordered migrations applied at bootstrap to ensure indexes. + +### 3.2 Indexing strategy + +* Hot path queries use exact `(vulnId, productKey)` and time‑bounded windows; compound indexes cover both. +* Providers list view by `lastObserved` for monitoring staleness. +* `vex.consensus` keyed by `(vulnId, productKey, policyRevision)` for deterministic reuse. + +--- + +## 4) Ingestion pipeline + +### 4.1 Connector contract + +```csharp +public interface IVexConnector +{ + string ProviderId { get; } + Task FetchAsync(VexConnectorContext ctx, CancellationToken ct); // raw docs + Task NormalizeAsync(VexConnectorContext ctx, CancellationToken ct); // raw -> VexClaim[] +} +``` + +* **Fetch** must implement: window scheduling, conditional GET (ETag/If‑Modified‑Since), rate limiting, retry/backoff. +* **Normalize** parses the format, validates schema, maps product identities deterministically, emits `VexClaim` records with **provenance**. + +### 4.2 Signature verification (per provider) + +* **cosign (keyless or keyful)** for OCI referrers or HTTP‑served JSON with Sigstore bundles. +* **PGP** (provider keyrings) for distro/vendor feeds that sign docs. +* **x509** (mutual TLS / provider‑pinned certs) where applicable. +* Signature state is stored on **vex.raw.sig** and copied into **provenance.signatureState** on claims. + +> Claims from sources failing signature policy are marked `"signatureState.verified=false"` and **policy** can down‑weight or ignore them. + +### 4.3 Time discipline + +* For each doc, prefer **provider’s document timestamp**; if absent, use fetch time. +* Claims carry `lastObserved` which drives **tie‑breaking** within equal weight tiers. + +--- + +## 5) Normalization: product & status semantics + +### 5.1 Product mapping + +* **purl** first; **cpe** second; OS package NVRA/EVR mapping helpers (distro connectors) produce purls via canonical tables (e.g., rpm→purl:rpm, deb→purl:deb). +* Where a provider publishes **platform‑level** VEX (e.g., “RHEL 9 not affected”), connectors expand to known product inventory rules (e.g., map to sets of packages/components shipped in the platform). Expansion tables are versioned and kept per provider; every expansion emits **evidence** indicating the rule applied. +* If expansion would be speculative, the claim remains **platform‑scoped** with `productKey="platform:redhat:rhel:9"` and is flagged **non‑joinable**; backend can decide to use platform VEX only when Scanner proves the platform runtime. + +### 5.2 Status + justification mapping + +* Canonical **status**: `affected | not_affected | fixed | under_investigation`. +* **Justifications** normalized to a controlled vocabulary (CISA‑aligned), e.g.: + + * `component_not_present` + * `vulnerable_code_not_in_execute_path` + * `vulnerable_configuration_unused` + * `inline_mitigation_applied` + * `fix_available` (with `fixedVersion`) + * `under_investigation` +* Providers with free‑text justifications are mapped by deterministic tables; raw text preserved as `evidence`. + +--- + +## 6) Consensus algorithm + +**Goal:** produce a **stable**, explainable `rollupStatus` per `(vulnId, productKey)` given possibly conflicting claims. + +### 6.1 Inputs + +* Set **S** of `VexClaim` for the key. +* **Vexer policy snapshot**: + + * **weights** per provider tier and per provider overrides. + * **justification gates** (e.g., require justification for `not_affected` to be acceptable). + * **minEvidence** rules (e.g., `not_affected` must come from ≥1 vendor or 2 distros). + * **signature requirements** (e.g., require verified signature for ‘fixed’ to be considered). + +### 6.2 Steps + +1. **Filter invalid** claims by signature policy & justification gates → set `S'`. +2. **Score** each claim: + `score = weight(provider) * freshnessFactor(lastObserved)` where freshnessFactor ∈ [0.8, 1.0] for staleness decay (configurable; small effect). +3. **Aggregate** scores per status: `W(status) = Σ score(claims with that status)`. +4. **Pick** `rollupStatus = argmax_status W(status)`. +5. **Tie‑breakers** (in order): + + * Higher **max single** provider score wins (vendor > distro > platform > hub). + * More **recent** lastObserved wins. + * Deterministic lexicographic order of status (`fixed` > `not_affected` > `under_investigation` > `affected`) as final tiebreaker. +6. **Explain**: mark accepted sources (`accepted=true; reason="weight"`/`"freshness"`), mark rejected sources with explicit `reason` (`"insufficient_justification"`, `"signature_unverified"`, `"lower_weight"`). + +> The algorithm is **pure** given S and policy snapshot; result is reproducible and hashed into `consensusDigest`. + +--- + +## 7) Query & export APIs + +All endpoints are versioned under `/api/v1/vex`. + +### 7.1 Query (online) + +``` +POST /claims/search + body: { vulnIds?: string[], productKeys?: string[], providers?: string[], since?: timestamp, limit?: int, pageToken?: string } + → { claims[], nextPageToken? } + +POST /consensus/search + body: { vulnIds?: string[], productKeys?: string[], policyRevisionId?: string, since?: timestamp, limit?: int, pageToken?: string } + → { entries[], nextPageToken? } + +POST /resolve + body: { purls: string[], vulnIds: string[], policyRevisionId?: string } + → { results: [ { vulnId, productKey, rollupStatus, sources[] } ] } +``` + +### 7.2 Exports (cacheable snapshots) + +``` +POST /exports + body: { signature: { vulnFilter?, productFilter?, providers?, since? }, format: raw|consensus|index, policyRevisionId?: string, force?: bool } + → { exportId, artifactSha256, rekor? } + +GET /exports/{exportId} → bytes (application/json or binary index) +GET /exports/{exportId}/meta → { signature, policyRevisionId, createdAt, artifactSha256, rekor? } +``` + +### 7.3 Provider operations + +``` +GET /providers → provider list & signature policy +POST /providers/{id}/refresh → trigger fetch/normalize window +GET /providers/{id}/status → last fetch, doc counts, signature stats +``` + +**Auth:** service‑to‑service via Authority tokens; operator operations via UI/CLI with RBAC. + +--- + +## 8) Attestation integration + +* Exports can be **DSSE‑signed** via **Signer** and logged to **Rekor v2** via **Attestor** (optional but recommended for regulated pipelines). +* `vex.exports.rekor` stores `{uuid, index, url}` when present. +* **Predicate type**: `https://stella-ops.org/attestations/vex-export/1` with fields: + + * `querySignature`, `policyRevisionId`, `artifactSha256`, `createdAt`. + +--- + +## 9) Configuration (YAML) + +```yaml +vexer: + mongo: { uri: "mongodb://mongo/vexer" } + s3: + endpoint: http://minio:9000 + bucket: stellaops + policy: + weights: + vendor: 1.0 + distro: 0.9 + platform: 0.7 + hub: 0.5 + attestation: 0.6 + providerOverrides: + redhat: 1.0 + suse: 0.95 + requireJustificationForNotAffected: true + signatureRequiredForFixed: true + minEvidence: + not_affected: + vendorOrTwoDistros: true + connectors: + - providerId: redhat + kind: csaf + baseUrl: https://access.redhat.com/security/data/csaf/v2/ + signaturePolicy: { type: pgp, keys: [ "…redhat-pgp-key…" ] } + windowDays: 7 + - providerId: suse + kind: csaf + baseUrl: https://ftp.suse.com/pub/projects/security/csaf/ + signaturePolicy: { type: pgp, keys: [ "…suse-pgp-key…" ] } + - providerId: ubuntu + kind: openvex + baseUrl: https://…/vex/ + signaturePolicy: { type: none } + - providerId: vendorX + kind: cyclonedx-vex + ociRef: ghcr.io/vendorx/vex@sha256:… + signaturePolicy: { type: cosign, cosignKeylessRoots: [ "sigstore-root" ] } +``` + +--- + +## 10) Security model + +* **Input signature verification** enforced per provider policy (PGP, cosign, x509). +* **Connector allowlists**: outbound fetch constrained to configured domains. +* **Tenant isolation**: per‑tenant DB prefixes or separate DBs; per‑tenant S3 prefixes; per‑tenant policies. +* **AuthN/Z**: Authority‑issued OpToks; RBAC roles (`vex.read`, `vex.admin`, `vex.export`). +* **No secrets in logs**; deterministic logging contexts include providerId, docDigest, claim keys. + +--- + +## 11) Performance & scale + +* **Targets:** + + * Normalize 10k VEX claims/minute/core. + * Consensus compute ≤ 50 ms for 1k unique `(vuln, product)` pairs in hot cache. + * Export (consensus) 1M rows in ≤ 60 s on 8 cores with streaming writer. + +* **Scaling:** + + * WebService handles control APIs; **Worker** background services (same image) execute fetch/normalize in parallel with rate‑limits; Mongo writes batched; upserts by natural keys. + * Exports stream straight to S3 (MinIO) with rolling buffers. + +* **Caching:** + + * `vex.cache` maps query signatures → export; TTL to avoid stampedes; optimistic reuse unless `force`. + +--- + +## 12) Observability + +* **Metrics:** + + * `vex.ingest.docs_total{provider}` + * `vex.normalize.claims_total{provider}` + * `vex.signature.failures_total{provider,method}` + * `vex.consensus.conflicts_total{vulnId}` + * `vex.exports.bytes{format}` / `vex.exports.latency_seconds` +* **Tracing:** spans for fetch, verify, parse, map, consensus, export. +* **Dashboards:** provider staleness, top conflicting vulns/components, signature posture, export cache hit‑rate. + +--- + +## 13) Testing matrix + +* **Connectors:** golden raw docs → deterministic claims (fixtures per provider/format). +* **Signature policies:** valid/invalid PGP/cosign/x509 samples; ensure rejects are recorded but not accepted. +* **Normalization edge cases:** platform‑only claims, free‑text justifications, non‑purl products. +* **Consensus:** conflict scenarios across tiers; check tie‑breakers; justification gates. +* **Performance:** 1M‑row export timing; memory ceilings; stream correctness. +* **Determinism:** same inputs + policy → identical `consensusDigest` and export bytes. +* **API contract tests:** pagination, filters, RBAC, rate limits. + +--- + +## 14) Integration points + +* **Backend Policy Engine** (in Scanner.WebService): calls `POST /resolve` with batched `(purl, vulnId)` pairs to fetch `rollupStatus + sources`. +* **Feedser**: provides alias graph (CVE↔vendor IDs) and may supply VEX‑adjacent metadata (e.g., KEV flag) for policy escalation. +* **UI**: VEX explorer screens use `/claims/search` and `/consensus/search`; show conflicts & provenance. +* **CLI**: `stellaops vex export --consensus --since 7d --out vex.json` for audits. + +--- + +## 15) Failure modes & fallback + +* **Provider unreachable:** stale thresholds trigger warnings; policy can down‑weight stale providers automatically (freshness factor). +* **Signature outage:** continue to ingest but mark `signatureState.verified=false`; consensus will likely exclude or down‑weight per policy. +* **Schema drift:** unknown fields are preserved as `evidence`; normalization rejects only on **invalid identity** or **status**. + +--- + +## 16) Rollout plan (incremental) + +1. **MVP**: OpenVEX + CSAF connectors for 3 major providers (e.g., Red Hat/SUSE/Ubuntu), normalization + consensus + `/resolve`. +2. **Signature policies**: PGP for distros; cosign for OCI. +3. **Exports + optional attestation**. +4. **CycloneDX VEX** connectors; platform claim expansion tables; UI explorer. +5. **Scale hardening**: export indexes; conflict analytics. + +--- + +## 17) Appendix — canonical JSON (stable ordering) + +All exports and consensus entries are serialized via `VexCanonicalJsonSerializer`: + +* UTF‑8 without BOM; +* keys sorted (ASCII); +* arrays sorted by `(providerId, vulnId, productKey, lastObserved)` unless semantic order mandated; +* timestamps in `YYYY‑MM‑DDThh:mm:ssZ`; +* no insignificant whitespace. -This architecture keeps Vexer aligned with StellaOps' deterministic, offline-operable design while layering VEX-specific consensus and attestation capabilities on top of the Feedser foundations. diff --git a/docs/ARCHITECTURE_ZASTAVA.md b/docs/ARCHITECTURE_ZASTAVA.md new file mode 100644 index 00000000..60f017ae --- /dev/null +++ b/docs/ARCHITECTURE_ZASTAVA.md @@ -0,0 +1,451 @@ +# component_architecture_zastava.md — **Stella Ops Zastava** (2025Q4) + +> **Scope.** Implementation‑ready architecture for **Zastava**: the **runtime inspector/enforcer** that watches real workloads, detects drift from the scanned baseline, verifies image/SBOM/attestation posture, and (optionally) **admits/blocks** deployments. Includes Kubernetes & plain‑Docker topologies, data contracts, APIs, security posture, performance targets, test matrices, and failure modes. + +--- + +## 0) Mission & boundaries + +**Mission.** Give operators **ground‑truth** from running environments and a **fast guardrail** before workloads land: + +* **Observer:** inventory containers, entrypoints actually executed, and DSOs actually loaded; verify **image signature**, **SBOM referrers**, and **attestation** presence; detect **drift** (unexpected processes/paths) and **policy violations**; publish **runtime events** to Scanner.WebService. +* **Admission (optional):** Kubernetes ValidatingAdmissionWebhook that enforces minimal posture (signed images, SBOM availability, known base images, policy PASS) **pre‑flight**. + +**Boundaries.** + +* Zastava **does not** compute SBOMs and does not sign; it **consumes** Scanner/WebService outputs and **enforces** backend policy verdicts. +* Zastava can **request** a delta scan when the baseline is missing/stale, but scanning is done by **Scanner.Worker**. +* On non‑K8s Docker hosts, Zastava runs as a host service with **observer‑only** features. + +--- + +## 1) Topology & processes + +### 1.1 Components (Kubernetes) + +``` +stellaops/zastava-observer # DaemonSet on every node (read-only host mounts) +stellaops/zastava-webhook # ValidatingAdmissionWebhook (Deployment, 2+ replicas) +``` + +### 1.2 Components (Docker/VM) + +``` +stellaops/zastava-agent # System service; watch Docker events; observer only +``` + +### 1.3 Dependencies + +* **Authority** (OIDC): short OpToks (DPoP/mTLS) for API calls to Scanner.WebService. +* **Scanner.WebService**: `/runtime/events` ingestion; `/policy/runtime` fetch. +* **OCI Registry** (optional): for direct referrers/sig checks if not delegated to backend. +* **Container runtime**: containerd/CRI‑O/Docker (read interfaces only). +* **Kubernetes API** (watch Pods in cluster; validating webhook). +* **Host mounts** (K8s DaemonSet): `/proc`, `/var/lib/containerd` (or CRI‑O), `/run/containerd/containerd.sock` (optional, read‑only). + +--- + +## 2) Data contracts + +### 2.1 Runtime event (observer → Scanner.WebService) + +```json +{ + "eventId": "9f6a…", + "when": "2025-10-17T12:34:56Z", + "kind": "CONTAINER_START|CONTAINER_STOP|DRIFT|POLICY_VIOLATION|ATTESTATION_STATUS", + "tenant": "tenant-01", + "node": "ip-10-0-1-23", + "runtime": { "engine": "containerd", "version": "1.7.19" }, + "workload": { + "platform": "kubernetes", + "namespace": "payments", + "pod": "api-7c9fbbd8b7-ktd84", + "container": "api", + "containerId": "containerd://...", + "imageRef": "ghcr.io/acme/api@sha256:abcd…", + "owner": { "kind": "Deployment", "name": "api" } + }, + "process": { + "pid": 12345, + "entrypoint": ["/entrypoint.sh", "--serve"], + "entryTrace": [ + {"file":"/entrypoint.sh","line":3,"op":"exec","target":"/usr/bin/python3"}, + {"file":"","op":"python","target":"/opt/app/server.py"} + ] + }, + "loadedLibs": [ + { "path": "/lib/x86_64-linux-gnu/libssl.so.3", "inode": 123456, "sha256": "…"}, + { "path": "/usr/lib/x86_64-linux-gnu/libcrypto.so.3", "inode": 123457, "sha256": "…"} + ], + "posture": { + "imageSigned": true, + "sbomReferrer": "present|missing", + "attestation": { "uuid": "rekor-uuid", "verified": true } + }, + "delta": { + "baselineImageDigest": "sha256:abcd…", + "changedFiles": ["/opt/app/server.py"], // optional quick signal + "newBinaries": [{ "path":"/usr/local/bin/helper","sha256":"…" }] + }, + "evidence": [ + {"signal":"procfs.maps","value":"/lib/.../libssl.so.3@0x7f..."}, + {"signal":"cri.task.inspect","value":"pid=12345"}, + {"signal":"registry.referrers","value":"sbom: application/vnd.cyclonedx+json"} + ] +} +``` + +### 2.2 Admission decision (webhook → API server) + +```json +{ + "admissionId": "…", + "namespace": "payments", + "podSpecDigest": "sha256:…", + "images": [ + { + "name": "ghcr.io/acme/api:1.2.3", + "resolved": "ghcr.io/acme/api@sha256:abcd…", + "signed": true, + "hasSbomReferrers": true, + "policyVerdict": "pass|warn|fail", + "reasons": ["unsigned base image", "missing SBOM"] + } + ], + "decision": "Allow|Deny", + "ttlSeconds": 300 +} +``` + +--- + +## 3) Observer — node agent (DaemonSet) + +### 3.1 Responsibilities + +* **Watch** container lifecycle (start/stop) via CRI (`/run/containerd/containerd.sock` gRPC read‑only) or `/var/log/containers/*.log` tail fallback. +* **Resolve** container → image digest, mount point rootfs. +* **Trace entrypoint**: attach **short‑lived** nsenter/exec to PID 1 in container, parse shell for `exec` chain (bounded depth), record **terminal program**. +* **Sample loaded libs**: read `/proc//maps` and `exe` symlink to collect **actually loaded** DSOs; compute **sha256** for each mapped file (bounded count/size). +* **Posture check** (cheap): + + * Image signature presence (if cosign policies are local; else ask backend). + * SBOM **referrers** presence (HEAD to registry, optional). + * Rekor UUID known (query Scanner.WebService by image digest). +* **Publish runtime events** to Scanner.WebService `/runtime/events` (batch & compress). +* **Request delta scan** if: no SBOM in catalog OR base differs from known baseline. + +### 3.2 Privileges & mounts (K8s) + +* **SecurityContext:** `runAsUser: 0`, `readOnlyRootFilesystem: true`, `allowPrivilegeEscalation: false`. +* **Capabilities:** `CAP_SYS_PTRACE` (optional if using nsenter trace), `CAP_DAC_READ_SEARCH`. +* **Host mounts (read‑only):** + + * `/proc` (host) → `/host/proc` + * `/run/containerd/containerd.sock` (or CRI‑O socket) + * `/var/lib/containerd/io.containerd.runtime.v2.task` (rootfs paths & pids) +* **Networking:** cluster‑internal egress to Scanner.WebService only. +* **Rate limits:** hard caps for bytes hashed and file count per container to avoid noisy tenants. + +### 3.3 Event batching + +* Buffer ND‑JSON; flush by **N events** or **2 s**. +* Backpressure: local disk ring buffer (50 MB default) if Scanner is temporarily unavailable; drop oldest after cap with **metrics** and **warning** event. + +--- + +## 4) Admission Webhook (Kubernetes) + +### 4.1 Gate criteria + +Configurable policy (fetched from backend and cached): + +* **Image signature**: must be cosign‑verifiable to configured key(s) or keyless identities. +* **SBOM availability**: at least one **CycloneDX** referrer or **Scanner.WebService** catalog entry. +* **Scanner policy verdict**: backend `PASS` required for namespaces/labels matching rules; allow `WARN` if configured. +* **Registry allowlists/denylists**. +* **Tag bans** (e.g., `:latest`). +* **Base image allowlists** (by digest). + +### 4.2 Flow + +```mermaid +sequenceDiagram + autonumber + participant K8s as API Server + participant WH as Zastava Webhook + participant SW as Scanner.WebService + + K8s->>WH: AdmissionReview(Pod) + WH->>WH: Resolve images to digests (remote HEAD/pull if needed) + WH->>SW: POST /policy/runtime { digests, namespace, labels } + SW-->>WH: { per-image: {signed, hasSbom, verdict, reasons}, ttl } + alt All pass + WH-->>K8s: AdmissionResponse(Allow, ttl) + else Any fail (enforce=true) + WH-->>K8s: AdmissionResponse(Deny, message) + end +``` + +**Caching:** Per‑digest result cached `ttlSeconds` (default 300 s). **Fail‑open** or **fail‑closed** is configurable per namespace. + +### 4.3 TLS & HA + +* Webhook has its own **serving cert** signed by cluster CA (or custom cert + CA bundle on configuration). +* Deployment ≥ 2 replicas; **leaderless**; stateless. + +--- + +## 5) Backend integration (Scanner.WebService) + +### 5.1 Ingestion endpoint + +`POST /api/v1/scanner/runtime/events` *(OpTok + DPoP/mTLS)* + +* Validates event schema; enforces rate caps by tenant/node; persists to **Mongo** (`runtime.events` capped collection or regular with TTL). +* Performs **correlation**: + + * Attach nearest **image SBOM** (inventory/usage) and **BOM‑Index** if known. + * If unknown/missing, schedule **delta scan** and return `202 Accepted`. +* Emits **derived signals** (usedByEntrypoint per component based on `/proc//maps`). + +### 5.2 Policy decision API (for webhook) + +`POST /api/v1/scanner/policy/runtime` + +Request: + +```json +{ + "namespace": "payments", + "labels": { "app": "api", "env": "prod" }, + "images": ["ghcr.io/acme/api@sha256:...", "ghcr.io/acme/nginx@sha256:..."] +} +``` + +Response: + +```json +{ + "ttlSeconds": 300, + "results": { + "ghcr.io/acme/api@sha256:...": { + "signed": true, + "hasSbom": true, + "policyVerdict": "pass", + "reasons": [], + "rekor": { "uuid": "..." } + }, + "ghcr.io/acme/nginx@sha256:...": { + "signed": false, + "hasSbom": false, + "policyVerdict": "fail", + "reasons": ["unsigned", "missing SBOM"] + } + } +} +``` + +--- + +## 6) Configuration (YAML) + +```yaml +zastava: + mode: + observer: true + webhook: true + authority: + issuer: "https://authority.internal" + aud: ["scanner","zastava"] # tokens for backend and self-id + backend: + url: "https://scanner-web.internal" + connectTimeoutMs: 500 + requestTimeoutMs: 1500 + retry: { attempts: 3, backoffMs: 200 } + runtime: + engine: "auto" # containerd|cri-o|docker|auto + procfs: "/host/proc" + collect: + entryTrace: true + loadedLibs: true + maxLibs: 256 + maxHashBytesPerContainer: 64_000_000 + maxDepth: 48 + admission: + enforce: true + failOpenNamespaces: ["dev", "test"] + verify: + imageSignature: true + sbomReferrer: true + scannerPolicyPass: true + cacheTtlSeconds: 300 + resolveTags: true # do remote digest resolution for tag-only images + limits: + eventsPerSecond: 50 + burst: 200 + perNodeQueue: 10_000 + security: + mounts: + containerdSock: "/run/containerd/containerd.sock:ro" + proc: "/proc:/host/proc:ro" + runtimeState: "/var/lib/containerd:ro" +``` + +--- + +## 7) Security posture + +* **AuthN/Z**: Authority OpToks (DPoP preferred) to backend; webhook does **not** require client auth from API server (K8s handles). +* **Least privileges**: read‑only host mounts; optional `CAP_SYS_PTRACE`; **no** host networking; **no** write mounts. +* **Isolation**: never exec untrusted code; nsenter only to **read** `/proc/`. +* **Data minimization**: do not exfiltrate env vars or command arguments unless policy explicitly enables diagnostic mode. +* **Rate limiting**: per‑node caps; per‑tenant caps at backend. +* **Hard caps**: bytes hashed, files inspected, depth of shell parsing. + +--- + +## 8) Metrics, logs, tracing + +**Observer** + +* `zastava.events_emitted_total{kind}` +* `zastava.proc_maps_samples_total{result}` +* `zastava.entrytrace_depth{p99}` +* `zastava.hash_bytes_total` +* `zastava.buffer_drops_total` + +**Webhook** + +* `zastava.admission_requests_total{decision}` +* `zastava.admission_latency_seconds` +* `zastava.cache_hits_total` +* `zastava.backend_failures_total` + +**Logs** (structured): node, pod, image digest, decision, reasons. +**Tracing**: spans for observe→batch→post; webhook request→resolve→respond. + +--- + +## 9) Performance & scale targets + +* **Observer**: ≤ **30 ms** to sample `/proc//maps` and compute quick hashes for ≤ 64 files; ≤ **200 ms** for full library set (256 libs). +* **Webhook**: P95 ≤ **8 ms** with warm cache; ≤ **50 ms** with one backend round‑trip. +* **Throughput**: 1k admission requests/min/replica; 5k runtime events/min/node with batching. + +--- + +## 10) Drift detection model + +**Signals** + +* **Process drift**: terminal program differs from **EntryTrace** baseline. +* **Library drift**: loaded DSOs not present in **Usage** SBOM view. +* **Filesystem drift**: new executable files under `/usr/local/bin`, `/opt`, `/app` with **mtime** after image creation. +* **Network drift** (optional): listening sockets on unexpected ports (from policy). + +**Action** + +* Emit `DRIFT` event with evidence; backend can **auto‑queue** a delta scan; policy may **escalate** to alert/block (Admission cannot block already‑running pods; rely on K8s policies/PodSecurity or operator action). + +--- + +## 11) Test matrix + +* **Engines**: containerd, CRI‑O, Docker; ensure PID resolution and rootfs mapping. +* **EntryTrace**: bash features (case, if, run‑parts, `.`/`source`), language launchers (python/node/java). +* **Procfs**: multiple arches, musl/glibc images; static binaries (maps minimal). +* **Admission**: unsigned images, missing SBOM referrers, tag‑only images, digest resolution, backend latency, cache TTL. +* **Perf/soak**: 500 Pods/node churn; webhook under HPA growth. +* **Security**: attempt privilege escalation disabled, read‑only mounts enforced, rate‑limit abuse. +* **Failure injection**: backend down (observer buffers, webhook fail‑open/closed), registry throttling, containerd socket unavailable. + +--- + +## 12) Failure modes & responses + +| Condition | Observer behavior | Webhook behavior | +| ------------------------------- | ---------------------------------------------- | ------------------------------------------------------ | +| Backend unreachable | Buffer to disk; drop after cap; emit metric | **Fail‑open/closed** per namespace config | +| PID vanished mid‑sample | Retry once; emit partial evidence | N/A | +| CRI socket missing | Fallback to K8s events only (reduced fidelity) | N/A | +| Registry digest resolve blocked | Defer to backend; mark `resolve=unknown` | Deny or allow per `resolveTags` & `failOpenNamespaces` | +| Excessive events | Apply local rate limit, coalesce | N/A | + +--- + +## 13) Deployment notes (K8s) + +**DaemonSet (snippet):** + +```yaml +apiVersion: apps/v1 +kind: DaemonSet +metadata: { name: zastava-observer, namespace: stellaops } +spec: + template: + spec: + serviceAccountName: zastava + hostPID: true + containers: + - name: observer + image: stellaops/zastava-observer:2.3 + securityContext: + runAsUser: 0 + readOnlyRootFilesystem: true + allowPrivilegeEscalation: false + capabilities: { add: ["SYS_PTRACE","DAC_READ_SEARCH"] } + volumeMounts: + - { name: proc, mountPath: /host/proc, readOnly: true } + - { name: containerd-sock, mountPath: /run/containerd/containerd.sock, readOnly: true } + - { name: containerd-state, mountPath: /var/lib/containerd, readOnly: true } + volumes: + - { name: proc, hostPath: { path: /proc } } + - { name: containerd-sock, hostPath: { path: /run/containerd/containerd.sock } } + - { name: containerd-state, hostPath: { path: /var/lib/containerd } } +``` + +**Webhook (snippet):** + +```yaml +apiVersion: admissionregistration.k8s.io/v1 +kind: ValidatingWebhookConfiguration +webhooks: +- name: gate.zastava.stella-ops.org + admissionReviewVersions: ["v1"] + sideEffects: None + failurePolicy: Ignore # or Fail + rules: + - operations: ["CREATE","UPDATE"] + apiGroups: [""] + apiVersions: ["v1"] + resources: ["pods"] + clientConfig: + service: + namespace: stellaops + name: zastava-webhook + path: /admit + caBundle: +``` + +--- + +## 14) Implementation notes + +* **Language**: Rust (observer) for low‑latency `/proc` parsing; Go/.NET viable too. Webhook can be .NET 10 for parity with backend. +* **CRI drivers**: pluggable (`containerd`, `cri-o`, `docker`). Prefer CRI over parsing logs. +* **Shell parser**: re‑use Scanner.EntryTrace grammar for consistent results (compile to WASM if observer is Rust/Go). +* **Hashing**: `BLAKE3` for speed locally, then convert to `sha256` (or compute `sha256` directly when budget allows). +* **Resilience**: never block container start; observer is **passive**; only webhook decides allow/deny. + +--- + +## 15) Roadmap + +* **eBPF** option for syscall/library load tracing (kernel‑level, opt‑in). +* **Windows containers** support (ETW providers, loaded modules). +* **Network posture** checks: listening ports vs policy. +* **Live **used‑by‑entrypoint** synthesis**: send compact bitset diff to backend to tighten Usage view. +* **Admission dry‑run** dashboards (simulate block lists before enforcing). + diff --git a/docs/README.md b/docs/README.md index b10929c1..884b0798 100755 --- a/docs/README.md +++ b/docs/README.md @@ -31,17 +31,26 @@ Everything here is open‑source and versioned — when you check out a git ta - **03 – [Vision & Road‑map](03_VISION.md)** - **04 – [Feature Matrix](04_FEATURE_MATRIX.md)** -### Reference & concepts -- **05 – [System Requirements Specification](05_SYSTEM_REQUIREMENTS_SPEC.md)** -- **07 – [High‑Level Architecture](40_ARCHITECTURE_OVERVIEW.md)** -- **08 – Module Specifications** - - [README](08_MODULE_SPECIFICATIONS/README.md) - - [`backend_api.md`](08_MODULE_SPECIFICATIONS/backend_api.md) - - [`zastava_scanner.md`](08_MODULE_SPECIFICATIONS/zastava_scanner.md) - - [`registry_scanner.md`](08_MODULE_SPECIFICATIONS/registry_scanner.md) - - [`nightly_scheduler.md`](08_MODULE_SPECIFICATIONS/nightly_scheduler.md) +### Reference & concepts +- **05 – [System Requirements Specification](05_SYSTEM_REQUIREMENTS_SPEC.md)** +- **07 – [High‑Level Architecture](07_HIGH_LEVEL_ARCHITECTURE.md)** +- **08 – Module Architecture Dossiers** + - [Scanner](ARCHITECTURE_SCANNER.md) + - [Feedser](ARCHITECTURE_FEEDSER.md) + - [Vexer](ARCHITECTURE_VEXER.md) + - [Signer](ARCHITECTURE_SIGNER.md) + - [Attestor](ARCHITECTURE_ATTESTOR.md) + - [Authority](ARCHITECTURE_AUTHORITY.md) + - [CLI](ARCHITECTURE_CLI.md) + - [Web UI](ARCHITECTURE_UI.md) + - [Zastava Runtime](ARCHITECTURE_ZASTAVA.md) + - [Release & Operations](ARCHITECTURE_DEVOPS.md) - **09 – [API & CLI Reference](09_API_CLI_REFERENCE.md)** - **10 – [Plug‑in SDK Guide](10_PLUGIN_SDK_GUIDE.md)** +- **10 – [Feedser CLI Quickstart](10_FEEDSER_CLI_QUICKSTART.md)** +- **30 – [Vexer Connector Packaging Guide](dev/30_VEXER_CONNECTOR_GUIDE.md)** +- **30 – Developer Templates** + - [Vexer Connector Skeleton](dev/templates/vexer-connector/) - **11 – [Authority Service](11_AUTHORITY.md)** - **11 – [Data Schemas](11_DATA_SCHEMAS.md)** - **12 – [Performance Workbook](12_PERFORMANCE_WORKBOOK.md)** @@ -57,7 +66,7 @@ Everything here is open‑source and versioned — when you check out a git ta - **21 – [Install Guide](21_INSTALL_GUIDE.md)** - **22 – [CI/CD Recipes Library](ci/20_CI_RECIPES.md)** - **23 – [FAQ](23_FAQ_MATRIX.md)** -- **24 – [Offline Update Kit Admin Guide](24_OUK_ADMIN_GUIDE.md)** +- **24 – [Offline Update Kit Admin Guide](24_OFFLINE_KIT.md)** - **25 – [Feedser Apple Connector Operations](ops/feedser-apple-operations.md)** - **26 – [Authority Key Rotation Playbook](ops/authority-key-rotation.md)** - **27 – [Feedser CCCS Connector Operations](ops/feedser-cccs-operations.md)** diff --git a/docs/TASKS.md b/docs/TASKS.md index 6291fe92..2b14c092 100644 --- a/docs/TASKS.md +++ b/docs/TASKS.md @@ -2,6 +2,7 @@ | ID | Status | Owner(s) | Depends on | Description | Exit Criteria | |----|--------|----------|------------|-------------|---------------| +| DOC7.README-INDEX | DONE (2025-10-17) | Docs Guild | — | Refresh index docs (docs/README.md + root README) after architecture dossier split and Offline Kit overhaul. | ✅ ToC reflects new component architecture docs; ✅ root README highlights updated doc set; ✅ Offline Kit guide linked correctly. | | DOC4.AUTH-PDG | REVIEW | Docs Guild, Plugin Team | PLG6.DOC | Copy-edit `docs/dev/31_AUTHORITY_PLUGIN_DEVELOPER_GUIDE.md`, export lifecycle diagram, add LDAP RFC cross-link. | ✅ PR merged with polish; ✅ Diagram committed; ✅ Slack handoff posted. | | DOC1.AUTH | DONE (2025-10-12) | Docs Guild, Authority Core | CORE5B.DOC | Draft `docs/11_AUTHORITY.md` covering architecture, configuration, bootstrap flows. | ✅ Architecture + config sections approved by Core; ✅ Samples reference latest options; ✅ Offline note added. | | DOC3.Feedser-Authority | DONE (2025-10-12) | Docs Guild, DevEx | FSR4 | Polish operator/runbook sections (DOC3/DOC5) to document Feedser authority rollout, bypass logging, and enforcement checklist. | ✅ DOC3/DOC5 updated with audit runbook references; ✅ enforcement deadline highlighted; ✅ Docs guild sign-off. | diff --git a/docs/dev/30_VEXER_CONNECTOR_GUIDE.md b/docs/dev/30_VEXER_CONNECTOR_GUIDE.md new file mode 100644 index 00000000..4affd54f --- /dev/null +++ b/docs/dev/30_VEXER_CONNECTOR_GUIDE.md @@ -0,0 +1,220 @@ +# Vexer Connector Packaging Guide + +> **Audience:** teams implementing new Vexer provider plug‑ins (CSAF feeds, +> OpenVEX attestations, etc.) +> **Prerequisites:** read `docs/ARCHITECTURE_VEXER.md` and the module +> `AGENTS.md` in `src/StellaOps.Vexer.Connectors.Abstractions/`. + +The Vexer connector SDK gives you: + +- `VexConnectorBase` – deterministic logging, SHA‑256 helpers, time provider. +- `VexConnectorOptionsBinder` – strongly typed YAML/JSON configuration binding. +- `IVexConnectorOptionsValidator` – custom validation hooks (offline defaults, auth invariants). +- `VexConnectorDescriptor` & metadata helpers for consistent telemetry. + +This guide explains how to package a connector so the Vexer Worker/WebService +can load it via the plugin host. + +--- + +## 1. Project layout + +Start from the template under +`docs/dev/templates/vexer-connector/`. It contains: + +``` +Vexer.MyConnector/ +├── src/ +│ ├── Vexer.MyConnector.csproj +│ ├── MyConnectorOptions.cs +│ ├── MyConnector.cs +│ └── MyConnectorPlugin.cs +└── manifest/ + └── connector.manifest.yaml +``` + +Key points: + +- Target `net10.0`, enable `TreatWarningsAsErrors`, reference the + `StellaOps.Vexer.Connectors.Abstractions` project (or NuGet once published). +- Keep project ID prefix `StellaOps.Vexer.Connectors.` so the + plugin loader can discover it with the default search pattern. + +### 1.1 csproj snippet + +```xml + + + net10.0 + enable + enable + true + + + + + +``` + +Adjust the `ProjectReference` for your checkout (or switch to a NuGet package +once published). + +--- + +## 2. Implement the connector + +1. **Options model** – create an options POCO with data-annotation attributes. + Bind it via `VexConnectorOptionsBinder.Bind` in your connector + constructor or `ValidateAsync`. +2. **Validator** – implement `IVexConnectorOptionsValidator` to add + complex checks (e.g., ensure both `clientId` and `clientSecret` are present). +3. **Connector** – inherit from `VexConnectorBase`. Implement: + - `ValidateAsync` – run binder/validators, log configuration summary. + - `FetchAsync` – stream raw documents to `context.RawSink`. + - `NormalizeAsync` – convert raw documents into `VexClaimBatch` via + format-specific normalizers (`context.Normalizers`). +4. **Plugin adapter** – expose the connector via a plugin entry point so the + host can instantiate it. + +### 2.1 Options binding example + +```csharp +public sealed class MyConnectorOptions +{ + [Required] + [Url] + public string CatalogUri { get; set; } = default!; + + [Required] + public string ApiKey { get; set; } = default!; + + [Range(1, 64)] + public int MaxParallelRequests { get; set; } = 4; +} + +public sealed class MyConnectorOptionsValidator : IVexConnectorOptionsValidator +{ + public void Validate(VexConnectorDescriptor descriptor, MyConnectorOptions options, IList errors) + { + if (!options.CatalogUri.StartsWith("https://", StringComparison.OrdinalIgnoreCase)) + { + errors.Add("CatalogUri must use HTTPS."); + } + } +} +``` + +Bind inside the connector: + +```csharp +private readonly MyConnectorOptions _options; + +public MyConnector(VexConnectorDescriptor descriptor, ILogger logger, TimeProvider timeProvider) + : base(descriptor, logger, timeProvider) +{ + // `settings` comes from the orchestrator; validators registered via DI. + _options = VexConnectorOptionsBinder.Bind( + descriptor, + VexConnectorSettings.Empty, + validators: new[] { new MyConnectorOptionsValidator() }); +} +``` + +Replace `VexConnectorSettings.Empty` with the actual settings from context +inside `ValidateAsync`. + +--- + +## 3. Plugin adapter & manifest + +Create a simple plugin class that implements +`StellaOps.Plugin.IConnectorPlugin`. The Worker/WebService plugin host uses +this contract today. + +```csharp +public sealed class MyConnectorPlugin : IConnectorPlugin +{ + private static readonly VexConnectorDescriptor Descriptor = + new("vexer:my-provider", VexProviderKind.Vendor, "My Provider VEX"); + + public string Name => Descriptor.DisplayName; + + public bool IsAvailable(IServiceProvider services) => true; // inject feature flags if needed + + public IFeedConnector Create(IServiceProvider services) + { + var logger = services.GetRequiredService>(); + var timeProvider = services.GetRequiredService(); + return new MyConnector(Descriptor, logger, timeProvider); + } +} +``` + +> **Note:** the Vexer Worker currently instantiates connectors through the +> shared `IConnectorPlugin` contract. Once a dedicated Vexer plugin interface +> lands you simply swap the base interface; the descriptor/connector code +> remains unchanged. + +Provide a manifest describing the assembly for operational tooling: + +```yaml +# manifest/connector.manifest.yaml +id: vexer-my-provider +assembly: StellaOps.Vexer.Connectors.MyProvider.dll +entryPoint: StellaOps.Vexer.Connectors.MyProvider.MyConnectorPlugin +description: > + Official VEX feed for ExampleCorp products (CSAF JSON, daily updates). +tags: + - vexer + - csaf + - vendor +``` + +Store manifests under `/opt/stella/vexer/plugins//manifest/` in +production so the deployment tooling can inventory and verify plug‑ins. + +--- + +## 4. Packaging workflow + +1. `dotnet publish -c Release` → copy the published DLLs to + `/opt/stella/vexer/plugins//`. +2. Place `connector.manifest.yaml` next to the binaries. +3. Restart the Vexer Worker or WebService (hot reload not supported yet). +4. Verify logs: `VEX-ConnectorLoader` should list the connector descriptor. + +### 4.1 Offline kits + +- Add the connector folder (binaries + manifest) to the Offline Kit bundle. +- Include a `settings.sample.yaml` demonstrating offline-friendly defaults. +- Document any external dependencies (e.g., SHA mirrors) in the manifest `notes` + field. + +--- + +## 5. Testing checklist + +- Unit tests around options binding & validators. +- Integration tests (future `StellaOps.Vexer.Connectors.Abstractions.Tests`) + verifying deterministic logging scopes: + `logger.BeginScope` should produce `vex.connector.id`, `vex.connector.kind`, + and `vex.connector.operation`. +- Deterministic SHA tests: repeated `CreateRawDocument` calls with identical + content must return the same digest. + +--- + +## 6. Reference template + +See `docs/dev/templates/vexer-connector/` for the full quick‑start including: + +- Sample options class + validator. +- Connector implementation inheriting from `VexConnectorBase`. +- Plugin adapter + manifest. + +Copy the directory, rename namespaces/IDs, then iterate on provider-specific +logic. + +--- + +*Last updated: 2025-10-17* diff --git a/docs/dev/templates/vexer-connector/manifest/connector.manifest.yaml b/docs/dev/templates/vexer-connector/manifest/connector.manifest.yaml new file mode 100644 index 00000000..307b4bd5 --- /dev/null +++ b/docs/dev/templates/vexer-connector/manifest/connector.manifest.yaml @@ -0,0 +1,8 @@ +id: vexer-my-provider +assembly: StellaOps.Vexer.Connectors.MyProvider.dll +entryPoint: StellaOps.Vexer.Connectors.MyProvider.MyConnectorPlugin +description: | + Example connector template. Replace metadata before shipping. +tags: + - vexer + - template diff --git a/docs/dev/templates/vexer-connector/src/MyConnector.cs b/docs/dev/templates/vexer-connector/src/MyConnector.cs new file mode 100644 index 00000000..d7a8eccd --- /dev/null +++ b/docs/dev/templates/vexer-connector/src/MyConnector.cs @@ -0,0 +1,72 @@ +using System.Collections.Generic; +using System.Collections.Immutable; +using System.Runtime.CompilerServices; +using Microsoft.Extensions.Logging; +using StellaOps.Vexer.Connectors.Abstractions; +using StellaOps.Vexer.Core; + +namespace StellaOps.Vexer.Connectors.MyProvider; + +public sealed class MyConnector : VexConnectorBase +{ + private readonly IEnumerable> _validators; + private MyConnectorOptions? _options; + + public MyConnector(VexConnectorDescriptor descriptor, ILogger logger, TimeProvider timeProvider, IEnumerable> validators) + : base(descriptor, logger, timeProvider) + { + _validators = validators; + } + + public override ValueTask ValidateAsync(VexConnectorSettings settings, CancellationToken cancellationToken) + { + _options = VexConnectorOptionsBinder.Bind( + Descriptor, + settings, + validators: _validators); + + LogConnectorEvent(LogLevel.Information, "validate", "MyConnector configuration loaded.", + new Dictionary + { + ["catalogUri"] = _options.CatalogUri, + ["maxParallelRequests"] = _options.MaxParallelRequests, + }); + + return ValueTask.CompletedTask; + } + + public override IAsyncEnumerable FetchAsync(VexConnectorContext context, CancellationToken cancellationToken) + { + if (_options is null) + { + throw new InvalidOperationException("Connector not validated."); + } + + return FetchInternalAsync(context, cancellationToken); + } + + private async IAsyncEnumerable FetchInternalAsync(VexConnectorContext context, [EnumeratorCancellation] CancellationToken cancellationToken) + { + LogConnectorEvent(LogLevel.Information, "fetch", "Fetching catalog window..."); + + // Replace with real HTTP logic. + await Task.Delay(10, cancellationToken); + + var metadata = BuildMetadata(builder => builder + .Add("sourceUri", _options!.CatalogUri) + .Add("window", context.Since?.ToString("O") ?? "full")); + + yield return CreateRawDocument( + VexDocumentFormat.CsafJson, + new Uri($"{_options.CatalogUri.TrimEnd('/')}/sample.json"), + new byte[] { 0x7B, 0x7D }, + metadata); + } + + public override ValueTask NormalizeAsync(VexRawDocument document, CancellationToken cancellationToken) + { + var claims = ImmutableArray.Empty; + var diagnostics = ImmutableDictionary.Empty; + return ValueTask.FromResult(new VexClaimBatch(document, claims, diagnostics)); + } +} diff --git a/docs/dev/templates/vexer-connector/src/MyConnectorOptions.cs b/docs/dev/templates/vexer-connector/src/MyConnectorOptions.cs new file mode 100644 index 00000000..c8cd857e --- /dev/null +++ b/docs/dev/templates/vexer-connector/src/MyConnectorOptions.cs @@ -0,0 +1,16 @@ +using System.ComponentModel.DataAnnotations; + +namespace StellaOps.Vexer.Connectors.MyProvider; + +public sealed class MyConnectorOptions +{ + [Required] + [Url] + public string CatalogUri { get; set; } = default!; + + [Required] + public string ApiKey { get; set; } = default!; + + [Range(1, 32)] + public int MaxParallelRequests { get; set; } = 4; +} diff --git a/docs/dev/templates/vexer-connector/src/MyConnectorOptionsValidator.cs b/docs/dev/templates/vexer-connector/src/MyConnectorOptionsValidator.cs new file mode 100644 index 00000000..04723e5b --- /dev/null +++ b/docs/dev/templates/vexer-connector/src/MyConnectorOptionsValidator.cs @@ -0,0 +1,15 @@ +using System.Collections.Generic; +using StellaOps.Vexer.Connectors.Abstractions; + +namespace StellaOps.Vexer.Connectors.MyProvider; + +public sealed class MyConnectorOptionsValidator : IVexConnectorOptionsValidator +{ + public void Validate(VexConnectorDescriptor descriptor, MyConnectorOptions options, IList errors) + { + if (!options.CatalogUri.StartsWith("https://", StringComparison.OrdinalIgnoreCase)) + { + errors.Add("CatalogUri must use HTTPS."); + } + } +} diff --git a/docs/dev/templates/vexer-connector/src/MyConnectorPlugin.cs b/docs/dev/templates/vexer-connector/src/MyConnectorPlugin.cs new file mode 100644 index 00000000..d12b573d --- /dev/null +++ b/docs/dev/templates/vexer-connector/src/MyConnectorPlugin.cs @@ -0,0 +1,27 @@ +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Logging; +using StellaOps.Plugin; +using StellaOps.Vexer.Connectors.Abstractions; +using StellaOps.Vexer.Core; + +namespace StellaOps.Vexer.Connectors.MyProvider; + +public sealed class MyConnectorPlugin : IConnectorPlugin +{ + private static readonly VexConnectorDescriptor Descriptor = new( + id: "vexer:my-provider", + kind: VexProviderKind.Vendor, + displayName: "My Provider VEX"); + + public string Name => Descriptor.DisplayName; + + public bool IsAvailable(IServiceProvider services) => true; + + public IFeedConnector Create(IServiceProvider services) + { + var logger = services.GetRequiredService>(); + var timeProvider = services.GetRequiredService(); + var validators = services.GetServices>(); + return new MyConnector(Descriptor, logger, timeProvider, validators); + } +} diff --git a/docs/dev/templates/vexer-connector/src/Vexer.MyConnector.csproj b/docs/dev/templates/vexer-connector/src/Vexer.MyConnector.csproj new file mode 100644 index 00000000..3f18fde3 --- /dev/null +++ b/docs/dev/templates/vexer-connector/src/Vexer.MyConnector.csproj @@ -0,0 +1,12 @@ + + + net10.0 + enable + enable + true + + + + + + diff --git a/docs/ops/feedser-certbund-operations.md b/docs/ops/feedser-certbund-operations.md index 3a4f0a2b..0b09a906 100644 --- a/docs/ops/feedser-certbund-operations.md +++ b/docs/ops/feedser-certbund-operations.md @@ -1,6 +1,6 @@ # Feedser CERT-Bund Connector Operations -_Last updated: 2025-10-15_ +_Last updated: 2025-10-17_ Germany’s Federal Office for Information Security (BSI) operates the Warn- und Informationsdienst (WID) portal. The Feedser CERT-Bund connector (`source:cert-bund:*`) ingests the public RSS feed, hydrates the portal’s JSON detail endpoint, and maps the result into canonical advisories while preserving the original German content. @@ -96,18 +96,30 @@ curl -s -b cookies.txt \ Iterate `page` until the response `content` array is empty. Pages 0–9 currently cover 2014→present. Persist JSON responses (plus SHA256) for Offline Kit parity. +> **Shortcut** – run `python tools/certbund_offline_snapshot.py --output seed-data/cert-bund` +> to bootstrap the session, capture the paginated search responses, and regenerate +> the manifest/checksum files automatically. Supply `--cookie-file` and `--xsrf-token` +> if the portal requires a browser-derived session (see options via `--help`). + ### 3.3 Export bundles ```bash -curl -s -b cookies.txt \ - -H "Accept: application/json" \ - -H "X-XSRF-TOKEN: ${XSRF}" \ - "https://wid.cert-bund.de/portal/api/securityadvisory/export?format=json&from=2020-01-01" \ - > certbund-2020-2025.json +python tools/certbund_offline_snapshot.py \ + --output seed-data/cert-bund \ + --start-year 2014 \ + --end-year "$(date -u +%Y)" ``` -Split long ranges per year and record provenance (`from`, `to`, SHA, capturedAt). Feedser can ingest these JSON payloads directly when operating offline. -Task `FEEDCONN-CERTBUND-02-009` tracks turning this workflow into a shipped Offline Kit artefact with manifests and documentation updates—coordinate with the Docs guild before publishing. +The helper stores yearly exports under `seed-data/cert-bund/export/`, +captures paginated search snapshots in `seed-data/cert-bund/search/`, +and generates the manifest + SHA files in `seed-data/cert-bund/manifest/`. +Split ranges according to your compliance window (default: one file per +calendar year). Feedser can ingest these JSON payloads directly when +operating offline. + +> When automatic bootstrap fails (e.g. portal introduces CAPTCHA), run the +> manual `curl` flow above, then rerun the helper with `--skip-fetch` to +> rebuild the manifest from the existing files. ### 3.4 Connector-driven catch-up