Rewrite architecture docs and add Vexer connector template
This commit is contained in:
@@ -25,4 +25,10 @@ Pipeline note: deployment workflows should template `etc/feedser.yaml` during CI
|
||||
injecting environment-specific Mongo credentials and telemetry endpoints. Upcoming
|
||||
releases will add Microsoft OAuth (Entra ID) authentication support—track the quickstart
|
||||
for integration steps once available.
|
||||
|
||||
## Documentation
|
||||
|
||||
- `docs/README.md` now consolidates the platform index and points to the updated high-level architecture.
|
||||
- Module architecture dossiers live under `docs/ARCHITECTURE_*.md`; the most relevant here are `docs/ARCHITECTURE_FEEDSER.md` (service layout, merge engine, exports) and `docs/ARCHITECTURE_CLI.md` (command surface, AOT packaging, auth flows). Related services such as the Signer, Attestor, Authority, Scanner, UI, Vexer, Zastava, and DevOps pipeline each have their own dossier.
|
||||
- Offline operation guidance moved to `docs/24_OFFLINE_KIT.md`, which details bundle composition, verification, and delta workflows. Feedser-specific connector operations stay in `docs/ops/feedser-certbund-operations.md` and companion runbooks under `docs/ops/`.
|
||||
|
||||
|
||||
@@ -1,388 +1,430 @@
|
||||
# 7 · High‑Level Architecture — **Stella Ops**
|
||||
Below is the **revised, consolidated** `high_level_architecture.md`.
|
||||
It **absorbs** all content from `components.md` so you have a single, authoritative file. No separate components doc is required.
|
||||
|
||||
---
|
||||
|
||||
## 0 Purpose & Scope
|
||||
# High‑Level Architecture — **Stella Ops** (Consolidated • 2025Q4)
|
||||
|
||||
Give contributors, DevOps engineers and auditors a **complete yet readable map** of the Core:
|
||||
|
||||
* Major runtime components and message paths.
|
||||
* Where plug‑ins, CLI helpers and runtime agents attach.
|
||||
* Technology choices that enable the sub‑5 second SBOM goal.
|
||||
* Typical operational scenarios (pipeline scan, mute, nightly re‑scan, etc.).
|
||||
|
||||
Anything enterprise‑only (signed PDF, custom/regulated TLS, LDAP, enforcement) **must arrive as a plug‑in**; the Core never hard‑codes those concerns.
|
||||
---
|
||||
## 1 Component Overview
|
||||
|
||||
| # | Component | Responsibility |
|
||||
|---|-----------|---------------|
|
||||
| 1 | **API Gateway** | REST endpoints (`/scan`, `/quota`, **`/token/offline`**); token auth; quota enforcement |
|
||||
| 2 | **Scan Service** | SBOM parsing, Delta‑SBOM cache, vulnerability lookup |
|
||||
| 3 | **Policy Engine** | YAML / (optional) Rego rule evaluation; verdict assembly |
|
||||
| 4 | **Quota Service** | Per‑token counters; **333 scans/day**; waits & HTTP 429 |
|
||||
| 5 | **Client‑JWT Issuer** | Issues 30‑day offline tokens; bundles them into OUK |
|
||||
| 5 | **Registry** | Anonymous internal Docker registry for agents, SBOM uploads |
|
||||
| 6 | **Web UI** | React/Blazor SPA; dashboards, policy editor, quota banner |
|
||||
| 7 | **Data Stores** | **Redis** (cache, quota) & **MongoDB** (SBOMs, findings, audit) |
|
||||
| 8 | **Plugin Host** | Hot‑load .NET DLLs; isolates community plug‑ins |
|
||||
| 9 | **Agents** | `sbom‑builder`, `Stella CLI` scanner CLI, future `StellaOpsAttestor` |
|
||||
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
subgraph "External Actors"
|
||||
DEV["Developer / DevSecOps / Manager"]
|
||||
CI["CI/CD Pipeline (e.g., Stella CLI)"]
|
||||
K8S["Kubernetes Cluster (e.g., Zastava Agent)"]
|
||||
end
|
||||
|
||||
subgraph "Stella Ops Runtime"
|
||||
subgraph "Core Services"
|
||||
CORE["Stella Core<br>(REST + gRPC APIs, Orchestration)"]
|
||||
REDIS[("Redis<br>(Cache, Queues, Trivy DB Mirror)")]
|
||||
MONGO[("MongoDB<br>(Optional: Long-term Storage)")]
|
||||
POL["Mute Policies<br>(OPA & YAML Evaluator)"]
|
||||
REG["StellaOps Registry<br>(Docker Registry v2)"]
|
||||
ATT["StellaOps Attestor<br>(SLSA + Rekor)"]
|
||||
end
|
||||
|
||||
subgraph "Agents & Builders"
|
||||
SB["SBOM Builder<br>(Go Binary: Extracts Layers, Generates SBOMs)"]
|
||||
SA["Stella CLI<br>(Pipeline Helper: Invokes Builder, Triggers Scans)"]
|
||||
ZA["Zastava Agent<br>(K8s Webhook: Enforces Policies, Inventories Containers)"]
|
||||
end
|
||||
|
||||
subgraph "Scanners & UI"
|
||||
TRIVY["Trivy Scanner<br>(Plugin Container: Vulnerability Scanning)"]
|
||||
UI["Web UI<br>(Vue3 + Tailwind: Dashboards, Policy Editor)"]
|
||||
CLI["Stella CLI<br>(CLI Helper: Triggers Scans, Mutes)"]
|
||||
end
|
||||
end
|
||||
|
||||
DEV -->|Browses Findings, Mutes CVEs| UI
|
||||
DEV -->|Triggers Scans| CLI
|
||||
CI -->|Generates SBOM, Calls /scan| SA
|
||||
K8S -->|Inventories Containers, Enforces Gates| ZA
|
||||
|
||||
UI -- "REST" --> CORE
|
||||
CLI -- "REST/gRPC" --> CORE
|
||||
SA -->|Scan Requests| CORE
|
||||
SB -->|Uploads SBOMs| CORE
|
||||
ZA -->|Policy Gates| CORE
|
||||
|
||||
CORE -- "Queues, Caches" --> REDIS
|
||||
CORE -- "Persists Data" --> MONGO
|
||||
CORE -->|Evaluates Policies| POL
|
||||
CORE -->|Attests Provenance| ATT
|
||||
CORE -->|Scans Vulnerabilities| TRIVY
|
||||
|
||||
SB -- "Pulls Images" --> REG
|
||||
SA -- "Pulls Images" --> REG
|
||||
ZA -- "Pulls Images" --> REG
|
||||
|
||||
style DEV fill:#f9f,stroke:#333
|
||||
style CI fill:#f9f,stroke:#333
|
||||
style K8S fill:#f9f,stroke:#333
|
||||
style CORE fill:#ddf,stroke:#333
|
||||
style REDIS fill:#fdd,stroke:#333
|
||||
style MONGO fill:#fdd,stroke:#333
|
||||
style POL fill:#dfd,stroke:#333
|
||||
style REG fill:#dfd,stroke:#333
|
||||
style ATT fill:#dfd,stroke:#333
|
||||
style SB fill:#fdf,stroke:#333
|
||||
style SA fill:#fdf,stroke:#333
|
||||
style ZA fill:#fdf,stroke:#333
|
||||
style TRIVY fill:#ffd,stroke:#333
|
||||
style UI fill:#ffd,stroke:#333
|
||||
style CLI fill:#ffd,stroke:#333
|
||||
```
|
||||
|
||||
* **Developer / DevSecOps / Manager** – browses findings, mutes CVEs, triggers scans.
|
||||
* **Stella CLI** – generates SBOMs and calls `/scan` during CI.
|
||||
* **Zastava Agent** – inventories live containers; Core ships it in *passive* mode only (no kill).
|
||||
|
||||
### 1.1 Client‑JWT Lifecycle (offline aware)
|
||||
|
||||
1. **Online instance** – user signs in → `/connect/token` issues JWT valid 12 h.
|
||||
2. **Offline instance** – JWT with `exp ≈ 30 days` ships in OUK; backend
|
||||
**re‑signs** and stores it during import.
|
||||
3. Tokens embed a `tier` claim (“Free”) and `maxScansPerDay: 333`.
|
||||
4. On expiry the UI surfaces a red toast **7 days** in advance.
|
||||
> **Purpose.** A complete, implementation‑ready map of Stella Ops: product vision, all runtime components, trust boundaries, tokens/licensing, control/data flows, storage, APIs, security, scale, DevOps, and verification logic.
|
||||
> **Scope.** This file **replaces** the separate `components.md`; all component details now live here.
|
||||
|
||||
---
|
||||
|
||||
## 2 · Component Responsibilities (runtime view)
|
||||
## 0) Product vision & principles
|
||||
|
||||
| Component | Core Responsibility | Implementation Highlights |
|
||||
| -------------------------- | ---------------------------------------------------------------------------------------------------------- | --------------------------------------------------------- |
|
||||
| **Stella Core** | Orchestrates scans, persists SBOM blobs, serves REST/gRPC APIs, fans out jobs to scanners & policy engine. | .NET {{ dotnet }}, CQRS, Redis Streams; pluggable runner interfaces. |
|
||||
| **SBOM Builder** | Extracts image layers, queries Core for *missing* layers, generates SBOMs (multi‑format), uploads blobs. | Go binary; wraps Trivy & Syft libs. |
|
||||
| **Stella CLI** | Pipeline‑side helper; invokes Builder, triggers scan, streams progress back to CI/CD. | Static musl build. |
|
||||
| **Zastava Agent** | K8s admission webhook enforcing policy verdicts before Pod creation. | Rust for sub‑10 ms latencies. |
|
||||
| **UI** | Angular 17 SPA for dashboards, settings, policy editor. | Tailwind CSS; Webpack module federation (future). |
|
||||
| **Redis** | Cache, queue, Trivy‑DB mirror, layer diffing. | Single instance or Sentinel. |
|
||||
| **MongoDB** (opt.) | Long‑term SBOM & policy audit storage (> 180 days). | Optional; enabled via flag. |
|
||||
| **StellaOps.Registry** | Anonymous read‑only Docker v2 registry with optional Cosign verification. | `registry :2` behind nginx reverse proxy. |
|
||||
| **StellaOps.MutePolicies** | YAML/Rego evaluator, policy version store, `/policy/*` API. | Embeds OPA‑WASM; falls back to `opa exec`. |
|
||||
| **StellaOpsAttestor** | Generate SLSA provenance & Rekor signatures; verify on demand. | Side‑car container; DSSE + Rekor CLI. |
|
||||
**Vision.** Stella Ops is a **deterministic SBOM + VEX platform** for CI/CD and runtime, tuned for **speed** (per‑layer deltas), **quiet output** (usage‑scoped views), and **verifiability** (DSSE + Rekor v2). It is **self‑hostable**, **air‑gap capable**, and **commercially enforceable**: only licensed installations can produce **Stella Ops‑verified** attestations.
|
||||
|
||||
All cross‑component calls use dependency‑injected interfaces—no
|
||||
intra‑component reach‑ins.
|
||||
**Operating principles.**
|
||||
|
||||
* **Scanner‑owned SBOMs.** We generate our own BOMs; we do not warehouse third‑party SBOM content (we can **link** to attested SBOMs).
|
||||
* **Deterministic evidence.** Facts come from package DBs, installed metadata, linkers, and verified attestations; no fuzzy guessing in the core.
|
||||
* **Per‑layer caching.** Cache fragments by **layer digest** and compose image SBOMs via **CycloneDX BOM‑Link** / **SPDX ExternalRef**.
|
||||
* **Inventory vs Usage.** Always record the full **inventory** of what exists; separately present **usage** (entrypoint closure + loaded libs).
|
||||
* **Backend decides.** PASS/FAIL is produced by **Policy** + **VEX** + **Advisories**. The scanner reports facts.
|
||||
* **Attest or it didn’t happen.** Every export is signed as **in‑toto/DSSE** and logged in **Rekor v2**.
|
||||
* **Sovereign‑ready.** Cloud is used only for licensing and optional endorsement; everything else is first‑party and self‑hostable.
|
||||
|
||||
---
|
||||
|
||||
## 3 · Principal Backend Modules & Plug‑in Hooks
|
||||
## 1) Service topology & trust boundaries
|
||||
|
||||
| Namespace | Responsibility | Built‑in Tech / Default | Plug‑in Contract |
|
||||
| --------------- | -------------------------------------------------- | ----------------------- | ------------------------------------------------- |
|
||||
| `configuration` | Parse env/JSON, health‑check endpoint | .NET {{ dotnet }} Options | `IConfigValidator` |
|
||||
| `identity` | Embedded OAuth2/OIDC (OpenIddict 6) | MIT OpenIddict | `IIdentityProvider` for LDAP/SAML/JWT gateway |
|
||||
| `pluginloader` | Discover DLLs, SemVer gate, optional Cosign verify | Reflection + Cosign | `IPluginLifecycleHook` for telemetry |
|
||||
| `scanning` | SBOM‑ & image‑flow orchestration; runner pool | Trivy CLI (default) | `IScannerRunner` – e.g., Grype, Copacetic, Clair |
|
||||
| `feedser` (vulnerability ingest/merge/export service) | Nightly NVD merge & feed enrichment | Hangfire job | drop-in `*.Schedule.dll` for OSV, GHSA, NVD 2.0, CNNVD, CNVD, ENISA, JVN and BDU feeds |
|
||||
| `tls` | TLS provider abstraction | OpenSSL | `ITlsProvider` for custom suites (incl. **SM2**, where law or security requires it) |
|
||||
| `reporting` | Render HTML/PDF reports | RazorLight | `IReportRenderer` |
|
||||
| `ui` | Angular SPA & i18n | Angular {{ angular }} | new locales via `/locales/{lang}.json` |
|
||||
| `scheduling` | Cron + retries | Hangfire | any recurrent job via `*.Schedule.dll` |
|
||||
### 1.1 Runtime inventory (first‑party)
|
||||
|
||||
```mermaid
|
||||
classDiagram
|
||||
class configuration
|
||||
class identity
|
||||
class pluginloader
|
||||
class scanning
|
||||
class feedser
|
||||
class tls
|
||||
class reporting
|
||||
class ui
|
||||
class scheduling
|
||||
| Service / Tool | Container image | Core role | Scale pattern |
|
||||
| ------------------------------- | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------- |
|
||||
| **Scanner.WebService** | `stellaops/scanner-web` | Control plane for scans; catalog; SBOM composition (inventory & usage); diff; exports. | Stateless; N replicas behind LB. |
|
||||
| **Scanner.Worker** | `stellaops/scanner-worker` | Runs analyzers (OS, Lang: Java/Node/Python/Go/.NET/Rust, Native ELF/PE/Mach‑O, EntryTrace); emits per‑layer SBOMs and composes image SBOMs. | Horizontal; queue‑driven; sharded by layer digest. |
|
||||
| **Scanner.Sbomer.BuildXPlugin** | `stellaops/sbom-indexer` | BuildKit **generator** for build‑time SBOMs as OCI **referrers**. | CI‑side; ephemeral. |
|
||||
| **Scanner.Sbomer.DockerImage** | `stellaops/scanner-cli` | CLI‑orchestrated scanner container for post‑build scans. | Local/CI; ephemeral. |
|
||||
| **Feedser.WebService** | `stellaops/feedser-web` | Vulnerability ingest/normalize/merge/export (JSON + Trivy DB). | HA via Mongo locks. |
|
||||
| **Vexer.WebService** | `stellaops/vexer-web` | VEX ingest/normalize/consensus; conflict retention; exports. | HA via Mongo locks. |
|
||||
| **Policy Engine** | (in `scanner-web`) | YAML DSL evaluator (waivers, vendor preferences, KEV/EPSS, license, usage‑gating); produces **policy digest**. | In‑process; cache per digest. |
|
||||
| **Signer** | `stellaops/signer` | **Hard gate:** validates entitlement + release integrity; mints signing cert (Fulcio keyless) or uses KMS; signs DSSE. | Stateless; HPA by QPS. |
|
||||
| **Attestor** | `stellaops/attestor` | Posts DSSE bundles to **Rekor v2**; verification endpoints. | Stateless; HPA by QPS. |
|
||||
| **Authority** | `stellaops/authority` | On‑prem OIDC issuing **short‑lived OpToks** with DPoP/mTLS sender constraint. | HA behind LB. |
|
||||
| **Zastava** (Runtime) | `stellaops/zastava` | Runtime inspector/enforcer (observer + optional Admission Webhook). | DaemonSet + Webhook. |
|
||||
| **Web UI** | `stellaops/ui` | Angular app for scans, diffs, policy, VEX, runtime, reports. | Stateless. |
|
||||
| **StellaOps.Cli** | `stellaops/cli` | CLI for init/scan/export/diff/policy/report/verify; Buildx helper. | Local/CI. |
|
||||
|
||||
class AllModules
|
||||
### 1.2 Third‑party (self‑hosted)
|
||||
|
||||
configuration ..> identity : Uses
|
||||
identity ..> pluginloader : Authenticates Plugins
|
||||
pluginloader ..> scanning : Loads Scanner Runners
|
||||
scanning ..> feedser : Triggers Feed Merges
|
||||
tls ..> AllModules : Provides TLS Abstraction
|
||||
reporting ..> ui : Renders Reports for UI
|
||||
scheduling ..> feedser : Schedules Nightly Jobs
|
||||
* **Fulcio** (Sigstore CA) — issues short‑lived signing certs (keyless).
|
||||
* **Rekor v2** (tile‑backed transparency log).
|
||||
* **MinIO** — S3‑compatible object store with lifecycle & Object Lock.
|
||||
* **MongoDB** — catalog, advisories, VEX.
|
||||
* **Queue** — Redis Streams / NATS / RabbitMQ (pluggable).
|
||||
* **OCI Registry** — must support **Referrers API** (discover SBOMs/signatures).
|
||||
|
||||
note for scanning "Pluggable: ISScannerRunner<br>e.g., Trivy, Grype"
|
||||
note for feedser "Pluggable: *.Schedule.dll<br>e.g., OSV, GHSA Feeds"
|
||||
note for identity "Pluggable: IIdentityProvider<br>e.g., LDAP, SAML"
|
||||
note for reporting "Pluggable: IReportRenderer<br>e.g., Custom PDF"
|
||||
```
|
||||
### 1.3 Cloud licensing (Stella Ops)
|
||||
|
||||
**When remaining = 0:**
|
||||
API returns `429 Too Many Requests`, `Retry‑After: <UTC‑midnight>` (sequence omitted for brevity).
|
||||
* **Licensing Service** (`www.stella-ops.org`) — issues long‑lived **License Tokens (LT)**; exchanges LT → **Proof‑of‑Entitlement (PoE)** bound to an installation key; revoke/introspect PoE; optional cross‑log **endorsement**.
|
||||
|
||||
---
|
||||
|
||||
## 4 · Data Flows
|
||||
|
||||
### 4.1 SBOM‑First (≤ 5 s P95)
|
||||
|
||||
Builder produces SBOM locally, so Core never touches the Docker
|
||||
socket.
|
||||
Trivy path hits ≤ 5 s on alpine:3.19 with warmed DB.
|
||||
Image‑unpack fallback stays ≤ 10 s for 200 MB images.
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant CI as CI/CD Pipeline (Stella CLI)
|
||||
participant SB as SBOM Builder
|
||||
participant CORE as Stella Core
|
||||
participant REDIS as Redis Queue
|
||||
participant RUN as Scanner Runner (e.g., Trivy)
|
||||
participant POL as Policy Evaluator
|
||||
|
||||
CI->>SB: Invoke SBOM Generation
|
||||
SB->>CORE: Check Missing Layers (/layers/missing)
|
||||
CORE->>REDIS: Query Layer Diff (SDIFF)
|
||||
REDIS-->>CORE: Missing Layers List
|
||||
CORE-->>SB: Return Missing Layers
|
||||
SB->>SB: Generate Delta SBOM
|
||||
SB->>CORE: Upload SBOM Blob (POST /scan(sbom))
|
||||
CORE->>REDIS: Enqueue Scan Job
|
||||
REDIS->>RUN: Fan Out to Runner
|
||||
RUN->>RUN: Perform Vulnerability Scan
|
||||
RUN-->>CORE: Return Scan Results
|
||||
CORE->>POL: Evaluate Mute Policies
|
||||
POL-->>CORE: Policy Verdict
|
||||
CORE-->>CI: JSON Verdict & Progress Stream
|
||||
Note over CORE,CI: Achieves ≤5s P95 with Warmed DB
|
||||
```
|
||||
|
||||
### 4.2 Delta SBOM
|
||||
|
||||
Builder collects layer digests.
|
||||
`POST /layers/missing` → Redis SDIFF → missing layer list (< 20 ms).
|
||||
SBOM generated only for those layers and uploaded.
|
||||
|
||||
### 4.3 Feedser Harvest & Export
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant SCHED as Feedser Scheduler
|
||||
participant CONN as Source Connector Plug-in
|
||||
participant FEEDSER as Feedser Core
|
||||
participant MONGO as MongoDB (Canonical Advisories)
|
||||
participant EXPORT as Exporter (JSON / Trivy DB)
|
||||
participant ART as Artifact Store / Offline Kit
|
||||
|
||||
SCHED->>CONN: Trigger window (init/resume)
|
||||
CONN->>CONN: Fetch source documents + metadata
|
||||
CONN->>FEEDSER: Submit raw document for parsing
|
||||
FEEDSER->>FEEDSER: Parse & normalize to DTO
|
||||
FEEDSER->>FEEDSER: Merge & deduplicate canonical advisory
|
||||
FEEDSER->>MONGO: Write advisory, provenance, merge_event
|
||||
FEEDSER->>EXPORT: Queue export delta request
|
||||
EXPORT->>MONGO: Read canonical snapshot/deltas
|
||||
EXPORT->>EXPORT: Build deterministic JSON & Trivy DB artifacts
|
||||
EXPORT->>ART: Publish artifacts / Offline Kit bundle
|
||||
ART-->>FEEDSER: Record export state + digests
|
||||
```
|
||||
|
||||
### 4.4 Identity & Auth Flow
|
||||
|
||||
OpenIddict issues JWTs via client‑credentials or password grant.
|
||||
An IIdentityProvider plug‑in can delegate to LDAP, SAML or external OIDC
|
||||
without Core changes.
|
||||
---
|
||||
## 5 · Runtime Helpers
|
||||
|
||||
| Helper | Form | Purpose | Extensible Bits |
|
||||
|-----------|---------------------------------------|--------------------------------------------------------------------|-------------------------------------------|
|
||||
| **Stella CLI** | Distroless CLI | Generates SBOM, calls `/scan`, honours threshold flag | `--engine`, `--pdf-out` piped to plug‑ins |
|
||||
| **Zastava** | Static Go binary / DaemonSet | Watches Docker/CRI‑O events; uploads SBOMs; can enforce gate | Policy plug‑in could alter thresholds |
|
||||
|
||||
---
|
||||
|
||||
## 6 · Persistence & Cache Strategy
|
||||
|
||||
| Store | Primary Use | Why chosen |
|
||||
|----------------|-----------------------------------------------|--------------------------------|
|
||||
| **MongoDB** | Feedser canonical advisories, merge events, export state | Deterministic canonical store with flexible schema |
|
||||
| **Redis 7** | CLI quotas, short-lived job scheduling, layer diff cache | Sub-1 ms P99 latency for hot-path coordination |
|
||||
| **Local tmpfs**| Trivy layer cache (`/var/cache/trivy`) | Keeps disk I/O off hot path |
|
||||
### 1.4 Diagram (control/data planes & trust)
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph "Persistence Layers"
|
||||
REDIS[(Redis: Quotas & Short-lived Queues<br>Sub-1ms P99)]
|
||||
MONGO[(MongoDB: Canonical Advisories<br>Merge Events & Export State)]
|
||||
TMPFS[(Local tmpfs: Trivy Layer Cache<br>Low I/O Overhead)]
|
||||
end
|
||||
subgraph Cloud["www.stella-ops.org (Cloud)"]
|
||||
LS[Licensing Service<br/>LT→PoE / revoke / introspect]
|
||||
end
|
||||
|
||||
CORE["Stella Core"] -- Queues & SBOM Cache --> REDIS
|
||||
CORE -- Long-term Storage --> MONGO
|
||||
TRIVY["Trivy Scanner"] -- Layer Unpack Cache --> TMPFS
|
||||
subgraph OnPrem["Customer Site (Self-hosted)"]
|
||||
Auth[Authority (OIDC)\nOpTok (DPoP/mTLS)]
|
||||
SW[Scanner.WebService]
|
||||
WK[Scanner.Worker xN]
|
||||
FEED[Feedser]
|
||||
VEX[Vexer]
|
||||
POL[Policy Engine (in Scanner.Web)]
|
||||
SGN[Signer\n(entitlement + signing)]
|
||||
ATT[Attestor\n(Rekor v2 submit/verify)]
|
||||
UI[Web UI (Angular)]
|
||||
Z[Zastava\n(Runtime Inspector/Enforcer)]
|
||||
MIN[(MinIO S3)]
|
||||
MGO[(MongoDB)]
|
||||
QUE[(Queue/Streams)]
|
||||
end
|
||||
|
||||
style REDIS fill:#fdd,stroke:#333
|
||||
style MONGO fill:#dfd,stroke:#333
|
||||
style TMPFS fill:#ffd,stroke:#333
|
||||
CLI[StellaOps.Cli / Buildx Plugin]
|
||||
REG[(OCI Registry with Referrers)]
|
||||
FUL[ Fulcio ]
|
||||
REK[ Rekor v2 (tiles) ]
|
||||
|
||||
CLI -->|scan/build| SW
|
||||
SW -->|jobs| QUE
|
||||
QUE --> WK
|
||||
WK --> MIN
|
||||
SW --> MGO
|
||||
FEED --> MGO
|
||||
VEX --> MGO
|
||||
UI --> SW
|
||||
Z --> SW
|
||||
|
||||
SGN <--> Auth
|
||||
SGN --> FUL
|
||||
SGN -->|mTLS| ATT
|
||||
ATT --> REK
|
||||
|
||||
SGN <-->|verify referrers| REG
|
||||
```
|
||||
|
||||
**Trust boundaries.** Only **Signer** can sign; only **Attestor** can write to **Rekor v2**. Scanner/UI never sign.
|
||||
|
||||
---
|
||||
|
||||
## 2) Licensing & tokens (installation‑ready, theft‑resistant)
|
||||
|
||||
**Two‑token model.**
|
||||
|
||||
* **License Token (LT)** — long‑lived JWT from **Licensing Service**; used **once** to enroll the installation; never used in hot path.
|
||||
* **Proof‑of‑Entitlement (PoE)** — bound to the installation key (mTLS client cert **or** DPoP‑bound JWT with `cnf`); medium‑lived; renewable; revocable.
|
||||
* **Operational token (OpTok)** — 2–5 min OIDC token from **Authority**, **sender‑constrained** (DPoP or mTLS). Used to authenticate to **Signer**/**Scanner.WebService**.
|
||||
|
||||
**Signer enforces both:** PoE proves entitlement; OpTok proves “who is calling now”. It also **independently verifies** the **scanner image digest** is **Stella Ops‑signed** via **Referrers + cosign** before signing anything.
|
||||
|
||||
**Enrollment sequence (LT → PoE).**
|
||||
|
||||
```plantuml
|
||||
@startuml
|
||||
actor Operator
|
||||
participant "Install Agent" as IA
|
||||
participant "Licensing Service" as LS
|
||||
Operator -> IA: Provide LT
|
||||
IA -> IA: Generate K_inst
|
||||
IA -> LS: /license/enroll {LT, pub(K_inst)}
|
||||
LS --> IA: PoE (mTLS client cert or JWT with cnf=K_inst), CRL/OCSP/introspect
|
||||
@enduml
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7 · Typical Scenarios
|
||||
## 3) Scanner subsystem (facts engine)
|
||||
|
||||
| # | Flow | Steps |
|
||||
|---------|----------------------------|-------------------------------------------------------------------------------------------------|
|
||||
| **S‑1** | Pipeline Scan & Alert | Stella CLI → SBOM → `/scan` → policy verdict → CI exit code & link to *Scan Detail* |
|
||||
| **S‑2** | Mute Noisy CVE | Dev toggles **Mute** in UI → rule stored in Redis → next build passes |
|
||||
| **S‑3** | Nightly Re‑scan | `SbomNightly.Schedule` re‑queues SBOMs (mask‑filter) → dashboard highlights new Criticals |
|
||||
| **S‑4** | Feed Update Cycle | `Feedser (vulnerability ingest/merge/export service)` refreshes feeds → UI *Feed Age* tile turns green |
|
||||
| **S‑5** | Custom Report Generation | Plug‑in registers `IReportRenderer` → `/report/custom/{digest}` → CI downloads artifact |
|
||||
### 3.1 Analyzers (deterministic only)
|
||||
|
||||
* **OS packages:** apk/dpkg/rpm (Linux); Windows MSI/SxS/GAC (M2).
|
||||
* **Language (installed state):**
|
||||
|
||||
* Java (pom.properties / MANIFEST) → `pkg:maven/...`
|
||||
* Node (`node_modules/*/package.json`) → `pkg:npm/...`
|
||||
* Python (`*.dist-info/METADATA`) → `pkg:pypi/...`
|
||||
* Go (buildinfo) → `pkg:golang/...`
|
||||
* .NET (`*.deps.json`) → `pkg:nuget/...`
|
||||
* **Rust:** deterministic **language markers** (symbol mangling) and crates only when present; otherwise `bin:{sha256}`.
|
||||
* **Native:** ELF/PE/Mach‑O imports, DT_NEEDED, RPATH/RUNPATH, symbol versions, PE version info.
|
||||
* **EntryTrace:** parse `ENTRYPOINT`/`CMD`; shell AST; resolve launchers (Java/Node/Python) to terminal program; record file:line chain.
|
||||
|
||||
### 3.2 Caching & composition
|
||||
|
||||
* **Layer cache:** `{layerDigest → SBOM fragment + analyzer meta}`.
|
||||
* **File CAS:** `{sha256(file) → parse result (ELF/JAR metadata/etc.)}`.
|
||||
* **Composition:** build **image SBOMs** from fragments via **BOM‑Link/ExternalRef**; emit **two views**:
|
||||
|
||||
* **Inventory** (complete filesystem inventory).
|
||||
* **Usage** (entrypoint closure + linked libs).
|
||||
* **Transport:** JSON **and** **CycloneDX Protobuf** (compact, fast to parse).
|
||||
* **Index:** BOM‑Index sidecar with purl table + roaring bitmap + `usedByEntrypoint` flag for fast joins.
|
||||
|
||||
### 3.3 Diff (image → layer → package)
|
||||
|
||||
* Added / Removed / Version‑changed changes, **attributed** to the layer that caused them.
|
||||
* Raw diffs preserved; backend view applies **VEX + Policy**.
|
||||
|
||||
### 3.4 Build‑time SBOMs (fast CI path)
|
||||
|
||||
* Buildx **generator** runs analyzers during `docker buildx build --attest=type=sbom,generator=stellaops/sbom-indexer`, attaches SBOMs as **OCI referrers**.
|
||||
* Scanner.WebService can trust these (policy‑configurable) and **skip** re‑scan; DSSE + Rekor v2 can be done either at build time or post‑push via Signer/Attestor.
|
||||
|
||||
---
|
||||
|
||||
## 4) Backend evaluation (decider)
|
||||
|
||||
### 4.1 Feedser (advisories)
|
||||
|
||||
* Ingests vendor, distro, OSS feeds; normalizes & merges; persists canonical advisories in Mongo; exports **deterministic JSON** and **Trivy DB**.
|
||||
* Offline kit bundles for air‑gapped sites.
|
||||
|
||||
### 4.2 Vexer (VEX)
|
||||
|
||||
* Ingests **OpenVEX / CSAF VEX / CycloneDX VEX**; normalizes claims; retains conflicts; computes **consensus** with provider trust weights and justification gates.
|
||||
|
||||
### 4.3 Policy Engine (YAML DSL)
|
||||
|
||||
* Matchers: `image/repo/env/purl/cve/vendor/source/path/layerDigest/usedByEntrypoint`
|
||||
* Actions: `ignore(until, justification)`, `fail`, `warn`, `defer`, `requireVEX{vendors, justifications}`, `escalate {sev, KEV, EPSS}`, license constraints.
|
||||
* Produces a **policy digest** (SHA‑256 of canonicalized policy).
|
||||
|
||||
### 4.4 PASS/FAIL flow
|
||||
|
||||
1. SBOM (Inventory / Usage) → join with **Feedser** advisories.
|
||||
2. Apply **Vexer** consensus (statuses & justifications).
|
||||
3. Apply **Policy**; compute PASS/FAIL with waiver TTLs.
|
||||
4. Sign the **final report** (DSSE via **Signer**) and log to **Rekor v2** via **Attestor**.
|
||||
|
||||
---
|
||||
|
||||
## 5) Runtime enforcement (Zastava)
|
||||
|
||||
* **Observer:** inventories running containers, checks image signatures, SBOM presence (referrers), detects drift (entrypoint chain divergence), flags unapproved images.
|
||||
* **Admission Webhook (optional):** blocks policy‑fail pods (dry‑run first).
|
||||
* **Integration:** posts runtime events to Scanner.WebService; can request **delta scans** on changed layers.
|
||||
|
||||
---
|
||||
|
||||
## 6) Storage & catalogs (MinIO/Mongo)
|
||||
|
||||
**MinIO layout**
|
||||
|
||||
```
|
||||
s3://stellaops/
|
||||
layers/<sha256>/sbom.cdx.json.zst
|
||||
layers/<sha256>/sbom.spdx.json.zst
|
||||
images/<imgDigest>/inventory.cdx.pb
|
||||
images/<imgDigest>/usage.cdx.pb
|
||||
indexes/<imgDigest>/bom-index.bin
|
||||
attest/<artifactSha256>.dsse.json
|
||||
```
|
||||
|
||||
**Catalog (Mongo)**
|
||||
|
||||
* `artifacts` (type/format/sha/size/rekor/ttl/immutable/refCount/createdAt)
|
||||
* `images`, `layers`, `links`, `lifecycleRules`
|
||||
|
||||
**Retention**
|
||||
|
||||
* MinIO **ILM** for coarse TTL; Scanner.WebService GC decrements `refCount` and deletes unreferenced metadata; **Object Lock** for immutable classes (auditable artifacts).
|
||||
|
||||
---
|
||||
|
||||
## 7) APIs (consolidated surface)
|
||||
|
||||
### 7.1 Scanner.WebService
|
||||
|
||||
```
|
||||
POST /api/scans { imageRef|digest, force? } → { scanId }
|
||||
GET /api/scans/{id} → { status, digests, artifacts[] }
|
||||
GET /api/sboms/{imageDigest} ?format=cdx-json|cdx-pb|spdx-json&view=inventory|usage
|
||||
GET /api/diff?old=<digest>&new=<digest> → { added[], removed[], changed[], byLayer[] }
|
||||
POST /api/exports { imageDigest, format, view } → { artifactId, rekorUrl }
|
||||
POST /api/reports { imageDigest, policyRevision? } → { reportId, rekorUrl }
|
||||
GET /api/catalog/artifacts/{id} → { size, ttl, immutable, rekor, refs }
|
||||
GET /healthz | /readyz | /metrics
|
||||
```
|
||||
|
||||
### 7.2 Signer (mTLS; hard gate)
|
||||
|
||||
```
|
||||
POST /sign/dsse # body: {subjectHash, imageDigest, predicate}; headers: OpTok (DPoP/mTLS) + PoE
|
||||
GET /verify/referrers?imageDigest=sha256:... # is this image StellaOps-signed?
|
||||
```
|
||||
|
||||
### 7.3 Attestor (mTLS)
|
||||
|
||||
```
|
||||
POST /rekor/entries # DSSE bundle → {uuid, index, proof, logURL}
|
||||
GET /rekor/entries/{uuid}
|
||||
```
|
||||
|
||||
### 7.4 Authority (OIDC)
|
||||
|
||||
* `/.well-known/openid-configuration`, `/oauth/token` (DPoP/mTLS), `/oauth/introspect`, `/jwks`
|
||||
|
||||
### 7.5 Licensing (cloud)
|
||||
|
||||
```
|
||||
POST /license/enroll { LT, pubKey } → PoE + introspection endpoints
|
||||
POST /license/revoke { license_id } → ok
|
||||
POST /license/introspect { poe } → { active, claims, exp }
|
||||
POST /attest/endorse { bundle } → endorsement bundle (optional)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8) Security & verifiability
|
||||
|
||||
* **Sender‑constrained tokens.** All operational calls use **DPoP** (RFC 9449) or **mTLS‑bound** tokens (RFC 8705).
|
||||
* **Entitlement.** **PoE** is mandatory; revocation honored online.
|
||||
* **Release integrity.** **Signer** independently verifies **scanner image digest** via **Referrers + cosign** before signing.
|
||||
* **Separation of duties.** Scanner/UI cannot sign; only **Signer** can sign; only **Attestor** can write to **Rekor v2**.
|
||||
* **Verifiers.** Anyone can verify: DSSE signature → certificate chain to **Stella Ops Fulcio/KMS root** → **Rekor v2** inclusion.
|
||||
* **Community vs Authorized.** Free/community runs throttled with no official attestations; authorized runs full speed and produce **Stella Ops‑verified** bundles.
|
||||
|
||||
**DSSE predicate (SBOM/report)**
|
||||
|
||||
```json
|
||||
{
|
||||
"predicateType": "https://stella-ops.org/attestations/sbom/1",
|
||||
"subject": [{ "name": "s3://stellaops/images/<digest>/inventory.cdx.pb", "digest": { "sha256": "<sha256>" } }],
|
||||
"predicate": {
|
||||
"image_digest": "<sha256:...>",
|
||||
"stellaops_version": "2.3.1 (2027.04)",
|
||||
"license_id": "LIC-9F2A...",
|
||||
"customer_id": "CUST-ACME",
|
||||
"plan": "pro",
|
||||
"policy_digest": "sha256:...",
|
||||
"views": ["inventory","usage"],
|
||||
"created": "2025-10-17T12:34:56Z"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**BOM‑Index sidecar**
|
||||
Binary header + purl table + roaring bitmaps; optional `usedByEntrypoint` flags for fast policy joins.
|
||||
|
||||
---
|
||||
|
||||
## 9) Scale, performance & quotas
|
||||
|
||||
* **Workers:** horizontal; **distributed lock per layer digest**; global CAS in MinIO.
|
||||
* **Queues:** Redis Streams / NATS / RabbitMQ. HPA by queue depth, CPU, memory.
|
||||
* **Registry throttling:** per‑registry concurrency budgets.
|
||||
* **Targets:**
|
||||
|
||||
* Build‑time path P95 ≤ 3–5 s on warmed bases.
|
||||
* Post‑build delta scan P95 ≤ 10 s for 200 MB images.
|
||||
* Policy + VEX evaluation ≤ 500 ms for 5k components using BOM‑Index.
|
||||
* **Quotas:** license plan enforces QPS/concurrency/size; **Signer** throttles and can deny DSSE.
|
||||
|
||||
---
|
||||
|
||||
## 10) DevOps & distribution
|
||||
|
||||
* **Releases:** all first‑party images **cosign‑signed**; labels embed `org.stellaops.version` and `org.stellaops.release_date`.
|
||||
* **Channels:**
|
||||
|
||||
* **Community** (public registry): throttled, non‑attesting.
|
||||
* **Authorized** (private registry): full speed, DSSE enabled.
|
||||
* **Client update flow:** containers self‑verify signatures at boot; report version; **Signer** enforces `valid_release_year` / `max_version` from PoE before signing.
|
||||
* **Compose skeleton:**
|
||||
|
||||
```yaml
|
||||
services:
|
||||
authority: { image: stellaops/authority }
|
||||
fulcio: { image: sigstore/fulcio }
|
||||
rekor: { image: sigstore/rekor-v2 }
|
||||
minio: { image: minio/minio, command: server /data --console-address ":9001" }
|
||||
mongo: { image: mongo:7 }
|
||||
signer: { image: stellaops/signer, depends_on: [authority, fulcio] }
|
||||
attestor: { image: stellaops/attestor, depends_on: [rekor, signer] }
|
||||
scanner-web:{ image: stellaops/scanner-web, depends_on: [mongo, minio, signer, attestor] }
|
||||
scanner-worker:
|
||||
image: stellaops/scanner-worker
|
||||
deploy: { replicas: 4 }
|
||||
depends_on: [scanner-web]
|
||||
feedser: { image: stellaops/feedser-web, depends_on: [mongo] }
|
||||
vexer: { image: stellaops/vexer-web, depends_on: [mongo] }
|
||||
ui: { image: stellaops/ui, depends_on: [scanner-web, feedser, vexer] }
|
||||
```
|
||||
|
||||
* **Backups:** Mongo dumps; MinIO versioned buckets & replication; Rekor v2 DB snapshots; JWKS/Fulcio/KMS key rotation.
|
||||
|
||||
---
|
||||
|
||||
## 11) Observability & audit
|
||||
|
||||
* **Metrics:** scan latency, layer cache hit %, artifact bytes, DSSE/Rekor latency, policy evaluation time, queue depth, admission decisions (Zastava).
|
||||
* **Tracing:** per‑stage spans; correlation IDs across Scanner→Signer→Attestor.
|
||||
* **Audit logs:** every signing records `license_id`, `image_digest`, `policy_digest`, and Rekor UUID.
|
||||
* **Compliance:** MinIO **Object Lock** for immutable artifacts; reproducible outputs via policy digest + SBOM digest in predicate.
|
||||
|
||||
---
|
||||
|
||||
## 12) Roadmap (anchored to this architecture)
|
||||
|
||||
* M2: Windows MSI/SxS/GAC analyzers; deeper Rust (DWARF enrichers).
|
||||
* M2: Buildx generator certified flows; cross‑registry trust policies.
|
||||
* M3: Patch‑Presence plugin (signature‑based backport detection), opt‑in.
|
||||
* M3: Zastava Admission control GA with policy presets and dry‑run→enforce stages.
|
||||
* Continuous: Policy UX (waiver TTLs, vendor rules), Vexer connectors expansion.
|
||||
|
||||
---
|
||||
|
||||
## 13) Canonical sequences (verification & signing)
|
||||
|
||||
**Sign & log (OpTok + PoE, image verify, DSSE, Rekor).**
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant DEV as Developer
|
||||
participant UI as Web UI
|
||||
participant CORE as Stella Core
|
||||
participant REDIS as Redis
|
||||
participant RUN as Scanner Runner
|
||||
autonumber
|
||||
participant Scan as Scanner.WebService
|
||||
participant Auth as Authority (OIDC)
|
||||
participant Sign as Signer
|
||||
participant Reg as OCI Registry
|
||||
participant Ful as Fulcio/KMS
|
||||
participant Att as Attestor
|
||||
participant Rek as Rekor v2
|
||||
|
||||
DEV->>UI: Toggle Mute for CVE
|
||||
UI->>CORE: Update Mute Rule (POST /policy/mute)
|
||||
CORE->>REDIS: Store Mute Policy
|
||||
Note over CORE,REDIS: YAML/Rego Evaluator Updates
|
||||
|
||||
alt Next Pipeline Build
|
||||
CI->>CORE: Trigger Scan (POST /scan)
|
||||
CORE->>RUN: Enqueue & Scan
|
||||
RUN-->>CORE: Raw Findings
|
||||
CORE->>REDIS: Apply Mute Policies
|
||||
REDIS-->>CORE: Filtered Verdict (Passes)
|
||||
CORE-->>CI: Success Exit Code
|
||||
end
|
||||
Scan->>Auth: Get OpTok (DPoP/mTLS)
|
||||
Scan->>Sign: sign(request) + OpTok + PoE + DPoP proof
|
||||
Sign->>Auth: Validate OpTok & sender-constraint
|
||||
Sign->>Sign: Validate PoE (introspect/revocation)
|
||||
Sign->>Reg: Verify scanner image is StellaOps-signed (Referrers + cosign)
|
||||
alt OK
|
||||
Sign->>Ful: Get signing cert (keyless) or use KMS key
|
||||
Sign-->>Scan: DSSE bundle (cert chain)
|
||||
Scan->>Att: Submit bundle
|
||||
Att-->>Rek: Create entry
|
||||
Rek-->>Att: {uuid,index,proof}
|
||||
Att-->>Scan: Rekor URL
|
||||
else Deny
|
||||
Sign-->>Scan: 403 (no attestation)
|
||||
end
|
||||
```
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant CRON as SbomNightly.Schedule
|
||||
participant CORE as Stella Core
|
||||
participant REDIS as Redis Queue
|
||||
participant RUN as Scanner Runner
|
||||
participant UI as Dashboard
|
||||
**Verification (third party).**
|
||||
|
||||
CRON->>CORE: Re-queue SBOMs (Mask-Filter)
|
||||
CORE->>REDIS: Enqueue Filtered Jobs
|
||||
REDIS->>RUN: Fan Out to Runners
|
||||
RUN-->>CORE: New Scan Results
|
||||
CORE->>UI: Highlight New Criticals
|
||||
Note over CORE,UI: Focus on Changes Since Last Scan
|
||||
```plantuml
|
||||
@startuml
|
||||
actor Verifier
|
||||
participant "stellaops verify" as Tool
|
||||
database "Fulcio/KMS root" as Root
|
||||
participant "Rekor v2" as R2
|
||||
Verifier -> Tool: bundle (URL/file)
|
||||
Tool -> Tool: Verify DSSE signature
|
||||
Tool -> Root: Verify cert chain to StellaOps root
|
||||
Tool -> R2: Verify inclusion proof / query by UUID
|
||||
Tool -> Verifier: OK + claims (license_id, policy_digest, version)
|
||||
@enduml
|
||||
```
|
||||
---
|
||||
|
||||
## 8 · UI Fast Facts
|
||||
|
||||
* **Stack** – Angular 17 + Vite dev server; Tailwind CSS.
|
||||
* **State** – Signals + RxJS for live scan progress.
|
||||
* **i18n / l10n** – JSON bundles served from `/locales/{lang}.json`.
|
||||
* **Module Structure** – Lazy‑loaded feature modules (`dashboard`, `scans`, `settings`); runtime route injection by UI plug‑ins (road‑map Q2‑2026).
|
||||
|
||||
---
|
||||
|
||||
## 9 · Cross‑Cutting Concerns
|
||||
|
||||
* **Security** – containers run non‑root, `CAP_DROP:ALL`, read‑only FS, hardened seccomp profiles.
|
||||
* **Observability** – Serilog JSON, OpenTelemetry OTLP exporter, Prometheus `/metrics`.
|
||||
* **Upgrade Policy** – `/api/v1` endpoints & CLI flags stable across a minor; breaking changes bump major.
|
||||
|
||||
---
|
||||
|
||||
## 10 · Performance & Scalability
|
||||
|
||||
| Scenario | P95 target | Bottleneck | Mitigation |
|
||||
|-----------------|-----------:|-----------------|-------------------------------------------------|
|
||||
| SBOM‑first | ≤ 5 s | Redis queue | More CPU, increase `ScannerPool.Workers` |
|
||||
| Image‑unpack | ≤ 10 s | Layer unpack | Prefer SBOM path, warm Docker cache |
|
||||
| High concurrency| 40 rps | Runner CPU | Scale Core replicas + side‑car scanner services |
|
||||
|
||||
---
|
||||
|
||||
## 11 · Future Architectural Anchors
|
||||
|
||||
* **ScanService micro‑split (gRPC)** – isolate heavy runners for large clusters.
|
||||
* **UI route plug‑ins** – dynamic Angular module loader (road‑map Q2‑2026).
|
||||
* **Redis Cluster** – transparently sharded cache once sustained > 100 rps.
|
||||
|
||||
---
|
||||
|
||||
## 12 · Assumptions & Trade‑offs
|
||||
|
||||
Requires Docker/CRI‑O runtime; .NET 9 available on hosts; Windows containers are out‑of‑scope this cycle.
|
||||
Embedded auth simplifies deployment but may need plug‑ins for enterprise IdPs.
|
||||
Speed is prioritised over exhaustive feature parity with heavyweight commercial scanners.
|
||||
|
||||
---
|
||||
|
||||
## 13 · References & Further Reading
|
||||
|
||||
* **C4 Model** – <https://c4model.com>
|
||||
* **.NET Architecture Guides** – <https://learn.microsoft.com/dotnet/architecture>
|
||||
* **OSS Examples** – Kubernetes Architecture docs, Prometheus design papers, Backstage.
|
||||
|
||||
*(End of High‑Level Architecture v2.2)*
|
||||
**End of `high_level_architecture.md` (Consolidated).**
|
||||
|
||||
@@ -1,208 +0,0 @@
|
||||
# 8 · Detailed Module Specifications — **Stella Ops Feedser**
|
||||
_This document describes the Feedser service, its supporting libraries, connectors, exporters, and test assets that live in the OSS repository._
|
||||
|
||||
---
|
||||
|
||||
## 0 Scope
|
||||
|
||||
Feedser is the vulnerability ingest/merge/export subsystem of Stella Ops. It
|
||||
fetches primary advisories, normalizes and deduplicates them into MongoDB, and
|
||||
produces deterministic JSON and Trivy DB exports. This document lists the
|
||||
projects that make up that workflow, the extension points they expose, and the
|
||||
artefacts they ship.
|
||||
|
||||
---
|
||||
|
||||
## 1 Repository layout (current)
|
||||
|
||||
```text
|
||||
src/
|
||||
├─ Directory.Build.props / Directory.Build.targets
|
||||
├─ StellaOps.Plugin/
|
||||
├─ StellaOps.Feedser.Core/
|
||||
├─ StellaOps.Feedser.Core.Tests/
|
||||
├─ StellaOps.Feedser.Models/ (+ .Tests/)
|
||||
├─ StellaOps.Feedser.Normalization/ (+ .Tests/)
|
||||
├─ StellaOps.Feedser.Merge/ (+ .Tests/)
|
||||
├─ StellaOps.Feedser.Storage.Mongo/ (+ .Tests/)
|
||||
├─ StellaOps.Feedser.Exporter.Json/ (+ .Tests/)
|
||||
├─ StellaOps.Feedser.Exporter.TrivyDb/ (+ .Tests/)
|
||||
├─ StellaOps.Feedser.Source.* / StellaOps.Feedser.Source.*.Tests/
|
||||
├─ StellaOps.Feedser.Testing/
|
||||
├─ StellaOps.Feedser.Tests.Shared/
|
||||
├─ StellaOps.Feedser.WebService/ (+ .Tests/)
|
||||
├─ PluginBinaries/
|
||||
└─ StellaOps.Feedser.sln
|
||||
```
|
||||
|
||||
Each folder is a .NET project (or set of projects) referenced by
|
||||
`StellaOps.Feedser.sln`. Build assets are shared through the root
|
||||
`Directory.Build.props/targets` so conventions stay consistent.
|
||||
|
||||
---
|
||||
|
||||
## 2 Shared libraries
|
||||
|
||||
| Project | Purpose | Key extension points |
|
||||
|---------|---------|----------------------|
|
||||
| `StellaOps.Plugin` | Base contracts for connectors, exporters, and DI routines plus Cosign validation helpers. | `IFeedConnector`, `IExporterPlugin`, `IDependencyInjectionRoutine` |
|
||||
| `StellaOps.DependencyInjection` | Composable service registrations for Feedser and plug-ins. | `IDependencyInjectionRoutine` discovery |
|
||||
| `StellaOps.Feedser.Testing` | Common fixtures, builders, and harnesses for integration/unit tests. | `FeedserMongoFixture`, test builders |
|
||||
| `StellaOps.Feedser.Tests.Shared` | Shared assembly metadata and fixtures wired in via `Directory.Build.props`. | Test assembly references |
|
||||
|
||||
---
|
||||
|
||||
## 3 Core projects
|
||||
|
||||
| Project | Responsibility | Extensibility |
|
||||
|---------|----------------|---------------|
|
||||
| `StellaOps.Feedser.WebService` | ASP.NET Core minimal API hosting Feedser jobs, status endpoints, and scheduler. | DI-based plug-in discovery; configuration binding |
|
||||
| `StellaOps.Feedser.Core` | Job orchestration, connector pipelines, merge workflows, export coordination. | `IFeedConnector`, `IExportJob`, deterministic merge policies |
|
||||
| `StellaOps.Feedser.Models` | Canonical advisory DTOs and enums persisted in MongoDB and exported artefacts. | Partial classes for source-specific metadata |
|
||||
| `StellaOps.Feedser.Normalization` | Version comparison, CVSS normalization, text utilities for canonicalization. | Helpers consumed by connectors/merge |
|
||||
| `StellaOps.Feedser.Merge` | Precedence evaluation, alias graph maintenance, merge-event hashing. | Policy extensions via DI |
|
||||
| `StellaOps.Feedser.Storage.Mongo` | Repository layer for documents, DTOs, advisories, merge events, export state. | Connection string/config via options |
|
||||
| `StellaOps.Feedser.Exporter.Json` | Deterministic vuln-list JSON export pipeline. | Dependency injection for storage + plugin to host |
|
||||
| `StellaOps.Feedser.Exporter.TrivyDb` | Builds Trivy DB artefacts from canonical advisories. | Optional ORAS push routines |
|
||||
|
||||
### 3.1 StellaOps.Feedser.WebService
|
||||
|
||||
* Hosts minimal API endpoints (`/health`, `/status`, `/jobs`).
|
||||
* Runs the scheduler that triggers connectors and exporters according to
|
||||
configured windows.
|
||||
* Applies dependency-injection routines from `PluginBinaries/` at startup only
|
||||
(restart-time plug-ins).
|
||||
|
||||
### 3.2 StellaOps.Feedser.Core
|
||||
|
||||
* Defines job primitives (fetch, parse, map, merge, export) used by connectors.
|
||||
* Coordinates deterministic merge flows and writes `merge_event` documents.
|
||||
* Provides telemetry/log scopes consumed by WebService and exporters.
|
||||
|
||||
### 3.3 StellaOps.Feedser.Storage.Mongo
|
||||
|
||||
* Persists raw documents, DTO records, canonical advisories, aliases, affected
|
||||
packages, references, merge events, export state, and job leases.
|
||||
* Exposes repository helpers for exporters to stream full/delta snapshots.
|
||||
|
||||
### 3.4 StellaOps.Feedser.Exporter.*
|
||||
|
||||
* `Exporter.Json` mirrors the Aqua vuln-list tree with canonical ordering.
|
||||
* `Exporter.TrivyDb` builds Trivy DB Bolt archives and optional OCI bundles.
|
||||
* Both exporters honour deterministic hashing and respect export cursors.
|
||||
|
||||
---
|
||||
|
||||
## 4 Source connectors
|
||||
|
||||
Connectors live under `StellaOps.Feedser.Source.*` and conform to the interfaces
|
||||
in `StellaOps.Plugin`.
|
||||
|
||||
| Family | Project(s) | Notes |
|
||||
|--------|------------|-------|
|
||||
| Distro PSIRTs | `StellaOps.Feedser.Source.Distro.*` | Debian, Red Hat, SUSE, Ubuntu connectors with NEVRA/EVR helpers. |
|
||||
| Vendor PSIRTs | `StellaOps.Feedser.Source.Vndr.*` | Adobe, Apple, Cisco, Chromium, Microsoft, Oracle, VMware. |
|
||||
| Regional CERTs | `StellaOps.Feedser.Source.Cert*`, `Source.Ru.*`, `Source.Ics.*`, `Source.Kisa` | Provide enrichment metadata while preserving vendor precedence. |
|
||||
| OSS ecosystems | `StellaOps.Feedser.Source.Ghsa`, `Source.Osv`, `Source.Cve`, `Source.Kev`, `Source.Acsc`, `Source.Cccs`, `Source.Jvn` | Emit SemVer/alias-rich advisories. |
|
||||
|
||||
Each connector ships fixtures/tests under the matching `*.Tests` project.
|
||||
|
||||
---
|
||||
|
||||
## 5 · Module Details
|
||||
|
||||
> _Focus on the Feedser-specific services that replace the legacy FeedMerge cron._
|
||||
|
||||
### 5.1 Feedser.Core
|
||||
|
||||
* Owns the fetch → parse → merge → export job pipeline and enforces deterministic
|
||||
merge hashes (`merge_event`).
|
||||
* Provides `JobSchedulerBuilder`, job coordinator, and telemetry scopes consumed
|
||||
by the WebService and exporters.
|
||||
|
||||
### 5.2 Feedser.Storage.Mongo
|
||||
|
||||
* Bootstrapper creates collections/indexes (documents, dto, advisory, alias,
|
||||
affected, merge_event, export_state, jobs, locks).
|
||||
* Repository APIs surface full/delta advisory reads for exporters, plus
|
||||
SourceState and job lease persistence.
|
||||
|
||||
### 5.3 Feedser.Exporter.Json / Feedser.Exporter.TrivyDb
|
||||
|
||||
* JSON exporter mirrors vuln-list layout with per-file digests and manifest.
|
||||
* Trivy DB exporter shells or native-builds Bolt archives, optionally pushes OCI
|
||||
layers, and records export cursors. Delta runs reuse unchanged blobs from the
|
||||
previous full baseline, annotating `metadata.json` with `mode`, `baseExportId`,
|
||||
`baseManifestDigest`, `resetBaseline`, and `delta.changedFiles[]`/`delta.removedPaths[]`.
|
||||
ORAS pushes honour `publishFull` / `publishDelta`, and offline bundles respect
|
||||
`includeFull` / `includeDelta` for air-gapped syncs.
|
||||
|
||||
### 5.4 Feedser.WebService
|
||||
|
||||
* Minimal API host exposing `/health`, `/ready`, `/jobs` and wiring telemetry.
|
||||
* Loads restart-time plug-ins from `PluginBinaries/`, executes Mongo bootstrap,
|
||||
and registers built-in connectors/exporters with the scheduler.
|
||||
|
||||
### 5.5 Plugin host & DI bridge
|
||||
|
||||
* `StellaOps.Plugin` + `StellaOps.DependencyInjection` provide the contracts and
|
||||
helper routines for connectors/exporters to integrate with the WebService.
|
||||
|
||||
---
|
||||
|
||||
## 6 · Plug-ins & Agents
|
||||
|
||||
* **Plug-in discovery** – restart-only; the WebService enumerates
|
||||
`PluginBinaries/` (or configured directories) and executes the contained
|
||||
`IDependencyInjectionRoutine` implementations.
|
||||
* **Connector/exporter packages** – each source/exporter can ship as a plug-in
|
||||
assembly with its own options and HttpClient configuration, keeping the core
|
||||
image minimal.
|
||||
* **StellaOps CLI (agent)** – new `StellaOps.Cli` module that exposes
|
||||
`scanner`, `scan`, and `db` verbs (via System.CommandLine 2.0) to download
|
||||
scanner container bundles, install them locally, execute scans against target
|
||||
directories, automatically upload results, and trigger Feedser jobs (`db
|
||||
fetch/merge/export`) aligned with the SBOM-first workflow described in
|
||||
`AGENTS.md`.
|
||||
* **Offline Kit** – bundles Feedser plug-ins, JSON tree, Trivy DB, and export
|
||||
manifests so air-gapped sites can load the latest vulnerability data without
|
||||
outbound connectivity.
|
||||
|
||||
---
|
||||
|
||||
## 7 · Docker & Distribution Artefacts
|
||||
|
||||
| Artefact | Path / Identifier | Notes |
|
||||
|----------|-------------------|-------|
|
||||
| Feedser WebService image | `containers/feedser/Dockerfile` (built via CI) | Self-contained ASP.NET runtime hosting scheduler/endpoints. |
|
||||
| Plugin bundle | `PluginBinaries/` | Mounted or baked-in assemblies for connectors/exporters. |
|
||||
| Offline Kit tarball | Produced by CI release pipeline | Contains JSON tree, Trivy DB OCI layout, export manifest, and plug-ins. |
|
||||
| Local dev compose | `scripts/` + future compose overlays | Developers can run MongoDB, Redis (optional), and WebService locally. |
|
||||
|
||||
---
|
||||
|
||||
## 8 · Performance Budget
|
||||
|
||||
| Scenario | Budget | Source |
|
||||
|----------|--------|--------|
|
||||
| Advisory upsert (large advisory) | ≤ 500 ms/advisory | `AdvisoryStorePerformanceTests` (Mongo) |
|
||||
| Advisory fetch (`GetRecent`) | ≤ 200 ms/advisory | Same performance test harness |
|
||||
| Advisory point lookup (`Find`) | ≤ 200 ms/advisory | Same performance test harness |
|
||||
| Bulk upsert/fetch cycle | ≤ 28 s total for 30 large advisories | Same performance test harness |
|
||||
| Feedser job scheduling | Deterministic cron execution via `JobSchedulerHostedService` | `StellaOps.Feedser.Core` tests |
|
||||
| Trivy DB export | Deterministic digests across runs (ongoing TODO for end-to-end test) | `Exporter.TrivyDb` backlog |
|
||||
|
||||
Budgets are enforced in automated tests where available; outstanding TODO/DOING
|
||||
items (see task boards) continue tracking gaps such as exporter determinism.
|
||||
|
||||
---
|
||||
|
||||
## 9 Testing
|
||||
|
||||
* Unit and integration tests live alongside each component (`*.Tests`).
|
||||
* Shared fixtures come from `StellaOps.Feedser.Testing` and
|
||||
`StellaOps.Feedser.Tests.Shared` (linked via `Directory.Build.props`).
|
||||
* Integration suites use ephemeral MongoDB and Redis via Testcontainers to
|
||||
validate end-to-end flow without external dependencies.
|
||||
|
||||
---
|
||||
@@ -107,6 +107,7 @@ See the detailed rules in
|
||||
|
||||
## 6 · Related documentation
|
||||
|
||||
* **Install guide:** `/install/#air-gapped`
|
||||
* **Sovereign mode rationale:** `/sovereign/`
|
||||
* **Security policy:** `/security/#reporting-a-vulnerability`
|
||||
* **Install guide:** `/install/#air-gapped`
|
||||
* **Sovereign mode rationale:** `/sovereign/`
|
||||
* **Security policy:** `/security/#reporting-a-vulnerability`
|
||||
* **CERT-Bund snapshots:** `python tools/certbund_offline_snapshot.py --help` (see `docs/ops/feedser-certbund-operations.md`)
|
||||
|
||||
@@ -11,6 +11,7 @@ Produce and maintain offline-friendly documentation for StellaOps modules, cover
|
||||
## Operating Principles
|
||||
- Keep guides deterministic and in sync with shipped configuration samples.
|
||||
- Prefer tables/checklists for operator steps; flag security-sensitive actions.
|
||||
- When work involves a specific `StellaOps.<Component>` project, consult both `docs/07_HIGH_LEVEL_ARCHITECTURE.md` and the matching dossier `docs/ARCHITECTURE_<COMPONENT>.md` before drafting or editing content.
|
||||
- Update `docs/TASKS.md` whenever work items change status (TODO/DOING/REVIEW/DONE/BLOCKED).
|
||||
|
||||
## Coordination
|
||||
|
||||
384
docs/ARCHITECTURE_ATTESTOR.md
Normal file
384
docs/ARCHITECTURE_ATTESTOR.md
Normal file
@@ -0,0 +1,384 @@
|
||||
# component_architecture_attestor.md — **Stella Ops Attestor** (2025Q4)
|
||||
|
||||
> **Scope.** Implementation‑ready architecture for the **Attestor**: the service that **submits** DSSE envelopes to **Rekor v2**, retrieves/validates inclusion proofs, caches results, and exposes verification APIs. It accepts DSSE **only** from the **Signer** over mTLS, enforces chain‑of‑trust to Stella Ops roots, and returns `{uuid, index, proof, logURL}` to calling services (Scanner.WebService for SBOMs; backend for final reports; Vexer exports when configured).
|
||||
|
||||
---
|
||||
|
||||
## 0) Mission & boundaries
|
||||
|
||||
**Mission.** Turn a signed DSSE envelope from the Signer into a **transparency‑logged, verifiable fact** with a durable, replayable proof (Merkle inclusion + (optional) checkpoint anchoring). Provide **fast verification** for downstream consumers and a stable retrieval interface for UI/CLI.
|
||||
|
||||
**Boundaries.**
|
||||
|
||||
* Attestor **does not sign**; it **must not** accept unsigned or third‑party‑signed bundles.
|
||||
* Attestor **does not decide PASS/FAIL**; it logs attestations for SBOMs, reports, and export artifacts.
|
||||
* Rekor v2 backends may be **local** (self‑hosted) or **remote**; Attestor handles both with retries, backoff, and idempotency.
|
||||
|
||||
---
|
||||
|
||||
## 1) Topology & dependencies
|
||||
|
||||
**Process shape:** single stateless service `stellaops/attestor` behind mTLS.
|
||||
|
||||
**Dependencies:**
|
||||
|
||||
* **Signer** (caller) — authenticated via **mTLS** and **Authority** OpToks.
|
||||
* **Rekor v2** — tile‑backed transparency log endpoint(s).
|
||||
* **MinIO (S3)** — optional archive store for DSSE envelopes & verification bundles.
|
||||
* **MongoDB** — local cache of `{uuid, index, proof, artifactSha256, bundleSha256}`; job state; audit.
|
||||
* **Redis** — dedupe/idempotency keys and short‑lived rate‑limit buckets.
|
||||
* **Licensing Service (optional)** — “endorse” call for cross‑log publishing when customer opts‑in.
|
||||
|
||||
Trust boundary: **Only the Signer** is allowed to call submission endpoints; enforced by **mTLS peer cert allowlist** + `aud=attestor` OpTok.
|
||||
|
||||
---
|
||||
|
||||
## 2) Data model (Mongo)
|
||||
|
||||
Database: `attestor`
|
||||
|
||||
**Collections & schemas**
|
||||
|
||||
* `entries`
|
||||
|
||||
```
|
||||
{ _id: "<rekor-uuid>",
|
||||
artifact: { sha256: "<sha256>", kind: "sbom|report|vex-export", imageDigest?, subjectUri? },
|
||||
bundleSha256: "<sha256>", // canonicalized DSSE
|
||||
index: <int>, // log index/sequence if provided by backend
|
||||
proof: { // inclusion proof
|
||||
checkpoint: { origin, size, rootHash, timestamp },
|
||||
inclusion: { leafHash, path[] } // Merkle path (tiles)
|
||||
},
|
||||
log: { url, logId? },
|
||||
createdAt, status: "included|pending|failed",
|
||||
signerIdentity: { mode: "keyless|kms", issuer, san?, kid? }
|
||||
}
|
||||
```
|
||||
|
||||
* `dedupe`
|
||||
|
||||
```
|
||||
{ key: "bundle:<sha256>", rekorUuid, createdAt, ttlAt } // idempotency key
|
||||
```
|
||||
|
||||
* `audit`
|
||||
|
||||
```
|
||||
{ _id, ts, caller: { cn, mTLSThumbprint, sub, aud }, // from mTLS + OpTok
|
||||
action: "submit|verify|fetch",
|
||||
artifactSha256, bundleSha256, rekorUuid?, index?, result, latencyMs, backend }
|
||||
```
|
||||
|
||||
Indexes:
|
||||
|
||||
* `entries` on `artifact.sha256`, `bundleSha256`, `createdAt`, and `{status:1, createdAt:-1}`.
|
||||
* `dedupe.key` unique (TTL 24–48h).
|
||||
* `audit.ts` for time‑range queries.
|
||||
|
||||
---
|
||||
|
||||
## 3) Input contract (from Signer)
|
||||
|
||||
**Attestor accepts only** DSSE envelopes that satisfy all of:
|
||||
|
||||
1. **mTLS** peer certificate maps to `signer` service (CA‑pinned).
|
||||
2. **Authority** OpTok with `aud=attestor`, `scope=attestor.write`, DPoP or mTLS bound.
|
||||
3. DSSE envelope is **signed by the Signer’s key** (or includes a **Fulcio‑issued** cert chain) and **chains to configured roots** (Fulcio/KMS).
|
||||
4. **Predicate type** is one of Stella Ops types (sbom/report/vex‑export) with valid schema.
|
||||
5. `subject[*].digest.sha256` is present and canonicalized.
|
||||
|
||||
**Wire shape (JSON):**
|
||||
|
||||
```json
|
||||
{
|
||||
"bundle": { "dsse": { "payloadType": "application/vnd.in-toto+json", "payload": "<b64>", "signatures": [ ... ] },
|
||||
"certificateChain": [ "-----BEGIN CERTIFICATE-----..." ],
|
||||
"mode": "keyless" },
|
||||
"meta": {
|
||||
"artifact": { "sha256": "<subject sha256>", "kind": "sbom|report|vex-export", "imageDigest": "sha256:..." },
|
||||
"bundleSha256": "<sha256 of canonical dsse>",
|
||||
"logPreference": "primary", // "primary" | "mirror" | "both"
|
||||
"archive": true // whether Attestor should archive bundle to S3
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4) APIs
|
||||
|
||||
### 4.1 Submission
|
||||
|
||||
`POST /api/v1/rekor/entries` *(mTLS + OpTok required)*
|
||||
|
||||
* **Body**: as above.
|
||||
* **Behavior**:
|
||||
|
||||
* Verify caller (mTLS + OpTok).
|
||||
* Validate DSSE bundle (signature, cert chain to Fulcio/KMS; DSSE structure; payloadType allowed).
|
||||
* Idempotency: compute `bundleSha256`; check `dedupe`. If present, return existing `rekorUuid`.
|
||||
* Submit canonicalized bundle to Rekor v2 (primary or mirror according to `logPreference`).
|
||||
* Retrieve **inclusion proof** (blocking until inclusion or up to `proofTimeoutMs`); if backend returns promise only, return `status=pending` and retry asynchronously.
|
||||
* Persist `entries` record; archive DSSE to S3 if `archive=true`.
|
||||
* **Response 200**:
|
||||
|
||||
```json
|
||||
{
|
||||
"uuid": "…",
|
||||
"index": 123456,
|
||||
"proof": {
|
||||
"checkpoint": { "origin": "rekor@site", "size": 987654, "rootHash": "…", "timestamp": "…" },
|
||||
"inclusion": { "leafHash": "…", "path": ["…","…"] }
|
||||
},
|
||||
"logURL": "https://rekor…/api/v2/log/…/entries/…",
|
||||
"status": "included"
|
||||
}
|
||||
```
|
||||
* **Errors**: `401 invalid_token`, `403 not_signer|chain_untrusted`, `409 duplicate_bundle` (with existing `uuid`), `502 rekor_unavailable`, `504 proof_timeout`.
|
||||
|
||||
### 4.2 Proof retrieval
|
||||
|
||||
`GET /api/v1/rekor/entries/{uuid}`
|
||||
|
||||
* Returns `entries` row (refreshes proof from Rekor if stale/missing).
|
||||
* Accepts `?refresh=true` to force backend query.
|
||||
|
||||
### 4.3 Verification (third‑party or internal)
|
||||
|
||||
`POST /api/v1/rekor/verify`
|
||||
|
||||
* **Body** (one of):
|
||||
|
||||
* `{ "uuid": "…" }`
|
||||
* `{ "bundle": { …DSSE… } }`
|
||||
* `{ "artifactSha256": "…" }` *(looks up most recent entry)*
|
||||
|
||||
* **Checks**:
|
||||
|
||||
1. **Bundle signature** → cert chain to Fulcio/KMS roots configured.
|
||||
2. **Inclusion proof** → recompute leaf hash; verify Merkle path against checkpoint root.
|
||||
3. Optionally verify **checkpoint** against local trust anchors (if Rekor signs checkpoints).
|
||||
4. Confirm **subject.digest** matches caller‑provided hash (when given).
|
||||
|
||||
* **Response**:
|
||||
|
||||
```json
|
||||
{ "ok": true, "uuid": "…", "index": 123, "logURL": "…", "checkedAt": "…" }
|
||||
```
|
||||
|
||||
### 4.4 Batch submission (optional)
|
||||
|
||||
`POST /api/v1/rekor/batch` accepts an array of submission objects; processes with per‑item results.
|
||||
|
||||
---
|
||||
|
||||
## 5) Rekor v2 driver (backend)
|
||||
|
||||
* **Canonicalization**: DSSE envelopes are **normalized** (stable JSON ordering, no insignificant whitespace) before hashing and submission.
|
||||
* **Transport**: HTTP/2 with retries (exponential backoff, jitter), budgeted timeouts.
|
||||
* **Idempotency**: if backend returns “already exists,” map to existing `uuid`.
|
||||
* **Proof acquisition**:
|
||||
|
||||
* In synchronous mode, poll the log for inclusion up to `proofTimeoutMs`.
|
||||
* In asynchronous mode, return `pending` and schedule a **proof fetcher** job (Mongo job doc + backoff).
|
||||
* **Mirrors/dual logs**:
|
||||
|
||||
* When `logPreference="both"`, submit to primary and mirror; store **both** UUIDs (primary canonical).
|
||||
* Optional **cloud endorsement**: POST to the Stella Ops cloud `/attest/endorse` with `{uuid, artifactSha256}`; store returned endorsement id.
|
||||
|
||||
---
|
||||
|
||||
## 6) Security model
|
||||
|
||||
* **mTLS required** for submission from **Signer** (CA‑pinned).
|
||||
* **Authority token** with `aud=attestor` and DPoP/mTLS binding must be presented; Attestor verifies both.
|
||||
* **Bundle acceptance policy**:
|
||||
|
||||
* DSSE signature must chain to the configured **Fulcio** (keyless) or **KMS/HSM** roots.
|
||||
* SAN (Subject Alternative Name) must match **Signer identity** policy (e.g., `urn:stellaops:signer` or pinned OIDC issuer).
|
||||
* Predicate `predicateType` must be on allowlist (sbom/report/vex-export).
|
||||
* `subject.digest.sha256` values must be present and well‑formed (hex).
|
||||
* **No public submission** path. **Never** accept bundles from untrusted clients.
|
||||
* **Rate limits**: per mTLS thumbprint/license (from Signer‑forwarded claims) to avoid flooding the log.
|
||||
* **Redaction**: Attestor never logs secret material; DSSE payloads **should** be public by design (SBOMs/reports). If customers require redaction, enforce policy at Signer (predicate minimization) **before** Attestor.
|
||||
|
||||
---
|
||||
|
||||
## 7) Storage & archival
|
||||
|
||||
* **Entries** in Mongo provide a local ledger keyed by `rekorUuid` and **artifact sha256** for quick reverse lookups.
|
||||
* **S3 archival** (if enabled):
|
||||
|
||||
```
|
||||
s3://stellaops/attest/
|
||||
dsse/<bundleSha256>.json
|
||||
proof/<rekorUuid>.json
|
||||
bundle/<artifactSha256>.zip # optional verification bundle
|
||||
```
|
||||
* **Verification bundles** (zip):
|
||||
|
||||
* DSSE (`*.dsse.json`), proof (`*.proof.json`), `chain.pem` (certs), `README.txt` with verification steps & hashes.
|
||||
|
||||
---
|
||||
|
||||
## 8) Observability & audit
|
||||
|
||||
**Metrics** (Prometheus):
|
||||
|
||||
* `attestor.submit_total{result,backend}`
|
||||
* `attestor.submit_latency_seconds{backend}`
|
||||
* `attestor.proof_fetch_total{result}`
|
||||
* `attestor.verify_total{result}`
|
||||
* `attestor.dedupe_hits_total`
|
||||
* `attestor.errors_total{type}`
|
||||
|
||||
**Tracing**:
|
||||
|
||||
* Spans: `validate`, `rekor.submit`, `rekor.poll`, `persist`, `archive`, `verify`.
|
||||
|
||||
**Audit**:
|
||||
|
||||
* Immutable `audit` rows (ts, caller, action, hashes, uuid, index, backend, result, latency).
|
||||
|
||||
---
|
||||
|
||||
## 9) Configuration (YAML)
|
||||
|
||||
```yaml
|
||||
attestor:
|
||||
listen: "https://0.0.0.0:8444"
|
||||
security:
|
||||
mtls:
|
||||
caBundle: /etc/ssl/signer-ca.pem
|
||||
requireClientCert: true
|
||||
authority:
|
||||
issuer: "https://authority.internal"
|
||||
jwksUrl: "https://authority.internal/jwks"
|
||||
requireSenderConstraint: "dpop" # or "mtls"
|
||||
signerIdentity:
|
||||
mode: ["keyless","kms"]
|
||||
fulcioRoots: ["/etc/fulcio/root.pem"]
|
||||
allowedSANs: ["urn:stellaops:signer"]
|
||||
kmsKeys: ["kms://cluster-kms/stellaops-signer"]
|
||||
rekor:
|
||||
primary:
|
||||
url: "https://rekor-v2.internal"
|
||||
proofTimeoutMs: 15000
|
||||
pollIntervalMs: 250
|
||||
maxAttempts: 60
|
||||
mirror:
|
||||
enabled: false
|
||||
url: "https://rekor-v2.mirror"
|
||||
mongo:
|
||||
uri: "mongodb://mongo/attestor"
|
||||
s3:
|
||||
enabled: true
|
||||
endpoint: "http://minio:9000"
|
||||
bucket: "stellaops"
|
||||
prefix: "attest/"
|
||||
objectLock: "governance"
|
||||
redis:
|
||||
url: "redis://redis:6379/2"
|
||||
quotas:
|
||||
perCaller:
|
||||
qps: 50
|
||||
burst: 100
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 10) End‑to‑end sequences
|
||||
|
||||
**A) Submit & include (happy path)**
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
participant SW as Scanner.WebService
|
||||
participant SG as Signer
|
||||
participant AT as Attestor
|
||||
participant RK as Rekor v2
|
||||
|
||||
SW->>SG: POST /sign/dsse (OpTok+PoE)
|
||||
SG-->>SW: DSSE bundle (+certs)
|
||||
SW->>AT: POST /rekor/entries (mTLS + OpTok)
|
||||
AT->>AT: Validate DSSE (chain to Fulcio/KMS; signer identity)
|
||||
AT->>RK: submit(bundle)
|
||||
RK-->>AT: {uuid, index?}
|
||||
AT->>RK: poll inclusion until proof or timeout
|
||||
RK-->>AT: inclusion proof (checkpoint + path)
|
||||
AT-->>SW: {uuid, index, proof, logURL}
|
||||
```
|
||||
|
||||
**B) Verify by artifact digest (CLI)**
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
participant CLI as stellaops verify
|
||||
participant SW as Scanner.WebService
|
||||
participant AT as Attestor
|
||||
|
||||
CLI->>SW: GET /catalog/artifacts/{id}
|
||||
SW-->>CLI: {artifactSha256, rekor: {uuid}}
|
||||
CLI->>AT: POST /rekor/verify { uuid }
|
||||
AT-->>CLI: { ok: true, index, logURL }
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 11) Failure modes & responses
|
||||
|
||||
| Condition | Return | Details | | |
|
||||
| ------------------------------------- | ----------------------- | --------------------------------------------------------- | -------- | ------------ |
|
||||
| mTLS/OpTok invalid | `401 invalid_token` | Include `WWW-Authenticate` DPoP challenge when applicable | | |
|
||||
| Bundle not signed by trusted identity | `403 chain_untrusted` | DSSE accepted only from Signer identities | | |
|
||||
| Duplicate bundle | `409 duplicate_bundle` | Return existing `uuid` (idempotent) | | |
|
||||
| Rekor unreachable/timeout | `502 rekor_unavailable` | Retry with backoff; surface `Retry-After` | | |
|
||||
| Inclusion proof timeout | `202 accepted` | `status=pending`, background job continues to fetch proof | | |
|
||||
| Archive failure | `207 multi-status` | Entry recorded; archive will retry asynchronously | | |
|
||||
| Verification mismatch | `400 verify_failed` | Include reason: chain | leafHash | rootMismatch |
|
||||
|
||||
---
|
||||
|
||||
## 12) Performance & scale
|
||||
|
||||
* Stateless; scale horizontally.
|
||||
* **Targets**:
|
||||
|
||||
* Submit+proof P95 ≤ **300 ms** (warm log; local Rekor).
|
||||
* Verify P95 ≤ **30 ms** from cache; ≤ **120 ms** with live proof fetch.
|
||||
* 1k submissions/minute per replica sustained.
|
||||
* **Hot caches**: `dedupe` (bundle hash → uuid), recent `entries` by artifact sha256.
|
||||
|
||||
---
|
||||
|
||||
## 13) Testing matrix
|
||||
|
||||
* **Happy path**: valid DSSE, inclusion within timeout.
|
||||
* **Idempotency**: resubmit same `bundleSha256` → same `uuid`.
|
||||
* **Security**: reject non‑Signer mTLS, wrong `aud`, DPoP replay, untrusted cert chain, forbidden predicateType.
|
||||
* **Rekor variants**: promise‑then‑proof, proof delayed, mirror dual‑submit, mirror failure.
|
||||
* **Verification**: corrupt leaf path, wrong root, tampered bundle.
|
||||
* **Throughput**: soak test with 10k submissions; latency SLOs, zero drops.
|
||||
|
||||
---
|
||||
|
||||
## 14) Implementation notes
|
||||
|
||||
* Language: **.NET 10** minimal API; `HttpClient` with **sockets handler** tuned for HTTP/2.
|
||||
* JSON: **canonical writer** for DSSE payload hashing.
|
||||
* Crypto: use **BouncyCastle**/**System.Security.Cryptography**; PEM parsing for cert chains.
|
||||
* Rekor client: pluggable driver; treat backend errors as retryable/non‑retryable with granular mapping.
|
||||
* Safety: size caps on bundles; decompress bombs guarded; strict UTF‑8.
|
||||
* CLI integration: `stellaops verify attestation <uuid|bundle|artifact>` calls `/rekor/verify`.
|
||||
|
||||
---
|
||||
|
||||
## 15) Optional features
|
||||
|
||||
* **Dual‑log** write (primary + mirror) and **cross‑log proof** packaging.
|
||||
* **Cloud endorsement**: send `{uuid, artifactSha256}` to Stella Ops cloud; store returned endorsement id for marketing/chain‑of‑custody.
|
||||
* **Checkpoint pinning**: periodically pin latest Rekor checkpoints to an external audit store for independent monitoring.
|
||||
|
||||
394
docs/ARCHITECTURE_AUTHORITY.md
Normal file
394
docs/ARCHITECTURE_AUTHORITY.md
Normal file
@@ -0,0 +1,394 @@
|
||||
# component_architecture_authority.md — **Stella Ops Authority** (2025Q4)
|
||||
|
||||
> **Scope.** Implementation‑ready architecture for **Stella Ops Authority**: the on‑prem **OIDC/OAuth2** service that issues **short‑lived, sender‑constrained operational tokens (OpToks)** to first‑party services and tools. Covers protocols (DPoP & mTLS binding), token shapes, endpoints, storage, rotation, HA, RBAC, audit, and testing. This component is the trust anchor for *who* is calling inside a Stella Ops installation. (Entitlement is proven separately by **PoE** from the cloud Licensing Service; Authority does not issue PoE.)
|
||||
|
||||
---
|
||||
|
||||
## 0) Mission & boundaries
|
||||
|
||||
**Mission.** Provide **fast, local, verifiable** authentication for Stella Ops microservices and tools by minting **very short‑lived** OAuth2/OIDC tokens that are **sender‑constrained** (DPoP or mTLS‑bound). Support RBAC scopes, multi‑tenant claims, and deterministic validation for APIs (Scanner, Signer, Attestor, Vexer, Feedser, UI, CLI, Zastava).
|
||||
|
||||
**Boundaries.**
|
||||
|
||||
* Authority **does not** validate entitlements/licensing. That’s enforced by **Signer** using **PoE** with the cloud Licensing Service.
|
||||
* Authority tokens are **operational only** (2–5 min TTL) and must not be embedded in long‑lived artifacts or stored in SBOMs.
|
||||
* Authority is **stateless for validation** (JWT) and **optional introspection** for services that prefer online checks.
|
||||
|
||||
---
|
||||
|
||||
## 1) Protocols & cryptography
|
||||
|
||||
* **OIDC Discovery**: `/.well-known/openid-configuration`
|
||||
* **OAuth2** grant types:
|
||||
|
||||
* **Client Credentials** (service↔service, with mTLS or private_key_jwt)
|
||||
* **Device Code** (CLI login on headless agents; optional)
|
||||
* **Authorization Code + PKCE** (browser login for UI; optional)
|
||||
* **Sender constraint options** (choose per caller or per audience):
|
||||
|
||||
* **DPoP** (Demonstration of Proof‑of‑Possession): proof JWT on each HTTP request, bound to the access token via `cnf.jkt`.
|
||||
* **OAuth 2.0 mTLS** (certificate‑bound tokens): token bound to client certificate thumbprint via `cnf.x5t#S256`.
|
||||
* **Signing algorithms**: **EdDSA (Ed25519)** preferred; fallback **ES256 (P‑256)**. Rotation is supported via **kid** in JWKS.
|
||||
* **Token format**: **JWT** access tokens (compact), optionally opaque reference tokens for services that insist on introspection.
|
||||
* **Clock skew tolerance**: ±60 s; issue `nbf`, `iat`, `exp` accordingly.
|
||||
|
||||
---
|
||||
|
||||
## 2) Token model
|
||||
|
||||
### 2.1 Access token (OpTok) — short‑lived (120–300 s)
|
||||
|
||||
**Registered claims**
|
||||
|
||||
```
|
||||
iss = https://authority.<domain>
|
||||
sub = <client_id or user_id>
|
||||
aud = <service audience: signer|scanner|attestor|feedser|vexer|ui|zastava>
|
||||
exp = <unix ts> (<= 300 s from iat)
|
||||
iat = <unix ts>
|
||||
nbf = iat - 30
|
||||
jti = <uuid>
|
||||
scope = "scanner.scan scanner.export signer.sign ..."
|
||||
```
|
||||
|
||||
**Sender‑constraint (`cnf`)**
|
||||
|
||||
* **DPoP**:
|
||||
|
||||
```json
|
||||
"cnf": { "jkt": "<base64url(SHA-256(JWK))>" }
|
||||
```
|
||||
* **mTLS**:
|
||||
|
||||
```json
|
||||
"cnf": { "x5t#S256": "<base64url(SHA-256(client_cert_der))>" }
|
||||
```
|
||||
|
||||
**Install/tenant context (custom claims)**
|
||||
|
||||
```
|
||||
tid = <tenant id> // multi-tenant
|
||||
inst = <installation id> // unique installation
|
||||
roles = [ "svc.scanner", "svc.signer", "ui.admin", ... ]
|
||||
plan? = <plan name> // optional hint for UIs; not used for enforcement
|
||||
```
|
||||
|
||||
> **Note**: Do **not** copy PoE claims into OpTok; OpTok ≠ entitlement. Only **Signer** checks PoE.
|
||||
|
||||
### 2.2 Refresh tokens (optional)
|
||||
|
||||
* Default **disabled**. If enabled (for UI interactive logins), pair with **DPoP‑bound** refresh tokens or **mTLS** client sessions; short TTL (≤ 8 h), rotating on use (replay‑safe).
|
||||
|
||||
### 2.3 ID tokens (optional)
|
||||
|
||||
* Issued for UI/browser OIDC flows (Authorization Code + PKCE); not used for service auth.
|
||||
|
||||
---
|
||||
|
||||
## 3) Endpoints & flows
|
||||
|
||||
### 3.1 OIDC discovery & keys
|
||||
|
||||
* `GET /.well-known/openid-configuration` → endpoints, algs, jwks_uri
|
||||
* `GET /jwks` → JSON Web Key Set (rotating, at least 2 active keys during transition)
|
||||
|
||||
### 3.2 Token issuance
|
||||
|
||||
* `POST /oauth/token`
|
||||
|
||||
* **Client Credentials** (service→service):
|
||||
|
||||
* **mTLS**: mutual TLS + `client_id` → bound token (`cnf.x5t#S256`)
|
||||
* **private_key_jwt**: JWT‑based client auth + **DPoP** header (preferred for tools and CLI)
|
||||
* **Device Code** (CLI): `POST /oauth/device/code` + `POST /oauth/token` poll
|
||||
* **Authorization Code + PKCE** (UI): standard
|
||||
|
||||
**DPoP handshake (example)**
|
||||
|
||||
1. Client prepares **JWK** (ephemeral keypair).
|
||||
2. Client sends **DPoP proof** header with fields:
|
||||
|
||||
```
|
||||
htm=POST
|
||||
htu=https://authority.../oauth/token
|
||||
iat=<now>
|
||||
jti=<uuid>
|
||||
```
|
||||
|
||||
signed with the DPoP private key; header carries JWK.
|
||||
3. Authority validates proof; issues access token with `cnf.jkt=<thumbprint(JWK)>`.
|
||||
4. Client uses the same DPoP key to sign **every subsequent API request** to services (Signer, Scanner, …).
|
||||
|
||||
**mTLS flow**
|
||||
|
||||
* Mutual TLS at the connection; Authority extracts client cert, validates chain; token carries `cnf.x5t#S256`.
|
||||
|
||||
### 3.3 Introspection & revocation (optional)
|
||||
|
||||
* `POST /oauth/introspect` → `{ active, sub, scope, aud, exp, cnf, ... }`
|
||||
* `POST /oauth/revoke` → revokes refresh tokens or opaque access tokens.
|
||||
* **Replay prevention**: maintain **DPoP `jti` cache** (TTL ≤ 10 min) to reject duplicate proofs when services supply DPoP nonces (Signer requires nonce for high‑value operations).
|
||||
|
||||
### 3.4 UserInfo (optional for UI)
|
||||
|
||||
* `GET /userinfo` (ID token context).
|
||||
|
||||
---
|
||||
|
||||
## 4) Audiences, scopes & RBAC
|
||||
|
||||
### 4.1 Audiences
|
||||
|
||||
* `signer` — only the **Signer** service should accept tokens with `aud=signer`.
|
||||
* `attestor`, `scanner`, `feedser`, `vexer`, `ui`, `zastava` similarly.
|
||||
|
||||
Services **must** verify `aud` and **sender constraint** (DPoP/mTLS) per their policy.
|
||||
|
||||
### 4.2 Core scopes
|
||||
|
||||
| Scope | Service | Operation |
|
||||
| ---------------------------------- | ------------------ | -------------------------- |
|
||||
| `signer.sign` | Signer | Request DSSE signing |
|
||||
| `attestor.write` | Attestor | Submit Rekor entries |
|
||||
| `scanner.scan` | Scanner.WebService | Submit scan jobs |
|
||||
| `scanner.export` | Scanner.WebService | Export SBOMs |
|
||||
| `scanner.read` | Scanner.WebService | Read catalog/SBOMs |
|
||||
| `vex.read` / `vex.admin` | Vexer | Query/operate |
|
||||
| `feedser.read` / `feedser.export` | Feedser | Query/exports |
|
||||
| `ui.read` / `ui.admin` | UI | View/admin |
|
||||
| `zastava.emit` / `zastava.enforce` | Scanner/Zastava | Runtime events / admission |
|
||||
|
||||
**Roles → scopes mapping** is configured centrally (Authority policy) and pushed during token issuance.
|
||||
|
||||
---
|
||||
|
||||
## 5) Storage & state
|
||||
|
||||
* **Configuration DB** (PostgreSQL/MySQL): clients, audiences, role→scope maps, tenant/installation registry, device code grants, persistent consents (if any).
|
||||
* **Cache** (Redis):
|
||||
|
||||
* DPoP **jti** replay cache (short TTL)
|
||||
* **Nonce** store (per resource server, if they demand nonce)
|
||||
* Device code pollers, rate limiting buckets
|
||||
* **JWKS**: key material in HSM/KMS or encrypted at rest; JWKS served from memory.
|
||||
|
||||
---
|
||||
|
||||
## 6) Key management & rotation
|
||||
|
||||
* Maintain **at least 2 signing keys** active during rotation; tokens carry `kid`.
|
||||
* Prefer **Ed25519** for compact tokens; maintain **ES256** fallback for FIPS contexts.
|
||||
* Rotation cadence: 30–90 days; emergency rotation supported.
|
||||
* Publish new JWKS **before** issuing tokens with the new `kid` to avoid cold‑start validation misses.
|
||||
* Keep **old keys** available **at least** for max token TTL + 5 minutes.
|
||||
|
||||
---
|
||||
|
||||
## 7) HA & performance
|
||||
|
||||
* **Stateless issuance** (except device codes/refresh) → scale horizontally behind a load‑balancer.
|
||||
* **DB** only for client metadata and optional flows; token checks are JWT‑local; introspection endpoints hit cache/DB minimally.
|
||||
* **Targets**:
|
||||
|
||||
* Token issuance P95 ≤ **20 ms** under warm cache.
|
||||
* DPoP proof validation ≤ **1 ms** extra per request at resource servers (Signer/Scanner).
|
||||
* 99.9% uptime; HPA on CPU/latency.
|
||||
|
||||
---
|
||||
|
||||
## 8) Security posture
|
||||
|
||||
* **Strict TLS** (1.3 preferred); HSTS; modern cipher suites.
|
||||
* **mTLS** enabled where required (Signer/Attestor paths).
|
||||
* **Replay protection**: DPoP `jti` cache, nonce support for **Signer** (add `DPoP-Nonce` header on 401; clients re‑sign).
|
||||
* **Rate limits** per client & per IP; exponential backoff on failures.
|
||||
* **Secrets**: clients use **private_key_jwt** or **mTLS**; never basic secrets over the wire.
|
||||
* **CSP/CSRF** hardening on UI flows; `SameSite=Lax` cookies; PKCE enforced.
|
||||
* **Logs** redact `Authorization` and DPoP proofs; store `sub`, `aud`, `scopes`, `inst`, `tid`, `cnf` thumbprints, not full keys.
|
||||
|
||||
---
|
||||
|
||||
## 9) Multi‑tenancy & installations
|
||||
|
||||
* **Tenant (`tid`)** and **Installation (`inst`)** registries define which audiences/scopes a client can request.
|
||||
* Cross‑tenant isolation enforced at issuance (disallow rogue `aud`), and resource servers **must** check that `tid` matches their configured tenant.
|
||||
|
||||
---
|
||||
|
||||
## 10) Admin & operations APIs
|
||||
|
||||
All under `/admin` (mTLS + `authority.admin` scope).
|
||||
|
||||
```
|
||||
POST /admin/clients # create/update client (confidential/public)
|
||||
POST /admin/audiences # register audience resource URIs
|
||||
POST /admin/roles # define role→scope mappings
|
||||
POST /admin/tenants # create tenant/install entries
|
||||
POST /admin/keys/rotate # rotate signing key (zero-downtime)
|
||||
GET /admin/metrics # Prometheus exposition (token issue rates, errors)
|
||||
GET /admin/healthz|readyz # health/readiness
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 11) Integration hard lines (what resource servers must enforce)
|
||||
|
||||
Every Stella Ops service that consumes Authority tokens **must**:
|
||||
|
||||
1. Verify JWT signature (`kid` in JWKS), `iss`, `aud`, `exp`, `nbf`.
|
||||
2. Enforce **sender‑constraint**:
|
||||
|
||||
* **DPoP**: validate DPoP proof (`htu`, `htm`, `iat`, `jti`) and match `cnf.jkt`; cache `jti` for replay defense; honor nonce challenges.
|
||||
* **mTLS**: match presented client cert thumbprint to token `cnf.x5t#S256`.
|
||||
3. Check **scopes**; optionally map to internal roles.
|
||||
4. Check **tenant** (`tid`) and **installation** (`inst`) as appropriate.
|
||||
5. For **Signer** only: require **both** OpTok and **PoE** in the request (enforced by Signer, not Authority).
|
||||
|
||||
---
|
||||
|
||||
## 12) Error surfaces & UX
|
||||
|
||||
* Token endpoint errors follow OAuth2 (`invalid_client`, `invalid_grant`, `invalid_scope`, `unauthorized_client`).
|
||||
* Resource servers use RFC 6750 style (`WWW-Authenticate: DPoP error="invalid_token", error_description="…", dpop_nonce="…" `).
|
||||
* For DPoP nonce challenges, clients retry with the server‑supplied nonce once.
|
||||
|
||||
---
|
||||
|
||||
## 13) Observability & audit
|
||||
|
||||
* **Metrics**:
|
||||
|
||||
* `authority.tokens_issued_total{grant,aud}`
|
||||
* `authority.dpop_validations_total{result}`
|
||||
* `authority.mtls_bindings_total{result}`
|
||||
* `authority.jwks_rotations_total`
|
||||
* `authority.errors_total{type}`
|
||||
* **Audit log** (immutable sink): token issuance (`sub`, `aud`, `scopes`, `tid`, `inst`, `cnf thumbprint`, `jti`), revocations, admin changes.
|
||||
* **Tracing**: token flows, DB reads, JWKS cache.
|
||||
|
||||
---
|
||||
|
||||
## 14) Configuration (YAML)
|
||||
|
||||
```yaml
|
||||
authority:
|
||||
issuer: "https://authority.internal"
|
||||
keys:
|
||||
algs: [ "EdDSA", "ES256" ]
|
||||
rotationDays: 60
|
||||
storage: kms://cluster-kms/authority-signing
|
||||
tokens:
|
||||
accessTtlSeconds: 180
|
||||
enableRefreshTokens: false
|
||||
clockSkewSeconds: 60
|
||||
dpop:
|
||||
enable: true
|
||||
nonce:
|
||||
enable: true
|
||||
ttlSeconds: 600
|
||||
mtls:
|
||||
enable: true
|
||||
caBundleFile: /etc/ssl/mtls/clients-ca.pem
|
||||
clients:
|
||||
- clientId: scanner-web
|
||||
grantTypes: [ "client_credentials" ]
|
||||
audiences: [ "scanner" ]
|
||||
auth: { type: "private_key_jwt", jwkFile: "/secrets/scanner-web.jwk" }
|
||||
senderConstraint: "dpop"
|
||||
scopes: [ "scanner.scan", "scanner.export", "scanner.read" ]
|
||||
- clientId: signer
|
||||
grantTypes: [ "client_credentials" ]
|
||||
audiences: [ "signer" ]
|
||||
auth: { type: "mtls" }
|
||||
senderConstraint: "mtls"
|
||||
scopes: [ "signer.sign" ]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 15) Testing matrix
|
||||
|
||||
* **JWT validation**: wrong `aud`, expired `exp`, skewed `nbf`, stale `kid`.
|
||||
* **DPoP**: invalid `htu`/`htm`, replayed `jti`, stale `iat`, wrong `jkt`, nonce dance.
|
||||
* **mTLS**: wrong client cert, wrong CA, thumbprint mismatch.
|
||||
* **RBAC**: scope enforcement per audience; over‑privileged client denied.
|
||||
* **Rotation**: JWKS rotation while load‑testing; zero‑downtime verification.
|
||||
* **HA**: kill one Authority instance; verify issuance continues; JWKS served by peers.
|
||||
* **Performance**: 1k token issuance/sec on 2 cores with Redis enabled for jti caching.
|
||||
|
||||
---
|
||||
|
||||
## 16) Threat model & mitigations (summary)
|
||||
|
||||
| Threat | Vector | Mitigation |
|
||||
| ------------------- | ---------------- | ------------------------------------------------------------------------------------------ |
|
||||
| Token theft | Copy of JWT | **Short TTL**, **sender‑constraint** (DPoP/mTLS); replay blocked by `jti` cache and nonces |
|
||||
| Replay across hosts | Reuse DPoP proof | Enforce `htu`/`htm`, `iat` freshness, `jti` uniqueness; services may require **nonce** |
|
||||
| Impersonation | Fake client | mTLS or `private_key_jwt` with pinned JWK; client registration & rotation |
|
||||
| Key compromise | Signing key leak | HSM/KMS storage, key rotation, audit; emergency key revoke path; narrow token TTL |
|
||||
| Cross‑tenant abuse | Scope elevation | Enforce `aud`, `tid`, `inst` at issuance and resource servers |
|
||||
| Downgrade to bearer | Strip DPoP | Resource servers require DPoP/mTLS based on `aud`; reject bearer without `cnf` |
|
||||
|
||||
---
|
||||
|
||||
## 17) Deployment & HA
|
||||
|
||||
* **Stateless** microservice, containerized; run ≥ 2 replicas behind LB.
|
||||
* **DB**: HA Postgres (or MySQL) for clients/roles; **Redis** for device codes, DPoP nonces/jtis.
|
||||
* **Secrets**: mount client JWKs via K8s Secrets/HashiCorp Vault; signing keys via KMS.
|
||||
* **Backups**: DB daily; Redis not critical (ephemeral).
|
||||
* **Disaster recovery**: export/import of client registry; JWKS rehydrate from KMS.
|
||||
* **Compliance**: TLS audit; penetration testing for OIDC flows.
|
||||
|
||||
---
|
||||
|
||||
## 18) Implementation notes
|
||||
|
||||
* Reference stack: **.NET 10** + **OpenIddict 6** (or IdentityServer if licensed) with custom DPoP validator and mTLS binding middleware.
|
||||
* Keep the DPoP/JTI cache pluggable; allow Redis/Memcached.
|
||||
* Provide **client SDKs** for C# and Go: DPoP key mgmt, proof generation, nonce handling, token refresh helper.
|
||||
|
||||
---
|
||||
|
||||
## 19) Quick reference — wire examples
|
||||
|
||||
**Access token (payload excerpt)**
|
||||
|
||||
```json
|
||||
{
|
||||
"iss": "https://authority.internal",
|
||||
"sub": "scanner-web",
|
||||
"aud": "signer",
|
||||
"exp": 1760668800,
|
||||
"iat": 1760668620,
|
||||
"nbf": 1760668620,
|
||||
"jti": "9d9c3f01-6e1a-49f1-8f77-9b7e6f7e3c50",
|
||||
"scope": "signer.sign",
|
||||
"tid": "tenant-01",
|
||||
"inst": "install-7A2B",
|
||||
"cnf": { "jkt": "KcVb2V...base64url..." }
|
||||
}
|
||||
```
|
||||
|
||||
**DPoP proof header fields (for POST /sign/dsse)**
|
||||
|
||||
```json
|
||||
{
|
||||
"htu": "https://signer.internal/sign/dsse",
|
||||
"htm": "POST",
|
||||
"iat": 1760668620,
|
||||
"jti": "4b1c9b3c-8a95-4c58-8a92-9c6cfb4a6a0b"
|
||||
}
|
||||
```
|
||||
|
||||
Signer validates that `hash(JWK)` in the proof matches `cnf.jkt` in the token.
|
||||
|
||||
---
|
||||
|
||||
## 20) Rollout plan
|
||||
|
||||
1. **MVP**: Client Credentials (private_key_jwt + DPoP), JWKS, short OpToks, per‑audience scopes.
|
||||
2. **Add**: mTLS‑bound tokens for Signer/Attestor; device code for CLI; optional introspection.
|
||||
3. **Hardening**: DPoP nonce support; full audit pipeline; HA tuning.
|
||||
4. **UX**: Tenant/installation admin UI; role→scope editors; client bootstrap wizards.
|
||||
|
||||
389
docs/ARCHITECTURE_CLI.md
Normal file
389
docs/ARCHITECTURE_CLI.md
Normal file
@@ -0,0 +1,389 @@
|
||||
# component_architecture_cli.md — **Stella Ops CLI** (2025Q4)
|
||||
|
||||
> **Scope.** Implementation‑ready architecture for **Stella Ops CLI**: command surface, process model, auth (Authority/DPoP), integration with Scanner/Vexer/Feedser/Signer/Attestor, Buildx plug‑in management, offline kit behavior, packaging, observability, security posture, and CI ergonomics.
|
||||
|
||||
---
|
||||
|
||||
## 0) Mission & boundaries
|
||||
|
||||
**Mission.** Provide a **fast, deterministic, CI‑friendly** command‑line interface to drive Stella Ops workflows:
|
||||
|
||||
* Build‑time SBOM generation via **Buildx generator** orchestration.
|
||||
* Post‑build **scan/compose/diff/export** against **Scanner.WebService**.
|
||||
* **Policy** operations and **VEX/Vuln** data pulls (operator tasks).
|
||||
* **Verification** (attestation, referrers, signatures) for audits.
|
||||
* Air‑gapped/offline **kit** administration.
|
||||
|
||||
**Boundaries.**
|
||||
|
||||
* CLI **never** signs; it only calls **Signer**/**Attestor** via backend APIs when needed (e.g., `report --attest`).
|
||||
* CLI **does not** store long‑lived credentials beyond OS keychain; tokens are **short** (Authority OpToks).
|
||||
* Heavy work (scanning, merging, policy) is executed **server‑side** (Scanner/Vexer/Feedser).
|
||||
|
||||
---
|
||||
|
||||
## 1) Solution layout & runtime form
|
||||
|
||||
```
|
||||
src/
|
||||
├─ StellaOps.Cli/ # net10.0 (Native AOT) single binary
|
||||
├─ StellaOps.Cli.Core/ # verb plumbing, config, HTTP, auth
|
||||
├─ StellaOps.Cli.Plugins/ # optional verbs packaged as plugins
|
||||
├─ StellaOps.Cli.Tests/ # unit + golden-output tests
|
||||
└─ packaging/
|
||||
├─ msix / msi / deb / rpm / brew formula
|
||||
└─ scoop manifest / winget manifest
|
||||
```
|
||||
|
||||
**Language/runtime**: .NET 10 **Native AOT** for speed/startup; Linux builds use **musl** static when possible.
|
||||
|
||||
**OS targets**: linux‑x64/arm64, windows‑x64/arm64, macOS‑x64/arm64.
|
||||
|
||||
---
|
||||
|
||||
## 2) Command surface (verbs)
|
||||
|
||||
> All verbs default to **JSON** output when `--json` is set (CI mode). Human output is concise, deterministic.
|
||||
|
||||
### 2.1 Auth & profile
|
||||
|
||||
* `auth login`
|
||||
|
||||
* Modes: **device‑code** (default), **client‑credentials** (service principal).
|
||||
* Produces **Authority** access token (OpTok) + stores **DPoP** keypair in OS keychain.
|
||||
* `auth status` — show current issuer, subject, audiences, expiry.
|
||||
* `auth logout` — wipe cached tokens/keys.
|
||||
|
||||
### 2.2 Build‑time SBOM (Buildx)
|
||||
|
||||
* `buildx install` — install/update the **StellaOps.Scanner.Sbomer.BuildXPlugin** on the host.
|
||||
* `buildx verify` — ensure generator is usable.
|
||||
* `buildx build` — thin wrapper around `docker buildx build --attest=type=sbom,generator=stellaops/sbom-indexer` with convenience flags:
|
||||
|
||||
* `--attest` (request Signer/Attestor via backend post‑push)
|
||||
* `--provenance` pass‑through (optional)
|
||||
|
||||
### 2.3 Scanning & artifacts
|
||||
|
||||
* `scan image <ref|digest>`
|
||||
|
||||
* Options: `--force`, `--wait`, `--view=inventory|usage|both`, `--format=cdx-json|cdx-pb|spdx-json`, `--attest` (ask backend to sign/log).
|
||||
* Streams progress; exits early unless `--wait`.
|
||||
* `diff image --old <digest> --new <digest> [--view ...]` — show layer‑attributed changes.
|
||||
* `export sbom <digest> [--view ... --format ... --out file]` — download artifact.
|
||||
* `report final <digest> [--policy-revision ... --attest]` — request PASS/FAIL report from backend (policy+vex) and optional attestation.
|
||||
|
||||
### 2.4 Policy & data
|
||||
|
||||
* `policy get/set/apply` — fetch active policy, apply staged policy, compute digest.
|
||||
* `feedser export` — trigger/export canonical JSON or Trivy DB (admin).
|
||||
* `vexer export` — trigger/export consensus/raw claims (admin).
|
||||
|
||||
### 2.5 Verification
|
||||
|
||||
* `verify attestation --uuid <rekor-uuid> | --artifact <sha256> | --bundle <path>` — call **Attestor /verify** and print proof summary.
|
||||
* `verify referrers <digest>` — ask **Signer /verify/referrers** (is image Stella‑signed?).
|
||||
* `verify image-signature <ref|digest>` — standalone cosign verification (optional, local).
|
||||
|
||||
### 2.6 Runtime (Zastava helper)
|
||||
|
||||
* `runtime policy test --images <digest,...> [--ns <name> --labels k=v,...]` — ask backend `/policy/runtime` like the webhook would.
|
||||
|
||||
### 2.7 Offline kit
|
||||
|
||||
* `offline kit pull` — fetch latest **Feedser JSON + Trivy DB + Vexer exports** as a tarball from a mirror.
|
||||
* `offline kit import <tar>` — upload the kit to on‑prem services (Feedser/Vexer).
|
||||
* `offline kit status` — list current seed versions.
|
||||
|
||||
### 2.8 Utilities
|
||||
|
||||
* `config set/get` — endpoint & defaults.
|
||||
* `whoami` — short auth display.
|
||||
* `version` — CLI + protocol versions; release channel.
|
||||
|
||||
---
|
||||
|
||||
## 3) AuthN: Authority + DPoP
|
||||
|
||||
### 3.1 Token acquisition
|
||||
|
||||
* **Device‑code**: the CLI opens an OIDC device code flow against **Authority**; the browser login is optional for service principals.
|
||||
* **Client‑credentials**: service principals use **private_key_jwt** or **mTLS** to get tokens.
|
||||
|
||||
### 3.2 DPoP key management
|
||||
|
||||
* On first login, the CLI generates an **ephemeral JWK** (Ed25519) and stores it in the **OS keychain** (Keychain/DPAPI/KWallet/Gnome Keyring).
|
||||
* Every request to backend services includes a **DPoP proof**; CLI refreshes tokens as needed.
|
||||
|
||||
### 3.3 Multi‑audience & scopes
|
||||
|
||||
* CLI requests **audiences** as needed per verb:
|
||||
|
||||
* `scanner` for scan/export/report/diff
|
||||
* `signer` (indirect; usually backend calls Signer)
|
||||
* `attestor` for verify
|
||||
* `feedser`/`vexer` for admin verbs
|
||||
|
||||
CLI rejects verbs if required scopes are missing.
|
||||
|
||||
---
|
||||
|
||||
## 4) Process model & reliability
|
||||
|
||||
### 4.1 HTTP client
|
||||
|
||||
* Single **http2** client with connection pooling, DNS pinning, retry/backoff (idempotent GET/POST marked safe).
|
||||
* **DPoP nonce** handling: on `401` with nonce challenge, CLI replays once.
|
||||
|
||||
### 4.2 Streaming
|
||||
|
||||
* `scan` and `report` support **server‑sent JSON lines** (progress events).
|
||||
* `--json` prints machine events; human mode shows compact spinners and crucial updates only.
|
||||
|
||||
### 4.3 Exit codes (CI‑safe)
|
||||
|
||||
| Code | Meaning |
|
||||
| ---- | ------------------------------------------- |
|
||||
| 0 | Success |
|
||||
| 2 | Policy fail (final report verdict=fail) |
|
||||
| 3 | Verification failed (attestation/signature) |
|
||||
| 4 | Auth error (invalid/missing token/DPoP) |
|
||||
| 5 | Resource not found (image/SBOM) |
|
||||
| 6 | Rate limited / quota exceeded |
|
||||
| 7 | Backend unavailable (retryable) |
|
||||
| 9 | Invalid arguments |
|
||||
|
||||
---
|
||||
|
||||
## 5) Configuration model
|
||||
|
||||
**Precedence:** CLI flags → env vars → config file → defaults.
|
||||
|
||||
**Config file**: `${XDG_CONFIG_HOME}/stellaops/config.yaml` (Windows: `%APPDATA%\StellaOps\config.yaml`)
|
||||
|
||||
```yaml
|
||||
cli:
|
||||
authority: "https://authority.internal"
|
||||
backend:
|
||||
scanner: "https://scanner-web.internal"
|
||||
attestor: "https://attestor.internal"
|
||||
feedser: "https://feedser-web.internal"
|
||||
vexer: "https://vexer-web.internal"
|
||||
auth:
|
||||
audienceDefault: "scanner"
|
||||
deviceCode: true
|
||||
output:
|
||||
json: false
|
||||
color: auto
|
||||
tls:
|
||||
caBundle: "/etc/ssl/certs/ca-bundle.crt"
|
||||
offline:
|
||||
kitMirror: "s3://mirror/stellaops-kit"
|
||||
```
|
||||
|
||||
Environment variables: `STELLAOPS_AUTHORITY`, `STELLAOPS_SCANNER_URL`, etc.
|
||||
|
||||
---
|
||||
|
||||
## 6) Buildx generator orchestration
|
||||
|
||||
* `buildx install` locates the Docker root directory, writes the **generator** plugin manifest, and pulls `stellaops/sbom-indexer` image (pinned digest).
|
||||
* `buildx build` wrapper injects:
|
||||
|
||||
* `--attest=type=sbom,generator=stellaops/sbom-indexer`
|
||||
* `--label org.stellaops.request=sbom`
|
||||
* Post‑build: CLI optionally calls **Scanner.WebService** to **verify referrers**, **compose** image SBOMs, and **attest** via Signer/Attestor.
|
||||
|
||||
**Detection**: If Buildx or generator unavailable, CLI falls back to **post‑build scan** with a warning.
|
||||
|
||||
---
|
||||
|
||||
## 7) Artifact handling
|
||||
|
||||
* **Downloads** (`export sbom`, `report final`): stream to file; compute sha256 on the fly; write sidecar `.sha256` and optional **verification bundle** (if `--bundle`).
|
||||
* **Uploads** (`offline kit import`): chunked upload; retry on transient errors; show progress bar (unless `--json`).
|
||||
|
||||
---
|
||||
|
||||
## 8) Security posture
|
||||
|
||||
* **DPoP private keys** stored in **OS keychain**; metadata cached in config.
|
||||
* **No plaintext tokens** on disk; short‑lived **OpToks** held in memory.
|
||||
* **TLS**: verify backend certificates; allow custom CA bundle for on‑prem.
|
||||
* **Redaction**: CLI logs remove `Authorization`, DPoP headers, PoE tokens.
|
||||
* **Supply chain**: CLI distribution binaries are **cosign‑signed**; `stellaops version --verify` checks its own signature.
|
||||
|
||||
---
|
||||
|
||||
## 9) Observability
|
||||
|
||||
* `--verbose` adds request IDs, timings, and retry traces.
|
||||
* **Metrics** (optional, disabled by default): Prometheus text file exporter for local monitoring in long‑running agents.
|
||||
* **Structured logs** (`--json`): per‑event JSON lines with `ts`, `verb`, `status`, `latencyMs`.
|
||||
|
||||
---
|
||||
|
||||
## 10) Performance targets
|
||||
|
||||
* Startup ≤ **20 ms** (AOT).
|
||||
* `scan image` request/response overhead ≤ **5 ms** (excluding server work).
|
||||
* Buildx wrapper overhead negligible (<1 ms).
|
||||
* Large artifact download (100 MB) sustained ≥ **80 MB/s** on local networks.
|
||||
|
||||
---
|
||||
|
||||
## 11) Tests & golden outputs
|
||||
|
||||
* **Unit tests**: argument parsing, config precedence, URL resolution, DPoP proof creation.
|
||||
* **Integration tests** (Testcontainers): mock Authority/Scanner/Attestor; CI pipeline with fake registry.
|
||||
* **Golden outputs**: verb snapshots for `--json` across OSes; kept in `tests/golden/…`.
|
||||
* **Contract tests**: ensure API shapes match service OpenAPI; fail build if incompatible.
|
||||
|
||||
---
|
||||
|
||||
## 12) Error envelopes (human + JSON)
|
||||
|
||||
**Human:**
|
||||
|
||||
```
|
||||
✖ Policy FAIL: 3 high, 1 critical (VEX suppressed 12)
|
||||
- pkg:rpm/openssl (CVE-2025-12345) — affected (vendor) — fixed in 3.0.14
|
||||
- pkg:npm/lodash (GHSA-xxxx) — affected — no fix
|
||||
See: https://ui.internal/scans/sha256:...
|
||||
Exit code: 2
|
||||
```
|
||||
|
||||
**JSON (`--json`):**
|
||||
|
||||
```json
|
||||
{ "event":"report", "status":"fail", "critical":1, "high":3, "url":"https://ui..." }
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 13) Admin & advanced flags
|
||||
|
||||
* `--authority`, `--scanner`, `--attestor`, `--feedser`, `--vexer` override config URLs.
|
||||
* `--no-color`, `--quiet`, `--json`.
|
||||
* `--timeout`, `--retries`, `--retry-backoff-ms`.
|
||||
* `--ca-bundle`, `--insecure` (dev only; prints warning).
|
||||
* `--trace` (dump HTTP traces to file; scrubbed).
|
||||
|
||||
---
|
||||
|
||||
## 14) Interop with other tools
|
||||
|
||||
* Emits **CycloneDX Protobuf** directly to stdout when `export sbom --format cdx-pb --out -`.
|
||||
* Pipes to `jq`/`yq` cleanly in JSON mode.
|
||||
* Can act as a **credential helper** for scripts: `stellaops auth token --aud scanner` prints a one‑shot token for curl.
|
||||
|
||||
---
|
||||
|
||||
## 15) Packaging & distribution
|
||||
|
||||
* **Installers**: deb/rpm (postinst registers completions), Homebrew, Scoop, Winget, MSI/MSIX.
|
||||
* **Shell completions**: bash/zsh/fish/pwsh.
|
||||
* **Update channel**: `stellaops self-update` (optional) fetches cosign‑signed release manifest; corporate environments can disable.
|
||||
|
||||
---
|
||||
|
||||
## 16) Security hard lines
|
||||
|
||||
* Refuse to print token values; redact Authorization headers in verbose output.
|
||||
* Disallow `--insecure` unless `STELLAOPS_CLI_ALLOW_INSECURE=1` set (double opt‑in).
|
||||
* Enforce **short token TTL**; refresh proactively when <30 s left.
|
||||
* Device‑code cache binding to **machine** and **user** (protect against copy to other machines).
|
||||
|
||||
---
|
||||
|
||||
## 17) Wire sequences
|
||||
|
||||
**A) Scan & wait with attestation**
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
participant CLI
|
||||
participant Auth as Authority
|
||||
participant SW as Scanner.WebService
|
||||
participant SG as Signer
|
||||
participant AT as Attestor
|
||||
|
||||
CLI->>Auth: device code flow (DPoP)
|
||||
Auth-->>CLI: OpTok (aud=scanner)
|
||||
|
||||
CLI->>SW: POST /scans { imageRef, attest:true }
|
||||
SW-->>CLI: { scanId }
|
||||
CLI->>SW: GET /scans/{id} (poll)
|
||||
SW-->>CLI: { status: completed, artifacts, rekor? } # if attested
|
||||
|
||||
alt attestation pending
|
||||
SW->>SG: POST /sign/dsse (server-side)
|
||||
SG-->>SW: DSSE
|
||||
SW->>AT: POST /rekor/entries
|
||||
AT-->>SW: { uuid, proof }
|
||||
end
|
||||
|
||||
CLI->>SW: GET /sboms/<digest>?format=cdx-pb&view=usage
|
||||
SW-->>CLI: bytes
|
||||
```
|
||||
|
||||
**B) Verify attestation by artifact**
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
participant CLI
|
||||
participant AT as Attestor
|
||||
|
||||
CLI->>AT: POST /rekor/verify { artifactSha256 }
|
||||
AT-->>CLI: { ok:true, uuid, index, logURL }
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 18) Roadmap (CLI)
|
||||
|
||||
* `scan fs <path>` (local filesystem tree) → upload to backend for analysis.
|
||||
* `policy test --sbom <file>` (simulate policy results offline using local policy bundle).
|
||||
* `runtime capture` (developer mode) — capture small `/proc/<pid>/maps` for troubleshooting.
|
||||
* Pluggable output renderers for SARIF/HTML (admin‑controlled).
|
||||
|
||||
---
|
||||
|
||||
## 19) Example CI snippets
|
||||
|
||||
**GitHub Actions (post‑build)**
|
||||
|
||||
```yaml
|
||||
- name: Login (device code w/ OIDC broker)
|
||||
run: stellaops auth login --json --authority ${{ secrets.AUTHORITY_URL }}
|
||||
|
||||
- name: Scan
|
||||
run: stellaops scan image ${{ steps.build.outputs.digest }} --wait --json
|
||||
|
||||
- name: Export (usage view, protobuf)
|
||||
run: stellaops export sbom ${{ steps.build.outputs.digest }} --view usage --format cdx-pb --out sbom.pb
|
||||
|
||||
- name: Verify attestation
|
||||
run: stellaops verify attestation --artifact $(sha256sum sbom.pb | cut -d' ' -f1) --json
|
||||
```
|
||||
|
||||
**GitLab (buildx generator)**
|
||||
|
||||
```yaml
|
||||
script:
|
||||
- stellaops buildx install
|
||||
- docker buildx build --attest=type=sbom,generator=stellaops/sbom-indexer -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA .
|
||||
- stellaops scan image $CI_REGISTRY_IMAGE@$IMAGE_DIGEST --wait --json
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 20) Test matrix (OS/arch)
|
||||
|
||||
* Linux: ubuntu‑20.04/22.04/24.04 (x64, arm64), alpine (musl).
|
||||
* macOS: 13–15 (x64, arm64).
|
||||
* Windows: 10/11, Server 2019/2022 (x64, arm64).
|
||||
* Docker engines: Docker Desktop, containerd‑based runners.
|
||||
|
||||
462
docs/ARCHITECTURE_DEVOPS.md
Normal file
462
docs/ARCHITECTURE_DEVOPS.md
Normal file
@@ -0,0 +1,462 @@
|
||||
# component_architecture_devops.md — **Stella Ops Release & Operations** (2025Q4)
|
||||
|
||||
> **Scope.** Implementation‑ready blueprint for **how Stella Ops is built, versioned, signed, distributed, upgraded, licensed (PoE)**, and operated in customer environments (online and air‑gapped). Covers reproducible builds, supply‑chain attestations, registries, offline kits, migration/rollback, artifact lifecycle (MinIO/Mongo), monitoring SLOs, and customer activation.
|
||||
|
||||
---
|
||||
|
||||
## 0) Product vision (operations lens)
|
||||
|
||||
Stella Ops must be **trustable at a glance** and **boringly operable**:
|
||||
|
||||
* Every release ships with **first‑party SBOMs, provenance, and signatures**; services verify **each other’s** integrity at runtime.
|
||||
* Customers can deploy by **digest** and stay aligned with **LTS/stable/edge** channels.
|
||||
* Paid customers receive **attestation authority** (Signer accepts their PoE) while the core platform remains **free to run**.
|
||||
* Air‑gapped customers receive **offline kits** with verifiable digests and deterministic import.
|
||||
* Artifacts expire predictably; operators know what’s kept, for how long, and why.
|
||||
|
||||
---
|
||||
|
||||
## 1) Release trains & versioning
|
||||
|
||||
### 1.1 Channels
|
||||
|
||||
* **LTS** (12‑month support window): quarterly cadence (Q1/Q2/Q3/Q4).
|
||||
* **Stable** (default): monthly rollup (bug fixes + compatible features).
|
||||
* **Edge**: weekly; for early adopters, no guarantees.
|
||||
|
||||
### 1.2 Version strings
|
||||
|
||||
Semantic core + calendar tag:
|
||||
|
||||
```
|
||||
<MAJOR>.<MINOR>.<PATCH> (<YYYY>.<MM>) e.g., 2.4.1 (2027.06)
|
||||
```
|
||||
|
||||
* **MAJOR**: breaking API/DB changes (rare).
|
||||
* **MINOR**: new features, compatible schema migrations (expand/contract pattern).
|
||||
* **PATCH**: bug fixes, perf and security updates.
|
||||
* **Calendar tag** exposes **release year** used by Signer for **PoE window checks**.
|
||||
|
||||
### 1.3 Component alignment
|
||||
|
||||
A release is a **bundle** of image digests + charts + manifests. All services in a bundle are **wire‑compatible**. Mixed minor versions are allowed within a bounded skew:
|
||||
|
||||
* **Web UI ↔ backend**: `±1 minor`.
|
||||
* **Scanner ↔ Policy/Vexer/Feedser**: `±1 minor`.
|
||||
* **Authority/Signer/Attestor triangle**: **must** be same minor (crypto and DPoP/mTLS binding rules).
|
||||
|
||||
At startup, services **self‑advertise** their semver & channel; the UI surfaces **mismatch warnings**.
|
||||
|
||||
---
|
||||
|
||||
## 2) Supply‑chain pipeline (how a release is built)
|
||||
|
||||
### 2.1 Deterministic builds
|
||||
|
||||
* **Builders**: isolated **BuildKit** workers with pinned base images (digest only).
|
||||
* **Pinning**: lock files or `go.mod`, `package-lock.json`, `global.json`, `Directory.Packages.props` are **frozen** at tag.
|
||||
* **Reproducibility**: timestamps normalized; source date epoch; deterministic zips/tars.
|
||||
* **Multi‑arch**: linux/amd64 + linux/arm64 (Windows images track M2 roadmap).
|
||||
|
||||
### 2.2 First‑party SBOMs & provenance
|
||||
|
||||
* Each image gets **CycloneDX (JSON+Protobuf) SBOM** and **SLSA‑style provenance** attached as **OCI referrers**.
|
||||
* Scanner’s **Buildx generator** is used to produce SBOMs *during* build; a separate post‑build scan verifies parity (red flag if drift).
|
||||
* **Release manifest** (see §6.1) lists all digests and SBOM/attestation refs.
|
||||
|
||||
### 2.3 Signing & transparency
|
||||
|
||||
* Images are **cosign‑signed** (keyless) with a Stella Ops release identity; inclusion in a **transparency log** (Rekor) is required.
|
||||
* SBOM and provenance attestations are **DSSE** and also transparency‑logged.
|
||||
* Release keys (Fulcio roots or public keys) are embedded in **Signer** policy (for **scanner‑release validation** at customer side).
|
||||
|
||||
### 2.4 Gates & tests
|
||||
|
||||
* **Static**: linters, codegen checks, protobuf API freeze (backward‑compat tests).
|
||||
* **Unit/integration**: per‑component, plus **end‑to‑end** flows (scan→vex→policy→sign→attest).
|
||||
* **Perf SLOs**: hot paths (SBOM compose, diff, export) measured against budgets.
|
||||
* **Security**: dependency audit vs Feedser export; container hardening tests; minimal caps.
|
||||
* **Canary cohort**: internal staging + selected customers; one week on **edge** before **stable** tag.
|
||||
|
||||
---
|
||||
|
||||
## 3) Distribution & activation
|
||||
|
||||
### 3.1 Registries
|
||||
|
||||
* **Primary**: `registry.stella-ops.org` (OCI v2, supports Referrers API).
|
||||
* **Mirrors**: GHCR (read‑only), regional mirrors for latency.
|
||||
* **Pull by digest only** in Kubernetes/Compose manifests.
|
||||
|
||||
**Gating policy**:
|
||||
|
||||
* **Core images** (Authority, Scanner, Feedser, Vexer, Attestor, UI): public **read**.
|
||||
* **Enterprise add‑ons** (if any) and **pre‑release**: private repos via OAuth2 token service.
|
||||
|
||||
> Monetization lever is **signing** (PoE gate), not image pulls, so the core remains simple to consume.
|
||||
|
||||
### 3.2 OAuth2 token service (for private repos)
|
||||
|
||||
* Docker Registry’s token flow backed by **Authority**:
|
||||
|
||||
1. Client hits registry (`401` with `WWW-Authenticate: Bearer realm=…`).
|
||||
2. Client gets an **access token** from the token service (validated by Authority) with `scope=repository:…:pull`.
|
||||
3. Registry allows pull for the requested repo.
|
||||
* Tokens are **short‑lived** (60–300 s) and **DPoP‑bound**.
|
||||
|
||||
### 3.3 Offline kits (air‑gapped)
|
||||
|
||||
* Tarball per release channel:
|
||||
|
||||
```
|
||||
stellaops-kit-<ver>-<channel>.tar.zst
|
||||
/images/ OCI layout with all first-party images (multi-arch)
|
||||
/sboms/ CycloneDX JSON+PB for each image
|
||||
/attest/ DSSE bundles + Rekor proofs
|
||||
/charts/ Helm charts + values templates
|
||||
/compose/ docker-compose.yml + .env template
|
||||
/plugins/ Feedser/Vexer connectors (restart-time)
|
||||
/policy/ example policies
|
||||
/manifest/ release.yaml (see §6.1)
|
||||
```
|
||||
* Import via CLI `offline kit import`; checks digests and signatures before load.
|
||||
|
||||
---
|
||||
|
||||
## 4) Licensing (PoE) & monetization
|
||||
|
||||
**Principle**: **Only paid Stella Ops issues valid signed attestations.** Running the stack is free; signing requires PoE.
|
||||
|
||||
### 4.1 PoE issuance
|
||||
|
||||
* Customers purchase a plan and obtain a **PoE artifact** from `www.stella-ops.org`:
|
||||
|
||||
* **PoE‑JWT** (DPoP/mTLS‑bound) **or** **PoE mTLS client certificate**.
|
||||
* Contains: `license_id`, `plan`, `valid_release_year`, `max_version`, `exp`, optional `tenant/customer` IDs.
|
||||
|
||||
### 4.2 Online enforcement
|
||||
|
||||
* **Signer** calls **Licensing /license/introspect** on every signing request (see signer doc).
|
||||
* If **revoked/expired/out‑of‑window** → deny with machine‑readable reason.
|
||||
* All **valid** bundles are DSSE‑signed and **Attestor** logs them; Rekor UUID returned.
|
||||
* UI badges: “**Verified by Stella Ops**” with link to the public log.
|
||||
|
||||
### 4.3 Air‑gapped / offline
|
||||
|
||||
* Customers obtain a **time‑boxed PoE lease** (signed JSON, 7–30 days).
|
||||
* Signer accepts the lease and emits **provisional** attestations (clearly labeled).
|
||||
* When connectivity returns, a background job **endorses** the provisional entries with the cloud service, updating their status to **verified**.
|
||||
* Operators can export a **verification bundle** for auditors even before endorsement (contains DSSE + local Rekor proof + lease snapshot).
|
||||
|
||||
### 4.4 Stolen/abused PoE
|
||||
|
||||
* Customers report theft; **Licensing** flags `license_id` as **revoked**.
|
||||
* Subsequent Signer requests **deny**; previous attestations remain but can be marked **contested** (UI shows badge, optional re‑sign path upon new PoE).
|
||||
|
||||
---
|
||||
|
||||
## 5) Deployment path (customer side)
|
||||
|
||||
### 5.1 First install
|
||||
|
||||
* **Helm** (Kubernetes) or **Compose** (VMs). Example (K8s):
|
||||
|
||||
```bash
|
||||
helm repo add stellaops https://charts.stella-ops.org
|
||||
helm install stella stellaops/platform \
|
||||
--version 2.4.0 \
|
||||
--set global.channel=stable \
|
||||
--set authority.issuer=https://authority.stella.local \
|
||||
--set scanner.minio.endpoint=http://minio.stella.local:9000 \
|
||||
--set scanner.mongo.uri=mongodb://mongo/scanner \
|
||||
--set feedser.mongo.uri=mongodb://mongo/feedser \
|
||||
--set vexer.mongo.uri=mongodb://mongo/vexer
|
||||
```
|
||||
|
||||
* Post‑install job registers **Authority clients** (Scanner, Signer, Attestor, UI) and prints **bootstrap** URLs and client credentials (sealed secrets).
|
||||
* UI banner shows **release bundle** and verification state (cosign OK? Rekor OK?).
|
||||
|
||||
### 5.2 Updates
|
||||
|
||||
* **Blue/green**: pull new bundle by **digest**; deploy side‑by‑side; cut traffic.
|
||||
|
||||
* **Rolling**: upgrade stateful components in safe order:
|
||||
|
||||
1. Authority (stateless, dual‑key rotation ready)
|
||||
2. Signer/Attestor (same minor)
|
||||
3. Scanner WebService & Workers
|
||||
4. Feedser, then Vexer (schema migrations are expand/contract)
|
||||
5. UI last
|
||||
|
||||
* **DB migrations** are **expand/contract**:
|
||||
|
||||
* Phase A (release N): **add** new fields/indexes, write old+new.
|
||||
* Phase B (N+1): **read** new fields; **drop** old.
|
||||
* Rollback is a matter of redeploying previous images and keeping both schemas valid.
|
||||
|
||||
### 5.3 Rollback
|
||||
|
||||
* Images referenced by **digest**; keep previous release manifest `K` versions back.
|
||||
* `helm rollback` or compose `docker compose -f release-K.yml up -d`.
|
||||
* Mongo migrations are additive; **no destructive changes** within a single minor.
|
||||
|
||||
---
|
||||
|
||||
## 6) Release payloads & manifests
|
||||
|
||||
### 6.1 Release manifest (`release.yaml`)
|
||||
|
||||
```yaml
|
||||
release:
|
||||
version: "2.4.1"
|
||||
channel: "stable"
|
||||
date: "2027-06-20T12:00:00Z"
|
||||
calendar: "2027.06"
|
||||
components:
|
||||
- name: scanner-webservice
|
||||
image: registry.stella-ops.org/stellaops/scanner-web@sha256:aa..bb
|
||||
sbom: oci://.../referrers/cdx-json@sha256:11..22
|
||||
provenance: oci://.../attest/provenance@sha256:33..44
|
||||
signature: { rekorUUID: "…" }
|
||||
- name: signer
|
||||
image: registry.stella-ops.org/stellaops/signer@sha256:cc..dd
|
||||
signature: { rekorUUID: "…" }
|
||||
charts:
|
||||
- name: platform
|
||||
version: "2.4.1"
|
||||
digest: "sha256:ee..ff"
|
||||
compose:
|
||||
file: "docker-compose.yml"
|
||||
digest: "sha256:77..88"
|
||||
checksums:
|
||||
sha256: "… digest of this release.yaml …"
|
||||
```
|
||||
|
||||
The manifest is **cosign‑signed**; UI/CLI can verify a bundle without talking to registries.
|
||||
|
||||
### 6.2 Image labels (release metadata)
|
||||
|
||||
Each image sets OCI labels:
|
||||
|
||||
```
|
||||
org.opencontainers.image.version = "2.4.1"
|
||||
org.opencontainers.image.revision = "<git sha>"
|
||||
org.opencontainers.image.created = "2027-06-20T12:00:00Z"
|
||||
org.stellaops.release.calendar = "2027.06"
|
||||
org.stellaops.release.channel = "stable"
|
||||
org.stellaops.build.slsaProvenance = "oci://…"
|
||||
```
|
||||
|
||||
Signer validates **scanner** image’s cosign identity + calendar tag for **release window** checks.
|
||||
|
||||
---
|
||||
|
||||
## 7) Artifact lifecycle & storage (MinIO/Mongo)
|
||||
|
||||
### 7.1 Buckets & prefixes (MinIO)
|
||||
|
||||
```
|
||||
s3://stellaops/
|
||||
scanner/
|
||||
layers/<sha256>/sbom.cdx.json.zst
|
||||
images/<imgDigest>/inventory.cdx.pb
|
||||
images/<imgDigest>/usage.cdx.pb
|
||||
diffs/<old>_<new>/diff.json.zst
|
||||
attest/<artifactSha256>.dsse.json
|
||||
feedser/
|
||||
json/<exportId>/...
|
||||
trivy/<exportId>/...
|
||||
vexer/
|
||||
exports/<exportId>/...
|
||||
attestor/
|
||||
dsse/<bundleSha256>.json
|
||||
proof/<rekorUuid>.json
|
||||
```
|
||||
|
||||
### 7.2 ILM classes
|
||||
|
||||
* **`short`**: working artifacts (diffs, queues) — TTL 7–14 days.
|
||||
* **`default`**: SBOMs & indexes — TTL 90–180 days (configurable).
|
||||
* **`compliance`**: signed reports & attested exports — **Object Lock** (governance/compliance) 1–7 years.
|
||||
|
||||
### 7.3 Artifact Lifecycle Controller (ALC)
|
||||
|
||||
* A background worker (part of Scanner.WebService) enforces **TTL** and **reference counting**:
|
||||
|
||||
* Artifacts referenced by **reports** or **tickets** are pinned.
|
||||
* ILM actions logged; UI shows per‑class usage & upcoming purges.
|
||||
|
||||
### 7.4 Mongo retention
|
||||
|
||||
* **Scanner**: `runtime.events` use TTL (e.g., 30–90 days); **catalog** permanent.
|
||||
* **Feedser/Vexer**: raw docs keep **last N windows**; canonical stores permanent.
|
||||
* **Attestor**: `entries` permanent; `dedupe` TTL 24–48h.
|
||||
|
||||
---
|
||||
|
||||
## 8) Observability & SLOs (operations)
|
||||
|
||||
* **Uptime SLO**: 99.9% for Signer/Authority/Attestor; 99.5% for Scanner WebService; Vexer/Feedser 99.0%.
|
||||
* **Error budgets**: tracked per month; dashboards show burn rates.
|
||||
* **Golden signals**:
|
||||
|
||||
* **Latency**: token issuance, sign→attest round‑trip, scan enqueue→emit, export build.
|
||||
* **Saturation**: queue depth, Mongo write IOPS, MinIO net throughput.
|
||||
* **Traffic**: scans/min, attestations/min, webhook admits/min.
|
||||
* **Errors**: 5xx rates, cosign verification failures, Rekor timeouts.
|
||||
|
||||
Prometheus + OTLP; Grafana dashboards ship in the charts.
|
||||
|
||||
---
|
||||
|
||||
## 9) Security & compliance operations
|
||||
|
||||
* **Key rotation**:
|
||||
|
||||
* Authority JWKS: 60‑day cadence, dual‑key overlap.
|
||||
* Release signing identities: rotate per minor or quarterly.
|
||||
* Sigstore roots mirrored and pinned; alarms on drift.
|
||||
|
||||
* **FIPS mode** (Gov build):
|
||||
|
||||
* Enforce `ES256` + KMS/HSM; disable Ed25519; MLS ciphers only.
|
||||
* Local **Rekor v2** and **Fulcio** alternatives; **air‑gapped** CA.
|
||||
|
||||
* **Vulnerability response**:
|
||||
|
||||
* Feedser red‑flag advisories trigger accelerated **stable** patch rollout; UI/CLI “security patch available” notice.
|
||||
|
||||
* **Backups/DR**:
|
||||
|
||||
* Mongo nightly snapshots; MinIO versioning + replication (if configured).
|
||||
* Restore runbooks tested quarterly with synthetic data.
|
||||
|
||||
---
|
||||
|
||||
## 10) Customer update flow (how versions are fetched & activated)
|
||||
|
||||
### 10.1 Online clusters
|
||||
|
||||
* **UI** surfaces update banner with **release manifest** diff and risk notes.
|
||||
* Operator approves → **Controller** pulls new images by digest; health‑checks; moves traffic; deprecates old revision.
|
||||
* Post‑switch, **schema Phase B** migrations (if any) run automatically.
|
||||
|
||||
### 10.2 Air‑gapped clusters
|
||||
|
||||
* Operator downloads **offline kit** from a mirror → `stellaops offline kit import`.
|
||||
* Controller validates bundle checksums and **cosign signatures**; applies charts/compose by digest.
|
||||
* After install, **verify** page shows green checks: image sigs, SBOMs attached, provenance logged.
|
||||
|
||||
### 10.3 CLI self‑update (optional)
|
||||
|
||||
* `stellaops self-update` pulls a **signed release manifest** and verifies the **CLI binary** with cosign before swapping (admin can disable).
|
||||
|
||||
---
|
||||
|
||||
## 11) Compatibility & deprecation policy
|
||||
|
||||
* **APIs** are stable within a **major**; breaking changes imply **MAJOR++** and deprecation period of one minor.
|
||||
* **Storage**: expand/contract; “drop old fields” only after one minor grace.
|
||||
* **Config**: feature flags (default off) for risky features (e.g., eBPF).
|
||||
|
||||
---
|
||||
|
||||
## 12) Runbooks (selected)
|
||||
|
||||
### 12.1 Lost PoE
|
||||
|
||||
1. Suspend **automatic attestation** jobs.
|
||||
2. Use CLI `stellaops signer status` to confirm `entitlement_denied`.
|
||||
3. Obtain new PoE from portal; verify on Signer `/poe/verify`.
|
||||
4. Re‑enable; optionally **re‑sign** last N reports (UI button → batch).
|
||||
|
||||
### 12.2 Rekor outage (self‑hosted)
|
||||
|
||||
* Attestor returns `202 (pending)` with queued proof fetch.
|
||||
* Keep DSSE bundles locally; re‑submit on schedule; UI badge shows **Pending**.
|
||||
* If outage > SLA, you can switch to a **mirror** log in config; Attestor writes to both when restored.
|
||||
|
||||
### 12.3 Emergency downgrade
|
||||
|
||||
* Identify prior release manifest (UI → Admin → Releases).
|
||||
* `helm rollback stella <revision>` (or compose apply previous file).
|
||||
* Services tolerate skew per §1.3; ensure **Signer/Authority/Attestor** are rolled together.
|
||||
|
||||
---
|
||||
|
||||
## 13) Example: cluster bootstrap (Compose)
|
||||
|
||||
```yaml
|
||||
version: "3.9"
|
||||
services:
|
||||
authority:
|
||||
image: registry.stella-ops.org/stellaops/authority@sha256:...
|
||||
env_file: ./env/authority.env
|
||||
ports: ["8440:8440"]
|
||||
signer:
|
||||
image: registry.stella-ops.org/stellaops/signer@sha256:...
|
||||
depends_on: [authority]
|
||||
environment:
|
||||
- SIGNER__POE__LICENSING__INTROSPECTURL=https://www.stella-ops.org/api/v1/license/introspect
|
||||
attestor:
|
||||
image: registry.stella-ops.org/stellaops/attestor@sha256:...
|
||||
depends_on: [signer]
|
||||
scanner-web:
|
||||
image: registry.stella-ops.org/stellaops/scanner-web@sha256:...
|
||||
environment:
|
||||
- SCANNER__S3__ENDPOINT=http://minio:9000
|
||||
scanner-worker:
|
||||
image: registry.stella-ops.org/stellaops/scanner-worker@sha256:...
|
||||
deploy: { replicas: 4 }
|
||||
feedser:
|
||||
image: registry.stella-ops.org/stellaops/feedser@sha256:...
|
||||
vexer:
|
||||
image: registry.stella-ops.org/stellaops/vexer@sha256:...
|
||||
web-ui:
|
||||
image: registry.stella-ops.org/stellaops/web-ui@sha256:...
|
||||
mongo:
|
||||
image: mongo:7
|
||||
minio:
|
||||
image: minio/minio:RELEASE.2025-07-10T00-00-00Z
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 14) Governance & keys (who owns the trust root)
|
||||
|
||||
* **Release key policy**: only the Release Engineering group can push signed releases; 4‑eyes approval; TUF‑style manifest possible in future.
|
||||
* **Signer acceptance policy**: embedded release identities are updated **only** via minor upgrade; emergency CRL supported.
|
||||
* **Customer keys**: none needed for core use; enterprise add‑ons may require per‑customer registries and keys.
|
||||
|
||||
---
|
||||
|
||||
## 15) Roadmap (Ops)
|
||||
|
||||
* **Windows containers GA** (Scanner + Zastava).
|
||||
* **Key Transparency** for Signer certs.
|
||||
* **Delta‑kit** (offline) for incremental updates.
|
||||
* **Operator CRDs** (K8s) to manage policy and ILM declaratively.
|
||||
* **SBOM **protobuf** as default transport at rest (smaller, faster).
|
||||
|
||||
---
|
||||
|
||||
### Appendix A — Minimal SLO monitors
|
||||
|
||||
* `authority.tokens_issued_total` slope ≈ normal.
|
||||
* `signer.requests_total{result="success"}/minute` > 0 (when scans occur).
|
||||
* `attestor.submit_latency_seconds{quantile=0.95}` < 0.3.
|
||||
* `scanner.scan_latency_seconds{quantile=0.95}` < target per image size.
|
||||
* `feedser.export.duration_seconds` stable; `vexer.consensus.conflicts_total` not exploding after policy changes.
|
||||
* MinIO `s3_requests_errors_total` near zero; Mongo `opcounters` hit expected baseline.
|
||||
|
||||
### Appendix B — Upgrade safety checklist
|
||||
|
||||
* Verify **release manifest** signature.
|
||||
* Ensure **Signer/Authority/Attestor** are same minor.
|
||||
* Verify **DB backups** < 24h old.
|
||||
* Confirm **ILM** won’t purge compliance artifacts during upgrade window.
|
||||
* Roll **one component** at a time; watch SLOs; abort on regression.
|
||||
|
||||
---
|
||||
|
||||
**End — component_architecture_devops.md**
|
||||
@@ -1,190 +1,433 @@
|
||||
# ARCHITECTURE.md — **StellaOps.Feedser**
|
||||
# component_architecture_feedser.md — **Stella Ops Feedser** (2025Q4)
|
||||
|
||||
> **Goal**: Build a sovereign-ready, self-hostable **feed-merge service** that ingests authoritative vulnerability sources, normalizes and de-duplicates them into **MongoDB**, and exports **JSON** and **Trivy-compatible DB** artifacts.
|
||||
> **Form factor**: Long-running **Web Service** with **REST APIs** (health, status, control) and an embedded **internal cron scheduler**. Controllable by StellaOps.Cli (# stella db ...)
|
||||
> **No signing inside Feedser** (signing is a separate pipeline step).
|
||||
> **Runtime SDK baseline**: .NET 10 Preview 7 (SDK 10.0.100-preview.7.25380.108) targeting `net10.0`, aligned with the deployed api.stella-ops.org service.
|
||||
> **Four explicit stages**:
|
||||
>
|
||||
> 1. **Source Download** → raw documents.
|
||||
> 2. **Parse & Normalize** → schema-validated DTOs enriched with canonical identifiers.
|
||||
> 3. **Merge & Deduplicate** → precedence-aware canonical records persisted to MongoDB.
|
||||
> 4. **Export** → JSON or TrivyDB (full or delta), then (externally) sign/publish.
|
||||
> **Scope.** Implementation‑ready architecture for **Feedser**: the vulnerability ingest/normalize/merge/export subsystem that produces deterministic advisory data for the Scanner + Policy + Vexer pipeline. Covers domain model, connectors, merge rules, storage schema, exports, APIs, performance, security, and test matrices.
|
||||
|
||||
---
|
||||
|
||||
## 1) Naming & Solution Layout
|
||||
## 0) Mission & boundaries
|
||||
|
||||
**Source connectors** namespace prefix: `StellaOps.Feedser.Source.*`
|
||||
**Exporters**:
|
||||
**Mission.** Acquire authoritative **vulnerability advisories** (vendor PSIRTs, distros, OSS ecosystems, CERTs), normalize them into a **canonical model**, reconcile aliases and version ranges, and export **deterministic artifacts** (JSON, Trivy DB) for fast backend joins.
|
||||
|
||||
* `StellaOps.Feedser.Exporter.Json`
|
||||
* `StellaOps.Feedser.Exporter.TrivyDb`
|
||||
**Boundaries.**
|
||||
|
||||
**Projects** (`/src`):
|
||||
* Feedser **does not** sign with private keys. When attestation is required, the export artifact is handed to the **Signer**/**Attestor** pipeline (out‑of‑process).
|
||||
* Feedser **does not** decide PASS/FAIL; it provides data to the **Policy** engine.
|
||||
* Online operation is **allowlist‑only**; air‑gapped deployments use the **Offline Kit**.
|
||||
|
||||
---
|
||||
|
||||
## 1) Topology & processes
|
||||
|
||||
**Process shape:** single ASP.NET Core service `StellaOps.Feedser.WebService` hosting:
|
||||
|
||||
* **Scheduler** with distributed locks (Mongo backed).
|
||||
* **Connectors** (fetch/parse/map).
|
||||
* **Merger** (canonical record assembly + precedence).
|
||||
* **Exporters** (JSON, Trivy DB).
|
||||
* **Minimal REST** for health/status/trigger/export.
|
||||
|
||||
**Scale:** HA by running N replicas; **locks** prevent overlapping jobs per source/exporter.
|
||||
|
||||
---
|
||||
|
||||
## 2) Canonical domain model
|
||||
|
||||
> Stored in MongoDB (database `feedser`), serialized with a **canonical JSON** writer (stable order, camelCase, normalized timestamps).
|
||||
|
||||
### 2.1 Core entities
|
||||
|
||||
**Advisory**
|
||||
|
||||
```
|
||||
StellaOps.Feedser.WebService/ # ASP.NET Core (Minimal API, net10.0 preview) WebService + embedded scheduler
|
||||
StellaOps.Feedser.Core/ # Domain models, pipelines, merge/dedupe engine, jobs orchestration
|
||||
StellaOps.Feedser.Models/ # Canonical POCOs, JSON Schemas, enums
|
||||
StellaOps.Feedser.Storage.Mongo/ # Mongo repositories, GridFS access, indexes, resume "flags"
|
||||
StellaOps.Feedser.Source.Common/ # HTTP clients, rate-limiters, schema validators, parsers utils
|
||||
StellaOps.Feedser.Source.Cve/
|
||||
StellaOps.Feedser.Source.Nvd/
|
||||
StellaOps.Feedser.Source.Ghsa/
|
||||
StellaOps.Feedser.Source.Osv/
|
||||
StellaOps.Feedser.Source.Jvn/
|
||||
StellaOps.Feedser.Source.CertCc/
|
||||
StellaOps.Feedser.Source.Kev/
|
||||
StellaOps.Feedser.Source.Kisa/
|
||||
StellaOps.Feedser.Source.CertIn/
|
||||
StellaOps.Feedser.Source.CertFr/
|
||||
StellaOps.Feedser.Source.CertBund/
|
||||
StellaOps.Feedser.Source.Acsc/
|
||||
StellaOps.Feedser.Source.Cccs/
|
||||
StellaOps.Feedser.Source.Ru.Bdu/ # HTML→schema with LLM fallback (gated)
|
||||
StellaOps.Feedser.Source.Ru.Nkcki/ # PDF/HTML bulletins → structured
|
||||
StellaOps.Feedser.Source.Vndr.Msrc/
|
||||
StellaOps.Feedser.Source.Vndr.Cisco/
|
||||
StellaOps.Feedser.Source.Vndr.Oracle/
|
||||
StellaOps.Feedser.Source.Vndr.Adobe/ # APSB ingest; emits vendor RangePrimitives with adobe.track/platform/priority telemetry + fixed-status provenance.
|
||||
StellaOps.Feedser.Source.Vndr.Apple/
|
||||
StellaOps.Feedser.Source.Vndr.Chromium/
|
||||
StellaOps.Feedser.Source.Vndr.Vmware/
|
||||
StellaOps.Feedser.Source.Distro.RedHat/
|
||||
StellaOps.Feedser.Source.Distro.Debian/ # Fetches DSA list + detail HTML, emits EVR RangePrimitives with per-release provenance and telemetry.
|
||||
StellaOps.Feedser.Source.Distro.Ubuntu/ # Ubuntu Security Notices connector (JSON index → EVR ranges with ubuntu.pocket telemetry).
|
||||
StellaOps.Feedser.Source.Distro.Suse/ # CSAF fetch pipeline emitting NEVRA RangePrimitives with suse.status vendor telemetry.
|
||||
StellaOps.Feedser.Source.Ics.Cisa/
|
||||
StellaOps.Feedser.Source.Ics.Kaspersky/
|
||||
StellaOps.Feedser.Normalization/ # Canonical mappers, validators, version-range normalization
|
||||
StellaOps.Feedser.Merge/ # Identity graph, precedence, deterministic merge
|
||||
StellaOps.Feedser.Exporter.Json/
|
||||
StellaOps.Feedser.Exporter.TrivyDb/
|
||||
StellaOps.Feedser.<Component>.Tests/ # Component-scoped unit/integration suites (Core, Storage.Mongo, Source.*, Exporter.*, WebService, etc.)
|
||||
advisoryId // internal GUID
|
||||
advisoryKey // stable string key (e.g., CVE-2025-12345 or vendor ID)
|
||||
title // short title (best-of from sources)
|
||||
summary // normalized summary (English; i18n optional)
|
||||
published // earliest source timestamp
|
||||
modified // latest source timestamp
|
||||
severity // normalized {none, low, medium, high, critical}
|
||||
cvss // {v2?, v3?, v4?} objects (vector, baseScore, severity, source)
|
||||
exploitKnown // bool (e.g., KEV/active exploitation flags)
|
||||
references[] // typed links (advisory, kb, patch, vendor, exploit, blog)
|
||||
sources[] // provenance for traceability (doc digests, URIs)
|
||||
```
|
||||
|
||||
---
|
||||
**Alias**
|
||||
|
||||
## 2) Runtime Shape
|
||||
```
|
||||
advisoryId
|
||||
scheme // CVE, GHSA, RHSA, DSA, USN, MSRC, etc.
|
||||
value // e.g., "CVE-2025-12345"
|
||||
```
|
||||
|
||||
**Process**: single service (`StellaOps.Feedser.WebService`)
|
||||
**Affected**
|
||||
|
||||
* `Program.cs`: top-level entry using **Generic Host**, **DI**, **Options** binding from `appsettings.json` + environment + optional `feedser.yaml`.
|
||||
* Built-in **scheduler** (cron-like) + **job manager** with **distributed locks** in Mongo to prevent overlaps, enforce timeouts, allow cancel/kill.
|
||||
* **REST APIs** for health/readiness/progress/trigger/kill/status.
|
||||
```
|
||||
advisoryId
|
||||
productKey // canonical product identity (see 2.2)
|
||||
rangeKind // semver | evr | nvra | apk | rpm | deb | generic | exact
|
||||
introduced? // string (format depends on rangeKind)
|
||||
fixed? // string (format depends on rangeKind)
|
||||
lastKnownSafe? // optional explicit safe floor
|
||||
arch? // arch or platform qualifier if source declares (x86_64, aarch64)
|
||||
distro? // distro qualifier when applicable (rhel:9, debian:12, alpine:3.19)
|
||||
ecosystem? // npm|pypi|maven|nuget|golang|…
|
||||
notes? // normalized notes per source
|
||||
```
|
||||
|
||||
**Key NuGet concepts** (indicative): `MongoDB.Driver`, `Polly` (retry/backoff), `System.Threading.Channels`, `Microsoft.Extensions.Http`, `Microsoft.Extensions.Hosting`, `Serilog`, `OpenTelemetry`.
|
||||
**Reference**
|
||||
|
||||
```
|
||||
advisoryId
|
||||
url
|
||||
kind // advisory | patch | kb | exploit | mitigation | blog | cvrf | csaf
|
||||
sourceTag // e.g., vendor/redhat, distro/debian, oss/ghsa
|
||||
```
|
||||
|
||||
**MergeEvent**
|
||||
|
||||
```
|
||||
advisoryKey
|
||||
beforeHash // canonical JSON hash before merge
|
||||
afterHash // canonical JSON hash after merge
|
||||
mergedAt
|
||||
inputs[] // source doc digests that contributed
|
||||
```
|
||||
|
||||
**ExportState**
|
||||
|
||||
```
|
||||
exportKind // json | trivydb
|
||||
baseExportId? // last full baseline
|
||||
baseDigest? // digest of last full baseline
|
||||
lastFullDigest? // digest of last full export
|
||||
lastDeltaDigest? // digest of last delta export
|
||||
cursor // per-kind incremental cursor
|
||||
files[] // last manifest snapshot (path → sha256)
|
||||
```
|
||||
|
||||
### 2.2 Product identity (`productKey`)
|
||||
|
||||
* **Primary:** `purl` (Package URL).
|
||||
* **OS packages:** RPM (NEVRA→purl:rpm), DEB (dpkg→purl:deb), APK (apk→purl:alpine), with **EVR/NVRA** preserved.
|
||||
* **Secondary:** `cpe` retained for compatibility; advisory records may carry both.
|
||||
* **Image/platform:** `oci:<registry>/<repo>@<digest>` for image‑level advisories (rare).
|
||||
* **Unmappable:** if a source is non‑deterministic, keep native string under `productKey="native:<provider>:<id>"` and mark **non‑joinable**.
|
||||
|
||||
---
|
||||
|
||||
## 3) Data Storage — **MongoDB** (single source of truth)
|
||||
## 3) Source families & precedence
|
||||
|
||||
**Database**: `feedser`
|
||||
**Write concern**: `majority` for merge/export state, `acknowledged` for raw docs.
|
||||
**Collections** (with “flags”/resume points):
|
||||
### 3.1 Families
|
||||
|
||||
* `source`
|
||||
* `_id`, `name`, `type`, `baseUrl`, `auth`, `notes`.
|
||||
* `source_state`
|
||||
* Keys: `sourceName` (unique), `enabled`, `cursor`, `lastSuccess`, `failCount`, `backoffUntil`, `paceOverrides`, `paused`.
|
||||
* Drives incremental fetch/parse/map resume and operator pause/pace controls.
|
||||
* `document`
|
||||
* `_id`, `sourceName`, `uri`, `fetchedAt`, `sha256`, `contentType`, `status`, `metadata`, `gridFsId`, `etag`, `lastModified`.
|
||||
* Index `{sourceName:1, uri:1}` unique; optional TTL for superseded versions.
|
||||
* `dto`
|
||||
* `_id`, `sourceName`, `documentId`, `schemaVer`, `payload` (BSON), `validatedAt`.
|
||||
* Index `{sourceName:1, documentId:1}`.
|
||||
* `advisory`
|
||||
* `_id`, `advisoryKey`, `title`, `summary`, `lang`, `published`, `modified`, `severity`, `exploitKnown`.
|
||||
* Unique `{advisoryKey:1}` plus indexes on `modified` and `published`.
|
||||
* `alias`
|
||||
* `advisoryId`, `scheme`, `value` with index `{scheme:1, value:1}`.
|
||||
* `affected`
|
||||
* `advisoryId`, `platform`, `name`, `versionRange`, `cpe`, `purl`, `fixedBy`, `introducedVersion`.
|
||||
* Index `{platform:1, name:1}`, `{advisoryId:1}`.
|
||||
* `reference`
|
||||
* `advisoryId`, `url`, `kind`, `sourceTag` (e.g., advisory/patch/kb).
|
||||
* Flags collections: `kev_flag`, `ru_flags`, `jp_flags`, `psirt_flags` keyed by `advisoryId`.
|
||||
* `merge_event`
|
||||
* `_id`, `advisoryKey`, `beforeHash`, `afterHash`, `mergedAt`, `inputs` (document ids).
|
||||
* `export_state`
|
||||
* `_id` (`json`/`trivydb`), `baseExportId`, `baseDigest`, `lastFullDigest`, `lastDeltaDigest`, `exportCursor`, `targetRepo`, `exporterVersion`.
|
||||
* `locks`
|
||||
* `_id` (`jobKey`), `holder`, `acquiredAt`, `heartbeatAt`, `leaseMs`, `ttlAt` (TTL index cleans dead locks).
|
||||
* `jobs`
|
||||
* `_id`, `type`, `args`, `state`, `startedAt`, `endedAt`, `error`, `owner`, `heartbeatAt`, `timeoutMs`.
|
||||
* **Vendor PSIRTs**: Microsoft, Oracle, Cisco, Adobe, Apple, VMware, Chromium…
|
||||
* **Linux distros**: Red Hat, SUSE, Ubuntu, Debian, Alpine…
|
||||
* **OSS ecosystems**: OSV, GHSA (GitHub Security Advisories), PyPI, npm, Maven, NuGet, Go.
|
||||
* **CERTs / national CSIRTs**: CISA (KEV, ICS), JVN, ACSC, CCCS, KISA, CERT‑FR/BUND, etc.
|
||||
|
||||
**GridFS buckets**: `fs.documents` for raw large payloads; referenced by `document.gridFsId`.
|
||||
### 3.2 Precedence (when claims conflict)
|
||||
|
||||
1. **Vendor PSIRT** (authoritative for their product).
|
||||
2. **Distro** (authoritative for packages they ship, including backports).
|
||||
3. **Ecosystem** (OSV/GHSA) for library semantics.
|
||||
4. **CERTs/aggregators** for enrichment (KEV/known exploited).
|
||||
|
||||
> Precedence affects **Affected** ranges and **fixed** info; **severity** is normalized to the **maximum** credible severity unless policy overrides. Conflicts are retained with **source provenance**.
|
||||
|
||||
---
|
||||
|
||||
## 4) Job & Scheduler Model
|
||||
## 4) Connectors & normalization
|
||||
|
||||
* Scheduler stores cron expressions per source/exporter in config; persists next-run pointers in Mongo.
|
||||
* Jobs acquire locks (`locks` collection) to ensure singleton execution per source/exporter.
|
||||
* Supports manual triggers via API endpoints (`POST /jobs/{type}`) and pause/resume toggles per source.
|
||||
|
||||
---
|
||||
|
||||
## 5) Connector Contracts
|
||||
|
||||
Connectors implement:
|
||||
### 4.1 Connector contract
|
||||
|
||||
```csharp
|
||||
public interface IFeedConnector {
|
||||
string SourceName { get; }
|
||||
Task FetchAsync(IServiceProvider sp, CancellationToken ct);
|
||||
Task ParseAsync(IServiceProvider sp, CancellationToken ct);
|
||||
Task MapAsync(IServiceProvider sp, CancellationToken ct);
|
||||
string SourceName { get; }
|
||||
Task FetchAsync(IServiceProvider sp, CancellationToken ct); // -> document collection
|
||||
Task ParseAsync(IServiceProvider sp, CancellationToken ct); // -> dto collection (validated)
|
||||
Task MapAsync(IServiceProvider sp, CancellationToken ct); // -> advisory/alias/affected/reference
|
||||
}
|
||||
```
|
||||
|
||||
* Fetch populates `document` rows respecting rate limits, conditional GET, and `source_state.cursor`.
|
||||
* Parse validates schema (JSON Schema, XSD) and writes sanitized DTO payloads.
|
||||
* Map produces canonical advisory rows + provenance entries; must be idempotent.
|
||||
* Base helpers in `StellaOps.Feedser.Source.Common` provide HTTP clients, retry policies, and watermark utilities.
|
||||
* **Fetch**: windowed (cursor), conditional GET (ETag/Last‑Modified), retry/backoff, rate limiting.
|
||||
* **Parse**: schema validation (JSON Schema, XSD/CSAF), content type checks; write **DTO** with normalized casing.
|
||||
* **Map**: build canonical records; all outputs carry **provenance** (doc digest, URI, anchors).
|
||||
|
||||
### 4.2 Version range normalization
|
||||
|
||||
* **SemVer** ecosystems (npm, pypi, maven, nuget, golang): normalize to `introduced`/`fixed` semver ranges (use `~`, `^`, `<`, `>=` canonicalized to intervals).
|
||||
* **RPM EVR**: `epoch:version-release` with `rpmvercmp` semantics; store raw EVR strings and also **computed order keys** for query.
|
||||
* **DEB**: dpkg version comparison semantics mirrored; store computed keys.
|
||||
* **APK**: Alpine version semantics; compute order keys.
|
||||
* **Generic**: if provider uses text, retain raw; do **not** invent ranges.
|
||||
|
||||
### 4.3 Severity & CVSS
|
||||
|
||||
* Normalize **CVSS v2/v3/v4** where available (vector, baseScore, severity).
|
||||
* If multiple CVSS sources exist, track them all; **effective severity** defaults to **max** by policy (configurable).
|
||||
* **ExploitKnown** toggled by KEV and equivalent sources; store **evidence** (source, date).
|
||||
|
||||
---
|
||||
|
||||
## 6) Merge & Normalization
|
||||
## 5) Merge engine
|
||||
|
||||
* Canonical model stored in `StellaOps.Feedser.Models` with serialization contracts used by storage/export layers.
|
||||
* `StellaOps.Feedser.Normalization` handles NEVRA/EVR/PURL range parsing, CVSS normalization, localization.
|
||||
* `StellaOps.Feedser.Merge` builds alias graphs keyed by CVE first, then falls back to vendor/regional IDs.
|
||||
* Precedence rules: PSIRT/OVAL overrides generic ranges; KEV only toggles exploitation; regional feeds enrich severity but don’t override vendor truth.
|
||||
* Determinism enforced via canonical JSON hashing logged in `merge_event`.
|
||||
### 5.1 Keying & identity
|
||||
|
||||
* Identity graph: **CVE** is primary node; vendor/distro IDs resolved via **Alias** edges (from connectors and Feedser’s alias tables).
|
||||
* `advisoryKey` is the canonical primary key (CVE if present, else vendor/distro key).
|
||||
|
||||
### 5.2 Merge algorithm (deterministic)
|
||||
|
||||
1. **Gather** all rows for `advisoryKey` (across sources).
|
||||
2. **Select title/summary** by precedence source (vendor>distro>ecosystem>cert).
|
||||
3. **Union aliases** (dedupe by scheme+value).
|
||||
4. **Merge `Affected`** with rules:
|
||||
|
||||
* Prefer **vendor** ranges for vendor products; prefer **distro** for **distro‑shipped** packages.
|
||||
* If both exist for same `productKey`, keep **both**; mark `sourceTag` and `precedence` so **Policy** can decide.
|
||||
* Never collapse range semantics across different families (e.g., rpm EVR vs semver).
|
||||
5. **CVSS/severity**: record all CVSS sets; compute **effectiveSeverity** = max (unless policy override).
|
||||
6. **References**: union with type precedence (advisory > patch > kb > exploit > blog); dedupe by URL; preserve `sourceTag`.
|
||||
7. Produce **canonical JSON**; compute **afterHash**; store **MergeEvent** with inputs and hashes.
|
||||
|
||||
> The merge is **pure** given inputs. Any change in inputs or precedence matrices changes the **hash** predictably.
|
||||
|
||||
---
|
||||
|
||||
## 6) Storage schema (MongoDB)
|
||||
|
||||
**Collections & indexes**
|
||||
|
||||
* `source` `{_id, type, baseUrl, enabled, notes}`
|
||||
* `source_state` `{sourceName(unique), enabled, cursor, lastSuccess, backoffUntil, paceOverrides}`
|
||||
* `document` `{_id, sourceName, uri, fetchedAt, sha256, contentType, status, metadata, gridFsId?, etag?, lastModified?}`
|
||||
|
||||
* Index: `{sourceName:1, uri:1}` unique, `{fetchedAt:-1}`
|
||||
* `dto` `{_id, sourceName, documentId, schemaVer, payload, validatedAt}`
|
||||
|
||||
* Index: `{sourceName:1, documentId:1}`
|
||||
* `advisory` `{_id, advisoryKey, title, summary, published, modified, severity, cvss, exploitKnown, sources[]}`
|
||||
|
||||
* Index: `{advisoryKey:1}` unique, `{modified:-1}`, `{severity:1}`, text index (title, summary)
|
||||
* `alias` `{advisoryId, scheme, value}`
|
||||
|
||||
* Index: `{scheme:1,value:1}`, `{advisoryId:1}`
|
||||
* `affected` `{advisoryId, productKey, rangeKind, introduced?, fixed?, arch?, distro?, ecosystem?}`
|
||||
|
||||
* Index: `{productKey:1}`, `{advisoryId:1}`, `{productKey:1, rangeKind:1}`
|
||||
* `reference` `{advisoryId, url, kind, sourceTag}`
|
||||
|
||||
* Index: `{advisoryId:1}`, `{kind:1}`
|
||||
* `merge_event` `{advisoryKey, beforeHash, afterHash, mergedAt, inputs[]}`
|
||||
|
||||
* Index: `{advisoryKey:1, mergedAt:-1}`
|
||||
* `export_state` `{_id(exportKind), baseExportId?, baseDigest?, lastFullDigest?, lastDeltaDigest?, cursor, files[]}`
|
||||
* `locks` `{_id(jobKey), holder, acquiredAt, heartbeatAt, leaseMs, ttlAt}` (TTL cleans dead locks)
|
||||
* `jobs` `{_id, type, args, state, startedAt, heartbeatAt, endedAt, error}`
|
||||
|
||||
**GridFS buckets**: `fs.documents` for raw payloads.
|
||||
|
||||
---
|
||||
|
||||
## 7) Exporters
|
||||
|
||||
* JSON exporter mirrors `aquasecurity/vuln-list` layout with deterministic ordering and reproducible timestamps.
|
||||
* Trivy DB exporter shells out to `trivy-db build`, produces Bolt archives, and reuses unchanged blobs from the last full baseline when running in delta mode. The exporter annotates `metadata.json` with `mode`, `baseExportId`, `baseManifestDigest`, `resetBaseline`, and `delta.changedFiles[]`/`delta.removedPaths[]`, and honours `publishFull` / `publishDelta` (ORAS) plus `includeFull` / `includeDelta` (offline bundle) toggles.
|
||||
* `StellaOps.Feedser.Storage.Mongo` provides cursors for delta exports based on `export_state.exportCursor` and the persisted per-file manifest (`export_state.files`).
|
||||
* Export jobs produce OCI tarballs (layer media type `application/vnd.aquasec.trivy.db.layer.v1.tar+gzip`) and optionally push via ORAS; `metadata.json` accompanies each layout so mirrors can decide between full refreshes and deltas.
|
||||
### 7.1 Deterministic JSON (vuln‑list style)
|
||||
|
||||
* Folder structure mirroring `/<scheme>/<first-two>/<rest>/…` with one JSON per advisory; deterministic ordering, stable timestamps, normalized whitespace.
|
||||
* `manifest.json` lists all files with SHA‑256 and a top‑level **export digest**.
|
||||
|
||||
### 7.2 Trivy DB exporter
|
||||
|
||||
* Builds Bolt DB archives compatible with Trivy; supports **full** and **delta** modes.
|
||||
* In delta, unchanged blobs are reused from the base; metadata captures:
|
||||
|
||||
```
|
||||
{
|
||||
"mode": "delta|full",
|
||||
"baseExportId": "...",
|
||||
"baseManifestDigest": "sha256:...",
|
||||
"changed": ["path1", "path2"],
|
||||
"removed": ["path3"]
|
||||
}
|
||||
```
|
||||
* Optional ORAS push (OCI layout) for registries.
|
||||
* Offline kit bundles include Trivy DB + JSON tree + export manifest.
|
||||
|
||||
### 7.3 Hand‑off to Signer/Attestor (optional)
|
||||
|
||||
* On export completion, if `attest: true` is set in job args, Feedser **posts** the artifact metadata to **Signer**/**Attestor**; Feedser itself **does not** hold signing keys.
|
||||
* Export record stores returned `{ uuid, index, url }` from **Rekor v2**.
|
||||
|
||||
---
|
||||
|
||||
## 8) Observability
|
||||
## 8) REST APIs
|
||||
|
||||
* Serilog structured logging with enrichment fields (`source`, `uri`, `stage`, `durationMs`).
|
||||
* OpenTelemetry traces around fetch/parse/map/export; metrics for rate limit hits, schema failures, dedupe ratios, package size. Connector HTTP metrics are emitted via the shared `feedser.source.http.*` instruments tagged with `feedser.source=<connector>` so per-source dashboards slice on that label instead of bespoke metric names.
|
||||
* Prometheus scraping endpoint served by WebService.
|
||||
All under `/api/v1/feedser`.
|
||||
|
||||
**Health & status**
|
||||
|
||||
```
|
||||
GET /healthz | /readyz
|
||||
GET /status → sources, last runs, export cursors
|
||||
```
|
||||
|
||||
**Sources & jobs**
|
||||
|
||||
```
|
||||
GET /sources → list of configured sources
|
||||
POST /sources/{name}/trigger → { jobId }
|
||||
POST /sources/{name}/pause | /resume → toggle
|
||||
GET /jobs/{id} → job status
|
||||
```
|
||||
|
||||
**Exports**
|
||||
|
||||
```
|
||||
POST /exports/json { full?:bool, force?:bool, attest?:bool } → { exportId, digest, rekor? }
|
||||
POST /exports/trivy { full?:bool, force?:bool, publish?:bool, attest?:bool } → { exportId, digest, rekor? }
|
||||
GET /exports/{id} → export metadata (kind, digest, createdAt, rekor?)
|
||||
```
|
||||
|
||||
**Search (operator debugging)**
|
||||
|
||||
```
|
||||
GET /advisories/{key}
|
||||
GET /advisories?scheme=CVE&value=CVE-2025-12345
|
||||
GET /affected?productKey=pkg:rpm/openssl&limit=100
|
||||
```
|
||||
|
||||
**AuthN/Z:** Authority tokens (OpTok) with roles: `feedser.read`, `feedser.admin`, `feedser.export`.
|
||||
|
||||
---
|
||||
|
||||
## 9) Security Considerations
|
||||
## 9) Configuration (YAML)
|
||||
|
||||
* Offline-first: connectors only reach allowlisted hosts.
|
||||
* BDU LLM fallback gated by config flag; logs audit trail with confidence score.
|
||||
* No secrets written to logs; secrets loaded via environment or mounted files.
|
||||
* Signing handled outside Feedser pipeline.
|
||||
```yaml
|
||||
feedser:
|
||||
mongo: { uri: "mongodb://mongo/feedser" }
|
||||
s3:
|
||||
endpoint: "http://minio:9000"
|
||||
bucket: "stellaops-feedser"
|
||||
scheduler:
|
||||
windowSeconds: 30
|
||||
maxParallelSources: 4
|
||||
sources:
|
||||
- name: redhat
|
||||
kind: csaf
|
||||
baseUrl: https://access.redhat.com/security/data/csaf/v2/
|
||||
signature: { type: pgp, keys: [ "…redhat PGP…" ] }
|
||||
enabled: true
|
||||
windowDays: 7
|
||||
- name: suse
|
||||
kind: csaf
|
||||
baseUrl: https://ftp.suse.com/pub/projects/security/csaf/
|
||||
signature: { type: pgp, keys: [ "…suse PGP…" ] }
|
||||
- name: ubuntu
|
||||
kind: usn-json
|
||||
baseUrl: https://ubuntu.com/security/notices.json
|
||||
signature: { type: none }
|
||||
- name: osv
|
||||
kind: osv
|
||||
baseUrl: https://api.osv.dev/v1/
|
||||
signature: { type: none }
|
||||
- name: ghsa
|
||||
kind: ghsa
|
||||
baseUrl: https://api.github.com/graphql
|
||||
auth: { tokenRef: "env:GITHUB_TOKEN" }
|
||||
exporters:
|
||||
json:
|
||||
enabled: true
|
||||
output: s3://stellaops-feedser/json/
|
||||
trivy:
|
||||
enabled: true
|
||||
mode: full
|
||||
output: s3://stellaops-feedser/trivy/
|
||||
oras:
|
||||
enabled: false
|
||||
repo: ghcr.io/org/feedser
|
||||
precedence:
|
||||
vendorWinsOverDistro: true
|
||||
distroWinsOverOsv: true
|
||||
severity:
|
||||
policy: max # or 'vendorPreferred' / 'distroPreferred'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 10) Deployment Notes
|
||||
## 10) Security & compliance
|
||||
|
||||
* **Outbound allowlist** per connector (domains, protocols); proxy support; TLS pinning where possible.
|
||||
* **Signature verification** for raw docs (PGP/cosign/x509) with results stored in `document.metadata.sig`. Docs failing verification may still be ingested but flagged; **merge** can down‑weight or ignore them by config.
|
||||
* **No secrets in logs**; auth material via `env:` or mounted files; HTTP redaction of `Authorization` headers.
|
||||
* **Multi‑tenant**: per‑tenant DBs or prefixes; per‑tenant S3 prefixes; tenant‑scoped API tokens.
|
||||
* **Determinism**: canonical JSON writer; export digests stable across runs given same inputs.
|
||||
|
||||
---
|
||||
|
||||
## 11) Performance targets & scale
|
||||
|
||||
* **Ingest**: ≥ 5k documents/min on 4 cores (CSAF/OpenVEX/JSON).
|
||||
* **Normalize/map**: ≥ 50k `Affected` rows/min on 4 cores.
|
||||
* **Merge**: ≤ 10 ms P95 per advisory at steady‑state updates.
|
||||
* **Export**: 1M advisories JSON in ≤ 90 s (streamed, zstd), Trivy DB in ≤ 60 s on 8 cores.
|
||||
* **Memory**: hard cap per job; chunked streaming writers; backpressure to avoid GC spikes.
|
||||
|
||||
**Scale pattern**: add Feedser replicas; Mongo scaling via indices and read/write concerns; GridFS only for oversized docs.
|
||||
|
||||
---
|
||||
|
||||
## 12) Observability
|
||||
|
||||
* **Metrics**
|
||||
|
||||
* `feedser.fetch.docs_total{source}`
|
||||
* `feedser.fetch.bytes_total{source}`
|
||||
* `feedser.parse.failures_total{source}`
|
||||
* `feedser.map.affected_total{source}`
|
||||
* `feedser.merge.changed_total`
|
||||
* `feedser.export.bytes{kind}`
|
||||
* `feedser.export.duration_seconds{kind}`
|
||||
* **Tracing** around fetch/parse/map/merge/export.
|
||||
* **Logs**: structured with `source`, `uri`, `docDigest`, `advisoryKey`, `exportId`.
|
||||
|
||||
---
|
||||
|
||||
## 13) Testing matrix
|
||||
|
||||
* **Connectors:** fixture suites for each provider/format (happy path; malformed; signature fail).
|
||||
* **Version semantics:** EVR vs dpkg vs semver edge cases (epoch bumps, tilde versions, pre‑releases).
|
||||
* **Merge:** conflicting sources (vendor vs distro vs OSV); verify precedence & dual retention.
|
||||
* **Export determinism:** byte‑for‑byte stable outputs across runs; digest equality.
|
||||
* **Performance:** soak tests with 1M advisories; cap memory; verify backpressure.
|
||||
* **API:** pagination, filters, RBAC, error envelopes (RFC 7807).
|
||||
* **Offline kit:** bundle build & import correctness.
|
||||
|
||||
---
|
||||
|
||||
## 14) Failure modes & recovery
|
||||
|
||||
* **Source outages:** scheduler backs off with exponential delay; `source_state.backoffUntil`; alerts on staleness.
|
||||
* **Schema drifts:** parse stage marks DTO invalid; job fails with clear diagnostics; connector version flags track supported schema ranges.
|
||||
* **Partial exports:** exporters write to temp prefix; **manifest commit** is atomic; only then move to final prefix and update `export_state`.
|
||||
* **Resume:** all stages idempotent; `source_state.cursor` supports window resume.
|
||||
|
||||
---
|
||||
|
||||
## 15) Operator runbook (quick)
|
||||
|
||||
* **Trigger all sources:** `POST /api/v1/feedser/sources/*/trigger`
|
||||
* **Force full export JSON:** `POST /api/v1/feedser/exports/json { "full": true, "force": true }`
|
||||
* **Force Trivy DB delta publish:** `POST /api/v1/feedser/exports/trivy { "full": false, "publish": true }`
|
||||
* **Inspect advisory:** `GET /api/v1/feedser/advisories?scheme=CVE&value=CVE-2025-12345`
|
||||
* **Pause noisy source:** `POST /api/v1/feedser/sources/osv/pause`
|
||||
|
||||
---
|
||||
|
||||
## 16) Rollout plan
|
||||
|
||||
1. **MVP**: Red Hat (CSAF), SUSE (CSAF), Ubuntu (USN JSON), OSV; JSON export.
|
||||
2. **Add**: GHSA GraphQL, Debian (DSA HTML/JSON), Alpine secdb; Trivy DB export.
|
||||
3. **Attestation hand‑off**: integrate with **Signer/Attestor** (optional).
|
||||
4. **Scale & diagnostics**: provider dashboards, staleness alerts, export cache reuse.
|
||||
5. **Offline kit**: end‑to‑end verified bundles for air‑gap.
|
||||
|
||||
* Default storage MongoDB; for air-gapped, bundle Mongo image + seeded data backup.
|
||||
* Horizontal scale achieved via multiple web service instances sharing Mongo locks.
|
||||
* Provide `feedser.yaml` template describing sources, rate limits, and export settings.
|
||||
|
||||
413
docs/ARCHITECTURE_SCANNER.md
Normal file
413
docs/ARCHITECTURE_SCANNER.md
Normal file
@@ -0,0 +1,413 @@
|
||||
# component_architecture_scanner.md — **Stella Ops Scanner** (2025Q4)
|
||||
|
||||
> **Scope.** Implementation‑ready architecture for the **Scanner** subsystem: WebService, Workers, analyzers, SBOM assembly (inventory & usage), per‑layer caching, three‑way diffs, artifact catalog (MinIO+Mongo), attestation hand‑off, and scale/security posture. This document is the contract between the scanning plane and everything else (Policy, Vexer, Feedser, UI, CLI).
|
||||
|
||||
---
|
||||
|
||||
## 0) Mission & boundaries
|
||||
|
||||
**Mission.** Produce **deterministic**, **explainable** SBOMs and diffs for container images and filesystems, quickly and repeatedly, without guessing. Emit two views: **Inventory** (everything present) and **Usage** (entrypoint closure + actually linked libs). Attach attestations through **Signer→Attestor→Rekor v2**.
|
||||
|
||||
**Boundaries.**
|
||||
|
||||
* Scanner **does not** produce PASS/FAIL. The backend (Policy + Vexer + Feedser) decides presentation and verdicts.
|
||||
* Scanner **does not** keep third‑party SBOM warehouses. It may **bind** to existing attestations for exact hashes.
|
||||
* Core analyzers are **deterministic** (no fuzzy identity). Optional heuristic plug‑ins (e.g., patch‑presence) run under explicit flags and never contaminate the core SBOM.
|
||||
|
||||
---
|
||||
|
||||
## 1) Solution & project layout
|
||||
|
||||
```
|
||||
src/
|
||||
├─ StellaOps.Scanner.WebService/ # REST control plane, catalog, diff, exports
|
||||
├─ StellaOps.Scanner.Worker/ # queue consumer; executes analyzers
|
||||
├─ StellaOps.Scanner.Models/ # DTOs, evidence, graph nodes, CDX/SPDX adapters
|
||||
├─ StellaOps.Scanner.Storage/ # Mongo repositories; MinIO object client; ILM/GC
|
||||
├─ StellaOps.Scanner.Queue/ # queue abstraction (Redis/NATS/RabbitMQ)
|
||||
├─ StellaOps.Scanner.Cache/ # layer cache; file CAS; bloom/bitmap indexes
|
||||
├─ StellaOps.Scanner.EntryTrace/ # ENTRYPOINT/CMD → terminal program resolver (shell AST)
|
||||
├─ StellaOps.Scanner.Analyzers.OS.[Apk|Dpkg|Rpm]/
|
||||
├─ StellaOps.Scanner.Analyzers.Lang.[Java|Node|Python|Go|DotNet|Rust]/
|
||||
├─ StellaOps.Scanner.Analyzers.Native.[ELF|PE|MachO]/ # PE/Mach-O planned (M2)
|
||||
├─ StellaOps.Scanner.Emit.CDX/ # CycloneDX (JSON + Protobuf)
|
||||
├─ StellaOps.Scanner.Emit.SPDX/ # SPDX 3.0.1 JSON
|
||||
├─ StellaOps.Scanner.Diff/ # image→layer→component three‑way diff
|
||||
├─ StellaOps.Scanner.Index/ # BOM‑Index sidecar (purls + roaring bitmaps)
|
||||
├─ StellaOps.Scanner.Tests.* # unit/integration/e2e fixtures
|
||||
└─ tools/
|
||||
├─ StellaOps.Scanner.Sbomer.BuildXPlugin/ # BuildKit generator (image referrer SBOMs)
|
||||
└─ StellaOps.Scanner.Sbomer.DockerImage/ # CLI‑driven scanner container
|
||||
```
|
||||
|
||||
**Runtime form‑factor:** two deployables
|
||||
|
||||
* **Scanner.WebService** (stateless REST)
|
||||
* **Scanner.Worker** (N replicas; queue‑driven)
|
||||
|
||||
---
|
||||
|
||||
## 2) External dependencies
|
||||
|
||||
* **OCI registry** with **Referrers API** (discover attached SBOMs/signatures).
|
||||
* **MinIO** (S3‑compatible) for SBOM artifacts; **Object Lock** for immutable classes; **ILM** for TTL.
|
||||
* **MongoDB** for catalog, job state, diffs, ILM rules.
|
||||
* **Queue** (Redis Streams/NATS/RabbitMQ).
|
||||
* **Authority** (on‑prem OIDC) for **OpToks** (DPoP/mTLS).
|
||||
* **Signer** + **Attestor** (+ **Fulcio/KMS** + **Rekor v2**) for DSSE + transparency.
|
||||
|
||||
---
|
||||
|
||||
## 3) Contracts & data model
|
||||
|
||||
### 3.1 Evidence‑first component model
|
||||
|
||||
**Nodes**
|
||||
|
||||
* `Image`, `Layer`, `File`
|
||||
* `Component` (`purl?`, `name`, `version?`, `type`, `id` — may be `bin:{sha256}`)
|
||||
* `Executable` (ELF/PE/Mach‑O), `Library` (native or managed), `EntryScript` (shell/launcher)
|
||||
|
||||
**Edges** (all carry **Evidence**)
|
||||
|
||||
* `contains(Image|Layer → File)`
|
||||
* `installs(PackageDB → Component)` (OS database row)
|
||||
* `declares(InstalledMetadata → Component)` (dist‑info, pom.properties, deps.json…)
|
||||
* `links_to(Executable → Library)` (ELF `DT_NEEDED`, PE imports)
|
||||
* `calls(EntryScript → Program)` (file:line from shell AST)
|
||||
* `attests(Rekor → Component|Image)` (SBOM/predicate binding)
|
||||
* `bound_from_attestation(Component_attested → Component_observed)` (hash equality proof)
|
||||
|
||||
**Evidence**
|
||||
|
||||
```
|
||||
{ source: enum, locator: (path|offset|line), sha256?, method: enum, timestamp }
|
||||
```
|
||||
|
||||
No confidences. Either a fact is proven with listed mechanisms, or it is not claimed.
|
||||
|
||||
### 3.2 Catalog schema (Mongo)
|
||||
|
||||
* `artifacts`
|
||||
|
||||
```
|
||||
{ _id, type: layer-bom|image-bom|diff|index,
|
||||
format: cdx-json|cdx-pb|spdx-json,
|
||||
bytesSha256, size, rekor: { uuid,index,url }?,
|
||||
ttlClass, immutable, refCount, createdAt }
|
||||
```
|
||||
* `images { imageDigest, repo, tag?, arch, createdAt, lastSeen }`
|
||||
* `layers { layerDigest, mediaType, size, createdAt, lastSeen }`
|
||||
* `links { fromType, fromDigest, artifactId }` // image/layer -> artifact
|
||||
* `jobs { _id, kind, args, state, startedAt, heartbeatAt, endedAt, error }`
|
||||
* `lifecycleRules { ruleId, scope, ttlDays, retainIfReferenced, immutable }`
|
||||
|
||||
### 3.3 Object store layout (MinIO)
|
||||
|
||||
```
|
||||
layers/<sha256>/sbom.cdx.json.zst
|
||||
layers/<sha256>/sbom.spdx.json.zst
|
||||
images/<imgDigest>/inventory.cdx.pb # CycloneDX Protobuf
|
||||
images/<imgDigest>/usage.cdx.pb
|
||||
indexes/<imgDigest>/bom-index.bin # purls + roaring bitmaps
|
||||
diffs/<old>_<new>/diff.json.zst
|
||||
attest/<artifactSha256>.dsse.json # DSSE bundle (cert chain + Rekor proof)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4) REST API (Scanner.WebService)
|
||||
|
||||
All under `/api/v1/scanner`. Auth: **OpTok** (DPoP/mTLS); RBAC scopes.
|
||||
|
||||
```
|
||||
POST /scans { imageRef|digest, force?:bool } → { scanId }
|
||||
GET /scans/{id} → { status, imageDigest, artifacts[], rekor? }
|
||||
GET /sboms/{imageDigest} ?format=cdx-json|cdx-pb|spdx-json&view=inventory|usage → bytes
|
||||
GET /diff?old=<digest>&new=<digest>&view=inventory|usage → diff.json
|
||||
POST /exports { imageDigest, format, view, attest?:bool } → { artifactId, rekor? }
|
||||
POST /reports { imageDigest, policyRevision? } → { reportId, rekor? } # delegates to backend policy+vex
|
||||
GET /catalog/artifacts/{id} → { meta }
|
||||
GET /healthz | /readyz | /metrics
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5) Execution flow (Worker)
|
||||
|
||||
### 5.1 Acquire & verify
|
||||
|
||||
1. **Resolve image** (prefer `repo@sha256:…`).
|
||||
2. **(Optional) verify image signature** per policy (cosign).
|
||||
3. **Pull blobs**, compute layer digests; record metadata.
|
||||
|
||||
### 5.2 Layer union FS
|
||||
|
||||
* Apply whiteouts; materialize final filesystem; map **file → first introducing layer**.
|
||||
* Windows layers (MSI/SxS/GAC) planned in **M2**.
|
||||
|
||||
### 5.3 Evidence harvest (parallel analyzers; deterministic only)
|
||||
|
||||
**A) OS packages**
|
||||
|
||||
* **apk**: `/lib/apk/db/installed`
|
||||
* **dpkg**: `/var/lib/dpkg/status`, `/var/lib/dpkg/info/*.list`
|
||||
* **rpm**: `/var/lib/rpm/Packages` (via librpm or parser)
|
||||
* Record `name`, `version` (epoch/revision), `arch`, source package where present, and **declared file lists**.
|
||||
|
||||
**B) Language ecosystems (installed state only)**
|
||||
|
||||
* **Java**: `META-INF/maven/*/pom.properties`, MANIFEST → `pkg:maven/...`
|
||||
* **Node**: `node_modules/**/package.json` → `pkg:npm/...`
|
||||
* **Python**: `*.dist-info/{METADATA,RECORD}` → `pkg:pypi/...`
|
||||
* **Go**: Go **buildinfo** in binaries → `pkg:golang/...`
|
||||
* **.NET**: `*.deps.json` + assembly metadata → `pkg:nuget/...`
|
||||
* **Rust**: crates only when **explicitly present** (embedded metadata or cargo/registry traces); otherwise binaries reported as `bin:{sha256}`.
|
||||
|
||||
> **Rule:** We only report components proven **on disk** with authoritative metadata. Lockfiles are evidence only.
|
||||
|
||||
**C) Native link graph**
|
||||
|
||||
* **ELF**: parse `PT_INTERP`, `DT_NEEDED`, RPATH/RUNPATH, **GNU symbol versions**; map **SONAMEs** to file paths; link executables → libs.
|
||||
* **PE/Mach‑O** (planned M2): import table, delay‑imports; version resources; code signatures.
|
||||
* Map libs back to **OS packages** if possible (via file lists); else emit `bin:{sha256}` components.
|
||||
|
||||
**D) EntryTrace (ENTRYPOINT/CMD → terminal program)**
|
||||
|
||||
* Read image config; parse shell (POSIX/Bash subset) with AST: `source`/`.` includes; `case/if`; `exec`/`command`; `run‑parts`.
|
||||
* Resolve commands via **PATH** within the **built rootfs**; follow language launchers (Java/Node/Python) to identify the terminal program (ELF/JAR/venv script).
|
||||
* Record **file:line** and choices for each hop; output chain graph.
|
||||
* Unresolvable dynamic constructs are recorded as **unknown** edges with reasons (e.g., `$FOO` unresolved).
|
||||
|
||||
**E) Attestation & SBOM bind (optional)**
|
||||
|
||||
* For each **file hash** or **binary hash**, query local cache of **Rekor v2** indices; if an SBOM attestation is found for **exact hash**, bind it to the component (origin=`attested`).
|
||||
* For the **image** digest, likewise bind SBOM attestations (build‑time referrers).
|
||||
|
||||
### 5.4 Component normalization (exact only)
|
||||
|
||||
* Create `Component` nodes only with deterministic identities: purl, or **`bin:{sha256}`** for unlabeled binaries.
|
||||
* Record **origin** (OS DB, installed metadata, linker, attestation).
|
||||
|
||||
### 5.5 SBOM assembly & emit
|
||||
|
||||
* **Per‑layer SBOM fragments**: components introduced by the layer (+ relationships).
|
||||
* **Image SBOMs**: merge fragments; refer back to them via **CycloneDX BOM‑Link** (or SPDX ExternalRef).
|
||||
* Emit both **Inventory** & **Usage** views.
|
||||
* Serialize **CycloneDX JSON** and **CycloneDX Protobuf**; optionally **SPDX 3.0.1 JSON**.
|
||||
* Build **BOM‑Index** sidecar: purl table + roaring bitmap; flag `usedByEntrypoint` components for fast backend joins.
|
||||
|
||||
### 5.6 DSSE attestation (via Signer/Attestor)
|
||||
|
||||
* WebService constructs **predicate** with `image_digest`, `stellaops_version`, `license_id`, `policy_digest?` (when emitting **final reports**), timestamps.
|
||||
* Calls **Signer** (requires **OpTok + PoE**); Signer verifies **entitlement + scanner image integrity** and returns **DSSE bundle**.
|
||||
* **Attestor** logs to **Rekor v2**; returns `{uuid,index,proof}` → stored in `artifacts.rekor`.
|
||||
|
||||
---
|
||||
|
||||
## 6) Three‑way diff (image → layer → component)
|
||||
|
||||
### 6.1 Keys & classification
|
||||
|
||||
* Component key: **purl** when present; else `bin:{sha256}`.
|
||||
* Diff classes: `added`, `removed`, `version_changed` (`upgraded|downgraded`), `metadata_changed` (e.g., origin from attestation vs observed).
|
||||
* Layer attribution: for each change, resolve the **introducing/removing layer**.
|
||||
|
||||
### 6.2 Algorithm (outline)
|
||||
|
||||
```
|
||||
A = components(imageOld, key)
|
||||
B = components(imageNew, key)
|
||||
|
||||
added = B \ A
|
||||
removed = A \ B
|
||||
changed = { k in A∩B : version(A[k]) != version(B[k]) || origin changed }
|
||||
|
||||
for each item in added/removed/changed:
|
||||
layer = attribute_to_layer(item, imageOld|imageNew)
|
||||
usageFlag = usedByEntrypoint(item, imageNew)
|
||||
emit diff.json (grouped by layer with badges)
|
||||
```
|
||||
|
||||
Diffs are stored as artifacts and feed **UI** and **CLI**.
|
||||
|
||||
---
|
||||
|
||||
## 7) Build‑time SBOMs (fast CI path)
|
||||
|
||||
**Scanner.Sbomer.BuildXPlugin** can act as a BuildKit **generator**:
|
||||
|
||||
* During `docker buildx build --attest=type=sbom,generator=stellaops/sbom-indexer`, run analyzers on the build context/output; attach SBOMs as OCI **referrers** to the built image.
|
||||
* Optionally request **Signer/Attestor** to produce **Stella Ops‑verified** attestation immediately; else, Scanner.WebService can verify and re‑attest post‑push.
|
||||
* Scanner.WebService trusts build‑time SBOMs per policy, enabling **no‑rescan** for unchanged bases.
|
||||
|
||||
---
|
||||
|
||||
## 8) Configuration (YAML)
|
||||
|
||||
```yaml
|
||||
scanner:
|
||||
queue:
|
||||
kind: redis
|
||||
url: "redis://queue:6379/0"
|
||||
mongo:
|
||||
uri: "mongodb://mongo/scanner"
|
||||
s3:
|
||||
endpoint: "http://minio:9000"
|
||||
bucket: "stellaops"
|
||||
objectLock: "governance" # or 'compliance'
|
||||
analyzers:
|
||||
os: { apk: true, dpkg: true, rpm: true }
|
||||
lang: { java: true, node: true, python: true, go: true, dotnet: true, rust: true }
|
||||
native: { elf: true, pe: false, macho: false } # PE/Mach-O in M2
|
||||
entryTrace: { enabled: true, shellMaxDepth: 64, followRunParts: true }
|
||||
emit:
|
||||
cdx: { json: true, protobuf: true }
|
||||
spdx: { json: true }
|
||||
compress: "zstd"
|
||||
rekor:
|
||||
url: "https://rekor-v2.internal"
|
||||
signer:
|
||||
url: "https://signer.internal"
|
||||
limits:
|
||||
maxParallel: 8
|
||||
perRegistryConcurrency: 2
|
||||
policyHints:
|
||||
verifyImageSignature: false
|
||||
trustBuildTimeSboms: true
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 9) Scale & performance
|
||||
|
||||
* **Parallelism**: per‑analyzer concurrency; bounded directory walkers; file CAS dedupe by sha256.
|
||||
* **Distributed locks** per **layer digest** to prevent duplicate work across Workers.
|
||||
* **Registry throttles**: per‑host concurrency budgets; exponential backoff on 429/5xx.
|
||||
* **Targets**:
|
||||
|
||||
* **Build‑time**: P95 ≤ 3–5 s on warmed bases (CI generator).
|
||||
* **Post‑build delta**: P95 ≤ 10 s for 200 MB images with cache hit.
|
||||
* **Emit**: CycloneDX Protobuf ≤ 150 ms for 5k components; JSON ≤ 500 ms.
|
||||
* **Diff**: ≤ 200 ms for 5k vs 5k components.
|
||||
|
||||
---
|
||||
|
||||
## 10) Security posture
|
||||
|
||||
* **AuthN**: Authority‑issued short OpToks (DPoP/mTLS).
|
||||
* **AuthZ**: scopes (`scanner.scan`, `scanner.export`, `scanner.catalog.read`).
|
||||
* **mTLS** to **Signer**/**Attestor**; only **Signer** can sign.
|
||||
* **No network fetches** during analysis (except registry pulls and optional Rekor index reads).
|
||||
* **Sandboxing**: non‑root containers; read‑only FS; seccomp profiles; disable execution of scanned content.
|
||||
* **Release integrity**: all first‑party images are **cosign‑signed**; Workers/WebService self‑verify at startup.
|
||||
|
||||
---
|
||||
|
||||
## 11) Observability & audit
|
||||
|
||||
* **Metrics**:
|
||||
|
||||
* `scanner.jobs_inflight`, `scanner.scan_latency_seconds`
|
||||
* `scanner.layer_cache_hits_total`, `scanner.file_cas_hits_total`
|
||||
* `scanner.artifact_bytes_total{format}`
|
||||
* `scanner.attestation_latency_seconds`, `scanner.rekor_failures_total`
|
||||
* **Tracing**: spans for acquire→union→analyzers→compose→emit→sign→log.
|
||||
* **Audit logs**: DSSE requests log `license_id`, `image_digest`, `artifactSha256`, `policy_digest?`, Rekor UUID on success.
|
||||
|
||||
---
|
||||
|
||||
## 12) Testing matrix
|
||||
|
||||
* **Determinism:** given same image + analyzers → byte‑identical **CDX Protobuf**; JSON normalized.
|
||||
* **OS packages:** ground‑truth images per distro; compare to package DB.
|
||||
* **Lang ecosystems:** sample images per ecosystem (Java/Node/Python/Go/.NET/Rust) with installed metadata; negative tests w/ lockfile‑only.
|
||||
* **Native & EntryTrace:** ELF graph correctness; shell AST cases (includes, run‑parts, exec, case/if).
|
||||
* **Diff:** layer attribution against synthetic two‑image sequences.
|
||||
* **Performance:** cold vs warm cache; large `node_modules` and `site‑packages`.
|
||||
* **Security:** ensure no code execution from image; fuzz parser inputs; path traversal resistance on layer extract.
|
||||
|
||||
---
|
||||
|
||||
## 13) Failure modes & degradations
|
||||
|
||||
* **Missing OS DB** (files exist, DB removed): record **files**; do **not** fabricate package components; emit `bin:{sha256}` where unavoidable; flag in evidence.
|
||||
* **Unreadable metadata** (corrupt dist‑info): record file evidence; skip component creation; annotate.
|
||||
* **Dynamic shell constructs**: mark unresolved edges with reasons (env var unknown) and continue; **Usage** view may be partial.
|
||||
* **Registry rate limits**: honor backoff; queue job retries with jitter.
|
||||
* **Signer refusal** (license/plan/version): scan completes; artifact produced; **no attestation**; WebService marks result as **unverified**.
|
||||
|
||||
---
|
||||
|
||||
## 14) Optional plug‑ins (off by default)
|
||||
|
||||
* **Patch‑presence detector** (signature‑based backport checks). Reads curated function‑level signatures from advisories; inspects binaries for patched code snippets to lower false‑positives for backported fixes. Runs as a sidecar analyzer that **annotates** components; never overrides core identities.
|
||||
* **Runtime probes** (with Zastava): when allowed, compare **/proc/<pid>/maps** (DSOs actually loaded) with static **Usage** view for precision.
|
||||
|
||||
---
|
||||
|
||||
## 15) DevOps & operations
|
||||
|
||||
* **HA**: WebService horizontal scale; Workers autoscale by queue depth & CPU; distributed locks on layers.
|
||||
* **Retention**: ILM rules per artifact class (`short`, `default`, `compliance`); **Object Lock** for compliance artifacts (reports, signed SBOMs).
|
||||
* **Upgrades**: bump **cache schema** when analyzer outputs change; WebService triggers refresh of dependent artifacts.
|
||||
* **Backups**: Mongo (daily dumps); MinIO (versioned buckets, replication); Rekor v2 DB snapshots.
|
||||
|
||||
---
|
||||
|
||||
## 16) CLI & UI touch points
|
||||
|
||||
* **CLI**: `stellaops scan <ref>`, `stellaops diff --old --new`, `stellaops export`, `stellaops verify attestation <bundle|url>`.
|
||||
* **UI**: Scan detail shows **Inventory/Usage** toggles, **Diff by Layer**, **Attestation badge** (verified/unverified), Rekor link, and **EntryTrace** chain with file:line breadcrumbs.
|
||||
|
||||
---
|
||||
|
||||
## 17) Roadmap (Scanner)
|
||||
|
||||
* **M2**: Windows containers (MSI/SxS/GAC analyzers), PE/Mach‑O native analyzer, deeper Rust metadata.
|
||||
* **M2**: Buildx generator GA (certified external registries), cross‑registry trust policies.
|
||||
* **M3**: Patch‑presence plug‑in GA (opt‑in), cross‑image corpus clustering (evidence‑only; not identity).
|
||||
* **M3**: Advanced EntryTrace (POSIX shell features breadth, busybox detection).
|
||||
|
||||
---
|
||||
|
||||
### Appendix A — EntryTrace resolution (pseudo)
|
||||
|
||||
```csharp
|
||||
ResolveEntrypoint(ImageConfig cfg, RootFs fs):
|
||||
cmd = Normalize(cfg.ENTRYPOINT, cfg.CMD)
|
||||
stack = [ Script(cmd, path=FindOnPath(cmd[0], fs)) ]
|
||||
visited = set()
|
||||
|
||||
while stack not empty and depth < MAX:
|
||||
cur = stack.pop()
|
||||
if cur in visited: continue
|
||||
visited.add(cur)
|
||||
|
||||
if IsShellScript(cur.path):
|
||||
ast = ParseShell(cur.path)
|
||||
foreach directive in ast:
|
||||
if directive is Source include:
|
||||
p = ResolveInclude(include.path, cur.env, fs)
|
||||
stack.push(Script(p))
|
||||
if directive is Exec call:
|
||||
p = ResolveExec(call.argv[0], cur.env, fs)
|
||||
stack.push(Program(p, argv=call.argv))
|
||||
if directive is Interpreter (python -m / node / java -jar):
|
||||
term = ResolveInterpreterTarget(call, fs)
|
||||
stack.push(Program(term))
|
||||
else:
|
||||
return Terminal(cur.path)
|
||||
|
||||
return Unknown(reason)
|
||||
```
|
||||
|
||||
### Appendix B — BOM‑Index sidecar
|
||||
|
||||
```
|
||||
struct Header { magic, version, imageDigest, createdAt }
|
||||
vector<string> purls
|
||||
map<purlIndex, roaring_bitmap> components
|
||||
optional map<purlIndex, roaring_bitmap> usedByEntrypoint
|
||||
```
|
||||
|
||||
418
docs/ARCHITECTURE_SIGNER.md
Normal file
418
docs/ARCHITECTURE_SIGNER.md
Normal file
@@ -0,0 +1,418 @@
|
||||
# component_architecture_signer.md — **Stella Ops Signer** (2025Q4)
|
||||
|
||||
> **Scope.** Implementation‑ready architecture for the **Signer**: the *only* service allowed to produce **Stella Ops‑verified** signatures over SBOMs and reports. It enforces **entitlement** (PoE), **release integrity** (scanner provenance), **sender‑constrained auth** (DPoP/mTLS), and emits **in‑toto/DSSE** bundles suitable for **Rekor v2** logging by the Attestor. Includes APIs, data flow, storage, quotas, security, and test matrices.
|
||||
|
||||
---
|
||||
|
||||
## 0) Mission & boundaries
|
||||
|
||||
**Mission.** Convert authenticated signing requests from trusted Stella Ops services into **verifiable** DSSE bundles while enforcing **license policy** and **supply‑chain integrity**.
|
||||
|
||||
**Boundaries.**
|
||||
|
||||
* **Signer does not push to Rekor** — it returns DSSE to the caller; **Attestor** logs to **Rekor v2**.
|
||||
* **Signer does not compute PASS/FAIL** — it signs SBOMs/reports produced by Scanner/WebService after backend evaluation.
|
||||
* **Signer is stateless for hot path** — long‑term storage is limited to audit events; all secrets/keys live in KMS/HSM or are ephemeral (keyless).
|
||||
|
||||
---
|
||||
|
||||
## 1) Responsibilities (contract)
|
||||
|
||||
1. **Authenticate** caller with **OpTok** (Authority OIDC, DPoP or mTLS‑bound).
|
||||
2. **Authorize** scopes (`signer.sign`) + audience (`aud=signer`) + tenant/installation.
|
||||
3. **Validate entitlement** via **PoE** (Proof‑of‑Entitlement) against Cloud Licensing `/license/introspect`.
|
||||
4. **Verify release integrity** of the **scanner** image digest presented in the request: must be **cosign‑signed** by Stella Ops release key, discoverable via **OCI Referrers API**.
|
||||
5. **Enforce plan & quotas** (concurrency/QPS/artifact size/rate caps).
|
||||
6. **Mint signing identity**:
|
||||
|
||||
* **Keyless** (default): get a short‑lived X.509 cert from **Fulcio** using the Signer’s OIDC identity and sign the DSSE.
|
||||
* **Keyful** (optional): sign with an HSM/KMS key.
|
||||
7. **Return DSSE bundle** (subject digests + predicate + cert chain or KMS key id).
|
||||
8. **Audit** every decision; expose metrics.
|
||||
|
||||
---
|
||||
|
||||
## 2) External dependencies
|
||||
|
||||
* **Authority** (on‑prem OIDC): validates OpToks (JWKS/introspection) and DPoP/mTLS.
|
||||
* **Licensing Service (cloud)**: `/license/introspect` to verify PoE (active, claims, expiry, revocation).
|
||||
* **Fulcio** (Sigstore) *or* **KMS/HSM**: to obtain certs or perform signatures.
|
||||
* **OCI Registry (Referrers API)**: to verify **scanner** image release signature.
|
||||
* **Attestor**: downstream service that writes DSSE bundles to **Rekor v2**.
|
||||
* **Config/state stores**: Redis (caches, rate buckets), Mongo/Postgres (audit log).
|
||||
|
||||
---
|
||||
|
||||
## 3) API surface (mTLS; DPoP supported)
|
||||
|
||||
Base path: `/api/v1/signer`. **All endpoints require**:
|
||||
|
||||
* Access token (JWT) from **Authority** with `aud=signer`, `scope=signer.sign`.
|
||||
* **Sender constraint**: DPoP proof per request or mTLS client cert.
|
||||
* **PoE** presented as either:
|
||||
|
||||
* **Client TLS cert** (if PoE is mTLS‑style) chained to Licensing CA, *or*
|
||||
* **PoE JWT** (DPoP/mTLS‑bound) in `X-PoE` header or request body.
|
||||
|
||||
### 3.1 `POST /sign/dsse`
|
||||
|
||||
Request (JSON):
|
||||
|
||||
```json
|
||||
{
|
||||
"subject": [
|
||||
{ "name": "s3://stellaops/images/sha256:.../inventory.cdx.pb",
|
||||
"digest": { "sha256": "..." } }
|
||||
],
|
||||
"predicateType": "https://stella-ops.org/attestations/sbom/1",
|
||||
"predicate": {
|
||||
"image_digest": "sha256:...",
|
||||
"stellaops_version": "2.3.1 (2027.04)",
|
||||
"license_id": "LIC-9F2A...",
|
||||
"customer_id": "CUST-ACME",
|
||||
"plan": "pro",
|
||||
"policy_digest": "sha256:...", // optional for final reports
|
||||
"views": ["inventory", "usage"],
|
||||
"created": "2025-10-17T12:34:56Z"
|
||||
},
|
||||
"scannerImageDigest": "sha256:sc-web-or-worker-digest",
|
||||
"poe": {
|
||||
"format": "jwt", // or "mtls"
|
||||
"value": "eyJhbGciOi..." // PoE JWT when not using mTLS PoE
|
||||
},
|
||||
"options": {
|
||||
"signingMode": "keyless", // "keyless" | "kms"
|
||||
"expirySeconds": 600, // cert lifetime hint (keyless)
|
||||
"returnBundle": "dsse+cert" // dsse (default) | dsse+cert
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Response 200:
|
||||
|
||||
```json
|
||||
{
|
||||
"bundle": {
|
||||
"dsse": { "payloadType": "application/vnd.in-toto+json", "payload": "<base64>", "signatures": [ ... ] },
|
||||
"certificateChain": [ "-----BEGIN CERTIFICATE-----...", "... root ..." ],
|
||||
"mode": "keyless",
|
||||
"signingIdentity": { "issuer": "https://fulcio.internal", "san": "urn:stellaops:signer", "certExpiry": "2025-10-17T12:44:56Z" }
|
||||
},
|
||||
"policy": { "plan": "pro", "maxArtifactBytes": 104857600, "qpsRemaining": 97 },
|
||||
"auditId": "a7c9e3f2-1b7a-4e87-8c3a-90d7d2c3ad12"
|
||||
}
|
||||
```
|
||||
|
||||
Errors (RFC 7807):
|
||||
|
||||
* `401 invalid_token` (JWT/DPoP/mTLS failure)
|
||||
* `403 entitlement_denied` (PoE invalid/revoked/expired; release year mismatch)
|
||||
* `403 release_untrusted` (scanner image not Stella‑signed)
|
||||
* `429 plan_throttled` (license plan caps)
|
||||
* `413 artifact_too_large` (size cap)
|
||||
* `400 invalid_request` (schema/predicate/type invalid)
|
||||
* `500 signing_unavailable` (Fulcio/KMS outage)
|
||||
|
||||
### 3.2 `GET /verify/referrers?imageDigest=<sha256>`
|
||||
|
||||
Checks whether the **image** at digest is signed by **Stella Ops release key**.
|
||||
|
||||
Response:
|
||||
|
||||
```json
|
||||
{ "trusted": true, "signatures": [ { "type": "cosign", "digest": "sha256:...", "signedBy": "StellaOps Release 2027 Q2" } ] }
|
||||
```
|
||||
|
||||
> **Note:** This endpoint is also used internally by Signer before issuing signatures.
|
||||
|
||||
---
|
||||
|
||||
## 4) Validation pipeline (hot path)
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
participant Client as Scanner.WebService
|
||||
participant Auth as Authority (OIDC)
|
||||
participant Sign as Signer
|
||||
participant Lic as Licensing Service (cloud)
|
||||
participant Reg as OCI Registry (Referrers)
|
||||
participant Ful as Fulcio/KMS
|
||||
|
||||
Client->>Sign: POST /sign/dsse (OpTok + DPoP/mTLS, PoE, request)
|
||||
Note over Sign: 1) Validate OpTok, audience, scope, DPoP/mTLS binding
|
||||
Sign->>Lic: /license/introspect(PoE)
|
||||
Lic-->>Sign: { active, claims: {license_id, plan, valid_release_year, max_version}, exp }
|
||||
Note over Sign: 2) Enforce plan/version window and revocation
|
||||
|
||||
Sign->>Reg: Verify scannerImageDigest signed (Referrers + cosign)
|
||||
Reg-->>Sign: OK with signer identity
|
||||
Note over Sign: 3) Enforce release integrity
|
||||
|
||||
Note over Sign: 4) Enforce quotas (QPS/concurrency/size)
|
||||
Sign->>Ful: Mint cert (keyless) or sign via KMS
|
||||
Ful-->>Sign: Cert or signature
|
||||
|
||||
Sign-->>Client: DSSE bundle (+cert chain), policy counters, auditId
|
||||
```
|
||||
|
||||
**DPoP nonce dance (when enabled for high‑value ops):**
|
||||
|
||||
* If DPoP proof lacks a valid nonce, Signer replies `401` with `WWW-Authenticate: DPoP error="use_dpop_nonce", dpop_nonce="<nonce>"`.
|
||||
* Client retries with new proof including the nonce; Signer validates nonce and `jti` uniqueness (Redis TTL cache).
|
||||
|
||||
---
|
||||
|
||||
## 5) Entitlement enforcement (PoE)
|
||||
|
||||
* **Accepted forms**:
|
||||
|
||||
* **mTLS PoE**: client presents a **PoE client cert** at TLS handshake; Signer validates chain to **Licensing CA** (CA bundle configured) and calls `/license/introspect` with cert thumbprint + serial.
|
||||
* **JWT PoE**: `X-PoE` bearer token (DPoP/mTLS‑bound) is validated (sig + `cnf`) locally (Licensing JWKS) and then **introspected** for status and claims.
|
||||
|
||||
* **Claims required**:
|
||||
|
||||
* `license_id`, `plan` (free|pro|enterprise|gov), `valid_release_year`, `max_version`, `exp`.
|
||||
* Optional: `tenant_id`, `customer_id`, `entitlements[]`.
|
||||
|
||||
* **Enforcements**:
|
||||
|
||||
* Reject if **revoked**, **expired**, **plan mismatch** or **release outside window** (`stellaops_version` in predicate exceeds `max_version` or release date beyond `valid_release_year`).
|
||||
* Apply plan **throttles** (QPS/concurrency/artifact bytes) via token‑bucket in Redis keyed by `license_id`.
|
||||
|
||||
---
|
||||
|
||||
## 6) Release integrity (scanner provenance)
|
||||
|
||||
* **Input**: `scannerImageDigest` representing the actual Scanner component that produced the artifact.
|
||||
|
||||
* **Check**:
|
||||
|
||||
1. Use **OCI Referrers API** to enumerate signatures of that digest.
|
||||
2. Verify **cosign** signatures against the configured **Stella Ops Release** keyring (keyless Fulcio roots *or* keyful public keys).
|
||||
3. Optionally require Rekor inclusion for those signatures.
|
||||
|
||||
* **Policy**:
|
||||
|
||||
* If not signed by an authorized **Stella Ops Release** identity → **deny**.
|
||||
* If signed but **release year** > PoE `valid_release_year` → **deny**.
|
||||
|
||||
* **Cache**: LRU of digest → verification result (TTL 10–30 min) to avoid registry thrash.
|
||||
|
||||
---
|
||||
|
||||
## 7) Signing modes
|
||||
|
||||
### 7.1 Keyless (default; Sigstore Fulcio)
|
||||
|
||||
* Signer authenticates to **Fulcio** using its on‑prem OIDC identity (client credentials) and requests a **short‑lived cert** (5–10 min).
|
||||
* Generates **ephemeral keypair**, gets cert for the public key, signs DSSE with the **private key**.
|
||||
* DSSE **bundle** includes **certificate chain**; verifiers validate to Fulcio root.
|
||||
|
||||
### 7.2 Keyful (optional; KMS/HSM)
|
||||
|
||||
* Signer uses a configured **KMS** key (AWS KMS, GCP KMS, Azure Key Vault, Vault Transit, or HSM).
|
||||
* DSSE bundle includes **key metadata** (kid, cert chain if x509).
|
||||
* Recommended for FIPS/sovereign environments.
|
||||
|
||||
---
|
||||
|
||||
## 8) Predicates & schema
|
||||
|
||||
Supported **predicate types** (extensible):
|
||||
|
||||
* `https://stella-ops.org/attestations/sbom/1` (SBOM emissions)
|
||||
* `https://stella-ops.org/attestations/report/1` (final PASS/FAIL reports)
|
||||
* `https://stella-ops.org/attestations/vex-export/1` (Vexer exports; optional)
|
||||
|
||||
**Validation**:
|
||||
|
||||
* JSON‑Schema per predicate type; **canonical property order**.
|
||||
* `subject[*].digest` must include `sha256`.
|
||||
* `predicate.stellaops_version` must parse and match policy windows.
|
||||
|
||||
---
|
||||
|
||||
## 9) Quotas & throttling
|
||||
|
||||
Per `license_id` (from PoE):
|
||||
|
||||
* **QPS** (token bucket), **concurrency** (semaphore), **artifact bytes** (sliding window).
|
||||
* On exceed → `429 plan_throttled` with `Retry-After`.
|
||||
* Free/community plan may also receive **randomized delay** to disincentivize farmed signing.
|
||||
|
||||
---
|
||||
|
||||
## 10) Storage & caches
|
||||
|
||||
* **Redis**:
|
||||
|
||||
* DPoP nonce & `jti` replay cache (TTL ≤ 10 min).
|
||||
* PoE introspection cache (short TTL, e.g., 60–120 s).
|
||||
* Release‑verify cache (`scannerImageDigest` → { trusted, ts }).
|
||||
|
||||
* **Audit store** (Mongo or Postgres): `signer.audit_events`
|
||||
|
||||
```
|
||||
{ _id, ts, tenantId, installationId, licenseId, customerId,
|
||||
plan, actor{sub,cnf}, request{predicateType, subjectSha256[], imageDigest},
|
||||
poe{type, thumbprint|jwtKid, exp, introspectSnapshot},
|
||||
release{digest, signerId, policy},
|
||||
mode: "keyless"|"kms",
|
||||
result: "success"|"deny:<reason>"|"error:<reason>",
|
||||
bundleSha256? }
|
||||
```
|
||||
|
||||
* **Config**: Stella Ops release signing keyring, Fulcio roots, Licensing CA bundle.
|
||||
|
||||
---
|
||||
|
||||
## 11) Security & privacy
|
||||
|
||||
* **mTLS** on all Signer endpoints.
|
||||
* **No bearer fallbacks** — DPoP/mTLS enforced for `aud=signer`.
|
||||
* **PoE** is never persisted beyond audit snapshots (minimized fields).
|
||||
* **Secrets**: no long‑lived private keys on disk (keyless) or handled via KMS APIs.
|
||||
* **Input hardening**: schema‑validate predicates; cap payload sizes; zstd/gzip decompression bombs guarded.
|
||||
* **Logging**: redact PoE JWTs, access tokens, DPoP proofs; log only hashes and identifiers.
|
||||
|
||||
---
|
||||
|
||||
## 12) Metrics & observability
|
||||
|
||||
* `signer.requests_total{result}`
|
||||
* `signer.latency_seconds{stage=auth|introspect|release_verify|sign}`
|
||||
* `signer.poe_failures_total{reason}`
|
||||
* `signer.release_verify_failures_total{reason}`
|
||||
* `signer.plan_throttle_total{license_id}`
|
||||
* `signer.bundle_bytes_total`
|
||||
* `signer.keyless_certs_issued_total` / `signer.kms_sign_total`
|
||||
* OTEL traces across stages; correlation id (`auditId`) returned to client.
|
||||
|
||||
---
|
||||
|
||||
## 13) Configuration (YAML)
|
||||
|
||||
```yaml
|
||||
signer:
|
||||
listen: "https://0.0.0.0:8443"
|
||||
authority:
|
||||
issuer: "https://authority.internal"
|
||||
jwksUrl: "https://authority.internal/jwks"
|
||||
require: "dpop" # "dpop" | "mtls"
|
||||
poe:
|
||||
mode: "both" # "jwt" | "mtls" | "both"
|
||||
licensing:
|
||||
introspectUrl: "https://www.stella-ops.org/api/v1/license/introspect"
|
||||
caBundle: "/etc/ssl/licensing-ca.pem"
|
||||
cacheTtlSeconds: 90
|
||||
release:
|
||||
referrers:
|
||||
allowRekorVerified: true
|
||||
keyrings:
|
||||
- type: "cosign-keyless"
|
||||
fulcioRoots: ["/etc/fulcio/root.pem"]
|
||||
identities:
|
||||
- san: "mailto:release@stella-ops.org"
|
||||
- san: "https://sigstore.dev/oidc/stellaops"
|
||||
signing:
|
||||
mode: "keyless" # "keyless" | "kms"
|
||||
fulcio:
|
||||
issuer: "https://fulcio.internal"
|
||||
oidcClientId: "signer"
|
||||
oidcClientSecretRef: "env:FULCIO_CLIENT_SECRET"
|
||||
certTtlSeconds: 600
|
||||
kms:
|
||||
provider: "aws-kms"
|
||||
keyId: "arn:aws:kms:...:key/..."
|
||||
quotas:
|
||||
default:
|
||||
qps: 100
|
||||
concurrency: 20
|
||||
maxArtifactBytes: 104857600
|
||||
free:
|
||||
qps: 5
|
||||
concurrency: 1
|
||||
maxArtifactBytes: 1048576
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 14) Testing matrix
|
||||
|
||||
* **Auth & DPoP**: bad `aud`, wrong `jkt`, replayed `jti`, missing nonce, mTLS mismatch.
|
||||
* **PoE**: expired, revoked, plan mismatch, release year gate, max_version gate.
|
||||
* **Release verify**: unsigned digest, wrong signer, Rekor‑absent (when required), referrers unreachable.
|
||||
* **Signing**: Fulcio outage; KMS timeouts; bundle correctness (verifier harness).
|
||||
* **Quotas**: burst above QPS, artifact over size, concurrency overflow.
|
||||
* **Schema**: invalid predicate types/required fields.
|
||||
* **Determinism**: same request → identical DSSE (aside from cert validity period).
|
||||
* **Perf**: P95 end‑to‑end under 120 ms with caches warm (excluding network to Fulcio).
|
||||
|
||||
---
|
||||
|
||||
## 15) Failure modes & responses
|
||||
|
||||
| Failure | HTTP | Problem type | Notes |
|
||||
| ----------------------- | ---- | --------------------- | -------------------------------------------- |
|
||||
| Invalid OpTok / DPoP | 401 | `invalid_token` | `WWW-Authenticate` with DPoP nonce if needed |
|
||||
| PoE invalid/revoked | 403 | `entitlement_denied` | Include `license_id` (hashed) and reason |
|
||||
| Scanner image untrusted | 403 | `release_untrusted` | Include digest and required identity |
|
||||
| Plan throttle | 429 | `plan_throttled` | Include limits and `Retry-After` |
|
||||
| Artifact too large | 413 | `artifact_too_large` | Include cap |
|
||||
| Fulcio/KMS down | 503 | `signing_unavailable` | Retry‑After with jitter |
|
||||
|
||||
---
|
||||
|
||||
## 16) Deployment & HA
|
||||
|
||||
* Run ≥ 2 replicas; front with L7 LB; **sticky** not required.
|
||||
* Redis for replay/quota caches (HA).
|
||||
* Audit sink (Mongo/Postgres) in primary region; asynchronous write with local fallback buffer.
|
||||
* Fulcio/KMS clients configured with retries/backoff; circuit breakers.
|
||||
|
||||
---
|
||||
|
||||
## 17) Implementation notes
|
||||
|
||||
* **.NET 10** minimal API + Kestrel mTLS; custom DPoP middleware; JWT/JWKS cache.
|
||||
* **Cosign verification** via sigstore libraries; Referrers queries over registry API with retries.
|
||||
* **DSSE** via in‑toto libs; canonical JSON writer for predicates.
|
||||
* **Backpressure** paths: refuse at auth/quota stages before any expensive network calls.
|
||||
|
||||
---
|
||||
|
||||
## 18) Examples (wire)
|
||||
|
||||
**Request (free plan; expect throttle if burst):**
|
||||
|
||||
```http
|
||||
POST /api/v1/signer/sign/dsse HTTP/1.1
|
||||
Authorization: DPoP <JWT>
|
||||
DPoP: <proof>
|
||||
Content-Type: application/json
|
||||
|
||||
```
|
||||
|
||||
**Error (release untrusted):**
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "https://stella-ops.org/problems/release_untrusted",
|
||||
"title": "Scanner image not signed by StellaOps",
|
||||
"status": 403,
|
||||
"detail": "sha256:abcd... not in trusted keyring",
|
||||
"instance": "urn:audit:a7c9e3f2-..."
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 19) Roadmap
|
||||
|
||||
* **Key Transparency**: optional publication of Signer’s *own* certs to a KT log.
|
||||
* **Attested Build**: SLSA‑style provenance for Signer container itself, checked at startup.
|
||||
* **FIPS mode**: enforce `ES256` + KMS/HSM only; disallow Ed25519.
|
||||
* **Dual attestation**: optional immediate push to **Attestor** (sync mode) with timeout budget, returning Rekor UUID inline.
|
||||
|
||||
|
||||
342
docs/ARCHITECTURE_UI.md
Normal file
342
docs/ARCHITECTURE_UI.md
Normal file
@@ -0,0 +1,342 @@
|
||||
# component_architecture_web_ui.md — **Stella Ops Web UI** (2025Q4)
|
||||
|
||||
> **Scope.** Implementation‑ready architecture for the **Angular SPA** that operators and developers use to drive Stella Ops. This document defines UX surfaces, module boundaries, data flows, auth, RBAC, real‑time updates, performance targets, i18n/a11y, security headers, testing and deployment. The UI is a *consumer* of backend APIs (Scanner, Policy, Vexer, Feedser, Attestor, Authority) and never performs scanning, merging, or signing on its own.
|
||||
|
||||
---
|
||||
|
||||
## 0) Mission & non‑goals
|
||||
|
||||
**Mission.** Provide a **fast, explainable** console for:
|
||||
|
||||
* Scans (status, SBOMs, diffs, EntryTrace, attestation).
|
||||
* Policy management (rules, exemptions, VEX consumption view).
|
||||
* Vulnerability intel (Feedser status), VEX consensus exploration (Vexer).
|
||||
* Runtime posture (Zastava observer + admission).
|
||||
* Admin operations (tenants, tokens, quotas, licensing posture).
|
||||
|
||||
**Non‑goals.** No client‑side crypto signing; no Docker/CRI access; no direct registry access beyond fetching static assets or OCI referrer summaries exposed by backend.
|
||||
|
||||
---
|
||||
|
||||
## 1) Technology baseline
|
||||
|
||||
* **Framework**: Angular 17+ (Stand‑alone APIs / Signals), TypeScript 5.
|
||||
* **Styling**: Tailwind CSS + headless component patterns; CSS variables for theming.
|
||||
* **Charts**: Lightweight SVG (uPlot or Apache ECharts via on‑demand import).
|
||||
* **State**: Angular **Signals** + `@ngrx/signals` store for cross‑page slices.
|
||||
* **Transport**: `fetch` + RxJS interop; **SSE** (EventSource) for progress streams.
|
||||
* **Build**: Angular CLI + Vite builder.
|
||||
* **Testing**: Jest + Testing Library, Playwright for e2e.
|
||||
* **Packaging**: Containerized NGINX (immutable assets, ETag + content hashing).
|
||||
|
||||
---
|
||||
|
||||
## 2) High‑level module map
|
||||
|
||||
```
|
||||
/app
|
||||
├─ core/ # bootstrap, config, auth, http, error boundary, i18n
|
||||
├─ shared/ # UI kit (tables, code-viewers, badges), pipes
|
||||
├─ dashboard/ # live tiles, fleet KPIs, feed/vex age, queue depth
|
||||
├─ scans/ # scan list, detail, SBOM viewer, diff-by-layer, EntryTrace
|
||||
├─ runtime/ # Zastava posture, drift events, admission decisions
|
||||
├─ policy/ # rules editor (YAML/Rego), exemptions, previews
|
||||
├─ vex/ # VEX explorer (claims, consensus, conflicts)
|
||||
├─ feedser/ # source health, export cursors, rebuild/export triggers
|
||||
├─ attest/ # attestation proofs, verification bundles, Rekor links
|
||||
├─ admin/ # tenants, roles, clients, quotas, licensing posture
|
||||
└─ plugins/ # route plug-ins (lazy remote modules, governed)
|
||||
```
|
||||
|
||||
Each feature folder builds as a **standalone route** (lazy loaded). All HTTP shapes live in `core/api/` clients with shared DTOs.
|
||||
|
||||
---
|
||||
|
||||
## 3) Navigation & key views
|
||||
|
||||
### 3.1 Dashboard
|
||||
|
||||
* **Tiles**: “New criticals (24h)”, “VEX suppressions applied”, “Attested SBOMs (7d)”, “Feed age per provider”, “Scanner queue depth”, “Admission events”.
|
||||
* **Trends**: sparkline for vulns/day, pass/fail rates, attestation throughput.
|
||||
|
||||
### 3.2 Scans
|
||||
|
||||
* **Scan list** with status, image digest, repo, time, artifacts, attestation badge.
|
||||
* **Scan detail**:
|
||||
|
||||
* **SBOM viewer**: Inventory/Usage toggle; component table (virtualized), filters by package type, severity, source.
|
||||
* **Diff by layer**: A→B change grid (added/removed/upgraded), grouped by introducing/removing layer; tooltips show provenance and links to layer SBOM fragment.
|
||||
* **EntryTrace**: shell chain with file:line breadcrumbs; jump‑to source viewer (read‑only, hexdump fallback).
|
||||
* **Attestation**: Rekor UUID, index, inclusion proof; **Verify** button calls Attestor `/verify`.
|
||||
* **Export**: download buttons (CycloneDX JSON, Protobuf, SPDX JSON); size shown; SHA‑256 inline.
|
||||
|
||||
### 3.3 Runtime (Zastava)
|
||||
|
||||
* **Observer timeline**: container start/stop, drift, policy violations; faceted by namespace/owner.
|
||||
* **Live process view**: top N processes, loaded libs summary vs Usage SBOM.
|
||||
* **Admission decisions**: per‑namespace rules, allow/deny events, cache TTL, reasons.
|
||||
|
||||
### 3.4 Policy
|
||||
|
||||
* **Policy bundles**: active vs staged; diff viewer with change summary.
|
||||
* **Editors**:
|
||||
|
||||
* YAML rules (ignore lists, thresholds, vendor precedence overrides).
|
||||
* Rego blocks (advanced gates) with **WASM** preview evaluator (client‑side sandbox) for “preview” against sample SBOMs.
|
||||
* **VEX inclusion controls**: weight sliders (visualization only), provider allow/deny toggles.
|
||||
* **Preview**: select SBOM (or image digest) → show verdict under staged policy.
|
||||
|
||||
### 3.5 Vexer
|
||||
|
||||
* **Claims explorer**: search by vulnId/productKey/provider; show raw claim (status, justification, evidence).
|
||||
* **Consensus view**: rollup per (vuln, product) with accepted/rejected sources, weights, timestamps.
|
||||
* **Conflicts**: grid of top conflicts; filters for justification gates failed.
|
||||
|
||||
### 3.6 Feedser
|
||||
|
||||
* **Sources** table: staleness, last run, errors.
|
||||
* **Advisory search**: by CVE/alias; show normalized affected ranges.
|
||||
* **Exports**: trigger full/delta JSON/Trivy DB; show manifest digests and Rekor link if attested.
|
||||
|
||||
### 3.7 Attest
|
||||
|
||||
* **Proofs list**: last 7 days Rekor entries; filter by kind (sbom/report/vex).
|
||||
* **Verification**: paste UUID or upload bundle → verify; result with explanations (chain, Merkle path).
|
||||
|
||||
### 3.8 Admin
|
||||
|
||||
* **Tenants/Installations**: view/edit, isolation hints.
|
||||
* **Clients & roles**: Authority clients, role→scope mapping, rotation hints.
|
||||
* **Quotas**: per license plan, counters, throttle events.
|
||||
* **Licensing posture**: last PoE introspection snapshot (redacted), release window.
|
||||
|
||||
---
|
||||
|
||||
## 4) Auth, sessions & RBAC
|
||||
|
||||
### 4.1 OIDC flow
|
||||
|
||||
* **Authorization Code + PKCE** to **Authority**.
|
||||
* **ID Token** for UX identity; **Access Token** (OpTok) for APIs (2–5 min TTL).
|
||||
* **DPoP (browser)**: generate ephemeral **WebCrypto** keypair; store public JWK in memory, private key in **IndexedDB** (non‑exportable if platform allows). Access token includes `cnf.jkt`; each API call adds `DPoP` proof; handle nonce challenges automatically.
|
||||
* **Refresh**: optional DPoP‑bound refresh tokens with rotation; otherwise silent renew.
|
||||
|
||||
### 4.2 RBAC
|
||||
|
||||
* Roles (`ui.read`, `ui.admin`, plus service roles) are embedded in ID token or fetched via `/me` endpoint.
|
||||
* **Route guards** enforce access; **feature flags** hide admin pages for non‑admins.
|
||||
|
||||
### 4.3 Session storage
|
||||
|
||||
* Access tokens & refresh metadata in memory; persist **only** minimal session (subject, expiries) in `sessionStorage`. Never persist raw JWTs to `localStorage`. Use **SameSite=Lax** cookies for anti‑CSRF if cookies are required (prefer pure bearer headers + DPoP).
|
||||
|
||||
---
|
||||
|
||||
## 5) HTTP layer & API clients
|
||||
|
||||
* **`core/http/api-client.ts`** centralizes:
|
||||
|
||||
* Base URLs (Scanner, Vexer, Feedser, Attestor).
|
||||
* **Retry** policies on idempotent GETs (backoff + jitter).
|
||||
* **Problem+JSON** parser → uniform error toasts with correlation ID.
|
||||
* **SSE** helper (EventSource) with auto‑reconnect & backpressure.
|
||||
* **DPoP** injector & nonce handling.
|
||||
|
||||
* Typed API clients (DTOs in `core/api/models.ts`):
|
||||
|
||||
* `ScannerApi`, `PolicyApi`, `VexerApi`, `FeedserApi`, `AttestorApi`, `AuthorityApi`.
|
||||
|
||||
**DTO examples (abbrev):**
|
||||
|
||||
```ts
|
||||
export type ImageDigest = `sha256:${string}`;
|
||||
export interface ScanSummary {
|
||||
imageDigest: ImageDigest; createdAt: string;
|
||||
artifacts: { view: 'inventory'|'usage'; format: 'cdx-json'|'cdx-pb'|'spdx-json'; sha256: string; size: number }[];
|
||||
status: 'queued'|'running'|'completed'|'error';
|
||||
rekor?: { uuid: string; index?: number; url?: string };
|
||||
}
|
||||
export interface DiffEntry {
|
||||
key: string; change: 'added'|'removed'|'upgraded'|'downgraded';
|
||||
fromVersion?: string; toVersion?: string; layer: string; usedByEntrypoint?: boolean;
|
||||
}
|
||||
export interface VexConsensus {
|
||||
vulnId: string; productKey: string; rollupStatus: 'affected'|'not_affected'|'fixed'|'under_investigation';
|
||||
sources: { providerId: string; status: string; weight: number; accepted: boolean; reason: string }[];
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6) State, caching & real‑time
|
||||
|
||||
* **Per‑page stores** (Signals) for list filters, pagination, and selected entities.
|
||||
* **Normalized caches** keyed by `(imageDigest, view, format)`; artifacts are downloaded via pre‑signed URLs from Scanner and streamed; SHA‑256 verified client‑side before exposing “verified” badge.
|
||||
* **SSE channels**:
|
||||
|
||||
* `/scans/{id}/events` → progress log.
|
||||
* `/runtime/events/stream` (optional) → live drift/admission feed (rate‑limited).
|
||||
* **Cache invalidation** on job completion or explicit “refresh”.
|
||||
|
||||
---
|
||||
|
||||
## 7) SBOM viewing & diff UX
|
||||
|
||||
* **Huge tables** rendered with **virtual scrolling** (CDK Virtual Scroll); sort/filter performed client‑side for ≤ 20k rows; beyond that, server‑side queries via BOM‑Index endpoints.
|
||||
* **Component row** shows purl, version, origin (OS pkg / metadata / linker / attested), licenses, and **used** badge (Usage view).
|
||||
* **Diff**: compact heatmap per layer; clicking opens a right‑pane with evidence: introducing paths, file hashes, VEX notes (from Vexer consensus) and links to advisories (Feedser).
|
||||
|
||||
---
|
||||
|
||||
## 8) Policy editor & VEX integration
|
||||
|
||||
* **YAML editor**: Monaco‑based with schema hints; previews show which rules matched.
|
||||
* **Rego editor**: Monaco with WASM‑OPA sandbox for read‑only evaluation on sample SBOMs.
|
||||
* **VEX toggles**: per‑provider enable/disable; “explain” drawer shows why a claim was accepted/rejected (justification gate, signature state, weight).
|
||||
* **Staged → Active** promotion is a two‑step flow with confirmation and automatic policy digest computation.
|
||||
|
||||
---
|
||||
|
||||
## 9) Accessibility, i18n & theming
|
||||
|
||||
* **A11y**: WCAG 2.2 AA; keyboard navigation, focus management, ARIA roles; color‑contrast tokens verified by unit tests.
|
||||
* **I18n**: Angular i18n + runtime translation loader (`/locales/{lang}.json`); dates/numbers localized via `Intl`.
|
||||
* **Languages**: English default; Bulgarian, German, Japanese as initial additions.
|
||||
* **Theming**: dark/light via CSS variables; persisted in `prefers-color-scheme` aware store.
|
||||
|
||||
---
|
||||
|
||||
## 10) Performance budgets
|
||||
|
||||
* **TTI** ≤ 1.5 s on 4G/slow CPU (first visit), ≤ 0.6 s repeat (HTTP/2, cached).
|
||||
* **JS** initial < 300 KB gz (lazy routes).
|
||||
* **SBOM list**: render 10k rows in < 70 ms with virtualization; filter in < 150 ms.
|
||||
* **Diff view**: compute client‑side grouping for 5k changes in < 120 ms.
|
||||
|
||||
Techniques: route‑level code splitting; `ChangeDetectionStrategy.OnPush`; Signals; server compression (zstd/gzip), immutable assets with long max‑age (cache busting via hashes).
|
||||
|
||||
---
|
||||
|
||||
## 11) Security headers & CSP
|
||||
|
||||
* **CSP**: `default-src 'self'; connect-src 'self' https://*.internal; img-src 'self' data:; script-src 'self'; style-src 'self' 'unsafe-inline'; frame-ancestors 'none';`
|
||||
* **HSTS** enabled at gateway.
|
||||
* **Referrer Policy**: `no-referrer`.
|
||||
* **X‑Frame‑Options**: `DENY`.
|
||||
* **COOP/COEP** to enable faster cross‑origin isolation (for WASM OPA).
|
||||
* **Subresource Integrity (SRI)** for third‑party fonts (minimize third‑party).
|
||||
|
||||
---
|
||||
|
||||
## 12) Error handling & UX hygiene
|
||||
|
||||
* **Global error boundary** surfaces Problem+JSON `title/detail/instance` with correlation ID.
|
||||
* **Retry toast** for 429 (quota throttles) with backoff timer.
|
||||
* **Auth expiry**: pre‑emptive refresh; unobtrusive banner when < 60 s TTL; re‑login modal if refresh fails.
|
||||
* **Network down**: offline banner with queued actions (idempotent resubmits).
|
||||
* **File verify**: show SHA‑256 mismatch warnings if artifact altered in transit.
|
||||
|
||||
---
|
||||
|
||||
## 13) Observability
|
||||
|
||||
* **Front‑end telemetry** (OpenTelemetry Web): route timings, API latency by service, error counts; sampled to 1–5% and shipped to backend OTLP endpoint.
|
||||
* **User actions** logged anonymously (no PII): “policy promote”, “scan export”, “attest verify”.
|
||||
* **Metrics dash** in admin shows SLOs and recent front‑end errors.
|
||||
|
||||
---
|
||||
|
||||
## 14) Testing strategy
|
||||
|
||||
* **Unit**: pure component logic via Jest + Testing Library (no TestBed when possible).
|
||||
* **Component harness** for table, code viewer, diff heatmap.
|
||||
* **Contract tests**: OpenAPI schemas pulled at build time; DTOs validated; breaking changes fail CI.
|
||||
* **e2e**: Playwright scenarios (login, scan detail, diff, policy edit, admit deny).
|
||||
* **A11y**: axe-core CI checks; color‑contrast lints.
|
||||
* **i18n**: key coverage tests (no missing translations in supported locales).
|
||||
|
||||
---
|
||||
|
||||
## 15) Deployment & ops
|
||||
|
||||
* **Container**: `stellaops/web-ui:<ver>-<rev>`; NGINX with `gzip_static` + brotli; immutable assets under `/static/<hash>/…`.
|
||||
* **Config**: `/config.json` served by gateway (injected at runtime): API base URLs, authority issuer, telemetry sampling.
|
||||
* **Version banner**: footer shows UI & backend versions; warns on major mismatches.
|
||||
* **CDN** (optional): cache static bundle; APIs stay behind internal gateway.
|
||||
* **Feature flags**: environment gates (staged policies, eBPF runtime) readable from config.
|
||||
|
||||
---
|
||||
|
||||
## 16) Plugin system (route plug‑ins)
|
||||
|
||||
* **Manifest**: Backend provides a signed plug‑in manifest with remote module URLs and **cosign signature** per JS bundle.
|
||||
* **Loader**: dynamic import with **SRI** and signature verification (WebCrypto).
|
||||
* **Sandbox**: plug‑ins are routed modules receiving a limited **UI SDK** (navigation, theme, API gateway). No direct token access; API calls proxied through the UI SDK which enforces RBAC.
|
||||
* **Examples**: custom reports, vendor dashboards, regulated TLS config UIs.
|
||||
|
||||
---
|
||||
|
||||
## 17) Wire sequences (representative)
|
||||
|
||||
**A) View scan progress**
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
participant UI
|
||||
participant Auth as Authority
|
||||
participant SW as Scanner.WebService
|
||||
|
||||
UI->>Auth: /authorize (PKCE)
|
||||
Auth-->>UI: code → token (DPoP-bound)
|
||||
UI->>SW: GET /scans/{id} (Authorization+DPoP)
|
||||
SW-->>UI: { status: running }
|
||||
UI->>SW: (SSE) GET /scans/{id}/events
|
||||
SW-->>UI: progress events …
|
||||
SW-->>UI: terminal event { status: completed, artifacts[] }
|
||||
```
|
||||
|
||||
**B) Verify attestation**
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
participant UI
|
||||
participant AT as Attestor
|
||||
UI->>AT: POST /rekor/verify { uuid }
|
||||
AT-->>UI: { ok:true, index, logURL }
|
||||
```
|
||||
|
||||
**C) Promote policy & preview**
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
participant UI
|
||||
participant BE as Scanner.WebService (Policy endpoint)
|
||||
UI->>BE: POST /policy/stage { yaml, rego }
|
||||
BE-->>UI: { policyRevision, diagnostics }
|
||||
UI->>BE: POST /policy/preview { imageDigest, policyRevision }
|
||||
BE-->>UI: { verdict: pass|fail, reasons[] }
|
||||
UI->>BE: POST /policy/promote { policyRevision }
|
||||
BE-->>UI: { ok:true }
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 18) Security hard lines
|
||||
|
||||
* Never store JWTs in `localStorage`.
|
||||
* Enforce DPoP for API calls; if DPoP unsupported for a service, require **SameSite=Lax** cookies with CSRF token header.
|
||||
* Block mixed‑content; only HTTPS origins allowed.
|
||||
* Validate and render only **escaped** user content; code viewer uses safe highlighter.
|
||||
* Downloaded artifacts are treated as **opaque binaries**; no HTML rendering.
|
||||
|
||||
---
|
||||
|
||||
## 19) Roadmap
|
||||
|
||||
* **PWA** offline shell (read‑only) for dashboards and cached scan details.
|
||||
* **SBOM graph** visualization (force‑directed) for small components sets.
|
||||
* **Runtime session replay** (privacy‑safe) to debug operator workflows (opt‑in).
|
||||
* **Assistive wizards** for policy creation with guided templates.
|
||||
@@ -1,85 +1,463 @@
|
||||
# StellaOps Vexer Architecture
|
||||
# component_architecture_vexer.md — **Stella Ops Vexer** (2025Q4)
|
||||
|
||||
Vexer is StellaOps' vulnerability-exploitability (VEX) platform. It ingests VEX statements from multiple providers, normalizes them into canonical claims, projects trust-weighted consensus, and delivers deterministic export artifacts with signed attestations. This document summarizes the target architecture and how the current implementation maps to those goals.
|
||||
> **Scope.** This document specifies the **Vexer** service: its purpose, trust model, data structures, APIs, plug‑in contracts, storage schema, normalization/consensus algorithms, performance budgets, testing matrix, and how it integrates with Scanner, Policy, Feedser, and the attestation chain. It is implementation‑ready.
|
||||
|
||||
## 1. Solution topology
|
||||
---
|
||||
|
||||
| Module | Purpose | Key contracts |
|
||||
| --- | --- | --- |
|
||||
| `StellaOps.Vexer.Core` | Domain models (`VexClaim`, `VexConsensus`, `VexExportManifest`), deterministic JSON helpers, shared abstractions (connectors, exporters, attestations). | `IVexConnector`, `IVexExporter`, `IVexAttestationClient`, `VexCanonicalJsonSerializer` |
|
||||
| `StellaOps.Vexer.Policy` | Loads operator policy (weights, overrides, justification gates) and exposes snapshots for consensus. | `IVexPolicyProvider`, `IVexPolicyEvaluator`, `VexPolicyOptions` |
|
||||
| `StellaOps.Vexer.Storage.Mongo` | Persistence layer for providers, raw docs, claims, consensus, exports, cache. | `IVexRawStore`, `IVexExportStore`, Mongo class maps |
|
||||
| `StellaOps.Vexer.Export` | Orchestrates export pipeline (query signature → cache lookup → snapshot build → attestation handoff). | `IExportEngine`, `IVexExportDataSource` |
|
||||
| `StellaOps.Vexer.Attestation` *(planned)* | Builds in-toto/DSSE envelopes and communicates with Sigstore/Rekor. | `IVexAttestationClient` |
|
||||
| `StellaOps.Vexer.WebService` *(planned)* | Minimal API host for ingest/export endpoints. | `AddVexerWebService()` |
|
||||
| `StellaOps.Vexer.Worker` *(planned)* | Background executor for scheduled pulls, verification, reconciliation, cache GC. | Hosted services |
|
||||
## 0) Mission & role in the platform
|
||||
|
||||
All modules target .NET 10 preview and follow the same deterministic logging and serialization conventions as Feedser.
|
||||
**Mission.** Convert heterogeneous **VEX** statements (OpenVEX, CSAF VEX, CycloneDX VEX; vendor/distro/platform sources) into **canonical, queryable claims**; compute **deterministic consensus** per *(vuln, product)*; preserve **conflicts with provenance**; publish **stable, attestable exports** that the backend uses to suppress non‑exploitable findings, prioritize remaining risk, and explain decisions.
|
||||
|
||||
## 2. Data model
|
||||
**Boundaries.**
|
||||
|
||||
MongoDB acts as the canonical store; collections (with logical responsibilities) are:
|
||||
* Vexer **does not** decide PASS/FAIL. It supplies **evidence** (statuses + justifications + provenance weights).
|
||||
* Vexer preserves **conflicting claims** unchanged; consensus encodes how we would pick, but the raw set is always exportable.
|
||||
* VEX consumption is **backend‑only**: Scanner never applies VEX. The backend’s **Policy Engine** asks Vexer for status evidence and then decides what to show.
|
||||
|
||||
- `vex.providers` – provider metadata, trust tiers, discovery endpoints, and cosign/PGP details.
|
||||
- `vex.raw` – immutable raw documents (CSAF, CycloneDX VEX, OpenVEX, OCI attestations) with digests, retrieval metadata, and signature state.
|
||||
- `vex.claims` – normalized `VexClaim` rows; deduped on `(providerId, vulnId, productKey, docDigest)`.
|
||||
- `vex.consensus` – consensus projections per `(vulnId, productKey)` capturing rollup status, source weights, conflicts, and policy revision.
|
||||
- `vex.exports` – export manifests containing artifact digests, cache metadata, and attestation pointers.
|
||||
- `vex.cache` – index from `querySignature`/`format` to export digest for fast reuse.
|
||||
- `vex.migrations` – tracks applied storage migrations (index bootstrap, future schema updates).
|
||||
---
|
||||
|
||||
GridFS is used for large raw payloads when necessary, and artifact stores (S3/MinIO/file) hold serialized exports referenced by `vex.exports`.
|
||||
## 1) Inputs, outputs & canonical domain
|
||||
|
||||
## 3. Ingestion and reconciliation flow
|
||||
### 1.1 Accepted input formats (ingest)
|
||||
|
||||
1. **Discovery & configuration** – connectors load YAML/JSON settings via `StellaOps.Vexer.Policy` (provider enablement, trust overrides).
|
||||
2. **Fetch** – each `IVexConnector` pulls source windows, writing raw documents through `IVexRawDocumentSink` (Mongo-backed) with dedupe on digest.
|
||||
3. **Verification** – signatures/attestations validated through `IVexSignatureVerifier`; metadata stored alongside raw records.
|
||||
4. **Normalization** – format-specific `IVexNormalizer` instances translate raw payloads to canonical `VexClaim` batches.
|
||||
5. **Consensus** – `VexConsensusResolver` (Core) consumes claims with policy weights supplied by `IVexPolicyEvaluator`, producing deterministic consensus entries and conflict annotations.
|
||||
6. **Export** – query requests pass through `VexExportEngine`, generating `VexExportManifest` instances, caching by `VexQuerySignature`, and emitting artifacts for attestation/signature.
|
||||
7. **Attestation & transparency** *(planned)* – `IVexAttestationClient` signs exports (in-toto/DSSE) and records bundles in Rekor v2.
|
||||
* **OpenVEX** JSON documents (attested or raw).
|
||||
* **CSAF VEX** 2.x (vendor PSIRTs and distros commonly publish CSAF).
|
||||
* **CycloneDX VEX** 1.4+ (standalone VEX or embedded VEX blocks).
|
||||
* **OCI‑attached attestations** (VEX statements shipped as OCI referrers) — optional connectors.
|
||||
|
||||
The Worker coordinates the long-running steps (fetch/verify/normalize/export), while the WebService exposes synchronous APIs for on-demand operations and status lookups.
|
||||
All connectors register **source metadata**: provider identity, trust tier, signature expectations (PGP/cosign/PKI), fetch windows, rate limits, and time anchors.
|
||||
|
||||
## 4. Policy semantics
|
||||
### 1.2 Canonical model (normalized)
|
||||
|
||||
- **Weights** – default tiers (`vendor=1.0`, `distro=0.9`, `platform=0.7`, `hub=0.5`, `attestation=0.6`) loaded via `VexPolicyOptions.Weights`, with per-provider overrides.
|
||||
- **Justification gates** – policy enforces that `not_affected` claims must provide a recognized justification; rejected claims are preserved as conflicts with reason metadata.
|
||||
- **Diagnostics** – policy snapshots carry structured issues for misconfigurations (out-of-range weights, empty overrides) surfaced to operators via logs and future CLI/Web endpoints.
|
||||
Every incoming statement becomes a set of **VexClaim** records:
|
||||
|
||||
Policy snapshots are immutable and versioned so consensus records capture the policy revision used during evaluation.
|
||||
```
|
||||
VexClaim
|
||||
- providerId // 'redhat', 'suse', 'ubuntu', 'github', 'vendorX'
|
||||
- vulnId // 'CVE-2025-12345', 'GHSA-xxxx', canonicalized
|
||||
- productKey // canonical product identity (see §2.2)
|
||||
- status // affected | not_affected | fixed | under_investigation
|
||||
- justification? // for 'not_affected'/'affected' where provided
|
||||
- introducedVersion? // semantics per provider (range or exact)
|
||||
- fixedVersion? // where provided (range or exact)
|
||||
- lastObserved // timestamp from source or fetch time
|
||||
- provenance // doc digest, signature status, fetch URI, line/offset anchors
|
||||
- evidence[] // raw source snippets for explainability
|
||||
- supersedes? // optional cross-doc chain (docDigest → docDigest)
|
||||
```
|
||||
|
||||
## 5. Determinism & caching
|
||||
### 1.3 Exports (consumption)
|
||||
|
||||
- JSON serialization uses `VexCanonicalJsonSerializer`, enforcing property ordering and camelCase naming for reproducible snapshots and test fixtures.
|
||||
- `VexQuerySignature` produces canonical filter/order strings and SHA-256 digests, enabling cache keys shared across services.
|
||||
- Export manifests reuse cached artifacts when the same signature/format is requested unless `ForceRefresh` is explicitly set.
|
||||
- For scorring multiple sources on same VEX topic use - `VEXER_SCORRING.md`
|
||||
* **VexConsensus** per `(vulnId, productKey)` with:
|
||||
|
||||
## 6. Observability & offline posture
|
||||
* `rollupStatus` (after policy weights/justification gates),
|
||||
* `sources[]` (winning + losing claims with weights & reasons),
|
||||
* `policyRevisionId` (identifier of the Vexer policy used),
|
||||
* `consensusDigest` (stable SHA‑256 over canonical JSON).
|
||||
* **Raw claims** export for auditing (unchanged, with provenance).
|
||||
* **Provider snapshots** (per source, last N days) for operator debugging.
|
||||
* **Index** optimized for backend joins: `(productKey, vulnId) → (status, confidence, sourceSet)`.
|
||||
|
||||
- Structured logs (`ILogger`) capture correlation IDs, query signatures, provider IDs, and policy revisions. Metrics/OTel instrumentation will mirror Feedser once tracing hooks are added.
|
||||
- Offline-first: connectors, policy bundles, and export caches can be bundled inside the Offline Kit; no mandatory outbound calls beyond configured provider allowlists.
|
||||
- Operator tooling (CLI/WebService) will expose diagnostics (policy issues, verification failures, cache status) so air-gapped deployments maintain visibility without external telemetry.
|
||||
All exports are **deterministic**, and (optionally) **attested** via DSSE and logged to Rekor v2.
|
||||
|
||||
## 7. Roadmap highlights
|
||||
---
|
||||
|
||||
- Complete storage mappings for providers/consensus/cache and add migrations/indices per collection.
|
||||
- Implement Rekor/in-toto attestation clients and wire export engine to produce signed bundles.
|
||||
- Build WebService endpoints (`/vexer/status`, `/vexer/claims`, `/vexer/exports`) plus CLI verbs mirroring Feedser patterns.
|
||||
- Provide CSAF, CycloneDX VEX, and OpenVEX normalizers along with vendor-specific connectors (Red Hat, Cisco, SUSE, MSRC, Oracle, Ubuntu, OCI attestation).
|
||||
- Extend policy diagnostics with schema validation, change tracking, and operator-facing diff reports.
|
||||
- Mongo bootstrapper runs ordered migrations (`vex.migrations`) to ensure indexes for raw documents, providers, consensus snapshots, exports, and cache entries.
|
||||
## 2) Identity model — products & joins
|
||||
|
||||
## Appendix A – Policy diagnostics workflow
|
||||
### 2.1 Vuln identity
|
||||
|
||||
- `StellaOps.Vexer.Policy` now exposes `IVexPolicyDiagnostics`, producing deterministic diagnostics reports with timestamp, severity counts, active provider overrides, and the full issue list surfaced by `IVexPolicyProvider`.
|
||||
- CLI/WebService layers should call `IVexPolicyDiagnostics.GetDiagnostics()` to display operator-friendly summaries (`vexer policy diagnostics` and `/vexer/policy/diagnostics` are the planned entry points).
|
||||
- Recommendations in the report guide operators to resolve blocking errors, review warnings, and audit override usage before consensus runs—embed them directly in UX copy instead of re-deriving logic.
|
||||
- Export/consensus telemetry should log the diagnostic `Version` alongside `policyRevisionId` so dashboards can correlate policy changes with consensus decisions.
|
||||
- Offline installations can persist the diagnostics report (JSON) in the Offline Kit to document policy headroom during audits; the output is deterministic and diff-friendly.
|
||||
- Use `VexPolicyBinder` when ingesting operator-supplied YAML/JSON bundles; it normalizes weight/override values, reports deterministic issues, and returns the consensus-ready `VexConsensusPolicyOptions` used by `VexPolicyProvider`.
|
||||
- Reload telemetry emits `vex.policy.reloads` (tags: `revision`, `version`, `issues`) whenever a new digest is observed—feed this into dashboards to correlate policy changes with consensus outcomes.
|
||||
* Accepts **CVE**, **GHSA**, vendor IDs (MSRC, RHSA…), distro IDs (DSA/USN/RHSA…) — normalized to `vulnId` with alias sets.
|
||||
* **Alias graph** maintained (from Feedser) to map vendor/distro IDs → CVE (primary) and to **GHSA** where applicable.
|
||||
|
||||
### 2.2 Product identity (`productKey`)
|
||||
|
||||
* **Primary:** `purl` (Package URL).
|
||||
* **Secondary links:** `cpe`, **OS package NVRA/EVR**, NuGet/Maven/Golang identity, and **OS package name** when purl unavailable.
|
||||
* **Fallback:** `oci:<registry>/<repo>@<digest>` for image‑level VEX.
|
||||
* **Special cases:** kernel modules, firmware, platforms → provider‑specific mapping helpers (connector captures provider’s product taxonomy → canonical `productKey`).
|
||||
|
||||
> Vexer does not invent identities. If a provider cannot be mapped to purl/CPE/NVRA deterministically, we keep the native **product string** and mark the claim as **non‑joinable**; the backend will ignore it unless a policy explicitly whitelists that provider mapping.
|
||||
|
||||
---
|
||||
|
||||
## 3) Storage schema (MongoDB)
|
||||
|
||||
Database: `vexer`
|
||||
|
||||
### 3.1 Collections
|
||||
|
||||
**`vex.providers`**
|
||||
|
||||
```
|
||||
_id: providerId
|
||||
name, homepage, contact
|
||||
trustTier: enum {vendor, distro, platform, hub, attestation}
|
||||
signaturePolicy: { type: pgp|cosign|x509|none, keys[], certs[], cosignKeylessRoots[] }
|
||||
fetch: { baseUrl, kind: http|oci|file, rateLimit, etagSupport, windowDays }
|
||||
enabled: bool
|
||||
createdAt, modifiedAt
|
||||
```
|
||||
|
||||
**`vex.raw`** (immutable raw documents)
|
||||
|
||||
```
|
||||
_id: sha256(doc bytes)
|
||||
providerId
|
||||
uri
|
||||
ingestedAt
|
||||
contentType
|
||||
sig: { verified: bool, method: pgp|cosign|x509|none, keyId|certSubject, bundle? }
|
||||
payload: GridFS pointer (if large)
|
||||
disposition: kept|replaced|superseded
|
||||
correlation: { replaces?: sha256, replacedBy?: sha256 }
|
||||
```
|
||||
|
||||
**`vex.claims`** (normalized rows; dedupe on providerId+vulnId+productKey+docDigest)
|
||||
|
||||
```
|
||||
_id
|
||||
providerId
|
||||
vulnId
|
||||
productKey
|
||||
status
|
||||
justification?
|
||||
introducedVersion?
|
||||
fixedVersion?
|
||||
lastObserved
|
||||
docDigest
|
||||
provenance { uri, line?, pointer?, signatureState }
|
||||
evidence[] { key, value, locator }
|
||||
indices:
|
||||
- {vulnId:1, productKey:1}
|
||||
- {providerId:1, lastObserved:-1}
|
||||
- {status:1}
|
||||
- text index (optional) on evidence.value for debugging
|
||||
```
|
||||
|
||||
**`vex.consensus`** (rollups)
|
||||
|
||||
```
|
||||
_id: sha256(canonical(vulnId, productKey, policyRevision))
|
||||
vulnId
|
||||
productKey
|
||||
rollupStatus
|
||||
sources[]: [
|
||||
{ providerId, status, justification?, weight, lastObserved, accepted:bool, reason }
|
||||
]
|
||||
policyRevisionId
|
||||
evaluatedAt
|
||||
consensusDigest // same as _id
|
||||
indices:
|
||||
- {vulnId:1, productKey:1}
|
||||
- {policyRevisionId:1, evaluatedAt:-1}
|
||||
```
|
||||
|
||||
**`vex.exports`** (manifest of emitted artifacts)
|
||||
|
||||
```
|
||||
_id
|
||||
querySignature
|
||||
format: raw|consensus|index
|
||||
artifactSha256
|
||||
rekor { uuid, index, url }?
|
||||
createdAt
|
||||
policyRevisionId
|
||||
cacheable: bool
|
||||
```
|
||||
|
||||
**`vex.cache`**
|
||||
|
||||
```
|
||||
querySignature -> exportId (for fast reuse)
|
||||
ttl, hits
|
||||
```
|
||||
|
||||
**`vex.migrations`**
|
||||
|
||||
* ordered migrations applied at bootstrap to ensure indexes.
|
||||
|
||||
### 3.2 Indexing strategy
|
||||
|
||||
* Hot path queries use exact `(vulnId, productKey)` and time‑bounded windows; compound indexes cover both.
|
||||
* Providers list view by `lastObserved` for monitoring staleness.
|
||||
* `vex.consensus` keyed by `(vulnId, productKey, policyRevision)` for deterministic reuse.
|
||||
|
||||
---
|
||||
|
||||
## 4) Ingestion pipeline
|
||||
|
||||
### 4.1 Connector contract
|
||||
|
||||
```csharp
|
||||
public interface IVexConnector
|
||||
{
|
||||
string ProviderId { get; }
|
||||
Task FetchAsync(VexConnectorContext ctx, CancellationToken ct); // raw docs
|
||||
Task NormalizeAsync(VexConnectorContext ctx, CancellationToken ct); // raw -> VexClaim[]
|
||||
}
|
||||
```
|
||||
|
||||
* **Fetch** must implement: window scheduling, conditional GET (ETag/If‑Modified‑Since), rate limiting, retry/backoff.
|
||||
* **Normalize** parses the format, validates schema, maps product identities deterministically, emits `VexClaim` records with **provenance**.
|
||||
|
||||
### 4.2 Signature verification (per provider)
|
||||
|
||||
* **cosign (keyless or keyful)** for OCI referrers or HTTP‑served JSON with Sigstore bundles.
|
||||
* **PGP** (provider keyrings) for distro/vendor feeds that sign docs.
|
||||
* **x509** (mutual TLS / provider‑pinned certs) where applicable.
|
||||
* Signature state is stored on **vex.raw.sig** and copied into **provenance.signatureState** on claims.
|
||||
|
||||
> Claims from sources failing signature policy are marked `"signatureState.verified=false"` and **policy** can down‑weight or ignore them.
|
||||
|
||||
### 4.3 Time discipline
|
||||
|
||||
* For each doc, prefer **provider’s document timestamp**; if absent, use fetch time.
|
||||
* Claims carry `lastObserved` which drives **tie‑breaking** within equal weight tiers.
|
||||
|
||||
---
|
||||
|
||||
## 5) Normalization: product & status semantics
|
||||
|
||||
### 5.1 Product mapping
|
||||
|
||||
* **purl** first; **cpe** second; OS package NVRA/EVR mapping helpers (distro connectors) produce purls via canonical tables (e.g., rpm→purl:rpm, deb→purl:deb).
|
||||
* Where a provider publishes **platform‑level** VEX (e.g., “RHEL 9 not affected”), connectors expand to known product inventory rules (e.g., map to sets of packages/components shipped in the platform). Expansion tables are versioned and kept per provider; every expansion emits **evidence** indicating the rule applied.
|
||||
* If expansion would be speculative, the claim remains **platform‑scoped** with `productKey="platform:redhat:rhel:9"` and is flagged **non‑joinable**; backend can decide to use platform VEX only when Scanner proves the platform runtime.
|
||||
|
||||
### 5.2 Status + justification mapping
|
||||
|
||||
* Canonical **status**: `affected | not_affected | fixed | under_investigation`.
|
||||
* **Justifications** normalized to a controlled vocabulary (CISA‑aligned), e.g.:
|
||||
|
||||
* `component_not_present`
|
||||
* `vulnerable_code_not_in_execute_path`
|
||||
* `vulnerable_configuration_unused`
|
||||
* `inline_mitigation_applied`
|
||||
* `fix_available` (with `fixedVersion`)
|
||||
* `under_investigation`
|
||||
* Providers with free‑text justifications are mapped by deterministic tables; raw text preserved as `evidence`.
|
||||
|
||||
---
|
||||
|
||||
## 6) Consensus algorithm
|
||||
|
||||
**Goal:** produce a **stable**, explainable `rollupStatus` per `(vulnId, productKey)` given possibly conflicting claims.
|
||||
|
||||
### 6.1 Inputs
|
||||
|
||||
* Set **S** of `VexClaim` for the key.
|
||||
* **Vexer policy snapshot**:
|
||||
|
||||
* **weights** per provider tier and per provider overrides.
|
||||
* **justification gates** (e.g., require justification for `not_affected` to be acceptable).
|
||||
* **minEvidence** rules (e.g., `not_affected` must come from ≥1 vendor or 2 distros).
|
||||
* **signature requirements** (e.g., require verified signature for ‘fixed’ to be considered).
|
||||
|
||||
### 6.2 Steps
|
||||
|
||||
1. **Filter invalid** claims by signature policy & justification gates → set `S'`.
|
||||
2. **Score** each claim:
|
||||
`score = weight(provider) * freshnessFactor(lastObserved)` where freshnessFactor ∈ [0.8, 1.0] for staleness decay (configurable; small effect).
|
||||
3. **Aggregate** scores per status: `W(status) = Σ score(claims with that status)`.
|
||||
4. **Pick** `rollupStatus = argmax_status W(status)`.
|
||||
5. **Tie‑breakers** (in order):
|
||||
|
||||
* Higher **max single** provider score wins (vendor > distro > platform > hub).
|
||||
* More **recent** lastObserved wins.
|
||||
* Deterministic lexicographic order of status (`fixed` > `not_affected` > `under_investigation` > `affected`) as final tiebreaker.
|
||||
6. **Explain**: mark accepted sources (`accepted=true; reason="weight"`/`"freshness"`), mark rejected sources with explicit `reason` (`"insufficient_justification"`, `"signature_unverified"`, `"lower_weight"`).
|
||||
|
||||
> The algorithm is **pure** given S and policy snapshot; result is reproducible and hashed into `consensusDigest`.
|
||||
|
||||
---
|
||||
|
||||
## 7) Query & export APIs
|
||||
|
||||
All endpoints are versioned under `/api/v1/vex`.
|
||||
|
||||
### 7.1 Query (online)
|
||||
|
||||
```
|
||||
POST /claims/search
|
||||
body: { vulnIds?: string[], productKeys?: string[], providers?: string[], since?: timestamp, limit?: int, pageToken?: string }
|
||||
→ { claims[], nextPageToken? }
|
||||
|
||||
POST /consensus/search
|
||||
body: { vulnIds?: string[], productKeys?: string[], policyRevisionId?: string, since?: timestamp, limit?: int, pageToken?: string }
|
||||
→ { entries[], nextPageToken? }
|
||||
|
||||
POST /resolve
|
||||
body: { purls: string[], vulnIds: string[], policyRevisionId?: string }
|
||||
→ { results: [ { vulnId, productKey, rollupStatus, sources[] } ] }
|
||||
```
|
||||
|
||||
### 7.2 Exports (cacheable snapshots)
|
||||
|
||||
```
|
||||
POST /exports
|
||||
body: { signature: { vulnFilter?, productFilter?, providers?, since? }, format: raw|consensus|index, policyRevisionId?: string, force?: bool }
|
||||
→ { exportId, artifactSha256, rekor? }
|
||||
|
||||
GET /exports/{exportId} → bytes (application/json or binary index)
|
||||
GET /exports/{exportId}/meta → { signature, policyRevisionId, createdAt, artifactSha256, rekor? }
|
||||
```
|
||||
|
||||
### 7.3 Provider operations
|
||||
|
||||
```
|
||||
GET /providers → provider list & signature policy
|
||||
POST /providers/{id}/refresh → trigger fetch/normalize window
|
||||
GET /providers/{id}/status → last fetch, doc counts, signature stats
|
||||
```
|
||||
|
||||
**Auth:** service‑to‑service via Authority tokens; operator operations via UI/CLI with RBAC.
|
||||
|
||||
---
|
||||
|
||||
## 8) Attestation integration
|
||||
|
||||
* Exports can be **DSSE‑signed** via **Signer** and logged to **Rekor v2** via **Attestor** (optional but recommended for regulated pipelines).
|
||||
* `vex.exports.rekor` stores `{uuid, index, url}` when present.
|
||||
* **Predicate type**: `https://stella-ops.org/attestations/vex-export/1` with fields:
|
||||
|
||||
* `querySignature`, `policyRevisionId`, `artifactSha256`, `createdAt`.
|
||||
|
||||
---
|
||||
|
||||
## 9) Configuration (YAML)
|
||||
|
||||
```yaml
|
||||
vexer:
|
||||
mongo: { uri: "mongodb://mongo/vexer" }
|
||||
s3:
|
||||
endpoint: http://minio:9000
|
||||
bucket: stellaops
|
||||
policy:
|
||||
weights:
|
||||
vendor: 1.0
|
||||
distro: 0.9
|
||||
platform: 0.7
|
||||
hub: 0.5
|
||||
attestation: 0.6
|
||||
providerOverrides:
|
||||
redhat: 1.0
|
||||
suse: 0.95
|
||||
requireJustificationForNotAffected: true
|
||||
signatureRequiredForFixed: true
|
||||
minEvidence:
|
||||
not_affected:
|
||||
vendorOrTwoDistros: true
|
||||
connectors:
|
||||
- providerId: redhat
|
||||
kind: csaf
|
||||
baseUrl: https://access.redhat.com/security/data/csaf/v2/
|
||||
signaturePolicy: { type: pgp, keys: [ "…redhat-pgp-key…" ] }
|
||||
windowDays: 7
|
||||
- providerId: suse
|
||||
kind: csaf
|
||||
baseUrl: https://ftp.suse.com/pub/projects/security/csaf/
|
||||
signaturePolicy: { type: pgp, keys: [ "…suse-pgp-key…" ] }
|
||||
- providerId: ubuntu
|
||||
kind: openvex
|
||||
baseUrl: https://…/vex/
|
||||
signaturePolicy: { type: none }
|
||||
- providerId: vendorX
|
||||
kind: cyclonedx-vex
|
||||
ociRef: ghcr.io/vendorx/vex@sha256:…
|
||||
signaturePolicy: { type: cosign, cosignKeylessRoots: [ "sigstore-root" ] }
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 10) Security model
|
||||
|
||||
* **Input signature verification** enforced per provider policy (PGP, cosign, x509).
|
||||
* **Connector allowlists**: outbound fetch constrained to configured domains.
|
||||
* **Tenant isolation**: per‑tenant DB prefixes or separate DBs; per‑tenant S3 prefixes; per‑tenant policies.
|
||||
* **AuthN/Z**: Authority‑issued OpToks; RBAC roles (`vex.read`, `vex.admin`, `vex.export`).
|
||||
* **No secrets in logs**; deterministic logging contexts include providerId, docDigest, claim keys.
|
||||
|
||||
---
|
||||
|
||||
## 11) Performance & scale
|
||||
|
||||
* **Targets:**
|
||||
|
||||
* Normalize 10k VEX claims/minute/core.
|
||||
* Consensus compute ≤ 50 ms for 1k unique `(vuln, product)` pairs in hot cache.
|
||||
* Export (consensus) 1M rows in ≤ 60 s on 8 cores with streaming writer.
|
||||
|
||||
* **Scaling:**
|
||||
|
||||
* WebService handles control APIs; **Worker** background services (same image) execute fetch/normalize in parallel with rate‑limits; Mongo writes batched; upserts by natural keys.
|
||||
* Exports stream straight to S3 (MinIO) with rolling buffers.
|
||||
|
||||
* **Caching:**
|
||||
|
||||
* `vex.cache` maps query signatures → export; TTL to avoid stampedes; optimistic reuse unless `force`.
|
||||
|
||||
---
|
||||
|
||||
## 12) Observability
|
||||
|
||||
* **Metrics:**
|
||||
|
||||
* `vex.ingest.docs_total{provider}`
|
||||
* `vex.normalize.claims_total{provider}`
|
||||
* `vex.signature.failures_total{provider,method}`
|
||||
* `vex.consensus.conflicts_total{vulnId}`
|
||||
* `vex.exports.bytes{format}` / `vex.exports.latency_seconds`
|
||||
* **Tracing:** spans for fetch, verify, parse, map, consensus, export.
|
||||
* **Dashboards:** provider staleness, top conflicting vulns/components, signature posture, export cache hit‑rate.
|
||||
|
||||
---
|
||||
|
||||
## 13) Testing matrix
|
||||
|
||||
* **Connectors:** golden raw docs → deterministic claims (fixtures per provider/format).
|
||||
* **Signature policies:** valid/invalid PGP/cosign/x509 samples; ensure rejects are recorded but not accepted.
|
||||
* **Normalization edge cases:** platform‑only claims, free‑text justifications, non‑purl products.
|
||||
* **Consensus:** conflict scenarios across tiers; check tie‑breakers; justification gates.
|
||||
* **Performance:** 1M‑row export timing; memory ceilings; stream correctness.
|
||||
* **Determinism:** same inputs + policy → identical `consensusDigest` and export bytes.
|
||||
* **API contract tests:** pagination, filters, RBAC, rate limits.
|
||||
|
||||
---
|
||||
|
||||
## 14) Integration points
|
||||
|
||||
* **Backend Policy Engine** (in Scanner.WebService): calls `POST /resolve` with batched `(purl, vulnId)` pairs to fetch `rollupStatus + sources`.
|
||||
* **Feedser**: provides alias graph (CVE↔vendor IDs) and may supply VEX‑adjacent metadata (e.g., KEV flag) for policy escalation.
|
||||
* **UI**: VEX explorer screens use `/claims/search` and `/consensus/search`; show conflicts & provenance.
|
||||
* **CLI**: `stellaops vex export --consensus --since 7d --out vex.json` for audits.
|
||||
|
||||
---
|
||||
|
||||
## 15) Failure modes & fallback
|
||||
|
||||
* **Provider unreachable:** stale thresholds trigger warnings; policy can down‑weight stale providers automatically (freshness factor).
|
||||
* **Signature outage:** continue to ingest but mark `signatureState.verified=false`; consensus will likely exclude or down‑weight per policy.
|
||||
* **Schema drift:** unknown fields are preserved as `evidence`; normalization rejects only on **invalid identity** or **status**.
|
||||
|
||||
---
|
||||
|
||||
## 16) Rollout plan (incremental)
|
||||
|
||||
1. **MVP**: OpenVEX + CSAF connectors for 3 major providers (e.g., Red Hat/SUSE/Ubuntu), normalization + consensus + `/resolve`.
|
||||
2. **Signature policies**: PGP for distros; cosign for OCI.
|
||||
3. **Exports + optional attestation**.
|
||||
4. **CycloneDX VEX** connectors; platform claim expansion tables; UI explorer.
|
||||
5. **Scale hardening**: export indexes; conflict analytics.
|
||||
|
||||
---
|
||||
|
||||
## 17) Appendix — canonical JSON (stable ordering)
|
||||
|
||||
All exports and consensus entries are serialized via `VexCanonicalJsonSerializer`:
|
||||
|
||||
* UTF‑8 without BOM;
|
||||
* keys sorted (ASCII);
|
||||
* arrays sorted by `(providerId, vulnId, productKey, lastObserved)` unless semantic order mandated;
|
||||
* timestamps in `YYYY‑MM‑DDThh:mm:ssZ`;
|
||||
* no insignificant whitespace.
|
||||
|
||||
This architecture keeps Vexer aligned with StellaOps' deterministic, offline-operable design while layering VEX-specific consensus and attestation capabilities on top of the Feedser foundations.
|
||||
|
||||
451
docs/ARCHITECTURE_ZASTAVA.md
Normal file
451
docs/ARCHITECTURE_ZASTAVA.md
Normal file
@@ -0,0 +1,451 @@
|
||||
# component_architecture_zastava.md — **Stella Ops Zastava** (2025Q4)
|
||||
|
||||
> **Scope.** Implementation‑ready architecture for **Zastava**: the **runtime inspector/enforcer** that watches real workloads, detects drift from the scanned baseline, verifies image/SBOM/attestation posture, and (optionally) **admits/blocks** deployments. Includes Kubernetes & plain‑Docker topologies, data contracts, APIs, security posture, performance targets, test matrices, and failure modes.
|
||||
|
||||
---
|
||||
|
||||
## 0) Mission & boundaries
|
||||
|
||||
**Mission.** Give operators **ground‑truth** from running environments and a **fast guardrail** before workloads land:
|
||||
|
||||
* **Observer:** inventory containers, entrypoints actually executed, and DSOs actually loaded; verify **image signature**, **SBOM referrers**, and **attestation** presence; detect **drift** (unexpected processes/paths) and **policy violations**; publish **runtime events** to Scanner.WebService.
|
||||
* **Admission (optional):** Kubernetes ValidatingAdmissionWebhook that enforces minimal posture (signed images, SBOM availability, known base images, policy PASS) **pre‑flight**.
|
||||
|
||||
**Boundaries.**
|
||||
|
||||
* Zastava **does not** compute SBOMs and does not sign; it **consumes** Scanner/WebService outputs and **enforces** backend policy verdicts.
|
||||
* Zastava can **request** a delta scan when the baseline is missing/stale, but scanning is done by **Scanner.Worker**.
|
||||
* On non‑K8s Docker hosts, Zastava runs as a host service with **observer‑only** features.
|
||||
|
||||
---
|
||||
|
||||
## 1) Topology & processes
|
||||
|
||||
### 1.1 Components (Kubernetes)
|
||||
|
||||
```
|
||||
stellaops/zastava-observer # DaemonSet on every node (read-only host mounts)
|
||||
stellaops/zastava-webhook # ValidatingAdmissionWebhook (Deployment, 2+ replicas)
|
||||
```
|
||||
|
||||
### 1.2 Components (Docker/VM)
|
||||
|
||||
```
|
||||
stellaops/zastava-agent # System service; watch Docker events; observer only
|
||||
```
|
||||
|
||||
### 1.3 Dependencies
|
||||
|
||||
* **Authority** (OIDC): short OpToks (DPoP/mTLS) for API calls to Scanner.WebService.
|
||||
* **Scanner.WebService**: `/runtime/events` ingestion; `/policy/runtime` fetch.
|
||||
* **OCI Registry** (optional): for direct referrers/sig checks if not delegated to backend.
|
||||
* **Container runtime**: containerd/CRI‑O/Docker (read interfaces only).
|
||||
* **Kubernetes API** (watch Pods in cluster; validating webhook).
|
||||
* **Host mounts** (K8s DaemonSet): `/proc`, `/var/lib/containerd` (or CRI‑O), `/run/containerd/containerd.sock` (optional, read‑only).
|
||||
|
||||
---
|
||||
|
||||
## 2) Data contracts
|
||||
|
||||
### 2.1 Runtime event (observer → Scanner.WebService)
|
||||
|
||||
```json
|
||||
{
|
||||
"eventId": "9f6a…",
|
||||
"when": "2025-10-17T12:34:56Z",
|
||||
"kind": "CONTAINER_START|CONTAINER_STOP|DRIFT|POLICY_VIOLATION|ATTESTATION_STATUS",
|
||||
"tenant": "tenant-01",
|
||||
"node": "ip-10-0-1-23",
|
||||
"runtime": { "engine": "containerd", "version": "1.7.19" },
|
||||
"workload": {
|
||||
"platform": "kubernetes",
|
||||
"namespace": "payments",
|
||||
"pod": "api-7c9fbbd8b7-ktd84",
|
||||
"container": "api",
|
||||
"containerId": "containerd://...",
|
||||
"imageRef": "ghcr.io/acme/api@sha256:abcd…",
|
||||
"owner": { "kind": "Deployment", "name": "api" }
|
||||
},
|
||||
"process": {
|
||||
"pid": 12345,
|
||||
"entrypoint": ["/entrypoint.sh", "--serve"],
|
||||
"entryTrace": [
|
||||
{"file":"/entrypoint.sh","line":3,"op":"exec","target":"/usr/bin/python3"},
|
||||
{"file":"<argv>","op":"python","target":"/opt/app/server.py"}
|
||||
]
|
||||
},
|
||||
"loadedLibs": [
|
||||
{ "path": "/lib/x86_64-linux-gnu/libssl.so.3", "inode": 123456, "sha256": "…"},
|
||||
{ "path": "/usr/lib/x86_64-linux-gnu/libcrypto.so.3", "inode": 123457, "sha256": "…"}
|
||||
],
|
||||
"posture": {
|
||||
"imageSigned": true,
|
||||
"sbomReferrer": "present|missing",
|
||||
"attestation": { "uuid": "rekor-uuid", "verified": true }
|
||||
},
|
||||
"delta": {
|
||||
"baselineImageDigest": "sha256:abcd…",
|
||||
"changedFiles": ["/opt/app/server.py"], // optional quick signal
|
||||
"newBinaries": [{ "path":"/usr/local/bin/helper","sha256":"…" }]
|
||||
},
|
||||
"evidence": [
|
||||
{"signal":"procfs.maps","value":"/lib/.../libssl.so.3@0x7f..."},
|
||||
{"signal":"cri.task.inspect","value":"pid=12345"},
|
||||
{"signal":"registry.referrers","value":"sbom: application/vnd.cyclonedx+json"}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### 2.2 Admission decision (webhook → API server)
|
||||
|
||||
```json
|
||||
{
|
||||
"admissionId": "…",
|
||||
"namespace": "payments",
|
||||
"podSpecDigest": "sha256:…",
|
||||
"images": [
|
||||
{
|
||||
"name": "ghcr.io/acme/api:1.2.3",
|
||||
"resolved": "ghcr.io/acme/api@sha256:abcd…",
|
||||
"signed": true,
|
||||
"hasSbomReferrers": true,
|
||||
"policyVerdict": "pass|warn|fail",
|
||||
"reasons": ["unsigned base image", "missing SBOM"]
|
||||
}
|
||||
],
|
||||
"decision": "Allow|Deny",
|
||||
"ttlSeconds": 300
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3) Observer — node agent (DaemonSet)
|
||||
|
||||
### 3.1 Responsibilities
|
||||
|
||||
* **Watch** container lifecycle (start/stop) via CRI (`/run/containerd/containerd.sock` gRPC read‑only) or `/var/log/containers/*.log` tail fallback.
|
||||
* **Resolve** container → image digest, mount point rootfs.
|
||||
* **Trace entrypoint**: attach **short‑lived** nsenter/exec to PID 1 in container, parse shell for `exec` chain (bounded depth), record **terminal program**.
|
||||
* **Sample loaded libs**: read `/proc/<pid>/maps` and `exe` symlink to collect **actually loaded** DSOs; compute **sha256** for each mapped file (bounded count/size).
|
||||
* **Posture check** (cheap):
|
||||
|
||||
* Image signature presence (if cosign policies are local; else ask backend).
|
||||
* SBOM **referrers** presence (HEAD to registry, optional).
|
||||
* Rekor UUID known (query Scanner.WebService by image digest).
|
||||
* **Publish runtime events** to Scanner.WebService `/runtime/events` (batch & compress).
|
||||
* **Request delta scan** if: no SBOM in catalog OR base differs from known baseline.
|
||||
|
||||
### 3.2 Privileges & mounts (K8s)
|
||||
|
||||
* **SecurityContext:** `runAsUser: 0`, `readOnlyRootFilesystem: true`, `allowPrivilegeEscalation: false`.
|
||||
* **Capabilities:** `CAP_SYS_PTRACE` (optional if using nsenter trace), `CAP_DAC_READ_SEARCH`.
|
||||
* **Host mounts (read‑only):**
|
||||
|
||||
* `/proc` (host) → `/host/proc`
|
||||
* `/run/containerd/containerd.sock` (or CRI‑O socket)
|
||||
* `/var/lib/containerd/io.containerd.runtime.v2.task` (rootfs paths & pids)
|
||||
* **Networking:** cluster‑internal egress to Scanner.WebService only.
|
||||
* **Rate limits:** hard caps for bytes hashed and file count per container to avoid noisy tenants.
|
||||
|
||||
### 3.3 Event batching
|
||||
|
||||
* Buffer ND‑JSON; flush by **N events** or **2 s**.
|
||||
* Backpressure: local disk ring buffer (50 MB default) if Scanner is temporarily unavailable; drop oldest after cap with **metrics** and **warning** event.
|
||||
|
||||
---
|
||||
|
||||
## 4) Admission Webhook (Kubernetes)
|
||||
|
||||
### 4.1 Gate criteria
|
||||
|
||||
Configurable policy (fetched from backend and cached):
|
||||
|
||||
* **Image signature**: must be cosign‑verifiable to configured key(s) or keyless identities.
|
||||
* **SBOM availability**: at least one **CycloneDX** referrer or **Scanner.WebService** catalog entry.
|
||||
* **Scanner policy verdict**: backend `PASS` required for namespaces/labels matching rules; allow `WARN` if configured.
|
||||
* **Registry allowlists/denylists**.
|
||||
* **Tag bans** (e.g., `:latest`).
|
||||
* **Base image allowlists** (by digest).
|
||||
|
||||
### 4.2 Flow
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
participant K8s as API Server
|
||||
participant WH as Zastava Webhook
|
||||
participant SW as Scanner.WebService
|
||||
|
||||
K8s->>WH: AdmissionReview(Pod)
|
||||
WH->>WH: Resolve images to digests (remote HEAD/pull if needed)
|
||||
WH->>SW: POST /policy/runtime { digests, namespace, labels }
|
||||
SW-->>WH: { per-image: {signed, hasSbom, verdict, reasons}, ttl }
|
||||
alt All pass
|
||||
WH-->>K8s: AdmissionResponse(Allow, ttl)
|
||||
else Any fail (enforce=true)
|
||||
WH-->>K8s: AdmissionResponse(Deny, message)
|
||||
end
|
||||
```
|
||||
|
||||
**Caching:** Per‑digest result cached `ttlSeconds` (default 300 s). **Fail‑open** or **fail‑closed** is configurable per namespace.
|
||||
|
||||
### 4.3 TLS & HA
|
||||
|
||||
* Webhook has its own **serving cert** signed by cluster CA (or custom cert + CA bundle on configuration).
|
||||
* Deployment ≥ 2 replicas; **leaderless**; stateless.
|
||||
|
||||
---
|
||||
|
||||
## 5) Backend integration (Scanner.WebService)
|
||||
|
||||
### 5.1 Ingestion endpoint
|
||||
|
||||
`POST /api/v1/scanner/runtime/events` *(OpTok + DPoP/mTLS)*
|
||||
|
||||
* Validates event schema; enforces rate caps by tenant/node; persists to **Mongo** (`runtime.events` capped collection or regular with TTL).
|
||||
* Performs **correlation**:
|
||||
|
||||
* Attach nearest **image SBOM** (inventory/usage) and **BOM‑Index** if known.
|
||||
* If unknown/missing, schedule **delta scan** and return `202 Accepted`.
|
||||
* Emits **derived signals** (usedByEntrypoint per component based on `/proc/<pid>/maps`).
|
||||
|
||||
### 5.2 Policy decision API (for webhook)
|
||||
|
||||
`POST /api/v1/scanner/policy/runtime`
|
||||
|
||||
Request:
|
||||
|
||||
```json
|
||||
{
|
||||
"namespace": "payments",
|
||||
"labels": { "app": "api", "env": "prod" },
|
||||
"images": ["ghcr.io/acme/api@sha256:...", "ghcr.io/acme/nginx@sha256:..."]
|
||||
}
|
||||
```
|
||||
|
||||
Response:
|
||||
|
||||
```json
|
||||
{
|
||||
"ttlSeconds": 300,
|
||||
"results": {
|
||||
"ghcr.io/acme/api@sha256:...": {
|
||||
"signed": true,
|
||||
"hasSbom": true,
|
||||
"policyVerdict": "pass",
|
||||
"reasons": [],
|
||||
"rekor": { "uuid": "..." }
|
||||
},
|
||||
"ghcr.io/acme/nginx@sha256:...": {
|
||||
"signed": false,
|
||||
"hasSbom": false,
|
||||
"policyVerdict": "fail",
|
||||
"reasons": ["unsigned", "missing SBOM"]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6) Configuration (YAML)
|
||||
|
||||
```yaml
|
||||
zastava:
|
||||
mode:
|
||||
observer: true
|
||||
webhook: true
|
||||
authority:
|
||||
issuer: "https://authority.internal"
|
||||
aud: ["scanner","zastava"] # tokens for backend and self-id
|
||||
backend:
|
||||
url: "https://scanner-web.internal"
|
||||
connectTimeoutMs: 500
|
||||
requestTimeoutMs: 1500
|
||||
retry: { attempts: 3, backoffMs: 200 }
|
||||
runtime:
|
||||
engine: "auto" # containerd|cri-o|docker|auto
|
||||
procfs: "/host/proc"
|
||||
collect:
|
||||
entryTrace: true
|
||||
loadedLibs: true
|
||||
maxLibs: 256
|
||||
maxHashBytesPerContainer: 64_000_000
|
||||
maxDepth: 48
|
||||
admission:
|
||||
enforce: true
|
||||
failOpenNamespaces: ["dev", "test"]
|
||||
verify:
|
||||
imageSignature: true
|
||||
sbomReferrer: true
|
||||
scannerPolicyPass: true
|
||||
cacheTtlSeconds: 300
|
||||
resolveTags: true # do remote digest resolution for tag-only images
|
||||
limits:
|
||||
eventsPerSecond: 50
|
||||
burst: 200
|
||||
perNodeQueue: 10_000
|
||||
security:
|
||||
mounts:
|
||||
containerdSock: "/run/containerd/containerd.sock:ro"
|
||||
proc: "/proc:/host/proc:ro"
|
||||
runtimeState: "/var/lib/containerd:ro"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7) Security posture
|
||||
|
||||
* **AuthN/Z**: Authority OpToks (DPoP preferred) to backend; webhook does **not** require client auth from API server (K8s handles).
|
||||
* **Least privileges**: read‑only host mounts; optional `CAP_SYS_PTRACE`; **no** host networking; **no** write mounts.
|
||||
* **Isolation**: never exec untrusted code; nsenter only to **read** `/proc/<pid>`.
|
||||
* **Data minimization**: do not exfiltrate env vars or command arguments unless policy explicitly enables diagnostic mode.
|
||||
* **Rate limiting**: per‑node caps; per‑tenant caps at backend.
|
||||
* **Hard caps**: bytes hashed, files inspected, depth of shell parsing.
|
||||
|
||||
---
|
||||
|
||||
## 8) Metrics, logs, tracing
|
||||
|
||||
**Observer**
|
||||
|
||||
* `zastava.events_emitted_total{kind}`
|
||||
* `zastava.proc_maps_samples_total{result}`
|
||||
* `zastava.entrytrace_depth{p99}`
|
||||
* `zastava.hash_bytes_total`
|
||||
* `zastava.buffer_drops_total`
|
||||
|
||||
**Webhook**
|
||||
|
||||
* `zastava.admission_requests_total{decision}`
|
||||
* `zastava.admission_latency_seconds`
|
||||
* `zastava.cache_hits_total`
|
||||
* `zastava.backend_failures_total`
|
||||
|
||||
**Logs** (structured): node, pod, image digest, decision, reasons.
|
||||
**Tracing**: spans for observe→batch→post; webhook request→resolve→respond.
|
||||
|
||||
---
|
||||
|
||||
## 9) Performance & scale targets
|
||||
|
||||
* **Observer**: ≤ **30 ms** to sample `/proc/<pid>/maps` and compute quick hashes for ≤ 64 files; ≤ **200 ms** for full library set (256 libs).
|
||||
* **Webhook**: P95 ≤ **8 ms** with warm cache; ≤ **50 ms** with one backend round‑trip.
|
||||
* **Throughput**: 1k admission requests/min/replica; 5k runtime events/min/node with batching.
|
||||
|
||||
---
|
||||
|
||||
## 10) Drift detection model
|
||||
|
||||
**Signals**
|
||||
|
||||
* **Process drift**: terminal program differs from **EntryTrace** baseline.
|
||||
* **Library drift**: loaded DSOs not present in **Usage** SBOM view.
|
||||
* **Filesystem drift**: new executable files under `/usr/local/bin`, `/opt`, `/app` with **mtime** after image creation.
|
||||
* **Network drift** (optional): listening sockets on unexpected ports (from policy).
|
||||
|
||||
**Action**
|
||||
|
||||
* Emit `DRIFT` event with evidence; backend can **auto‑queue** a delta scan; policy may **escalate** to alert/block (Admission cannot block already‑running pods; rely on K8s policies/PodSecurity or operator action).
|
||||
|
||||
---
|
||||
|
||||
## 11) Test matrix
|
||||
|
||||
* **Engines**: containerd, CRI‑O, Docker; ensure PID resolution and rootfs mapping.
|
||||
* **EntryTrace**: bash features (case, if, run‑parts, `.`/`source`), language launchers (python/node/java).
|
||||
* **Procfs**: multiple arches, musl/glibc images; static binaries (maps minimal).
|
||||
* **Admission**: unsigned images, missing SBOM referrers, tag‑only images, digest resolution, backend latency, cache TTL.
|
||||
* **Perf/soak**: 500 Pods/node churn; webhook under HPA growth.
|
||||
* **Security**: attempt privilege escalation disabled, read‑only mounts enforced, rate‑limit abuse.
|
||||
* **Failure injection**: backend down (observer buffers, webhook fail‑open/closed), registry throttling, containerd socket unavailable.
|
||||
|
||||
---
|
||||
|
||||
## 12) Failure modes & responses
|
||||
|
||||
| Condition | Observer behavior | Webhook behavior |
|
||||
| ------------------------------- | ---------------------------------------------- | ------------------------------------------------------ |
|
||||
| Backend unreachable | Buffer to disk; drop after cap; emit metric | **Fail‑open/closed** per namespace config |
|
||||
| PID vanished mid‑sample | Retry once; emit partial evidence | N/A |
|
||||
| CRI socket missing | Fallback to K8s events only (reduced fidelity) | N/A |
|
||||
| Registry digest resolve blocked | Defer to backend; mark `resolve=unknown` | Deny or allow per `resolveTags` & `failOpenNamespaces` |
|
||||
| Excessive events | Apply local rate limit, coalesce | N/A |
|
||||
|
||||
---
|
||||
|
||||
## 13) Deployment notes (K8s)
|
||||
|
||||
**DaemonSet (snippet):**
|
||||
|
||||
```yaml
|
||||
apiVersion: apps/v1
|
||||
kind: DaemonSet
|
||||
metadata: { name: zastava-observer, namespace: stellaops }
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
serviceAccountName: zastava
|
||||
hostPID: true
|
||||
containers:
|
||||
- name: observer
|
||||
image: stellaops/zastava-observer:2.3
|
||||
securityContext:
|
||||
runAsUser: 0
|
||||
readOnlyRootFilesystem: true
|
||||
allowPrivilegeEscalation: false
|
||||
capabilities: { add: ["SYS_PTRACE","DAC_READ_SEARCH"] }
|
||||
volumeMounts:
|
||||
- { name: proc, mountPath: /host/proc, readOnly: true }
|
||||
- { name: containerd-sock, mountPath: /run/containerd/containerd.sock, readOnly: true }
|
||||
- { name: containerd-state, mountPath: /var/lib/containerd, readOnly: true }
|
||||
volumes:
|
||||
- { name: proc, hostPath: { path: /proc } }
|
||||
- { name: containerd-sock, hostPath: { path: /run/containerd/containerd.sock } }
|
||||
- { name: containerd-state, hostPath: { path: /var/lib/containerd } }
|
||||
```
|
||||
|
||||
**Webhook (snippet):**
|
||||
|
||||
```yaml
|
||||
apiVersion: admissionregistration.k8s.io/v1
|
||||
kind: ValidatingWebhookConfiguration
|
||||
webhooks:
|
||||
- name: gate.zastava.stella-ops.org
|
||||
admissionReviewVersions: ["v1"]
|
||||
sideEffects: None
|
||||
failurePolicy: Ignore # or Fail
|
||||
rules:
|
||||
- operations: ["CREATE","UPDATE"]
|
||||
apiGroups: [""]
|
||||
apiVersions: ["v1"]
|
||||
resources: ["pods"]
|
||||
clientConfig:
|
||||
service:
|
||||
namespace: stellaops
|
||||
name: zastava-webhook
|
||||
path: /admit
|
||||
caBundle: <base64 CA>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 14) Implementation notes
|
||||
|
||||
* **Language**: Rust (observer) for low‑latency `/proc` parsing; Go/.NET viable too. Webhook can be .NET 10 for parity with backend.
|
||||
* **CRI drivers**: pluggable (`containerd`, `cri-o`, `docker`). Prefer CRI over parsing logs.
|
||||
* **Shell parser**: re‑use Scanner.EntryTrace grammar for consistent results (compile to WASM if observer is Rust/Go).
|
||||
* **Hashing**: `BLAKE3` for speed locally, then convert to `sha256` (or compute `sha256` directly when budget allows).
|
||||
* **Resilience**: never block container start; observer is **passive**; only webhook decides allow/deny.
|
||||
|
||||
---
|
||||
|
||||
## 15) Roadmap
|
||||
|
||||
* **eBPF** option for syscall/library load tracing (kernel‑level, opt‑in).
|
||||
* **Windows containers** support (ETW providers, loaded modules).
|
||||
* **Network posture** checks: listening ports vs policy.
|
||||
* **Live **used‑by‑entrypoint** synthesis**: send compact bitset diff to backend to tighten Usage view.
|
||||
* **Admission dry‑run** dashboards (simulate block lists before enforcing).
|
||||
|
||||
@@ -31,17 +31,26 @@ Everything here is open‑source and versioned — when you check out a git ta
|
||||
- **03 – [Vision & Road‑map](03_VISION.md)**
|
||||
- **04 – [Feature Matrix](04_FEATURE_MATRIX.md)**
|
||||
|
||||
### Reference & concepts
|
||||
- **05 – [System Requirements Specification](05_SYSTEM_REQUIREMENTS_SPEC.md)**
|
||||
- **07 – [High‑Level Architecture](40_ARCHITECTURE_OVERVIEW.md)**
|
||||
- **08 – Module Specifications**
|
||||
- [README](08_MODULE_SPECIFICATIONS/README.md)
|
||||
- [`backend_api.md`](08_MODULE_SPECIFICATIONS/backend_api.md)
|
||||
- [`zastava_scanner.md`](08_MODULE_SPECIFICATIONS/zastava_scanner.md)
|
||||
- [`registry_scanner.md`](08_MODULE_SPECIFICATIONS/registry_scanner.md)
|
||||
- [`nightly_scheduler.md`](08_MODULE_SPECIFICATIONS/nightly_scheduler.md)
|
||||
### Reference & concepts
|
||||
- **05 – [System Requirements Specification](05_SYSTEM_REQUIREMENTS_SPEC.md)**
|
||||
- **07 – [High‑Level Architecture](07_HIGH_LEVEL_ARCHITECTURE.md)**
|
||||
- **08 – Module Architecture Dossiers**
|
||||
- [Scanner](ARCHITECTURE_SCANNER.md)
|
||||
- [Feedser](ARCHITECTURE_FEEDSER.md)
|
||||
- [Vexer](ARCHITECTURE_VEXER.md)
|
||||
- [Signer](ARCHITECTURE_SIGNER.md)
|
||||
- [Attestor](ARCHITECTURE_ATTESTOR.md)
|
||||
- [Authority](ARCHITECTURE_AUTHORITY.md)
|
||||
- [CLI](ARCHITECTURE_CLI.md)
|
||||
- [Web UI](ARCHITECTURE_UI.md)
|
||||
- [Zastava Runtime](ARCHITECTURE_ZASTAVA.md)
|
||||
- [Release & Operations](ARCHITECTURE_DEVOPS.md)
|
||||
- **09 – [API & CLI Reference](09_API_CLI_REFERENCE.md)**
|
||||
- **10 – [Plug‑in SDK Guide](10_PLUGIN_SDK_GUIDE.md)**
|
||||
- **10 – [Feedser CLI Quickstart](10_FEEDSER_CLI_QUICKSTART.md)**
|
||||
- **30 – [Vexer Connector Packaging Guide](dev/30_VEXER_CONNECTOR_GUIDE.md)**
|
||||
- **30 – Developer Templates**
|
||||
- [Vexer Connector Skeleton](dev/templates/vexer-connector/)
|
||||
- **11 – [Authority Service](11_AUTHORITY.md)**
|
||||
- **11 – [Data Schemas](11_DATA_SCHEMAS.md)**
|
||||
- **12 – [Performance Workbook](12_PERFORMANCE_WORKBOOK.md)**
|
||||
@@ -57,7 +66,7 @@ Everything here is open‑source and versioned — when you check out a git ta
|
||||
- **21 – [Install Guide](21_INSTALL_GUIDE.md)**
|
||||
- **22 – [CI/CD Recipes Library](ci/20_CI_RECIPES.md)**
|
||||
- **23 – [FAQ](23_FAQ_MATRIX.md)**
|
||||
- **24 – [Offline Update Kit Admin Guide](24_OUK_ADMIN_GUIDE.md)**
|
||||
- **24 – [Offline Update Kit Admin Guide](24_OFFLINE_KIT.md)**
|
||||
- **25 – [Feedser Apple Connector Operations](ops/feedser-apple-operations.md)**
|
||||
- **26 – [Authority Key Rotation Playbook](ops/authority-key-rotation.md)**
|
||||
- **27 – [Feedser CCCS Connector Operations](ops/feedser-cccs-operations.md)**
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
|
||||
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|
||||
|----|--------|----------|------------|-------------|---------------|
|
||||
| DOC7.README-INDEX | DONE (2025-10-17) | Docs Guild | — | Refresh index docs (docs/README.md + root README) after architecture dossier split and Offline Kit overhaul. | ✅ ToC reflects new component architecture docs; ✅ root README highlights updated doc set; ✅ Offline Kit guide linked correctly. |
|
||||
| DOC4.AUTH-PDG | REVIEW | Docs Guild, Plugin Team | PLG6.DOC | Copy-edit `docs/dev/31_AUTHORITY_PLUGIN_DEVELOPER_GUIDE.md`, export lifecycle diagram, add LDAP RFC cross-link. | ✅ PR merged with polish; ✅ Diagram committed; ✅ Slack handoff posted. |
|
||||
| DOC1.AUTH | DONE (2025-10-12) | Docs Guild, Authority Core | CORE5B.DOC | Draft `docs/11_AUTHORITY.md` covering architecture, configuration, bootstrap flows. | ✅ Architecture + config sections approved by Core; ✅ Samples reference latest options; ✅ Offline note added. |
|
||||
| DOC3.Feedser-Authority | DONE (2025-10-12) | Docs Guild, DevEx | FSR4 | Polish operator/runbook sections (DOC3/DOC5) to document Feedser authority rollout, bypass logging, and enforcement checklist. | ✅ DOC3/DOC5 updated with audit runbook references; ✅ enforcement deadline highlighted; ✅ Docs guild sign-off. |
|
||||
|
||||
220
docs/dev/30_VEXER_CONNECTOR_GUIDE.md
Normal file
220
docs/dev/30_VEXER_CONNECTOR_GUIDE.md
Normal file
@@ -0,0 +1,220 @@
|
||||
# Vexer Connector Packaging Guide
|
||||
|
||||
> **Audience:** teams implementing new Vexer provider plug‑ins (CSAF feeds,
|
||||
> OpenVEX attestations, etc.)
|
||||
> **Prerequisites:** read `docs/ARCHITECTURE_VEXER.md` and the module
|
||||
> `AGENTS.md` in `src/StellaOps.Vexer.Connectors.Abstractions/`.
|
||||
|
||||
The Vexer connector SDK gives you:
|
||||
|
||||
- `VexConnectorBase` – deterministic logging, SHA‑256 helpers, time provider.
|
||||
- `VexConnectorOptionsBinder` – strongly typed YAML/JSON configuration binding.
|
||||
- `IVexConnectorOptionsValidator<T>` – custom validation hooks (offline defaults, auth invariants).
|
||||
- `VexConnectorDescriptor` & metadata helpers for consistent telemetry.
|
||||
|
||||
This guide explains how to package a connector so the Vexer Worker/WebService
|
||||
can load it via the plugin host.
|
||||
|
||||
---
|
||||
|
||||
## 1. Project layout
|
||||
|
||||
Start from the template under
|
||||
`docs/dev/templates/vexer-connector/`. It contains:
|
||||
|
||||
```
|
||||
Vexer.MyConnector/
|
||||
├── src/
|
||||
│ ├── Vexer.MyConnector.csproj
|
||||
│ ├── MyConnectorOptions.cs
|
||||
│ ├── MyConnector.cs
|
||||
│ └── MyConnectorPlugin.cs
|
||||
└── manifest/
|
||||
└── connector.manifest.yaml
|
||||
```
|
||||
|
||||
Key points:
|
||||
|
||||
- Target `net10.0`, enable `TreatWarningsAsErrors`, reference the
|
||||
`StellaOps.Vexer.Connectors.Abstractions` project (or NuGet once published).
|
||||
- Keep project ID prefix `StellaOps.Vexer.Connectors.<Provider>` so the
|
||||
plugin loader can discover it with the default search pattern.
|
||||
|
||||
### 1.1 csproj snippet
|
||||
|
||||
```xml
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
<PropertyGroup>
|
||||
<TargetFramework>net10.0</TargetFramework>
|
||||
<Nullable>enable</Nullable>
|
||||
<ImplicitUsings>enable</ImplicitUsings>
|
||||
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
|
||||
</PropertyGroup>
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="..\..\..\src\StellaOps.Vexer.Connectors.Abstractions\StellaOps.Vexer.Connectors.Abstractions.csproj" />
|
||||
</ItemGroup>
|
||||
</Project>
|
||||
```
|
||||
|
||||
Adjust the `ProjectReference` for your checkout (or switch to a NuGet package
|
||||
once published).
|
||||
|
||||
---
|
||||
|
||||
## 2. Implement the connector
|
||||
|
||||
1. **Options model** – create an options POCO with data-annotation attributes.
|
||||
Bind it via `VexConnectorOptionsBinder.Bind<TOptions>` in your connector
|
||||
constructor or `ValidateAsync`.
|
||||
2. **Validator** – implement `IVexConnectorOptionsValidator<TOptions>` to add
|
||||
complex checks (e.g., ensure both `clientId` and `clientSecret` are present).
|
||||
3. **Connector** – inherit from `VexConnectorBase`. Implement:
|
||||
- `ValidateAsync` – run binder/validators, log configuration summary.
|
||||
- `FetchAsync` – stream raw documents to `context.RawSink`.
|
||||
- `NormalizeAsync` – convert raw documents into `VexClaimBatch` via
|
||||
format-specific normalizers (`context.Normalizers`).
|
||||
4. **Plugin adapter** – expose the connector via a plugin entry point so the
|
||||
host can instantiate it.
|
||||
|
||||
### 2.1 Options binding example
|
||||
|
||||
```csharp
|
||||
public sealed class MyConnectorOptions
|
||||
{
|
||||
[Required]
|
||||
[Url]
|
||||
public string CatalogUri { get; set; } = default!;
|
||||
|
||||
[Required]
|
||||
public string ApiKey { get; set; } = default!;
|
||||
|
||||
[Range(1, 64)]
|
||||
public int MaxParallelRequests { get; set; } = 4;
|
||||
}
|
||||
|
||||
public sealed class MyConnectorOptionsValidator : IVexConnectorOptionsValidator<MyConnectorOptions>
|
||||
{
|
||||
public void Validate(VexConnectorDescriptor descriptor, MyConnectorOptions options, IList<string> errors)
|
||||
{
|
||||
if (!options.CatalogUri.StartsWith("https://", StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
errors.Add("CatalogUri must use HTTPS.");
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Bind inside the connector:
|
||||
|
||||
```csharp
|
||||
private readonly MyConnectorOptions _options;
|
||||
|
||||
public MyConnector(VexConnectorDescriptor descriptor, ILogger<MyConnector> logger, TimeProvider timeProvider)
|
||||
: base(descriptor, logger, timeProvider)
|
||||
{
|
||||
// `settings` comes from the orchestrator; validators registered via DI.
|
||||
_options = VexConnectorOptionsBinder.Bind<MyConnectorOptions>(
|
||||
descriptor,
|
||||
VexConnectorSettings.Empty,
|
||||
validators: new[] { new MyConnectorOptionsValidator() });
|
||||
}
|
||||
```
|
||||
|
||||
Replace `VexConnectorSettings.Empty` with the actual settings from context
|
||||
inside `ValidateAsync`.
|
||||
|
||||
---
|
||||
|
||||
## 3. Plugin adapter & manifest
|
||||
|
||||
Create a simple plugin class that implements
|
||||
`StellaOps.Plugin.IConnectorPlugin`. The Worker/WebService plugin host uses
|
||||
this contract today.
|
||||
|
||||
```csharp
|
||||
public sealed class MyConnectorPlugin : IConnectorPlugin
|
||||
{
|
||||
private static readonly VexConnectorDescriptor Descriptor =
|
||||
new("vexer:my-provider", VexProviderKind.Vendor, "My Provider VEX");
|
||||
|
||||
public string Name => Descriptor.DisplayName;
|
||||
|
||||
public bool IsAvailable(IServiceProvider services) => true; // inject feature flags if needed
|
||||
|
||||
public IFeedConnector Create(IServiceProvider services)
|
||||
{
|
||||
var logger = services.GetRequiredService<ILogger<MyConnector>>();
|
||||
var timeProvider = services.GetRequiredService<TimeProvider>();
|
||||
return new MyConnector(Descriptor, logger, timeProvider);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
> **Note:** the Vexer Worker currently instantiates connectors through the
|
||||
> shared `IConnectorPlugin` contract. Once a dedicated Vexer plugin interface
|
||||
> lands you simply swap the base interface; the descriptor/connector code
|
||||
> remains unchanged.
|
||||
|
||||
Provide a manifest describing the assembly for operational tooling:
|
||||
|
||||
```yaml
|
||||
# manifest/connector.manifest.yaml
|
||||
id: vexer-my-provider
|
||||
assembly: StellaOps.Vexer.Connectors.MyProvider.dll
|
||||
entryPoint: StellaOps.Vexer.Connectors.MyProvider.MyConnectorPlugin
|
||||
description: >
|
||||
Official VEX feed for ExampleCorp products (CSAF JSON, daily updates).
|
||||
tags:
|
||||
- vexer
|
||||
- csaf
|
||||
- vendor
|
||||
```
|
||||
|
||||
Store manifests under `/opt/stella/vexer/plugins/<connector>/manifest/` in
|
||||
production so the deployment tooling can inventory and verify plug‑ins.
|
||||
|
||||
---
|
||||
|
||||
## 4. Packaging workflow
|
||||
|
||||
1. `dotnet publish -c Release` → copy the published DLLs to
|
||||
`/opt/stella/vexer/plugins/<Provider>/`.
|
||||
2. Place `connector.manifest.yaml` next to the binaries.
|
||||
3. Restart the Vexer Worker or WebService (hot reload not supported yet).
|
||||
4. Verify logs: `VEX-ConnectorLoader` should list the connector descriptor.
|
||||
|
||||
### 4.1 Offline kits
|
||||
|
||||
- Add the connector folder (binaries + manifest) to the Offline Kit bundle.
|
||||
- Include a `settings.sample.yaml` demonstrating offline-friendly defaults.
|
||||
- Document any external dependencies (e.g., SHA mirrors) in the manifest `notes`
|
||||
field.
|
||||
|
||||
---
|
||||
|
||||
## 5. Testing checklist
|
||||
|
||||
- Unit tests around options binding & validators.
|
||||
- Integration tests (future `StellaOps.Vexer.Connectors.Abstractions.Tests`)
|
||||
verifying deterministic logging scopes:
|
||||
`logger.BeginScope` should produce `vex.connector.id`, `vex.connector.kind`,
|
||||
and `vex.connector.operation`.
|
||||
- Deterministic SHA tests: repeated `CreateRawDocument` calls with identical
|
||||
content must return the same digest.
|
||||
|
||||
---
|
||||
|
||||
## 6. Reference template
|
||||
|
||||
See `docs/dev/templates/vexer-connector/` for the full quick‑start including:
|
||||
|
||||
- Sample options class + validator.
|
||||
- Connector implementation inheriting from `VexConnectorBase`.
|
||||
- Plugin adapter + manifest.
|
||||
|
||||
Copy the directory, rename namespaces/IDs, then iterate on provider-specific
|
||||
logic.
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2025-10-17*
|
||||
@@ -0,0 +1,8 @@
|
||||
id: vexer-my-provider
|
||||
assembly: StellaOps.Vexer.Connectors.MyProvider.dll
|
||||
entryPoint: StellaOps.Vexer.Connectors.MyProvider.MyConnectorPlugin
|
||||
description: |
|
||||
Example connector template. Replace metadata before shipping.
|
||||
tags:
|
||||
- vexer
|
||||
- template
|
||||
72
docs/dev/templates/vexer-connector/src/MyConnector.cs
Normal file
72
docs/dev/templates/vexer-connector/src/MyConnector.cs
Normal file
@@ -0,0 +1,72 @@
|
||||
using System.Collections.Generic;
|
||||
using System.Collections.Immutable;
|
||||
using System.Runtime.CompilerServices;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using StellaOps.Vexer.Connectors.Abstractions;
|
||||
using StellaOps.Vexer.Core;
|
||||
|
||||
namespace StellaOps.Vexer.Connectors.MyProvider;
|
||||
|
||||
public sealed class MyConnector : VexConnectorBase
|
||||
{
|
||||
private readonly IEnumerable<IVexConnectorOptionsValidator<MyConnectorOptions>> _validators;
|
||||
private MyConnectorOptions? _options;
|
||||
|
||||
public MyConnector(VexConnectorDescriptor descriptor, ILogger<MyConnector> logger, TimeProvider timeProvider, IEnumerable<IVexConnectorOptionsValidator<MyConnectorOptions>> validators)
|
||||
: base(descriptor, logger, timeProvider)
|
||||
{
|
||||
_validators = validators;
|
||||
}
|
||||
|
||||
public override ValueTask ValidateAsync(VexConnectorSettings settings, CancellationToken cancellationToken)
|
||||
{
|
||||
_options = VexConnectorOptionsBinder.Bind(
|
||||
Descriptor,
|
||||
settings,
|
||||
validators: _validators);
|
||||
|
||||
LogConnectorEvent(LogLevel.Information, "validate", "MyConnector configuration loaded.",
|
||||
new Dictionary<string, object?>
|
||||
{
|
||||
["catalogUri"] = _options.CatalogUri,
|
||||
["maxParallelRequests"] = _options.MaxParallelRequests,
|
||||
});
|
||||
|
||||
return ValueTask.CompletedTask;
|
||||
}
|
||||
|
||||
public override IAsyncEnumerable<VexRawDocument> FetchAsync(VexConnectorContext context, CancellationToken cancellationToken)
|
||||
{
|
||||
if (_options is null)
|
||||
{
|
||||
throw new InvalidOperationException("Connector not validated.");
|
||||
}
|
||||
|
||||
return FetchInternalAsync(context, cancellationToken);
|
||||
}
|
||||
|
||||
private async IAsyncEnumerable<VexRawDocument> FetchInternalAsync(VexConnectorContext context, [EnumeratorCancellation] CancellationToken cancellationToken)
|
||||
{
|
||||
LogConnectorEvent(LogLevel.Information, "fetch", "Fetching catalog window...");
|
||||
|
||||
// Replace with real HTTP logic.
|
||||
await Task.Delay(10, cancellationToken);
|
||||
|
||||
var metadata = BuildMetadata(builder => builder
|
||||
.Add("sourceUri", _options!.CatalogUri)
|
||||
.Add("window", context.Since?.ToString("O") ?? "full"));
|
||||
|
||||
yield return CreateRawDocument(
|
||||
VexDocumentFormat.CsafJson,
|
||||
new Uri($"{_options.CatalogUri.TrimEnd('/')}/sample.json"),
|
||||
new byte[] { 0x7B, 0x7D },
|
||||
metadata);
|
||||
}
|
||||
|
||||
public override ValueTask<VexClaimBatch> NormalizeAsync(VexRawDocument document, CancellationToken cancellationToken)
|
||||
{
|
||||
var claims = ImmutableArray<VexClaim>.Empty;
|
||||
var diagnostics = ImmutableDictionary<string, string>.Empty;
|
||||
return ValueTask.FromResult(new VexClaimBatch(document, claims, diagnostics));
|
||||
}
|
||||
}
|
||||
16
docs/dev/templates/vexer-connector/src/MyConnectorOptions.cs
Normal file
16
docs/dev/templates/vexer-connector/src/MyConnectorOptions.cs
Normal file
@@ -0,0 +1,16 @@
|
||||
using System.ComponentModel.DataAnnotations;
|
||||
|
||||
namespace StellaOps.Vexer.Connectors.MyProvider;
|
||||
|
||||
public sealed class MyConnectorOptions
|
||||
{
|
||||
[Required]
|
||||
[Url]
|
||||
public string CatalogUri { get; set; } = default!;
|
||||
|
||||
[Required]
|
||||
public string ApiKey { get; set; } = default!;
|
||||
|
||||
[Range(1, 32)]
|
||||
public int MaxParallelRequests { get; set; } = 4;
|
||||
}
|
||||
@@ -0,0 +1,15 @@
|
||||
using System.Collections.Generic;
|
||||
using StellaOps.Vexer.Connectors.Abstractions;
|
||||
|
||||
namespace StellaOps.Vexer.Connectors.MyProvider;
|
||||
|
||||
public sealed class MyConnectorOptionsValidator : IVexConnectorOptionsValidator<MyConnectorOptions>
|
||||
{
|
||||
public void Validate(VexConnectorDescriptor descriptor, MyConnectorOptions options, IList<string> errors)
|
||||
{
|
||||
if (!options.CatalogUri.StartsWith("https://", StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
errors.Add("CatalogUri must use HTTPS.");
|
||||
}
|
||||
}
|
||||
}
|
||||
27
docs/dev/templates/vexer-connector/src/MyConnectorPlugin.cs
Normal file
27
docs/dev/templates/vexer-connector/src/MyConnectorPlugin.cs
Normal file
@@ -0,0 +1,27 @@
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using StellaOps.Plugin;
|
||||
using StellaOps.Vexer.Connectors.Abstractions;
|
||||
using StellaOps.Vexer.Core;
|
||||
|
||||
namespace StellaOps.Vexer.Connectors.MyProvider;
|
||||
|
||||
public sealed class MyConnectorPlugin : IConnectorPlugin
|
||||
{
|
||||
private static readonly VexConnectorDescriptor Descriptor = new(
|
||||
id: "vexer:my-provider",
|
||||
kind: VexProviderKind.Vendor,
|
||||
displayName: "My Provider VEX");
|
||||
|
||||
public string Name => Descriptor.DisplayName;
|
||||
|
||||
public bool IsAvailable(IServiceProvider services) => true;
|
||||
|
||||
public IFeedConnector Create(IServiceProvider services)
|
||||
{
|
||||
var logger = services.GetRequiredService<ILogger<MyConnector>>();
|
||||
var timeProvider = services.GetRequiredService<TimeProvider>();
|
||||
var validators = services.GetServices<IVexConnectorOptionsValidator<MyConnectorOptions>>();
|
||||
return new MyConnector(Descriptor, logger, timeProvider, validators);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,12 @@
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
<PropertyGroup>
|
||||
<TargetFramework>net10.0</TargetFramework>
|
||||
<Nullable>enable</Nullable>
|
||||
<ImplicitUsings>enable</ImplicitUsings>
|
||||
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
|
||||
</PropertyGroup>
|
||||
<ItemGroup>
|
||||
<!-- Adjust the relative path when copying this template into a repo -->
|
||||
<ProjectReference Include="..\..\..\..\src\StellaOps.Vexer.Connectors.Abstractions\StellaOps.Vexer.Connectors.Abstractions.csproj" />
|
||||
</ItemGroup>
|
||||
</Project>
|
||||
@@ -1,6 +1,6 @@
|
||||
# Feedser CERT-Bund Connector Operations
|
||||
|
||||
_Last updated: 2025-10-15_
|
||||
_Last updated: 2025-10-17_
|
||||
|
||||
Germany’s Federal Office for Information Security (BSI) operates the Warn- und Informationsdienst (WID) portal. The Feedser CERT-Bund connector (`source:cert-bund:*`) ingests the public RSS feed, hydrates the portal’s JSON detail endpoint, and maps the result into canonical advisories while preserving the original German content.
|
||||
|
||||
@@ -96,18 +96,30 @@ curl -s -b cookies.txt \
|
||||
|
||||
Iterate `page` until the response `content` array is empty. Pages 0–9 currently cover 2014→present. Persist JSON responses (plus SHA256) for Offline Kit parity.
|
||||
|
||||
> **Shortcut** – run `python tools/certbund_offline_snapshot.py --output seed-data/cert-bund`
|
||||
> to bootstrap the session, capture the paginated search responses, and regenerate
|
||||
> the manifest/checksum files automatically. Supply `--cookie-file` and `--xsrf-token`
|
||||
> if the portal requires a browser-derived session (see options via `--help`).
|
||||
|
||||
### 3.3 Export bundles
|
||||
|
||||
```bash
|
||||
curl -s -b cookies.txt \
|
||||
-H "Accept: application/json" \
|
||||
-H "X-XSRF-TOKEN: ${XSRF}" \
|
||||
"https://wid.cert-bund.de/portal/api/securityadvisory/export?format=json&from=2020-01-01" \
|
||||
> certbund-2020-2025.json
|
||||
python tools/certbund_offline_snapshot.py \
|
||||
--output seed-data/cert-bund \
|
||||
--start-year 2014 \
|
||||
--end-year "$(date -u +%Y)"
|
||||
```
|
||||
|
||||
Split long ranges per year and record provenance (`from`, `to`, SHA, capturedAt). Feedser can ingest these JSON payloads directly when operating offline.
|
||||
Task `FEEDCONN-CERTBUND-02-009` tracks turning this workflow into a shipped Offline Kit artefact with manifests and documentation updates—coordinate with the Docs guild before publishing.
|
||||
The helper stores yearly exports under `seed-data/cert-bund/export/`,
|
||||
captures paginated search snapshots in `seed-data/cert-bund/search/`,
|
||||
and generates the manifest + SHA files in `seed-data/cert-bund/manifest/`.
|
||||
Split ranges according to your compliance window (default: one file per
|
||||
calendar year). Feedser can ingest these JSON payloads directly when
|
||||
operating offline.
|
||||
|
||||
> When automatic bootstrap fails (e.g. portal introduces CAPTCHA), run the
|
||||
> manual `curl` flow above, then rerun the helper with `--skip-fetch` to
|
||||
> rebuild the manifest from the existing files.
|
||||
|
||||
### 3.4 Connector-driven catch-up
|
||||
|
||||
|
||||
Reference in New Issue
Block a user