stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 2 Releases Wiki Activity

save progress

Browse Source
This commit is contained in:
StellaOps Bot
2026-01-03 00:47:24 +02:00
parent 3f197814c5
commit ca578801fd
319 changed files with 32478 additions and 2202 deletions
Download Patch File Download Diff File Expand all files Collapse all files

16
docs/technical/architecture/README.md
View File

@@ -17,6 +17,22 @@ Use this index to locate platform-level architecture references and per-module d
- [Data isolation model](data-isolation.md)
- [Security boundaries](security-boundaries.md)
## User-centric views (NEW)
- [User flows (UML diagrams)](user-flows.md) - End-to-end flows from user perspective
- [Module matrix](module-matrix.md) - Complete 46-module inventory with categorization
- [Data flows](data-flows.md) - SBOM, advisory, VEX, and policy data lifecycles
- [Schema mapping](schema-mapping.md) - PostgreSQL, Valkey, and RustFS storage reference
## End-to-end workflow flows
Comprehensive flow documentation for all major StellaOps workflows: [flows/](../../flows/)
| Category | Flows |
|----------|-------|
| **Core Platform** | Dashboard, Scan Submission, SBOM Generation, Policy Evaluation, Notification, Export |
| **Advanced** | CI/CD Gate, Advisory Drift Re-scan, VEX Auto-Generation, Evidence Bundle Export |
| **Enterprise** | Multi-Tenant Policy Rollout, Exception Approval, Risk Score Dashboard |
| **Specialized** | Binary Delta Attestation, Offline Sync, Reachability Drift Alert |
## Module catalogue
Each module directory bundles an ownership charter (`AGENTS.md`), current work (`TASKS.md`), an architecture dossier, and an implementation plan. Operations guides live under `operations/` where applicable.

550
docs/technical/architecture/data-flows.md Normal file
View File

@@ -0,0 +1,550 @@
# Data Flows
This document details the data flows for SBOM generation, advisory ingestion, policy evaluation, and VEX processing in StellaOps. All flows are designed for deterministic, offline-first operation.
## Table of Contents
- [1. SBOM Data Lifecycle](#1-sbom-data-lifecycle)
- [2. Advisory Data Flow](#2-advisory-data-flow)
- [3. VEX Data Flow](#3-vex-data-flow)
- [4. Policy Evaluation Data Flow](#4-policy-evaluation-data-flow)
- [5. Event-Driven Flows](#5-event-driven-flows)
- [6. Offline/Air-Gap Data Flow](#6-offlineair-gap-data-flow)
---
## 1. SBOM Data Lifecycle
### 1.1 Generation Phase (Scanner)
```
Image Input (OCI reference)
|
v
+--------------------------------------------------------------------------------------------+
| Scanner.Worker |
| |
| +----------------------+ +----------------------+ +----------------------+ |
| | Layer Extraction |---->| Delta Cache Check |---->| Analyzer Execution | |
| | (OCI manifest) | | (Valkey layers:*) | | (11 language + OS) | |
| +----------------------+ +----------------------+ +----------------------+ |
| | | |
| | Cache Hit | Cache Miss |
| v v |
| +------------------+ +------------------+ |
| | Stitch Existing | | Full Analysis | |
| | SBOM Fragments | | (20ms fast path) | |
| +------------------+ +------------------+ |
| | | |
| +-------------+---------------+ |
| | |
| v |
| +-----------------------------------------------------------------------------------------+
| | Component Discovery |
| | |
| | +---------------+ +---------------+ +---------------+ +---------------+ |
| | | OS Packages | | Language Deps | | Native Bins | | Call Graphs | |
| | | (Apk/Dpkg/Rpm)| | (11 ecosystems)| | (ELF/PE/Mach) | | (Reachability)| |
| | +---------------+ +---------------+ +---------------+ +---------------+ |
| +-----------------------------------------------------------------------------------------+
| | |
| v |
| +-----------------------------------------------------------------------------------------+
| | SBOM Generation (Two Views) |
| | |
| | Inventory View: Usage View: |
| | - All components in filesystem - Entrypoint closure only |
| | - Declared + transitive + vendored - Actually linked libraries |
| | - Path: images/{digest}/inventory.* - Path: images/{digest}/usage.* |
| +-----------------------------------------------------------------------------------------+
| | |
| v |
| +-----------------------------------------------------------------------------------------+
| | Format Output |
| | |
| | +-------------------+ +-------------------+ +-------------------+ |
| | | CycloneDX 1.6 JSON| | CycloneDX Protobuf| | SPDX 3.0.1 JSON | |
| | | (.cdx.json) | | (.cdx.pb, compact)| | (.spdx.json) | |
| | +-------------------+ +-------------------+ +-------------------+ |
| +-----------------------------------------------------------------------------------------+
+--------------------------------------------------------------------------------------------+
```
### 1.2 Storage Phase
```
+--------------------------------------------------------------------------------------------+
| Dual-Write Coordination |
| |
| +------------------------------------------+ +------------------------------------------+|
| | PostgreSQL (scanner schema) | | RustFS (S3 API) ||
| | | | ||
| | artifacts table: | | Blob Layout: ||
| | - artifact_id (sha256) | | blobs/{sha256_prefix}/ ||
| | - image_digest | | sbom.json (payload) ||
| | - format (cdx-json, spdx-json, etc.) | | sbom.meta.json (wrapper) ||
| | - created_at | | sbom.cdx.pb (binary) ||
| | - rekor_proof (optional) | | ||
| | | | Wrapper Envelope: ||
| | images table: | | { ||
| | - image_digest | | "id": "sha256:417f...", ||
| | - repository | | "imageDigest": "sha256:e2b9...", ||
| | - tag | | "format": "cdx-json", ||
| | - architecture | | "layers": ["sha256:..."], ||
| | | | "partial": false, ||
| | layers table: | | "provenanceId": "prov_0291" ||
| | - layer_digest | | } ||
| | - media_type | | ||
| | - size | | ||
| +------------------------------------------+ +------------------------------------------+|
+--------------------------------------------------------------------------------------------+
```
### 1.3 Index & Cache Phase (Valkey)
```
+--------------------------------------------------------------------------------------------+
| Valkey Keyspace for SBOM |
| |
| +------------------------------------------+ +------------------------------------------+|
| | Key Pattern | | Purpose ||
| +------------------------------------------+ +------------------------------------------+|
| | scan:{digest} | | Last scan JSON result ||
| | layers:{digest} | | Set of layer digests (90d TTL) ||
| | locator:{imageDigest} | | sbomBlobId mapping (30d TTL) ||
| +------------------------------------------+ +------------------------------------------+|
| |
| Delta SBOM Flow: |
| 1. Check layers:{digest} for cached layers |
| 2. Scan only missing layers (partial=true) |
| 3. Stitch new data onto cached full SBOM |
| 4. Update locator mapping |
| 5. Fast path: 20ms for unchanged layers |
+--------------------------------------------------------------------------------------------+
```
### 1.4 Consumption Phase
```
+--------------------------------------------------------------------------------------------+
| SBOM Consumers |
| |
| +-----------------+ +-----------------+ +-----------------+ |
| | Policy Engine | | Export Center | | Replay Engine | |
| +-----------------+ +-----------------+ +-----------------+ |
| | | | |
| v v v |
| +-------------+ +-------------+ +-------------+ |
| | Read SBOM | | Retrieve | | Link to | |
| | from RustFS | | by digest | | scan ID | |
| +-------------+ +-------------+ +-------------+ |
| | | | |
| v v v |
| +-------------+ +-------------+ +-------------+ |
| | Join with | | Convert | | Replay with | |
| | advisories | | format | | same inputs | |
| +-------------+ +-------------+ +-------------+ |
| | | | |
| v v v |
| +-------------+ +-------------+ +-------------+ |
| | Apply | | Generate | | Verify | |
| | reachability| | SARIF | | determinism | |
| +-------------+ +-------------+ +-------------+ |
+--------------------------------------------------------------------------------------------+
```
---
## 2. Advisory Data Flow
### 2.1 Ingestion (Concelier)
```
+--------------------------------------------------------------------------------------------+
| Advisory Ingestion Pipeline |
| |
| External Sources Concelier.Worker |
| +---------------+ +-------------------------------------------+ |
| | NVD |----------------------->| | |
| | Red Hat | | 1. Fetch advisories (HTTP/mirror) | |
| | OSV | | 2. For air-gap: use mirror bundles first | |
| | GHSA | | 3. Validate schema conformance | |
| | CSAF sources | | 4. Normalize to canonical observations | |
| +---------------+ | 5. Apply AOC (Aggregation-Only Contract) | |
| | 6. Persist raw documents (append-only) | |
| | 7. Build linksets (advisory -> PURL) | |
| | 8. Publish delta event | |
| +-------------------------------------------+ |
| | |
| v |
| +-----------------------------------------------------------------------------------------+
| | PostgreSQL (vuln schema) |
| | |
| | advisory_raw (append-only): linksets: |
| | - raw_document (JSON as-received) - advisory_id -> purl[] |
| | - source (NVD, RED_HAT, OSV, etc.) - Used for SBOM join in Policy Engine |
| | - advisory_id (CVE-2024-xxxx) |
| | - affected_purls observations: |
| | - published_at (UTC) - Normalized advisory metadata |
| | - revision - Severity, description, references |
| +-----------------------------------------------------------------------------------------+
| | |
| v |
| +-----------------------------------------------------------------------------------------+
| | Event: concelier:drift (Valkey Stream) |
| | |
| | Triggers: |
| | - Scheduler: identifies affected scans |
| | - Policy Engine: re-evaluation of impacted findings |
| | - Notify: critical vuln alerts |
| +-----------------------------------------------------------------------------------------+
+--------------------------------------------------------------------------------------------+
```
### 2.2 Advisory Data Model
```
+--------------------------------------------------------------------------------------------+
| Raw Layer (Immutable - AOC Enforced) |
| |
| { |
| "advisoryId": "CVE-2024-1234", |
| "source": "NVD", |
| "rawDocument": { /* original JSON as received */ }, |
| "publishedAt": "2024-01-15T10:00:00Z", |
| "revision": 2, |
| "affectedPurls": [ |
| "pkg:npm/lodash@4.17.20", |
| "pkg:maven/org.apache.struts/struts2-core@2.5.30" |
| ], |
| "severity": { |
| "cvssV3": { "baseScore": 9.8, "vector": "..." }, |
| "cvssV4": { "baseScore": 9.2, "vector": "..." } |
| } |
| } |
| |
| Key Constraints: |
| - Raw documents are NEVER modified after ingestion |
| - Conflicts are preserved, not collapsed |
| - Multiple sources for same CVE stored separately |
| - Provenance tracked per observation |
+--------------------------------------------------------------------------------------------+
```
---
## 3. VEX Data Flow
### 3.1 VEX Ingestion (Excititor)
```
+--------------------------------------------------------------------------------------------+
| VEX Ingestion Pipeline |
| |
| External Sources Excititor.Worker |
| +---------------+ +-------------------------------------------+ |
| | OpenVEX |----------------------->| | |
| | CSAF VEX | | 1. Fetch VEX statements | |
| | SBOM referrers| | 2. For air-gap: use offline bundles | |
| | Vendor feeds | | 3. Verify signatures (if signed) | |
| +---------------+ | 4. Normalize to canonical shape | |
| | 5. Persist immutable raw statements | |
| | 6. Publish to VexLens for consensus | |
| +-------------------------------------------+ |
| | |
| v |
| +-----------------------------------------------------------------------------------------+
| | PostgreSQL (vex schema) |
| | |
| | vex_raw (append-only): |
| | - raw_statement (OpenVEX JSON as-received) |
| | - issuer_id (vendor or trust issuer) |
| | - component_purl |
| | - vulnerability_id (CVE or GHSA) |
| | - status (not_affected, affected, under_investigation) |
| | - justification (component_not_present, vulnerable_code_not_present, etc.) |
| | - published_at |
| | - signature (DSSE envelope if signed) |
| +-----------------------------------------------------------------------------------------+
+--------------------------------------------------------------------------------------------+
```
### 3.2 VEX Consensus (VexLens)
```
+--------------------------------------------------------------------------------------------+
| VEX Consensus Pipeline |
| |
| +--------------------+ |
| | Multiple VEX | |
| | statements for | |
| | same (CVE, PURL) | |
| +--------------------+ |
| | |
| v |
| +--------------------------------------------------------------------------------------------+
| | VexLens Consensus Engine |
| | |
| | 1. Merge observations by component identity (PURL) |
| | 2. Apply issuer priority rules: |
| | - Vendor > Distro > Researcher > Community |
| | 3. Apply trust scores (from IssuerDirectory) |
| | 4. Detect conflicts (multiple issuers disagree) |
| | 5. Preserve conflict state (K4 lattice: True + False = Conflict) |
| | 6. Export consensus outcomes |
| +--------------------------------------------------------------------------------------------+
| | |
| v |
| +--------------------------------------------------------------------------------------------+
| | Policy Engine Integration |
| | |
| | VEX gates override severity thresholds: |
| | - not_affected -> PASS (with evidence) |
| | - affected -> normal evaluation continues |
| | - under_investigation -> WARN (pending) |
| +--------------------------------------------------------------------------------------------+
+--------------------------------------------------------------------------------------------+
```
---
## 4. Policy Evaluation Data Flow
### 4.1 Input Assembly
```
+--------------------------------------------------------------------------------------------+
| Policy Engine Input Sources (All Immutable - AOC Enforced) |
| |
| +------------------+ +------------------+ +------------------+ +------------------+ |
| | SBOM | | Advisory | | VEX | | Reachability | |
| | (from RustFS) | | (from vuln.*) | | (from vex.*) | | (from Scanner) | |
| +------------------+ +------------------+ +------------------+ +------------------+ |
| | | | | |
| +--------------------+--------------------+--------------------+ |
| | |
| v |
| +-----------------------------------------------------------------------------------------+
| | Selection Layer |
| | |
| | Deterministic Joining: |
| | - SBOM <-> Advisory (by PURL matching) |
| | - Advisory <-> VEX (by CVE + PURL) |
| | - Component <-> Reachability (by identity) |
| | |
| | Batch Ordering: |
| | - Sort by (tenant, policyId, vulnerabilityId, productKey, source) |
| | - Enables incremental cursor-based processing |
| +-----------------------------------------------------------------------------------------+
+--------------------------------------------------------------------------------------------+
```
### 4.2 Evaluation Pipeline
```
+--------------------------------------------------------------------------------------------+
| Policy Evaluation Pipeline |
| |
| +-----------------------------------------------------------------------------------------+
| | 1. Load Policy IR (cached by policyId+version hash) |
| +-----------------------------------------------------------------------------------------+
| | |
| v |
| +-----------------------------------------------------------------------------------------+
| | 2. For Each Batch (component, vulnerability): |
| | |
| | +------------------+ |
| | | Evidence-Weighted| severityWeight (from advisory CVSS) |
| | | Score Compute |---> trustWeight (from VEX issuer) |
| | +------------------+ reachabilityWeight (from Scanner entrypoint closure) |
| | | runtimeWeight (from Zastava signals) |
| | v |
| | +------------------+ |
| | | Policy Rules | First-match semantics |
| | | Execution |---> Actions: assign, annotate, escalate, warn |
| | +------------------+ |
| | | |
| | v |
| | +------------------+ |
| | | Exception Apply | Specificity-ranked |
| | | |---> Effects: suppress, defer, downgrade, require-control |
| | +------------------+ |
| | | |
| | v |
| | +------------------+ |
| | | Unknown Budget | Per-environment limits |
| | | Check |---> Block if exceeded, Warn if approaching |
| | +------------------+ |
| | | |
| | v |
| | +------------------+ |
| | | Confidence Calc | 5 factors: reachability, runtime, VEX, provenance, policy |
| | | |---> Final = Clamp01(Sum(Weight x RawValue)) |
| | +------------------+ Tiers: VeryHigh(>=0.9), High(>=0.7), Medium(>=0.5), etc. |
| | | |
| | v |
| | +------------------+ |
| | | VEX Decision | Emit OpenVEX statements for verdict changes |
| | | Emission |---> DSSE-signed, logged to Rekor v2 |
| | +------------------+ |
| +-----------------------------------------------------------------------------------------+
+--------------------------------------------------------------------------------------------+
```
### 4.3 Output Materialization
```
+--------------------------------------------------------------------------------------------+
| Finding Materialization |
| |
| +------------------------------------------+ +------------------------------------------+|
| | Current Snapshot | | History (Audit Trail) ||
| | policy.effective_finding_{policyId} | | policy.effective_finding_{policyId}_history|
| +------------------------------------------+ +------------------------------------------+|
| | - finding_key (deterministic digest) | | - All snapshots with timestamps ||
| | - severity (CRITICAL, HIGH, etc.) | | - Previous verdicts ||
| | - source (NVD, vendor, research) | | - Provenance chain ||
| | - advisory_raw_ids (back-link) | | ||
| | - vex_raw_ids (if overridden) | | ||
| | - sbom_component_id (inventory link) | | ||
| | - verdict (PASS, BLOCK, WARN, FAIL) | | ||
| | - confidence_score (0-100) | | ||
| | - explained_trace (policy rule hits) | | ||
| +------------------------------------------+ +------------------------------------------+|
| |
| Determinism Hash: |
| SHA256(policyVersion + batchCursor + inputsHash) |
| -> Same inputs always produce same outputs |
+--------------------------------------------------------------------------------------------+
```
---
## 5. Event-Driven Flows
### 5.1 Event Bus Architecture
```
+--------------------------------------------------------------------------------------------+
| Valkey Streams / NATS JetStream Event Bus |
| |
| +-----------------------------------------------------------------------------------------+
| | Stream: scanner:events |
| | Events: scan.submitted, scan.started, scan.completed, scan.failed |
| | Consumers: Policy, Notify, TimelineIndexer, ExportCenter |
| +-----------------------------------------------------------------------------------------+
| |
| +-----------------------------------------------------------------------------------------+
| | Stream: concelier:drift |
| | Events: advisory.new, advisory.updated, advisory.withdrawn |
| | Consumers: Scheduler, Policy, Notify |
| +-----------------------------------------------------------------------------------------+
| |
| +-----------------------------------------------------------------------------------------+
| | Stream: policy:evaluated |
| | Events: evaluation.completed, verdict.changed, exception.applied |
| | Consumers: Notify, Findings, ExportCenter |
| +-----------------------------------------------------------------------------------------+
| |
| +-----------------------------------------------------------------------------------------+
| | Stream: scheduler:jobs |
| | Events: run.started, run.completed, run.failed, rescan.triggered |
| | Consumers: Scanner, Notify, TimelineIndexer |
| +-----------------------------------------------------------------------------------------+
| |
| +-----------------------------------------------------------------------------------------+
| | Stream: notify:delivery |
| | Events: notification.sent, notification.failed, notification.throttled |
| | Consumers: Audit, TimelineIndexer |
| +-----------------------------------------------------------------------------------------+
+--------------------------------------------------------------------------------------------+
```
### 5.2 Scan Completion Event Flow
```
scan.completed event
|
+-------------------+-------------------+-------------------+
| | | |
v v v v
+----------------+ +----------------+ +----------------+ +----------------+
| Policy.Engine | | Notify.Worker | |TimelineIndexer | | ExportCenter |
+----------------+ +----------------+ +----------------+ +----------------+
| | | | | | | |
| Evaluate with | | Check rules | | Index event | | Generate SARIF |
| new SBOM data | | for matches | | for audit | | if configured |
| | | | | | | |
+-------+--------+ +-------+--------+ +-------+--------+ +----------------+
| | |
v v v
policy:evaluated notification.sent event indexed
| |
+-------------------+
|
v
Downstream consumers
```
---
## 6. Offline/Air-Gap Data Flow
### 6.1 Offline Kit Contents
```
+--------------------------------------------------------------------------------------------+
| Offline Update Kit Structure |
| |
| offline-kit-2025-01-02/ |
| +-- feeds/ |
| | +-- nvd/ # NVD advisory snapshots |
| | +-- osv/ # OSV advisory snapshots |
| | +-- ghsa/ # GHSA advisory snapshots |
| | +-- vex/ # VEX statement bundles |
| +-- images/ # Container images for platform services |
| +-- sboms/ # Pre-generated SBOMs for bundled images |
| +-- signatures/ # DSSE-signed bundles |
| | +-- feeds.dsse # Signed feed manifest |
| | +-- images.dsse # Signed image manifest |
| +-- trust-roots/ # CA certificates, JWKS |
| +-- policies/ # Default policy definitions |
| +-- manifest.json # Kit contents and checksums |
| +-- manifest.dsse # Signed manifest |
+--------------------------------------------------------------------------------------------+
```
### 6.2 Offline Ingestion Flow
```
+--------------------------------------------------------------------------------------------+
| Air-Gap Ingestion Pipeline |
| |
| USB/Portable Media AirGap.Importer |
| +---------------+ +-------------------------------------------+ |
| | Offline Kit |----------------------->| | |
| | (signed) | | 1. Verify manifest signature (DSSE) | |
| +---------------+ | 2. Validate checksums | |
| | 3. Import feeds to Concelier | |
| | 4. Import VEX to Excititor | |
| | 5. Update trust roots | |
| | 6. Trigger re-evaluation | |
| +-------------------------------------------+ |
| |
| Key Guarantees: |
| - No network calls during import |
| - All verification is local |
| - Deterministic outputs match online mode |
| - Full audit trail preserved |
+--------------------------------------------------------------------------------------------+
```
---
## Related Documentation
- [User Flows](user-flows.md)
- [Module Matrix](module-matrix.md)
- [Schema Mapping](schema-mapping.md)
- [Data Schemas](../../11_DATA_SCHEMAS.md)
- [Offline Kit](../../24_OFFLINE_KIT.md)

248
docs/technical/architecture/module-matrix.md Normal file
View File

@@ -0,0 +1,248 @@
# Complete Module Matrix
This document provides a comprehensive inventory of all 46+ modules in the StellaOps solution (`src/StellaOps.sln`), explaining the purpose of each module and how they relate to the documented architecture.
## Table of Contents
- [1. Module Count Explanation](#1-module-count-explanation)
- [2. Complete Module Inventory](#2-complete-module-inventory)
- [3. Module Categories](#3-module-categories)
- [4. Service Deployment Matrix](#4-service-deployment-matrix)
- [5. Module Dependencies](#5-module-dependencies)
---
## 1. Module Count Explanation
The solution contains **46 top-level modules** in `src/`. The architecture documentation covers the **~35 core user-facing modules**, while the remaining modules fall into:
| Category | Count | Description |
|----------|-------|-------------|
| Core Platform | 4 | Authority, Gateway, Router, Platform |
| Data Ingestion | 7 | Concelier, Excititor, VexLens, VexHub, IssuerDirectory, Feedser, Mirror |
| Scanning & Analysis | 5 | Scanner, BinaryIndex, AdvisoryAI, Symbols, ReachGraph |
| Artifacts & Evidence | 7 | Attestor, Signer, SbomService, EvidenceLocker, ExportCenter, Provenance, Provcache |
| Policy & Risk | 4 | Policy, RiskEngine, VulnExplorer, Unknowns |
| Operations | 8 | Scheduler, Orchestrator, TaskRunner, Notify, Notifier, PacksRegistry, TimelineIndexer, Replay |
| Integration | 5 | CLI, Zastava, Web, API, Registry |
| Infrastructure | 6 | Cryptography, Telemetry, Graph, Signals, AirGap, AOC |
| Testing & Benchmarks | 2 | Benchmark, Bench |
| Utility & Internal | 6+ | Cartographer, Findings, SrmRemote, Tools, PluginBinaries, etc. |
---
## 2. Complete Module Inventory
### Core Platform (4 modules)
| Module | Path | Purpose | WebService | Worker | Storage |
|--------|------|---------|------------|--------|---------|
| **Authority** | `src/Authority/` | Authentication, authorization, OAuth/OIDC, DPoP, tenant management | Yes | No | PostgreSQL (`authority`) |
| **Gateway** | `src/Gateway/` | API gateway with routing, TLS termination, transport abstraction | Yes | No | Stateless |
| **Router** | `src/Router/` | Transport-agnostic messaging (TCP/TLS/UDP/RabbitMQ/Valkey) | Yes | No | Valkey |
| **Platform** | `src/Platform/` | Platform Service aggregation APIs, console data composition | Yes | No | Aggregates |
### Data Ingestion (7 modules)
| Module | Path | Purpose | WebService | Worker | Storage |
|--------|------|---------|------------|--------|---------|
| **Concelier** | `src/Concelier/` | Vulnerability advisory ingestion (NVD, OSV, GHSA, CSAF), merge engine with AOC | Yes | Yes | PostgreSQL (`vuln`) |
| **Excititor** | `src/Excititor/` | VEX document ingestion and export (OpenVEX, CSAF VEX) | Yes | Yes | PostgreSQL (`vex`) |
| **VexLens** | `src/VexLens/` | VEX consensus computation across issuers, conflict analysis | Yes | No | PostgreSQL (cache) |
| **VexHub** | `src/VexHub/` | VEX distribution and exchange hub | Yes | No | PostgreSQL |
| **IssuerDirectory** | `src/IssuerDirectory/` | Issuer trust registry for CSAF publishers | Yes | No | PostgreSQL |
| **Feedser** | `src/Feedser/` | Evidence collection library for backport detection | Library | N/A | N/A |
| **Mirror** | `src/Mirror/` | Vulnerability feed mirror and distribution | Yes | Yes | RustFS |
### Scanning & Analysis (5 modules)
| Module | Path | Purpose | WebService | Worker | Storage |
|--------|------|---------|------------|--------|---------|
| **Scanner** | `src/Scanner/` | Container scanning with SBOM generation (11 language analyzers), call graphs | Yes | Yes | PostgreSQL (`scanner`) + RustFS |
| **BinaryIndex** | `src/BinaryIndex/` | Binary identity extraction and fingerprinting | Yes | No | PostgreSQL |
| **AdvisoryAI** | `src/AdvisoryAI/` | AI-assisted advisory analysis and summarization | Yes | No | PostgreSQL |
| **Symbols** | `src/Symbols/` | Symbol resolution and debug information | Yes | No | PostgreSQL |
| **ReachGraph** | `src/ReachGraph/` | Reachability graph service, CVE reachability analysis | Yes | No | PostgreSQL |
### Artifacts & Evidence (7 modules)
| Module | Path | Purpose | WebService | Worker | Storage |
|--------|------|---------|------------|--------|---------|
| **Attestor** | `src/Attestor/` | in-toto/DSSE attestation generation, Rekor v2 integration | Yes | No | PostgreSQL + RustFS |
| **Signer** | `src/Signer/` | Cryptographic signing operations (PKIX, DSSE) | Yes | No | PostgreSQL |
| **SbomService** | `src/SbomService/` | SBOM storage, versioning, and lineage ledger | Yes | No | PostgreSQL + RustFS |
| **EvidenceLocker** | `src/EvidenceLocker/` | Sealed evidence storage and export | Yes | No | RustFS |
| **ExportCenter** | `src/ExportCenter/` | Batch export and report generation (SARIF, SBOM, evidence bundles) | Yes | No | RustFS |
| **Provenance** | `src/Provenance/` | SLSA/DSSE attestation tooling | Library | N/A | N/A |
| **Provcache** | Library | Provenance cache utilities | Library | N/A | N/A |
### Policy & Risk (4 modules)
| Module | Path | Purpose | WebService | Worker | Storage |
|--------|------|---------|------------|--------|---------|
| **Policy** | `src/Policy/` | Policy engine with K4 lattice logic, confidence scoring, VEX emission | Yes | Yes | PostgreSQL (`policy`) |
| **RiskEngine** | `src/RiskEngine/` | Risk scoring runtime with pluggable providers | Yes | No | PostgreSQL |
| **VulnExplorer** | `src/VulnExplorer/` | Vulnerability exploration and triage UI backend | Yes | No | PostgreSQL (cache) |
| **Unknowns** | `src/Unknowns/` | Unknown component and symbol tracking registry | Yes | No | PostgreSQL |
### Operations (8 modules)
| Module | Path | Purpose | WebService | Worker | Storage |
|--------|------|---------|------------|--------|---------|
| **Scheduler** | `src/Scheduler/` | Job scheduling and queue management, cron-based rescan | Yes | No | PostgreSQL (`scheduler`) |
| **Orchestrator** | `src/Orchestrator/` | Workflow orchestration and task coordination | Yes | No | PostgreSQL (`orchestrator`) |
| **TaskRunner** | `src/TaskRunner/` | Task pack execution engine | Yes | Yes | PostgreSQL |
| **Notify** | `src/Notify/` | Notification toolkit (Email, Slack, Teams, Webhooks) - shared libraries | Library | N/A | N/A |
| **Notifier** | `src/Notifier/` | Notifications Studio host (WebService + Worker) | Yes | Yes | PostgreSQL (`notify`) |
| **PacksRegistry** | `src/PacksRegistry/` | Task packs registry and distribution | Yes | No | PostgreSQL |
| **TimelineIndexer** | `src/TimelineIndexer/` | Timeline event indexing for audit trails | Yes | No | PostgreSQL |
| **Replay** | `src/Replay/` | Deterministic replay engine | Yes | No | PostgreSQL |
### Integration (5 modules)
| Module | Path | Purpose | WebService | Worker | Storage |
|--------|------|---------|------------|--------|---------|
| **CLI** | `src/Cli/` | Command-line interface (Native AOT, multi-platform) | CLI | N/A | N/A |
| **Zastava** | `src/Zastava/` | Container registry webhook observer, admission control | Yes | No | PostgreSQL |
| **Web** | `src/Web/` | Angular 17 frontend SPA | Static | N/A | N/A |
| **API** | `src/Api/` | OpenAPI contracts and governance | Library | N/A | N/A |
| **Registry** | `src/Registry/` | Container registry integration, token service | Yes | No | PostgreSQL |
### Infrastructure (6 modules)
| Module | Path | Purpose | WebService | Worker | Storage |
|--------|------|---------|------------|--------|---------|
| **Cryptography** | `src/Cryptography/` | Crypto plugins (FIPS, eIDAS, GOST, SM, PQ) | Library | N/A | N/A |
| **Telemetry** | `src/Telemetry/` | OpenTelemetry traces, metrics, logging | Library | N/A | N/A |
| **Graph** | `src/Graph/` | Call graph and reachability data structures | Library | N/A | N/A |
| **Signals** | `src/Signals/` | Runtime signal collection and correlation | Library | N/A | N/A |
| **AirGap** | `src/AirGap/` | Air-gapped deployment support, Offline Kit bundling | Yes | Yes | RustFS |
| **AOC** | `src/Aoc/` | Append-Only Contract enforcement (Roslyn analyzers) | Library | N/A | N/A |
### Testing & Benchmarks (2 modules)
| Module | Path | Purpose | WebService | Worker | Storage |
|--------|------|---------|------------|--------|---------|
| **Benchmark** | Scanner library | Competitive benchmarking (accuracy comparison) | Tool | N/A | N/A |
| **Bench** | `src/Bench/` | Performance benchmarks | Tool | N/A | N/A |
### Utility & Internal (6+ modules)
| Module | Path | Purpose | Notes |
|--------|------|---------|-------|
| **Cartographer** | `src/Cartographer/` | Identity graphs from SBOM/advisory data | Feeds Graph Explorer |
| **Findings** | `src/Findings/` | Materializes effective findings from Policy outputs | Feeds UI/CLI/Notify |
| **SrmRemote** | `src/SrmRemote/` | SBOM remote operations | Integration utility |
| **Tools** | `src/Tools/` | Utility programs (fixture generators, migration scripts) | Dev tooling |
| **PluginBinaries** | Various | Authority and Concelier plugin binaries | Plugin hosting |
| **DevPortal** | `src/DevPortal/` | Developer onboarding portal | Documentation |
---
## 3. Module Categories
### By Runtime Type
| Type | Modules |
|------|---------|
| **WebService + Worker** | Scanner, Concelier, Excititor, Policy, Notifier, TaskRunner, AirGap, Mirror |
| **WebService Only** | Authority, Gateway, Router, Platform, VexLens, VexHub, IssuerDirectory, BinaryIndex, AdvisoryAI, Symbols, ReachGraph, Attestor, Signer, SbomService, EvidenceLocker, ExportCenter, RiskEngine, VulnExplorer, Unknowns, Scheduler, Orchestrator, PacksRegistry, TimelineIndexer, Replay, Zastava, Registry |
| **Library** | Feedser, Provenance, Provcache, Notify, API, Cryptography, Telemetry, Graph, Signals, AOC |
| **CLI/Tool** | CLI, Benchmark, Bench, Tools |
| **Static** | Web (Angular SPA) |
### By Data Store
| Store | Modules |
|-------|---------|
| **PostgreSQL** | Authority, Concelier, Excititor, VexLens, VexHub, IssuerDirectory, Scanner, BinaryIndex, AdvisoryAI, Symbols, ReachGraph, Attestor, Signer, SbomService, Policy, RiskEngine, VulnExplorer, Unknowns, Scheduler, Orchestrator, TaskRunner, Notifier, PacksRegistry, TimelineIndexer, Replay, Zastava, Registry |
| **RustFS (S3)** | Scanner, Attestor, SbomService, EvidenceLocker, ExportCenter, AirGap, Mirror |
| **Valkey** | Gateway, Router, Scanner, Policy, Scheduler, Notifier (for queues/cache) |
| **Stateless** | Gateway, Platform, CLI, Web |
### By Communication Pattern
| Pattern | Modules |
|---------|---------|
| **HTTP REST** | All WebService modules |
| **Binary Frame (Router)** | Gateway to all backend services |
| **Event Streams (Valkey/NATS)** | Scanner, Concelier, Excititor, Policy, Scheduler, Notifier |
| **Direct Library** | All Library modules |
---
## 4. Service Deployment Matrix
| Service | Container Image | Replicas | Storage | Queue | Observable |
|---------|-----------------|----------|---------|-------|------------|
| Authority | `stellaops/authority` | 2+ | PostgreSQL | Valkey (DPoP) | Yes |
| Gateway | `stellaops/gateway` | 2+ | Stateless | - | Yes |
| Scanner.Web | `stellaops/scanner-web` | 2+ | PostgreSQL + RustFS | - | Yes |
| Scanner.Worker | `stellaops/scanner-worker` | N | RustFS | Valkey/NATS | Yes |
| Concelier.Web | `stellaops/concelier-web` | 2+ | PostgreSQL | - | Yes |
| Concelier.Worker | `stellaops/concelier-worker` | N | PostgreSQL | Valkey/NATS | Yes |
| Policy.Engine | `stellaops/policy-engine` | 2+ | PostgreSQL | Valkey | Yes |
| Policy.Worker | `stellaops/policy-worker` | N | PostgreSQL | Valkey | Yes |
| Scheduler | `stellaops/scheduler` | 2+ | PostgreSQL | Valkey | Yes |
| Notifier | `stellaops/notifier` | 2+ | PostgreSQL | Valkey | Yes |
| ExportCenter | `stellaops/export-center` | 2+ | RustFS | - | Yes |
| Web (UI) | `stellaops/web` | 2+ | Static | - | Yes |
---
## 5. Module Dependencies
### Core Dependency Graph
```
+-------------+
| Authority |
+------+------+
|
+----------------------------+----------------------------+
| | |
v v v
+----------+ +----------+ +----------+
| Gateway |---------------->| Scanner |---------------->| Policy |
+----+-----+ +----+-----+ +----+-----+
| | |
| v v
| +----------+ +----------+
| | Concelier| | Signer |
| +----+-----+ +----+-----+
| | |
| v v
| +----------+ +----------+
| | Excititor| | Attestor|
| +----------+ +----------+
|
v
+----------+
| UI |
+----------+
```
### Key Integration Points
| From | To | Integration |
|------|----|-------------|
| Gateway | Authority | Token validation (JWKS) |
| Gateway | All Services | Binary frame routing |
| Scanner | Signer | SBOM signing |
| Scanner | Attestor | in-toto attestation |
| Policy | Concelier | Advisory data (read-only) |
| Policy | Excititor | VEX data (read-only) |
| Policy | Scanner | SBOM data (read-only) |
| Scheduler | Scanner | Trigger rescans |
| Scheduler | Concelier | Observe advisory deltas |
| Notifier | All Services | Consume events |
---
## Related Documentation
- [User Flows](user-flows.md)
- [Data Flows](data-flows.md)
- [Schema Mapping](schema-mapping.md)
- [Component Map](component-map.md)
- [Platform Topology](platform-topology.md)

1507
docs/technical/architecture/policy-engine-data-pipeline.md Normal file
View File

File diff suppressed because it is too large Load Diff

583
docs/technical/architecture/schema-mapping.md Normal file
View File

@@ -0,0 +1,583 @@
# Schema Mapping Reference
This document provides a comprehensive mapping of all data storage schemas across PostgreSQL, Valkey, and RustFS (S3), organized by module ownership.
## Table of Contents
- [1. Storage Overview](#1-storage-overview)
- [2. PostgreSQL Schema Ownership](#2-postgresql-schema-ownership)
- [3. Valkey Keyspace Reference](#3-valkey-keyspace-reference)
- [4. RustFS (S3) Path Conventions](#4-rustfs-s3-path-conventions)
- [5. Module-to-Storage Mapping](#5-module-to-storage-mapping)
- [6. Cross-References](#6-cross-references)
---
## 1. Storage Overview
StellaOps uses three primary storage systems:
| Storage | Purpose | Technology | Notes |
|---------|---------|------------|-------|
| **PostgreSQL** | Canonical persistent store | PostgreSQL v16+ | Per-module schema isolation |
| **Valkey** | Cache, queues, events | Valkey v8.0 (Redis-compatible) | DPoP nonces, task streams |
| **RustFS** | Object storage | S3-compatible API | Content-addressed blobs |
### Storage Principles
1. **Schema Isolation**: Each module owns its PostgreSQL schema
2. **Append-Only for Evidence**: Advisory, VEX, and SBOM raw data is immutable (AOC)
3. **Content-Addressable**: Blob storage uses SHA256 digest prefixes
4. **Deterministic Keys**: Valkey keys follow predictable patterns
5. **TTL Management**: Cache entries have explicit TTLs
---
## 2. PostgreSQL Schema Ownership
### Schema-to-Module Mapping
| Schema | Owner Module | Purpose | Key Tables |
|--------|--------------|---------|------------|
| `authority` | Authority | Identity, clients, keys, auth audit | `clients`, `keys`, `tokens`, `audit_trail` |
| `scanner` | Scanner | Scan manifests, triage, metadata | `scans`, `artifacts`, `images`, `layers` |
| `vuln` | Concelier | Advisory raw documents, linksets | `advisory_raw`, `linksets`, `observations` |
| `vex` | Excititor | VEX raw statements, consensus | `vex_raw`, `consensus`, `issuer_trust` |
| `policy` | Policy | Policies, exceptions, findings | `policies`, `exceptions`, `effective_finding_*` |
| `scheduler` | Scheduler | Jobs, runs, schedules | `schedules`, `runs`, `impact_snapshots` |
| `notify` | Notifier | Rules, channels, delivery history | `rules`, `channels`, `templates`, `delivery_log` |
| `orchestrator` | Orchestrator | Workflows, tasks | `workflows`, `tasks`, `task_runs` |
| `registry` | Registry | Token service, image metadata | `tokens`, `repositories` |
| `symbols` | Symbols | Symbol resolution | `symbols`, `debug_info` |
| `unknowns` | Unknowns | Unknown components | `unknown_components`, `tracking` |
### Detailed Schema Definitions
#### Schema: `authority`
```sql
-- Core identity tables
clients (
client_id UUID PRIMARY KEY,
tenant_id UUID NOT NULL,
name VARCHAR(255),
client_type VARCHAR(50), -- confidential, public, service
redirect_uris TEXT[],
scopes TEXT[],
created_at TIMESTAMPTZ,
updated_at TIMESTAMPTZ
);
keys (
key_id UUID PRIMARY KEY,
client_id UUID REFERENCES clients,
algorithm VARCHAR(50), -- RS256, ES256, EdDSA
public_key TEXT,
created_at TIMESTAMPTZ,
expires_at TIMESTAMPTZ
);
tokens (
token_id UUID PRIMARY KEY,
client_id UUID REFERENCES clients,
subject VARCHAR(255),
scopes TEXT[],
issued_at TIMESTAMPTZ,
expires_at TIMESTAMPTZ,
revoked_at TIMESTAMPTZ
);
audit_trail (
audit_id UUID PRIMARY KEY,
action VARCHAR(100),
actor_id VARCHAR(255),
resource_type VARCHAR(100),
resource_id VARCHAR(255),
occurred_at TIMESTAMPTZ,
metadata JSONB
);
```
#### Schema: `scanner`
```sql
-- Scan lifecycle tables
scans (
scan_id UUID PRIMARY KEY,
tenant_id UUID NOT NULL,
image_digest VARCHAR(100),
image_reference TEXT,
state VARCHAR(50), -- pending, acquired, running, completed, failed
created_at TIMESTAMPTZ,
started_at TIMESTAMPTZ,
completed_at TIMESTAMPTZ,
metadata JSONB
);
artifacts (
artifact_id VARCHAR(100) PRIMARY KEY, -- sha256:...
scan_id UUID REFERENCES scans,
format VARCHAR(50), -- cdx-json, spdx-json, cdx-pb
created_at TIMESTAMPTZ,
rekor_proof JSONB
);
images (
image_digest VARCHAR(100) PRIMARY KEY,
repository TEXT,
tag TEXT,
architecture VARCHAR(50),
os VARCHAR(50),
created_at TIMESTAMPTZ
);
layers (
layer_digest VARCHAR(100) PRIMARY KEY,
media_type VARCHAR(255),
size BIGINT,
created_at TIMESTAMPTZ
);
scan_artifacts (
scan_id UUID REFERENCES scans,
artifact_id VARCHAR(100) REFERENCES artifacts,
PRIMARY KEY (scan_id, artifact_id)
);
```
#### Schema: `vuln`
```sql
-- Advisory storage (append-only, AOC enforced)
advisory_raw (
raw_id UUID PRIMARY KEY,
advisory_id VARCHAR(100), -- CVE-2024-xxxx
source VARCHAR(50), -- NVD, RED_HAT, OSV, GHSA
raw_document JSONB NOT NULL, -- Original JSON as-received
published_at TIMESTAMPTZ,
revision INTEGER,
created_at TIMESTAMPTZ,
UNIQUE (advisory_id, source, revision)
);
linksets (
linkset_id UUID PRIMARY KEY,
advisory_id VARCHAR(100),
purl TEXT,
version_range JSONB,
created_at TIMESTAMPTZ
);
observations (
observation_id UUID PRIMARY KEY,
advisory_id VARCHAR(100),
severity_cvss3 JSONB,
severity_cvss4 JSONB,
description TEXT,
references JSONB,
created_at TIMESTAMPTZ
);
```
#### Schema: `vex`
```sql
-- VEX storage (append-only, AOC enforced)
vex_raw (
raw_id UUID PRIMARY KEY,
issuer_id VARCHAR(255),
component_purl TEXT,
vulnerability_id VARCHAR(100),
status VARCHAR(50), -- not_affected, affected, under_investigation
justification VARCHAR(100),
raw_statement JSONB NOT NULL,
published_at TIMESTAMPTZ,
signature JSONB, -- DSSE envelope if signed
created_at TIMESTAMPTZ
);
consensus (
consensus_id UUID PRIMARY KEY,
component_purl TEXT,
vulnerability_id VARCHAR(100),
resolved_status VARCHAR(50),
conflict_detected BOOLEAN,
contributing_vex_ids UUID[],
computed_at TIMESTAMPTZ
);
issuer_trust (
issuer_id VARCHAR(255) PRIMARY KEY,
trust_score DECIMAL(3,2), -- 0.00 to 1.00
priority INTEGER, -- Lower = higher priority
issuer_type VARCHAR(50), -- vendor, distro, researcher, community
updated_at TIMESTAMPTZ
);
```
#### Schema: `policy`
```sql
-- Policy definitions and lifecycle
policies (
policy_id UUID PRIMARY KEY,
tenant_id UUID NOT NULL,
name VARCHAR(255),
version INTEGER,
state VARCHAR(50), -- DRAFT, SHADOW, ACTIVE, ENFORCING
yaml_content TEXT,
compiled_ir JSONB,
ir_hash VARCHAR(100),
created_at TIMESTAMPTZ,
updated_at TIMESTAMPTZ
);
policy_runs (
run_id UUID PRIMARY KEY,
policy_id UUID REFERENCES policies,
tenant_id UUID NOT NULL,
cursor JSONB,
stats JSONB,
determinism_hash VARCHAR(100),
started_at TIMESTAMPTZ,
completed_at TIMESTAMPTZ
);
exceptions (
exception_id UUID PRIMARY KEY,
tenant_id UUID NOT NULL,
policy_id UUID,
component_purl TEXT,
vulnerability_id VARCHAR(100),
effect VARCHAR(50), -- suppress, defer, downgrade, require_control
approval_level VARCHAR(10), -- G0, G1, G2, G3, G4
expires_at TIMESTAMPTZ,
created_at TIMESTAMPTZ,
created_by VARCHAR(255)
);
exception_approval_audit (
audit_id UUID PRIMARY KEY,
exception_id UUID REFERENCES exceptions,
approval_level VARCHAR(10),
approver_id VARCHAR(255),
approved_at TIMESTAMPTZ,
comment TEXT
);
-- Dynamic per-policy finding tables (created dynamically)
-- effective_finding_{policyId} - Current snapshot
-- effective_finding_{policyId}_history - Audit trail
```
#### Schema: `scheduler`
```sql
-- Job scheduling
schedules (
schedule_id VARCHAR(100) PRIMARY KEY,
tenant_id UUID NOT NULL,
name VARCHAR(255),
enabled BOOLEAN,
cron_expression VARCHAR(100),
timezone VARCHAR(50),
mode VARCHAR(50),
selection JSONB,
notify JSONB,
limits JSONB,
created_at TIMESTAMPTZ,
updated_at TIMESTAMPTZ
);
runs (
run_id VARCHAR(100) PRIMARY KEY,
tenant_id UUID NOT NULL,
schedule_id VARCHAR(100) REFERENCES schedules,
trigger VARCHAR(50),
state VARCHAR(50),
stats JSONB,
deltas JSONB,
created_at TIMESTAMPTZ,
started_at TIMESTAMPTZ,
finished_at TIMESTAMPTZ,
error TEXT
);
run_summaries (
summary_id VARCHAR(200) PRIMARY KEY, -- tenant:schedule
tenant_id UUID NOT NULL,
schedule_id VARCHAR(100),
last_run JSONB,
recent JSONB[],
counters JSONB,
updated_at TIMESTAMPTZ
);
```
#### Schema: `notify`
```sql
-- Notification routing
rules (
rule_id UUID PRIMARY KEY,
tenant_id UUID NOT NULL,
name VARCHAR(255),
match JSONB,
actions JSONB,
enabled BOOLEAN,
created_at TIMESTAMPTZ,
updated_at TIMESTAMPTZ,
deleted_at TIMESTAMPTZ
);
channels (
channel_id UUID PRIMARY KEY,
tenant_id UUID NOT NULL,
name VARCHAR(255),
type VARCHAR(50), -- slack, teams, email, webhook
config JSONB,
created_at TIMESTAMPTZ,
updated_at TIMESTAMPTZ,
deleted_at TIMESTAMPTZ
);
templates (
template_id UUID PRIMARY KEY,
channel_type VARCHAR(50),
key VARCHAR(100),
locale VARCHAR(10),
render_mode VARCHAR(50),
body TEXT,
created_at TIMESTAMPTZ,
updated_at TIMESTAMPTZ
);
delivery_log (
delivery_id UUID PRIMARY KEY,
rule_id UUID REFERENCES rules,
channel_id UUID REFERENCES channels,
event_id UUID,
status VARCHAR(50),
delivered_at TIMESTAMPTZ,
error TEXT
);
```
---
## 3. Valkey Keyspace Reference
### Key Patterns by Module
| Module | Key Pattern | Type | TTL | Purpose |
|--------|-------------|------|-----|---------|
| **Authority** | `dpop:{jti}` | string | 5m | DPoP nonce cache (RFC 9449) |
| **Scanner** | `scan:{digest}` | string | Infinite | Last scan JSON result |
| **Scanner** | `layers:{digest}` | set | 90d | Layers with SBOMs (delta cache) |
| **Scanner** | `locator:{imageDigest}` | string | 30d | Image digest to sbomBlobId mapping |
| **Policy** | `policy:active` | string | Infinite | Active policy YAML/Rego |
| **Policy** | `policy:history` | list | Infinite | Change audit IDs |
| **Concelier** | `feed:nvd:json` | string | 24h | Normalized feed snapshot |
| **General** | `quota:{token}` | string | Until UTC midnight | Per-token scan counter |
| **Scheduler** | `scheduler:jobs` | stream | - | Job queue |
| **Notifier** | `notify:delivery` | stream | 7d | Delivery events |
| **All** | `events:*` | stream | 7d | Event streams |
| **All** | `queue:*` | stream | - | Task queues |
| **Telemetry** | `metrics:*` | various | - | Runtime metrics |
### Stream Definitions
| Stream | Producers | Consumers | Events |
|--------|-----------|-----------|--------|
| `scanner:events` | Scanner.Worker | Policy, Notify, TimelineIndexer, ExportCenter | scan.submitted, scan.completed, scan.failed |
| `concelier:drift` | Concelier.Worker | Scheduler, Policy, Notify | advisory.new, advisory.updated |
| `policy:evaluated` | Policy.Worker | Notify, Findings, ExportCenter | evaluation.completed, verdict.changed |
| `scheduler:jobs` | Scheduler | Scanner, Policy | run.started, rescan.triggered |
| `notify:delivery` | Notifier | Audit, TimelineIndexer | notification.sent, notification.failed |
### Valkey Configuration
```yaml
# Recommended Valkey configuration for StellaOps
maxmemory: 2gb
maxmemory-policy: volatile-lru
stream-node-max-bytes: 4096
stream-node-max-entries: 100
# Consumer groups for job processing
scanner:jobs:
consumer_group: scanner-workers
idle_timeout: 60s
notify:delivery:
consumer_group: notify-workers
idle_timeout: 30s
```
---
## 4. RustFS (S3) Path Conventions
### Blob Storage Layout
```
blobs/
+-- {sha256_prefix}/ # First 4 chars of digest
+-- {full_digest}/
+-- sbom.json # SBOM payload (any format)
+-- sbom.meta.json # Wrapper envelope
+-- sbom.cdx.pb # CycloneDX Protobuf (compact)
+-- attestation.dsse # DSSE envelope
+-- evidence.bundle # Evidence package
images/
+-- {imageDigest}/
+-- inventory.cdx.json # Inventory SBOM
+-- inventory.cdx.pb # Inventory (Protobuf)
+-- usage.cdx.json # Usage SBOM (entrypoint closure)
+-- usage.cdx.pb # Usage (Protobuf)
+-- call-graph.json # Call graph data
+-- reachability.json # Reachability analysis
evidence/
+-- {bundleId}/
+-- manifest.json # Bundle manifest
+-- manifest.dsse # Signed manifest
+-- sboms/ # SBOM files
+-- attestations/ # Attestation files
+-- proofs/ # Verification proofs
offline-kits/
+-- {kitId}/
+-- feeds/ # Advisory snapshots
+-- images/ # Container images
+-- signatures/ # DSSE signatures
+-- trust-roots/ # CA certificates
+-- manifest.json # Kit manifest
```
### SBOM Wrapper Envelope
```json
{
"id": "sha256:417f...",
"imageDigest": "sha256:e2b9...",
"created": "2025-01-02T15:30:00Z",
"format": "cdx-json",
"layers": [
"sha256:d38b...",
"sha256:af45..."
],
"partial": false,
"provenanceId": "prov_0291"
}
```
### Content Types
| Extension | Content-Type | Description |
|-----------|--------------|-------------|
| `.json` | `application/json` | JSON documents |
| `.cdx.json` | `application/vnd.cyclonedx+json` | CycloneDX JSON |
| `.cdx.pb` | `application/vnd.cyclonedx+protobuf` | CycloneDX Protobuf |
| `.spdx.json` | `application/spdx+json` | SPDX JSON |
| `.dsse` | `application/vnd.dsse+json` | DSSE envelope |
| `.bundle` | `application/zip` | Evidence bundle |
---
## 5. Module-to-Storage Mapping
### Complete Reference Table
| Module | PostgreSQL Schema | Valkey Keys | RustFS Paths |
|--------|------------------|-------------|--------------|
| **Authority** | `authority` | `dpop:{jti}` | - |
| **Gateway** | - (stateless) | - | - |
| **Router** | - | connection state | - |
| **Scanner** | `scanner` | `scan:{digest}`, `layers:{digest}`, `locator:{imageDigest}` | `blobs/`, `images/` |
| **Concelier** | `vuln` | `feed:*`, `concelier:drift` stream | - |
| **Excititor** | `vex` | - | - |
| **VexLens** | - (reads `vex`) | - | - |
| **VexHub** | `vex` (extension) | - | - |
| **IssuerDirectory** | - (reads `vex.issuer_trust`) | - | - |
| **Policy** | `policy` | `policy:active`, `policy:history` | - |
| **RiskEngine** | - (reads `policy`) | - | - |
| **Scheduler** | `scheduler` | `scheduler:jobs` stream | - |
| **Notifier** | `notify` | `notify:delivery` stream | - |
| **Orchestrator** | `orchestrator` | `orchestrator:*` streams | - |
| **Attestor** | - (uses `scanner`) | - | `blobs/*/attestation.dsse` |
| **Signer** | - (uses `authority`) | - | - |
| **SbomService** | - (reads `scanner`) | - | `blobs/`, `images/` |
| **EvidenceLocker** | - | - | `evidence/` |
| **ExportCenter** | - | - | `evidence/`, `offline-kits/` |
| **AirGap** | - | - | `offline-kits/` |
| **Registry** | `registry` | - | - |
| **Symbols** | `symbols` | - | - |
| **Unknowns** | `unknowns` | - | - |
| **TimelineIndexer** | - (writes to `scanner`, etc.) | - | - |
---
## 6. Cross-References
### Data Flow Dependencies
```
+---------+ +-----------+ +--------+ +--------+
| Scanner |---->| SbomService|---->| Policy |---->| Notify |
+---------+ +-----------+ +--------+ +--------+
| | | |
v v v v
+----------+ +-----------+ +-----------+ +---------+
| scanner | | blobs/ | | policy | | notify |
| (PG) | | (RustFS) | | (PG) | | (PG) |
+----------+ +-----------+ +-----------+ +---------+
+-----------+ +----------+
| Concelier |---->| Policy |
+-----------+ +----------+
| ^
v |
+----------+ +-----------+
| vuln | | Excititor |
| (PG) | +-----------+
+----------+ |
v
+----------+
| vex |
| (PG) |
+----------+
```
### Schema Version Tracking
All schemas support versioning:
```sql
-- Every schema has a version tracking table
schema_migrations (
version INTEGER PRIMARY KEY,
description TEXT,
applied_at TIMESTAMPTZ,
checksum VARCHAR(64)
);
```
### Backup Considerations
| Storage | Backup Strategy | Retention |
|---------|-----------------|-----------|
| PostgreSQL | pg_dump + WAL archiving | 30 days |
| Valkey | RDB snapshots + AOF | 7 days |
| RustFS | Object versioning | 90 days |
---
## Related Documentation
- [Data Schemas](../../11_DATA_SCHEMAS.md) - Detailed schema definitions
- [Data Flows](data-flows.md) - How data moves through the system
- [Data Isolation](data-isolation.md) - Per-tenant isolation model
- [Module Matrix](module-matrix.md) - Complete module inventory
- [Request Flows](request-flows.md) - HTTP/Binary protocol flows

577
docs/technical/architecture/user-flows.md Normal file
View File

@@ -0,0 +1,577 @@
# User Flow Architecture
This document provides detailed UML-style diagrams showing how users interact with StellaOps from the UI/CLI through the Gateway to backend services. All flows are documented from the user's perspective.
## Table of Contents
- [1. Architecture Overview](#1-architecture-overview)
- [2. Dashboard Data Flow](#2-dashboard-data-flow)
- [3. Scan Submission Flow](#3-scan-submission-flow)
- [4. SBOM Generation Flow](#4-sbom-generation-flow)
- [5. Policy Evaluation Flow](#5-policy-evaluation-flow)
- [6. Policy Setup Flow](#6-policy-setup-flow)
- [7. Notification Flow](#7-notification-flow)
- [8. Export Flow](#8-export-flow)
---
## 1. Architecture Overview
### Layered Architecture Diagram
```
+-----------------------------------------------------------------------------------------------------------+
| USER EXPERIENCE LAYER |
+-----------------------------------------------------------------------------------------------------------+
| |
| +----------------+ +----------------+ +----------------+ |
| | Web UI | | CLI | | CI/CD | |
| | (Angular 17) | | (Native AOT) | | (BuildX) | |
| +-------+--------+ +-------+--------+ +-------+--------+ |
| | | | |
| +----------------------+----------------------+ |
| | HTTPS + DPoP/mTLS |
| v |
| +-------------------------------------------------------------------------------------------+ |
| | GATEWAY (API Router) | |
| | +---------------+ +---------------+ +---------------+ +---------------+ +-------------+ | |
| | |CorrelationId |>| DPoP/mTLS |>|IdentityHeader |>| Authorization |>|Routing | | |
| | | Middleware | | Validation | | Policy | | Middleware | |Decision | | |
| | +---------------+ +---------------+ +---------------+ +---------------+ +-------------+ | |
| | | | |
| | +------------+------------+ | |
| | v v | |
| | Token Validation Binary Frame Protocol | |
| | via Authority (TCP/TLS/Valkey) | |
| +-------------------------------------------------------------------------------------------+ |
| |
+-----------------------------------------------------------------------------------------------------------+
|
+------------------+------------------+------------------+------------------+
v v v v v
+-----------------------------------------------------------------------------------------------------------+
| MICROSERVICES LAYER |
+-----------------------------------------------------------------------------------------------------------+
| |
| +------------------+ +------------------+ +------------------+ +------------------+ +--------------+ |
| | AUTHORITY | | SCANNER | | POLICY | | CONCELIER | | EXCITITOR | |
| | (OAuth/OIDC) | | (SBOM Gen) | | (Decisions) | | (Advisories) | | (VEX Ingest) | |
| +------------------+ +------------------+ +------------------+ +------------------+ +--------------+ |
| | * DPoP tokens | | * WebService | | * K4 Lattice | | * NVD/OSV/GHSA | | * OpenVEX | |
| | * mTLS certs | | * Worker pool | | * Confidence | | * CSAF sources | | * CSAF VEX | |
| | * Scopes/RBAC | | * 11 analyzers | | * Gates | | * Linksets | | * Consensus | |
| | * Tenant mgmt | | * Call graphs | | * VEX emission | | * AOC enforced | | * Signatures | |
| +--------+---------+ +--------+---------+ +--------+---------+ +--------+---------+ +------+-------+ |
| | | | | | |
| | +---------------+--------------------+--------------------+----------------------+ |
| | | | | | |
| v v v v v |
| +--------------------------------------------------------------------------------------------+ |
| | EVENT BUS (Valkey Streams / NATS JetStream) | |
| | scanner:events | concelier:drift | policy:evaluated | notify:delivery | scheduler:jobs | |
| +--------------------------------------------------------------------------------------------+ |
| | | | | |
| v v v v |
| +------------------+ +------------------+ +------------------+ +------------------+ |
| | SCHEDULER | | NOTIFY | | ATTESTOR | | EXPORTCENTER | |
| | (Job Orch) | | (Notifications) | | (DSSE/in-toto) | | (SARIF/SBOM) | |
| +------------------+ +------------------+ +------------------+ +------------------+ |
| | * Cron jobs | | * Slack/Teams | | * DSSE envelope | | * Format conv | |
| | * Delta rescan | | * Email/Webhook | | * Rekor v2 log | | * Batch export | |
| | * Lease/backoff | | * Templates | | * Verify chain | | * Evidence zip | |
| +------------------+ +------------------+ +--------+---------+ +------------------+ |
| | |
| v |
| +------------------+ |
| | SIGNER | |
| | (Crypto Ops) | |
| +------------------+ |
| | * PKIX signing | |
| | * FIPS/eIDAS | |
| | * GOST/SM/PQ | |
| +------------------+ |
| |
+-----------------------------------------------------------------------------------------------------------+
|
v
+-----------------------------------------------------------------------------------------------------------+
| PERSISTENCE LAYER |
+-----------------------------------------------------------------------------------------------------------+
| |
| +----------------------------------------------+ +----------------------------------------------+ |
| | PostgreSQL v16+ | | RustFS (S3 API) | |
| | | | | |
| | Schema: authority (identity, clients, keys) | | blobs/{sha256}/sbom.json | |
| | Schema: scanner (manifests, triage) | | blobs/{sha256}/sbom.cdx.pb | |
| | Schema: vuln (advisory_raw, linksets) | | blobs/{sha256}/attestation.dsse | |
| | Schema: vex (vex_raw, consensus) | | blobs/{sha256}/evidence.bundle | |
| | Schema: policy (exceptions, findings) | | | |
| | Schema: scheduler (jobs, runs) | | | |
| | Schema: notify (rules, channels, history) | | | |
| | Schema: orchestrator (workflows) | | | |
| +----------------------------------------------+ +----------------------------------------------+ |
| |
| +----------------------------------------------+ |
| | Valkey v8.0 (Cache/Queue) | |
| | | |
| | scan:{digest} -> Last scan result | |
| | layers:{digest} -> Delta cache (90d TTL) | |
| | dpop:{jti} -> DPoP nonce (5m TTL) | |
| | queue:* -> Task streams | |
| | events:* -> Event streams (7d TTL) | |
| +----------------------------------------------+ |
| |
+-----------------------------------------------------------------------------------------------------------+
```
---
## 2. Dashboard Data Flow
When a user opens the Web UI dashboard, the following data flow occurs:
```
User opens Web UI (Angular)
|
v
+--------------------------------------------------------------------------------------------+
| HomeDashboardComponent (src/Web/StellaOps.Web/src/app/features/home/) |
| |
| +--------------------------------------------------------------------------------------+ |
| | forkJoin([ | |
| | ConsoleVulnApi.getFacets() -> Severity breakdown (Critical/High/Med/Low) | |
| | RiskApi.stats() -> Risk score & trend (improving/worsening) | |
| | ReachabilityApi.getSummary() -> Reachable/Unreachable/Uncertain counts | |
| | ]) | |
| +--------------------------------------------------------------------------------------+ |
| | |
| | HTTPS |
| v |
| +--------------------------------------------------------------------------------------+ |
| | GATEWAY (routes by path) | |
| | | |
| | /api/console/vuln/facets -> Scanner.WebService | |
| | /api/risk/stats -> RiskEngine | |
| | /api/reachability/summary-> ReachGraph | |
| +--------------------------------------------------------------------------------------+ |
| | |
| v |
| +--------------------------------------------------------------------------------------+ |
| | Dashboard Widgets Rendered: | |
| | +------------+ +------------+ +------------+ +------------+ | |
| | | Severity | | Risk Score | |Reachability| | VEX Impact | | |
| | | Breakdown | | & Trend | | Donut | | Rate | | |
| | +------------+ +------------+ +------------+ +------------+ | |
| +--------------------------------------------------------------------------------------+ |
+--------------------------------------------------------------------------------------------+
Data Sources (PostgreSQL schemas):
- scanner.scan_results -> Aggregated severity counts
- policy.effective_finding -> Finding counts by status
- vuln.advisory_raw -> CVE metadata
- vex.vex_raw -> VEX suppression stats
```
### Dashboard API Endpoints
| Widget | API Endpoint | Module | Schema |
|--------|--------------|--------|--------|
| Severity Breakdown | `GET /api/console/vuln/facets` | Scanner | `scanner` |
| Risk Score & Trend | `GET /api/risk/stats` | RiskEngine | `policy` |
| Reachability Donut | `GET /api/reachability/summary` | ReachGraph | `scanner` |
| VEX Impact Rate | `GET /api/vex/suppression-rate` | Excititor | `vex` |
---
## 3. Scan Submission Flow
### CLI/API Scan Submission
```
User: stellaops scan --image registry.example.com/app:v1.0
|
v
+--------------------------------------------------------------------------------------------+
| POST /api/v1/scans/ |
| { |
| "image": { "reference": "registry.example.com/app:v1.0" }, |
| "metadata": { "tenant": "alpha" } |
| } |
+--------------------------------------------------------------------------------------------+
|
v
+--------------------------------------------------------------------------------------------+
| Scanner.WebService |
| 1. Validate request (ref OR digest required) |
| 2. Normalize ScanTarget |
| 3. Create ScanManifest in PostgreSQL (scanner schema) |
| 4. Enqueue job to Valkey Streams (scanner:jobs) |
| 5. Return 202 Accepted + scanId |
+--------------------------------------------------------------------------------------------+
|
v
+--------------------------------------------------------------------------------------------+
| Scanner.Worker (N replicas, queue-driven) |
| |
| 1. Consume job from Valkey Stream (with lease) |
| 2. Pull OCI image layers |
| 3. Check delta cache (Valkey layers:{digest}) |
| - If cached: stitch existing SBOM fragments (20ms fast path) |
| - If new: run full analysis |
| |
| 4. Execute 11 Language Analyzers: |
| +------------------------------------------------------------------------+ |
| | OS: Apk (Alpine), Dpkg (Debian), Rpm (RHEL) | |
| | Lang: Java, Node, Python, Go, .NET, Rust, Ruby, PHP, Bun, Deno | |
| | Native: ELF (Linux), PE (Windows M2), MachO (macOS M2) | |
| +------------------------------------------------------------------------+ |
| |
| 5. Extract call graphs (for reachability) |
| 6. Generate SBOM: |
| - Inventory view (all components) |
| - Usage view (entrypoint closure) |
| - Formats: CycloneDX 1.6 (JSON/Protobuf), SPDX 3.0.1 |
| |
| 7. Upload to RustFS: blobs/{sha256}/sbom.cdx.json |
| 8. Update PostgreSQL: scanner.artifacts, scanner.scan_status |
| 9. Publish event: scanner:events (scan.completed) |
+--------------------------------------------------------------------------------------------+
```
### Scan Status Polling
```
GET /api/v1/scans/{scanId}
|
v
+--------------------------------------------------------------------------------------------+
| Response: |
| { |
| "scanId": "scan-abc123", |
| "imageDigest": "sha256:...", |
| "state": "running|completed|failed", |
| "progress": { |
| "currentStage": "analyzing-java", |
| "percentage": 45, |
| "layersCurrent": 5, |
| "layersTotal": 12 |
| }, |
| "startedAt": "2025-01-02T15:30:00Z", |
| "completedAt": null, |
| "artifacts": { |
| "sbom": { "format": "cdx-json", "digest": "sha256:..." } |
| } |
| } |
+--------------------------------------------------------------------------------------------+
```
---
## 4. SBOM Generation Flow
### Docker BuildX Plugin Integration
```
docker buildx build --sbom=true ...
|
v
+--------------------------------------------------------------------------------------------+
| StellaOps.Scanner.Sbomer.BuildXPlugin (src/Scanner/...BuildXPlugin/) |
| |
| For each layer: |
| 1. Extract files |
| 2. Run analyzers (Node, Java, Python, Go, .NET, Rust, Ruby, PHP, Bun, Deno) |
| 3. Generate layer SBOM fragment |
| 4. Upload to local CAS (content-addressable store) |
| 5. Emit OCI annotation (referrer) |
| |
| CI overhead: <=300ms per layer |
+--------------------------------------------------------------------------------------------+
|
v
Image pushed to registry with SBOM referrer annotation
```
### BYOS (Bring Your Own SBOM) Upload
```
POST /api/v1/sboms/upload
{
"artifactRef": "app:v1.0",
"artifactDigest": "sha256:...",
"document": { /* CycloneDX or SPDX */ },
"format": "cdx-json"
}
|
v
+--------------------------------------------------------------------------------------------+
| SbomByosUploadService: |
| 1. Format detection + schema validation |
| 2. Component normalization (PURL, sort) |
| 3. Quality scoring + warnings |
| 4. Digest computation |
| 5. Register in artifact catalog |
| 6. Trigger scan coordination (same downstream flow) |
+--------------------------------------------------------------------------------------------+
```
---
## 5. Policy Evaluation Flow
### Scan Completion Triggers Policy
```
scan.completed event published to Valkey Streams
|
v
+--------------------------------------------------------------------------------------------+
| Event Subscribers (consume scanner:events) |
| |
| +----------------+ +----------------+ +----------------+ +----------------+ |
| | Policy.Engine | | Notify.Worker | |TimelineIndexer | | ExportCenter | |
| | (evaluate) | | (alert) | | (audit) | | (SARIF export) | |
| +-------+--------+ +----------------+ +----------------+ +----------------+ |
| | |
| v |
| +--------------------------------------------------------------------------------------+ |
| | Policy Engine Evaluation Pipeline | |
| | | |
| | Input Sources (immutable reads only - AOC enforced): | |
| | * SBOM from SbomService (RustFS) | |
| | * Advisory raw from PostgreSQL (vuln.advisory_raw) | |
| | * VEX raw from PostgreSQL (vex.vex_raw) | |
| | * Reachability lattice from Scanner call-graph | |
| | * Policy definition (stella-dsl@1) | |
| | | |
| | +--------------------------------------------------------------------------------+ | |
| | | K4 LATTICE LOGIC (Four-Valued Truth) | | |
| | | | | |
| | | Values: Unknown (bot) | True (T) | False (F) | Conflict (top) | | |
| | | | | |
| | | Knowledge Ordering: bot <= {T,F} <= top | | |
| | | | | |
| | | Conflict Detection: T join F = top | | |
| | | (Both evidence for and against -> conflict preserved, not collapsed) | | |
| | +--------------------------------------------------------------------------------+ | |
| | | |
| | Processing Steps: | |
| | 1. Load Policy IR (cached by policyId+version hash) | |
| | 2. Batch join: SBOM <-> Advisory <-> VEX (deterministic ordering) | |
| | 3. For each (component, vulnerability): | |
| | a. Compute Evidence-Weighted Score | |
| | b. Execute policy rules (first-match) | |
| | c. Apply exceptions (specificity-ranked) | |
| | d. Check unknown budget | |
| | e. Calculate confidence (5 factors): | |
| | +-----------------------------------------------------------------------+ | |
| | | Reachability (0.85 unreachable -> 0.1 confirmed reachable) | | |
| | | Runtime (0.9 supports -> 0.2 contradicts) | | |
| | | VEX (trust score x status weight) | | |
| | | Provenance (SBOM completeness) | | |
| | | Policy (exception coverage) | | |
| | | | | |
| | | Final = Clamp01(Sum of Weight x RawValue) | | |
| | +-----------------------------------------------------------------------+ | |
| | f. Emit OpenVEX decision (if verdict change) | |
| | | |
| | 4. Upsert effective findings: | |
| | - policy.effective_finding_{policyId} (current snapshot) | |
| | - policy.effective_finding_{policyId}_history (audit trail) | |
| | | |
| | 5. Compute determinism hash for replay verification | |
| | | |
| | Output: | |
| | verdict: PASS | BLOCK | WARN | FAIL | |
| | confidence: 0.0-1.0 with tier (VeryHigh/High/Medium/Low/VeryLow) | |
| | explain_trace: [rule hits + factor breakdown] | |
| | evidence_ids: [advisory_raw_ids, vex_raw_ids, sbom_digest] | |
| +--------------------------------------------------------------------------------------+ |
+--------------------------------------------------------------------------------------------+
|
v
+--------------------------------------------------------------------------------------------+
| Optional: DSSE Signing + Transparency |
| |
| Policy Engine -> Signer (DSSE envelope) |
| | |
| v |
| Attestor (in-toto predicate: stella.ops/vexDecision@v1) |
| | |
| v |
| Rekor v2 transparency log (optional) |
+--------------------------------------------------------------------------------------------+
```
---
## 6. Policy Setup Flow
```
User navigates to Policy Studio (/policy-studio)
|
v
+--------------------------------------------------------------------------------------------+
| Web UI: PolicyWorkspaceComponent |
| |
| +--------------------------------------------------------------------------------------+ |
| | Policy Studio Sub-Features: | |
| | | |
| | 1. Policy Editor (Monaco YAML) | |
| | - Visual rule builder | |
| | - stella-dsl@1 syntax | |
| | - Live validation | |
| | | |
| | 2. Policy Simulation | |
| | - Test against historical scan data | |
| | - What-if analysis | |
| | - Impact preview | |
| | | |
| | 3. Approval Workflow | |
| | - G0 (auto) -> G1 (peer) -> G2 (owner) -> G3 (manager) -> G4 (CISO) | |
| | - Multi-stage review | |
| | | |
| | 4. Policy Dashboard | |
| | - Execution history | |
| | - Metrics & trends | |
| | - Explain traces | |
| +--------------------------------------------------------------------------------------+ |
| | |
| v |
| +--------------------------------------------------------------------------------------+ |
| | Policy Schema (YAML v1.0) | |
| | | |
| | version: "1.0" | |
| | rules: | |
| | - name: Block Critical | |
| | severity: [Critical] | |
| | action: block | |
| | | |
| | - name: Ignore Low Dev | |
| | severity: [Low, None] | |
| | environments: [dev, staging] | |
| | action: ignore | |
| | expires: "2026-01-01" | |
| | | |
| | - name: Escalate Regional High | |
| | sources: [NVD, CNNVD, ENISA] | |
| | severity: [High, Critical] | |
| | action: escalate | |
| +--------------------------------------------------------------------------------------+ |
| | |
| v |
| POST /api/v1/policies/ -> Gateway -> Policy.Engine |
| | |
| v |
| +--------------------------------------------------------------------------------------+ |
| | Policy Engine Processing | |
| | | |
| | 1. Validate schema (JSON-Schema at src/Policy/__Libraries/.../Schemas/) | |
| | 2. Compile to IR (cached by policyId+version hash) | |
| | 3. Store in PostgreSQL (policy.policies) | |
| | 4. Set lifecycle state: DRAFT -> SHADOW -> ACTIVE -> ENFORCING | |
| | | |
| | Storage Tables: | |
| | * policy.policies - Policy versions, lifecycle states | |
| | * policy.policy_runs - Run records with cursors | |
| | * policy.exceptions - Approved exceptions (suppress/defer/downgrade) | |
| | * policy.exception_approval_audit - Approval chain | |
| +--------------------------------------------------------------------------------------+ |
+--------------------------------------------------------------------------------------------+
```
---
## 7. Notification Flow
```
Event published (e.g., scan.completed, advisory.delta)
|
v
+--------------------------------------------------------------------------------------------+
| Valkey Stream: scanner:events / concelier:drift / etc. |
+--------------------------------------------------------------------------------------------+
|
v
+--------------------------------------------------------------------------------------------+
| Notify.Worker consumes stream |
| 1. Evaluates notification rules (Notify.Rule) |
| 2. Matches event kind (scanner.report.ready, scheduler.rescan.delta) |
| 3. Queries user notification preferences |
| 4. Renders template (configurable per channel + locale) |
| 5. Sends via configured channels: |
| +-- Slack: POST to hooks.slack.com |
| +-- Teams: POST to graph.microsoft.com |
| +-- Email: SMTP send |
| +-- Webhook: POST to custom endpoint |
| 6. Records delivery receipt |
| 7. Idempotency tracking (prevents duplicate sends) |
| 8. Retry with deterministic backoff |
+--------------------------------------------------------------------------------------------+
```
---
## 8. Export Flow
```
User requests export
|
v
+--------------------------------------------------------------------------------------------+
| GET /api/v1/scans/{scan_id}/export?format=spdx |
+--------------------------------------------------------------------------------------------+
|
v
+--------------------------------------------------------------------------------------------+
| Scanner.Web or ExportCenter |
| 1. Queries scan metadata from PostgreSQL |
| 2. Retrieves SBOM artifact from RustFS |
| 3. Signs SBOM with Signer service (DSSE envelope) |
| 4. Creates in-toto attestation with Attestor |
| 5. Stores final bundle to RustFS |
| 6. Returns signed bundle (application/vnd.in-toto+json) |
+--------------------------------------------------------------------------------------------+
|
v
+--------------------------------------------------------------------------------------------+
| For air-gap export: |
| ExportCenter bundles: |
| - SBOMs (SPDX/CycloneDX) |
| - Offline Kit (vulnerability feeds, trust roots, signatures) |
| - Evidence bundles (DSSE-signed artifacts) |
| Generates mirror artefacts |
| Packages for portable/USB distribution |
+--------------------------------------------------------------------------------------------+
```
---
## API Endpoint Quick Reference
| User Action | Endpoint | Module | Schema |
|-------------|----------|--------|--------|
| **View Dashboard** | `GET /api/console/vuln/facets` | Scanner | `scanner` |
| | `GET /api/risk/stats` | RiskEngine | `policy` |
| | `GET /api/reachability/summary` | ReachGraph | `scanner` |
| **Submit Scan** | `POST /api/v1/scans/` | Scanner | `scanner` |
| **Check Status** | `GET /api/v1/scans/{id}` | Scanner | `scanner` |
| **Download SBOM** | `GET /api/v1/scans/{id}/sbom` | Scanner | RustFS |
| **Upload SBOM** | `POST /api/v1/sboms/upload` | Scanner | `scanner` |
| **View Findings** | `GET /api/v1/findings` | Policy | `policy` |
| **Create Policy** | `POST /api/v1/policies/` | Policy | `policy` |
| **Evaluate Policy** | `POST /api/v1/policies/evaluate` | Policy | `policy` + `vuln` + `vex` |
| **Manage Exceptions** | `POST /api/v1/exceptions/` | Policy | `policy` |
| **View VEX** | `GET /api/vex/statements` | Excititor | `vex` |
| **View Advisories** | `GET /api/advisories` | Concelier | `vuln` |
| **Export Evidence** | `GET /api/v1/export/bundle` | ExportCenter | RustFS |
---
## Related Documentation
- [Architecture Overview](../../40_ARCHITECTURE_OVERVIEW.md)
- [High-Level Architecture](../../07_HIGH_LEVEL_ARCHITECTURE.md)
- [Data Flows](data-flows.md)
- [Schema Mapping](schema-mapping.md)
- [Module Matrix](module-matrix.md)