# Request and data flows (detailed) This document describes the canonical end-to-end flows at a level useful for debugging and auditing. Exact endpoints and payloads are defined by each module dossier under `docs/modules/`. ## 1) Scan execution (happy path) 1. **Client -> Gateway**: submit scan request (authenticated; tenant-scoped). 2. **Gateway -> Scanner.WebService**: route request after auth/rate-limit checks. 3. **Scanner.WebService -> PostgreSQL**: persist scan manifest and initial status. 4. **Scanner.WebService -> queue/stream**: enqueue a scan job (transport is profile/config dependent; for example Valkey streams or NATS). 5. **Scanner.Worker -> queue/stream**: claim job, pull image, extract layers, run analyzers. 6. **Scanner.Worker -> RustFS/S3**: write SBOM fragments, composed SBOMs, and other scan artifacts. 7. **Scanner.Worker -> Concelier**: query linksets / observations needed for evaluation (deployment-dependent). 8. **Scanner.Worker -> Scanner.WebService**: heartbeat and completion callbacks. 9. **Scanner.WebService -> Policy**: request verdict evaluation using SBOM + advisory + VEX + policy inputs. 10. **Scanner.WebService -> Signer / Attestor (optional)**: create DSSE/in-toto evidence bundles and (optionally) attach transparency receipts. 11. **Scanner.WebService -> events stream**: publish completion events for notifications and downstream consumers. 12. **Notification engine -> channels**: render and deliver notifications with idempotency tracking. Offline note: for air-gapped deployments, step 6 writes to local object storage and step 7 relies on offline mirrors/bundles rather than public feeds. See `docs/24_OFFLINE_KIT.md` and `docs/airgap/overview.md`. ### Scan execution sequence diagram ``` ┌──────────────────────────────────────────────────────────────────────────────────┐ │ 1. CLIENT REQUEST (CLI or Web UI) │ │ $ stella scan docker://alpine:latest --sbom-format=spdx │ └───────────────────────────────────┬──────────────────────────────────────────────┘ │ HTTPS ▼ ┌──────────────────────────────────────────────────────────────────────────────────┐ │ 2. GATEWAY (API Router) │ │ - Terminates TLS │ │ - Routes to appropriate backend service │ │ - Load balancing (if multiple instances) │ └───────────────────────────────────┬──────────────────────────────────────────────┘ │ HTTP (internal) ▼ ┌──────────────────────────────────────────────────────────────────────────────────┐ │ 3. AUTHORITY (Authentication) │ │ - Validates OAuth2 access token (DPoP-bound) │ │ - Checks DPoP proof against Valkey nonce cache │ │ - Returns user identity and scopes │ │ │ │ ┌─────────────┐ │ │ │ Valkey │◀── DPoP nonce validation (GET/SET) │ │ │ (Cache) │ │ │ └─────────────┘ │ │ ┌─────────────┐ │ │ │ PostgreSQL │◀── User/client lookup (SELECT) │ │ └─────────────┘ │ └───────────────────────────────────┬──────────────────────────────────────────────┘ │ Authenticated request ▼ ┌──────────────────────────────────────────────────────────────────────────────────┐ │ 4. SCANNER.WEB (Scan API Controller) │ │ - Validates scan request parameters │ │ - Creates scan job record in PostgreSQL │ │ - Enqueues scan job to Valkey queue (default) or NATS (if configured) │ │ │ │ ┌─────────────┐ │ │ │ PostgreSQL │◀── INSERT scan_jobs (job_id, image_ref, status='pending') │ │ └─────────────┘ │ │ ┌─────────────┐ │ │ │ Valkey │◀── XADD scanner:jobs (enqueue job message) │ │ │ (Queue) │ │ │ └─────────────┘ │ │ │ │ Returns: HTTP 202 Accepted { "job_id": "scan-abc123", "status": "queued" } │ └───────────────────────────────────┬──────────────────────────────────────────────┘ │ │ (Client polls for status) │ ┌──────────────────────────────────────────────────────────────────────────────────┐ │ 5. SCANNER.WORKER (Background Processor) │ │ - Consumes job from Valkey queue (XREADGROUP scanner:jobs) │ │ - Updates job status to 'running' │ │ - Downloads container image from registry │ │ - Executes analyzers (OS packages, language deps, files) │ │ - Generates SBOM (SPDX/CycloneDX) │ │ - Stores artifacts to RustFS │ │ │ │ ┌─────────────┐ │ │ │ PostgreSQL │◀── UPDATE scan_jobs SET status='running' │ │ │ │◀── INSERT sbom_documents, packages, vulnerabilities │ │ │ │◀── UPDATE scan_jobs SET status='completed' │ │ └─────────────┘ │ │ ┌─────────────┐ │ │ │ RustFS │◀── PUT /artifacts/scan-abc123/sbom.spdx.json │ │ │ (S3 API) │◀── PUT /artifacts/scan-abc123/image-layers.tar.gz │ │ └─────────────┘ │ │ ┌─────────────┐ │ │ │ Valkey │◀── XADD scanner:events (publish scan.completed event) │ │ │(Event Stream│ │ │ └─────────────┘ │ └───────────────────────────────────┬──────────────────────────────────────────────┘ │ Event published ▼ ┌──────────────────────────────────────────────────────────────────────────────────┐ │ 6. EVENT PROPAGATION (Valkey Streams) │ │ │ │ ┌───────────────────────────────────────────────────────────────────┐ │ │ │ Valkey Event Stream: "scanner:events" │ │ │ │ Event: { "type": "scan.completed", "job_id": "scan-abc123", ... } │ │ │ └───────────────────────────────────────────────────────────────────┘ │ │ │ │ │ ┌──────────────┼──────────────┬───────────────┐ │ │ ▼ ▼ ▼ ▼ │ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │ │ Notify │ │Timeline │ │ Policy │ │ Export │ │ │ │ Worker │ │ Indexer │ │ Engine │ │ Center │ │ │ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │ │ │ │ │ │ │ │ │ (all subscribe to scanner:events via XREADGROUP) │ └─────────┼───────────────┼──────────────┼───────────────┼─────────────────────────┘ │ │ │ │ ▼ ▼ ▼ ▼ ┌──────────────┐ ┌──────────────┐ ┌──────────┐ ┌──────────────┐ │ 7a. NOTIFY │ │ 7b. TIMELINE │ │ 7c.POLICY│ │ 7d. EXPORT │ │ │ │ INDEXER │ │ ENGINE │ │ CENTER │ │ - Query scan │ │ │ │ │ │ │ │ results │ │ - Index event│ │ - Eval │ │ - Generate │ │ - Check user │ │ timeline │ │ policy │ │ SARIF │ │ notif prefs│ │ - Store in │ │ rules │ │ - Export to │ │ - Send Slack │ │ PostgreSQL │ │ - Block/ │ │ external │ │ message │ │ │ │ Allow │ │ systems │ │ │ │ │ │ │ │ │ │ PostgreSQL ◀─┤ │ PostgreSQL ◀─┤ │PostgreSQL│ │ RustFS ◀─┤ │ (user prefs) │ │ (timeline) │ │(policies)│ │ (exports) │ └──────────────┘ └──────────────┘ └──────────┘ └──────────────┘ ``` ## 2) Advisory ingestion (delta-driven) 1. **Concelier.Worker** fetches advisories from configured sources (mirrors first; no hidden outbound calls in air-gap profiles). 2. **Concelier** validates and normalizes advisories, producing canonical observations and linksets. 3. **Concelier -> PostgreSQL (`vuln`)** persists immutable raw documents (append-only patterns where required) plus derived linksets. 4. **Concelier -> Scheduler** notifies about deltas (new/updated advisories) via webhook/event. 5. **Scheduler** schedules impacted re-scans or evaluations based on the delta. ## 3) VEX ingestion and consensus 1. **Excititor.Worker** fetches VEX statements from configured sources (mirrors/bundles for offline). 2. **Excititor** verifies signatures where required and normalizes statements into a canonical shape. 3. **Excititor -> PostgreSQL (`vex`)** persists immutable raw statements and consensus outcomes. 4. **Excititor -> Scheduler / Policy** emits deltas so verdicts can be recomputed deterministically. ## 4) Policy evaluation (decision trace) 1. **Caller (Scanner/UI/CLI) -> Policy.Gateway** submits evaluation request. 2. **Policy.Gateway** loads exception objects and policy snapshots from its own store. 3. **Policy Engine** consumes advisory/VEX observations (by read model, replication, or API depending on deployment) and applies deterministic precedence/lattice rules. 4. **Policy.Gateway -> caller** returns a verdict plus a trace/explain payload suitable for audits. ## 5) Notification delivery 1. **Notification engine** consumes platform events (scan completed, advisory delta, etc.). 2. **Notification engine -> queue/stream** enqueues delivery tasks with idempotency keys (when a worker model is used). 3. **Delivery workers -> channels** deliver (email/chat/webhook), record results, and retry with deterministic backoff rules. ### Notification flow diagram (vulnerability alert) ``` ┌──────────────────────────────────────────────────────────────────────────────────┐ │ TRIGGER: New critical CVE detected in existing scan │ │ Source: Concelier advisory ingestion │ └───────────────────────────────────┬──────────────────────────────────────────────┘ │ ▼ ┌──────────────────────────────────────────────────────────────────────────────────┐ │ CONCELIER.WORKER (Advisory Processor) │ │ │ │ 1. Ingest new advisory from NVD/OSV/CSAF │ │ ┌─────────────┐ │ │ │ PostgreSQL │◀── INSERT INTO advisories (cve_id, severity, ...) │ │ └─────────────┘ │ │ │ │ 2. Match advisory against existing scans (PURL/CPE matching) │ │ ┌─────────────┐ │ │ │ PostgreSQL │◀── SELECT scans WHERE package_purl IN (affected_purls) │ │ └─────────────┘ │ │ │ │ 3. Publish drift event to Valkey │ │ ┌─────────────┐ │ │ │ Valkey │◀── XADD concelier:drift (new vulnerability found) │ │ └─────────────┘ │ └───────────────────────────────────┬──────────────────────────────────────────────┘ │ Event published ▼ ┌──────────────────────────────────────────────────────────────────────────────────┐ │ NOTIFY.WORKER (Notification Processor) │ │ │ │ 1. Consume drift event from Valkey stream │ │ ┌─────────────┐ │ │ │ Valkey │◀── XREADGROUP concelier:drift notify-workers │ │ └─────────────┘ │ │ │ │ 2. Query user notification preferences │ │ ┌─────────────┐ │ │ │ PostgreSQL │◀── SELECT * FROM user_notification_preferences │ │ │ │ WHERE user_id = scan_owner AND channel = 'slack' │ │ └─────────────┘ │ │ │ │ 3. Render notification template │ │ Template: "New critical CVE-2024-1234 affects alpine:latest scan" │ │ │ │ 4. Deliver notification via configured channels │ │ ┌─────────────────────────────────────────────────────────┐ │ │ │ External APIs │ │ │ │ - POST https://hooks.slack.com/services/T00/B00/xxx │ │ │ │ - POST https://graph.microsoft.com/v1.0/teams/channels │ │ │ │ - SMTP send (email) │ │ │ └─────────────────────────────────────────────────────────┘ │ │ │ │ 5. Store delivery receipt in PostgreSQL │ │ ┌─────────────┐ │ │ │ PostgreSQL │◀── INSERT INTO notification_deliveries (status, ...) │ │ └─────────────┘ │ └──────────────────────────────────────────────────────────────────────────────────┘ ``` ## 6) Export flow (SBOM distribution) ``` ┌──────────────────────────────────────────────────────────────────────────────────┐ │ EXPORT REQUEST: GET /api/v1/scans/{scan_id}/export?format=spdx │ └───────────────────────────────────┬──────────────────────────────────────────────┘ │ ▼ ┌──────────────────────────────────────────────────────────────────────────────────┐ │ SCANNER.WEB or EXPORT CENTER │ │ │ │ 1. Query scan metadata from PostgreSQL │ │ ┌─────────────┐ │ │ │ PostgreSQL │◀── SELECT * FROM scan_jobs WHERE job_id = $1 │ │ │ │◀── SELECT * FROM sbom_documents WHERE scan_id = $1 │ │ └─────────────┘ │ │ │ │ 2. Retrieve SBOM artifact from RustFS │ │ ┌─────────────┐ │ │ │ RustFS │◀── GET /artifacts/scan-abc123/sbom.spdx.json │ │ └─────────────┘ │ │ │ │ 3. Sign SBOM with Signer service │ │ ┌─────────────┐ │ │ │ Signer │◀── POST /api/v1/sign (SBOM payload) │ │ │ │──▶ Returns: DSSE envelope with signature │ │ └─────────────┘ │ │ │ │ 4. Create in-toto attestation with Attestor │ │ ┌─────────────┐ │ │ │ Attestor │◀── POST /api/v1/attest (signed SBOM) │ │ │ │──▶ Returns: in-toto attestation bundle │ │ └─────────────┘ │ │ │ │ 5. Store final bundle to RustFS │ │ ┌─────────────┐ │ │ │ RustFS │◀── PUT /artifacts/scan-abc123/bundle.jsonl │ │ └─────────────┘ │ │ │ │ 6. Return signed bundle to client │ │ Returns: HTTP 200 OK (application/vnd.in-toto+json) │ └──────────────────────────────────────────────────────────────────────────────────┘ ```