26 KiB
26 KiB
Request and data flows (detailed)
This document describes the canonical end-to-end flows at a level useful for debugging and auditing. Exact endpoints and payloads are defined by each module dossier under docs/modules/.
1) Scan execution (happy path)
- Client -> Gateway: submit scan request (authenticated; tenant-scoped).
- Gateway -> Scanner.WebService: route request after auth/rate-limit checks.
- Scanner.WebService -> PostgreSQL: persist scan manifest and initial status.
- Scanner.WebService -> queue/stream: enqueue a scan job (transport is profile/config dependent; for example Valkey streams or NATS).
- Scanner.Worker -> queue/stream: claim job, pull image, extract layers, run analyzers.
- Scanner.Worker -> RustFS/S3: write SBOM fragments, composed SBOMs, and other scan artifacts.
- Scanner.Worker -> Concelier: query linksets / observations needed for evaluation (deployment-dependent).
- Scanner.Worker -> Scanner.WebService: heartbeat and completion callbacks.
- Scanner.WebService -> Policy: request verdict evaluation using SBOM + advisory + VEX + policy inputs.
- Scanner.WebService -> Signer / Attestor (optional): create DSSE/in-toto evidence bundles and (optionally) attach transparency receipts.
- Scanner.WebService -> events stream: publish completion events for notifications and downstream consumers.
- Notification engine -> channels: render and deliver notifications with idempotency tracking.
Offline note: for air-gapped deployments, step 6 writes to local object storage and step 7 relies on offline mirrors/bundles rather than public feeds. See docs/24_OFFLINE_KIT.md and docs/airgap/overview.md.
Scan execution sequence diagram
┌──────────────────────────────────────────────────────────────────────────────────┐
│ 1. CLIENT REQUEST (CLI or Web UI) │
│ $ stella scan docker://alpine:latest --sbom-format=spdx │
└───────────────────────────────────┬──────────────────────────────────────────────┘
│ HTTPS
▼
┌──────────────────────────────────────────────────────────────────────────────────┐
│ 2. GATEWAY (API Router) │
│ - Terminates TLS │
│ - Routes to appropriate backend service │
│ - Load balancing (if multiple instances) │
└───────────────────────────────────┬──────────────────────────────────────────────┘
│ HTTP (internal)
▼
┌──────────────────────────────────────────────────────────────────────────────────┐
│ 3. AUTHORITY (Authentication) │
│ - Validates OAuth2 access token (DPoP-bound) │
│ - Checks DPoP proof against Valkey nonce cache │
│ - Returns user identity and scopes │
│ │
│ ┌─────────────┐ │
│ │ Valkey │◀── DPoP nonce validation (GET/SET) │
│ │ (Cache) │ │
│ └─────────────┘ │
│ ┌─────────────┐ │
│ │ PostgreSQL │◀── User/client lookup (SELECT) │
│ └─────────────┘ │
└───────────────────────────────────┬──────────────────────────────────────────────┘
│ Authenticated request
▼
┌──────────────────────────────────────────────────────────────────────────────────┐
│ 4. SCANNER.WEB (Scan API Controller) │
│ - Validates scan request parameters │
│ - Creates scan job record in PostgreSQL │
│ - Enqueues scan job to Valkey queue (default) or NATS (if configured) │
│ │
│ ┌─────────────┐ │
│ │ PostgreSQL │◀── INSERT scan_jobs (job_id, image_ref, status='pending') │
│ └─────────────┘ │
│ ┌─────────────┐ │
│ │ Valkey │◀── XADD scanner:jobs (enqueue job message) │
│ │ (Queue) │ │
│ └─────────────┘ │
│ │
│ Returns: HTTP 202 Accepted { "job_id": "scan-abc123", "status": "queued" } │
└───────────────────────────────────┬──────────────────────────────────────────────┘
│
│ (Client polls for status)
│
┌──────────────────────────────────────────────────────────────────────────────────┐
│ 5. SCANNER.WORKER (Background Processor) │
│ - Consumes job from Valkey queue (XREADGROUP scanner:jobs) │
│ - Updates job status to 'running' │
│ - Downloads container image from registry │
│ - Executes analyzers (OS packages, language deps, files) │
│ - Generates SBOM (SPDX/CycloneDX) │
│ - Stores artifacts to RustFS │
│ │
│ ┌─────────────┐ │
│ │ PostgreSQL │◀── UPDATE scan_jobs SET status='running' │
│ │ │◀── INSERT sbom_documents, packages, vulnerabilities │
│ │ │◀── UPDATE scan_jobs SET status='completed' │
│ └─────────────┘ │
│ ┌─────────────┐ │
│ │ RustFS │◀── PUT /artifacts/scan-abc123/sbom.spdx.json │
│ │ (S3 API) │◀── PUT /artifacts/scan-abc123/image-layers.tar.gz │
│ └─────────────┘ │
│ ┌─────────────┐ │
│ │ Valkey │◀── XADD scanner:events (publish scan.completed event) │
│ │(Event Stream│ │
│ └─────────────┘ │
└───────────────────────────────────┬──────────────────────────────────────────────┘
│ Event published
▼
┌──────────────────────────────────────────────────────────────────────────────────┐
│ 6. EVENT PROPAGATION (Valkey Streams) │
│ │
│ ┌───────────────────────────────────────────────────────────────────┐ │
│ │ Valkey Event Stream: "scanner:events" │ │
│ │ Event: { "type": "scan.completed", "job_id": "scan-abc123", ... } │ │
│ └───────────────────────────────────────────────────────────────────┘ │
│ │ │
│ ┌──────────────┼──────────────┬───────────────┐ │
│ ▼ ▼ ▼ ▼ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ Notify │ │Timeline │ │ Policy │ │ Export │ │
│ │ Worker │ │ Indexer │ │ Engine │ │ Center │ │
│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │
│ │ │ │ │ │
│ │ (all subscribe to scanner:events via XREADGROUP) │
└─────────┼───────────────┼──────────────┼───────────────┼─────────────────────────┘
│ │ │ │
▼ ▼ ▼ ▼
┌──────────────┐ ┌──────────────┐ ┌──────────┐ ┌──────────────┐
│ 7a. NOTIFY │ │ 7b. TIMELINE │ │ 7c.POLICY│ │ 7d. EXPORT │
│ │ │ INDEXER │ │ ENGINE │ │ CENTER │
│ - Query scan │ │ │ │ │ │ │
│ results │ │ - Index event│ │ - Eval │ │ - Generate │
│ - Check user │ │ timeline │ │ policy │ │ SARIF │
│ notif prefs│ │ - Store in │ │ rules │ │ - Export to │
│ - Send Slack │ │ PostgreSQL │ │ - Block/ │ │ external │
│ message │ │ │ │ Allow │ │ systems │
│ │ │ │ │ │ │ │
│ PostgreSQL ◀─┤ │ PostgreSQL ◀─┤ │PostgreSQL│ │ RustFS ◀─┤
│ (user prefs) │ │ (timeline) │ │(policies)│ │ (exports) │
└──────────────┘ └──────────────┘ └──────────┘ └──────────────┘
2) Advisory ingestion (delta-driven)
- Concelier.Worker fetches advisories from configured sources (mirrors first; no hidden outbound calls in air-gap profiles).
- Concelier validates and normalizes advisories, producing canonical observations and linksets.
- Concelier -> PostgreSQL (
vuln) persists immutable raw documents (append-only patterns where required) plus derived linksets. - Concelier -> Scheduler notifies about deltas (new/updated advisories) via webhook/event.
- Scheduler schedules impacted re-scans or evaluations based on the delta.
3) VEX ingestion and consensus
- Excititor.Worker fetches VEX statements from configured sources (mirrors/bundles for offline).
- Excititor verifies signatures where required and normalizes statements into a canonical shape.
- Excititor -> PostgreSQL (
vex) persists immutable raw statements and consensus outcomes. - Excititor -> Scheduler / Policy emits deltas so verdicts can be recomputed deterministically.
4) Policy evaluation (decision trace)
- Caller (Scanner/UI/CLI) -> Policy.Gateway submits evaluation request.
- Policy.Gateway loads exception objects and policy snapshots from its own store.
- Policy Engine consumes advisory/VEX observations (by read model, replication, or API depending on deployment) and applies deterministic precedence/lattice rules.
- Policy.Gateway -> caller returns a verdict plus a trace/explain payload suitable for audits.
5) Notification delivery
- Notification engine consumes platform events (scan completed, advisory delta, etc.).
- Notification engine -> queue/stream enqueues delivery tasks with idempotency keys (when a worker model is used).
- Delivery workers -> channels deliver (email/chat/webhook), record results, and retry with deterministic backoff rules.
Notification flow diagram (vulnerability alert)
┌──────────────────────────────────────────────────────────────────────────────────┐
│ TRIGGER: New critical CVE detected in existing scan │
│ Source: Concelier advisory ingestion │
└───────────────────────────────────┬──────────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────────────────────────┐
│ CONCELIER.WORKER (Advisory Processor) │
│ │
│ 1. Ingest new advisory from NVD/OSV/CSAF │
│ ┌─────────────┐ │
│ │ PostgreSQL │◀── INSERT INTO advisories (cve_id, severity, ...) │
│ └─────────────┘ │
│ │
│ 2. Match advisory against existing scans (PURL/CPE matching) │
│ ┌─────────────┐ │
│ │ PostgreSQL │◀── SELECT scans WHERE package_purl IN (affected_purls) │
│ └─────────────┘ │
│ │
│ 3. Publish drift event to Valkey │
│ ┌─────────────┐ │
│ │ Valkey │◀── XADD concelier:drift (new vulnerability found) │
│ └─────────────┘ │
└───────────────────────────────────┬──────────────────────────────────────────────┘
│ Event published
▼
┌──────────────────────────────────────────────────────────────────────────────────┐
│ NOTIFY.WORKER (Notification Processor) │
│ │
│ 1. Consume drift event from Valkey stream │
│ ┌─────────────┐ │
│ │ Valkey │◀── XREADGROUP concelier:drift notify-workers │
│ └─────────────┘ │
│ │
│ 2. Query user notification preferences │
│ ┌─────────────┐ │
│ │ PostgreSQL │◀── SELECT * FROM user_notification_preferences │
│ │ │ WHERE user_id = scan_owner AND channel = 'slack' │
│ └─────────────┘ │
│ │
│ 3. Render notification template │
│ Template: "New critical CVE-2024-1234 affects alpine:latest scan" │
│ │
│ 4. Deliver notification via configured channels │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ External APIs │ │
│ │ - POST https://hooks.slack.com/services/T00/B00/xxx │ │
│ │ - POST https://graph.microsoft.com/v1.0/teams/channels │ │
│ │ - SMTP send (email) │ │
│ └─────────────────────────────────────────────────────────┘ │
│ │
│ 5. Store delivery receipt in PostgreSQL │
│ ┌─────────────┐ │
│ │ PostgreSQL │◀── INSERT INTO notification_deliveries (status, ...) │
│ └─────────────┘ │
└──────────────────────────────────────────────────────────────────────────────────┘
6) Export flow (SBOM distribution)
┌──────────────────────────────────────────────────────────────────────────────────┐
│ EXPORT REQUEST: GET /api/v1/scans/{scan_id}/export?format=spdx │
└───────────────────────────────────┬──────────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────────────────────────┐
│ SCANNER.WEB or EXPORT CENTER │
│ │
│ 1. Query scan metadata from PostgreSQL │
│ ┌─────────────┐ │
│ │ PostgreSQL │◀── SELECT * FROM scan_jobs WHERE job_id = $1 │
│ │ │◀── SELECT * FROM sbom_documents WHERE scan_id = $1 │
│ └─────────────┘ │
│ │
│ 2. Retrieve SBOM artifact from RustFS │
│ ┌─────────────┐ │
│ │ RustFS │◀── GET /artifacts/scan-abc123/sbom.spdx.json │
│ └─────────────┘ │
│ │
│ 3. Sign SBOM with Signer service │
│ ┌─────────────┐ │
│ │ Signer │◀── POST /api/v1/sign (SBOM payload) │
│ │ │──▶ Returns: DSSE envelope with signature │
│ └─────────────┘ │
│ │
│ 4. Create in-toto attestation with Attestor │
│ ┌─────────────┐ │
│ │ Attestor │◀── POST /api/v1/attest (signed SBOM) │
│ │ │──▶ Returns: in-toto attestation bundle │
│ └─────────────┘ │
│ │
│ 5. Store final bundle to RustFS │
│ ┌─────────────┐ │
│ │ RustFS │◀── PUT /artifacts/scan-abc123/bundle.jsonl │
│ └─────────────┘ │
│ │
│ 6. Return signed bundle to client │
│ Returns: HTTP 200 OK (application/vnd.in-toto+json) │
└──────────────────────────────────────────────────────────────────────────────────┘