265 lines
26 KiB
Markdown
265 lines
26 KiB
Markdown
# Request and data flows (detailed)
|
|
|
|
This document describes the canonical end-to-end flows at a level useful for debugging and auditing. Exact endpoints and payloads are defined by each module dossier under `docs/modules/`.
|
|
|
|
## 1) Scan execution (happy path)
|
|
|
|
1. **Client -> Gateway**: submit scan request (authenticated; tenant-scoped).
|
|
2. **Gateway -> Scanner.WebService**: route request after auth/rate-limit checks.
|
|
3. **Scanner.WebService -> PostgreSQL**: persist scan manifest and initial status.
|
|
4. **Scanner.WebService -> queue/stream**: enqueue a scan job (transport is profile/config dependent; for example Valkey streams or NATS).
|
|
5. **Scanner.Worker -> queue/stream**: claim job, pull image, extract layers, run analyzers.
|
|
6. **Scanner.Worker -> RustFS/S3**: write SBOM fragments, composed SBOMs, and other scan artifacts.
|
|
7. **Scanner.Worker -> Concelier**: query linksets / observations needed for evaluation (deployment-dependent).
|
|
8. **Scanner.Worker -> Scanner.WebService**: heartbeat and completion callbacks.
|
|
9. **Scanner.WebService -> Policy**: request verdict evaluation using SBOM + advisory + VEX + policy inputs.
|
|
10. **Scanner.WebService -> Signer / Attestor (optional)**: create DSSE/in-toto evidence bundles and (optionally) attach transparency receipts.
|
|
11. **Scanner.WebService -> events stream**: publish completion events for notifications and downstream consumers.
|
|
12. **Notification engine -> channels**: render and deliver notifications with idempotency tracking.
|
|
|
|
Offline note: for air-gapped deployments, step 6 writes to local object storage and step 7 relies on offline mirrors/bundles rather than public feeds. See `docs/24_OFFLINE_KIT.md` and `docs/airgap/overview.md`.
|
|
|
|
### Scan execution sequence diagram
|
|
|
|
```
|
|
┌──────────────────────────────────────────────────────────────────────────────────┐
|
|
│ 1. CLIENT REQUEST (CLI or Web UI) │
|
|
│ $ stella scan docker://alpine:latest --sbom-format=spdx │
|
|
└───────────────────────────────────┬──────────────────────────────────────────────┘
|
|
│ HTTPS
|
|
▼
|
|
┌──────────────────────────────────────────────────────────────────────────────────┐
|
|
│ 2. GATEWAY (API Router) │
|
|
│ - Terminates TLS │
|
|
│ - Routes to appropriate backend service │
|
|
│ - Load balancing (if multiple instances) │
|
|
└───────────────────────────────────┬──────────────────────────────────────────────┘
|
|
│ HTTP (internal)
|
|
▼
|
|
┌──────────────────────────────────────────────────────────────────────────────────┐
|
|
│ 3. AUTHORITY (Authentication) │
|
|
│ - Validates OAuth2 access token (DPoP-bound) │
|
|
│ - Checks DPoP proof against Valkey nonce cache │
|
|
│ - Returns user identity and scopes │
|
|
│ │
|
|
│ ┌─────────────┐ │
|
|
│ │ Valkey │◀── DPoP nonce validation (GET/SET) │
|
|
│ │ (Cache) │ │
|
|
│ └─────────────┘ │
|
|
│ ┌─────────────┐ │
|
|
│ │ PostgreSQL │◀── User/client lookup (SELECT) │
|
|
│ └─────────────┘ │
|
|
└───────────────────────────────────┬──────────────────────────────────────────────┘
|
|
│ Authenticated request
|
|
▼
|
|
┌──────────────────────────────────────────────────────────────────────────────────┐
|
|
│ 4. SCANNER.WEB (Scan API Controller) │
|
|
│ - Validates scan request parameters │
|
|
│ - Creates scan job record in PostgreSQL │
|
|
│ - Enqueues scan job to Valkey queue (default) or NATS (if configured) │
|
|
│ │
|
|
│ ┌─────────────┐ │
|
|
│ │ PostgreSQL │◀── INSERT scan_jobs (job_id, image_ref, status='pending') │
|
|
│ └─────────────┘ │
|
|
│ ┌─────────────┐ │
|
|
│ │ Valkey │◀── XADD scanner:jobs (enqueue job message) │
|
|
│ │ (Queue) │ │
|
|
│ └─────────────┘ │
|
|
│ │
|
|
│ Returns: HTTP 202 Accepted { "job_id": "scan-abc123", "status": "queued" } │
|
|
└───────────────────────────────────┬──────────────────────────────────────────────┘
|
|
│
|
|
│ (Client polls for status)
|
|
│
|
|
┌──────────────────────────────────────────────────────────────────────────────────┐
|
|
│ 5. SCANNER.WORKER (Background Processor) │
|
|
│ - Consumes job from Valkey queue (XREADGROUP scanner:jobs) │
|
|
│ - Updates job status to 'running' │
|
|
│ - Downloads container image from registry │
|
|
│ - Executes analyzers (OS packages, language deps, files) │
|
|
│ - Generates SBOM (SPDX/CycloneDX) │
|
|
│ - Stores artifacts to RustFS │
|
|
│ │
|
|
│ ┌─────────────┐ │
|
|
│ │ PostgreSQL │◀── UPDATE scan_jobs SET status='running' │
|
|
│ │ │◀── INSERT sbom_documents, packages, vulnerabilities │
|
|
│ │ │◀── UPDATE scan_jobs SET status='completed' │
|
|
│ └─────────────┘ │
|
|
│ ┌─────────────┐ │
|
|
│ │ RustFS │◀── PUT /artifacts/scan-abc123/sbom.spdx.json │
|
|
│ │ (S3 API) │◀── PUT /artifacts/scan-abc123/image-layers.tar.gz │
|
|
│ └─────────────┘ │
|
|
│ ┌─────────────┐ │
|
|
│ │ Valkey │◀── XADD scanner:events (publish scan.completed event) │
|
|
│ │(Event Stream│ │
|
|
│ └─────────────┘ │
|
|
└───────────────────────────────────┬──────────────────────────────────────────────┘
|
|
│ Event published
|
|
▼
|
|
┌──────────────────────────────────────────────────────────────────────────────────┐
|
|
│ 6. EVENT PROPAGATION (Valkey Streams) │
|
|
│ │
|
|
│ ┌───────────────────────────────────────────────────────────────────┐ │
|
|
│ │ Valkey Event Stream: "scanner:events" │ │
|
|
│ │ Event: { "type": "scan.completed", "job_id": "scan-abc123", ... } │ │
|
|
│ └───────────────────────────────────────────────────────────────────┘ │
|
|
│ │ │
|
|
│ ┌──────────────┼──────────────┬───────────────┐ │
|
|
│ ▼ ▼ ▼ ▼ │
|
|
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
|
|
│ │ Notify │ │Timeline │ │ Policy │ │ Export │ │
|
|
│ │ Worker │ │ Indexer │ │ Engine │ │ Center │ │
|
|
│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │
|
|
│ │ │ │ │ │
|
|
│ │ (all subscribe to scanner:events via XREADGROUP) │
|
|
└─────────┼───────────────┼──────────────┼───────────────┼─────────────────────────┘
|
|
│ │ │ │
|
|
▼ ▼ ▼ ▼
|
|
┌──────────────┐ ┌──────────────┐ ┌──────────┐ ┌──────────────┐
|
|
│ 7a. NOTIFY │ │ 7b. TIMELINE │ │ 7c.POLICY│ │ 7d. EXPORT │
|
|
│ │ │ INDEXER │ │ ENGINE │ │ CENTER │
|
|
│ - Query scan │ │ │ │ │ │ │
|
|
│ results │ │ - Index event│ │ - Eval │ │ - Generate │
|
|
│ - Check user │ │ timeline │ │ policy │ │ SARIF │
|
|
│ notif prefs│ │ - Store in │ │ rules │ │ - Export to │
|
|
│ - Send Slack │ │ PostgreSQL │ │ - Block/ │ │ external │
|
|
│ message │ │ │ │ Allow │ │ systems │
|
|
│ │ │ │ │ │ │ │
|
|
│ PostgreSQL ◀─┤ │ PostgreSQL ◀─┤ │PostgreSQL│ │ RustFS ◀─┤
|
|
│ (user prefs) │ │ (timeline) │ │(policies)│ │ (exports) │
|
|
└──────────────┘ └──────────────┘ └──────────┘ └──────────────┘
|
|
```
|
|
|
|
## 2) Advisory ingestion (delta-driven)
|
|
|
|
1. **Concelier.Worker** fetches advisories from configured sources (mirrors first; no hidden outbound calls in air-gap profiles).
|
|
2. **Concelier** validates and normalizes advisories, producing canonical observations and linksets.
|
|
3. **Concelier -> PostgreSQL (`vuln`)** persists immutable raw documents (append-only patterns where required) plus derived linksets.
|
|
4. **Concelier -> Scheduler** notifies about deltas (new/updated advisories) via webhook/event.
|
|
5. **Scheduler** schedules impacted re-scans or evaluations based on the delta.
|
|
|
|
## 3) VEX ingestion and consensus
|
|
|
|
1. **Excititor.Worker** fetches VEX statements from configured sources (mirrors/bundles for offline).
|
|
2. **Excititor** verifies signatures where required and normalizes statements into a canonical shape.
|
|
3. **Excititor -> PostgreSQL (`vex`)** persists immutable raw statements and consensus outcomes.
|
|
4. **Excititor -> Scheduler / Policy** emits deltas so verdicts can be recomputed deterministically.
|
|
|
|
## 4) Policy evaluation (decision trace)
|
|
|
|
1. **Caller (Scanner/UI/CLI) -> Policy.Gateway** submits evaluation request.
|
|
2. **Policy.Gateway** loads exception objects and policy snapshots from its own store.
|
|
3. **Policy Engine** consumes advisory/VEX observations (by read model, replication, or API depending on deployment) and applies deterministic precedence/lattice rules.
|
|
4. **Policy.Gateway -> caller** returns a verdict plus a trace/explain payload suitable for audits.
|
|
|
|
## 5) Notification delivery
|
|
|
|
1. **Notification engine** consumes platform events (scan completed, advisory delta, etc.).
|
|
2. **Notification engine -> queue/stream** enqueues delivery tasks with idempotency keys (when a worker model is used).
|
|
3. **Delivery workers -> channels** deliver (email/chat/webhook), record results, and retry with deterministic backoff rules.
|
|
|
|
### Notification flow diagram (vulnerability alert)
|
|
|
|
```
|
|
┌──────────────────────────────────────────────────────────────────────────────────┐
|
|
│ TRIGGER: New critical CVE detected in existing scan │
|
|
│ Source: Concelier advisory ingestion │
|
|
└───────────────────────────────────┬──────────────────────────────────────────────┘
|
|
│
|
|
▼
|
|
┌──────────────────────────────────────────────────────────────────────────────────┐
|
|
│ CONCELIER.WORKER (Advisory Processor) │
|
|
│ │
|
|
│ 1. Ingest new advisory from NVD/OSV/CSAF │
|
|
│ ┌─────────────┐ │
|
|
│ │ PostgreSQL │◀── INSERT INTO advisories (cve_id, severity, ...) │
|
|
│ └─────────────┘ │
|
|
│ │
|
|
│ 2. Match advisory against existing scans (PURL/CPE matching) │
|
|
│ ┌─────────────┐ │
|
|
│ │ PostgreSQL │◀── SELECT scans WHERE package_purl IN (affected_purls) │
|
|
│ └─────────────┘ │
|
|
│ │
|
|
│ 3. Publish drift event to Valkey │
|
|
│ ┌─────────────┐ │
|
|
│ │ Valkey │◀── XADD concelier:drift (new vulnerability found) │
|
|
│ └─────────────┘ │
|
|
└───────────────────────────────────┬──────────────────────────────────────────────┘
|
|
│ Event published
|
|
▼
|
|
┌──────────────────────────────────────────────────────────────────────────────────┐
|
|
│ NOTIFY.WORKER (Notification Processor) │
|
|
│ │
|
|
│ 1. Consume drift event from Valkey stream │
|
|
│ ┌─────────────┐ │
|
|
│ │ Valkey │◀── XREADGROUP concelier:drift notify-workers │
|
|
│ └─────────────┘ │
|
|
│ │
|
|
│ 2. Query user notification preferences │
|
|
│ ┌─────────────┐ │
|
|
│ │ PostgreSQL │◀── SELECT * FROM user_notification_preferences │
|
|
│ │ │ WHERE user_id = scan_owner AND channel = 'slack' │
|
|
│ └─────────────┘ │
|
|
│ │
|
|
│ 3. Render notification template │
|
|
│ Template: "New critical CVE-2024-1234 affects alpine:latest scan" │
|
|
│ │
|
|
│ 4. Deliver notification via configured channels │
|
|
│ ┌─────────────────────────────────────────────────────────┐ │
|
|
│ │ External APIs │ │
|
|
│ │ - POST https://hooks.slack.com/services/T00/B00/xxx │ │
|
|
│ │ - POST https://graph.microsoft.com/v1.0/teams/channels │ │
|
|
│ │ - SMTP send (email) │ │
|
|
│ └─────────────────────────────────────────────────────────┘ │
|
|
│ │
|
|
│ 5. Store delivery receipt in PostgreSQL │
|
|
│ ┌─────────────┐ │
|
|
│ │ PostgreSQL │◀── INSERT INTO notification_deliveries (status, ...) │
|
|
│ └─────────────┘ │
|
|
└──────────────────────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
## 6) Export flow (SBOM distribution)
|
|
|
|
```
|
|
┌──────────────────────────────────────────────────────────────────────────────────┐
|
|
│ EXPORT REQUEST: GET /api/v1/scans/{scan_id}/export?format=spdx │
|
|
└───────────────────────────────────┬──────────────────────────────────────────────┘
|
|
│
|
|
▼
|
|
┌──────────────────────────────────────────────────────────────────────────────────┐
|
|
│ SCANNER.WEB or EXPORT CENTER │
|
|
│ │
|
|
│ 1. Query scan metadata from PostgreSQL │
|
|
│ ┌─────────────┐ │
|
|
│ │ PostgreSQL │◀── SELECT * FROM scan_jobs WHERE job_id = $1 │
|
|
│ │ │◀── SELECT * FROM sbom_documents WHERE scan_id = $1 │
|
|
│ └─────────────┘ │
|
|
│ │
|
|
│ 2. Retrieve SBOM artifact from RustFS │
|
|
│ ┌─────────────┐ │
|
|
│ │ RustFS │◀── GET /artifacts/scan-abc123/sbom.spdx.json │
|
|
│ └─────────────┘ │
|
|
│ │
|
|
│ 3. Sign SBOM with Signer service │
|
|
│ ┌─────────────┐ │
|
|
│ │ Signer │◀── POST /api/v1/sign (SBOM payload) │
|
|
│ │ │──▶ Returns: DSSE envelope with signature │
|
|
│ └─────────────┘ │
|
|
│ │
|
|
│ 4. Create in-toto attestation with Attestor │
|
|
│ ┌─────────────┐ │
|
|
│ │ Attestor │◀── POST /api/v1/attest (signed SBOM) │
|
|
│ │ │──▶ Returns: in-toto attestation bundle │
|
|
│ └─────────────┘ │
|
|
│ │
|
|
│ 5. Store final bundle to RustFS │
|
|
│ ┌─────────────┐ │
|
|
│ │ RustFS │◀── PUT /artifacts/scan-abc123/bundle.jsonl │
|
|
│ └─────────────┘ │
|
|
│ │
|
|
│ 6. Return signed bundle to client │
|
|
│ Returns: HTTP 200 OK (application/vnd.in-toto+json) │
|
|
└──────────────────────────────────────────────────────────────────────────────────┘
|
|
```
|