docs consolidation
This commit is contained in:
StellaOps Bot
@@ -7,7 +7,7 @@ This document describes the canonical end-to-end flows at a level useful for deb
|
||||
1. **Client -> Gateway**: submit scan request (authenticated; tenant-scoped).
|
||||
2. **Gateway -> Scanner.WebService**: route request after auth/rate-limit checks.
|
||||
3. **Scanner.WebService -> PostgreSQL**: persist scan manifest and initial status.
|
||||
4. **Scanner.WebService -> queue/stream**: enqueue a scan job (Valkey streams by default; optional alternative transports exist).
|
||||
4. **Scanner.WebService -> queue/stream**: enqueue a scan job (transport is profile/config dependent; for example Valkey streams or NATS).
|
||||
5. **Scanner.Worker -> queue/stream**: claim job, pull image, extract layers, run analyzers.
|
||||
6. **Scanner.Worker -> RustFS/S3**: write SBOM fragments, composed SBOMs, and other scan artifacts.
|
||||
7. **Scanner.Worker -> Concelier**: query linksets / observations needed for evaluation (deployment-dependent).
|
||||
@@ -15,10 +15,121 @@ This document describes the canonical end-to-end flows at a level useful for deb
|
||||
9. **Scanner.WebService -> Policy**: request verdict evaluation using SBOM + advisory + VEX + policy inputs.
|
||||
10. **Scanner.WebService -> Signer / Attestor (optional)**: create DSSE/in-toto evidence bundles and (optionally) attach transparency receipts.
|
||||
11. **Scanner.WebService -> events stream**: publish completion events for notifications and downstream consumers.
|
||||
12. **Notify.WebService/Worker -> channels**: render and deliver notifications with idempotency tracking.
|
||||
12. **Notification engine -> channels**: render and deliver notifications with idempotency tracking.
|
||||
|
||||
Offline note: for air-gapped deployments, step 6 writes to local object storage and step 7 relies on offline mirrors/bundles rather than public feeds. See `docs/24_OFFLINE_KIT.md` and `docs/airgap/overview.md`.
|
||||
|
||||
### Scan execution sequence diagram
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────────────────────────────────────┐
|
||||
│ 1. CLIENT REQUEST (CLI or Web UI) │
|
||||
│ $ stella scan docker://alpine:latest --sbom-format=spdx │
|
||||
└───────────────────────────────────┬──────────────────────────────────────────────┘
|
||||
│ HTTPS
|
||||
▼
|
||||
┌──────────────────────────────────────────────────────────────────────────────────┐
|
||||
│ 2. GATEWAY (API Router) │
|
||||
│ - Terminates TLS │
|
||||
│ - Routes to appropriate backend service │
|
||||
│ - Load balancing (if multiple instances) │
|
||||
└───────────────────────────────────┬──────────────────────────────────────────────┘
|
||||
│ HTTP (internal)
|
||||
▼
|
||||
┌──────────────────────────────────────────────────────────────────────────────────┐
|
||||
│ 3. AUTHORITY (Authentication) │
|
||||
│ - Validates OAuth2 access token (DPoP-bound) │
|
||||
│ - Checks DPoP proof against Valkey nonce cache │
|
||||
│ - Returns user identity and scopes │
|
||||
│ │
|
||||
│ ┌─────────────┐ │
|
||||
│ │ Valkey │◀── DPoP nonce validation (GET/SET) │
|
||||
│ │ (Cache) │ │
|
||||
│ └─────────────┘ │
|
||||
│ ┌─────────────┐ │
|
||||
│ │ PostgreSQL │◀── User/client lookup (SELECT) │
|
||||
│ └─────────────┘ │
|
||||
└───────────────────────────────────┬──────────────────────────────────────────────┘
|
||||
│ Authenticated request
|
||||
▼
|
||||
┌──────────────────────────────────────────────────────────────────────────────────┐
|
||||
│ 4. SCANNER.WEB (Scan API Controller) │
|
||||
│ - Validates scan request parameters │
|
||||
│ - Creates scan job record in PostgreSQL │
|
||||
│ - Enqueues scan job to Valkey queue (default) or NATS (if configured) │
|
||||
│ │
|
||||
│ ┌─────────────┐ │
|
||||
│ │ PostgreSQL │◀── INSERT scan_jobs (job_id, image_ref, status='pending') │
|
||||
│ └─────────────┘ │
|
||||
│ ┌─────────────┐ │
|
||||
│ │ Valkey │◀── XADD scanner:jobs (enqueue job message) │
|
||||
│ │ (Queue) │ │
|
||||
│ └─────────────┘ │
|
||||
│ │
|
||||
│ Returns: HTTP 202 Accepted { "job_id": "scan-abc123", "status": "queued" } │
|
||||
└───────────────────────────────────┬──────────────────────────────────────────────┘
|
||||
│
|
||||
│ (Client polls for status)
|
||||
│
|
||||
┌──────────────────────────────────────────────────────────────────────────────────┐
|
||||
│ 5. SCANNER.WORKER (Background Processor) │
|
||||
│ - Consumes job from Valkey queue (XREADGROUP scanner:jobs) │
|
||||
│ - Updates job status to 'running' │
|
||||
│ - Downloads container image from registry │
|
||||
│ - Executes analyzers (OS packages, language deps, files) │
|
||||
│ - Generates SBOM (SPDX/CycloneDX) │
|
||||
│ - Stores artifacts to RustFS │
|
||||
│ │
|
||||
│ ┌─────────────┐ │
|
||||
│ │ PostgreSQL │◀── UPDATE scan_jobs SET status='running' │
|
||||
│ │ │◀── INSERT sbom_documents, packages, vulnerabilities │
|
||||
│ │ │◀── UPDATE scan_jobs SET status='completed' │
|
||||
│ └─────────────┘ │
|
||||
│ ┌─────────────┐ │
|
||||
│ │ RustFS │◀── PUT /artifacts/scan-abc123/sbom.spdx.json │
|
||||
│ │ (S3 API) │◀── PUT /artifacts/scan-abc123/image-layers.tar.gz │
|
||||
│ └─────────────┘ │
|
||||
│ ┌─────────────┐ │
|
||||
│ │ Valkey │◀── XADD scanner:events (publish scan.completed event) │
|
||||
│ │(Event Stream│ │
|
||||
│ └─────────────┘ │
|
||||
└───────────────────────────────────┬──────────────────────────────────────────────┘
|
||||
│ Event published
|
||||
▼
|
||||
┌──────────────────────────────────────────────────────────────────────────────────┐
|
||||
│ 6. EVENT PROPAGATION (Valkey Streams) │
|
||||
│ │
|
||||
│ ┌───────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ Valkey Event Stream: "scanner:events" │ │
|
||||
│ │ Event: { "type": "scan.completed", "job_id": "scan-abc123", ... } │ │
|
||||
│ └───────────────────────────────────────────────────────────────────┘ │
|
||||
│ │ │
|
||||
│ ┌──────────────┼──────────────┬───────────────┐ │
|
||||
│ ▼ ▼ ▼ ▼ │
|
||||
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
|
||||
│ │ Notify │ │Timeline │ │ Policy │ │ Export │ │
|
||||
│ │ Worker │ │ Indexer │ │ Engine │ │ Center │ │
|
||||
│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │
|
||||
│ │ │ │ │ │
|
||||
│ │ (all subscribe to scanner:events via XREADGROUP) │
|
||||
└─────────┼───────────────┼──────────────┼───────────────┼─────────────────────────┘
|
||||
│ │ │ │
|
||||
▼ ▼ ▼ ▼
|
||||
┌──────────────┐ ┌──────────────┐ ┌──────────┐ ┌──────────────┐
|
||||
│ 7a. NOTIFY │ │ 7b. TIMELINE │ │ 7c.POLICY│ │ 7d. EXPORT │
|
||||
│ │ │ INDEXER │ │ ENGINE │ │ CENTER │
|
||||
│ - Query scan │ │ │ │ │ │ │
|
||||
│ results │ │ - Index event│ │ - Eval │ │ - Generate │
|
||||
│ - Check user │ │ timeline │ │ policy │ │ SARIF │
|
||||
│ notif prefs│ │ - Store in │ │ rules │ │ - Export to │
|
||||
│ - Send Slack │ │ PostgreSQL │ │ - Block/ │ │ external │
|
||||
│ message │ │ │ │ Allow │ │ systems │
|
||||
│ │ │ │ │ │ │ │
|
||||
│ PostgreSQL ◀─┤ │ PostgreSQL ◀─┤ │PostgreSQL│ │ RustFS ◀─┤
|
||||
│ (user prefs) │ │ (timeline) │ │(policies)│ │ (exports) │
|
||||
└──────────────┘ └──────────────┘ └──────────┘ └──────────────┘
|
||||
```
|
||||
|
||||
## 2) Advisory ingestion (delta-driven)
|
||||
|
||||
1. **Concelier.Worker** fetches advisories from configured sources (mirrors first; no hidden outbound calls in air-gap profiles).
|
||||
@@ -43,6 +154,111 @@ Offline note: for air-gapped deployments, step 6 writes to local object storage
|
||||
|
||||
## 5) Notification delivery
|
||||
|
||||
1. **Notify.WebService** consumes platform events (scan completed, advisory delta, etc.).
|
||||
2. **Notify.WebService -> queue/stream** enqueues delivery tasks with idempotency keys.
|
||||
3. **Notify.Worker -> channels** delivers (email/chat/webhook), records results, and retries with deterministic backoff rules.
|
||||
1. **Notification engine** consumes platform events (scan completed, advisory delta, etc.).
|
||||
2. **Notification engine -> queue/stream** enqueues delivery tasks with idempotency keys (when a worker model is used).
|
||||
3. **Delivery workers -> channels** deliver (email/chat/webhook), record results, and retry with deterministic backoff rules.
|
||||
|
||||
### Notification flow diagram (vulnerability alert)
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────────────────────────────────────┐
|
||||
│ TRIGGER: New critical CVE detected in existing scan │
|
||||
│ Source: Concelier advisory ingestion │
|
||||
└───────────────────────────────────┬──────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────────────────────────────────────────────────────────────────────────┐
|
||||
│ CONCELIER.WORKER (Advisory Processor) │
|
||||
│ │
|
||||
│ 1. Ingest new advisory from NVD/OSV/CSAF │
|
||||
│ ┌─────────────┐ │
|
||||
│ │ PostgreSQL │◀── INSERT INTO advisories (cve_id, severity, ...) │
|
||||
│ └─────────────┘ │
|
||||
│ │
|
||||
│ 2. Match advisory against existing scans (PURL/CPE matching) │
|
||||
│ ┌─────────────┐ │
|
||||
│ │ PostgreSQL │◀── SELECT scans WHERE package_purl IN (affected_purls) │
|
||||
│ └─────────────┘ │
|
||||
│ │
|
||||
│ 3. Publish drift event to Valkey │
|
||||
│ ┌─────────────┐ │
|
||||
│ │ Valkey │◀── XADD concelier:drift (new vulnerability found) │
|
||||
│ └─────────────┘ │
|
||||
└───────────────────────────────────┬──────────────────────────────────────────────┘
|
||||
│ Event published
|
||||
▼
|
||||
┌──────────────────────────────────────────────────────────────────────────────────┐
|
||||
│ NOTIFY.WORKER (Notification Processor) │
|
||||
│ │
|
||||
│ 1. Consume drift event from Valkey stream │
|
||||
│ ┌─────────────┐ │
|
||||
│ │ Valkey │◀── XREADGROUP concelier:drift notify-workers │
|
||||
│ └─────────────┘ │
|
||||
│ │
|
||||
│ 2. Query user notification preferences │
|
||||
│ ┌─────────────┐ │
|
||||
│ │ PostgreSQL │◀── SELECT * FROM user_notification_preferences │
|
||||
│ │ │ WHERE user_id = scan_owner AND channel = 'slack' │
|
||||
│ └─────────────┘ │
|
||||
│ │
|
||||
│ 3. Render notification template │
|
||||
│ Template: "New critical CVE-2024-1234 affects alpine:latest scan" │
|
||||
│ │
|
||||
│ 4. Deliver notification via configured channels │
|
||||
│ ┌─────────────────────────────────────────────────────────┐ │
|
||||
│ │ External APIs │ │
|
||||
│ │ - POST https://hooks.slack.com/services/T00/B00/xxx │ │
|
||||
│ │ - POST https://graph.microsoft.com/v1.0/teams/channels │ │
|
||||
│ │ - SMTP send (email) │ │
|
||||
│ └─────────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ 5. Store delivery receipt in PostgreSQL │
|
||||
│ ┌─────────────┐ │
|
||||
│ │ PostgreSQL │◀── INSERT INTO notification_deliveries (status, ...) │
|
||||
│ └─────────────┘ │
|
||||
└──────────────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## 6) Export flow (SBOM distribution)
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────────────────────────────────────┐
|
||||
│ EXPORT REQUEST: GET /api/v1/scans/{scan_id}/export?format=spdx │
|
||||
└───────────────────────────────────┬──────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────────────────────────────────────────────────────────────────────────┐
|
||||
│ SCANNER.WEB or EXPORT CENTER │
|
||||
│ │
|
||||
│ 1. Query scan metadata from PostgreSQL │
|
||||
│ ┌─────────────┐ │
|
||||
│ │ PostgreSQL │◀── SELECT * FROM scan_jobs WHERE job_id = $1 │
|
||||
│ │ │◀── SELECT * FROM sbom_documents WHERE scan_id = $1 │
|
||||
│ └─────────────┘ │
|
||||
│ │
|
||||
│ 2. Retrieve SBOM artifact from RustFS │
|
||||
│ ┌─────────────┐ │
|
||||
│ │ RustFS │◀── GET /artifacts/scan-abc123/sbom.spdx.json │
|
||||
│ └─────────────┘ │
|
||||
│ │
|
||||
│ 3. Sign SBOM with Signer service │
|
||||
│ ┌─────────────┐ │
|
||||
│ │ Signer │◀── POST /api/v1/sign (SBOM payload) │
|
||||
│ │ │──▶ Returns: DSSE envelope with signature │
|
||||
│ └─────────────┘ │
|
||||
│ │
|
||||
│ 4. Create in-toto attestation with Attestor │
|
||||
│ ┌─────────────┐ │
|
||||
│ │ Attestor │◀── POST /api/v1/attest (signed SBOM) │
|
||||
│ │ │──▶ Returns: in-toto attestation bundle │
|
||||
│ └─────────────┘ │
|
||||
│ │
|
||||
│ 5. Store final bundle to RustFS │
|
||||
│ ┌─────────────┐ │
|
||||
│ │ RustFS │◀── PUT /artifacts/scan-abc123/bundle.jsonl │
|
||||
│ └─────────────┘ │
|
||||
│ │
|
||||
│ 6. Return signed bundle to client │
|
||||
│ Returns: HTTP 200 OK (application/vnd.in-toto+json) │
|
||||
└──────────────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user