# StellaOps Developer Onboarding Guide > **Target Audience:** DevOps operators with developer knowledge who need to understand, deploy, and debug the StellaOps platform. ## Table of Contents 1. [Architecture Overview](#architecture-overview) 2. [Prerequisites](#prerequisites) 3. [Quick Start - Full Platform in Docker](#quick-start) 4. [Hybrid Debugging Workflow](#hybrid-debugging-workflow) 5. [Service-by-Service Debugging Guide](#service-by-service-debugging-guide) 6. [Configuration Deep Dive](#configuration-deep-dive) 7. [Common Development Workflows](#common-development-workflows) 8. [Troubleshooting](#troubleshooting) --- ## Architecture Overview StellaOps is a deterministic SBOM + VEX platform built as a microservices architecture with 36+ services organized into functional domains. **πŸ“– For detailed component architecture with communication patterns, see [ARCHITECTURE_DETAILED.md](./ARCHITECTURE_DETAILED.md)** ### Quick Reference - Component Topology ``` CLIENT LAYER β”œβ”€ stella CLI β†’ Gateway (JWT + DPoP auth) β”œβ”€ Web UI (Angular) β†’ Gateway (JWT + DPoP auth) β”œβ”€ CI/CD Pipelines β†’ Gateway (JWT + DPoP auth) └─ Zastava Observer β†’ Scanner (runtime scans) INFRASTRUCTURE (REQUIRED) β”œβ”€ PostgreSQL v16+ β†’ Primary database (ALL services) β”œβ”€ Valkey v8.0 β†’ Cache, DPoP, queues, events └─ RustFS β†’ Object storage (S3 API) INFRASTRUCTURE (OPTIONAL) └─ NATS JetStream β†’ Alternative messaging (Valkey is default) GATEWAY LAYER └─ Gateway.WebService β†’ Auth, routing, rate limiting AUTH & CRYPTO β”œβ”€ Authority β†’ OAuth2/OIDC, OpTok issuance β”œβ”€ Signer β†’ DSSE signing (FIPS/GOST/SM) └─ Attestor β†’ Rekor v2 transparency log CORE ENGINES β”œβ”€ Scanner.WebService β†’ Scan orchestration β”œβ”€ Scanner.Worker β†’ Image analysis, SBOM generation β”œβ”€ Concelier.WebService β†’ Advisory ingestion (NVD, Red Hat, etc.) β”œβ”€ Excititor.WebService β†’ VEX ingestion + consensus β”œβ”€ Policy.Gateway β†’ OPA/Rego policy evaluation β”œβ”€ Scheduler.WebService β†’ Re-scan orchestration β”œβ”€ Notify.WebService β†’ Notification orchestration β”œβ”€ Notify.Worker β†’ Slack/Teams/Email delivery └─ Orchestrator.WebService β†’ DAG workflows, pack runs SUPPORTING └─ IssuerDirectory β†’ VEX issuer trust registry ``` ### Service Categories | Category | Services | Purpose | |----------|----------|---------| | **Gateway** | Gateway.WebService | API routing, auth enforcement | | **Auth & Security** | Authority, Signer, Attestor | OAuth2, signing, transparency | | **Scanning** | Scanner.Web, Scanner.Worker | Container analysis, SBOM | | **Advisory** | Concelier.Web, Concelier.Worker | Vulnerability ingestion | | **VEX** | Excititor.Web, Excititor.Worker | Exploitability statements | | **Policy** | Policy.Gateway, Policy Engine | OPA/Rego evaluation | | **Orchestration** | Scheduler, Orchestrator | Job coordination | | **Notifications** | Notify.Web, Notify.Worker | Delivery to Slack/Teams/Email | ### Runtime Topology - Infrastructure Dependencies ``` β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ INFRASTRUCTURE LAYER β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β”‚ PostgreSQL β”‚ β”‚ Valkey β”‚ β”‚ RustFS β”‚ β”‚ β”‚ β”‚ (v16+ ONLY) β”‚ β”‚ (Redis-compat) β”‚ β”‚ (S3-like API) β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ - Caching β”‚ β”‚ - Artifacts β”‚ β”‚ β”‚ β”‚ All services use β”‚ β”‚ - DPoP nonces β”‚ β”‚ - SBOMs β”‚ β”‚ β”‚ β”‚ PostgreSQL for β”‚ β”‚ - Event queues β”‚ β”‚ - Signatures β”‚ β”‚ β”‚ β”‚ persistent data β”‚ β”‚ - Rate limitingβ”‚ β”‚ β”‚ β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β”‚ Optional: NATS JetStream (alternative transport for queues) β”‚ β”‚ β”‚ β”‚ Only used if explicitly configured in appsettings β”‚ β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β–² β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ AUTHENTICATION & SIGNING β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β”‚ Authority │─▢│ Signer │─▢│ Attestor β”‚ β”‚ β”‚ β”‚ (OAuth2/OIDC)β”‚ β”‚(DSSE/PKIX) β”‚ β”‚(in-toto/DSSE)β”‚ β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β–² β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ INGESTION & AGGREGATION β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β”‚ Concelier β”‚ β”‚ Excititor β”‚ β”‚IssuerDirectryβ”‚ β”‚ β”‚ β”‚(Advisories) β”‚ β”‚ (VEX) β”‚ β”‚(CSAF Pubshrs)β”‚ β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β–² β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ SCANNING & ANALYSIS β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β”‚Scanner.Web β”‚ β”‚Scanner.Workerβ”‚ β”‚ AdvisoryAI β”‚ β”‚ β”‚ β”‚(API/Control) β”‚ β”‚(Analyzers) β”‚ β”‚(ML Analysis) β”‚ β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β”‚ RiskEngine β”‚ β”‚ Policy β”‚ β”‚ β”‚ β”‚ (Scoring) β”‚ β”‚ (Engine) β”‚ β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β–² β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ ORCHESTRATION & WORKFLOW β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β”‚ Scheduler β”‚ β”‚ Orchestrator β”‚ β”‚ TaskRunner β”‚ β”‚ β”‚ β”‚(Job Sched) β”‚ β”‚(Coordinator) β”‚ β”‚(Executor) β”‚ β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β–² β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ EVENTS & NOTIFICATIONS β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β”‚ Notify β”‚ β”‚ Notifier β”‚ β”‚TimelineIndex β”‚ β”‚ β”‚ β”‚(Slack/Teams) β”‚ β”‚ (Advanced) β”‚ β”‚ (Events) β”‚ β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β–² β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ DATA & EXPORT β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β”‚ExportCenter β”‚ β”‚EvidenceLockerβ”‚ β”‚FindingsLedgerβ”‚ β”‚ β”‚ β”‚(SARIF/SBOM) β”‚ β”‚(Artifacts) β”‚ β”‚(Audit Trail) β”‚ β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β–² β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ USER EXPERIENCE β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β”‚ Gateway β”‚ β”‚ Web (UI) β”‚ β”‚ CLI β”‚ β”‚ β”‚ β”‚ (API Router) β”‚ β”‚ (Angular v17)β”‚ β”‚(Multi-plat) β”‚ β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` ### Detailed Request Flow - Scan Execution Example This diagram shows a complete scan request lifecycle with detailed routing through services: ``` β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ 1. CLIENT REQUEST (CLI or Web UI) β”‚ β”‚ $ stella scan docker://alpine:latest --sbom-format=spdx β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ HTTPS β–Ό β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ 2. GATEWAY (API Router) β”‚ β”‚ - Terminates TLS β”‚ β”‚ - Routes to appropriate backend service β”‚ β”‚ - Load balancing (if multiple instances) β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ HTTP (internal) β–Ό β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ 3. AUTHORITY (Authentication) β”‚ β”‚ - Validates OAuth2 access token (DPoP-bound) β”‚ β”‚ - Checks DPoP proof against Valkey nonce cache β”‚ β”‚ - Returns user identity and scopes β”‚ β”‚ β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β”‚ Valkey │◀── DPoP nonce validation (GET/SET) β”‚ β”‚ β”‚ (Cache) β”‚ β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β”‚ PostgreSQL │◀── User/client lookup (SELECT) β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ Authenticated request β–Ό β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ 4. SCANNER.WEB (Scan API Controller) β”‚ β”‚ - Validates scan request parameters β”‚ β”‚ - Creates scan job record in PostgreSQL β”‚ β”‚ - Enqueues scan job to Valkey queue (default) or NATS (if configured) β”‚ β”‚ β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β”‚ PostgreSQL │◀── INSERT scan_jobs (job_id, image_ref, status='pending') β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β”‚ Valkey │◀── XADD scanner:jobs (enqueue job message) β”‚ β”‚ β”‚ (Queue) β”‚ β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ β”‚ β”‚ Returns: HTTP 202 Accepted { "job_id": "scan-abc123", "status": "queued" } β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ (Client polls for status) β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ 5. SCANNER.WORKER (Background Processor) β”‚ β”‚ - Consumes job from Valkey queue (XREADGROUP scanner:jobs) β”‚ β”‚ - Updates job status to 'running' β”‚ β”‚ - Downloads container image from registry β”‚ β”‚ - Executes analyzers (OS packages, language deps, files) β”‚ β”‚ - Generates SBOM (SPDX/CycloneDX) β”‚ β”‚ - Stores artifacts to RustFS β”‚ β”‚ β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β”‚ PostgreSQL │◀── UPDATE scan_jobs SET status='running' β”‚ β”‚ β”‚ │◀── INSERT sbom_documents, packages, vulnerabilities β”‚ β”‚ β”‚ │◀── UPDATE scan_jobs SET status='completed' β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β”‚ RustFS │◀── PUT /artifacts/scan-abc123/sbom.spdx.json β”‚ β”‚ β”‚ (S3 API) │◀── PUT /artifacts/scan-abc123/image-layers.tar.gz β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β”‚ Valkey │◀── XADD scanner:events (publish scan.completed event) β”‚ β”‚ β”‚(Event Streamβ”‚ β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ Event published β–Ό β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ 6. EVENT PROPAGATION (Valkey Streams) β”‚ β”‚ β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β”‚ Valkey Event Stream: "scanner:events" β”‚ β”‚ β”‚ β”‚ Event: { "type": "scan.completed", "job_id": "scan-abc123", ... } β”‚ β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ β”‚ β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β–Ό β–Ό β–Ό β–Ό β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β”‚ Notify β”‚ β”‚Timeline β”‚ β”‚ Policy β”‚ β”‚ Export β”‚ β”‚ β”‚ β”‚ Worker β”‚ β”‚ Indexer β”‚ β”‚ Engine β”‚ β”‚ Center β”‚ β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ (all subscribe to scanner:events via XREADGROUP) β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ β”‚ β”‚ β–Ό β–Ό β–Ό β–Ό β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ 7a. NOTIFY β”‚ β”‚ 7b. TIMELINE β”‚ β”‚ 7c.POLICYβ”‚ β”‚ 7d. EXPORT β”‚ β”‚ β”‚ β”‚ INDEXER β”‚ β”‚ ENGINE β”‚ β”‚ CENTER β”‚ β”‚ - Query scan β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ results β”‚ β”‚ - Index eventβ”‚ β”‚ - Eval β”‚ β”‚ - Generate β”‚ β”‚ - Check user β”‚ β”‚ timeline β”‚ β”‚ policy β”‚ β”‚ SARIF β”‚ β”‚ notif prefsβ”‚ β”‚ - Store in β”‚ β”‚ rules β”‚ β”‚ - Export to β”‚ β”‚ - Send Slack β”‚ β”‚ PostgreSQL β”‚ β”‚ - Block/ β”‚ β”‚ external β”‚ β”‚ message β”‚ β”‚ β”‚ β”‚ Allow β”‚ β”‚ systems β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ PostgreSQL ◀── β”‚ PostgreSQL ◀── β”‚PostgreSQLβ”‚ β”‚ RustFS ◀── β”‚ (user prefs) β”‚ β”‚ (timeline) β”‚ β”‚(policies)β”‚ β”‚ (exports) β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` ### Export Flow - SBOM Distribution ``` β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ EXPORT REQUEST: GET /api/v1/scans/{scan_id}/export?format=spdx β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β–Ό β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ SCANNER.WEB or EXPORT CENTER β”‚ β”‚ β”‚ β”‚ 1. Query scan metadata from PostgreSQL β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β”‚ PostgreSQL │◀── SELECT * FROM scan_jobs WHERE job_id = $1 β”‚ β”‚ β”‚ │◀── SELECT * FROM sbom_documents WHERE scan_id = $1 β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ β”‚ β”‚ 2. Retrieve SBOM artifact from RustFS β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β”‚ RustFS │◀── GET /artifacts/scan-abc123/sbom.spdx.json β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ β”‚ β”‚ 3. Sign SBOM with Signer service β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β”‚ Signer │◀── POST /api/v1/sign (SBOM payload) β”‚ β”‚ β”‚ │──▢ Returns: DSSE envelope with signature β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ β”‚ β”‚ 4. Create in-toto attestation with Attestor β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β”‚ Attestor │◀── POST /api/v1/attest (signed SBOM) β”‚ β”‚ β”‚ │──▢ Returns: in-toto attestation bundle β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ β”‚ β”‚ 5. Store final bundle to RustFS β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β”‚ RustFS │◀── PUT /artifacts/scan-abc123/bundle.jsonl β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ β”‚ β”‚ 6. Return signed bundle to client β”‚ β”‚ Returns: HTTP 200 OK (application/vnd.in-toto+json) β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` ### Notification Flow - Vulnerability Alert ``` β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ TRIGGER: New critical CVE detected in existing scan β”‚ β”‚ Source: Concelier advisory ingestion β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β–Ό β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ CONCELIER.WORKER (Advisory Processor) β”‚ β”‚ β”‚ β”‚ 1. Ingest new advisory from NVD/OSV/CSAF β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β”‚ PostgreSQL │◀── INSERT INTO advisories (cve_id, severity, ...) β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ β”‚ β”‚ 2. Match advisory against existing scans (PURL/CPE matching) β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β”‚ PostgreSQL │◀── SELECT scans WHERE package_purl IN (affected_purls) β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ β”‚ β”‚ 3. Publish drift event to Valkey β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β”‚ Valkey │◀── XADD concelier:drift (new vulnerability found) β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ Event published β–Ό β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ NOTIFY.WORKER (Notification Processor) β”‚ β”‚ β”‚ β”‚ 1. Consume drift event from Valkey stream β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β”‚ Valkey │◀── XREADGROUP concelier:drift notify-workers β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ β”‚ β”‚ 2. Query user notification preferences β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β”‚ PostgreSQL │◀── SELECT * FROM user_notification_preferences β”‚ β”‚ β”‚ β”‚ WHERE user_id = scan_owner AND channel = 'slack' β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ β”‚ β”‚ 3. Render notification template β”‚ β”‚ Template: "🚨 New critical CVE-2024-1234 affects alpine:latest scan" β”‚ β”‚ β”‚ β”‚ 4. Deliver notification via configured channels β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β”‚ External APIs β”‚ β”‚ β”‚ β”‚ - POST https://hooks.slack.com/services/T00/B00/xxx β”‚ β”‚ β”‚ β”‚ - POST https://graph.microsoft.com/v1.0/teams/channels β”‚ β”‚ β”‚ β”‚ - SMTP send (email) β”‚ β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ β”‚ β”‚ 5. Store delivery receipt in PostgreSQL β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β”‚ PostgreSQL │◀── INSERT INTO notification_deliveries (status, ...) β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` ### Key Architectural Principles 1. **Deterministic Evidence** - Same inputs always produce same outputs 2. **VEX-First Decisioning** - Policy decisions based on OpenVEX statements 3. **Offline-First** - Full air-gap operation supported 4. **Plugin Architecture** - Extensible connectors for advisories, analyzers, auth 5. **Sovereign Crypto** - FIPS, eIDAS, GOST, SM support 6. **Schema Isolation** - Per-module PostgreSQL schemas ### Service Categories | Category | Services | Purpose | |----------|----------|---------| | **Infrastructure** | PostgreSQL v16+, Valkey 8.0, RustFS, NATS (optional) | Database, cache/messaging, object storage, optional queue transport | | **Auth & Signing** | Authority, Signer, Attestor | OAuth2/OIDC with DPoP, cryptographic signing, in-toto attestations | | **Ingestion** | Concelier, Excititor, IssuerDirectory | Advisory/VEX ingestion, normalization, merging, CSAF publisher discovery | | **Scanning** | Scanner.Web, Scanner.Worker, AdvisoryAI | Container scanning, SBOM generation (SPDX/CDX), ML vulnerability analysis | | **Policy & Risk** | Policy Engine, RiskEngine | OPA/Rego policy evaluation, risk scoring, exploitability assessment | | **Orchestration** | Scheduler, Orchestrator, TaskRunner | Job scheduling, workflow coordination, distributed task execution | | **Notifications** | Notify, Notifier, TimelineIndexer | Event delivery (Slack/Teams/Email), notification management, timeline tracking | | **Data & Export** | ExportCenter, EvidenceLocker, FindingsLedger | SARIF/SBOM export, evidence storage, immutable audit trail | | **User Experience** | Gateway, Web UI, CLI | API routing, Angular v17 UI, multi-platform command-line tools | --- ## Prerequisites ### Required Software 1. **Docker Desktop** (Windows/Mac) or **Docker Engine + Docker Compose** (Linux) - Version: 20.10+ recommended - Enable WSL2 backend (Windows) 2. **.NET 10 SDK** - Download: https://dotnet.microsoft.com/download/dotnet/10.0 - Verify: `dotnet --version` (should show 10.0.x) 3. **Visual Studio 2022** (v17.12+) or **Visual Studio Code** - Workload: ASP.NET and web development - Workload: .NET desktop development - Extension (VS Code): C# Dev Kit 4. **Git** - Version: 2.30+ recommended ### Optional Tools - **PostgreSQL Client** (psql, pgAdmin, DBeaver) - for database inspection - **Redis Insight** or **Another Redis Desktop Manager** - for Valkey inspection (Valkey is Redis-compatible) - **Postman/Insomnia** - for API testing - **AWS CLI or s3cmd** - for RustFS (S3-compatible) inspection ### System Requirements - **RAM:** 16 GB minimum, 32 GB recommended - **Disk:** 50 GB free space (for Docker images, volumes, build artifacts) - **CPU:** 4 cores minimum, 8 cores recommended --- ## Quick Start ### Step 1: Clone the Repository ```bash cd C:\dev\ git clone https://git.stella-ops.org/stella-ops.org/git.stella-ops.org cd git.stella-ops.org ``` ### Step 2: Prepare Environment Configuration ```bash # Copy the development environment template cd deploy\compose copy env\dev.env.example .env # Edit .env with your preferred text editor notepad .env ``` **Key settings to configure:** ```bash # PostgreSQL Database POSTGRES_USER=stellaops POSTGRES_PASSWORD=your_secure_password_here POSTGRES_DB=stellaops_platform POSTGRES_PORT=5432 # Valkey (Redis-compatible cache and messaging) VALKEY_PORT=6379 # RustFS Object Storage RUSTFS_HTTP_PORT=8080 # Service ports (adjust if conflicts exist) AUTHORITY_PORT=8440 SIGNER_PORT=8441 ATTESTOR_PORT=8442 CONCELIER_PORT=8445 SCANNER_WEB_PORT=8444 NOTIFY_WEB_PORT=8446 # Scanner configuration (Valkey default, can switch to NATS if needed) SCANNER_EVENTS_DRIVER=valkey SCANNER_EVENTS_DSN=valkey:6379 # Scheduler configuration (Valkey default, can switch to NATS if needed) SCHEDULER_QUEUE_KIND=Valkey SCHEDULER_QUEUE_VALKEY_URL=valkey:6379 # Authority configuration AUTHORITY_ISSUER=https://authority:8440 SIGNER_POE_INTROSPECT_URL=https://www.stella-ops.org/license/introspect ``` ### Step 3: Start the Full Platform ```bash # From deploy/compose directory docker compose -f docker-compose.dev.yaml up -d ``` **This will start all infrastructure and services:** - PostgreSQL v16+ (port 5432) - Primary database for all services - Valkey 8.0 (port 6379) - Cache, DPoP nonces, event streams, rate limiting - RustFS (port 8080) - S3-compatible object storage for artifacts/SBOMs - NATS JetStream (port 4222) - Optional transport (only if configured) - Authority (port 8440) - OAuth2/OIDC authentication - Signer (port 8441) - Cryptographic signing - Attestor (port 8442) - in-toto attestation generation - Scanner.Web (port 8444) - Scan API - Concelier (port 8445) - Advisory ingestion - And 30+ more services... ### Step 4: Verify Services Are Running ```bash # Check all services are up docker compose -f docker-compose.dev.yaml ps # Check logs for a specific service docker compose -f docker-compose.dev.yaml logs -f scanner-web # Check infrastructure health docker compose -f docker-compose.dev.yaml logs postgres docker compose -f docker-compose.dev.yaml logs valkey docker compose -f docker-compose.dev.yaml logs rustfs ``` ### Step 5: Access the Platform Open your browser and navigate to: - **RustFS:** http://localhost:8080 (S3-compatible object storage) - **Scanner API:** http://localhost:8444/swagger (if Swagger enabled) - **Concelier API:** http://localhost:8445/swagger - **Authority:** http://localhost:8440/.well-known/openid-configuration (OIDC discovery) --- ## Hybrid Debugging Workflow The hybrid workflow allows you to: 1. Run infrastructure (databases, queues) in Docker 2. Run most services in Docker 3. **Selectively debug** one or two services in Visual Studio ### Workflow Overview ``` β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ DOCKER ENVIRONMENT β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β”‚PostgreSQLβ”‚ β”‚ Valkey β”‚ β”‚ RustFS β”‚ β”‚ β”‚ β”‚ (DB) β”‚ β”‚(Cache/Msgβ”‚ β”‚(Storage) β”‚ β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β”‚ Authorityβ”‚ β”‚ Signer β”‚ β”‚ Attestor β”‚ β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β”‚Concelier β”‚ β”‚ Excititorβ”‚ ← Running normally β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β–² β”‚ HTTP calls + Valkey streams β–Ό β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ VISUAL STUDIO (F5) β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β”‚ Scanner.WebService β”‚ β”‚ β”‚ β”‚ Running on http://localhost:5210 β”‚ β”‚ β”‚ β”‚ (Breakpoints, hot reload, debugging) β”‚ ← YOU DEBUG HERE β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` ### Step-by-Step: Debug Scanner.WebService #### 1. Stop the Docker Container for Scanner ```bash cd deploy\compose docker compose -f docker-compose.dev.yaml stop scanner-web ``` **Verify it's stopped:** ```bash docker compose -f docker-compose.dev.yaml ps scanner-web # Should show: State = "exited" ``` #### 2. Configure Local Development Settings Create or modify the service's `appsettings.Development.json`: ```bash cd C:\dev\New folder\git.stella-ops.org\src\Scanner\StellaOps.Scanner.WebService ``` **Create `appsettings.Development.json`:** ```json { "Logging": { "LogLevel": { "Default": "Information", "Microsoft.AspNetCore": "Warning", "StellaOps": "Debug" } }, "ConnectionStrings": { "DefaultConnection": "Host=localhost;Port=5432;Database=stellaops_platform;Username=stellaops;Password=your_password_here;Include Error Detail=true" }, "Scanner": { "Storage": { "Mongo": { "ConnectionString": "mongodb://stellaops:your_password_here@localhost:27017" } }, "ArtifactStore": { "Driver": "rustfs", "Endpoint": "http://localhost:8080/api/v1", "Bucket": "scanner-artifacts", "TimeoutSeconds": 30 }, "Queue": { "Broker": "nats://localhost:4222" }, "Events": { "Enabled": false } }, "Authority": { "Issuer": "https://localhost:8440", "BaseUrl": "https://localhost:8440", "BypassNetworks": ["127.0.0.1", "::1"] } } ``` **Note:** Adjust connection strings to match your Docker infrastructure ports. If PostgreSQL is on Docker's bridge network, you may need to expose it on `localhost:5432`. #### 3. Expose Docker Services to localhost For services running in Docker to be accessible from your host machine, ensure ports are mapped in `docker-compose.dev.yaml`: ```yaml # Already configured in docker-compose.dev.yaml postgres: ports: - "${POSTGRES_PORT:-5432}:5432" mongo: ports: - "27017:27017" nats: ports: - "${NATS_CLIENT_PORT:-4222}:4222" rustfs: ports: - "${RUSTFS_HTTP_PORT:-8080}:8080" ``` **Verify connectivity:** ```bash # Test PostgreSQL psql -h localhost -U stellaops -d stellaops_platform # Test NATS telnet localhost 4222 # Test RustFS curl http://localhost:8080/health ``` #### 4. Open Solution in Visual Studio ```bash # Open the solution cd C:\dev\New folder\git.stella-ops.org start src\StellaOps.sln ``` **In Visual Studio:** 1. Right-click `StellaOps.Scanner.WebService` project 2. Select **"Set as Startup Project"** 3. Press **F5** to start debugging **Expected output:** ``` info: Microsoft.Hosting.Lifetime[14] Now listening on: http://localhost:5210 info: Microsoft.Hosting.Lifetime[0] Application started. Press Ctrl+C to shut down. ``` #### 5. Update Other Services to Call localhost Since you're running Scanner.WebService on `localhost:5210` instead of `scanner-web:8444`, you need to update any services that call it. **Option A: Environment Variables (Docker containers)** Update `.env` file: ```bash # Use host.docker.internal to reach host machine from Docker SCANNER_WEB_BASEURL=http://host.docker.internal:5210 ``` Restart dependent services: ```bash docker compose -f docker-compose.dev.yaml restart scheduler-web ``` **Option B: Modify docker-compose.dev.yaml** ```yaml scheduler-web: environment: SCHEDULER__WORKER__RUNNER__SCANNER__BASEADDRESS: "http://host.docker.internal:5210" ``` Then restart: ```bash docker compose -f docker-compose.dev.yaml up -d scheduler-web ``` #### 6. Set Breakpoints and Debug 1. Navigate to `Program.cs` in Scanner.WebService 2. Set a breakpoint on a line in a controller or service method 3. Trigger the endpoint using: - Swagger UI (if enabled): http://localhost:5210/swagger - Postman/curl - CLI command **Example curl:** ```bash curl -X POST http://localhost:5210/api/scans \ -H "Content-Type: application/json" \ -d '{"imageRef": "alpine:latest"}' ``` Your breakpoint should hit, and you can step through code. #### 7. Return to Docker Mode When you're done debugging: ```bash # Stop Visual Studio debugger (Shift+F5) # Restart the Docker container cd deploy\compose docker compose -f docker-compose.dev.yaml start scanner-web # Verify it's running docker compose -f docker-compose.dev.yaml ps scanner-web ``` --- ## Service-by-Service Debugging Guide ### Authority (OAuth2/OIDC Provider) **Project:** `src/Authority/StellaOps.Authority/StellaOps.Authority.csproj` **Stop Docker container:** ```bash docker compose -f docker-compose.dev.yaml stop authority ``` **Configuration:** `appsettings.Development.json` ```json { "ConnectionStrings": { "DefaultConnection": "Host=localhost;Port=5432;Database=stellaops_platform;Username=stellaops;Password=your_password" }, "StellaOps_Authority": { "Issuer": "https://localhost:5001", "Mongo": { "ConnectionString": "mongodb://stellaops:your_password@localhost:27017" } }, "Kestrel": { "Endpoints": { "Https": { "Url": "https://localhost:5001" } } } } ``` **Run in Visual Studio:** F5 on `StellaOps.Authority` project **Default URL:** https://localhost:5001 **Update dependent services:** ```bash # In .env AUTHORITY_ISSUER=https://host.docker.internal:5001 ``` ### Concelier (Advisory Ingestion) **Project:** `src/Concelier/StellaOps.Concelier.WebService/StellaOps.Concelier.WebService.csproj` **Stop Docker:** ```bash docker compose -f docker-compose.dev.yaml stop concelier ``` **Configuration:** `appsettings.Development.json` ```json { "ConnectionStrings": { "DefaultConnection": "Host=localhost;Port=5432;Database=stellaops_platform;Username=stellaops;Password=your_password" }, "Concelier": { "Storage": { "Mongo": { "ConnectionString": "mongodb://stellaops:your_password@localhost:27017" }, "S3": { "Endpoint": "http://localhost:9000", "AccessKeyId": "stellaops", "SecretAccessKey": "your_password" } }, "Authority": { "BaseUrl": "https://localhost:8440" } } } ``` **Run:** F5 on `StellaOps.Concelier.WebService` **Default URL:** http://localhost:5000 ### Scanner.Worker (Background Analyzer) **Project:** `src/Scanner/StellaOps.Scanner.Worker/StellaOps.Scanner.Worker.csproj` **Stop Docker:** ```bash docker compose -f docker-compose.dev.yaml stop scanner-worker ``` **Configuration:** Same as Scanner.WebService (shares settings) **Run:** F5 on `StellaOps.Scanner.Worker` **Note:** Worker has no HTTP endpoint - it consumes from NATS queue ### Scheduler.WebService **Project:** `src/Scheduler/StellaOps.Scheduler.WebService/StellaOps.Scheduler.WebService.csproj` **Stop Docker:** ```bash docker compose -f docker-compose.dev.yaml stop scheduler-web ``` **Configuration:** `appsettings.Development.json` ```json { "ConnectionStrings": { "DefaultConnection": "Host=localhost;Port=5432;Database=stellaops_orchestrator;Username=stellaops;Password=your_password" }, "Scheduler": { "Queue": { "Kind": "Nats", "Nats": { "Url": "nats://localhost:4222" } }, "Worker": { "Runner": { "Scanner": { "BaseAddress": "http://localhost:5210" } } } } } ``` **Run:** F5 on `StellaOps.Scheduler.WebService` ### Notify.WebService **Project:** `src/Notify/StellaOps.Notify.WebService/StellaOps.Notify.WebService.csproj` **Stop Docker:** ```bash docker compose -f docker-compose.dev.yaml stop notify-web ``` **Configuration:** Uses `etc/notify.dev.yaml` **Run:** F5 on `StellaOps.Notify.WebService` --- ## Configuration Deep Dive ### Configuration Hierarchy All services follow this configuration priority (highest to lowest): 1. **Environment Variables** - `STELLAOPS__` or `__` 2. **appsettings.{Environment}.json** - `appsettings.Development.json`, `appsettings.Production.json` 3. **appsettings.json** - Base configuration 4. **YAML files** - `../etc/.yaml`, `../etc/.local.yaml` ### Common Configuration Patterns #### PostgreSQL Connection Strings ```json { "ConnectionStrings": { "DefaultConnection": "Host=localhost;Port=5432;Database=;Username=stellaops;Password=;Pooling=true;Minimum Pool Size=1;Maximum Pool Size=100;Command Timeout=60" } } ``` **Database names by service:** - Scanner: `stellaops_platform` or `scanner_*` - Orchestrator: `stellaops_orchestrator` - Authority: `stellaops_platform` (shared, schema-isolated) - Concelier: `stellaops_platform` (vuln schema) - Notify: `stellaops_platform` (notify schema) #### Valkey Configuration (Default Transport) ```json { "Scanner": { "Events": { "Driver": "valkey", "Dsn": "localhost:6379" }, "Cache": { "Redis": { "ConnectionString": "localhost:6379" } } }, "Scheduler": { "Queue": { "Kind": "Valkey", "Valkey": { "Url": "localhost:6379" } } } } ``` #### NATS Queue Configuration (Optional Alternative Transport) ```json { "Scanner": { "Events": { "Driver": "nats", "Dsn": "nats://localhost:4222" } }, "Scheduler": { "Queue": { "Kind": "Nats", "Nats": { "Url": "nats://localhost:4222" } } } } ``` #### RustFS Configuration (S3-Compatible Object Storage) ```json { "Scanner": { "Storage": { "RustFS": { "Endpoint": "http://localhost:8080", "AccessKeyId": "stellaops", "SecretAccessKey": "your_password", "BucketName": "scanner-artifacts", "Region": "us-east-1", "ForcePathStyle": true } } } } ``` #### RustFS Configuration ```json { "Scanner": { "ArtifactStore": { "Driver": "rustfs", "Endpoint": "http://localhost:8080/api/v1", "Bucket": "scanner-artifacts", "TimeoutSeconds": 30 } } } ``` ### Environment Variable Mapping ASP.NET Core uses `__` (double underscore) for nested configuration: ```bash # This JSON configuration: { "Scanner": { "Queue": { "Broker": "nats://localhost:4222" } } } # Can be set via environment variable: SCANNER__QUEUE__BROKER=nats://localhost:4222 # Or with STELLAOPS_ prefix: STELLAOPS_SCANNER__QUEUE__BROKER=nats://localhost:4222 ``` --- ## Common Development Workflows ### Workflow 1: Debug a Single Service with Full Stack **Scenario:** You need to debug Scanner.WebService while all other services run normally. ```bash # 1. Start full platform cd deploy\compose docker compose -f docker-compose.dev.yaml up -d # 2. Stop the service you want to debug docker compose -f docker-compose.dev.yaml stop scanner-web # 3. Open Visual Studio cd C:\dev\New folder\git.stella-ops.org start src\StellaOps.sln # 4. Set Scanner.WebService as startup project and F5 # 5. Test the service curl -X POST http://localhost:5210/api/scans -H "Content-Type: application/json" -d '{"imageRef":"alpine:latest"}' # 6. When done, stop VS debugger and restart Docker container docker compose -f docker-compose.dev.yaml start scanner-web ``` ### Workflow 2: Debug Multiple Services Together **Scenario:** Debug Scanner.WebService and Scanner.Worker together. ```bash # 1. Stop both containers docker compose -f docker-compose.dev.yaml stop scanner-web scanner-worker # 2. In Visual Studio, configure multiple startup projects: # - Right-click solution > Properties # - Set "Multiple startup projects" # - Select Scanner.WebService: Start # - Select Scanner.Worker: Start # 3. Press F5 to debug both simultaneously ``` ### Workflow 3: Test Integration with Modified Code **Scenario:** You modified Concelier and want to test how Scanner integrates with it. ```bash # 1. Build Concelier locally cd src\Concelier\StellaOps.Concelier.WebService dotnet build # 2. Stop Docker Concelier cd ..\..\..\deploy\compose docker compose -f docker-compose.dev.yaml stop concelier # 3. Run Concelier in Visual Studio (F5) # 4. Keep Scanner in Docker, but point it to localhost Concelier # Update .env: CONCELIER_BASEURL=http://host.docker.internal:5000 # 5. Restart Scanner to pick up new config docker compose -f docker-compose.dev.yaml restart scanner-web ``` ### Workflow 4: Reset Database State **Scenario:** You need a clean database to test migrations or start fresh. ```bash # 1. Stop all services docker compose -f docker-compose.dev.yaml down # 2. Remove database volumes docker volume rm compose_postgres-data docker volume rm compose_mongo-data # 3. Restart platform (will recreate volumes and databases) docker compose -f docker-compose.dev.yaml up -d # 4. Wait for migrations to run docker compose -f docker-compose.dev.yaml logs -f postgres # Look for migration completion messages ``` ### Workflow 5: Test Offline/Air-Gap Mode **Scenario:** Test the platform in offline mode. ```bash # 1. Use the air-gap compose profile cd deploy\compose docker compose -f docker-compose.airgap.yaml up -d # 2. Verify no external network calls docker compose -f docker-compose.airgap.yaml logs | grep -i "external\|outbound\|internet" ``` --- ## Troubleshooting ### Common Issues #### 1. Port Already in Use **Error:** ``` Error starting userland proxy: listen tcp 0.0.0.0:5432: bind: address already in use ``` **Solutions:** **Option A: Change the port in .env** ```bash # Edit .env POSTGRES_PORT=5433 # Use a different port ``` **Option B: Stop the conflicting process** ```bash # Windows netstat -ano | findstr :5432 taskkill /PID /F # Linux/Mac lsof -i :5432 kill -9 ``` #### 2. Cannot Connect to PostgreSQL from Visual Studio **Error:** ``` Npgsql.NpgsqlException: Connection refused ``` **Solutions:** 1. **Verify PostgreSQL is accessible from host:** ```bash psql -h localhost -U stellaops -d stellaops_platform ``` 2. **Check Docker network:** ```bash docker network inspect compose_stellaops # Ensure your service has "host.docker.internal" DNS resolution ``` 3. **Update connection string:** ```json { "ConnectionStrings": { "DefaultConnection": "Host=localhost;Port=5432;Database=stellaops_platform;Username=stellaops;Password=your_password;Include Error Detail=true" } } ``` #### 3. NATS Connection Refused **Error:** ``` NATS connection error: connection refused ``` **Solution:** By default, services use **Valkey** for messaging, not NATS. Ensure Valkey is running: ```bash docker compose -f docker-compose.dev.yaml ps valkey # Should show: State = "Up" # Test connectivity telnet localhost 6379 ``` Update configuration to use Valkey (default): ```json { "Scanner": { "Events": { "Driver": "valkey", "Dsn": "localhost:6379" } }, "Scheduler": { "Queue": { "Kind": "Valkey", "Valkey": { "Url": "localhost:6379" } } } } ``` **If you explicitly want to use NATS** (optional): ```bash docker compose -f docker-compose.dev.yaml ps nats # Ensure NATS is running # Update appsettings.Development.json: { "Scanner": { "Events": { "Driver": "nats", "Dsn": "nats://localhost:4222" } } } ``` #### 4. Valkey Connection Refused **Error:** ``` StackExchange.Redis.RedisConnectionException: It was not possible to connect to the redis server(s) ``` **Solutions:** 1. **Check Valkey is running:** ```bash docker compose -f docker-compose.dev.yaml ps valkey # Should show: State = "Up" # Check logs docker compose -f docker-compose.dev.yaml logs valkey ``` 2. **Reset Valkey:** ```bash docker compose -f docker-compose.dev.yaml stop valkey docker volume rm compose_valkey-data docker compose -f docker-compose.dev.yaml up -d valkey ``` #### 5. Service Cannot Reach host.docker.internal **Error:** ``` Could not resolve host: host.docker.internal ``` **Solution (Windows/Mac):** Should work automatically with Docker Desktop. **Solution (Linux):** Add to docker-compose.dev.yaml: ```yaml services: scanner-web: extra_hosts: - "host.docker.internal:host-gateway" ``` Or use the host's IP address: ```bash # Find host IP ip addr show docker0 # Use that IP instead of host.docker.internal ``` #### 6. Certificate Validation Errors (Authority/HTTPS) **Error:** ``` The SSL connection could not be established ``` **Solution:** For development, disable certificate validation: ```json { "Authority": { "ValidateCertificate": false } } ``` Or trust the development certificate: ```bash dotnet dev-certs https --trust ``` #### 7. Build Errors - Missing SDK **Error:** ``` error MSB4236: The SDK 'Microsoft.NET.Sdk.Web' specified could not be found ``` **Solution:** Install .NET 10 SDK: ```bash # Verify installation dotnet --list-sdks # Should show: # 10.0.xxx [C:\Program Files\dotnet\sdk] ``` #### 8. Hot Reload Not Working **Symptom:** Changes in code don't reflect when running in Visual Studio. **Solutions:** 1. Ensure Hot Reload is enabled: Tools > Options > Debugging > .NET Hot Reload > Enable Hot Reload 2. Rebuild the project: Ctrl+Shift+B 3. Restart debugging session: Shift+F5, then F5 #### 9. Docker Compose Fails to Parse .env **Error:** ``` invalid interpolation format ``` **Solution:** Ensure no spaces around `=` in .env: ```bash # Wrong POSTGRES_USER = stellaops # Correct POSTGRES_USER=stellaops ``` #### 10. Volume Permission Issues (Linux) **Error:** ``` Permission denied writing to /data/db ``` **Solution:** ```bash # Fix permissions on volume directories sudo chown -R $USER:$USER ./volumes # Or run Docker as root (not recommended for production) sudo docker compose -f docker-compose.dev.yaml up -d ``` --- ## Next Steps ### Learning Path 1. **Week 1: Infrastructure** - Understand PostgreSQL schema isolation (all services use PostgreSQL) - Learn Valkey streams for event queuing and caching - Study RustFS S3-compatible object storage - Optional: NATS JetStream as alternative transport 2. **Week 2: Core Services** - Deep dive into Scanner architecture (analyzers, workers, caching) - Understand Concelier advisory ingestion and merging - Study VEX workflow in Excititor 3. **Week 3: Authentication & Security** - Master OAuth2/OIDC flow in Authority - Understand signing flow (Signer β†’ Attestor β†’ Rekor) - Study policy evaluation engine 4. **Week 4: Integration** - Build end-to-end scan workflow - Implement custom Concelier connector - Create custom notification rules ### Key Documentation - **Architecture:** `docs/07_HIGH_LEVEL_ARCHITECTURE.md` - **Build Commands:** `CLAUDE.md` - **Database Spec:** `docs/db/SPECIFICATION.md` - **API Reference:** `docs/09_API_CLI_REFERENCE.md` - **Module Architecture:** `docs/modules//architecture.md` ### Support - **Issues:** https://git.stella-ops.org/stella-ops.org/git.stella-ops.org/issues - **Discussions:** Internal team channels - **Documentation:** `docs/` directory in the repository --- ## Quick Reference Card ### Essential Commands ```bash # Start full platform cd deploy\compose docker compose -f docker-compose.dev.yaml up -d # Stop a specific service for debugging docker compose -f docker-compose.dev.yaml stop # View logs docker compose -f docker-compose.dev.yaml logs -f # Restart a service docker compose -f docker-compose.dev.yaml restart # Stop all services docker compose -f docker-compose.dev.yaml down # Stop all services and remove volumes (DESTRUCTIVE) docker compose -f docker-compose.dev.yaml down -v # Build the solution cd C:\dev\New folder\git.stella-ops.org dotnet build src\StellaOps.sln # Run tests dotnet test src\StellaOps.sln # Run a specific project cd src\Scanner\StellaOps.Scanner.WebService dotnet run ``` ### Service Default Ports | Service | Port | URL | Notes | |---------|------|-----|-------| | **Infrastructure** | | PostgreSQL | 5432 | `localhost:5432` | Primary database (REQUIRED) | | Valkey | 6379 | `localhost:6379` | Cache/events/queues (REQUIRED) | | RustFS | 8080 | http://localhost:8080 | S3-compatible storage (REQUIRED) | | NATS | 4222 | `nats://localhost:4222` | Optional alternative transport | | **Services** | | Authority | 8440 | https://localhost:8440 | OAuth2/OIDC auth | | Signer | 8441 | https://localhost:8441 | Cryptographic signing | | Attestor | 8442 | https://localhost:8442 | in-toto attestations | | Scanner.Web | 8444 | http://localhost:8444 | Scan API | | Concelier | 8445 | http://localhost:8445 | Advisory ingestion | | Notify | 8446 | http://localhost:8446 | Notifications | | IssuerDirectory | 8447 | http://localhost:8447 | CSAF publisher discovery | ### Visual Studio Shortcuts | Action | Shortcut | |--------|----------| | Start Debugging | F5 | | Start Without Debugging | Ctrl+F5 | | Stop Debugging | Shift+F5 | | Step Over | F10 | | Step Into | F11 | | Step Out | Shift+F11 | | Toggle Breakpoint | F9 | | Build Solution | Ctrl+Shift+B | | Rebuild Solution | Ctrl+Shift+F5 | --- **Document Version:** 1.0 **Last Updated:** 2025-12-22 **Maintained By:** StellaOps Development Team