1464 lines
62 KiB
Markdown
1464 lines
62 KiB
Markdown
# StellaOps Developer Onboarding Guide
|
|
|
|
> **Target Audience:** DevOps operators with developer knowledge who need to understand, deploy, and debug the StellaOps platform.
|
|
|
|
## Table of Contents
|
|
|
|
1. [Architecture Overview](#architecture-overview)
|
|
2. [Prerequisites](#prerequisites)
|
|
3. [Quick Start - Full Platform in Docker](#quick-start)
|
|
4. [Hybrid Debugging Workflow](#hybrid-debugging-workflow)
|
|
5. [Service-by-Service Debugging Guide](#service-by-service-debugging-guide)
|
|
6. [Configuration Deep Dive](#configuration-deep-dive)
|
|
7. [Common Development Workflows](#common-development-workflows)
|
|
8. [Troubleshooting](#troubleshooting)
|
|
|
|
---
|
|
|
|
## Architecture Overview
|
|
|
|
StellaOps is a deterministic SBOM + VEX platform built as a microservices architecture with 36+ services organized into functional domains.
|
|
|
|
### Runtime Topology - High-Level
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────────┐
|
|
│ INFRASTRUCTURE LAYER │
|
|
│ ┌──────────────────┐ ┌──────────────────┐ ┌─────────────────┐ │
|
|
│ │ PostgreSQL │ │ Valkey │ │ RustFS │ │
|
|
│ │ (v16+ ONLY) │ │ (Redis-compat) │ │ (S3-like API) │ │
|
|
│ │ │ │ - Caching │ │ - Artifacts │ │
|
|
│ │ All services use │ │ - DPoP nonces │ │ - SBOMs │ │
|
|
│ │ PostgreSQL for │ │ - Event queues │ │ - Signatures │ │
|
|
│ │ persistent data │ │ - Rate limiting│ │ │ │
|
|
│ └──────────────────┘ └──────────────────┘ └─────────────────┘ │
|
|
│ │
|
|
│ ┌──────────────────────────────────────────────────────────────┐ │
|
|
│ │ Optional: NATS JetStream (alternative transport for queues) │ │
|
|
│ │ Only used if explicitly configured in appsettings │ │
|
|
│ └──────────────────────────────────────────────────────────────┘ │
|
|
└─────────────────────────────────────────────────────────────────────┘
|
|
▲
|
|
│
|
|
┌─────────────────────────────────────────────────────────────────────┐
|
|
│ AUTHENTICATION & SIGNING │
|
|
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
|
|
│ │ Authority │─▶│ Signer │─▶│ Attestor │ │
|
|
│ │ (OAuth2/OIDC)│ │(DSSE/PKIX) │ │(in-toto/DSSE)│ │
|
|
│ └──────────────┘ └──────────────┘ └──────────────┘ │
|
|
└─────────────────────────────────────────────────────────────────────┘
|
|
▲
|
|
│
|
|
┌─────────────────────────────────────────────────────────────────────┐
|
|
│ INGESTION & AGGREGATION │
|
|
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
|
|
│ │ Concelier │ │ Excititor │ │IssuerDirectry│ │
|
|
│ │(Advisories) │ │ (VEX) │ │(CSAF Pubshrs)│ │
|
|
│ └──────────────┘ └──────────────┘ └──────────────┘ │
|
|
└─────────────────────────────────────────────────────────────────────┘
|
|
▲
|
|
│
|
|
┌─────────────────────────────────────────────────────────────────────┐
|
|
│ SCANNING & ANALYSIS │
|
|
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
|
|
│ │Scanner.Web │ │Scanner.Worker│ │ AdvisoryAI │ │
|
|
│ │(API/Control) │ │(Analyzers) │ │(ML Analysis) │ │
|
|
│ └──────────────┘ └──────────────┘ └──────────────┘ │
|
|
│ ┌──────────────┐ ┌──────────────┐ │
|
|
│ │ RiskEngine │ │ Policy │ │
|
|
│ │ (Scoring) │ │ (Engine) │ │
|
|
│ └──────────────┘ └──────────────┘ │
|
|
└─────────────────────────────────────────────────────────────────────┘
|
|
▲
|
|
│
|
|
┌─────────────────────────────────────────────────────────────────────┐
|
|
│ ORCHESTRATION & WORKFLOW │
|
|
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
|
|
│ │ Scheduler │ │ Orchestrator │ │ TaskRunner │ │
|
|
│ │(Job Sched) │ │(Coordinator) │ │(Executor) │ │
|
|
│ └──────────────┘ └──────────────┘ └──────────────┘ │
|
|
└─────────────────────────────────────────────────────────────────────┘
|
|
▲
|
|
│
|
|
┌─────────────────────────────────────────────────────────────────────┘
|
|
│ EVENTS & NOTIFICATIONS │
|
|
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
|
|
│ │ Notify │ │ Notifier │ │TimelineIndex │ │
|
|
│ │(Slack/Teams) │ │ (Advanced) │ │ (Events) │ │
|
|
│ └──────────────┘ └──────────────┘ └──────────────┘ │
|
|
└─────────────────────────────────────────────────────────────────────┘
|
|
▲
|
|
│
|
|
┌─────────────────────────────────────────────────────────────────────┐
|
|
│ DATA & EXPORT │
|
|
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
|
|
│ │ExportCenter │ │EvidenceLocker│ │FindingsLedger│ │
|
|
│ │(SARIF/SBOM) │ │(Artifacts) │ │(Audit Trail) │ │
|
|
│ └──────────────┘ └──────────────┘ └──────────────┘ │
|
|
└─────────────────────────────────────────────────────────────────────┘
|
|
▲
|
|
│
|
|
┌─────────────────────────────────────────────────────────────────────┐
|
|
│ USER EXPERIENCE │
|
|
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
|
|
│ │ Gateway │ │ Web (UI) │ │ CLI │ │
|
|
│ │ (API Router) │ │ (Angular v17)│ │(Multi-plat) │ │
|
|
│ └──────────────┘ └──────────────┘ └──────────────┘ │
|
|
└─────────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
### Detailed Request Flow - Scan Execution Example
|
|
|
|
This diagram shows a complete scan request lifecycle with detailed routing through services:
|
|
|
|
```
|
|
┌──────────────────────────────────────────────────────────────────────────────────┐
|
|
│ 1. CLIENT REQUEST (CLI or Web UI) │
|
|
│ $ stella scan docker://alpine:latest --sbom-format=spdx │
|
|
└───────────────────────────────────┬──────────────────────────────────────────────┘
|
|
│ HTTPS
|
|
▼
|
|
┌──────────────────────────────────────────────────────────────────────────────────┐
|
|
│ 2. GATEWAY (API Router) │
|
|
│ - Terminates TLS │
|
|
│ - Routes to appropriate backend service │
|
|
│ - Load balancing (if multiple instances) │
|
|
└───────────────────────────────────┬──────────────────────────────────────────────┘
|
|
│ HTTP (internal)
|
|
▼
|
|
┌──────────────────────────────────────────────────────────────────────────────────┐
|
|
│ 3. AUTHORITY (Authentication) │
|
|
│ - Validates OAuth2 access token (DPoP-bound) │
|
|
│ - Checks DPoP proof against Valkey nonce cache │
|
|
│ - Returns user identity and scopes │
|
|
│ │
|
|
│ ┌─────────────┐ │
|
|
│ │ Valkey │◀── DPoP nonce validation (GET/SET) │
|
|
│ │ (Cache) │ │
|
|
│ └─────────────┘ │
|
|
│ ┌─────────────┐ │
|
|
│ │ PostgreSQL │◀── User/client lookup (SELECT) │
|
|
│ └─────────────┘ │
|
|
└───────────────────────────────────┬──────────────────────────────────────────────┘
|
|
│ Authenticated request
|
|
▼
|
|
┌──────────────────────────────────────────────────────────────────────────────────┐
|
|
│ 4. SCANNER.WEB (Scan API Controller) │
|
|
│ - Validates scan request parameters │
|
|
│ - Creates scan job record in PostgreSQL │
|
|
│ - Enqueues scan job to Valkey queue (default) or NATS (if configured) │
|
|
│ │
|
|
│ ┌─────────────┐ │
|
|
│ │ PostgreSQL │◀── INSERT scan_jobs (job_id, image_ref, status='pending') │
|
|
│ └─────────────┘ │
|
|
│ ┌─────────────┐ │
|
|
│ │ Valkey │◀── XADD scanner:jobs (enqueue job message) │
|
|
│ │ (Queue) │ │
|
|
│ └─────────────┘ │
|
|
│ │
|
|
│ Returns: HTTP 202 Accepted { "job_id": "scan-abc123", "status": "queued" } │
|
|
└───────────────────────────────────┬──────────────────────────────────────────────┘
|
|
│
|
|
│ (Client polls for status)
|
|
│
|
|
┌──────────────────────────────────────────────────────────────────────────────────┐
|
|
│ 5. SCANNER.WORKER (Background Processor) │
|
|
│ - Consumes job from Valkey queue (XREADGROUP scanner:jobs) │
|
|
│ - Updates job status to 'running' │
|
|
│ - Downloads container image from registry │
|
|
│ - Executes analyzers (OS packages, language deps, files) │
|
|
│ - Generates SBOM (SPDX/CycloneDX) │
|
|
│ - Stores artifacts to RustFS │
|
|
│ │
|
|
│ ┌─────────────┐ │
|
|
│ │ PostgreSQL │◀── UPDATE scan_jobs SET status='running' │
|
|
│ │ │◀── INSERT sbom_documents, packages, vulnerabilities │
|
|
│ │ │◀── UPDATE scan_jobs SET status='completed' │
|
|
│ └─────────────┘ │
|
|
│ ┌─────────────┐ │
|
|
│ │ RustFS │◀── PUT /artifacts/scan-abc123/sbom.spdx.json │
|
|
│ │ (S3 API) │◀── PUT /artifacts/scan-abc123/image-layers.tar.gz │
|
|
│ └─────────────┘ │
|
|
│ ┌─────────────┐ │
|
|
│ │ Valkey │◀── XADD scanner:events (publish scan.completed event) │
|
|
│ │(Event Stream│ │
|
|
│ └─────────────┘ │
|
|
└───────────────────────────────────┬──────────────────────────────────────────────┘
|
|
│ Event published
|
|
▼
|
|
┌──────────────────────────────────────────────────────────────────────────────────┐
|
|
│ 6. EVENT PROPAGATION (Valkey Streams) │
|
|
│ │
|
|
│ ┌───────────────────────────────────────────────────────────────────┐ │
|
|
│ │ Valkey Event Stream: "scanner:events" │ │
|
|
│ │ Event: { "type": "scan.completed", "job_id": "scan-abc123", ... } │ │
|
|
│ └───────────────────────────────────────────────────────────────────┘ │
|
|
│ │ │
|
|
│ ┌──────────────┼──────────────┬───────────────┐ │
|
|
│ ▼ ▼ ▼ ▼ │
|
|
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
|
|
│ │ Notify │ │Timeline │ │ Policy │ │ Export │ │
|
|
│ │ Worker │ │ Indexer │ │ Engine │ │ Center │ │
|
|
│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │
|
|
│ │ │ │ │ │
|
|
│ │ (all subscribe to scanner:events via XREADGROUP) │
|
|
└─────────┼───────────────┼──────────────┼───────────────┼──────────────────────┘
|
|
│ │ │ │
|
|
▼ ▼ ▼ ▼
|
|
┌──────────────┐ ┌──────────────┐ ┌──────────┐ ┌──────────────┐
|
|
│ 7a. NOTIFY │ │ 7b. TIMELINE │ │ 7c.POLICY│ │ 7d. EXPORT │
|
|
│ │ │ INDEXER │ │ ENGINE │ │ CENTER │
|
|
│ - Query scan │ │ │ │ │ │ │
|
|
│ results │ │ - Index event│ │ - Eval │ │ - Generate │
|
|
│ - Check user │ │ timeline │ │ policy │ │ SARIF │
|
|
│ notif prefs│ │ - Store in │ │ rules │ │ - Export to │
|
|
│ - Send Slack │ │ PostgreSQL │ │ - Block/ │ │ external │
|
|
│ message │ │ │ │ Allow │ │ systems │
|
|
│ │ │ │ │ │ │ │
|
|
│ PostgreSQL ◀─┤ │ PostgreSQL ◀─┤ │PostgreSQL│ │ RustFS ◀─┤
|
|
│ (user prefs) │ │ (timeline) │ │(policies)│ │ (exports) │
|
|
└──────────────┘ └──────────────┘ └──────────┘ └──────────────┘
|
|
```
|
|
|
|
### Export Flow - SBOM Distribution
|
|
|
|
```
|
|
┌──────────────────────────────────────────────────────────────────────────────────┐
|
|
│ EXPORT REQUEST: GET /api/v1/scans/{scan_id}/export?format=spdx │
|
|
└───────────────────────────────────┬──────────────────────────────────────────────┘
|
|
│
|
|
▼
|
|
┌──────────────────────────────────────────────────────────────────────────────────┐
|
|
│ SCANNER.WEB or EXPORT CENTER │
|
|
│ │
|
|
│ 1. Query scan metadata from PostgreSQL │
|
|
│ ┌─────────────┐ │
|
|
│ │ PostgreSQL │◀── SELECT * FROM scan_jobs WHERE job_id = $1 │
|
|
│ │ │◀── SELECT * FROM sbom_documents WHERE scan_id = $1 │
|
|
│ └─────────────┘ │
|
|
│ │
|
|
│ 2. Retrieve SBOM artifact from RustFS │
|
|
│ ┌─────────────┐ │
|
|
│ │ RustFS │◀── GET /artifacts/scan-abc123/sbom.spdx.json │
|
|
│ └─────────────┘ │
|
|
│ │
|
|
│ 3. Sign SBOM with Signer service │
|
|
│ ┌─────────────┐ │
|
|
│ │ Signer │◀── POST /api/v1/sign (SBOM payload) │
|
|
│ │ │──▶ Returns: DSSE envelope with signature │
|
|
│ └─────────────┘ │
|
|
│ │
|
|
│ 4. Create in-toto attestation with Attestor │
|
|
│ ┌─────────────┐ │
|
|
│ │ Attestor │◀── POST /api/v1/attest (signed SBOM) │
|
|
│ │ │──▶ Returns: in-toto attestation bundle │
|
|
│ └─────────────┘ │
|
|
│ │
|
|
│ 5. Store final bundle to RustFS │
|
|
│ ┌─────────────┐ │
|
|
│ │ RustFS │◀── PUT /artifacts/scan-abc123/bundle.jsonl │
|
|
│ └─────────────┘ │
|
|
│ │
|
|
│ 6. Return signed bundle to client │
|
|
│ Returns: HTTP 200 OK (application/vnd.in-toto+json) │
|
|
└──────────────────────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
### Notification Flow - Vulnerability Alert
|
|
|
|
```
|
|
┌──────────────────────────────────────────────────────────────────────────────────┐
|
|
│ TRIGGER: New critical CVE detected in existing scan │
|
|
│ Source: Concelier advisory ingestion │
|
|
└───────────────────────────────────┬──────────────────────────────────────────────┘
|
|
│
|
|
▼
|
|
┌──────────────────────────────────────────────────────────────────────────────────┐
|
|
│ CONCELIER.WORKER (Advisory Processor) │
|
|
│ │
|
|
│ 1. Ingest new advisory from NVD/OSV/CSAF │
|
|
│ ┌─────────────┐ │
|
|
│ │ PostgreSQL │◀── INSERT INTO advisories (cve_id, severity, ...) │
|
|
│ └─────────────┘ │
|
|
│ │
|
|
│ 2. Match advisory against existing scans (PURL/CPE matching) │
|
|
│ ┌─────────────┐ │
|
|
│ │ PostgreSQL │◀── SELECT scans WHERE package_purl IN (affected_purls) │
|
|
│ └─────────────┘ │
|
|
│ │
|
|
│ 3. Publish drift event to Valkey │
|
|
│ ┌─────────────┐ │
|
|
│ │ Valkey │◀── XADD concelier:drift (new vulnerability found) │
|
|
│ └─────────────┘ │
|
|
└───────────────────────────────────┬──────────────────────────────────────────────┘
|
|
│ Event published
|
|
▼
|
|
┌──────────────────────────────────────────────────────────────────────────────────┐
|
|
│ NOTIFY.WORKER (Notification Processor) │
|
|
│ │
|
|
│ 1. Consume drift event from Valkey stream │
|
|
│ ┌─────────────┐ │
|
|
│ │ Valkey │◀── XREADGROUP concelier:drift notify-workers │
|
|
│ └─────────────┘ │
|
|
│ │
|
|
│ 2. Query user notification preferences │
|
|
│ ┌─────────────┐ │
|
|
│ │ PostgreSQL │◀── SELECT * FROM user_notification_preferences │
|
|
│ │ │ WHERE user_id = scan_owner AND channel = 'slack' │
|
|
│ └─────────────┘ │
|
|
│ │
|
|
│ 3. Render notification template │
|
|
│ Template: "🚨 New critical CVE-2024-1234 affects alpine:latest scan" │
|
|
│ │
|
|
│ 4. Deliver notification via configured channels │
|
|
│ ┌─────────────────────────────────────────────────────────┐ │
|
|
│ │ External APIs │ │
|
|
│ │ - POST https://hooks.slack.com/services/T00/B00/xxx │ │
|
|
│ │ - POST https://graph.microsoft.com/v1.0/teams/channels │ │
|
|
│ │ - SMTP send (email) │ │
|
|
│ └─────────────────────────────────────────────────────────┘ │
|
|
│ │
|
|
│ 5. Store delivery receipt in PostgreSQL │
|
|
│ ┌─────────────┐ │
|
|
│ │ PostgreSQL │◀── INSERT INTO notification_deliveries (status, ...) │
|
|
│ └─────────────┘ │
|
|
└──────────────────────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
### Key Architectural Principles
|
|
|
|
1. **Deterministic Evidence** - Same inputs always produce same outputs
|
|
2. **VEX-First Decisioning** - Policy decisions based on OpenVEX statements
|
|
3. **Offline-First** - Full air-gap operation supported
|
|
4. **Plugin Architecture** - Extensible connectors for advisories, analyzers, auth
|
|
5. **Sovereign Crypto** - FIPS, eIDAS, GOST, SM support
|
|
6. **Schema Isolation** - Per-module PostgreSQL schemas
|
|
|
|
### Service Categories
|
|
|
|
| Category | Services | Purpose |
|
|
|----------|----------|---------|
|
|
| **Infrastructure** | PostgreSQL v16+, Valkey 8.0, RustFS, NATS (optional) | Database, cache/messaging, object storage, optional queue transport |
|
|
| **Auth & Signing** | Authority, Signer, Attestor | OAuth2/OIDC with DPoP, cryptographic signing, in-toto attestations |
|
|
| **Ingestion** | Concelier, Excititor, IssuerDirectory | Advisory/VEX ingestion, normalization, merging, CSAF publisher discovery |
|
|
| **Scanning** | Scanner.Web, Scanner.Worker, AdvisoryAI | Container scanning, SBOM generation (SPDX/CDX), ML vulnerability analysis |
|
|
| **Policy & Risk** | Policy Engine, RiskEngine | OPA/Rego policy evaluation, risk scoring, exploitability assessment |
|
|
| **Orchestration** | Scheduler, Orchestrator, TaskRunner | Job scheduling, workflow coordination, distributed task execution |
|
|
| **Notifications** | Notify, Notifier, TimelineIndexer | Event delivery (Slack/Teams/Email), notification management, timeline tracking |
|
|
| **Data & Export** | ExportCenter, EvidenceLocker, FindingsLedger | SARIF/SBOM export, evidence storage, immutable audit trail |
|
|
| **User Experience** | Gateway, Web UI, CLI | API routing, Angular v17 UI, multi-platform command-line tools |
|
|
|
|
---
|
|
|
|
## Prerequisites
|
|
|
|
### Required Software
|
|
|
|
1. **Docker Desktop** (Windows/Mac) or **Docker Engine + Docker Compose** (Linux)
|
|
- Version: 20.10+ recommended
|
|
- Enable WSL2 backend (Windows)
|
|
|
|
2. **.NET 10 SDK**
|
|
- Download: https://dotnet.microsoft.com/download/dotnet/10.0
|
|
- Verify: `dotnet --version` (should show 10.0.x)
|
|
|
|
3. **Visual Studio 2022** (v17.12+) or **Visual Studio Code**
|
|
- Workload: ASP.NET and web development
|
|
- Workload: .NET desktop development
|
|
- Extension (VS Code): C# Dev Kit
|
|
|
|
4. **Git**
|
|
- Version: 2.30+ recommended
|
|
|
|
### Optional Tools
|
|
|
|
- **PostgreSQL Client** (psql, pgAdmin, DBeaver) - for database inspection
|
|
- **Redis Insight** or **Another Redis Desktop Manager** - for Valkey inspection (Valkey is Redis-compatible)
|
|
- **Postman/Insomnia** - for API testing
|
|
- **AWS CLI or s3cmd** - for RustFS (S3-compatible) inspection
|
|
|
|
### System Requirements
|
|
|
|
- **RAM:** 16 GB minimum, 32 GB recommended
|
|
- **Disk:** 50 GB free space (for Docker images, volumes, build artifacts)
|
|
- **CPU:** 4 cores minimum, 8 cores recommended
|
|
|
|
---
|
|
|
|
## Quick Start
|
|
|
|
### Step 1: Clone the Repository
|
|
|
|
```bash
|
|
cd C:\dev\
|
|
git clone https://git.stella-ops.org/stella-ops.org/git.stella-ops.org
|
|
cd git.stella-ops.org
|
|
```
|
|
|
|
### Step 2: Prepare Environment Configuration
|
|
|
|
```bash
|
|
# Copy the development environment template
|
|
cd deploy\compose
|
|
copy env\dev.env.example .env
|
|
|
|
# Edit .env with your preferred text editor
|
|
notepad .env
|
|
```
|
|
|
|
**Key settings to configure:**
|
|
|
|
```bash
|
|
# PostgreSQL Database
|
|
POSTGRES_USER=stellaops
|
|
POSTGRES_PASSWORD=your_secure_password_here
|
|
POSTGRES_DB=stellaops_platform
|
|
POSTGRES_PORT=5432
|
|
|
|
# Valkey (Redis-compatible cache and messaging)
|
|
VALKEY_PORT=6379
|
|
|
|
# RustFS Object Storage
|
|
RUSTFS_HTTP_PORT=8080
|
|
|
|
# Service ports (adjust if conflicts exist)
|
|
AUTHORITY_PORT=8440
|
|
SIGNER_PORT=8441
|
|
ATTESTOR_PORT=8442
|
|
CONCELIER_PORT=8445
|
|
SCANNER_WEB_PORT=8444
|
|
NOTIFY_WEB_PORT=8446
|
|
|
|
# Scanner configuration (Valkey default, can switch to NATS if needed)
|
|
SCANNER_EVENTS_DRIVER=valkey
|
|
SCANNER_EVENTS_DSN=valkey:6379
|
|
|
|
# Scheduler configuration (Valkey default, can switch to NATS if needed)
|
|
SCHEDULER_QUEUE_KIND=Valkey
|
|
SCHEDULER_QUEUE_VALKEY_URL=valkey:6379
|
|
|
|
# Authority configuration
|
|
AUTHORITY_ISSUER=https://authority:8440
|
|
SIGNER_POE_INTROSPECT_URL=https://www.stella-ops.org/license/introspect
|
|
```
|
|
|
|
### Step 3: Start the Full Platform
|
|
|
|
```bash
|
|
# From deploy/compose directory
|
|
docker compose -f docker-compose.dev.yaml up -d
|
|
```
|
|
|
|
**This will start all infrastructure and services:**
|
|
- PostgreSQL v16+ (port 5432) - Primary database for all services
|
|
- Valkey 8.0 (port 6379) - Cache, DPoP nonces, event streams, rate limiting
|
|
- RustFS (port 8080) - S3-compatible object storage for artifacts/SBOMs
|
|
- NATS JetStream (port 4222) - Optional transport (only if configured)
|
|
- Authority (port 8440) - OAuth2/OIDC authentication
|
|
- Signer (port 8441) - Cryptographic signing
|
|
- Attestor (port 8442) - in-toto attestation generation
|
|
- Scanner.Web (port 8444) - Scan API
|
|
- Concelier (port 8445) - Advisory ingestion
|
|
- And 30+ more services...
|
|
|
|
### Step 4: Verify Services Are Running
|
|
|
|
```bash
|
|
# Check all services are up
|
|
docker compose -f docker-compose.dev.yaml ps
|
|
|
|
# Check logs for a specific service
|
|
docker compose -f docker-compose.dev.yaml logs -f scanner-web
|
|
|
|
# Check infrastructure health
|
|
docker compose -f docker-compose.dev.yaml logs postgres
|
|
docker compose -f docker-compose.dev.yaml logs valkey
|
|
docker compose -f docker-compose.dev.yaml logs rustfs
|
|
```
|
|
|
|
### Step 5: Access the Platform
|
|
|
|
Open your browser and navigate to:
|
|
|
|
- **RustFS:** http://localhost:8080 (S3-compatible object storage)
|
|
- **Scanner API:** http://localhost:8444/swagger (if Swagger enabled)
|
|
- **Concelier API:** http://localhost:8445/swagger
|
|
- **Authority:** http://localhost:8440/.well-known/openid-configuration (OIDC discovery)
|
|
|
|
---
|
|
|
|
## Hybrid Debugging Workflow
|
|
|
|
The hybrid workflow allows you to:
|
|
1. Run infrastructure (databases, queues) in Docker
|
|
2. Run most services in Docker
|
|
3. **Selectively debug** one or two services in Visual Studio
|
|
|
|
### Workflow Overview
|
|
|
|
```
|
|
┌────────────────────────────────────────────────────────────┐
|
|
│ DOCKER ENVIRONMENT │
|
|
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
|
|
│ │PostgreSQL│ │ Valkey │ │ RustFS │ │
|
|
│ │ (DB) │ │(Cache/Msg│ │(Storage) │ │
|
|
│ └──────────┘ └──────────┘ └──────────┘ │
|
|
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
|
|
│ │ Authority│ │ Signer │ │ Attestor │ │
|
|
│ └──────────┘ └──────────┘ └──────────┘ │
|
|
│ ┌──────────┐ ┌──────────┐ │
|
|
│ │Concelier │ │ Excititor│ ← Running normally │
|
|
│ └──────────┘ └──────────┘ │
|
|
└────────────────────────────────────────────────────────────┘
|
|
▲
|
|
│ HTTP calls + Valkey streams
|
|
▼
|
|
┌────────────────────────────────────────────────────────────┐
|
|
│ VISUAL STUDIO (F5) │
|
|
│ ┌────────────────────────────────────────────┐ │
|
|
│ │ Scanner.WebService │ │
|
|
│ │ Running on http://localhost:5210 │ │
|
|
│ │ (Breakpoints, hot reload, debugging) │ ← YOU DEBUG HERE
|
|
│ └────────────────────────────────────────────┘ │
|
|
└────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
### Step-by-Step: Debug Scanner.WebService
|
|
|
|
#### 1. Stop the Docker Container for Scanner
|
|
|
|
```bash
|
|
cd deploy\compose
|
|
docker compose -f docker-compose.dev.yaml stop scanner-web
|
|
```
|
|
|
|
**Verify it's stopped:**
|
|
```bash
|
|
docker compose -f docker-compose.dev.yaml ps scanner-web
|
|
# Should show: State = "exited"
|
|
```
|
|
|
|
#### 2. Configure Local Development Settings
|
|
|
|
Create or modify the service's `appsettings.Development.json`:
|
|
|
|
```bash
|
|
cd C:\dev\New folder\git.stella-ops.org\src\Scanner\StellaOps.Scanner.WebService
|
|
```
|
|
|
|
**Create `appsettings.Development.json`:**
|
|
|
|
```json
|
|
{
|
|
"Logging": {
|
|
"LogLevel": {
|
|
"Default": "Information",
|
|
"Microsoft.AspNetCore": "Warning",
|
|
"StellaOps": "Debug"
|
|
}
|
|
},
|
|
"ConnectionStrings": {
|
|
"DefaultConnection": "Host=localhost;Port=5432;Database=stellaops_platform;Username=stellaops;Password=your_password_here;Include Error Detail=true"
|
|
},
|
|
"Scanner": {
|
|
"Storage": {
|
|
"Mongo": {
|
|
"ConnectionString": "mongodb://stellaops:your_password_here@localhost:27017"
|
|
}
|
|
},
|
|
"ArtifactStore": {
|
|
"Driver": "rustfs",
|
|
"Endpoint": "http://localhost:8080/api/v1",
|
|
"Bucket": "scanner-artifacts",
|
|
"TimeoutSeconds": 30
|
|
},
|
|
"Queue": {
|
|
"Broker": "nats://localhost:4222"
|
|
},
|
|
"Events": {
|
|
"Enabled": false
|
|
}
|
|
},
|
|
"Authority": {
|
|
"Issuer": "https://localhost:8440",
|
|
"BaseUrl": "https://localhost:8440",
|
|
"BypassNetworks": ["127.0.0.1", "::1"]
|
|
}
|
|
}
|
|
```
|
|
|
|
**Note:** Adjust connection strings to match your Docker infrastructure ports. If PostgreSQL is on Docker's bridge network, you may need to expose it on `localhost:5432`.
|
|
|
|
#### 3. Expose Docker Services to localhost
|
|
|
|
For services running in Docker to be accessible from your host machine, ensure ports are mapped in `docker-compose.dev.yaml`:
|
|
|
|
```yaml
|
|
# Already configured in docker-compose.dev.yaml
|
|
postgres:
|
|
ports:
|
|
- "${POSTGRES_PORT:-5432}:5432"
|
|
|
|
mongo:
|
|
ports:
|
|
- "27017:27017"
|
|
|
|
nats:
|
|
ports:
|
|
- "${NATS_CLIENT_PORT:-4222}:4222"
|
|
|
|
rustfs:
|
|
ports:
|
|
- "${RUSTFS_HTTP_PORT:-8080}:8080"
|
|
```
|
|
|
|
**Verify connectivity:**
|
|
|
|
```bash
|
|
# Test PostgreSQL
|
|
psql -h localhost -U stellaops -d stellaops_platform
|
|
|
|
# Test NATS
|
|
telnet localhost 4222
|
|
|
|
# Test RustFS
|
|
curl http://localhost:8080/health
|
|
```
|
|
|
|
#### 4. Open Solution in Visual Studio
|
|
|
|
```bash
|
|
# Open the solution
|
|
cd C:\dev\New folder\git.stella-ops.org
|
|
start src\StellaOps.sln
|
|
```
|
|
|
|
**In Visual Studio:**
|
|
|
|
1. Right-click `StellaOps.Scanner.WebService` project
|
|
2. Select **"Set as Startup Project"**
|
|
3. Press **F5** to start debugging
|
|
|
|
**Expected output:**
|
|
```
|
|
info: Microsoft.Hosting.Lifetime[14]
|
|
Now listening on: http://localhost:5210
|
|
info: Microsoft.Hosting.Lifetime[0]
|
|
Application started. Press Ctrl+C to shut down.
|
|
```
|
|
|
|
#### 5. Update Other Services to Call localhost
|
|
|
|
Since you're running Scanner.WebService on `localhost:5210` instead of `scanner-web:8444`, you need to update any services that call it.
|
|
|
|
**Option A: Environment Variables (Docker containers)**
|
|
|
|
Update `.env` file:
|
|
```bash
|
|
# Use host.docker.internal to reach host machine from Docker
|
|
SCANNER_WEB_BASEURL=http://host.docker.internal:5210
|
|
```
|
|
|
|
Restart dependent services:
|
|
```bash
|
|
docker compose -f docker-compose.dev.yaml restart scheduler-web
|
|
```
|
|
|
|
**Option B: Modify docker-compose.dev.yaml**
|
|
|
|
```yaml
|
|
scheduler-web:
|
|
environment:
|
|
SCHEDULER__WORKER__RUNNER__SCANNER__BASEADDRESS: "http://host.docker.internal:5210"
|
|
```
|
|
|
|
Then restart:
|
|
```bash
|
|
docker compose -f docker-compose.dev.yaml up -d scheduler-web
|
|
```
|
|
|
|
#### 6. Set Breakpoints and Debug
|
|
|
|
1. Navigate to `Program.cs` in Scanner.WebService
|
|
2. Set a breakpoint on a line in a controller or service method
|
|
3. Trigger the endpoint using:
|
|
- Swagger UI (if enabled): http://localhost:5210/swagger
|
|
- Postman/curl
|
|
- CLI command
|
|
|
|
**Example curl:**
|
|
```bash
|
|
curl -X POST http://localhost:5210/api/scans \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"imageRef": "alpine:latest"}'
|
|
```
|
|
|
|
Your breakpoint should hit, and you can step through code.
|
|
|
|
#### 7. Return to Docker Mode
|
|
|
|
When you're done debugging:
|
|
|
|
```bash
|
|
# Stop Visual Studio debugger (Shift+F5)
|
|
|
|
# Restart the Docker container
|
|
cd deploy\compose
|
|
docker compose -f docker-compose.dev.yaml start scanner-web
|
|
|
|
# Verify it's running
|
|
docker compose -f docker-compose.dev.yaml ps scanner-web
|
|
```
|
|
|
|
---
|
|
|
|
## Service-by-Service Debugging Guide
|
|
|
|
### Authority (OAuth2/OIDC Provider)
|
|
|
|
**Project:** `src/Authority/StellaOps.Authority/StellaOps.Authority.csproj`
|
|
|
|
**Stop Docker container:**
|
|
```bash
|
|
docker compose -f docker-compose.dev.yaml stop authority
|
|
```
|
|
|
|
**Configuration:** `appsettings.Development.json`
|
|
```json
|
|
{
|
|
"ConnectionStrings": {
|
|
"DefaultConnection": "Host=localhost;Port=5432;Database=stellaops_platform;Username=stellaops;Password=your_password"
|
|
},
|
|
"StellaOps_Authority": {
|
|
"Issuer": "https://localhost:5001",
|
|
"Mongo": {
|
|
"ConnectionString": "mongodb://stellaops:your_password@localhost:27017"
|
|
}
|
|
},
|
|
"Kestrel": {
|
|
"Endpoints": {
|
|
"Https": {
|
|
"Url": "https://localhost:5001"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
**Run in Visual Studio:** F5 on `StellaOps.Authority` project
|
|
|
|
**Default URL:** https://localhost:5001
|
|
|
|
**Update dependent services:**
|
|
```bash
|
|
# In .env
|
|
AUTHORITY_ISSUER=https://host.docker.internal:5001
|
|
```
|
|
|
|
### Concelier (Advisory Ingestion)
|
|
|
|
**Project:** `src/Concelier/StellaOps.Concelier.WebService/StellaOps.Concelier.WebService.csproj`
|
|
|
|
**Stop Docker:**
|
|
```bash
|
|
docker compose -f docker-compose.dev.yaml stop concelier
|
|
```
|
|
|
|
**Configuration:** `appsettings.Development.json`
|
|
```json
|
|
{
|
|
"ConnectionStrings": {
|
|
"DefaultConnection": "Host=localhost;Port=5432;Database=stellaops_platform;Username=stellaops;Password=your_password"
|
|
},
|
|
"Concelier": {
|
|
"Storage": {
|
|
"Mongo": {
|
|
"ConnectionString": "mongodb://stellaops:your_password@localhost:27017"
|
|
},
|
|
"S3": {
|
|
"Endpoint": "http://localhost:9000",
|
|
"AccessKeyId": "stellaops",
|
|
"SecretAccessKey": "your_password"
|
|
}
|
|
},
|
|
"Authority": {
|
|
"BaseUrl": "https://localhost:8440"
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
**Run:** F5 on `StellaOps.Concelier.WebService`
|
|
|
|
**Default URL:** http://localhost:5000
|
|
|
|
### Scanner.Worker (Background Analyzer)
|
|
|
|
**Project:** `src/Scanner/StellaOps.Scanner.Worker/StellaOps.Scanner.Worker.csproj`
|
|
|
|
**Stop Docker:**
|
|
```bash
|
|
docker compose -f docker-compose.dev.yaml stop scanner-worker
|
|
```
|
|
|
|
**Configuration:** Same as Scanner.WebService (shares settings)
|
|
|
|
**Run:** F5 on `StellaOps.Scanner.Worker`
|
|
|
|
**Note:** Worker has no HTTP endpoint - it consumes from NATS queue
|
|
|
|
### Scheduler.WebService
|
|
|
|
**Project:** `src/Scheduler/StellaOps.Scheduler.WebService/StellaOps.Scheduler.WebService.csproj`
|
|
|
|
**Stop Docker:**
|
|
```bash
|
|
docker compose -f docker-compose.dev.yaml stop scheduler-web
|
|
```
|
|
|
|
**Configuration:** `appsettings.Development.json`
|
|
```json
|
|
{
|
|
"ConnectionStrings": {
|
|
"DefaultConnection": "Host=localhost;Port=5432;Database=stellaops_orchestrator;Username=stellaops;Password=your_password"
|
|
},
|
|
"Scheduler": {
|
|
"Queue": {
|
|
"Kind": "Nats",
|
|
"Nats": {
|
|
"Url": "nats://localhost:4222"
|
|
}
|
|
},
|
|
"Worker": {
|
|
"Runner": {
|
|
"Scanner": {
|
|
"BaseAddress": "http://localhost:5210"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
**Run:** F5 on `StellaOps.Scheduler.WebService`
|
|
|
|
### Notify.WebService
|
|
|
|
**Project:** `src/Notify/StellaOps.Notify.WebService/StellaOps.Notify.WebService.csproj`
|
|
|
|
**Stop Docker:**
|
|
```bash
|
|
docker compose -f docker-compose.dev.yaml stop notify-web
|
|
```
|
|
|
|
**Configuration:** Uses `etc/notify.dev.yaml`
|
|
|
|
**Run:** F5 on `StellaOps.Notify.WebService`
|
|
|
|
---
|
|
|
|
## Configuration Deep Dive
|
|
|
|
### Configuration Hierarchy
|
|
|
|
All services follow this configuration priority (highest to lowest):
|
|
|
|
1. **Environment Variables** - `STELLAOPS_<MODULE>_<SETTING>` or `<MODULE>__<SETTING>`
|
|
2. **appsettings.{Environment}.json** - `appsettings.Development.json`, `appsettings.Production.json`
|
|
3. **appsettings.json** - Base configuration
|
|
4. **YAML files** - `../etc/<service>.yaml`, `../etc/<service>.local.yaml`
|
|
|
|
### Common Configuration Patterns
|
|
|
|
#### PostgreSQL Connection Strings
|
|
|
|
```json
|
|
{
|
|
"ConnectionStrings": {
|
|
"DefaultConnection": "Host=localhost;Port=5432;Database=<db_name>;Username=stellaops;Password=<password>;Pooling=true;Minimum Pool Size=1;Maximum Pool Size=100;Command Timeout=60"
|
|
}
|
|
}
|
|
```
|
|
|
|
**Database names by service:**
|
|
- Scanner: `stellaops_platform` or `scanner_*`
|
|
- Orchestrator: `stellaops_orchestrator`
|
|
- Authority: `stellaops_platform` (shared, schema-isolated)
|
|
- Concelier: `stellaops_platform` (vuln schema)
|
|
- Notify: `stellaops_platform` (notify schema)
|
|
|
|
#### Valkey Configuration (Default Transport)
|
|
|
|
```json
|
|
{
|
|
"Scanner": {
|
|
"Events": {
|
|
"Driver": "valkey",
|
|
"Dsn": "localhost:6379"
|
|
},
|
|
"Cache": {
|
|
"Redis": {
|
|
"ConnectionString": "localhost:6379"
|
|
}
|
|
}
|
|
},
|
|
"Scheduler": {
|
|
"Queue": {
|
|
"Kind": "Valkey",
|
|
"Valkey": {
|
|
"Url": "localhost:6379"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
#### NATS Queue Configuration (Optional Alternative Transport)
|
|
|
|
```json
|
|
{
|
|
"Scanner": {
|
|
"Events": {
|
|
"Driver": "nats",
|
|
"Dsn": "nats://localhost:4222"
|
|
}
|
|
},
|
|
"Scheduler": {
|
|
"Queue": {
|
|
"Kind": "Nats",
|
|
"Nats": {
|
|
"Url": "nats://localhost:4222"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
#### RustFS Configuration (S3-Compatible Object Storage)
|
|
|
|
```json
|
|
{
|
|
"Scanner": {
|
|
"Storage": {
|
|
"RustFS": {
|
|
"Endpoint": "http://localhost:8080",
|
|
"AccessKeyId": "stellaops",
|
|
"SecretAccessKey": "your_password",
|
|
"BucketName": "scanner-artifacts",
|
|
"Region": "us-east-1",
|
|
"ForcePathStyle": true
|
|
}
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
#### RustFS Configuration
|
|
|
|
```json
|
|
{
|
|
"Scanner": {
|
|
"ArtifactStore": {
|
|
"Driver": "rustfs",
|
|
"Endpoint": "http://localhost:8080/api/v1",
|
|
"Bucket": "scanner-artifacts",
|
|
"TimeoutSeconds": 30
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
### Environment Variable Mapping
|
|
|
|
ASP.NET Core uses `__` (double underscore) for nested configuration:
|
|
|
|
```bash
|
|
# This JSON configuration:
|
|
{
|
|
"Scanner": {
|
|
"Queue": {
|
|
"Broker": "nats://localhost:4222"
|
|
}
|
|
}
|
|
}
|
|
|
|
# Can be set via environment variable:
|
|
SCANNER__QUEUE__BROKER=nats://localhost:4222
|
|
|
|
# Or with STELLAOPS_ prefix:
|
|
STELLAOPS_SCANNER__QUEUE__BROKER=nats://localhost:4222
|
|
```
|
|
|
|
---
|
|
|
|
## Common Development Workflows
|
|
|
|
### Workflow 1: Debug a Single Service with Full Stack
|
|
|
|
**Scenario:** You need to debug Scanner.WebService while all other services run normally.
|
|
|
|
```bash
|
|
# 1. Start full platform
|
|
cd deploy\compose
|
|
docker compose -f docker-compose.dev.yaml up -d
|
|
|
|
# 2. Stop the service you want to debug
|
|
docker compose -f docker-compose.dev.yaml stop scanner-web
|
|
|
|
# 3. Open Visual Studio
|
|
cd C:\dev\New folder\git.stella-ops.org
|
|
start src\StellaOps.sln
|
|
|
|
# 4. Set Scanner.WebService as startup project and F5
|
|
|
|
# 5. Test the service
|
|
curl -X POST http://localhost:5210/api/scans -H "Content-Type: application/json" -d '{"imageRef":"alpine:latest"}'
|
|
|
|
# 6. When done, stop VS debugger and restart Docker container
|
|
docker compose -f docker-compose.dev.yaml start scanner-web
|
|
```
|
|
|
|
### Workflow 2: Debug Multiple Services Together
|
|
|
|
**Scenario:** Debug Scanner.WebService and Scanner.Worker together.
|
|
|
|
```bash
|
|
# 1. Stop both containers
|
|
docker compose -f docker-compose.dev.yaml stop scanner-web scanner-worker
|
|
|
|
# 2. In Visual Studio, configure multiple startup projects:
|
|
# - Right-click solution > Properties
|
|
# - Set "Multiple startup projects"
|
|
# - Select Scanner.WebService: Start
|
|
# - Select Scanner.Worker: Start
|
|
|
|
# 3. Press F5 to debug both simultaneously
|
|
```
|
|
|
|
### Workflow 3: Test Integration with Modified Code
|
|
|
|
**Scenario:** You modified Concelier and want to test how Scanner integrates with it.
|
|
|
|
```bash
|
|
# 1. Build Concelier locally
|
|
cd src\Concelier\StellaOps.Concelier.WebService
|
|
dotnet build
|
|
|
|
# 2. Stop Docker Concelier
|
|
cd ..\..\..\deploy\compose
|
|
docker compose -f docker-compose.dev.yaml stop concelier
|
|
|
|
# 3. Run Concelier in Visual Studio (F5)
|
|
|
|
# 4. Keep Scanner in Docker, but point it to localhost Concelier
|
|
# Update .env:
|
|
CONCELIER_BASEURL=http://host.docker.internal:5000
|
|
|
|
# 5. Restart Scanner to pick up new config
|
|
docker compose -f docker-compose.dev.yaml restart scanner-web
|
|
```
|
|
|
|
### Workflow 4: Reset Database State
|
|
|
|
**Scenario:** You need a clean database to test migrations or start fresh.
|
|
|
|
```bash
|
|
# 1. Stop all services
|
|
docker compose -f docker-compose.dev.yaml down
|
|
|
|
# 2. Remove database volumes
|
|
docker volume rm compose_postgres-data
|
|
docker volume rm compose_mongo-data
|
|
|
|
# 3. Restart platform (will recreate volumes and databases)
|
|
docker compose -f docker-compose.dev.yaml up -d
|
|
|
|
# 4. Wait for migrations to run
|
|
docker compose -f docker-compose.dev.yaml logs -f postgres
|
|
# Look for migration completion messages
|
|
```
|
|
|
|
### Workflow 5: Test Offline/Air-Gap Mode
|
|
|
|
**Scenario:** Test the platform in offline mode.
|
|
|
|
```bash
|
|
# 1. Use the air-gap compose profile
|
|
cd deploy\compose
|
|
docker compose -f docker-compose.airgap.yaml up -d
|
|
|
|
# 2. Verify no external network calls
|
|
docker compose -f docker-compose.airgap.yaml logs | grep -i "external\|outbound\|internet"
|
|
```
|
|
|
|
---
|
|
|
|
## Troubleshooting
|
|
|
|
### Common Issues
|
|
|
|
#### 1. Port Already in Use
|
|
|
|
**Error:**
|
|
```
|
|
Error starting userland proxy: listen tcp 0.0.0.0:5432: bind: address already in use
|
|
```
|
|
|
|
**Solutions:**
|
|
|
|
**Option A: Change the port in .env**
|
|
```bash
|
|
# Edit .env
|
|
POSTGRES_PORT=5433 # Use a different port
|
|
```
|
|
|
|
**Option B: Stop the conflicting process**
|
|
```bash
|
|
# Windows
|
|
netstat -ano | findstr :5432
|
|
taskkill /PID <PID> /F
|
|
|
|
# Linux/Mac
|
|
lsof -i :5432
|
|
kill -9 <PID>
|
|
```
|
|
|
|
#### 2. Cannot Connect to PostgreSQL from Visual Studio
|
|
|
|
**Error:**
|
|
```
|
|
Npgsql.NpgsqlException: Connection refused
|
|
```
|
|
|
|
**Solutions:**
|
|
|
|
1. **Verify PostgreSQL is accessible from host:**
|
|
```bash
|
|
psql -h localhost -U stellaops -d stellaops_platform
|
|
```
|
|
|
|
2. **Check Docker network:**
|
|
```bash
|
|
docker network inspect compose_stellaops
|
|
# Ensure your service has "host.docker.internal" DNS resolution
|
|
```
|
|
|
|
3. **Update connection string:**
|
|
```json
|
|
{
|
|
"ConnectionStrings": {
|
|
"DefaultConnection": "Host=localhost;Port=5432;Database=stellaops_platform;Username=stellaops;Password=your_password;Include Error Detail=true"
|
|
}
|
|
}
|
|
```
|
|
|
|
#### 3. NATS Connection Refused
|
|
|
|
**Error:**
|
|
```
|
|
NATS connection error: connection refused
|
|
```
|
|
|
|
**Solution:**
|
|
|
|
By default, services use **Valkey** for messaging, not NATS. Ensure Valkey is running:
|
|
```bash
|
|
docker compose -f docker-compose.dev.yaml ps valkey
|
|
# Should show: State = "Up"
|
|
|
|
# Test connectivity
|
|
telnet localhost 6379
|
|
```
|
|
|
|
Update configuration to use Valkey (default):
|
|
```json
|
|
{
|
|
"Scanner": {
|
|
"Events": {
|
|
"Driver": "valkey",
|
|
"Dsn": "localhost:6379"
|
|
}
|
|
},
|
|
"Scheduler": {
|
|
"Queue": {
|
|
"Kind": "Valkey",
|
|
"Valkey": {
|
|
"Url": "localhost:6379"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
**If you explicitly want to use NATS** (optional):
|
|
```bash
|
|
docker compose -f docker-compose.dev.yaml ps nats
|
|
# Ensure NATS is running
|
|
|
|
# Update appsettings.Development.json:
|
|
{
|
|
"Scanner": {
|
|
"Events": {
|
|
"Driver": "nats",
|
|
"Dsn": "nats://localhost:4222"
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
#### 4. Valkey Connection Refused
|
|
|
|
**Error:**
|
|
```
|
|
StackExchange.Redis.RedisConnectionException: It was not possible to connect to the redis server(s)
|
|
```
|
|
|
|
**Solutions:**
|
|
|
|
1. **Check Valkey is running:**
|
|
```bash
|
|
docker compose -f docker-compose.dev.yaml ps valkey
|
|
# Should show: State = "Up"
|
|
|
|
# Check logs
|
|
docker compose -f docker-compose.dev.yaml logs valkey
|
|
```
|
|
|
|
2. **Reset Valkey:**
|
|
```bash
|
|
docker compose -f docker-compose.dev.yaml stop valkey
|
|
docker volume rm compose_valkey-data
|
|
docker compose -f docker-compose.dev.yaml up -d valkey
|
|
```
|
|
|
|
#### 5. Service Cannot Reach host.docker.internal
|
|
|
|
**Error:**
|
|
```
|
|
Could not resolve host: host.docker.internal
|
|
```
|
|
|
|
**Solution (Windows/Mac):**
|
|
|
|
Should work automatically with Docker Desktop.
|
|
|
|
**Solution (Linux):**
|
|
|
|
Add to docker-compose.dev.yaml:
|
|
```yaml
|
|
services:
|
|
scanner-web:
|
|
extra_hosts:
|
|
- "host.docker.internal:host-gateway"
|
|
```
|
|
|
|
Or use the host's IP address:
|
|
```bash
|
|
# Find host IP
|
|
ip addr show docker0
|
|
# Use that IP instead of host.docker.internal
|
|
```
|
|
|
|
#### 6. Certificate Validation Errors (Authority/HTTPS)
|
|
|
|
**Error:**
|
|
```
|
|
The SSL connection could not be established
|
|
```
|
|
|
|
**Solution:**
|
|
|
|
For development, disable certificate validation:
|
|
```json
|
|
{
|
|
"Authority": {
|
|
"ValidateCertificate": false
|
|
}
|
|
}
|
|
```
|
|
|
|
Or trust the development certificate:
|
|
```bash
|
|
dotnet dev-certs https --trust
|
|
```
|
|
|
|
#### 7. Build Errors - Missing SDK
|
|
|
|
**Error:**
|
|
```
|
|
error MSB4236: The SDK 'Microsoft.NET.Sdk.Web' specified could not be found
|
|
```
|
|
|
|
**Solution:**
|
|
|
|
Install .NET 10 SDK:
|
|
```bash
|
|
# Verify installation
|
|
dotnet --list-sdks
|
|
|
|
# Should show:
|
|
# 10.0.xxx [C:\Program Files\dotnet\sdk]
|
|
```
|
|
|
|
#### 8. Hot Reload Not Working
|
|
|
|
**Symptom:** Changes in code don't reflect when running in Visual Studio.
|
|
|
|
**Solutions:**
|
|
|
|
1. Ensure Hot Reload is enabled: Tools > Options > Debugging > .NET Hot Reload > Enable Hot Reload
|
|
2. Rebuild the project: Ctrl+Shift+B
|
|
3. Restart debugging session: Shift+F5, then F5
|
|
|
|
#### 9. Docker Compose Fails to Parse .env
|
|
|
|
**Error:**
|
|
```
|
|
invalid interpolation format
|
|
```
|
|
|
|
**Solution:**
|
|
|
|
Ensure no spaces around `=` in .env:
|
|
```bash
|
|
# Wrong
|
|
POSTGRES_USER = stellaops
|
|
|
|
# Correct
|
|
POSTGRES_USER=stellaops
|
|
```
|
|
|
|
#### 10. Volume Permission Issues (Linux)
|
|
|
|
**Error:**
|
|
```
|
|
Permission denied writing to /data/db
|
|
```
|
|
|
|
**Solution:**
|
|
|
|
```bash
|
|
# Fix permissions on volume directories
|
|
sudo chown -R $USER:$USER ./volumes
|
|
|
|
# Or run Docker as root (not recommended for production)
|
|
sudo docker compose -f docker-compose.dev.yaml up -d
|
|
```
|
|
|
|
---
|
|
|
|
## Next Steps
|
|
|
|
### Learning Path
|
|
|
|
1. **Week 1: Infrastructure**
|
|
- Understand PostgreSQL schema isolation (all services use PostgreSQL)
|
|
- Learn Valkey streams for event queuing and caching
|
|
- Study RustFS S3-compatible object storage
|
|
- Optional: NATS JetStream as alternative transport
|
|
|
|
2. **Week 2: Core Services**
|
|
- Deep dive into Scanner architecture (analyzers, workers, caching)
|
|
- Understand Concelier advisory ingestion and merging
|
|
- Study VEX workflow in Excititor
|
|
|
|
3. **Week 3: Authentication & Security**
|
|
- Master OAuth2/OIDC flow in Authority
|
|
- Understand signing flow (Signer → Attestor → Rekor)
|
|
- Study policy evaluation engine
|
|
|
|
4. **Week 4: Integration**
|
|
- Build end-to-end scan workflow
|
|
- Implement custom Concelier connector
|
|
- Create custom notification rules
|
|
|
|
### Key Documentation
|
|
|
|
- **Architecture:** `docs/07_HIGH_LEVEL_ARCHITECTURE.md`
|
|
- **Build Commands:** `CLAUDE.md`
|
|
- **Database Spec:** `docs/db/SPECIFICATION.md`
|
|
- **API Reference:** `docs/09_API_CLI_REFERENCE.md`
|
|
- **Module Architecture:** `docs/modules/<module>/architecture.md`
|
|
|
|
### Support
|
|
|
|
- **Issues:** https://git.stella-ops.org/stella-ops.org/git.stella-ops.org/issues
|
|
- **Discussions:** Internal team channels
|
|
- **Documentation:** `docs/` directory in the repository
|
|
|
|
---
|
|
|
|
## Quick Reference Card
|
|
|
|
### Essential Commands
|
|
|
|
```bash
|
|
# Start full platform
|
|
cd deploy\compose
|
|
docker compose -f docker-compose.dev.yaml up -d
|
|
|
|
# Stop a specific service for debugging
|
|
docker compose -f docker-compose.dev.yaml stop <service-name>
|
|
|
|
# View logs
|
|
docker compose -f docker-compose.dev.yaml logs -f <service-name>
|
|
|
|
# Restart a service
|
|
docker compose -f docker-compose.dev.yaml restart <service-name>
|
|
|
|
# Stop all services
|
|
docker compose -f docker-compose.dev.yaml down
|
|
|
|
# Stop all services and remove volumes (DESTRUCTIVE)
|
|
docker compose -f docker-compose.dev.yaml down -v
|
|
|
|
# Build the solution
|
|
cd C:\dev\New folder\git.stella-ops.org
|
|
dotnet build src\StellaOps.sln
|
|
|
|
# Run tests
|
|
dotnet test src\StellaOps.sln
|
|
|
|
# Run a specific project
|
|
cd src\Scanner\StellaOps.Scanner.WebService
|
|
dotnet run
|
|
```
|
|
|
|
### Service Default Ports
|
|
|
|
| Service | Port | URL | Notes |
|
|
|---------|------|-----|-------|
|
|
| **Infrastructure** |
|
|
| PostgreSQL | 5432 | `localhost:5432` | Primary database (REQUIRED) |
|
|
| Valkey | 6379 | `localhost:6379` | Cache/events/queues (REQUIRED) |
|
|
| RustFS | 8080 | http://localhost:8080 | S3-compatible storage (REQUIRED) |
|
|
| NATS | 4222 | `nats://localhost:4222` | Optional alternative transport |
|
|
| **Services** |
|
|
| Authority | 8440 | https://localhost:8440 | OAuth2/OIDC auth |
|
|
| Signer | 8441 | https://localhost:8441 | Cryptographic signing |
|
|
| Attestor | 8442 | https://localhost:8442 | in-toto attestations |
|
|
| Scanner.Web | 8444 | http://localhost:8444 | Scan API |
|
|
| Concelier | 8445 | http://localhost:8445 | Advisory ingestion |
|
|
| Notify | 8446 | http://localhost:8446 | Notifications |
|
|
| IssuerDirectory | 8447 | http://localhost:8447 | CSAF publisher discovery |
|
|
|
|
### Visual Studio Shortcuts
|
|
|
|
| Action | Shortcut |
|
|
|--------|----------|
|
|
| Start Debugging | F5 |
|
|
| Start Without Debugging | Ctrl+F5 |
|
|
| Stop Debugging | Shift+F5 |
|
|
| Step Over | F10 |
|
|
| Step Into | F11 |
|
|
| Step Out | Shift+F11 |
|
|
| Toggle Breakpoint | F9 |
|
|
| Build Solution | Ctrl+Shift+B |
|
|
| Rebuild Solution | Ctrl+Shift+F5 |
|
|
|
|
---
|
|
|
|
**Document Version:** 1.0
|
|
**Last Updated:** 2025-12-22
|
|
**Maintained By:** StellaOps Development Team
|