stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 2 Releases Wiki Activity
Files
3ba7157b00e8b08defbcdb810d023038ec0f8bc7
git.stella-ops.org/docs/DEVELOPER_ONBOARDING.md

1464 lines
62 KiB
Markdown
Raw Blame History

# StellaOps Developer Onboarding Guide
> **Target Audience:** DevOps operators with developer knowledge who need to understand, deploy, and debug the StellaOps platform.
## Table of Contents
1. [Architecture Overview](#architecture-overview)
2. [Prerequisites](#prerequisites)
3. [Quick Start - Full Platform in Docker](#quick-start)
4. [Hybrid Debugging Workflow](#hybrid-debugging-workflow)
5. [Service-by-Service Debugging Guide](#service-by-service-debugging-guide)
6. [Configuration Deep Dive](#configuration-deep-dive)
7. [Common Development Workflows](#common-development-workflows)
8. [Troubleshooting](#troubleshooting)
---
## Architecture Overview
StellaOps is a deterministic SBOM + VEX platform built as a microservices architecture with 36+ services organized into functional domains.
### Runtime Topology - High-Level
```
┌─────────────────────────────────────────────────────────────────────┐
│ INFRASTRUCTURE LAYER │
│ ┌──────────────────┐ ┌──────────────────┐ ┌─────────────────┐ │
│ │ PostgreSQL │ │ Valkey │ │ RustFS │ │
│ │ (v16+ ONLY) │ │ (Redis-compat) │ │ (S3-like API) │ │
│ │ │ │ - Caching │ │ - Artifacts │ │
│ │ All services use │ │ - DPoP nonces │ │ - SBOMs │ │
│ │ PostgreSQL for │ │ - Event queues │ │ - Signatures │ │
│ │ persistent data │ │ - Rate limiting│ │ │ │
│ └──────────────────┘ └──────────────────┘ └─────────────────┘ │
│ │
│ ┌──────────────────────────────────────────────────────────────┐ │
│ │ Optional: NATS JetStream (alternative transport for queues) │ │
│ │ Only used if explicitly configured in appsettings │ │
│ └──────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────┘
▲
│
┌─────────────────────────────────────────────────────────────────────┐
│ AUTHENTICATION & SIGNING │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ Authority │─▶│ Signer │─▶│ Attestor │ │
│ │ (OAuth2/OIDC)│ │(DSSE/PKIX) │ │(in-toto/DSSE)│ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
└─────────────────────────────────────────────────────────────────────┘
▲
│
┌─────────────────────────────────────────────────────────────────────┐
│ INGESTION & AGGREGATION │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ Concelier │ │ Excititor │ │IssuerDirectry│ │
│ │(Advisories) │ │ (VEX) │ │(CSAF Pubshrs)│ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
└─────────────────────────────────────────────────────────────────────┘
▲
│
┌─────────────────────────────────────────────────────────────────────┐
│ SCANNING & ANALYSIS │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │Scanner.Web │ │Scanner.Worker│ │ AdvisoryAI │ │
│ │(API/Control) │ │(Analyzers) │ │(ML Analysis) │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
│ ┌──────────────┐ ┌──────────────┐ │
│ │ RiskEngine │ │ Policy │ │
│ │ (Scoring) │ │ (Engine) │ │
│ └──────────────┘ └──────────────┘ │
└─────────────────────────────────────────────────────────────────────┘
▲
│
┌─────────────────────────────────────────────────────────────────────┐
│ ORCHESTRATION & WORKFLOW │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ Scheduler │ │ Orchestrator │ │ TaskRunner │ │
│ │(Job Sched) │ │(Coordinator) │ │(Executor) │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
└─────────────────────────────────────────────────────────────────────┘
▲
│
┌─────────────────────────────────────────────────────────────────────┘
│ EVENTS & NOTIFICATIONS │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ Notify │ │ Notifier │ │TimelineIndex │ │
│ │(Slack/Teams) │ │ (Advanced) │ │ (Events) │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
└─────────────────────────────────────────────────────────────────────┘
▲
│
┌─────────────────────────────────────────────────────────────────────┐
│ DATA & EXPORT │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ExportCenter │ │EvidenceLocker│ │FindingsLedger│ │
│ │(SARIF/SBOM) │ │(Artifacts) │ │(Audit Trail) │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
└─────────────────────────────────────────────────────────────────────┘
▲
│
┌─────────────────────────────────────────────────────────────────────┐
│ USER EXPERIENCE │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ Gateway │ │ Web (UI) │ │ CLI │ │
│ │ (API Router) │ │ (Angular v17)│ │(Multi-plat) │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
└─────────────────────────────────────────────────────────────────────┘
```
### Detailed Request Flow - Scan Execution Example
This diagram shows a complete scan request lifecycle with detailed routing through services:
```
┌──────────────────────────────────────────────────────────────────────────────────┐
│ 1. CLIENT REQUEST (CLI or Web UI) │
│ $ stella scan docker://alpine:latest --sbom-format=spdx │
└───────────────────────────────────┬──────────────────────────────────────────────┘
│ HTTPS
▼
┌──────────────────────────────────────────────────────────────────────────────────┐
│ 2. GATEWAY (API Router) │
│ - Terminates TLS │
│ - Routes to appropriate backend service │
│ - Load balancing (if multiple instances) │
└───────────────────────────────────┬──────────────────────────────────────────────┘
│ HTTP (internal)
▼
┌──────────────────────────────────────────────────────────────────────────────────┐
│ 3. AUTHORITY (Authentication) │
│ - Validates OAuth2 access token (DPoP-bound) │
│ - Checks DPoP proof against Valkey nonce cache │
│ - Returns user identity and scopes │
│ │
│ ┌─────────────┐ │
│ │ Valkey │◀── DPoP nonce validation (GET/SET) │
│ │ (Cache) │ │
│ └─────────────┘ │
│ ┌─────────────┐ │
│ │ PostgreSQL │◀── User/client lookup (SELECT) │
│ └─────────────┘ │
└───────────────────────────────────┬──────────────────────────────────────────────┘
│ Authenticated request
▼
┌──────────────────────────────────────────────────────────────────────────────────┐
│ 4. SCANNER.WEB (Scan API Controller) │
│ - Validates scan request parameters │
│ - Creates scan job record in PostgreSQL │
│ - Enqueues scan job to Valkey queue (default) or NATS (if configured) │
│ │
│ ┌─────────────┐ │
│ │ PostgreSQL │◀── INSERT scan_jobs (job_id, image_ref, status='pending') │
│ └─────────────┘ │
│ ┌─────────────┐ │
│ │ Valkey │◀── XADD scanner:jobs (enqueue job message) │
│ │ (Queue) │ │
│ └─────────────┘ │
│ │
│ Returns: HTTP 202 Accepted { "job_id": "scan-abc123", "status": "queued" } │
└───────────────────────────────────┬──────────────────────────────────────────────┘
│
│ (Client polls for status)
│
┌──────────────────────────────────────────────────────────────────────────────────┐
│ 5. SCANNER.WORKER (Background Processor) │
│ - Consumes job from Valkey queue (XREADGROUP scanner:jobs) │
│ - Updates job status to 'running' │
│ - Downloads container image from registry │
│ - Executes analyzers (OS packages, language deps, files) │
│ - Generates SBOM (SPDX/CycloneDX) │
│ - Stores artifacts to RustFS │
│ │
│ ┌─────────────┐ │
│ │ PostgreSQL │◀── UPDATE scan_jobs SET status='running' │
│ │ │◀── INSERT sbom_documents, packages, vulnerabilities │
│ │ │◀── UPDATE scan_jobs SET status='completed' │
│ └─────────────┘ │
│ ┌─────────────┐ │
│ │ RustFS │◀── PUT /artifacts/scan-abc123/sbom.spdx.json │
│ │ (S3 API) │◀── PUT /artifacts/scan-abc123/image-layers.tar.gz │
│ └─────────────┘ │
│ ┌─────────────┐ │
│ │ Valkey │◀── XADD scanner:events (publish scan.completed event) │
│ │(Event Stream│ │
│ └─────────────┘ │
└───────────────────────────────────┬──────────────────────────────────────────────┘
│ Event published
▼
┌──────────────────────────────────────────────────────────────────────────────────┐
│ 6. EVENT PROPAGATION (Valkey Streams) │
│ │
│ ┌───────────────────────────────────────────────────────────────────┐ │
│ │ Valkey Event Stream: "scanner:events" │ │
│ │ Event: { "type": "scan.completed", "job_id": "scan-abc123", ... } │ │
│ └───────────────────────────────────────────────────────────────────┘ │
│ │ │
│ ┌──────────────┼──────────────┬───────────────┐ │
│ ▼ ▼ ▼ ▼ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ Notify │ │Timeline │ │ Policy │ │ Export │ │
│ │ Worker │ │ Indexer │ │ Engine │ │ Center │ │
│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │
│ │ │ │ │ │
│ │ (all subscribe to scanner:events via XREADGROUP) │
└─────────┼───────────────┼──────────────┼───────────────┼──────────────────────┘
│ │ │ │
▼ ▼ ▼ ▼
┌──────────────┐ ┌──────────────┐ ┌──────────┐ ┌──────────────┐
│ 7a. NOTIFY │ │ 7b. TIMELINE │ │ 7c.POLICY│ │ 7d. EXPORT │
│ │ │ INDEXER │ │ ENGINE │ │ CENTER │
│ - Query scan │ │ │ │ │ │ │
│ results │ │ - Index event│ │ - Eval │ │ - Generate │
│ - Check user │ │ timeline │ │ policy │ │ SARIF │
│ notif prefs│ │ - Store in │ │ rules │ │ - Export to │
│ - Send Slack │ │ PostgreSQL │ │ - Block/ │ │ external │
│ message │ │ │ │ Allow │ │ systems │
│ │ │ │ │ │ │ │
│ PostgreSQL ◀─┤ │ PostgreSQL ◀─┤ │PostgreSQL│ │ RustFS ◀─┤
│ (user prefs) │ │ (timeline) │ │(policies)│ │ (exports) │
└──────────────┘ └──────────────┘ └──────────┘ └──────────────┘
```
### Export Flow - SBOM Distribution
```
┌──────────────────────────────────────────────────────────────────────────────────┐
│ EXPORT REQUEST: GET /api/v1/scans/{scan_id}/export?format=spdx │
└───────────────────────────────────┬──────────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────────────────────────┐
│ SCANNER.WEB or EXPORT CENTER │
│ │
│ 1. Query scan metadata from PostgreSQL │
│ ┌─────────────┐ │
│ │ PostgreSQL │◀── SELECT * FROM scan_jobs WHERE job_id = $1 │
│ │ │◀── SELECT * FROM sbom_documents WHERE scan_id = $1 │
│ └─────────────┘ │
│ │
│ 2. Retrieve SBOM artifact from RustFS │
│ ┌─────────────┐ │
│ │ RustFS │◀── GET /artifacts/scan-abc123/sbom.spdx.json │
│ └─────────────┘ │
│ │
│ 3. Sign SBOM with Signer service │
│ ┌─────────────┐ │
│ │ Signer │◀── POST /api/v1/sign (SBOM payload) │
│ │ │──▶ Returns: DSSE envelope with signature │
│ └─────────────┘ │
│ │
│ 4. Create in-toto attestation with Attestor │
│ ┌─────────────┐ │
│ │ Attestor │◀── POST /api/v1/attest (signed SBOM) │
│ │ │──▶ Returns: in-toto attestation bundle │
│ └─────────────┘ │
│ │
│ 5. Store final bundle to RustFS │
│ ┌─────────────┐ │
│ │ RustFS │◀── PUT /artifacts/scan-abc123/bundle.jsonl │
│ └─────────────┘ │
│ │
│ 6. Return signed bundle to client │
│ Returns: HTTP 200 OK (application/vnd.in-toto+json) │
└──────────────────────────────────────────────────────────────────────────────────┘
```
### Notification Flow - Vulnerability Alert
```
┌──────────────────────────────────────────────────────────────────────────────────┐
│ TRIGGER: New critical CVE detected in existing scan │
│ Source: Concelier advisory ingestion │
└───────────────────────────────────┬──────────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────────────────────────┐
│ CONCELIER.WORKER (Advisory Processor) │
│ │
│ 1. Ingest new advisory from NVD/OSV/CSAF │
│ ┌─────────────┐ │
│ │ PostgreSQL │◀── INSERT INTO advisories (cve_id, severity, ...) │
│ └─────────────┘ │
│ │
│ 2. Match advisory against existing scans (PURL/CPE matching) │
│ ┌─────────────┐ │
│ │ PostgreSQL │◀── SELECT scans WHERE package_purl IN (affected_purls) │
│ └─────────────┘ │
│ │
│ 3. Publish drift event to Valkey │
│ ┌─────────────┐ │
│ │ Valkey │◀── XADD concelier:drift (new vulnerability found) │
│ └─────────────┘ │
└───────────────────────────────────┬──────────────────────────────────────────────┘
│ Event published
▼
┌──────────────────────────────────────────────────────────────────────────────────┐
│ NOTIFY.WORKER (Notification Processor) │
│ │
│ 1. Consume drift event from Valkey stream │
│ ┌─────────────┐ │
│ │ Valkey │◀── XREADGROUP concelier:drift notify-workers │
│ └─────────────┘ │
│ │
│ 2. Query user notification preferences │
│ ┌─────────────┐ │
│ │ PostgreSQL │◀── SELECT * FROM user_notification_preferences │
│ │ │ WHERE user_id = scan_owner AND channel = 'slack' │
│ └─────────────┘ │
│ │
│ 3. Render notification template │
│ Template: "🚨 New critical CVE-2024-1234 affects alpine:latest scan" │
│ │
│ 4. Deliver notification via configured channels │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ External APIs │ │
│ │ - POST https://hooks.slack.com/services/T00/B00/xxx │ │
│ │ - POST https://graph.microsoft.com/v1.0/teams/channels │ │
│ │ - SMTP send (email) │ │
│ └─────────────────────────────────────────────────────────┘ │
│ │
│ 5. Store delivery receipt in PostgreSQL │
│ ┌─────────────┐ │
│ │ PostgreSQL │◀── INSERT INTO notification_deliveries (status, ...) │
│ └─────────────┘ │
└──────────────────────────────────────────────────────────────────────────────────┘
```
### Key Architectural Principles
1. **Deterministic Evidence** - Same inputs always produce same outputs
2. **VEX-First Decisioning** - Policy decisions based on OpenVEX statements
3. **Offline-First** - Full air-gap operation supported
4. **Plugin Architecture** - Extensible connectors for advisories, analyzers, auth
5. **Sovereign Crypto** - FIPS, eIDAS, GOST, SM support
6. **Schema Isolation** - Per-module PostgreSQL schemas
### Service Categories
| Category | Services | Purpose |
|----------|----------|---------|
| **Infrastructure** | PostgreSQL v16+, Valkey 8.0, RustFS, NATS (optional) | Database, cache/messaging, object storage, optional queue transport |
| **Auth & Signing** | Authority, Signer, Attestor | OAuth2/OIDC with DPoP, cryptographic signing, in-toto attestations |
| **Ingestion** | Concelier, Excititor, IssuerDirectory | Advisory/VEX ingestion, normalization, merging, CSAF publisher discovery |
| **Scanning** | Scanner.Web, Scanner.Worker, AdvisoryAI | Container scanning, SBOM generation (SPDX/CDX), ML vulnerability analysis |
| **Policy & Risk** | Policy Engine, RiskEngine | OPA/Rego policy evaluation, risk scoring, exploitability assessment |
| **Orchestration** | Scheduler, Orchestrator, TaskRunner | Job scheduling, workflow coordination, distributed task execution |
| **Notifications** | Notify, Notifier, TimelineIndexer | Event delivery (Slack/Teams/Email), notification management, timeline tracking |
| **Data & Export** | ExportCenter, EvidenceLocker, FindingsLedger | SARIF/SBOM export, evidence storage, immutable audit trail |
| **User Experience** | Gateway, Web UI, CLI | API routing, Angular v17 UI, multi-platform command-line tools |
---
## Prerequisites
### Required Software
1. **Docker Desktop** (Windows/Mac) or **Docker Engine + Docker Compose** (Linux)
- Version: 20.10+ recommended
- Enable WSL2 backend (Windows)
2. **.NET 10 SDK**
- Download: https://dotnet.microsoft.com/download/dotnet/10.0
- Verify: `dotnet --version` (should show 10.0.x)
3. **Visual Studio 2022** (v17.12+) or **Visual Studio Code**
- Workload: ASP.NET and web development
- Workload: .NET desktop development
- Extension (VS Code): C# Dev Kit
4. **Git**
- Version: 2.30+ recommended
### Optional Tools
- **PostgreSQL Client** (psql, pgAdmin, DBeaver) - for database inspection
- **Redis Insight** or **Another Redis Desktop Manager** - for Valkey inspection (Valkey is Redis-compatible)
- **Postman/Insomnia** - for API testing
- **AWS CLI or s3cmd** - for RustFS (S3-compatible) inspection
### System Requirements
- **RAM:** 16 GB minimum, 32 GB recommended
- **Disk:** 50 GB free space (for Docker images, volumes, build artifacts)
- **CPU:** 4 cores minimum, 8 cores recommended
---
## Quick Start
### Step 1: Clone the Repository
```bash
cd C:\dev\
git clone https://git.stella-ops.org/stella-ops.org/git.stella-ops.org
cd git.stella-ops.org
```
### Step 2: Prepare Environment Configuration
```bash
# Copy the development environment template
cd deploy\compose
copy env\dev.env.example .env
# Edit .env with your preferred text editor
notepad .env
```
**Key settings to configure:**
```bash
# PostgreSQL Database
POSTGRES_USER=stellaops
POSTGRES_PASSWORD=your_secure_password_here
POSTGRES_DB=stellaops_platform
POSTGRES_PORT=5432
# Valkey (Redis-compatible cache and messaging)
VALKEY_PORT=6379
# RustFS Object Storage
RUSTFS_HTTP_PORT=8080
# Service ports (adjust if conflicts exist)
AUTHORITY_PORT=8440
SIGNER_PORT=8441
ATTESTOR_PORT=8442
CONCELIER_PORT=8445
SCANNER_WEB_PORT=8444
NOTIFY_WEB_PORT=8446
# Scanner configuration (Valkey default, can switch to NATS if needed)
SCANNER_EVENTS_DRIVER=valkey
SCANNER_EVENTS_DSN=valkey:6379
# Scheduler configuration (Valkey default, can switch to NATS if needed)
SCHEDULER_QUEUE_KIND=Valkey
SCHEDULER_QUEUE_VALKEY_URL=valkey:6379
# Authority configuration
AUTHORITY_ISSUER=https://authority:8440
SIGNER_POE_INTROSPECT_URL=https://www.stella-ops.org/license/introspect
```
### Step 3: Start the Full Platform
```bash
# From deploy/compose directory
docker compose -f docker-compose.dev.yaml up -d
```
**This will start all infrastructure and services:**
- PostgreSQL v16+ (port 5432) - Primary database for all services
- Valkey 8.0 (port 6379) - Cache, DPoP nonces, event streams, rate limiting
- RustFS (port 8080) - S3-compatible object storage for artifacts/SBOMs
- NATS JetStream (port 4222) - Optional transport (only if configured)
- Authority (port 8440) - OAuth2/OIDC authentication
- Signer (port 8441) - Cryptographic signing
- Attestor (port 8442) - in-toto attestation generation
- Scanner.Web (port 8444) - Scan API
- Concelier (port 8445) - Advisory ingestion
- And 30+ more services...
### Step 4: Verify Services Are Running
```bash
# Check all services are up
docker compose -f docker-compose.dev.yaml ps
# Check logs for a specific service
docker compose -f docker-compose.dev.yaml logs -f scanner-web
# Check infrastructure health
docker compose -f docker-compose.dev.yaml logs postgres
docker compose -f docker-compose.dev.yaml logs valkey
docker compose -f docker-compose.dev.yaml logs rustfs
```
### Step 5: Access the Platform
Open your browser and navigate to:
- **RustFS:** http://localhost:8080 (S3-compatible object storage)
- **Scanner API:** http://localhost:8444/swagger (if Swagger enabled)
- **Concelier API:** http://localhost:8445/swagger
- **Authority:** http://localhost:8440/.well-known/openid-configuration (OIDC discovery)
---
## Hybrid Debugging Workflow
The hybrid workflow allows you to:
1. Run infrastructure (databases, queues) in Docker
2. Run most services in Docker
3. **Selectively debug** one or two services in Visual Studio
### Workflow Overview
```
┌────────────────────────────────────────────────────────────┐
│ DOCKER ENVIRONMENT │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │PostgreSQL│ │ Valkey │ │ RustFS │ │
│ │ (DB) │ │(Cache/Msg│ │(Storage) │ │
│ └──────────┘ └──────────┘ └──────────┘ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ Authority│ │ Signer │ │ Attestor │ │
│ └──────────┘ └──────────┘ └──────────┘ │
│ ┌──────────┐ ┌──────────┐ │
│ │Concelier │ │ Excititor│ ← Running normally │
│ └──────────┘ └──────────┘ │
└────────────────────────────────────────────────────────────┘
▲
│ HTTP calls + Valkey streams
▼
┌────────────────────────────────────────────────────────────┐
│ VISUAL STUDIO (F5) │
│ ┌────────────────────────────────────────────┐ │
│ │ Scanner.WebService │ │
│ │ Running on http://localhost:5210 │ │
│ │ (Breakpoints, hot reload, debugging) │ ← YOU DEBUG HERE
│ └────────────────────────────────────────────┘ │
└────────────────────────────────────────────────────────────┘
```
### Step-by-Step: Debug Scanner.WebService
#### 1. Stop the Docker Container for Scanner
```bash
cd deploy\compose
docker compose -f docker-compose.dev.yaml stop scanner-web
```
**Verify it's stopped:**
```bash
docker compose -f docker-compose.dev.yaml ps scanner-web
# Should show: State = "exited"
```
#### 2. Configure Local Development Settings
Create or modify the service's `appsettings.Development.json`:
```bash
cd C:\dev\New folder\git.stella-ops.org\src\Scanner\StellaOps.Scanner.WebService
```
**Create `appsettings.Development.json`:**
```json
{
"Logging": {
"LogLevel": {
"Default": "Information",
"Microsoft.AspNetCore": "Warning",
"StellaOps": "Debug"
}
},
"ConnectionStrings": {
"DefaultConnection": "Host=localhost;Port=5432;Database=stellaops_platform;Username=stellaops;Password=your_password_here;Include Error Detail=true"
},
"Scanner": {
"Storage": {
"Mongo": {
"ConnectionString": "mongodb://stellaops:your_password_here@localhost:27017"
}
},
"ArtifactStore": {
"Driver": "rustfs",
"Endpoint": "http://localhost:8080/api/v1",
"Bucket": "scanner-artifacts",
"TimeoutSeconds": 30
},
"Queue": {
"Broker": "nats://localhost:4222"
},
"Events": {
"Enabled": false
}
},
"Authority": {
"Issuer": "https://localhost:8440",
"BaseUrl": "https://localhost:8440",
"BypassNetworks": ["127.0.0.1", "::1"]
}
}
```
**Note:** Adjust connection strings to match your Docker infrastructure ports. If PostgreSQL is on Docker's bridge network, you may need to expose it on `localhost:5432`.
#### 3. Expose Docker Services to localhost
For services running in Docker to be accessible from your host machine, ensure ports are mapped in `docker-compose.dev.yaml`:
```yaml
# Already configured in docker-compose.dev.yaml
postgres:
ports:
- "${POSTGRES_PORT:-5432}:5432"
mongo:
ports:
- "27017:27017"
nats:
ports:
- "${NATS_CLIENT_PORT:-4222}:4222"
rustfs:
ports:
- "${RUSTFS_HTTP_PORT:-8080}:8080"
```
**Verify connectivity:**
```bash
# Test PostgreSQL
psql -h localhost -U stellaops -d stellaops_platform
# Test NATS
telnet localhost 4222
# Test RustFS
curl http://localhost:8080/health
```
#### 4. Open Solution in Visual Studio
```bash
# Open the solution
cd C:\dev\New folder\git.stella-ops.org
start src\StellaOps.sln
```
**In Visual Studio:**
1. Right-click `StellaOps.Scanner.WebService` project
2. Select **"Set as Startup Project"**
3. Press **F5** to start debugging
**Expected output:**
```
info: Microsoft.Hosting.Lifetime[14]
Now listening on: http://localhost:5210
info: Microsoft.Hosting.Lifetime[0]
Application started. Press Ctrl+C to shut down.
```
#### 5. Update Other Services to Call localhost
Since you're running Scanner.WebService on `localhost:5210` instead of `scanner-web:8444`, you need to update any services that call it.
**Option A: Environment Variables (Docker containers)**
Update `.env` file:
```bash
# Use host.docker.internal to reach host machine from Docker
SCANNER_WEB_BASEURL=http://host.docker.internal:5210
```
Restart dependent services:
```bash
docker compose -f docker-compose.dev.yaml restart scheduler-web
```
**Option B: Modify docker-compose.dev.yaml**
```yaml
scheduler-web:
environment:
SCHEDULER__WORKER__RUNNER__SCANNER__BASEADDRESS: "http://host.docker.internal:5210"
```
Then restart:
```bash
docker compose -f docker-compose.dev.yaml up -d scheduler-web
```
#### 6. Set Breakpoints and Debug
1. Navigate to `Program.cs` in Scanner.WebService
2. Set a breakpoint on a line in a controller or service method
3. Trigger the endpoint using:
- Swagger UI (if enabled): http://localhost:5210/swagger
- Postman/curl
- CLI command
**Example curl:**
```bash
curl -X POST http://localhost:5210/api/scans \
-H "Content-Type: application/json" \
-d '{"imageRef": "alpine:latest"}'
```
Your breakpoint should hit, and you can step through code.
#### 7. Return to Docker Mode
When you're done debugging:
```bash
# Stop Visual Studio debugger (Shift+F5)
# Restart the Docker container
cd deploy\compose
docker compose -f docker-compose.dev.yaml start scanner-web
# Verify it's running
docker compose -f docker-compose.dev.yaml ps scanner-web
```
---
## Service-by-Service Debugging Guide
### Authority (OAuth2/OIDC Provider)
**Project:** `src/Authority/StellaOps.Authority/StellaOps.Authority.csproj`
**Stop Docker container:**
```bash
docker compose -f docker-compose.dev.yaml stop authority
```
**Configuration:** `appsettings.Development.json`
```json
{
"ConnectionStrings": {
"DefaultConnection": "Host=localhost;Port=5432;Database=stellaops_platform;Username=stellaops;Password=your_password"
},
"StellaOps_Authority": {
"Issuer": "https://localhost:5001",
"Mongo": {
"ConnectionString": "mongodb://stellaops:your_password@localhost:27017"
}
},
"Kestrel": {
"Endpoints": {
"Https": {
"Url": "https://localhost:5001"
}
}
}
}
```
**Run in Visual Studio:** F5 on `StellaOps.Authority` project
**Default URL:** https://localhost:5001
**Update dependent services:**
```bash
# In .env
AUTHORITY_ISSUER=https://host.docker.internal:5001
```
### Concelier (Advisory Ingestion)
**Project:** `src/Concelier/StellaOps.Concelier.WebService/StellaOps.Concelier.WebService.csproj`
**Stop Docker:**
```bash
docker compose -f docker-compose.dev.yaml stop concelier
```
**Configuration:** `appsettings.Development.json`
```json
{
"ConnectionStrings": {
"DefaultConnection": "Host=localhost;Port=5432;Database=stellaops_platform;Username=stellaops;Password=your_password"
},
"Concelier": {
"Storage": {
"Mongo": {
"ConnectionString": "mongodb://stellaops:your_password@localhost:27017"
},
"S3": {
"Endpoint": "http://localhost:9000",
"AccessKeyId": "stellaops",
"SecretAccessKey": "your_password"
}
},
"Authority": {
"BaseUrl": "https://localhost:8440"
}
}
}
```
**Run:** F5 on `StellaOps.Concelier.WebService`
**Default URL:** http://localhost:5000
### Scanner.Worker (Background Analyzer)
**Project:** `src/Scanner/StellaOps.Scanner.Worker/StellaOps.Scanner.Worker.csproj`
**Stop Docker:**
```bash
docker compose -f docker-compose.dev.yaml stop scanner-worker
```
**Configuration:** Same as Scanner.WebService (shares settings)
**Run:** F5 on `StellaOps.Scanner.Worker`
**Note:** Worker has no HTTP endpoint - it consumes from NATS queue
### Scheduler.WebService
**Project:** `src/Scheduler/StellaOps.Scheduler.WebService/StellaOps.Scheduler.WebService.csproj`
**Stop Docker:**
```bash
docker compose -f docker-compose.dev.yaml stop scheduler-web
```
**Configuration:** `appsettings.Development.json`
```json
{
"ConnectionStrings": {
"DefaultConnection": "Host=localhost;Port=5432;Database=stellaops_orchestrator;Username=stellaops;Password=your_password"
},
"Scheduler": {
"Queue": {
"Kind": "Nats",
"Nats": {
"Url": "nats://localhost:4222"
}
},
"Worker": {
"Runner": {
"Scanner": {
"BaseAddress": "http://localhost:5210"
}
}
}
}
}
```
**Run:** F5 on `StellaOps.Scheduler.WebService`
### Notify.WebService
**Project:** `src/Notify/StellaOps.Notify.WebService/StellaOps.Notify.WebService.csproj`
**Stop Docker:**
```bash
docker compose -f docker-compose.dev.yaml stop notify-web
```
**Configuration:** Uses `etc/notify.dev.yaml`
**Run:** F5 on `StellaOps.Notify.WebService`
---
## Configuration Deep Dive
### Configuration Hierarchy
All services follow this configuration priority (highest to lowest):
1. **Environment Variables** - `STELLAOPS_<MODULE>_<SETTING>` or `<MODULE>__<SETTING>`
2. **appsettings.{Environment}.json** - `appsettings.Development.json`, `appsettings.Production.json`
3. **appsettings.json** - Base configuration
4. **YAML files** - `../etc/<service>.yaml`, `../etc/<service>.local.yaml`
### Common Configuration Patterns
#### PostgreSQL Connection Strings
```json
{
"ConnectionStrings": {
"DefaultConnection": "Host=localhost;Port=5432;Database=<db_name>;Username=stellaops;Password=<password>;Pooling=true;Minimum Pool Size=1;Maximum Pool Size=100;Command Timeout=60"
}
}
```
**Database names by service:**
- Scanner: `stellaops_platform` or `scanner_*`
- Orchestrator: `stellaops_orchestrator`
- Authority: `stellaops_platform` (shared, schema-isolated)
- Concelier: `stellaops_platform` (vuln schema)
- Notify: `stellaops_platform` (notify schema)
#### Valkey Configuration (Default Transport)
```json
{
"Scanner": {
"Events": {
"Driver": "valkey",
"Dsn": "localhost:6379"
},
"Cache": {
"Redis": {
"ConnectionString": "localhost:6379"
}
}
},
"Scheduler": {
"Queue": {
"Kind": "Valkey",
"Valkey": {
"Url": "localhost:6379"
}
}
}
}
```
#### NATS Queue Configuration (Optional Alternative Transport)
```json
{
"Scanner": {
"Events": {
"Driver": "nats",
"Dsn": "nats://localhost:4222"
}
},
"Scheduler": {
"Queue": {
"Kind": "Nats",
"Nats": {
"Url": "nats://localhost:4222"
}
}
}
}
```
#### RustFS Configuration (S3-Compatible Object Storage)
```json
{
"Scanner": {
"Storage": {
"RustFS": {
"Endpoint": "http://localhost:8080",
"AccessKeyId": "stellaops",
"SecretAccessKey": "your_password",
"BucketName": "scanner-artifacts",
"Region": "us-east-1",
"ForcePathStyle": true
}
}
}
}
```
#### RustFS Configuration
```json
{
"Scanner": {
"ArtifactStore": {
"Driver": "rustfs",
"Endpoint": "http://localhost:8080/api/v1",
"Bucket": "scanner-artifacts",
"TimeoutSeconds": 30
}
}
}
```
### Environment Variable Mapping
ASP.NET Core uses `__` (double underscore) for nested configuration:
```bash
# This JSON configuration:
{
"Scanner": {
"Queue": {
"Broker": "nats://localhost:4222"
}
}
}
# Can be set via environment variable:
SCANNER__QUEUE__BROKER=nats://localhost:4222
# Or with STELLAOPS_ prefix:
STELLAOPS_SCANNER__QUEUE__BROKER=nats://localhost:4222
```
---
## Common Development Workflows
### Workflow 1: Debug a Single Service with Full Stack
**Scenario:** You need to debug Scanner.WebService while all other services run normally.
```bash
# 1. Start full platform
cd deploy\compose
docker compose -f docker-compose.dev.yaml up -d
# 2. Stop the service you want to debug
docker compose -f docker-compose.dev.yaml stop scanner-web
# 3. Open Visual Studio
cd C:\dev\New folder\git.stella-ops.org
start src\StellaOps.sln
# 4. Set Scanner.WebService as startup project and F5
# 5. Test the service
curl -X POST http://localhost:5210/api/scans -H "Content-Type: application/json" -d '{"imageRef":"alpine:latest"}'
# 6. When done, stop VS debugger and restart Docker container
docker compose -f docker-compose.dev.yaml start scanner-web
```
### Workflow 2: Debug Multiple Services Together
**Scenario:** Debug Scanner.WebService and Scanner.Worker together.
```bash
# 1. Stop both containers
docker compose -f docker-compose.dev.yaml stop scanner-web scanner-worker
# 2. In Visual Studio, configure multiple startup projects:
# - Right-click solution > Properties
# - Set "Multiple startup projects"
# - Select Scanner.WebService: Start
# - Select Scanner.Worker: Start
# 3. Press F5 to debug both simultaneously
```
### Workflow 3: Test Integration with Modified Code
**Scenario:** You modified Concelier and want to test how Scanner integrates with it.
```bash
# 1. Build Concelier locally
cd src\Concelier\StellaOps.Concelier.WebService
dotnet build
# 2. Stop Docker Concelier
cd ..\..\..\deploy\compose
docker compose -f docker-compose.dev.yaml stop concelier
# 3. Run Concelier in Visual Studio (F5)
# 4. Keep Scanner in Docker, but point it to localhost Concelier
# Update .env:
CONCELIER_BASEURL=http://host.docker.internal:5000
# 5. Restart Scanner to pick up new config
docker compose -f docker-compose.dev.yaml restart scanner-web
```
### Workflow 4: Reset Database State
**Scenario:** You need a clean database to test migrations or start fresh.
```bash
# 1. Stop all services
docker compose -f docker-compose.dev.yaml down
# 2. Remove database volumes
docker volume rm compose_postgres-data
docker volume rm compose_mongo-data
# 3. Restart platform (will recreate volumes and databases)
docker compose -f docker-compose.dev.yaml up -d
# 4. Wait for migrations to run
docker compose -f docker-compose.dev.yaml logs -f postgres
# Look for migration completion messages
```
### Workflow 5: Test Offline/Air-Gap Mode
**Scenario:** Test the platform in offline mode.
```bash
# 1. Use the air-gap compose profile
cd deploy\compose
docker compose -f docker-compose.airgap.yaml up -d
# 2. Verify no external network calls
docker compose -f docker-compose.airgap.yaml logs | grep -i "external\|outbound\|internet"
```
---
## Troubleshooting
### Common Issues
#### 1. Port Already in Use
**Error:**
```
Error starting userland proxy: listen tcp 0.0.0.0:5432: bind: address already in use
```
**Solutions:**
**Option A: Change the port in .env**
```bash
# Edit .env
POSTGRES_PORT=5433 # Use a different port
```
**Option B: Stop the conflicting process**
```bash
# Windows
netstat -ano | findstr :5432
taskkill /PID <PID> /F
# Linux/Mac
lsof -i :5432
kill -9 <PID>
```
#### 2. Cannot Connect to PostgreSQL from Visual Studio
**Error:**
```
Npgsql.NpgsqlException: Connection refused
```
**Solutions:**
1. **Verify PostgreSQL is accessible from host:**
```bash
psql -h localhost -U stellaops -d stellaops_platform
```
2. **Check Docker network:**
```bash
docker network inspect compose_stellaops
# Ensure your service has "host.docker.internal" DNS resolution
```
3. **Update connection string:**
```json
{
"ConnectionStrings": {
"DefaultConnection": "Host=localhost;Port=5432;Database=stellaops_platform;Username=stellaops;Password=your_password;Include Error Detail=true"
}
}
```
#### 3. NATS Connection Refused
**Error:**
```
NATS connection error: connection refused
```
**Solution:**
By default, services use **Valkey** for messaging, not NATS. Ensure Valkey is running:
```bash
docker compose -f docker-compose.dev.yaml ps valkey
# Should show: State = "Up"
# Test connectivity
telnet localhost 6379
```
Update configuration to use Valkey (default):
```json
{
"Scanner": {
"Events": {
"Driver": "valkey",
"Dsn": "localhost:6379"
}
},
"Scheduler": {
"Queue": {
"Kind": "Valkey",
"Valkey": {
"Url": "localhost:6379"
}
}
}
}
```
**If you explicitly want to use NATS** (optional):
```bash
docker compose -f docker-compose.dev.yaml ps nats
# Ensure NATS is running
# Update appsettings.Development.json:
{
"Scanner": {
"Events": {
"Driver": "nats",
"Dsn": "nats://localhost:4222"
}
}
}
```
#### 4. Valkey Connection Refused
**Error:**
```
StackExchange.Redis.RedisConnectionException: It was not possible to connect to the redis server(s)
```
**Solutions:**
1. **Check Valkey is running:**
```bash
docker compose -f docker-compose.dev.yaml ps valkey
# Should show: State = "Up"
# Check logs
docker compose -f docker-compose.dev.yaml logs valkey
```
2. **Reset Valkey:**
```bash
docker compose -f docker-compose.dev.yaml stop valkey
docker volume rm compose_valkey-data
docker compose -f docker-compose.dev.yaml up -d valkey
```
#### 5. Service Cannot Reach host.docker.internal
**Error:**
```
Could not resolve host: host.docker.internal
```
**Solution (Windows/Mac):**
Should work automatically with Docker Desktop.
**Solution (Linux):**
Add to docker-compose.dev.yaml:
```yaml
services:
scanner-web:
extra_hosts:
- "host.docker.internal:host-gateway"
```
Or use the host's IP address:
```bash
# Find host IP
ip addr show docker0
# Use that IP instead of host.docker.internal
```
#### 6. Certificate Validation Errors (Authority/HTTPS)
**Error:**
```
The SSL connection could not be established
```
**Solution:**
For development, disable certificate validation:
```json
{
"Authority": {
"ValidateCertificate": false
}
}
```
Or trust the development certificate:
```bash
dotnet dev-certs https --trust
```
#### 7. Build Errors - Missing SDK
**Error:**
```
error MSB4236: The SDK 'Microsoft.NET.Sdk.Web' specified could not be found
```
**Solution:**
Install .NET 10 SDK:
```bash
# Verify installation
dotnet --list-sdks
# Should show:
# 10.0.xxx [C:\Program Files\dotnet\sdk]
```
#### 8. Hot Reload Not Working
**Symptom:** Changes in code don't reflect when running in Visual Studio.
**Solutions:**
1. Ensure Hot Reload is enabled: Tools > Options > Debugging > .NET Hot Reload > Enable Hot Reload
2. Rebuild the project: Ctrl+Shift+B
3. Restart debugging session: Shift+F5, then F5
#### 9. Docker Compose Fails to Parse .env
**Error:**
```
invalid interpolation format
```
**Solution:**
Ensure no spaces around `=` in .env:
```bash
# Wrong
POSTGRES_USER = stellaops
# Correct
POSTGRES_USER=stellaops
```
#### 10. Volume Permission Issues (Linux)
**Error:**
```
Permission denied writing to /data/db
```
**Solution:**
```bash
# Fix permissions on volume directories
sudo chown -R $USER:$USER ./volumes
# Or run Docker as root (not recommended for production)
sudo docker compose -f docker-compose.dev.yaml up -d
```
---
## Next Steps
### Learning Path
1. **Week 1: Infrastructure**
- Understand PostgreSQL schema isolation (all services use PostgreSQL)
- Learn Valkey streams for event queuing and caching
- Study RustFS S3-compatible object storage
- Optional: NATS JetStream as alternative transport
2. **Week 2: Core Services**
- Deep dive into Scanner architecture (analyzers, workers, caching)
- Understand Concelier advisory ingestion and merging
- Study VEX workflow in Excititor
3. **Week 3: Authentication & Security**
- Master OAuth2/OIDC flow in Authority
- Understand signing flow (Signer → Attestor → Rekor)
- Study policy evaluation engine
4. **Week 4: Integration**
- Build end-to-end scan workflow
- Implement custom Concelier connector
- Create custom notification rules
### Key Documentation
- **Architecture:** `docs/07_HIGH_LEVEL_ARCHITECTURE.md`
- **Build Commands:** `CLAUDE.md`
- **Database Spec:** `docs/db/SPECIFICATION.md`
- **API Reference:** `docs/09_API_CLI_REFERENCE.md`
- **Module Architecture:** `docs/modules/<module>/architecture.md`
### Support
- **Issues:** https://git.stella-ops.org/stella-ops.org/git.stella-ops.org/issues
- **Discussions:** Internal team channels
- **Documentation:** `docs/` directory in the repository
---
## Quick Reference Card
### Essential Commands
```bash
# Start full platform
cd deploy\compose
docker compose -f docker-compose.dev.yaml up -d
# Stop a specific service for debugging
docker compose -f docker-compose.dev.yaml stop <service-name>
# View logs
docker compose -f docker-compose.dev.yaml logs -f <service-name>
# Restart a service
docker compose -f docker-compose.dev.yaml restart <service-name>
# Stop all services
docker compose -f docker-compose.dev.yaml down
# Stop all services and remove volumes (DESTRUCTIVE)
docker compose -f docker-compose.dev.yaml down -v
# Build the solution
cd C:\dev\New folder\git.stella-ops.org
dotnet build src\StellaOps.sln
# Run tests
dotnet test src\StellaOps.sln
# Run a specific project
cd src\Scanner\StellaOps.Scanner.WebService
dotnet run
```
### Service Default Ports
| Service | Port | URL | Notes |
|---------|------|-----|-------|
| **Infrastructure** |
| PostgreSQL | 5432 | `localhost:5432` | Primary database (REQUIRED) |
| Valkey | 6379 | `localhost:6379` | Cache/events/queues (REQUIRED) |
| RustFS | 8080 | http://localhost:8080 | S3-compatible storage (REQUIRED) |
| NATS | 4222 | `nats://localhost:4222` | Optional alternative transport |
| **Services** |
| Authority | 8440 | https://localhost:8440 | OAuth2/OIDC auth |
| Signer | 8441 | https://localhost:8441 | Cryptographic signing |
| Attestor | 8442 | https://localhost:8442 | in-toto attestations |
| Scanner.Web | 8444 | http://localhost:8444 | Scan API |
| Concelier | 8445 | http://localhost:8445 | Advisory ingestion |
| Notify | 8446 | http://localhost:8446 | Notifications |
| IssuerDirectory | 8447 | http://localhost:8447 | CSAF publisher discovery |
### Visual Studio Shortcuts
| Action | Shortcut |
|--------|----------|
| Start Debugging | F5 |
| Start Without Debugging | Ctrl+F5 |
| Stop Debugging | Shift+F5 |
| Step Over | F10 |
| Step Into | F11 |
| Step Out | Shift+F11 |
| Toggle Breakpoint | F9 |
| Build Solution | Ctrl+Shift+B |
| Rebuild Solution | Ctrl+Shift+F5 |
---
**Document Version:** 1.0
**Last Updated:** 2025-12-22
**Maintained By:** StellaOps Development Team
Reference in New Issue View Git Blame Copy Permalink