stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 2 Releases Wiki Activity

docs consolidation

Browse Source
This commit is contained in:
StellaOps Bot
2025-12-24 12:38:14 +02:00
parent 7503c19b8f
commit 9a08d10b89
215 changed files with 2188 additions and 9623 deletions
Download Patch File Download Diff File Expand all files Collapse all files

834
docs/DEVELOPER_ONBOARDING.md
View File

@@ -17,394 +17,44 @@
## Architecture Overview
StellaOps is a deterministic SBOM + VEX platform built as a microservices architecture with 36+ services organized into functional domains.
StellaOps is a deterministic, offline-first SBOM + VEX platform built as a microservice architecture. The system is designed so every verdict can be replayed from concrete evidence (SBOM slices, advisory/VEX observations, policy decision traces, and optional attestations).
**📖 For detailed component architecture with communication patterns, see [ARCHITECTURE_DETAILED.md](./ARCHITECTURE_DETAILED.md)**
### Canonical references
- Architecture overview (10-minute tour): `docs/40_ARCHITECTURE_OVERVIEW.md`
- High-level reference map: `docs/07_HIGH_LEVEL_ARCHITECTURE.md`
- Detailed architecture index: `docs/technical/architecture/README.md`
- Topology: `docs/technical/architecture/platform-topology.md`
- Infrastructure: `docs/technical/architecture/infrastructure-dependencies.md`
- Flows: `docs/technical/architecture/request-flows.md`
- Data isolation: `docs/technical/architecture/data-isolation.md`
- Security boundaries: `docs/technical/architecture/security-boundaries.md`
### Quick Reference - Component Topology
### Key architectural principles
```
CLIENT LAYER
├─ stella CLI → Gateway (JWT + DPoP auth)
├─ Web UI (Angular) → Gateway (JWT + DPoP auth)
├─ CI/CD Pipelines → Gateway (JWT + DPoP auth)
└─ Zastava Observer → Scanner (runtime scans)
1. **Deterministic evidence**: the same inputs produce the same outputs (stable ordering, stable IDs, replayable artifacts).
2. **VEX-first decisioning**: policy decisions are driven by VEX inputs and issuer trust, not enumeration alone.
3. **Offline-first**: fully air-gapped workflows are supported (mirrors, bundles, importer/controller).
4. **Extensibility without drift**: connectors, plugins, and policy packs must preserve determinism.
5. **Sovereign posture**: bring-your-own trust roots and configurable crypto profiles where enabled.
6. **Isolation boundaries**: clear module ownership, schema boundaries, and tenant scoping.
INFRASTRUCTURE (REQUIRED)
├─ PostgreSQL v16+ → Primary database (ALL services)
├─ Valkey v8.0 → Cache, DPoP, queues, events
└─ RustFS → Object storage (S3 API)
### Service categories (orientation)
INFRASTRUCTURE (OPTIONAL)
└─ NATS JetStream → Alternative messaging (Valkey is default)
| Category | Examples | Purpose |
| --- | --- | --- |
| Infrastructure | PostgreSQL, Valkey, RustFS/S3, optional message broker | Durable state, coordination, artifact storage, transport abstraction. |
| Auth & signing | Authority, Signer, Attestor, issuer trust services | Identity, scopes/tenancy, evidence signing and attestation workflows. |
| Ingestion | Concelier, Excititor | Advisory and VEX ingestion/normalization with deterministic merges. |
| Scanning | Scanner (API + workers) | Container analysis, SBOM generation, artifact production. |
| Policy & risk | Policy engine + explain traces | Deterministic verdicts, waivers/exceptions, explainability for audits. |
| Orchestration | Scheduler, Orchestrator | Re-scan orchestration, workflows, pack runs. |
| Notifications | Notification engine(s) | Event delivery and idempotent notifications. |
| User experience | Gateway, Web UI, CLI | Authenticated access, routing, operator workflows. |
GATEWAY LAYER
└─ Gateway.WebService → Auth, routing, rate limiting
AUTH & CRYPTO
├─ Authority → OAuth2/OIDC, OpTok issuance
├─ Signer → DSSE signing (FIPS/GOST/SM)
└─ Attestor → Rekor v2 transparency log
CORE ENGINES
├─ Scanner.WebService → Scan orchestration
├─ Scanner.Worker → Image analysis, SBOM generation
├─ Concelier.WebService → Advisory ingestion (NVD, Red Hat, etc.)
├─ Excititor.WebService → VEX ingestion + consensus
├─ Policy.Gateway → OPA/Rego policy evaluation
├─ Scheduler.WebService → Re-scan orchestration
├─ Notify.WebService → Notification orchestration
├─ Notify.Worker → Slack/Teams/Email delivery
└─ Orchestrator.WebService → DAG workflows, pack runs
SUPPORTING
└─ IssuerDirectory → VEX issuer trust registry
```
### Service Categories
| Category | Services | Purpose |
|----------|----------|---------|
| **Gateway** | Gateway.WebService | API routing, auth enforcement |
| **Auth & Security** | Authority, Signer, Attestor | OAuth2, signing, transparency |
| **Scanning** | Scanner.Web, Scanner.Worker | Container analysis, SBOM |
| **Advisory** | Concelier.Web, Concelier.Worker | Vulnerability ingestion |
| **VEX** | Excititor.Web, Excititor.Worker | Exploitability statements |
| **Policy** | Policy.Gateway, Policy Engine | OPA/Rego evaluation |
| **Orchestration** | Scheduler, Orchestrator | Job coordination |
| **Notifications** | Notify.Web, Notify.Worker | Delivery to Slack/Teams/Email |
### Runtime Topology - Infrastructure Dependencies
```
┌─────────────────────────────────────────────────────────────────────┐
│ INFRASTRUCTURE LAYER │
│ ┌──────────────────┐ ┌──────────────────┐ ┌─────────────────┐ │
│ │ PostgreSQL │ │ Valkey │ │ RustFS │ │
│ │ (v16+ ONLY) │ │ (Redis-compat) │ │ (S3-like API) │ │
│ │ │ │ - Caching │ │ - Artifacts │ │
│ │ All services use │ │ - DPoP nonces │ │ - SBOMs │ │
│ │ PostgreSQL for │ │ - Event queues │ │ - Signatures │ │
│ │ persistent data │ │ - Rate limiting│ │ │ │
│ └──────────────────┘ └──────────────────┘ └─────────────────┘ │
│ │
│ ┌──────────────────────────────────────────────────────────────┐ │
│ │ Optional: NATS JetStream (alternative transport for queues) │ │
│ │ Only used if explicitly configured in appsettings │ │
│ └──────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────┘
▲
│
┌─────────────────────────────────────────────────────────────────────┐
│ AUTHENTICATION & SIGNING │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ Authority │─▶│ Signer │─▶│ Attestor │ │
│ │ (OAuth2/OIDC)│ │(DSSE/PKIX) │ │(in-toto/DSSE)│ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
└─────────────────────────────────────────────────────────────────────┘
▲
│
┌─────────────────────────────────────────────────────────────────────┐
│ INGESTION & AGGREGATION │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ Concelier │ │ Excititor │ │IssuerDirectry│ │
│ │(Advisories) │ │ (VEX) │ │(CSAF Pubshrs)│ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
└─────────────────────────────────────────────────────────────────────┘
▲
│
┌─────────────────────────────────────────────────────────────────────┐
│ SCANNING & ANALYSIS │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │Scanner.Web │ │Scanner.Worker│ │ AdvisoryAI │ │
│ │(API/Control) │ │(Analyzers) │ │(ML Analysis) │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
│ ┌──────────────┐ ┌──────────────┐ │
│ │ RiskEngine │ │ Policy │ │
│ │ (Scoring) │ │ (Engine) │ │
│ └──────────────┘ └──────────────┘ │
└─────────────────────────────────────────────────────────────────────┘
▲
│
┌─────────────────────────────────────────────────────────────────────┐
│ ORCHESTRATION & WORKFLOW │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ Scheduler │ │ Orchestrator │ │ TaskRunner │ │
│ │(Job Sched) │ │(Coordinator) │ │(Executor) │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
└─────────────────────────────────────────────────────────────────────┘
▲
│
┌─────────────────────────────────────────────────────────────────────┘
│ EVENTS & NOTIFICATIONS │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ Notify │ │ Notifier │ │TimelineIndex │ │
│ │(Slack/Teams) │ │ (Advanced) │ │ (Events) │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
└─────────────────────────────────────────────────────────────────────┘
▲
│
┌─────────────────────────────────────────────────────────────────────┐
│ DATA & EXPORT │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ExportCenter │ │EvidenceLocker│ │FindingsLedger│ │
│ │(SARIF/SBOM) │ │(Artifacts) │ │(Audit Trail) │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
└─────────────────────────────────────────────────────────────────────┘
▲
│
┌─────────────────────────────────────────────────────────────────────┐
│ USER EXPERIENCE │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ Gateway │ │ Web (UI) │ │ CLI │ │
│ │ (API Router) │ │ (Angular v17)│ │(Multi-plat) │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
└─────────────────────────────────────────────────────────────────────┘
```
### Detailed Request Flow - Scan Execution Example
This diagram shows a complete scan request lifecycle with detailed routing through services:
```
┌──────────────────────────────────────────────────────────────────────────────────┐
│ 1. CLIENT REQUEST (CLI or Web UI) │
│ $ stella scan docker://alpine:latest --sbom-format=spdx │
└───────────────────────────────────┬──────────────────────────────────────────────┘
│ HTTPS
▼
┌──────────────────────────────────────────────────────────────────────────────────┐
│ 2. GATEWAY (API Router) │
│ - Terminates TLS │
│ - Routes to appropriate backend service │
│ - Load balancing (if multiple instances) │
└───────────────────────────────────┬──────────────────────────────────────────────┘
│ HTTP (internal)
▼
┌──────────────────────────────────────────────────────────────────────────────────┐
│ 3. AUTHORITY (Authentication) │
│ - Validates OAuth2 access token (DPoP-bound) │
│ - Checks DPoP proof against Valkey nonce cache │
│ - Returns user identity and scopes │
│ │
│ ┌─────────────┐ │
│ │ Valkey │◀── DPoP nonce validation (GET/SET) │
│ │ (Cache) │ │
│ └─────────────┘ │
│ ┌─────────────┐ │
│ │ PostgreSQL │◀── User/client lookup (SELECT) │
│ └─────────────┘ │
└───────────────────────────────────┬──────────────────────────────────────────────┘
│ Authenticated request
▼
┌──────────────────────────────────────────────────────────────────────────────────┐
│ 4. SCANNER.WEB (Scan API Controller) │
│ - Validates scan request parameters │
│ - Creates scan job record in PostgreSQL │
│ - Enqueues scan job to Valkey queue (default) or NATS (if configured) │
│ │
│ ┌─────────────┐ │
│ │ PostgreSQL │◀── INSERT scan_jobs (job_id, image_ref, status='pending') │
│ └─────────────┘ │
│ ┌─────────────┐ │
│ │ Valkey │◀── XADD scanner:jobs (enqueue job message) │
│ │ (Queue) │ │
│ └─────────────┘ │
│ │
│ Returns: HTTP 202 Accepted { "job_id": "scan-abc123", "status": "queued" } │
└───────────────────────────────────┬──────────────────────────────────────────────┘
│
│ (Client polls for status)
│
┌──────────────────────────────────────────────────────────────────────────────────┐
│ 5. SCANNER.WORKER (Background Processor) │
│ - Consumes job from Valkey queue (XREADGROUP scanner:jobs) │
│ - Updates job status to 'running' │
│ - Downloads container image from registry │
│ - Executes analyzers (OS packages, language deps, files) │
│ - Generates SBOM (SPDX/CycloneDX) │
│ - Stores artifacts to RustFS │
│ │
│ ┌─────────────┐ │
│ │ PostgreSQL │◀── UPDATE scan_jobs SET status='running' │
│ │ │◀── INSERT sbom_documents, packages, vulnerabilities │
│ │ │◀── UPDATE scan_jobs SET status='completed' │
│ └─────────────┘ │
│ ┌─────────────┐ │
│ │ RustFS │◀── PUT /artifacts/scan-abc123/sbom.spdx.json │
│ │ (S3 API) │◀── PUT /artifacts/scan-abc123/image-layers.tar.gz │
│ └─────────────┘ │
│ ┌─────────────┐ │
│ │ Valkey │◀── XADD scanner:events (publish scan.completed event) │
│ │(Event Stream│ │
│ └─────────────┘ │
└───────────────────────────────────┬──────────────────────────────────────────────┘
│ Event published
▼
┌──────────────────────────────────────────────────────────────────────────────────┐
│ 6. EVENT PROPAGATION (Valkey Streams) │
│ │
│ ┌───────────────────────────────────────────────────────────────────┐ │
│ │ Valkey Event Stream: "scanner:events" │ │
│ │ Event: { "type": "scan.completed", "job_id": "scan-abc123", ... } │ │
│ └───────────────────────────────────────────────────────────────────┘ │
│ │ │
│ ┌──────────────┼──────────────┬───────────────┐ │
│ ▼ ▼ ▼ ▼ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ Notify │ │Timeline │ │ Policy │ │ Export │ │
│ │ Worker │ │ Indexer │ │ Engine │ │ Center │ │
│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │
│ │ │ │ │ │
│ │ (all subscribe to scanner:events via XREADGROUP) │
└─────────┼───────────────┼──────────────┼───────────────┼──────────────────────┘
│ │ │ │
▼ ▼ ▼ ▼
┌──────────────┐ ┌──────────────┐ ┌──────────┐ ┌──────────────┐
│ 7a. NOTIFY │ │ 7b. TIMELINE │ │ 7c.POLICY│ │ 7d. EXPORT │
│ │ │ INDEXER │ │ ENGINE │ │ CENTER │
│ - Query scan │ │ │ │ │ │ │
│ results │ │ - Index event│ │ - Eval │ │ - Generate │
│ - Check user │ │ timeline │ │ policy │ │ SARIF │
│ notif prefs│ │ - Store in │ │ rules │ │ - Export to │
│ - Send Slack │ │ PostgreSQL │ │ - Block/ │ │ external │
│ message │ │ │ │ Allow │ │ systems │
│ │ │ │ │ │ │ │
│ PostgreSQL ◀─┤ │ PostgreSQL ◀─┤ │PostgreSQL│ │ RustFS ◀─┤
│ (user prefs) │ │ (timeline) │ │(policies)│ │ (exports) │
└──────────────┘ └──────────────┘ └──────────┘ └──────────────┘
```
### Export Flow - SBOM Distribution
```
┌──────────────────────────────────────────────────────────────────────────────────┐
│ EXPORT REQUEST: GET /api/v1/scans/{scan_id}/export?format=spdx │
└───────────────────────────────────┬──────────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────────────────────────┐
│ SCANNER.WEB or EXPORT CENTER │
│ │
│ 1. Query scan metadata from PostgreSQL │
│ ┌─────────────┐ │
│ │ PostgreSQL │◀── SELECT * FROM scan_jobs WHERE job_id = $1 │
│ │ │◀── SELECT * FROM sbom_documents WHERE scan_id = $1 │
│ └─────────────┘ │
│ │
│ 2. Retrieve SBOM artifact from RustFS │
│ ┌─────────────┐ │
│ │ RustFS │◀── GET /artifacts/scan-abc123/sbom.spdx.json │
│ └─────────────┘ │
│ │
│ 3. Sign SBOM with Signer service │
│ ┌─────────────┐ │
│ │ Signer │◀── POST /api/v1/sign (SBOM payload) │
│ │ │──▶ Returns: DSSE envelope with signature │
│ └─────────────┘ │
│ │
│ 4. Create in-toto attestation with Attestor │
│ ┌─────────────┐ │
│ │ Attestor │◀── POST /api/v1/attest (signed SBOM) │
│ │ │──▶ Returns: in-toto attestation bundle │
│ └─────────────┘ │
│ │
│ 5. Store final bundle to RustFS │
│ ┌─────────────┐ │
│ │ RustFS │◀── PUT /artifacts/scan-abc123/bundle.jsonl │
│ └─────────────┘ │
│ │
│ 6. Return signed bundle to client │
│ Returns: HTTP 200 OK (application/vnd.in-toto+json) │
└──────────────────────────────────────────────────────────────────────────────────┘
```
### Notification Flow - Vulnerability Alert
```
┌──────────────────────────────────────────────────────────────────────────────────┐
│ TRIGGER: New critical CVE detected in existing scan │
│ Source: Concelier advisory ingestion │
└───────────────────────────────────┬──────────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────────────────────────┐
│ CONCELIER.WORKER (Advisory Processor) │
│ │
│ 1. Ingest new advisory from NVD/OSV/CSAF │
│ ┌─────────────┐ │
│ │ PostgreSQL │◀── INSERT INTO advisories (cve_id, severity, ...) │
│ └─────────────┘ │
│ │
│ 2. Match advisory against existing scans (PURL/CPE matching) │
│ ┌─────────────┐ │
│ │ PostgreSQL │◀── SELECT scans WHERE package_purl IN (affected_purls) │
│ └─────────────┘ │
│ │
│ 3. Publish drift event to Valkey │
│ ┌─────────────┐ │
│ │ Valkey │◀── XADD concelier:drift (new vulnerability found) │
│ └─────────────┘ │
└───────────────────────────────────┬──────────────────────────────────────────────┘
│ Event published
▼
┌──────────────────────────────────────────────────────────────────────────────────┐
│ NOTIFY.WORKER (Notification Processor) │
│ │
│ 1. Consume drift event from Valkey stream │
│ ┌─────────────┐ │
│ │ Valkey │◀── XREADGROUP concelier:drift notify-workers │
│ └─────────────┘ │
│ │
│ 2. Query user notification preferences │
│ ┌─────────────┐ │
│ │ PostgreSQL │◀── SELECT * FROM user_notification_preferences │
│ │ │ WHERE user_id = scan_owner AND channel = 'slack' │
│ └─────────────┘ │
│ │
│ 3. Render notification template │
│ Template: "🚨 New critical CVE-2024-1234 affects alpine:latest scan" │
│ │
│ 4. Deliver notification via configured channels │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ External APIs │ │
│ │ - POST https://hooks.slack.com/services/T00/B00/xxx │ │
│ │ - POST https://graph.microsoft.com/v1.0/teams/channels │ │
│ │ - SMTP send (email) │ │
│ └─────────────────────────────────────────────────────────┘ │
│ │
│ 5. Store delivery receipt in PostgreSQL │
│ ┌─────────────┐ │
│ │ PostgreSQL │◀── INSERT INTO notification_deliveries (status, ...) │
│ └─────────────┘ │
└──────────────────────────────────────────────────────────────────────────────────┘
```
### Key Architectural Principles
1. **Deterministic Evidence** - Same inputs always produce same outputs
2. **VEX-First Decisioning** - Policy decisions based on OpenVEX statements
3. **Offline-First** - Full air-gap operation supported
4. **Plugin Architecture** - Extensible connectors for advisories, analyzers, auth
5. **Sovereign Crypto** - FIPS, eIDAS, GOST, SM support
6. **Schema Isolation** - Per-module PostgreSQL schemas
### Service Categories
| Category | Services | Purpose |
|----------|----------|---------|
| **Infrastructure** | PostgreSQL v16+, Valkey 8.0, RustFS, NATS (optional) | Database, cache/messaging, object storage, optional queue transport |
| **Auth & Signing** | Authority, Signer, Attestor | OAuth2/OIDC with DPoP, cryptographic signing, in-toto attestations |
| **Ingestion** | Concelier, Excititor, IssuerDirectory | Advisory/VEX ingestion, normalization, merging, CSAF publisher discovery |
| **Scanning** | Scanner.Web, Scanner.Worker, AdvisoryAI | Container scanning, SBOM generation (SPDX/CDX), ML vulnerability analysis |
| **Policy & Risk** | Policy Engine, RiskEngine | OPA/Rego policy evaluation, risk scoring, exploitability assessment |
| **Orchestration** | Scheduler, Orchestrator, TaskRunner | Job scheduling, workflow coordination, distributed task execution |
| **Notifications** | Notify, Notifier, TimelineIndexer | Event delivery (Slack/Teams/Email), notification management, timeline tracking |
| **Data & Export** | ExportCenter, EvidenceLocker, FindingsLedger | SARIF/SBOM export, evidence storage, immutable audit trail |
| **User Experience** | Gateway, Web UI, CLI | API routing, Angular v17 UI, multi-platform command-line tools |
### Canonical flows
- Scan execution, ingestion updates, policy evaluation, and notification delivery are described in `docs/technical/architecture/request-flows.md`.
---
## Prerequisites
### Required Software
@@ -446,7 +96,7 @@ This diagram shows a complete scan request lifecycle with detailed routing throu
```bash
cd C:\dev\
git clone https://git.stella-ops.org/stella-ops.org/git.stella-ops.org
git clone https://git.stella-ops.org/stella-ops.org/git.stella-ops.org.git
cd git.stella-ops.org
```
@@ -462,40 +112,9 @@ notepad .env
```
**Key settings to configure:**
```bash
# PostgreSQL Database
POSTGRES_USER=stellaops
POSTGRES_PASSWORD=your_secure_password_here
POSTGRES_DB=stellaops_platform
POSTGRES_PORT=5432
# Valkey (Redis-compatible cache and messaging)
VALKEY_PORT=6379
# RustFS Object Storage
RUSTFS_HTTP_PORT=8080
# Service ports (adjust if conflicts exist)
AUTHORITY_PORT=8440
SIGNER_PORT=8441
ATTESTOR_PORT=8442
CONCELIER_PORT=8445
SCANNER_WEB_PORT=8444
NOTIFY_WEB_PORT=8446
# Scanner configuration (Valkey default, can switch to NATS if needed)
SCANNER_EVENTS_DRIVER=valkey
SCANNER_EVENTS_DSN=valkey:6379
# Scheduler configuration (Valkey default, can switch to NATS if needed)
SCHEDULER_QUEUE_KIND=Valkey
SCHEDULER_QUEUE_VALKEY_URL=valkey:6379
# Authority configuration
AUTHORITY_ISSUER=https://authority:8440
SIGNER_POE_INTROSPECT_URL=https://www.stella-ops.org/license/introspect
```
- Copy and edit the profile env file (`deploy/compose/env/dev.env.example` -> `.env`).
- Update at minimum `POSTGRES_PASSWORD` and any host port overrides needed for your machine.
- Treat `deploy/compose/env/*.env.example` as the authoritative list of variables for each profile (queue/transport knobs are profile-dependent).
### Step 3: Start the Full Platform
@@ -544,373 +163,36 @@ Open your browser and navigate to:
## Hybrid Debugging Workflow
The hybrid workflow allows you to:
1. Run infrastructure (databases, queues) in Docker
2. Run most services in Docker
3. **Selectively debug** one or two services in Visual Studio
Hybrid debugging runs the full platform in Docker, then stops one service container and runs that service locally under a debugger while it continues to use Docker-hosted dependencies.
### Workflow Overview
```
┌────────────────────────────────────────────────────────────┐
│ DOCKER ENVIRONMENT │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │PostgreSQL│ │ Valkey │ │ RustFS │ │
│ │ (DB) │ │(Cache/Msg│ │(Storage) │ │
│ └──────────┘ └──────────┘ └──────────┘ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ Authority│ │ Signer │ │ Attestor │ │
│ └──────────┘ └──────────┘ └──────────┘ │
│ ┌──────────┐ ┌──────────┐ │
│ │Concelier │ │ Excititor│ ← Running normally │
│ └──────────┘ └──────────┘ │
└────────────────────────────────────────────────────────────┘
▲
│ HTTP calls + Valkey streams
▼
┌────────────────────────────────────────────────────────────┐
│ VISUAL STUDIO (F5) │
│ ┌────────────────────────────────────────────┐ │
│ │ Scanner.WebService │ │
│ │ Running on http://localhost:5210 │ │
│ │ (Breakpoints, hot reload, debugging) │ ← YOU DEBUG HERE
│ └────────────────────────────────────────────┘ │
└────────────────────────────────────────────────────────────┘
```
### Step-by-Step: Debug Scanner.WebService
#### 1. Stop the Docker Container for Scanner
```bash
cd deploy\compose
docker compose -f docker-compose.dev.yaml stop scanner-web
```
**Verify it's stopped:**
```bash
docker compose -f docker-compose.dev.yaml ps scanner-web
# Should show: State = "exited"
```
#### 2. Configure Local Development Settings
Create or modify the service's `appsettings.Development.json`:
```bash
cd C:\dev\New folder\git.stella-ops.org\src\Scanner\StellaOps.Scanner.WebService
```
**Create `appsettings.Development.json`:**
```json
{
"Logging": {
"LogLevel": {
"Default": "Information",
"Microsoft.AspNetCore": "Warning",
"StellaOps": "Debug"
}
},
"ConnectionStrings": {
"DefaultConnection": "Host=localhost;Port=5432;Database=stellaops_platform;Username=stellaops;Password=your_password_here;Include Error Detail=true"
},
"Scanner": {
"Storage": {
"Mongo": {
"ConnectionString": "mongodb://stellaops:your_password_here@localhost:27017"
}
},
"ArtifactStore": {
"Driver": "rustfs",
"Endpoint": "http://localhost:8080/api/v1",
"Bucket": "scanner-artifacts",
"TimeoutSeconds": 30
},
"Queue": {
"Broker": "nats://localhost:4222"
},
"Events": {
"Enabled": false
}
},
"Authority": {
"Issuer": "https://localhost:8440",
"BaseUrl": "https://localhost:8440",
"BypassNetworks": ["127.0.0.1", "::1"]
}
}
```
**Note:** Adjust connection strings to match your Docker infrastructure ports. If PostgreSQL is on Docker's bridge network, you may need to expose it on `localhost:5432`.
#### 3. Expose Docker Services to localhost
For services running in Docker to be accessible from your host machine, ensure ports are mapped in `docker-compose.dev.yaml`:
```yaml
# Already configured in docker-compose.dev.yaml
postgres:
ports:
- "${POSTGRES_PORT:-5432}:5432"
mongo:
ports:
- "27017:27017"
nats:
ports:
- "${NATS_CLIENT_PORT:-4222}:4222"
rustfs:
ports:
- "${RUSTFS_HTTP_PORT:-8080}:8080"
```
**Verify connectivity:**
```bash
# Test PostgreSQL
psql -h localhost -U stellaops -d stellaops_platform
# Test NATS
telnet localhost 4222
# Test RustFS
curl http://localhost:8080/health
```
#### 4. Open Solution in Visual Studio
```bash
# Open the solution
cd C:\dev\New folder\git.stella-ops.org
start src\StellaOps.sln
```
**In Visual Studio:**
1. Right-click `StellaOps.Scanner.WebService` project
2. Select **"Set as Startup Project"**
3. Press **F5** to start debugging
**Expected output:**
```
info: Microsoft.Hosting.Lifetime[14]
Now listening on: http://localhost:5210
info: Microsoft.Hosting.Lifetime[0]
Application started. Press Ctrl+C to shut down.
```
#### 5. Update Other Services to Call localhost
Since you're running Scanner.WebService on `localhost:5210` instead of `scanner-web:8444`, you need to update any services that call it.
**Option A: Environment Variables (Docker containers)**
Update `.env` file:
```bash
# Use host.docker.internal to reach host machine from Docker
SCANNER_WEB_BASEURL=http://host.docker.internal:5210
```
Restart dependent services:
```bash
docker compose -f docker-compose.dev.yaml restart scheduler-web
```
**Option B: Modify docker-compose.dev.yaml**
```yaml
scheduler-web:
environment:
SCHEDULER__WORKER__RUNNER__SCANNER__BASEADDRESS: "http://host.docker.internal:5210"
```
Then restart:
```bash
docker compose -f docker-compose.dev.yaml up -d scheduler-web
```
#### 6. Set Breakpoints and Debug
1. Navigate to `Program.cs` in Scanner.WebService
2. Set a breakpoint on a line in a controller or service method
3. Trigger the endpoint using:
- Swagger UI (if enabled): http://localhost:5210/swagger
- Postman/curl
- CLI command
**Example curl:**
```bash
curl -X POST http://localhost:5210/api/scans \
-H "Content-Type: application/json" \
-d '{"imageRef": "alpine:latest"}'
```
Your breakpoint should hit, and you can step through code.
#### 7. Return to Docker Mode
When you're done debugging:
```bash
# Stop Visual Studio debugger (Shift+F5)
# Restart the Docker container
cd deploy\compose
docker compose -f docker-compose.dev.yaml start scanner-web
# Verify it's running
docker compose -f docker-compose.dev.yaml ps scanner-web
```
---
Canonical guide:
- `docs/QUICKSTART_HYBRID_DEBUG.md`
Related references:
- Compose profiles: `deploy/compose/README.md`
- Install guide: `docs/21_INSTALL_GUIDE.md`
- Service-specific runbooks: `docs/modules/<module>/operations/`
## Service-by-Service Debugging Guide
### Authority (OAuth2/OIDC Provider)
Service-specific debugging guidance lives with each module to avoid stale, copy-pasted configuration examples.
**Project:** `src/Authority/StellaOps.Authority/StellaOps.Authority.csproj`
Generic workflow:
1. Stop the service container in `deploy/compose` (for example: `docker compose -f docker-compose.dev.yaml stop <service>`).
2. Run the service locally under a debugger.
3. Update dependent services to call `host.docker.internal:<port>` (or your host IP) and restart them.
4. Use the module operations docs for required env vars, auth scopes, and health checks.
**Stop Docker container:**
```bash
docker compose -f docker-compose.dev.yaml stop authority
```
Start here:
- Hybrid debugging walkthrough: `docs/QUICKSTART_HYBRID_DEBUG.md`
- Architecture index: `docs/technical/architecture/README.md`
- Module dossiers and operations: `docs/modules/`
**Configuration:** `appsettings.Development.json`
```json
{
"ConnectionStrings": {
"DefaultConnection": "Host=localhost;Port=5432;Database=stellaops_platform;Username=stellaops;Password=your_password"
},
"StellaOps_Authority": {
"Issuer": "https://localhost:5001",
"Mongo": {
"ConnectionString": "mongodb://stellaops:your_password@localhost:27017"
}
},
"Kestrel": {
"Endpoints": {
"Https": {
"Url": "https://localhost:5001"
}
}
}
}
```
**Run in Visual Studio:** F5 on `StellaOps.Authority` project
**Default URL:** https://localhost:5001
**Update dependent services:**
```bash
# In .env
AUTHORITY_ISSUER=https://host.docker.internal:5001
```
### Concelier (Advisory Ingestion)
**Project:** `src/Concelier/StellaOps.Concelier.WebService/StellaOps.Concelier.WebService.csproj`
**Stop Docker:**
```bash
docker compose -f docker-compose.dev.yaml stop concelier
```
**Configuration:** `appsettings.Development.json`
```json
{
"ConnectionStrings": {
"DefaultConnection": "Host=localhost;Port=5432;Database=stellaops_platform;Username=stellaops;Password=your_password"
},
"Concelier": {
"Storage": {
"Mongo": {
"ConnectionString": "mongodb://stellaops:your_password@localhost:27017"
},
"S3": {
"Endpoint": "http://localhost:9000",
"AccessKeyId": "stellaops",
"SecretAccessKey": "your_password"
}
},
"Authority": {
"BaseUrl": "https://localhost:8440"
}
}
}
```
**Run:** F5 on `StellaOps.Concelier.WebService`
**Default URL:** http://localhost:5000
### Scanner.Worker (Background Analyzer)
**Project:** `src/Scanner/StellaOps.Scanner.Worker/StellaOps.Scanner.Worker.csproj`
**Stop Docker:**
```bash
docker compose -f docker-compose.dev.yaml stop scanner-worker
```
**Configuration:** Same as Scanner.WebService (shares settings)
**Run:** F5 on `StellaOps.Scanner.Worker`
**Note:** Worker has no HTTP endpoint - it consumes from NATS queue
### Scheduler.WebService
**Project:** `src/Scheduler/StellaOps.Scheduler.WebService/StellaOps.Scheduler.WebService.csproj`
**Stop Docker:**
```bash
docker compose -f docker-compose.dev.yaml stop scheduler-web
```
**Configuration:** `appsettings.Development.json`
```json
{
"ConnectionStrings": {
"DefaultConnection": "Host=localhost;Port=5432;Database=stellaops_orchestrator;Username=stellaops;Password=your_password"
},
"Scheduler": {
"Queue": {
"Kind": "Nats",
"Nats": {
"Url": "nats://localhost:4222"
}
},
"Worker": {
"Runner": {
"Scanner": {
"BaseAddress": "http://localhost:5210"
}
}
}
}
}
```
**Run:** F5 on `StellaOps.Scheduler.WebService`
### Notify.WebService
**Project:** `src/Notify/StellaOps.Notify.WebService/StellaOps.Notify.WebService.csproj`
**Stop Docker:**
```bash
docker compose -f docker-compose.dev.yaml stop notify-web
```
**Configuration:** Uses `etc/notify.dev.yaml`
**Run:** F5 on `StellaOps.Notify.WebService`
---
Common module runbooks:
- Authority: `docs/modules/authority/operations/`
- Scanner: `docs/modules/scanner/operations/`
- Concelier: `docs/modules/concelier/operations/`
- Scheduler: `docs/modules/scheduler/operations/`
- UI / Console: `docs/modules/ui/`
## Configuration Deep Dive
@@ -1419,7 +701,7 @@ sudo docker compose -f docker-compose.dev.yaml up -d
3. **Week 3: Authentication & Security**
- Master OAuth2/OIDC flow in Authority
- Understand signing flow (Signer → Attestor → Rekor)
- Understand signing flow (Signer -> Attestor -> Rekor)
- Study policy evaluation engine
4. **Week 4: Integration**