371 lines
13 KiB
Markdown
371 lines
13 KiB
Markdown
# Dev Environment Setup
|
|
|
|
Actionable checklist for getting a local Stella Ops development environment running.
|
|
For hybrid debugging workflows and service-specific guides, see [`docs/DEVELOPER_ONBOARDING.md`](../DEVELOPER_ONBOARDING.md).
|
|
|
|
---
|
|
|
|
## Quick Start (automated)
|
|
|
|
Setup scripts validate prerequisites, build solutions and Docker images, and launch the full platform.
|
|
|
|
**Windows (PowerShell 7):**
|
|
|
|
```powershell
|
|
.\scripts\setup.ps1 # full setup
|
|
.\scripts\setup.ps1 -InfraOnly # infrastructure only (PostgreSQL, Valkey, SeaweedFS, Zot; Rekor is opt-in)
|
|
.\scripts\setup.ps1 -SkipBuild # skip .NET builds, build images and start platform
|
|
.\scripts\setup.ps1 -SkipImages # build .NET but skip Docker images
|
|
.\scripts\setup.ps1 -ImagesOnly # only build Docker images
|
|
```
|
|
|
|
**Linux / macOS:**
|
|
|
|
```bash
|
|
./scripts/setup.sh # full setup
|
|
./scripts/setup.sh --infra-only # infrastructure only
|
|
./scripts/setup.sh --skip-build # skip .NET builds
|
|
./scripts/setup.sh --skip-images # skip Docker image builds
|
|
./scripts/setup.sh --images-only # only build Docker images
|
|
```
|
|
|
|
The scripts will check for required tools (dotnet 10.x, node 20+, npm 10+, docker, git), warn about missing hosts file entries, and copy `.env` from the example if needed. See the manual steps below for details on each stage.
|
|
|
|
### Quick validation + demo seed (first-run path)
|
|
|
|
```powershell
|
|
# 1) Bring platform up quickly (reuse existing images)
|
|
.\scripts\setup.ps1 -SkipBuild -SkipImages
|
|
|
|
# 2) Validate platform health
|
|
docker compose -f devops/compose/docker-compose.stella-ops.yml ps
|
|
|
|
# 3) Preview seed work
|
|
dotnet run --project src/Cli/StellaOps.Cli/StellaOps.Cli.csproj -- `
|
|
admin seed-demo --dry-run `
|
|
--connection "Host=127.1.1.1;Port=5432;Database=stellaops_platform;Username=stellaops;Password=stellaops"
|
|
|
|
# 4) Execute demo seeding
|
|
dotnet run --project src/Cli/StellaOps.Cli/StellaOps.Cli.csproj -- `
|
|
admin seed-demo --confirm `
|
|
--connection "Host=127.1.1.1;Port=5432;Database=stellaops_platform;Username=stellaops;Password=stellaops"
|
|
```
|
|
|
|
### Known warnings vs blocking failures
|
|
|
|
| Output | Class | Meaning | Action |
|
|
|---|---|---|---|
|
|
| `health=starting` (RustFS) | Warning | Service still warming up | Wait and recheck `docker compose ... ps` |
|
|
| `SM remote service probe failed (localhost:56080)` | Warning | Optional SM remote provider is unavailable | Ignore unless validating China SM remote crypto profile |
|
|
| `stellaops-dev-rekor restarting` without `--profile sigstore` | Warning | Optional Sigstore container from prior run | Ignore for default profile or remove stale container |
|
|
| `policy ... scheduler_exceptions_tenant_isolation already exists` | Blocking | Outdated Scheduler migration idempotency | Update code and rerun seeding |
|
|
| `POST /api/v1/admin/seed-demo` returns 500 after patching source | Blocking | Running stale platform container image | Rebuild/restart platform image |
|
|
|
|
---
|
|
|
|
## 1. Prerequisites
|
|
|
|
| Tool | Version | Verify |
|
|
|------|---------|--------|
|
|
| .NET 10 SDK | 10.0.100 (pinned in `global.json`) | `dotnet --version` |
|
|
| Node.js | ^20.19.0 \|\| ^22.12.0 \|\| ^24.0.0 | `node --version` |
|
|
| npm | >=10.2.0 | `npm --version` |
|
|
| Docker Desktop / Engine + Compose | 20.10+ | `docker --version` |
|
|
| Git | 2.30+ | `git --version` |
|
|
| PowerShell 7+ (Windows) or Bash | -- | `pwsh --version` / `bash --version` |
|
|
|
|
### Optional
|
|
|
|
- Visual Studio 2022 v17.12+ (ASP.NET and web development workload)
|
|
- VS Code + C# Dev Kit
|
|
- PostgreSQL client (`psql`, DBeaver, pgAdmin)
|
|
- `valkey-cli` or Redis Insight (Valkey is Redis-compatible)
|
|
- AWS CLI or `s3cmd` for RustFS inspection
|
|
|
|
### System requirements
|
|
|
|
- **RAM:** 16 GB minimum, 32 GB recommended
|
|
- **Disk:** 50 GB free (Docker images, volumes, build artifacts)
|
|
- **CPU:** 4 cores minimum, 8 cores recommended
|
|
|
|
---
|
|
|
|
## 1b. Runtime data assets
|
|
|
|
Some services need files that `dotnet build` does not produce. For local dev the
|
|
most impactful one is the ONNX embedding model used by AdvisoryAI for semantic
|
|
search. Without it the encoder falls back to a reduced-quality projection.
|
|
|
|
```bash
|
|
# Download the embedding model (~80 MB) — run once after cloning
|
|
./devops/runtime-assets/acquire.sh --models
|
|
|
|
# Optional: JDK + Ghidra for binary analysis (~1.6 GB)
|
|
./devops/runtime-assets/acquire.sh --ghidra
|
|
|
|
# Verify all assets
|
|
./devops/runtime-assets/acquire.sh --verify
|
|
```
|
|
|
|
Full inventory, Docker volume mounts, and air-gap packaging:
|
|
[`devops/runtime-assets/README.md`](../../devops/runtime-assets/README.md).
|
|
|
|
---
|
|
|
|
## 2. Hosts file setup
|
|
|
|
Each service binds to a unique loopback IP so all can use ports 443/80 without collisions.
|
|
Full details: [`docs/technical/architecture/port-registry.md`](../technical/architecture/port-registry.md).
|
|
|
|
### Automated (recommended)
|
|
|
|
The setup scripts (`scripts/setup.ps1` / `scripts/setup.sh`) will detect missing entries and offer to install them automatically.
|
|
|
|
### Manual
|
|
|
|
Append the contents of [`devops/compose/hosts.stellaops.local`](../../devops/compose/hosts.stellaops.local) to your hosts file:
|
|
|
|
- **Windows:** Run an elevated PowerShell and run:
|
|
```powershell
|
|
Get-Content devops\compose\hosts.stellaops.local | Add-Content C:\Windows\System32\drivers\etc\hosts
|
|
```
|
|
- **Linux / macOS:**
|
|
```bash
|
|
sudo sh -c 'cat devops/compose/hosts.stellaops.local >> /etc/hosts'
|
|
```
|
|
|
|
The file contains ~50 entries mapping services to unique loopback IPs (127.1.0.1 through 127.1.1.5).
|
|
See the file for the full list.
|
|
|
|
---
|
|
|
|
## 3. Start infrastructure (Docker)
|
|
|
|
```bash
|
|
cd devops/compose
|
|
cp env/stellaops.env.example .env # works out of the box; change POSTGRES_PASSWORD for production
|
|
docker compose -f docker-compose.dev.yml up -d
|
|
docker compose -f docker-compose.dev.yml ps
|
|
```
|
|
|
|
### Verify infrastructure
|
|
|
|
```bash
|
|
# PostgreSQL
|
|
psql -h db.stella-ops.local -U stellaops -d stellaops_dev -c "SELECT 1"
|
|
|
|
# Valkey
|
|
valkey-cli -h cache.stella-ops.local ping
|
|
```
|
|
|
|
Infrastructure versions (from `docker-compose.dev.yml`):
|
|
|
|
| Service | Version | Hostname | Port |
|
|
|---------|---------|----------|------|
|
|
| PostgreSQL | 18.1 | `db.stella-ops.local` | 5432 |
|
|
| Valkey | 9.0.1 | `cache.stella-ops.local` | 6379 |
|
|
| SeaweedFS (S3) | -- | `s3.stella-ops.local` | 8333 |
|
|
| Rekor v2 (optional `sigstore` profile) | -- | `rekor.stella-ops.local` | 3322 |
|
|
| Zot (OCI registry) | v2.1.3 | `registry.stella-ops.local` | 80 |
|
|
|
|
---
|
|
|
|
## 4. Build .NET modules
|
|
|
|
The codebase uses a **module-first** approach -- there is no root solution file used for builds. Each module has its own `.sln` under `src/<Module>/`.
|
|
|
|
### Single module
|
|
|
|
```powershell
|
|
dotnet build src\Scanner\StellaOps.Scanner.sln
|
|
dotnet test src\Scanner\StellaOps.Scanner.sln
|
|
```
|
|
|
|
### All modules
|
|
|
|
```powershell
|
|
# Windows (PowerShell 7)
|
|
.\scripts\build-all-solutions.ps1
|
|
|
|
# With tests
|
|
.\scripts\build-all-solutions.ps1 -Test
|
|
|
|
# Linux / macOS
|
|
./scripts/build-all-solutions.sh
|
|
|
|
# With tests
|
|
./scripts/build-all-solutions.sh --test
|
|
```
|
|
|
|
### Module solution index
|
|
|
|
See [`docs/dev/SOLUTION_BUILD_GUIDE.md`](SOLUTION_BUILD_GUIDE.md) for the authoritative list. Current modules (39):
|
|
|
|
| Module | Solution path |
|
|
|--------|---------------|
|
|
| AdvisoryAI | `src/AdvisoryAI/StellaOps.AdvisoryAI.sln` |
|
|
| AirGap | `src/AirGap/StellaOps.AirGap.sln` |
|
|
| Aoc | `src/Aoc/StellaOps.Aoc.sln` |
|
|
| Attestor | `src/Attestor/StellaOps.Attestor.sln` |
|
|
| Authority | `src/Authority/StellaOps.Authority.sln` |
|
|
| Bench | `src/Bench/StellaOps.Bench.sln` |
|
|
| BinaryIndex | `src/BinaryIndex/StellaOps.BinaryIndex.sln` |
|
|
| Cartographer (absorbed into Scanner) | `src/Scanner/StellaOps.Scanner.sln` |
|
|
| Cli | `src/Cli/StellaOps.Cli.sln` |
|
|
| Concelier | `src/Concelier/StellaOps.Concelier.sln` |
|
|
| EvidenceLocker | `src/EvidenceLocker/StellaOps.EvidenceLocker.sln` |
|
|
| Excititor | `src/Excititor/StellaOps.Excititor.sln` |
|
|
| ExportCenter | `src/ExportCenter/StellaOps.ExportCenter.sln` |
|
|
| Feedser | `src/Feedser/StellaOps.Feedser.sln` |
|
|
| Findings | `src/Findings/StellaOps.Findings.sln` |
|
|
| Router (Gateway) | `src/Router/StellaOps.Router.sln` |
|
|
| Graph | `src/Graph/StellaOps.Graph.sln` |
|
|
| IssuerDirectory | `src/IssuerDirectory/StellaOps.IssuerDirectory.sln` |
|
|
| Notifier | `src/Notifier/StellaOps.Notifier.sln` |
|
|
| Notify | `src/Notify/StellaOps.Notify.sln` |
|
|
| Orchestrator | `src/JobEngine/StellaOps.JobEngine.sln` |
|
|
| PacksRegistry | `src/PacksRegistry/StellaOps.PacksRegistry.sln` |
|
|
| Policy | `src/Policy/StellaOps.Policy.sln` |
|
|
| ReachGraph | `src/ReachGraph/StellaOps.ReachGraph.sln` |
|
|
| Registry | `src/Registry/StellaOps.Registry.sln` |
|
|
| Replay | `src/Replay/StellaOps.Replay.sln` |
|
|
| RiskEngine | `src/Findings/StellaOps.Findings.sln` (consolidated into Findings) |
|
|
| Router | `src/Router/StellaOps.Router.sln` |
|
|
| SbomService | `src/SbomService/StellaOps.SbomService.sln` |
|
|
| Scanner | `src/Scanner/StellaOps.Scanner.sln` |
|
|
| Scheduler | `src/Scheduler/StellaOps.Scheduler.sln` |
|
|
| Signer | `src/Signer/StellaOps.Signer.sln` |
|
|
| Signals | `src/Signals/StellaOps.Signals.sln` |
|
|
| SmRemote | `src/SmRemote/StellaOps.SmRemote.sln` |
|
|
| TaskRunner | `src/TaskRunner/StellaOps.TaskRunner.sln` |
|
|
| Telemetry | `src/Telemetry/StellaOps.Telemetry.sln` |
|
|
| Timeline (incl. TimelineIndexer) | `src/Timeline/` (no standalone sln; use root `StellaOps.sln`) |
|
|
| Tools | `src/Tools/StellaOps.Tools.sln` |
|
|
| VexHub | `src/VexHub/StellaOps.VexHub.sln` |
|
|
| VexLens | `src/VexLens/StellaOps.VexLens.sln` |
|
|
| VulnExplorer | `src/Findings/StellaOps.Findings.sln` (consolidated into Findings) |
|
|
| Zastava | `src/Zastava/StellaOps.Zastava.sln` |
|
|
|
|
---
|
|
|
|
## 5. Build Angular frontend
|
|
|
|
```bash
|
|
cd src/Web/StellaOps.Web
|
|
npm ci --prefer-offline --no-audit --no-fund
|
|
npm run start # dev server -> https://stella-ops.local
|
|
npm run build # production build
|
|
npm run test # unit tests (Vitest)
|
|
npm run test:e2e # Playwright E2E
|
|
```
|
|
|
|
Additional scripts:
|
|
|
|
| Command | Purpose |
|
|
|---------|---------|
|
|
| `npm run storybook` | Launch Storybook component explorer |
|
|
| `npm run analyze` | Bundle size visualization (esbuild-visualizer) |
|
|
| `npm run test:a11y` | Accessibility smoke tests |
|
|
|
|
---
|
|
|
|
## 6. Build Docker images
|
|
|
|
### Option A: Build all services (matrix-driven)
|
|
|
|
```bash
|
|
cd devops/docker
|
|
./build-all.sh
|
|
```
|
|
|
|
Uses `services-matrix.env` and `Dockerfile.hardened.template` for .NET services, `Dockerfile.console` for Angular.
|
|
|
|
### Option B: Build a single .NET service
|
|
|
|
```bash
|
|
docker build -f devops/docker/Dockerfile.hardened.template . \
|
|
--build-arg SDK_IMAGE=mcr.microsoft.com/dotnet/sdk:10.0-bookworm-slim \
|
|
--build-arg RUNTIME_IMAGE=mcr.microsoft.com/dotnet/aspnet:10.0-bookworm-slim \
|
|
--build-arg APP_PROJECT=src/Scanner/StellaOps.Scanner.WebService/StellaOps.Scanner.WebService.csproj \
|
|
--build-arg APP_BINARY=StellaOps.Scanner.WebService \
|
|
--build-arg APP_PORT=8080 \
|
|
-t stellaops/scanner-web:dev
|
|
```
|
|
|
|
### Option C: Build the Angular console image
|
|
|
|
```bash
|
|
docker build -f devops/docker/Dockerfile.console . \
|
|
--build-arg APP_DIR=src/Web/StellaOps.Web \
|
|
-t stellaops/console:dev
|
|
```
|
|
|
|
### Release-quality builds (distroless)
|
|
|
|
Release Dockerfiles live under `devops/release/docker/`:
|
|
|
|
- `Dockerfile.dotnet-service` -- .NET services
|
|
- `Dockerfile.angular-ui` -- Angular console
|
|
|
|
Component manifest: `devops/release/components.json`.
|
|
|
|
---
|
|
|
|
## 7. Run the full platform
|
|
|
|
```bash
|
|
# Core services
|
|
docker compose -f devops/compose/docker-compose.stella-ops.yml up -d
|
|
|
|
# With Sigstore transparency log
|
|
docker compose -f devops/compose/docker-compose.stella-ops.yml --profile sigstore up -d
|
|
|
|
# With telemetry stack
|
|
docker compose -f devops/compose/docker-compose.stella-ops.yml \
|
|
-f devops/compose/docker-compose.telemetry.yml up -d
|
|
```
|
|
|
|
Verify:
|
|
|
|
```bash
|
|
docker compose -f devops/compose/docker-compose.stella-ops.yml ps
|
|
```
|
|
|
|
---
|
|
|
|
## 8. Seed demo data and verify endpoint errors
|
|
|
|
Use the CLI seeder for local bootstraps and demo datasets:
|
|
|
|
```powershell
|
|
# dry-run
|
|
dotnet run --project src/Cli/StellaOps.Cli/StellaOps.Cli.csproj -- `
|
|
admin seed-demo --dry-run `
|
|
--connection "Host=127.1.1.1;Port=5432;Database=stellaops_platform;Username=stellaops;Password=stellaops"
|
|
|
|
# execute
|
|
dotnet run --project src/Cli/StellaOps.Cli/StellaOps.Cli.csproj -- `
|
|
admin seed-demo --confirm `
|
|
--connection "Host=127.1.1.1;Port=5432;Database=stellaops_platform;Username=stellaops;Password=stellaops"
|
|
```
|
|
|
|
Seed API behavior (`POST /api/v1/admin/seed-demo`) now returns deterministic non-500 errors for expected failure modes:
|
|
- `401/403` for auth policy failures (`platform.setup.admin`)
|
|
- `503` when demo seeding is disabled (`STELLAOPS_ENABLE_DEMO_SEED=false`)
|
|
- `400` for invalid module filters (for example, mixing `all` with specific modules)
|
|
- `503` when database connection settings are missing
|
|
|
|
---
|
|
|
|
## 9. Hybrid debugging (quick reference)
|
|
|
|
1. Start the full platform in Docker (section 7).
|
|
2. Stop the container for the service you want to debug:
|
|
```bash
|
|
docker compose -f devops/compose/docker-compose.stella-ops.yml stop <service-name>
|
|
```
|
|
3. Run that service locally from your IDE (F5 in Visual Studio, or `dotnet run`).
|
|
4. The local service uses `localhost` / `.stella-ops.local` hostnames to reach Docker-hosted infrastructure.
|
|
|
|
For detailed walkthroughs, configuration overrides, and multi-service debugging see [`docs/DEVELOPER_ONBOARDING.md`](../DEVELOPER_ONBOARDING.md).
|