394 lines
15 KiB
Markdown
394 lines
15 KiB
Markdown
# Dev Environment Setup
|
|
|
|
Actionable checklist for getting a local Stella Ops development environment running.
|
|
For hybrid debugging workflows and service-specific guides, see [`docs/DEVELOPER_ONBOARDING.md`](../DEVELOPER_ONBOARDING.md).
|
|
|
|
---
|
|
|
|
## Quick Start (automated)
|
|
|
|
Setup scripts validate prerequisites, build solutions and Docker images, and launch the full platform.
|
|
|
|
**Windows (PowerShell 7):**
|
|
|
|
```powershell
|
|
.\scripts\setup.ps1 # full setup
|
|
.\scripts\setup.ps1 -InfraOnly # infrastructure only (PostgreSQL, Valkey, SeaweedFS, Zot; Rekor is opt-in)
|
|
.\scripts\setup.ps1 -SkipBuild # skip .NET builds, build images and start platform
|
|
.\scripts\setup.ps1 -SkipImages # build .NET but skip Docker images
|
|
.\scripts\setup.ps1 -ImagesOnly # only build Docker images
|
|
```
|
|
|
|
**Linux / macOS:**
|
|
|
|
```bash
|
|
./scripts/setup.sh # full setup
|
|
./scripts/setup.sh --infra-only # infrastructure only
|
|
./scripts/setup.sh --skip-build # skip .NET builds
|
|
./scripts/setup.sh --skip-images # skip Docker image builds
|
|
./scripts/setup.sh --images-only # only build Docker images
|
|
```
|
|
|
|
The scripts will check for required tools (dotnet 10.x, node 20+, npm 10+, docker, git), warn about missing hosts file entries, copy `.env` from the example if needed, and stop repo-local host-run Stella services before the solution build so scratch bootstraps do not fail on locked `bin/Debug` outputs. Solution discovery is limited to repo-owned sources and skips generated trees such as `dist`, `coverage`, and `output`, so copied docs samples do not break scratch setup. A full setup now also performs one bounded restart pass for services that stay unhealthy after the first compose boot, waits for the first-user frontdoor bootstrap path (`/welcome`, `/envsettings.json`, OIDC discovery, `/connect/authorize`), and then runs an authenticated readiness probe that proves the topology inventory, notifications administration overrides, and promotion bootstrap routes load cleanly before the script prints success. See the manual steps below for details on each stage.
|
|
|
|
On Windows and Linux, the backend image builder now publishes each selected .NET service locally and builds the hardened runtime image from a small temporary context. That avoids repeatedly streaming the whole monorepo into Docker during scratch setup.
|
|
|
|
### Quick validation + demo seed (first-run path)
|
|
|
|
```powershell
|
|
# 1) Bring platform up quickly (reuse existing images)
|
|
.\scripts\setup.ps1 -SkipBuild -SkipImages
|
|
|
|
# 2) Validate platform health
|
|
docker compose -f devops/compose/docker-compose.stella-ops.yml ps
|
|
|
|
# 3) Preview seed work
|
|
dotnet run --project src/Cli/StellaOps.Cli/StellaOps.Cli.csproj -- `
|
|
admin seed-demo --dry-run `
|
|
--connection "Host=127.1.1.1;Port=5432;Database=stellaops_platform;Username=stellaops;Password=stellaops"
|
|
|
|
# 4) Execute demo seeding
|
|
dotnet run --project src/Cli/StellaOps.Cli/StellaOps.Cli.csproj -- `
|
|
admin seed-demo --confirm `
|
|
--connection "Host=127.1.1.1;Port=5432;Database=stellaops_platform;Username=stellaops;Password=stellaops"
|
|
```
|
|
|
|
### Known warnings vs blocking failures
|
|
|
|
| Output | Class | Meaning | Action |
|
|
|---|---|---|---|
|
|
| `GET http://127.1.1.3:8333/` returns `403` | Info | SeaweedFS S3 endpoint is live and rejecting anonymous root requests | Treat `403` as ready for the scratch setup smoke |
|
|
| `SM remote service probe failed (localhost:56080)` | Warning | Optional SM remote provider is unavailable | Ignore unless validating China SM remote crypto profile |
|
|
| `stellaops-dev-rekor restarting` without `--profile sigstore` | Warning | Optional Sigstore container from prior run | Ignore for default profile or remove stale container |
|
|
| `policy ... scheduler_exceptions_tenant_isolation already exists` | Blocking | Outdated Scheduler migration idempotency | Update code and rerun seeding |
|
|
| `POST /api/v1/admin/seed-demo` returns 500 after patching source | Blocking | Running stale platform container image | Rebuild/restart platform image |
|
|
|
|
---
|
|
|
|
## 1. Prerequisites
|
|
|
|
| Tool | Version | Verify |
|
|
|------|---------|--------|
|
|
| .NET 10 SDK | 10.0.100 (pinned in `global.json`) | `dotnet --version` |
|
|
| Node.js | ^20.19.0 \|\| ^22.12.0 \|\| ^24.0.0 | `node --version` |
|
|
| npm | >=10.2.0 | `npm --version` |
|
|
| Docker Desktop / Engine + Compose | 20.10+ | `docker --version` |
|
|
| Git | 2.30+ | `git --version` |
|
|
| PowerShell 7+ (Windows) or Bash | -- | `pwsh --version` / `bash --version` |
|
|
|
|
### Optional
|
|
|
|
- Visual Studio 2022 v17.12+ (ASP.NET and web development workload)
|
|
- VS Code + C# Dev Kit
|
|
- PostgreSQL client (`psql`, DBeaver, pgAdmin)
|
|
- `valkey-cli` or Redis Insight (Valkey is Redis-compatible)
|
|
- AWS CLI or `s3cmd` for RustFS inspection
|
|
|
|
### System requirements
|
|
|
|
- **RAM:** 16 GB minimum, 32 GB recommended
|
|
- **Disk:** 50 GB free (Docker images, volumes, build artifacts)
|
|
- **CPU:** 4 cores minimum, 8 cores recommended
|
|
|
|
---
|
|
|
|
## 1b. Runtime data assets
|
|
|
|
Some services need files that `dotnet build` does not produce. For local dev the
|
|
most impactful one is the ONNX embedding model used by AdvisoryAI for semantic
|
|
search. Without it the encoder falls back to a reduced-quality projection.
|
|
|
|
```bash
|
|
# Download the embedding model (~80 MB) — run once after cloning
|
|
./devops/runtime-assets/acquire.sh --models
|
|
|
|
# Optional: JDK + Ghidra for binary analysis (~1.6 GB)
|
|
./devops/runtime-assets/acquire.sh --ghidra
|
|
|
|
# Verify all assets
|
|
./devops/runtime-assets/acquire.sh --verify
|
|
```
|
|
|
|
Full inventory, Docker volume mounts, and air-gap packaging:
|
|
[`devops/runtime-assets/README.md`](../../devops/runtime-assets/README.md).
|
|
|
|
---
|
|
|
|
## 2. Hosts file setup
|
|
|
|
Each service binds to a unique loopback IP so all can use ports 443/80 without collisions.
|
|
Full details: [`docs/technical/architecture/port-registry.md`](../technical/architecture/port-registry.md).
|
|
|
|
### Automated (recommended)
|
|
|
|
The setup scripts (`scripts/setup.ps1` / `scripts/setup.sh`) will detect missing entries and offer to install them automatically.
|
|
|
|
### Manual
|
|
|
|
Append the contents of [`devops/compose/hosts.stellaops.local`](../../devops/compose/hosts.stellaops.local) to your hosts file:
|
|
|
|
- **Windows:** Run an elevated PowerShell and run:
|
|
```powershell
|
|
Get-Content devops\compose\hosts.stellaops.local | Add-Content C:\Windows\System32\drivers\etc\hosts
|
|
```
|
|
- **Linux / macOS:**
|
|
```bash
|
|
sudo sh -c 'cat devops/compose/hosts.stellaops.local >> /etc/hosts'
|
|
```
|
|
|
|
The file contains ~50 entries mapping services to unique loopback IPs (127.1.0.1 through 127.1.1.5).
|
|
See the file for the full list.
|
|
|
|
---
|
|
|
|
## 3. Start infrastructure (Docker)
|
|
|
|
```bash
|
|
cd devops/compose
|
|
cp env/stellaops.env.example .env # works out of the box; change POSTGRES_PASSWORD for production
|
|
docker compose -f docker-compose.dev.yml up -d
|
|
docker compose -f docker-compose.dev.yml ps
|
|
```
|
|
|
|
### Verify infrastructure
|
|
|
|
```bash
|
|
# PostgreSQL
|
|
psql -h db.stella-ops.local -U stellaops -d stellaops_dev -c "SELECT 1"
|
|
|
|
# Valkey
|
|
valkey-cli -h cache.stella-ops.local ping
|
|
|
|
# SeaweedFS S3 root returns 403 when unauthenticated; that still proves readiness
|
|
curl -I http://s3.stella-ops.local/
|
|
|
|
# Zot OCI registry
|
|
curl -I http://registry.stella-ops.local/v2/
|
|
```
|
|
|
|
Infrastructure versions (from `docker-compose.dev.yml`):
|
|
|
|
| Service | Version | Hostname | Port |
|
|
|---------|---------|----------|------|
|
|
| PostgreSQL | 18.1 | `db.stella-ops.local` | 5432 |
|
|
| Valkey | 9.0.1 | `cache.stella-ops.local` | 6379 |
|
|
| SeaweedFS (S3) | -- | `s3.stella-ops.local` | 8333 |
|
|
| Rekor v2 (optional `sigstore` profile) | -- | `rekor.stella-ops.local` | 3322 |
|
|
| Zot (OCI registry) | v2.1.3 | `registry.stella-ops.local` | 80 |
|
|
|
|
---
|
|
|
|
## 4. Build .NET modules
|
|
|
|
The codebase uses a **module-first** approach -- there is no root solution file used for builds. Each module has its own `.sln` under `src/<Module>/`.
|
|
|
|
### Single module
|
|
|
|
```powershell
|
|
dotnet build src\Scanner\StellaOps.Scanner.sln
|
|
dotnet test src\Scanner\StellaOps.Scanner.sln
|
|
```
|
|
|
|
### All modules
|
|
|
|
```powershell
|
|
# Windows (PowerShell 7)
|
|
.\scripts\build-all-solutions.ps1
|
|
|
|
# Stop repo-local host-run Stella services first if a prior debug session left binaries locked
|
|
.\scripts\build-all-solutions.ps1 -StopRepoHostProcesses
|
|
|
|
# With tests
|
|
.\scripts\build-all-solutions.ps1 -Test
|
|
|
|
# Linux / macOS
|
|
./scripts/build-all-solutions.sh
|
|
|
|
# With tests
|
|
./scripts/build-all-solutions.sh --test
|
|
```
|
|
|
|
### Targeted backend image rebuilds
|
|
|
|
```powershell
|
|
.\devops\docker\build-all.ps1 -Services notify-web,advisory-ai-web
|
|
```
|
|
|
|
```bash
|
|
SERVICES=notify-web,advisory-ai-web ./devops/docker/build-all.sh
|
|
```
|
|
|
|
Use this after scoped backend changes instead of re-running the full image matrix.
|
|
|
|
### Module solution index
|
|
|
|
See [`docs/dev/SOLUTION_BUILD_GUIDE.md`](SOLUTION_BUILD_GUIDE.md) for the authoritative list. Current modules (39):
|
|
|
|
| Module | Solution path |
|
|
|--------|---------------|
|
|
| AdvisoryAI | `src/AdvisoryAI/StellaOps.AdvisoryAI.sln` |
|
|
| AirGap | `src/AirGap/StellaOps.AirGap.sln` |
|
|
| Aoc | `src/Aoc/StellaOps.Aoc.sln` |
|
|
| Attestor | `src/Attestor/StellaOps.Attestor.sln` |
|
|
| Authority | `src/Authority/StellaOps.Authority.sln` |
|
|
| Bench | `src/Bench/StellaOps.Bench.sln` |
|
|
| BinaryIndex | `src/BinaryIndex/StellaOps.BinaryIndex.sln` |
|
|
| Cartographer (absorbed into Scanner) | `src/Scanner/StellaOps.Scanner.sln` |
|
|
| Cli | `src/Cli/StellaOps.Cli.sln` |
|
|
| Concelier | `src/Concelier/StellaOps.Concelier.sln` |
|
|
| EvidenceLocker | `src/EvidenceLocker/StellaOps.EvidenceLocker.sln` |
|
|
| Excititor | `src/Excititor/StellaOps.Excititor.sln` |
|
|
| ExportCenter | `src/ExportCenter/StellaOps.ExportCenter.sln` |
|
|
| Feedser | `src/Feedser/StellaOps.Feedser.sln` |
|
|
| Findings | `src/Findings/StellaOps.Findings.sln` |
|
|
| Router (Gateway) | `src/Router/StellaOps.Router.sln` |
|
|
| Graph | `src/Graph/StellaOps.Graph.sln` |
|
|
| IssuerDirectory | `src/IssuerDirectory/StellaOps.IssuerDirectory.sln` |
|
|
| Notifier | `src/Notifier/StellaOps.Notifier.sln` |
|
|
| Notify | `src/Notify/StellaOps.Notify.sln` |
|
|
| Orchestrator | `src/JobEngine/StellaOps.JobEngine.sln` |
|
|
| PacksRegistry | `src/PacksRegistry/StellaOps.PacksRegistry.sln` |
|
|
| Policy | `src/Policy/StellaOps.Policy.sln` |
|
|
| ReachGraph | `src/ReachGraph/StellaOps.ReachGraph.sln` |
|
|
| Registry | `src/Registry/StellaOps.Registry.sln` |
|
|
| Replay | `src/Replay/StellaOps.Replay.sln` |
|
|
| RiskEngine | `src/Findings/StellaOps.Findings.sln` (consolidated into Findings) |
|
|
| Router | `src/Router/StellaOps.Router.sln` |
|
|
| SbomService | `src/SbomService/StellaOps.SbomService.sln` |
|
|
| Scanner | `src/Scanner/StellaOps.Scanner.sln` |
|
|
| Scheduler | `src/Scheduler/StellaOps.Scheduler.sln` |
|
|
| Signer | `src/Signer/StellaOps.Signer.sln` |
|
|
| Signals | `src/Signals/StellaOps.Signals.sln` |
|
|
| SmRemote | `src/SmRemote/StellaOps.SmRemote.sln` |
|
|
| TaskRunner | `src/TaskRunner/StellaOps.TaskRunner.sln` |
|
|
| Telemetry | `src/Telemetry/StellaOps.Telemetry.sln` |
|
|
| Timeline (incl. TimelineIndexer) | `src/Timeline/` (no standalone sln; use root `StellaOps.sln`) |
|
|
| Tools | `src/Tools/StellaOps.Tools.sln` |
|
|
| VexHub | `src/VexHub/StellaOps.VexHub.sln` |
|
|
| VexLens | `src/VexLens/StellaOps.VexLens.sln` |
|
|
| VulnExplorer | `src/Findings/StellaOps.Findings.sln` (consolidated into Findings) |
|
|
| Zastava | `src/Zastava/StellaOps.Zastava.sln` |
|
|
|
|
---
|
|
|
|
## 5. Build Angular frontend
|
|
|
|
```bash
|
|
cd src/Web/StellaOps.Web
|
|
npm ci --prefer-offline --no-audit --no-fund
|
|
npm run start # dev server -> https://stella-ops.local
|
|
npm run build # production build
|
|
npm run test # unit tests (Vitest)
|
|
npm run test:e2e # Playwright E2E
|
|
```
|
|
|
|
Additional scripts:
|
|
|
|
| Command | Purpose |
|
|
|---------|---------|
|
|
| `npm run storybook` | Launch Storybook component explorer |
|
|
| `npm run analyze` | Bundle size visualization (esbuild-visualizer) |
|
|
| `npm run test:a11y` | Accessibility smoke tests |
|
|
|
|
---
|
|
|
|
## 6. Build Docker images
|
|
|
|
### Option A: Build all services (matrix-driven)
|
|
|
|
```bash
|
|
cd devops/docker
|
|
./build-all.sh
|
|
```
|
|
|
|
Uses `services-matrix.env` and `Dockerfile.hardened.template` for .NET services, `Dockerfile.console` for Angular.
|
|
|
|
### Option B: Build a single .NET service
|
|
|
|
```bash
|
|
docker build -f devops/docker/Dockerfile.hardened.template . \
|
|
--build-arg SDK_IMAGE=mcr.microsoft.com/dotnet/sdk:10.0-bookworm-slim \
|
|
--build-arg RUNTIME_IMAGE=mcr.microsoft.com/dotnet/aspnet:10.0-bookworm-slim \
|
|
--build-arg APP_PROJECT=src/Scanner/StellaOps.Scanner.WebService/StellaOps.Scanner.WebService.csproj \
|
|
--build-arg APP_BINARY=StellaOps.Scanner.WebService \
|
|
--build-arg APP_PORT=8080 \
|
|
-t stellaops/scanner-web:dev
|
|
```
|
|
|
|
### Option C: Build the Angular console image
|
|
|
|
```bash
|
|
docker build -f devops/docker/Dockerfile.console . \
|
|
--build-arg APP_DIR=src/Web/StellaOps.Web \
|
|
-t stellaops/console:dev
|
|
```
|
|
|
|
### Release-quality builds (distroless)
|
|
|
|
Release Dockerfiles live under `devops/release/docker/`:
|
|
|
|
- `Dockerfile.dotnet-service` -- .NET services
|
|
- `Dockerfile.angular-ui` -- Angular console
|
|
|
|
Component manifest: `devops/release/components.json`.
|
|
|
|
---
|
|
|
|
## 7. Run the full platform
|
|
|
|
```bash
|
|
# Core services
|
|
docker compose -f devops/compose/docker-compose.stella-ops.yml up -d
|
|
|
|
# With Sigstore transparency log
|
|
docker compose -f devops/compose/docker-compose.stella-ops.yml --profile sigstore up -d
|
|
|
|
# With telemetry stack
|
|
docker compose -f devops/compose/docker-compose.stella-ops.yml \
|
|
-f devops/compose/docker-compose.telemetry.yml up -d
|
|
```
|
|
|
|
Verify:
|
|
|
|
```bash
|
|
docker compose -f devops/compose/docker-compose.stella-ops.yml ps
|
|
```
|
|
|
|
---
|
|
|
|
## 8. Seed demo data and verify endpoint errors
|
|
|
|
Use the CLI seeder for local bootstraps and demo datasets:
|
|
|
|
```powershell
|
|
# dry-run
|
|
dotnet run --project src/Cli/StellaOps.Cli/StellaOps.Cli.csproj -- `
|
|
admin seed-demo --dry-run `
|
|
--connection "Host=127.1.1.1;Port=5432;Database=stellaops_platform;Username=stellaops;Password=stellaops"
|
|
|
|
# execute
|
|
dotnet run --project src/Cli/StellaOps.Cli/StellaOps.Cli.csproj -- `
|
|
admin seed-demo --confirm `
|
|
--connection "Host=127.1.1.1;Port=5432;Database=stellaops_platform;Username=stellaops;Password=stellaops"
|
|
```
|
|
|
|
Seed API behavior (`POST /api/v1/admin/seed-demo`) now returns deterministic non-500 errors for expected failure modes:
|
|
- `401/403` for auth policy failures (`platform.setup.admin`)
|
|
- `503` when demo seeding is disabled (`STELLAOPS_ENABLE_DEMO_SEED=false`)
|
|
- `400` for invalid module filters (for example, mixing `all` with specific modules)
|
|
- `503` when database connection settings are missing
|
|
|
|
---
|
|
|
|
## 9. Hybrid debugging (quick reference)
|
|
|
|
1. Start the full platform in Docker (section 7).
|
|
2. Stop the container for the service you want to debug:
|
|
```bash
|
|
docker compose -f devops/compose/docker-compose.stella-ops.yml stop <service-name>
|
|
```
|
|
3. Run that service locally from your IDE (F5 in Visual Studio, or `dotnet run`).
|
|
4. The local service uses `localhost` / `.stella-ops.local` hostnames to reach Docker-hosted infrastructure.
|
|
|
|
For detailed walkthroughs, configuration overrides, and multi-service debugging see [`docs/DEVELOPER_ONBOARDING.md`](../DEVELOPER_ONBOARDING.md).
|