stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 2 Releases Wiki Activity
Files
df07bcfd24c20c229d165aeba502c1a9b24a7480
git.stella-ops.org/docs/integrations/LOCAL_SERVICES.md
master 50abd2137f Update docs, sprint plans, and compose configuration 
Add 12 new sprint files (Integrations, Graph, JobEngine, FE, Router,
AdvisoryAI), archive completed scheduler UI sprint, update module
architecture docs (router, graph, jobengine, web, integrations),
and add Gitea entrypoint script for local dev.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 08:53:50 +03:00

413 lines
15 KiB
Markdown
Raw Blame History

# Local Integration Services
This guide covers the third-party services available for local integration testing with Stella Ops.
## Architecture Overview
```
stellaops network
+------------------------------------------------------------------+
| |
| STELLA OPS CORE INTEGRATION SERVICES |
| (docker-compose.stella-ops.yml) (docker-compose.integrations.yml)|
| |
| router-gateway ─────────> gitea (SCM) 127.1.2.1:3000 |
| concelier ─────────> jenkins (CI/CD) 127.1.2.2:8080 |
| integrations ─────────> nexus (Registry) 127.1.2.3:8081 |
| scanner ─────────> vault (Secrets) 127.1.2.4:8200 |
| evidence-locker ────────> docker-reg (Registry) 127.1.2.5:5000 |
| airgap-controller ──────> minio (S3) 127.1.2.6:9000 |
| gitlab [heavy](All-in-1) 127.1.2.7:8929 |
| |
| MOCK FIXTURES (docker-compose.integration-fixtures.yml) |
| harbor-fixture (Registry mock) 127.1.1.6:80 |
| github-app-fixture (SCM mock) 127.1.1.7:80 |
| advisory-fixture (Advisory mock) 127.1.1.8:80 |
+------------------------------------------------------------------+
```
## Quick Start
### Prerequisites
- Docker Desktop with 8 GB+ RAM allocated
- The main Stella Ops stack running (`docker-compose.stella-ops.yml`)
- Hosts file entries (see below)
### 1. Add hosts file entries
Add to `C:\Windows\System32\drivers\etc\hosts`:
```
127.1.2.1 gitea.stella-ops.local
127.1.2.2 jenkins.stella-ops.local
127.1.2.3 nexus.stella-ops.local
127.1.2.4 vault.stella-ops.local
127.1.2.5 registry.stella-ops.local
127.1.2.6 minio.stella-ops.local
127.1.2.7 gitlab.stella-ops.local
127.1.2.8 consul.stella-ops.local
```
### 2. Start services
```bash
cd devops/compose
# Start the default low-idle services (recommended)
docker compose -f docker-compose.integrations.yml up -d
# Or start specific services only
docker compose -f docker-compose.integrations.yml up -d gitea vault jenkins
# Start WITH mock fixtures (for full integration testing)
docker compose \
-f docker-compose.integrations.yml \
-f docker-compose.integration-fixtures.yml \
up -d
# Start Consul only when validating the Consul connector
docker compose -f docker-compose.integrations.yml --profile consul up -d consul
# Start GitLab CE (heavy, 4 GB+ RAM, ~3 min startup)
docker compose -f docker-compose.integrations.yml --profile heavy up -d gitlab
# Re-enable GitLab registry/package surfaces for registry-specific tests
GITLAB_ENABLE_REGISTRY=true GITLAB_ENABLE_PACKAGES=true \
docker compose -f docker-compose.integrations.yml --profile heavy up -d gitlab
```
### 3. Verify services
```bash
# Quick health check for all services
docker compose -f docker-compose.integrations.yml ps
# Gitea is only complete once the container is healthy
docker compose -f docker-compose.integrations.yml ps gitea
```
`docker-compose.testing.yml` is the separate infrastructure-test lane. It starts `postgres-test`, `valkey-test`, mocks, and an isolated Gitea profile on different ports; it does not start Consul or GitLab.
---
## Service Reference
### Gitea (SCM)
| Property | Value |
|----------|-------|
| URL | http://gitea.stella-ops.local:3000 |
| API | http://gitea.stella-ops.local:3000/api/v1 |
| SSH | gitea.stella-ops.local:2222 |
| Admin | `stellaops` / `Stella2026!` on fresh volumes |
| Bootstrap | Container entrypoint seeds the repo root and first admin before health goes green |
| Swagger | http://gitea.stella-ops.local:3000/api/swagger |
| Integration type | SCM (Gitea provider) |
| Docker DNS | `gitea.stella-ops.local` |
**Stella Ops integration config:**
- Endpoint: `http://gitea.stella-ops.local:3000`
- AuthRef: `authref://vault/gitea#api-token`
- Organization: *(your Gitea org name)*
**Create an API token:**
1. Log in to Gitea as `stellaops` / `Stella2026!` on a fresh volume, or use the existing admin user if this environment was already initialized.
2. Settings > Applications > Generate Token
3. Store in Vault at `secret/gitea` with key `api-token`
> The previous local-service flow was contradictory: the compose profile marked Gitea as install-locked while the docs still described a manual first-login admin creation path. The compose bootstrap now makes the service deterministic and leaves only PAT creation as a manual step.
---
### Jenkins (CI/CD)
| Property | Value |
|----------|-------|
| URL | http://jenkins.stella-ops.local:8080 |
| API | http://jenkins.stella-ops.local:8080/api/json |
| Admin | Setup wizard disabled; create user via script console |
| Agent port | 127.1.2.2:50000 |
| Integration type | CI/CD (Jenkins provider) |
| Docker DNS | `jenkins.stella-ops.local` |
**Stella Ops integration config:**
- Endpoint: `http://jenkins.stella-ops.local:8080`
- AuthRef: `authref://vault/jenkins#api-token`
**Create an API token:**
1. Open Jenkins > Manage Jenkins > Users > admin > Configure
2. Add API Token
3. Store in Vault at `secret/jenkins` with key `api-token`
---
### Nexus (Repository Manager)
| Property | Value |
|----------|-------|
| URL | http://nexus.stella-ops.local:8081 |
| API | http://nexus.stella-ops.local:8081/service/rest/v1/status |
| Docker hosted | nexus.stella-ops.local:8082 |
| Docker proxy | nexus.stella-ops.local:8083 |
| Admin | admin / *(see `/nexus-data/admin.password` on first run)* |
| Integration type | Registry (Nexus provider) |
| Docker DNS | `nexus.stella-ops.local` |
**Get initial admin password:**
```bash
docker exec stellaops-nexus cat /nexus-data/admin.password
```
**Stella Ops integration config:**
- Endpoint: `http://nexus.stella-ops.local:8081`
- AuthRef: `authref://vault/nexus#admin-password`
**Setup Docker hosted repository:**
1. Login to Nexus UI
2. Server Administration > Repositories > Create > docker (hosted)
3. HTTP port: 8082, Allow redeploy: true
4. Create a docker (proxy) repository pointing to Docker Hub, HTTP port: 8083
---
### HashiCorp Vault (Secrets)
| Property | Value |
|----------|-------|
| URL | http://vault.stella-ops.local:8200 |
| API | http://vault.stella-ops.local:8200/v1/sys/health |
| Root token | `stellaops-dev-root-token-2026` |
| Mode | Dev server (in-memory, unsealed) |
| Integration type | Secrets (Vault provider) |
| Docker DNS | `vault.stella-ops.local` |
**Stella Ops integration config:**
- Endpoint: `http://vault.stella-ops.local:8200`
- AuthRef: (Vault is the auth provider itself)
**Store integration credentials in Vault:**
```bash
# Enable KV v2 engine (already enabled in dev mode at secret/)
export VAULT_ADDR=http://vault.stella-ops.local:8200
export VAULT_TOKEN=stellaops-dev-root-token-2026
# Store Harbor credentials
vault kv put secret/harbor robot-account="harbor-robot-token"
# Store GitHub App credentials
vault kv put secret/github app-private-key="-----BEGIN RSA PRIVATE KEY-----..."
# Store Gitea API token
vault kv put secret/gitea api-token="your-gitea-token"
# Store Jenkins API token
vault kv put secret/jenkins api-token="your-jenkins-token"
# Store Nexus admin password
vault kv put secret/nexus admin-password="your-nexus-password"
```
---
### Consul (Optional KV / Settings Store)
| Property | Value |
|----------|-------|
| URL | http://consul.stella-ops.local:8500 |
| API | http://consul.stella-ops.local:8500/v1/status/leader |
| Auth | None (single-node local server) |
| Start mode | `--profile consul` only |
| Integration type | Settings / KV (`Consul` provider) |
| Docker DNS | `consul.stella-ops.local` |
**Start Consul only when needed:**
```bash
docker compose -f docker-compose.integrations.yml --profile consul up -d consul
```
**Why opt-in:** even in its lower-idle local mode, Consul is still an extra control-plane service that most local connector checks do not need. The default integration lane keeps it off unless you are explicitly validating the Consul connector.
**Runtime mode:** the local compose profile now runs Consul as a persistent single-node server with the UI enabled instead of `agent -dev`. That preserves the HTTP KV surface while materially lowering idle CPU.
**Stella Ops integration config:**
- Endpoint: `http://consul.stella-ops.local:8500`
- AuthRef: *(none required in local mode)*
---
### Docker Registry (OCI v2)
| Property | Value |
|----------|-------|
| URL | http://registry.stella-ops.local:5000 |
| API | http://registry.stella-ops.local:5000/v2/ |
| Auth | None (open dev registry) |
| Integration type | Registry (generic OCI) |
| Docker DNS | `registry.stella-ops.local` |
**Push a test image:**
```bash
docker tag alpine:latest registry.stella-ops.local:5000/test/alpine:latest
docker push registry.stella-ops.local:5000/test/alpine:latest
# List repositories
curl http://registry.stella-ops.local:5000/v2/_catalog
```
**Stella Ops integration config:**
- Endpoint: `http://registry.stella-ops.local:5000`
- AuthRef: *(none required for dev)*
---
### MinIO (S3 Storage)
| Property | Value |
|----------|-------|
| Console | http://minio.stella-ops.local:9001 |
| S3 API | http://minio.stella-ops.local:9000 |
| Access key | `stellaops` |
| Secret key | `Stella2026!` |
| Integration type | Object Storage (`S3Compatible` provider) |
| Docker DNS | `minio.stella-ops.local` |
**Create buckets for Stella Ops:**
```bash
# Install mc CLI
docker exec stellaops-minio mc alias set local http://localhost:9000 stellaops Stella2026!
# Create buckets
docker exec stellaops-minio mc mb local/evidence-locker
docker exec stellaops-minio mc mb local/airgap-bundles
docker exec stellaops-minio mc mb local/scan-results
docker exec stellaops-minio mc mb local/sbom-archive
```
**Stella Ops integration config:**
- Endpoint: `http://minio.stella-ops.local:9000`
- Type: `ObjectStorage`
- Provider: `S3Compatible`
- AuthRef: optional for the default local health probe
---
### GitLab CE (Heavy, Optional)
| Property | Value |
|----------|-------|
| URL | http://gitlab.stella-ops.local:8929 |
| Admin | root / `Stella2026!` |
| SSH | gitlab.stella-ops.local:2224 |
| Container Registry | gitlab.stella-ops.local:5050 (`GITLAB_ENABLE_REGISTRY=true` only) |
| RAM required | 4 GB+ |
| Startup time | ~3-5 minutes |
| Integration type | SCM + CI/CD + Registry |
| Docker DNS | `gitlab.stella-ops.local` |
**Start GitLab (uses `heavy` profile):**
```bash
docker compose -f docker-compose.integrations.yml --profile heavy up -d gitlab
```
**Default local tuning:**
- SCM/API coverage stays available.
- Registry and package surfaces are disabled by default to reduce idle CPU.
- Puma and Sidekiq run in reduced-concurrency mode for local connector checks.
**Enable registry/package coverage explicitly when needed:**
```bash
GITLAB_ENABLE_REGISTRY=true GITLAB_ENABLE_PACKAGES=true \
docker compose -f docker-compose.integrations.yml --profile heavy up -d gitlab
```
**Stella Ops integration config (SCM):**
- Endpoint: `http://gitlab.stella-ops.local:8929`
- AuthRef: `authref://vault/gitlab#access-token`
---
## Mock Fixtures
In addition to real services, lightweight nginx-based fixtures provide deterministic mock APIs for UI testing.
| Fixture | Mocks | Address | Compose file |
|---------|-------|---------|-------------|
| harbor-fixture | Harbor v2 API | 127.1.1.6:80 | docker-compose.integration-fixtures.yml |
| github-app-fixture | GitHub App API | 127.1.1.7:80 | docker-compose.integration-fixtures.yml |
| advisory-fixture | CERT-In, FSTEC, VEX Hub, StellaOps Mirror, etc. | 127.1.1.8:80 | docker-compose.integration-fixtures.yml |
```bash
# Start fixtures only
docker compose -f docker-compose.integration-fixtures.yml up -d
```
---
## IP Address Map
| IP | Service | Port(s) |
|----|---------|---------|
| 127.1.0.1 | stella-ops.local (gateway) | 443 |
| 127.1.0.4 | authority (OIDC) | 80 |
| 127.1.1.1 | postgres | 5432 |
| 127.1.1.2 | valkey | 6379 |
| 127.1.1.6 | harbor-fixture | 80 |
| 127.1.1.7 | github-app-fixture | 80 |
| 127.1.1.8 | advisory-fixture | 80 |
| 127.1.2.1 | gitea | 3000, 2222 |
| 127.1.2.2 | jenkins | 8080, 50000 |
| 127.1.2.3 | nexus | 8081, 8082, 8083 |
| 127.1.2.4 | vault | 8200 |
| 127.1.2.5 | docker-registry | 5000 |
| 127.1.2.6 | minio | 9000, 9001 |
| 127.1.2.7 | gitlab (heavy) | 8929, 2224, 5050 |
---
## Volumes
All service data persists in named Docker volumes. To reset a service:
```bash
# Stop and remove a specific service + its volume
docker compose -f docker-compose.integrations.yml down -v nexus
docker volume rm stellaops-nexus-data
# Reset ALL integration services
docker compose -f docker-compose.integrations.yml down -v
```
---
## Integration Matrix
| Stella Ops Category | Provider | Local Service | Status |
|---------------------|----------|---------------|--------|
| **Registry** | Harbor | harbor-fixture (mock) | Ready |
| **Registry** | Docker Hub / OCI | docker-registry | Ready |
| **Registry** | Nexus | nexus | Ready |
| **Registry** | GitLab Registry | gitlab (heavy) | Ready when `GITLAB_ENABLE_REGISTRY=true` |
| **SCM** | GitHub App | github-app-fixture (mock) | Ready |
| **SCM** | Gitea | gitea | Ready |
| **SCM** | GitLab Server | gitlab (heavy) | Ready with Vault-backed PAT |
| **CI/CD** | Jenkins | jenkins | Ready |
| **CI/CD** | GitLab CI | gitlab (heavy) | Ready with reduced local concurrency |
| **Secrets** | Vault | vault | Ready |
| **Secrets** | Consul | consul | Opt-in (`--profile consul`) |
| **Runtime Host** | eBPF Agent | runtime-host-fixture (mock) | Ready |
| **Feed Mirror** | StellaOps / NVD / OSV mirror | concelier | Ready |
| **Storage** | S3-compatible (MinIO) | minio | Ready |
| **Advisory & VEX** | 74 sources | advisory-fixture + live | 74/74 healthy |
> **Current provider list:** the local Integrations service currently reports connector plugins for Harbor,
> Docker Registry, GitLab Container Registry, Nexus, GitHub App, Gitea, GitLab Server,
> GitLab CI, Jenkins, Vault, Consul, eBPF Agent, the `S3Compatible` object-storage provider, the feed mirror providers
> (`StellaOpsMirror`, `NvdMirror`, `OsvMirror`), and the test-only InMemory provider.
>
> **Storage note:** the `S3Compatible` connector defaults to probing `/minio/health/live`
> when the configured endpoint is the service root, which matches the local MinIO fixture.
>
> **Auth caveat:** several connector plugins validate public health/version endpoints only. A green connection
> test proves reachability and the plugin wiring, but it does not guarantee that privileged API operations are
> fully configured unless you also provision the corresponding secret material in Vault.
Reference in New Issue View Git Blame Copy Permalink