4d932cc1ba
Some checks failed
Docs CI / lint-and-preview (push) Has been cancelled
- Introduced RunnerBackgroundService to handle execution of runner segments. - Added RunnerExecutionService for processing segments and aggregating results. - Implemented PlannerQueueDispatchService to manage dispatching of planner messages. - Created PlannerQueueDispatcherBackgroundService for leasing and processing planner queue messages. - Developed ScannerReportClient for interacting with the scanner service. - Enhanced observability with SchedulerWorkerMetrics for tracking planner and runner performance. - Added comprehensive documentation for the new runner execution pipeline and observability metrics. - Implemented event emission for rescan activity and scanner report readiness.
95 lines
5.7 KiB
Markdown
95 lines
5.7 KiB
Markdown
# Stella Ops Compose Profiles
|
||
|
||
These Compose bundles ship the minimum services required to exercise the scanner pipeline plus control-plane dependencies. Every profile is pinned to immutable image digests sourced from `deploy/releases/*.yaml` and is linted via `docker compose config` in CI.
|
||
|
||
## Layout
|
||
|
||
| Path | Purpose |
|
||
| ---- | ------- |
|
||
| `docker-compose.dev.yaml` | Edge/nightly stack tuned for laptops and iterative work. |
|
||
| `docker-compose.stage.yaml` | Stable channel stack mirroring pre-production clusters. |
|
||
| `docker-compose.prod.yaml` | Production cutover stack with front-door network hand-off and Notify events enabled. |
|
||
| `docker-compose.airgap.yaml` | Stable stack with air-gapped defaults (no outbound hostnames). |
|
||
| `docker-compose.mirror.yaml` | Managed mirror topology for `*.stella-ops.org` distribution (Concelier + Excititor + CDN gateway). |
|
||
| `docker-compose.telemetry.yaml` | Optional OpenTelemetry collector overlay (mutual TLS, OTLP ingest endpoints). |
|
||
| `docker-compose.telemetry-storage.yaml` | Prometheus/Tempo/Loki storage overlay with multi-tenant defaults. |
|
||
| `env/*.env.example` | Seed `.env` files that document required secrets and ports per profile. |
|
||
|
||
## Usage
|
||
|
||
```bash
|
||
cp env/dev.env.example dev.env
|
||
docker compose --env-file dev.env -f docker-compose.dev.yaml config
|
||
docker compose --env-file dev.env -f docker-compose.dev.yaml up -d
|
||
```
|
||
|
||
The stage and airgap variants behave the same way—swap the file names accordingly. All profiles expose 443/8443 for the UI and REST APIs, and they share a `stellaops` Docker network scoped to the compose project.
|
||
|
||
> **Graph Explorer reminder:** If you enable Cartographer or Graph API containers alongside these profiles, update `etc/authority.yaml` so the `cartographer-service` client is marked with `properties.serviceIdentity: "cartographer"` and carries a tenant hint. The Authority host now refuses `graph:write` tokens without that marker, so apply the configuration change before rolling out the updated images.
|
||
|
||
### Telemetry collector overlay
|
||
|
||
The OpenTelemetry collector overlay is optional and can be layered on top of any profile:
|
||
|
||
```bash
|
||
./ops/devops/telemetry/generate_dev_tls.sh
|
||
docker compose -f docker-compose.telemetry.yaml up -d
|
||
python ../../ops/devops/telemetry/smoke_otel_collector.py --host localhost
|
||
docker compose -f docker-compose.telemetry-storage.yaml up -d
|
||
```
|
||
|
||
The generator script creates a development CA plus server/client certificates under
|
||
`deploy/telemetry/certs/`. The smoke test sends OTLP/HTTP payloads using the generated
|
||
client certificate and asserts the collector reports accepted traces, metrics, and logs.
|
||
The storage overlay starts Prometheus, Tempo, and Loki with multitenancy enabled so you
|
||
can validate the end-to-end pipeline before promoting changes to staging. Adjust the
|
||
configs in `deploy/telemetry/storage/` before running in production.
|
||
Mount the same certificates when running workloads so the collector can enforce mutual TLS.
|
||
|
||
For production cutovers copy `env/prod.env.example` to `prod.env`, update the secret placeholders, and create the external network expected by the profile:
|
||
|
||
```bash
|
||
docker network create stellaops_frontdoor
|
||
docker compose --env-file prod.env -f docker-compose.prod.yaml config
|
||
```
|
||
|
||
### Scanner event stream settings
|
||
|
||
Scanner WebService can emit signed `scanner.report.*` events to Redis Streams when `SCANNER__EVENTS__ENABLED=true`. Each profile ships environment placeholders you can override in the `.env` file:
|
||
|
||
- `SCANNER_EVENTS_ENABLED` – toggle emission on/off (defaults to `false`).
|
||
- `SCANNER_EVENTS_DRIVER` – currently only `redis` is supported.
|
||
- `SCANNER_EVENTS_DSN` – Redis endpoint; leave blank to reuse the queue DSN when it uses `redis://`.
|
||
- `SCANNER_EVENTS_STREAM` – stream name (`stella.events` by default).
|
||
- `SCANNER_EVENTS_PUBLISH_TIMEOUT_SECONDS` – per-publish timeout window (defaults to `5`).
|
||
- `SCANNER_EVENTS_MAX_STREAM_LENGTH` – max stream length before Redis trims entries (defaults to `10000`).
|
||
|
||
Helm values mirror the same knobs under each service’s `env` map (see `deploy/helm/stellaops/values-*.yaml`).
|
||
|
||
### Scheduler worker configuration
|
||
|
||
Every Compose profile now provisions the `scheduler-worker` container (backed by the
|
||
`StellaOps.Scheduler.Worker.Host` entrypoint). The environment placeholders exposed
|
||
in the `.env` samples match the options bound by `AddSchedulerWorker`:
|
||
|
||
- `SCHEDULER_QUEUE_KIND` – queue transport (`Nats` or `Redis`).
|
||
- `SCHEDULER_QUEUE_NATS_URL` – NATS connection string used by planner/runner consumers.
|
||
- `SCHEDULER_STORAGE_DATABASE` – MongoDB database name for scheduler state.
|
||
- `SCHEDULER_SCANNER_BASEADDRESS` – base URL the runner uses when invoking Scanner’s
|
||
`/api/v1/reports` (defaults to the in-cluster `http://scanner-web:8444`).
|
||
|
||
Helm deployments inherit the same defaults from `services.scheduler-worker.env` in
|
||
`values.yaml`; override them per environment as needed.
|
||
|
||
### Front-door network hand-off
|
||
|
||
`docker-compose.prod.yaml` adds a `frontdoor` network so operators can attach Traefik, Envoy, or an on-prem load balancer that terminates TLS. Override `FRONTDOOR_NETWORK` in `prod.env` if your reverse proxy uses a different bridge name. Attach only the externally reachable services (Authority, Signer, Attestor, Concelier, Scanner Web, Notify Web, UI) to that network—internal infrastructure (Mongo, MinIO, RustFS, NATS) stays on the private `stellaops` network.
|
||
|
||
### Updating to a new release
|
||
|
||
1. Import the new manifest into `deploy/releases/` (see `deploy/README.md`).
|
||
2. Update image digests in the relevant Compose file(s).
|
||
3. Re-run `docker compose config` to confirm the bundle is deterministic.
|
||
|
||
Keep digests synchronized between Compose, Helm, and the release manifest to preserve reproducibility guarantees. `deploy/tools/validate-profiles.sh` performs a quick audit.
|