- Introduced RunnerBackgroundService to handle execution of runner segments. - Added RunnerExecutionService for processing segments and aggregating results. - Implemented PlannerQueueDispatchService to manage dispatching of planner messages. - Created PlannerQueueDispatcherBackgroundService for leasing and processing planner queue messages. - Developed ScannerReportClient for interacting with the scanner service. - Enhanced observability with SchedulerWorkerMetrics for tracking planner and runner performance. - Added comprehensive documentation for the new runner execution pipeline and observability metrics. - Implemented event emission for rescan activity and scanner report readiness.
5.7 KiB
Stella Ops Compose Profiles
These Compose bundles ship the minimum services required to exercise the scanner pipeline plus control-plane dependencies. Every profile is pinned to immutable image digests sourced from deploy/releases/*.yaml and is linted via docker compose config in CI.
Layout
| Path | Purpose |
|---|---|
docker-compose.dev.yaml |
Edge/nightly stack tuned for laptops and iterative work. |
docker-compose.stage.yaml |
Stable channel stack mirroring pre-production clusters. |
docker-compose.prod.yaml |
Production cutover stack with front-door network hand-off and Notify events enabled. |
docker-compose.airgap.yaml |
Stable stack with air-gapped defaults (no outbound hostnames). |
docker-compose.mirror.yaml |
Managed mirror topology for *.stella-ops.org distribution (Concelier + Excititor + CDN gateway). |
docker-compose.telemetry.yaml |
Optional OpenTelemetry collector overlay (mutual TLS, OTLP ingest endpoints). |
docker-compose.telemetry-storage.yaml |
Prometheus/Tempo/Loki storage overlay with multi-tenant defaults. |
env/*.env.example |
Seed .env files that document required secrets and ports per profile. |
Usage
cp env/dev.env.example dev.env
docker compose --env-file dev.env -f docker-compose.dev.yaml config
docker compose --env-file dev.env -f docker-compose.dev.yaml up -d
The stage and airgap variants behave the same way—swap the file names accordingly. All profiles expose 443/8443 for the UI and REST APIs, and they share a stellaops Docker network scoped to the compose project.
Graph Explorer reminder: If you enable Cartographer or Graph API containers alongside these profiles, update
etc/authority.yamlso thecartographer-serviceclient is marked withproperties.serviceIdentity: "cartographer"and carries a tenant hint. The Authority host now refusesgraph:writetokens without that marker, so apply the configuration change before rolling out the updated images.
Telemetry collector overlay
The OpenTelemetry collector overlay is optional and can be layered on top of any profile:
./ops/devops/telemetry/generate_dev_tls.sh
docker compose -f docker-compose.telemetry.yaml up -d
python ../../ops/devops/telemetry/smoke_otel_collector.py --host localhost
docker compose -f docker-compose.telemetry-storage.yaml up -d
The generator script creates a development CA plus server/client certificates under
deploy/telemetry/certs/. The smoke test sends OTLP/HTTP payloads using the generated
client certificate and asserts the collector reports accepted traces, metrics, and logs.
The storage overlay starts Prometheus, Tempo, and Loki with multitenancy enabled so you
can validate the end-to-end pipeline before promoting changes to staging. Adjust the
configs in deploy/telemetry/storage/ before running in production.
Mount the same certificates when running workloads so the collector can enforce mutual TLS.
For production cutovers copy env/prod.env.example to prod.env, update the secret placeholders, and create the external network expected by the profile:
docker network create stellaops_frontdoor
docker compose --env-file prod.env -f docker-compose.prod.yaml config
Scanner event stream settings
Scanner WebService can emit signed scanner.report.* events to Redis Streams when SCANNER__EVENTS__ENABLED=true. Each profile ships environment placeholders you can override in the .env file:
SCANNER_EVENTS_ENABLED– toggle emission on/off (defaults tofalse).SCANNER_EVENTS_DRIVER– currently onlyredisis supported.SCANNER_EVENTS_DSN– Redis endpoint; leave blank to reuse the queue DSN when it usesredis://.SCANNER_EVENTS_STREAM– stream name (stella.eventsby default).SCANNER_EVENTS_PUBLISH_TIMEOUT_SECONDS– per-publish timeout window (defaults to5).SCANNER_EVENTS_MAX_STREAM_LENGTH– max stream length before Redis trims entries (defaults to10000).
Helm values mirror the same knobs under each service’s env map (see deploy/helm/stellaops/values-*.yaml).
Scheduler worker configuration
Every Compose profile now provisions the scheduler-worker container (backed by the
StellaOps.Scheduler.Worker.Host entrypoint). The environment placeholders exposed
in the .env samples match the options bound by AddSchedulerWorker:
SCHEDULER_QUEUE_KIND– queue transport (NatsorRedis).SCHEDULER_QUEUE_NATS_URL– NATS connection string used by planner/runner consumers.SCHEDULER_STORAGE_DATABASE– MongoDB database name for scheduler state.SCHEDULER_SCANNER_BASEADDRESS– base URL the runner uses when invoking Scanner’s/api/v1/reports(defaults to the in-clusterhttp://scanner-web:8444).
Helm deployments inherit the same defaults from services.scheduler-worker.env in
values.yaml; override them per environment as needed.
Front-door network hand-off
docker-compose.prod.yaml adds a frontdoor network so operators can attach Traefik, Envoy, or an on-prem load balancer that terminates TLS. Override FRONTDOOR_NETWORK in prod.env if your reverse proxy uses a different bridge name. Attach only the externally reachable services (Authority, Signer, Attestor, Concelier, Scanner Web, Notify Web, UI) to that network—internal infrastructure (Mongo, MinIO, RustFS, NATS) stays on the private stellaops network.
Updating to a new release
- Import the new manifest into
deploy/releases/(seedeploy/README.md). - Update image digests in the relevant Compose file(s).
- Re-run
docker compose configto confirm the bundle is deterministic.
Keep digests synchronized between Compose, Helm, and the release manifest to preserve reproducibility guarantees. deploy/tools/validate-profiles.sh performs a quick audit.