- Introduced RunnerBackgroundService to handle execution of runner segments. - Added RunnerExecutionService for processing segments and aggregating results. - Implemented PlannerQueueDispatchService to manage dispatching of planner messages. - Created PlannerQueueDispatcherBackgroundService for leasing and processing planner queue messages. - Developed ScannerReportClient for interacting with the scanner service. - Enhanced observability with SchedulerWorkerMetrics for tracking planner and runner performance. - Added comprehensive documentation for the new runner execution pipeline and observability metrics. - Implemented event emission for rescan activity and scanner report readiness.
Deployment Profiles
This directory contains deterministic deployment bundles for the core Stella Ops stack. All manifests reference immutable image digests and map 1:1 to the release manifests stored under deploy/releases/.
Structure
releases/– canonical release manifests (edge, stable, airgap) used to source image digests.compose/– Docker Compose bundles for dev/stage/airgap targets plus.envseed files.compose/docker-compose.mirror.yaml– managed mirror bundle for*.stella-ops.orgwith gateway cache and multi-tenant auth.compose/docker-compose.telemetry.yaml– optional OpenTelemetry collector overlay (mutual TLS, OTLP pipelines).compose/docker-compose.telemetry-storage.yaml– optional Prometheus/Tempo/Loki stack for observability backends.helm/stellaops/– multi-profile Helm chart with values files for dev/stage/airgap.telemetry/– shared OpenTelemetry collector configuration and certificate artefacts (generated via tooling).tools/validate-profiles.sh– helper that runsdocker compose configandhelm lint/templatefor every profile.
Workflow
- Update or add a release manifest under
releases/with the new digests. - Mirror the digests into the Compose and Helm profiles that correspond to that channel.
- Run
deploy/tools/validate-profiles.sh(requires Docker CLI and Helm) to ensure the bundles lint and template cleanly. - If telemetry ingest is required for the release, generate development certificates using
./ops/devops/telemetry/generate_dev_tls.shand run the collector smoke test withpython ./ops/devops/telemetry/smoke_otel_collector.pyto verify the OTLP endpoints. - Commit the change alongside any documentation updates (e.g. install guide cross-links).
Maintaining the digest linkage keeps offline/air-gapped installs reproducible and avoids tag drift between environments.
Additional tooling
deploy/tools/check-channel-alignment.py– verifies that Helm/Compose profiles reference the exact images listed in a release manifest. Run it for each channel before promoting a release.ops/devops/telemetry/generate_dev_tls.sh– produces local CA/server/client certificates for Compose-based collector testing.ops/devops/telemetry/smoke_otel_collector.py– sends OTLP traffic and asserts the collector accepted traces, metrics, and logs.ops/devops/telemetry/package_offline_bundle.py– packages telemetry assets (config/Helm/Compose) into a signed tarball for air-gapped installs.docs/ops/deployment-upgrade-runbook.md– end-to-end instructions for upgrade, rollback, and channel promotion workflows (Helm + Compose).
CI smoke checks
The .gitea/workflows/build-test-deploy.yml pipeline includes a notify-smoke stage that validates scanner event propagation after staging deployments. Configure the following repository secrets (or environment-level secrets) so the job can connect to Redis and the Notify API:
NOTIFY_SMOKE_REDIS_DSN– Redis connection string (redis://user:pass@host:port/db).NOTIFY_SMOKE_NOTIFY_BASEURL– Base URL for the staging Notify WebService (e.g.https://notify.stage.stella-ops.internal).NOTIFY_SMOKE_NOTIFY_TOKEN– OAuth bearer token (service account) with permission to read deliveries.NOTIFY_SMOKE_NOTIFY_TENANT– Tenant identifier used for the smoke validation requests.- (Optional)
NOTIFY_SMOKE_NOTIFY_TENANT_HEADER– Override for the tenant header name (defaults toX-StellaOps-Tenant).
Define the following repository variables (or secrets) to drive the assertions performed by the smoke check:
NOTIFY_SMOKE_EXPECT_KINDS– Comma-separated event kinds the checker must observe (for examplescanner.report.ready,scanner.scan.completed).NOTIFY_SMOKE_LOOKBACK_MINUTES– Time window (in minutes) used when scanning the Redis stream for recent events (for example30).
All of the above values are required—the workflow fails fast with a descriptive error if any are missing or empty. Provide the variables at the organisation or repository scope before enabling the smoke stage.