885ce86af4
- Introduced a comprehensive CI job structure for VEX Lens, including build, test, linting, and load testing. - Defined load test parameters and SLOs for VEX Lens API and Issuer Directory. - Created Grafana dashboards and alerting mechanisms for monitoring API performance and error rates. - Established offline posture guidelines for CI jobs and load testing. feat: Implement deterministic projection verification script - Added `verify_projection.sh` script for verifying the integrity of projection exports against expected hashes. - Ensured robust error handling for missing files and hash mismatches. feat: Develop Vuln Explorer CI and Ops Plan - Created CI jobs for Vuln Explorer, including build, test, and replay verification. - Implemented backup and disaster recovery strategies for MongoDB and Redis. - Established Merkle anchoring verification and automation for ledger projector. feat: Introduce EventEnvelopeHasher for hashing event envelopes - Implemented `EventEnvelopeHasher` to compute SHA256 hashes for event envelopes. feat: Add Risk Store and Dashboard components - Developed `RiskStore` for managing risk data and state. - Created `RiskDashboardComponent` for displaying risk profiles with filtering capabilities. - Implemented unit tests for `RiskStore` and `RiskDashboardComponent`. feat: Enhance Vulnerability Detail Component - Developed `VulnerabilityDetailComponent` for displaying detailed information about vulnerabilities. - Implemented error handling for missing vulnerability IDs and loading failures.
67 lines
6.0 KiB
Markdown
67 lines
6.0 KiB
Markdown
# Deployment Profiles
|
||
|
||
This directory contains deterministic deployment bundles for the core Stella Ops stack. All manifests reference immutable image digests and map 1:1 to the release manifests stored under `deploy/releases/`.
|
||
|
||
## Structure
|
||
|
||
- `releases/` – canonical release manifests (edge, stable, airgap) used to source image digests.
|
||
- `compose/` – Docker Compose bundles for dev/stage/airgap targets plus `.env` seed files.
|
||
- `compose/docker-compose.mirror.yaml` – managed mirror bundle for `*.stella-ops.org` with gateway cache and multi-tenant auth.
|
||
- `compose/docker-compose.telemetry.yaml` – optional OpenTelemetry collector overlay (mutual TLS, OTLP pipelines).
|
||
- `compose/docker-compose.telemetry-storage.yaml` – optional Prometheus/Tempo/Loki stack for observability backends.
|
||
- `helm/stellaops/` – multi-profile Helm chart with values files for dev/stage/airgap.
|
||
- `telemetry/` – shared OpenTelemetry collector configuration and certificate artefacts (generated via tooling).
|
||
- `tools/validate-profiles.sh` – helper that runs `docker compose config` and `helm lint/template` for every profile.
|
||
|
||
## Workflow
|
||
|
||
1. Update or add a release manifest under `releases/` with the new digests.
|
||
2. Mirror the digests into the Compose and Helm profiles that correspond to that channel.
|
||
3. Run `deploy/tools/validate-profiles.sh` (requires Docker CLI and Helm) to ensure the bundles lint and template cleanly.
|
||
4. If telemetry ingest is required for the release, generate development certificates using
|
||
`./ops/devops/telemetry/generate_dev_tls.sh` and run the collector smoke test with
|
||
`python ./ops/devops/telemetry/smoke_otel_collector.py` to verify the OTLP endpoints.
|
||
5. Commit the change alongside any documentation updates (e.g. install guide cross-links).
|
||
|
||
Maintaining the digest linkage keeps offline/air-gapped installs reproducible and avoids tag drift between environments.
|
||
|
||
### Surface.Env rollout warnings
|
||
|
||
- Compose (`deploy/compose/env/*.env.example`) and Helm (`deploy/helm/stellaops/values-*.yaml`) now seed `SCANNER_SURFACE_*` _and_ `ZASTAVA_SURFACE_*` variables so Scanner Worker/WebService and Zastava Observer/Webhook resolve cache roots, Surface.FS endpoints, and secrets providers through `StellaOps.Scanner.Surface.Env`.
|
||
- During rollout, watch for structured log messages (and readiness output) prefixed with `surface.env.`—for example, `surface.env.cache_root_missing`, `surface.env.endpoint_unreachable`, or `surface.env.secrets_provider_invalid`.
|
||
- Treat these warnings as deployment blockers: update the endpoint/cache/secrets values or permissions before promoting the environment, otherwise workers will fail fast at startup.
|
||
- Air-gapped bundles default the secrets provider to `file` with `/etc/stellaops/secrets`; connected clusters default to `kubernetes`. Adjust the provider/root pair if your secrets manager differs.
|
||
- Secret provisioning workflows for Kubernetes/Compose/Offline Kit are documented in `ops/devops/secrets/surface-secrets-provisioning.md`; follow that for `Surface.Secrets` handles and RBAC/permissions.
|
||
|
||
### Mongo2Go OpenSSL prerequisites
|
||
|
||
- Linux runners that execute Mongo2Go-backed suites (Excititor, Scheduler, Graph, etc.) must expose OpenSSL 1.1 (`libcrypto.so.1.1`, `libssl.so.1.1`). The canonical copies live under `tests/native/openssl-1.1/linux-x64`.
|
||
- Export `LD_LIBRARY_PATH="$(git rev-parse --show-toplevel)/tests/native/openssl-1.1/linux-x64:${LD_LIBRARY_PATH:-}"` before invoking `dotnet test`. Example:\
|
||
`LD_LIBRARY_PATH="$(pwd)/tests/native/openssl-1.1/linux-x64" dotnet test src/Excititor/__Tests/StellaOps.Excititor.WebService.Tests/StellaOps.Excititor.WebService.Tests.csproj --nologo`.
|
||
- CI agents or Dockerfiles that host these tests should either mount the directory into the container or copy the two `.so` files into a directory that is already on the runtime library path.
|
||
|
||
### Additional tooling
|
||
|
||
- `deploy/tools/check-channel-alignment.py` – verifies that Helm/Compose profiles reference the exact images listed in a release manifest. Run it for each channel before promoting a release.
|
||
- `ops/devops/telemetry/generate_dev_tls.sh` – produces local CA/server/client certificates for Compose-based collector testing.
|
||
- `ops/devops/telemetry/smoke_otel_collector.py` – sends OTLP traffic and asserts the collector accepted traces, metrics, and logs.
|
||
- `ops/devops/telemetry/package_offline_bundle.py` – packages telemetry assets (config/Helm/Compose) into a signed tarball for air-gapped installs.
|
||
- `docs/modules/devops/runbooks/deployment-upgrade.md` – end-to-end instructions for upgrade, rollback, and channel promotion workflows (Helm + Compose).
|
||
|
||
## CI smoke checks
|
||
|
||
The `.gitea/workflows/build-test-deploy.yml` pipeline includes a `notify-smoke` stage that validates scanner event propagation after staging deployments. Configure the following repository secrets (or environment-level secrets) so the job can connect to Redis and the Notify API:
|
||
|
||
- `NOTIFY_SMOKE_REDIS_DSN` – Redis connection string (`redis://user:pass@host:port/db`).
|
||
- `NOTIFY_SMOKE_NOTIFY_BASEURL` – Base URL for the staging Notify WebService (e.g. `https://notify.stage.stella-ops.internal`).
|
||
- `NOTIFY_SMOKE_NOTIFY_TOKEN` – OAuth bearer token (service account) with permission to read deliveries.
|
||
- `NOTIFY_SMOKE_NOTIFY_TENANT` – Tenant identifier used for the smoke validation requests.
|
||
- *(Optional)* `NOTIFY_SMOKE_NOTIFY_TENANT_HEADER` – Override for the tenant header name (defaults to `X-StellaOps-Tenant`).
|
||
|
||
Define the following repository variables (or secrets) to drive the assertions performed by the smoke check:
|
||
|
||
- `NOTIFY_SMOKE_EXPECT_KINDS` – Comma-separated event kinds the checker must observe (for example `scanner.report.ready,scanner.scan.completed`).
|
||
- `NOTIFY_SMOKE_LOOKBACK_MINUTES` – Time window (in minutes) used when scanning the Redis stream for recent events (for example `30`).
|
||
|
||
All of the above values are required—the workflow fails fast with a descriptive error if any are missing or empty. Provide the variables at the organisation or repository scope before enabling the smoke stage.
|