71e9a56cfd
- Implemented `run-scanner-ci.sh` to build and run tests for the Scanner solution with a warmed NuGet cache. - Created `excititor-vex-traces.json` dashboard for monitoring Excititor VEX observations. - Added Docker Compose configuration for the OTLP span sink in `docker-compose.spansink.yml`. - Configured OpenTelemetry collector in `otel-spansink.yaml` to receive and process traces. - Developed `run-spansink.sh` script to run the OTLP span sink for Excititor traces. - Introduced `FileSystemRiskBundleObjectStore` for storing risk bundle artifacts in the filesystem. - Built `RiskBundleBuilder` for creating risk bundles with associated metadata and providers. - Established `RiskBundleJob` to execute the risk bundle creation and storage process. - Defined models for risk bundle inputs, entries, and manifests in `RiskBundleModels.cs`. - Implemented signing functionality for risk bundle manifests with `HmacRiskBundleManifestSigner`. - Created unit tests for `RiskBundleBuilder`, `RiskBundleJob`, and signing functionality to ensure correctness. - Added filesystem artifact reader tests to validate manifest parsing and artifact listing. - Included test manifests for egress scenarios in the task runner tests. - Developed timeline query service tests to verify tenant and event ID handling.
100 lines
5.1 KiB
Markdown
100 lines
5.1 KiB
Markdown
# DevOps Release Automation
|
||
|
||
The **release** workflow builds and signs the StellaOps service containers,
|
||
generates SBOM + provenance attestations, and emits a canonical
|
||
`release.yaml`. The logic lives under `ops/devops/release/` and is invoked
|
||
by the new `.gitea/workflows/release.yml` pipeline.
|
||
|
||
## Local dry run
|
||
|
||
```bash
|
||
./ops/devops/release/build_release.py \
|
||
--version 2025.10.0-edge \
|
||
--channel edge \
|
||
--dry-run
|
||
```
|
||
|
||
Outputs land under `out/release/`. Use `--no-push` to run full builds without
|
||
pushing to the registry.
|
||
|
||
After the build completes, run the verifier to validate recorded hashes and artefact
|
||
presence:
|
||
|
||
```bash
|
||
python ops/devops/release/verify_release.py --release-dir out/release
|
||
```
|
||
|
||
## Python analyzer smoke & signing
|
||
|
||
`dotnet run --project src/Tools/LanguageAnalyzerSmoke` exercises the Python language
|
||
analyzer plug-in against the golden fixtures (cold/warm timings, determinism). The
|
||
release workflow runs this harness automatically and then produces Cosign
|
||
signatures + SHA-256 sidecars for `StellaOps.Scanner.Analyzers.Lang.Python.dll`
|
||
and its `manifest.json`. Keep `COSIGN_KEY_REF`/`COSIGN_IDENTITY_TOKEN` populated so
|
||
the step can sign the artefacts; the generated `.sig`/`.sha256` files ship with the
|
||
Offline Kit bundle.
|
||
|
||
## Required tooling
|
||
|
||
- Docker 25+ with Buildx
|
||
- .NET 10 preview SDK (builds container stages and the SBOM generator)
|
||
- Node.js 20 (Angular UI build)
|
||
- Helm 3.16+
|
||
- Cosign 2.2+
|
||
|
||
Supply signing material via environment variables:
|
||
|
||
- `COSIGN_KEY_REF` – e.g. `file:./keys/cosign.key` or `azurekms://…`
|
||
- `COSIGN_PASSWORD` – password protecting the above key
|
||
|
||
The workflow defaults to multi-arch (`linux/amd64,linux/arm64`), SBOM in
|
||
CycloneDX, and SLSA provenance (`https://slsa.dev/provenance/v1`).
|
||
|
||
## Debug store extraction
|
||
|
||
`build_release.py` now exports stripped debug artefacts for every ELF discovered in the published images. The files land under `out/release/debug/.build-id/<aa>/<rest>.debug`, with metadata captured in `debug/debug-manifest.json` (and a `.sha256` sidecar). Use `jq` to inspect the manifest or `readelf -n` to spot-check a build-id. Offline Kit packaging should reuse the `debug/` directory as-is.
|
||
|
||
## UI auth smoke (Playwright)
|
||
|
||
As part of **DEVOPS-UI-13-006** the pipelines will execute the UI auth smoke
|
||
tests (`npm run test:e2e`) after building the Angular bundle. See
|
||
`docs/modules/ui/operations/auth-smoke.md` for the job design, environment stubs, and
|
||
offline runner considerations.
|
||
|
||
## NuGet preview bootstrap
|
||
|
||
`.NET 10` preview packages (Microsoft.Extensions.*, JwtBearer 10.0 RC, Sqlite 9 RC)
|
||
ship from the public `dotnet-public` Azure DevOps feed. We mirror them into
|
||
`./local-nuget` so restores succeed inside Offline Kit.
|
||
|
||
1. Run `./ops/devops/sync-preview-nuget.sh` whenever you update the manifest.
|
||
2. The script now understands the optional `SourceBase` column (V3 flat container)
|
||
and writes packages alongside their SHA-256 checks.
|
||
3. `NuGet.config` registers the mirror (`local`), dotnet-public, and nuget.org.
|
||
|
||
Use `python3 ops/devops/validate_restore_sources.py` to prove the repo still
|
||
prefers the local mirror and that `Directory.Build.props` enforces the same order.
|
||
The validator now runs automatically in the `build-test-deploy` and `release`
|
||
workflows so CI fails fast when a feed priority regression slips in.
|
||
|
||
Detailed operator instructions live in `docs/modules/devops/runbooks/nuget-preview-bootstrap.md`.
|
||
|
||
## CI harnesses (offline-friendly)
|
||
|
||
- **Concelier**: `ops/devops/concelier-ci-runner/run-concelier-ci.sh` builds `concelier-webservice.slnf` and runs WebService + Storage Mongo tests. Outputs binlog + TRX + summary under `ops/devops/artifacts/concelier-ci/<ts>/`.
|
||
- **Advisory AI**: `ops/devops/advisoryai-ci-runner/run-advisoryai-ci.sh` builds `src/AdvisoryAI/StellaOps.AdvisoryAI.sln`, runs `StellaOps.AdvisoryAI.Tests`, and emits binlog + TRX + summary under `ops/devops/artifacts/advisoryai-ci/<ts>/`. Warmed NuGet cache from `local-nugets` for offline parity.
|
||
- **Scanner**: `ops/devops/scanner-ci-runner/run-scanner-ci.sh` builds `src/Scanner/StellaOps.Scanner.sln` and runs core/analyzer/web/worker test buckets with binlog + TRX outputs under `ops/devops/artifacts/scanner-ci/<ts>/`.
|
||
|
||
## Telemetry collector tooling (DEVOPS-OBS-50-001)
|
||
|
||
- `ops/devops/telemetry/generate_dev_tls.sh` – generates a development CA and
|
||
client/server certificates for the OpenTelemetry collector overlay (mutual TLS).
|
||
- `ops/devops/telemetry/smoke_otel_collector.py` – sends OTLP traces/metrics/logs
|
||
over TLS and validates that the collector increments its receiver counters.
|
||
- `ops/devops/telemetry/package_offline_bundle.py` – re-packages collector assets for the Offline Kit.
|
||
- `ops/devops/telemetry/tenant_isolation_smoke.py` – verifies Tempo/Loki tenant isolation with mTLS and scoped headers.
|
||
- `deploy/compose/docker-compose.telemetry-storage.yaml` – Prometheus/Tempo/Loki stack for staging validation.
|
||
|
||
Combine these helpers with `deploy/compose/docker-compose.telemetry.yaml` to run
|
||
a secured collector locally before rolling out the Helm-based deployment.
|