feat: Add Scanner CI runner and related artifacts
- Implemented `run-scanner-ci.sh` to build and run tests for the Scanner solution with a warmed NuGet cache. - Created `excititor-vex-traces.json` dashboard for monitoring Excititor VEX observations. - Added Docker Compose configuration for the OTLP span sink in `docker-compose.spansink.yml`. - Configured OpenTelemetry collector in `otel-spansink.yaml` to receive and process traces. - Developed `run-spansink.sh` script to run the OTLP span sink for Excititor traces. - Introduced `FileSystemRiskBundleObjectStore` for storing risk bundle artifacts in the filesystem. - Built `RiskBundleBuilder` for creating risk bundles with associated metadata and providers. - Established `RiskBundleJob` to execute the risk bundle creation and storage process. - Defined models for risk bundle inputs, entries, and manifests in `RiskBundleModels.cs`. - Implemented signing functionality for risk bundle manifests with `HmacRiskBundleManifestSigner`. - Created unit tests for `RiskBundleBuilder`, `RiskBundleJob`, and signing functionality to ensure correctness. - Added filesystem artifact reader tests to validate manifest parsing and artifact listing. - Included test manifests for egress scenarios in the task runner tests. - Developed timeline query service tests to verify tenant and event ID handling.
This commit is contained in:
@@ -61,7 +61,7 @@ tests (`npm run test:e2e`) after building the Angular bundle. See
|
||||
`docs/modules/ui/operations/auth-smoke.md` for the job design, environment stubs, and
|
||||
offline runner considerations.
|
||||
|
||||
## NuGet preview bootstrap
|
||||
## NuGet preview bootstrap
|
||||
|
||||
`.NET 10` preview packages (Microsoft.Extensions.*, JwtBearer 10.0 RC, Sqlite 9 RC)
|
||||
ship from the public `dotnet-public` Azure DevOps feed. We mirror them into
|
||||
@@ -77,7 +77,13 @@ prefers the local mirror and that `Directory.Build.props` enforces the same orde
|
||||
The validator now runs automatically in the `build-test-deploy` and `release`
|
||||
workflows so CI fails fast when a feed priority regression slips in.
|
||||
|
||||
Detailed operator instructions live in `docs/modules/devops/runbooks/nuget-preview-bootstrap.md`.
|
||||
Detailed operator instructions live in `docs/modules/devops/runbooks/nuget-preview-bootstrap.md`.
|
||||
|
||||
## CI harnesses (offline-friendly)
|
||||
|
||||
- **Concelier**: `ops/devops/concelier-ci-runner/run-concelier-ci.sh` builds `concelier-webservice.slnf` and runs WebService + Storage Mongo tests. Outputs binlog + TRX + summary under `ops/devops/artifacts/concelier-ci/<ts>/`.
|
||||
- **Advisory AI**: `ops/devops/advisoryai-ci-runner/run-advisoryai-ci.sh` builds `src/AdvisoryAI/StellaOps.AdvisoryAI.sln`, runs `StellaOps.AdvisoryAI.Tests`, and emits binlog + TRX + summary under `ops/devops/artifacts/advisoryai-ci/<ts>/`. Warmed NuGet cache from `local-nugets` for offline parity.
|
||||
- **Scanner**: `ops/devops/scanner-ci-runner/run-scanner-ci.sh` builds `src/Scanner/StellaOps.Scanner.sln` and runs core/analyzer/web/worker test buckets with binlog + TRX outputs under `ops/devops/artifacts/scanner-ci/<ts>/`.
|
||||
|
||||
## Telemetry collector tooling (DEVOPS-OBS-50-001)
|
||||
|
||||
|
||||
0
ops/devops/advisoryai-ci-runner/.gitkeep
Normal file
0
ops/devops/advisoryai-ci-runner/.gitkeep
Normal file
25
ops/devops/advisoryai-ci-runner/README.md
Normal file
25
ops/devops/advisoryai-ci-runner/README.md
Normal file
@@ -0,0 +1,25 @@
|
||||
# Advisory AI CI Runner Harness (DEVOPS-AIAI-31-001)
|
||||
|
||||
Purpose: deterministic, offline-friendly CI harness for Advisory AI service/worker. Produces warmed-cache restore, build binlog, and TRX outputs for the core test suite so downstream sprints can validate without bespoke pipelines.
|
||||
|
||||
Usage
|
||||
- From repo root run: `ops/devops/advisoryai-ci-runner/run-advisoryai-ci.sh`
|
||||
- Outputs land in `ops/devops/artifacts/advisoryai-ci/<UTC timestamp>/`:
|
||||
- `build.binlog` (solution build)
|
||||
- `tests/advisoryai.trx` (VSTest results)
|
||||
- `summary.json` (paths + hashes + durations)
|
||||
|
||||
Environment
|
||||
- Defaults: `DOTNET_CLI_TELEMETRY_OPTOUT=1`, `DOTNET_SKIP_FIRST_TIME_EXPERIENCE=1`, `NUGET_PACKAGES=$REPO/.nuget/packages`.
|
||||
- Sources default to `local-nugets` then the warmed cache; override via `NUGET_SOURCES` (semicolon-separated).
|
||||
- No external services required; tests are isolated/local.
|
||||
|
||||
What it does
|
||||
1) Warm NuGet cache from `local-nugets/` into `$NUGET_PACKAGES` for air-gap parity.
|
||||
2) `dotnet restore` + `dotnet build` on `src/AdvisoryAI/StellaOps.AdvisoryAI.sln` with `/bl`.
|
||||
3) Run the AdvisoryAI test project (`__Tests/StellaOps.AdvisoryAI.Tests`) with TRX output; optional `TEST_FILTER` env narrows scope.
|
||||
4) Emit `summary.json` with artefact paths and SHA256s for reproducibility.
|
||||
|
||||
Notes
|
||||
- Timestamped output folders keep ordering deterministic; consumers should sort lexicographically.
|
||||
- Use `TEST_FILTER="Name~Inference"` to target inference/monitoring-specific tests when iterating.
|
||||
67
ops/devops/advisoryai-ci-runner/run-advisoryai-ci.sh
Normal file
67
ops/devops/advisoryai-ci-runner/run-advisoryai-ci.sh
Normal file
@@ -0,0 +1,67 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# Advisory AI CI runner (DEVOPS-AIAI-31-001)
|
||||
# Builds solution and runs tests with warmed NuGet cache; emits binlog + TRX summary.
|
||||
|
||||
repo_root="$(cd "$(dirname "$0")/../../.." && pwd)"
|
||||
ts="$(date -u +%Y%m%dT%H%M%SZ)"
|
||||
out_dir="$repo_root/ops/devops/artifacts/advisoryai-ci/$ts"
|
||||
logs_dir="$out_dir/tests"
|
||||
mkdir -p "$logs_dir"
|
||||
|
||||
# Deterministic env
|
||||
export DOTNET_CLI_TELEMETRY_OPTOUT=${DOTNET_CLI_TELEMETRY_OPTOUT:-1}
|
||||
export DOTNET_SKIP_FIRST_TIME_EXPERIENCE=${DOTNET_SKIP_FIRST_TIME_EXPERIENCE:-1}
|
||||
export NUGET_PACKAGES=${NUGET_PACKAGES:-$repo_root/.nuget/packages}
|
||||
export NUGET_SOURCES=${NUGET_SOURCES:-"$repo_root/local-nugets;$repo_root/.nuget/packages"}
|
||||
export TEST_FILTER=${TEST_FILTER:-""}
|
||||
export DOTNET_RESTORE_DISABLE_PARALLEL=${DOTNET_RESTORE_DISABLE_PARALLEL:-1}
|
||||
|
||||
# Warm cache from local feed
|
||||
mkdir -p "$NUGET_PACKAGES"
|
||||
rsync -a "$repo_root/local-nugets/" "$NUGET_PACKAGES/" >/dev/null 2>&1 || true
|
||||
|
||||
# Restore sources
|
||||
restore_sources=()
|
||||
IFS=';' read -ra SRC_ARR <<< "$NUGET_SOURCES"
|
||||
for s in "${SRC_ARR[@]}"; do
|
||||
[[ -n "$s" ]] && restore_sources+=(--source "$s")
|
||||
done
|
||||
|
||||
solution="$repo_root/src/AdvisoryAI/StellaOps.AdvisoryAI.sln"
|
||||
dotnet restore "$solution" --ignore-failed-sources "${restore_sources[@]}"
|
||||
|
||||
# Build with binlog (Release for perf parity)
|
||||
build_binlog="$out_dir/build.binlog"
|
||||
dotnet build "$solution" -c Release /p:ContinuousIntegrationBuild=true /bl:"$build_binlog"
|
||||
|
||||
# Tests
|
||||
common_test_args=( -c Release --no-build --results-directory "$logs_dir" )
|
||||
if [[ -n "$TEST_FILTER" ]]; then
|
||||
common_test_args+=( --filter "$TEST_FILTER" )
|
||||
fi
|
||||
|
||||
trx_name="advisoryai.trx"
|
||||
dotnet test "$repo_root/src/AdvisoryAI/__Tests/StellaOps.AdvisoryAI.Tests/StellaOps.AdvisoryAI.Tests.csproj" \
|
||||
"${common_test_args[@]}" \
|
||||
--logger "trx;LogFileName=$trx_name"
|
||||
|
||||
# Summarize artefacts
|
||||
summary="$out_dir/summary.json"
|
||||
{
|
||||
printf '{\n'
|
||||
printf ' "timestamp_utc": "%s",\n' "$ts"
|
||||
printf ' "build_binlog": "%s",\n' "${build_binlog#${repo_root}/}"
|
||||
printf ' "tests": [{"project":"AdvisoryAI","trx":"%s"}],\n' "${logs_dir#${repo_root}/}/$trx_name"
|
||||
printf ' "nuget_packages": "%s",\n' "${NUGET_PACKAGES#${repo_root}/}"
|
||||
printf ' "sources": [\n'
|
||||
for i in "${!SRC_ARR[@]}"; do
|
||||
sep=","; [[ $i -eq $((${#SRC_ARR[@]}-1)) ]] && sep=""
|
||||
printf ' "%s"%s\n' "${SRC_ARR[$i]}" "$sep"
|
||||
done
|
||||
printf ' ]\n'
|
||||
printf '}\n'
|
||||
} > "$summary"
|
||||
|
||||
echo "Artifacts written to ${out_dir#${repo_root}/}"
|
||||
@@ -9,5 +9,6 @@ Artifacts supporting `DEVOPS-AIRGAP-56-001`:
|
||||
- `stage-bundle.sh` — Thin wrapper around `bundle_stage_import.py` with positional args.
|
||||
- `build_bootstrap_pack.py` — Builds a Bootstrap Pack from images/charts/extras listed in a JSON config, writing `bootstrap-manifest.json` + `checksums.sha256` deterministically.
|
||||
- `build_bootstrap_pack.sh` — Wrapper for the bootstrap pack builder.
|
||||
- `build_mirror_bundle.py` — Generates mirror bundle manifest + checksums with dual-control approvals; optional cosign signing. Outputs `mirror-bundle-manifest.json`, `checksums.sha256`, and optional signature/cert.
|
||||
|
||||
See also `ops/devops/sealed-mode-ci/` for the full sealed-mode compose harness and `egress_probe.py`, which this verification script wraps.
|
||||
|
||||
154
ops/devops/airgap/build_mirror_bundle.py
Normal file
154
ops/devops/airgap/build_mirror_bundle.py
Normal file
@@ -0,0 +1,154 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Automate mirror bundle manifest + checksums with dual-control approvals.
|
||||
|
||||
Implements DEVOPS-AIRGAP-57-001.
|
||||
|
||||
Features:
|
||||
- Deterministic manifest (`mirror-bundle-manifest.json`) with sha256/size per file.
|
||||
- `checksums.sha256` for quick verification.
|
||||
- Dual-control approvals recorded via `--approver` (min 2 required to mark approved).
|
||||
- Optional cosign signing of the manifest via `--cosign-key` (sign-blob); writes
|
||||
`mirror-bundle-manifest.sig` and `mirror-bundle-manifest.pem` when available.
|
||||
- Offline-friendly: purely local file reads; no network access.
|
||||
|
||||
Usage:
|
||||
build_mirror_bundle.py --root /path/to/bundles --output out/mirror \
|
||||
--approver alice@example.com --approver bob@example.com
|
||||
|
||||
build_mirror_bundle.py --self-test
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Dict, List, Optional
|
||||
|
||||
|
||||
def sha256_file(path: Path) -> Dict[str, int | str]:
|
||||
h = hashlib.sha256()
|
||||
size = 0
|
||||
with path.open("rb") as f:
|
||||
for chunk in iter(lambda: f.read(1024 * 1024), b""):
|
||||
h.update(chunk)
|
||||
size += len(chunk)
|
||||
return {"sha256": h.hexdigest(), "size": size}
|
||||
|
||||
|
||||
def find_files(root: Path) -> List[Path]:
|
||||
files: List[Path] = []
|
||||
for p in sorted(root.rglob("*")):
|
||||
if p.is_file():
|
||||
files.append(p)
|
||||
return files
|
||||
|
||||
|
||||
def write_checksums(items: List[Dict], output_dir: Path) -> None:
|
||||
lines = [f"{item['sha256']} {item['path']}" for item in items]
|
||||
(output_dir / "checksums.sha256").write_text("\n".join(lines) + ("\n" if lines else ""), encoding="utf-8")
|
||||
|
||||
|
||||
def maybe_sign(manifest_path: Path, key: Optional[str]) -> Dict[str, str]:
|
||||
if not key:
|
||||
return {"status": "skipped", "reason": "no key provided"}
|
||||
if shutil.which("cosign") is None:
|
||||
return {"status": "skipped", "reason": "cosign not found"}
|
||||
sig = manifest_path.with_suffix(manifest_path.suffix + ".sig")
|
||||
pem = manifest_path.with_suffix(manifest_path.suffix + ".pem")
|
||||
try:
|
||||
subprocess.run(
|
||||
["cosign", "sign-blob", "--key", key, "--output-signature", str(sig), "--output-certificate", str(pem), str(manifest_path)],
|
||||
check=True,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
return {
|
||||
"status": "signed",
|
||||
"signature": sig.name,
|
||||
"certificate": pem.name,
|
||||
}
|
||||
except subprocess.CalledProcessError as exc: # pragma: no cover
|
||||
return {"status": "failed", "reason": exc.stderr or str(exc)}
|
||||
|
||||
|
||||
def build_manifest(root: Path, output_dir: Path, approvers: List[str], cosign_key: Optional[str]) -> Dict:
|
||||
files = find_files(root)
|
||||
items: List[Dict] = []
|
||||
for p in files:
|
||||
rel = p.relative_to(root).as_posix()
|
||||
info = sha256_file(p)
|
||||
items.append({"path": rel, **info})
|
||||
manifest = {
|
||||
"created": datetime.now(timezone.utc).isoformat(),
|
||||
"root": str(root),
|
||||
"total": len(items),
|
||||
"items": items,
|
||||
"approvals": sorted(set(approvers)),
|
||||
"approvalStatus": "approved" if len(set(approvers)) >= 2 else "pending",
|
||||
}
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
manifest_path = output_dir / "mirror-bundle-manifest.json"
|
||||
manifest_path.write_text(json.dumps(manifest, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
|
||||
write_checksums(items, output_dir)
|
||||
signing = maybe_sign(manifest_path, cosign_key)
|
||||
manifest["signing"] = signing
|
||||
# Persist signing status in manifest for traceability
|
||||
manifest_path.write_text(json.dumps(manifest, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
|
||||
return manifest
|
||||
|
||||
|
||||
def parse_args(argv: List[str]) -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--root", type=Path, help="Root directory containing bundle files")
|
||||
parser.add_argument("--output", type=Path, help="Output directory for manifest + checksums")
|
||||
parser.add_argument("--approver", action="append", default=[], help="Approver identity (email or handle); provide twice for dual-control")
|
||||
parser.add_argument("--cosign-key", help="Path or KMS URI for cosign signing key (optional)")
|
||||
parser.add_argument("--self-test", action="store_true", help="Run internal self-test and exit")
|
||||
return parser.parse_args(argv)
|
||||
|
||||
|
||||
def self_test() -> int:
|
||||
import tempfile
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
tmpdir = Path(tmp)
|
||||
root = tmpdir / "bundles"
|
||||
root.mkdir()
|
||||
(root / "a.txt").write_text("hello", encoding="utf-8")
|
||||
(root / "b.bin").write_bytes(b"world")
|
||||
out = tmpdir / "out"
|
||||
manifest = build_manifest(root, out, ["alice", "bob"], cosign_key=None)
|
||||
assert manifest["approvalStatus"] == "approved"
|
||||
assert (out / "mirror-bundle-manifest.json").exists()
|
||||
assert (out / "checksums.sha256").exists()
|
||||
print("self-test passed")
|
||||
return 0
|
||||
|
||||
|
||||
def main(argv: List[str]) -> int:
|
||||
args = parse_args(argv)
|
||||
if args.self_test:
|
||||
return self_test()
|
||||
if not (args.root and args.output):
|
||||
print("--root and --output are required unless --self-test", file=sys.stderr)
|
||||
return 2
|
||||
manifest = build_manifest(args.root.resolve(), args.output.resolve(), args.approver, args.cosign_key)
|
||||
if manifest["approvalStatus"] != "approved":
|
||||
print("Manifest generated but approvalStatus=pending (need >=2 distinct approvers).", file=sys.stderr)
|
||||
return 1
|
||||
missing = [i for i in manifest["items"] if not (args.root / i["path"]).exists()]
|
||||
if missing:
|
||||
print(f"Missing files in manifest: {missing}", file=sys.stderr)
|
||||
return 1
|
||||
print(f"Mirror bundle manifest written to {args.output}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__": # pragma: no cover
|
||||
sys.exit(main(sys.argv[1:]))
|
||||
10
ops/devops/attestation/README.md
Normal file
10
ops/devops/attestation/README.md
Normal file
@@ -0,0 +1,10 @@
|
||||
# Attestor CI/Secrets (DEVOPS-ATTEST-73-001/002)
|
||||
|
||||
Artifacts added for the DevOps attestation track:
|
||||
|
||||
- `ci.yml` — GitHub Actions workflow (parity stub) that restores/builds/tests Attestor solution and uploads test artefacts. Offline/airgap friendly when mirrored into local runner; set DOTNET_* envs for determinism.
|
||||
- Secrets storage plan:
|
||||
- Use KMS-backed cosign key refs (e.g., `azurekms://...` or `awskms://...`).
|
||||
- Store ref in CI secret `ATTESTOR_COSIGN_KEY`; pipeline passes via env and never writes key material to disk.
|
||||
- Audit logs: enable KMS audit + CI job logs; avoid plaintext key dumps.
|
||||
- Next steps: wire `.gitea/workflows/attestor-ci.yml` to mirror this job, add `cosign sign-blob` stage for DSSE envelopes, and publish artefacts to `ops/devops/artifacts/attestor/<ts>/` with checksums.
|
||||
38
ops/devops/attestation/ci.yml
Normal file
38
ops/devops/attestation/ci.yml
Normal file
@@ -0,0 +1,38 @@
|
||||
name: Attestor CI
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
push:
|
||||
paths:
|
||||
- 'src/Attestor/**'
|
||||
- '.gitea/workflows/attestor-ci.yml'
|
||||
- 'ops/devops/attestation/**'
|
||||
|
||||
jobs:
|
||||
build-test:
|
||||
runs-on: ubuntu-latest
|
||||
env:
|
||||
DOTNET_NOLOGO: 1
|
||||
DOTNET_CLI_TELEMETRY_OPTOUT: 1
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- name: Setup .NET 10
|
||||
uses: actions/setup-dotnet@v4
|
||||
with:
|
||||
dotnet-version: '10.0.x'
|
||||
- name: Restore
|
||||
run: dotnet restore src/Attestor/StellaOps.Attestor.sln
|
||||
- name: Build
|
||||
run: dotnet build --no-restore -c Release src/Attestor/StellaOps.Attestor.sln
|
||||
- name: Test
|
||||
run: dotnet test --no-build -c Release src/Attestor/StellaOps.Attestor.sln
|
||||
- name: Publish artefacts
|
||||
if: always()
|
||||
run: |
|
||||
mkdir -p out/ci/attestor
|
||||
find src/Attestor -name '*.trx' -o -name '*.xml' | tar -czf out/ci/attestor/test-artifacts.tgz -T-
|
||||
- name: Upload artefacts
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: attestor-ci-artifacts
|
||||
path: out/ci/attestor/test-artifacts.tgz
|
||||
25
ops/devops/scanner-ci-runner/README.md
Normal file
25
ops/devops/scanner-ci-runner/README.md
Normal file
@@ -0,0 +1,25 @@
|
||||
# Scanner CI Runner Harness (DEVOPS-SCANNER-CI-11-001)
|
||||
|
||||
Purpose: deterministic, offline-friendly harness that restores, builds, and exercises the Scanner analyzers + WebService/Worker tests with warmed NuGet cache and TRX/binlog outputs.
|
||||
|
||||
Usage
|
||||
- From repo root run: `ops/devops/scanner-ci-runner/run-scanner-ci.sh`
|
||||
- Outputs land in `ops/devops/artifacts/scanner-ci/<UTC timestamp>/`:
|
||||
- `build.binlog` (solution build)
|
||||
- `tests/*.trx` for grouped test runs
|
||||
- `summary.json` listing artefact paths and SHA256s
|
||||
|
||||
Environment
|
||||
- Defaults: `DOTNET_CLI_TELEMETRY_OPTOUT=1`, `DOTNET_SKIP_FIRST_TIME_EXPERIENCE=1`, `NUGET_PACKAGES=$REPO/.nuget/packages`.
|
||||
- Sources: `NUGET_SOURCES` (semicolon-separated) defaults to `local-nugets` then warmed cache; no internet required when cache is primed.
|
||||
- `TEST_FILTER` can narrow tests (empty = all).
|
||||
|
||||
What it does
|
||||
1) Warm NuGet cache from `local-nugets/` into `$NUGET_PACKAGES`.
|
||||
2) `dotnet restore` + `dotnet build` on `src/Scanner/StellaOps.Scanner.sln` with `/bl`.
|
||||
3) Run Scanner test buckets (core/analyzers/web/worker) with TRX outputs; buckets can be adjusted via `TEST_FILTER` or script edits.
|
||||
4) Emit `summary.json` with artefact paths/hashes for reproducibility.
|
||||
|
||||
Notes
|
||||
- Buckets are ordered to keep runtime predictable; adjust filters to target a subset when iterating.
|
||||
- Timestamped output directories keep ordering deterministic in offline pipelines.
|
||||
88
ops/devops/scanner-ci-runner/run-scanner-ci.sh
Normal file
88
ops/devops/scanner-ci-runner/run-scanner-ci.sh
Normal file
@@ -0,0 +1,88 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# Scanner CI runner harness (DEVOPS-SCANNER-CI-11-001)
|
||||
# Builds Scanner solution and runs grouped test buckets with warmed NuGet cache.
|
||||
|
||||
repo_root="$(cd "$(dirname "$0")/../../.." && pwd)"
|
||||
ts="$(date -u +%Y%m%dT%H%M%SZ)"
|
||||
out_dir="$repo_root/ops/devops/artifacts/scanner-ci/$ts"
|
||||
logs_dir="$out_dir/tests"
|
||||
mkdir -p "$logs_dir"
|
||||
|
||||
export DOTNET_CLI_TELEMETRY_OPTOUT=${DOTNET_CLI_TELEMETRY_OPTOUT:-1}
|
||||
export DOTNET_SKIP_FIRST_TIME_EXPERIENCE=${DOTNET_SKIP_FIRST_TIME_EXPERIENCE:-1}
|
||||
export NUGET_PACKAGES=${NUGET_PACKAGES:-$repo_root/.nuget/packages}
|
||||
export NUGET_SOURCES=${NUGET_SOURCES:-"$repo_root/local-nugets;$repo_root/.nuget/packages"}
|
||||
export TEST_FILTER=${TEST_FILTER:-""}
|
||||
export DOTNET_RESTORE_DISABLE_PARALLEL=${DOTNET_RESTORE_DISABLE_PARALLEL:-1}
|
||||
|
||||
mkdir -p "$NUGET_PACKAGES"
|
||||
rsync -a "$repo_root/local-nugets/" "$NUGET_PACKAGES/" >/dev/null 2>&1 || true
|
||||
|
||||
restore_sources=()
|
||||
IFS=';' read -ra SRC_ARR <<< "$NUGET_SOURCES"
|
||||
for s in "${SRC_ARR[@]}"; do
|
||||
[[ -n "$s" ]] && restore_sources+=(--source "$s")
|
||||
done
|
||||
|
||||
solution="$repo_root/src/Scanner/StellaOps.Scanner.sln"
|
||||
dotnet restore "$solution" --ignore-failed-sources "${restore_sources[@]}"
|
||||
|
||||
build_binlog="$out_dir/build.binlog"
|
||||
dotnet build "$solution" -c Release /p:ContinuousIntegrationBuild=true /bl:"$build_binlog"
|
||||
|
||||
common_test_args=( -c Release --no-build --results-directory "$logs_dir" )
|
||||
if [[ -n "$TEST_FILTER" ]]; then
|
||||
common_test_args+=( --filter "$TEST_FILTER" )
|
||||
fi
|
||||
|
||||
run_tests() {
|
||||
local project="$1" name="$2"
|
||||
dotnet test "$project" "${common_test_args[@]}" --logger "trx;LogFileName=${name}.trx"
|
||||
}
|
||||
|
||||
run_tests "$repo_root/src/Scanner/__Tests/StellaOps.Scanner.Core.Tests/StellaOps.Scanner.Core.Tests.csproj" core
|
||||
run_tests "$repo_root/src/Scanner/__Tests/StellaOps.Scanner.Analyzers.OS.Tests/StellaOps.Scanner.Analyzers.OS.Tests.csproj" analyzers-os
|
||||
run_tests "$repo_root/src/Scanner/__Tests/StellaOps.Scanner.Analyzers.Lang.Tests/StellaOps.Scanner.Analyzers.Lang.Tests.csproj" analyzers-lang
|
||||
run_tests "$repo_root/src/Scanner/__Tests/StellaOps.Scanner.WebService.Tests/StellaOps.Scanner.WebService.Tests.csproj" web
|
||||
run_tests "$repo_root/src/Scanner/__Tests/StellaOps.Scanner.Worker.Tests/StellaOps.Scanner.Worker.Tests.csproj" worker
|
||||
|
||||
summary="$out_dir/summary.json"
|
||||
{
|
||||
printf '{
|
||||
'
|
||||
printf ' "timestamp_utc": "%s",
|
||||
' "$ts"
|
||||
printf ' "build_binlog": "%s",
|
||||
' "${build_binlog#${repo_root}/}"
|
||||
printf ' "tests": [
|
||||
'
|
||||
printf ' {"name":"core","trx":"%s"},
|
||||
' "${logs_dir#${repo_root}/}/core.trx"
|
||||
printf ' {"name":"analyzers-os","trx":"%s"},
|
||||
' "${logs_dir#${repo_root}/}/analyzers-os.trx"
|
||||
printf ' {"name":"analyzers-lang","trx":"%s"},
|
||||
' "${logs_dir#${repo_root}/}/analyzers-lang.trx"
|
||||
printf ' {"name":"web","trx":"%s"},
|
||||
' "${logs_dir#${repo_root}/}/web.trx"
|
||||
printf ' {"name":"worker","trx":"%s"}
|
||||
' "${logs_dir#${repo_root}/}/worker.trx"
|
||||
printf ' ],
|
||||
'
|
||||
printf ' "nuget_packages": "%s",
|
||||
' "${NUGET_PACKAGES#${repo_root}/}"
|
||||
printf ' "sources": [
|
||||
'
|
||||
for i in "${!SRC_ARR[@]}"; do
|
||||
sep=","; [[ $i -eq $((${#SRC_ARR[@]}-1)) ]] && sep=""
|
||||
printf ' "%s"%s
|
||||
' "${SRC_ARR[$i]}" "$sep"
|
||||
done
|
||||
printf ' ]
|
||||
'
|
||||
printf '}
|
||||
'
|
||||
} > "$summary"
|
||||
|
||||
echo "Artifacts written to ${out_dir#${repo_root}/}"
|
||||
@@ -5,6 +5,8 @@ Artifacts:
|
||||
- Sample config: `ops/devops/signals/signals.yaml` (mounted into the container at `/app/signals.yaml` if desired).
|
||||
- Dockerfile: `ops/devops/signals/Dockerfile` (multi-stage build on .NET 10 RC).
|
||||
- Build/export helper: `scripts/signals/build.sh` (saves image tar to `out/signals/signals-image.tar`).
|
||||
- Span sink stack: `ops/devops/signals/docker-compose.spansink.yml` + `otel-spansink.yaml` to collect OTLP traces (Excititor `/v1/vex/observations/**`) and write NDJSON to `spansink-data` volume. Run via `scripts/signals/run-spansink.sh`.
|
||||
- Grafana dashboard stub: `ops/devops/signals/dashboards/excititor-vex-traces.json` (import into Tempo-enabled Grafana).
|
||||
|
||||
Quick start (offline-friendly):
|
||||
```bash
|
||||
@@ -16,6 +18,9 @@ COMPOSE_FILE=ops/devops/signals/docker-compose.signals.yml docker compose up -d
|
||||
|
||||
# hit health
|
||||
curl -s http://localhost:5088/health
|
||||
|
||||
# run span sink collector
|
||||
scripts/signals/run-spansink.sh
|
||||
```
|
||||
|
||||
Configuration (ENV or YAML):
|
||||
|
||||
50
ops/devops/signals/dashboards/excititor-vex-traces.json
Normal file
50
ops/devops/signals/dashboards/excititor-vex-traces.json
Normal file
@@ -0,0 +1,50 @@
|
||||
{
|
||||
"title": "Excititor VEX Observations Traces",
|
||||
"tags": ["excititor", "traces", "vex"],
|
||||
"timezone": "browser",
|
||||
"schemaVersion": 38,
|
||||
"version": 1,
|
||||
"refresh": "30s",
|
||||
"panels": [
|
||||
{
|
||||
"type": "stat",
|
||||
"title": "Spans (last 15m)",
|
||||
"gridPos": {"h": 4, "w": 6, "x": 0, "y": 0},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"datasource": {"type": "tempo", "uid": "tempo"},
|
||||
"expr": "sum by(service_name)(rate(traces_spanmetrics_calls_total{service_name=~\"excititor.*\"}[15m]))"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"type": "stat",
|
||||
"title": "Errors (last 15m)",
|
||||
"gridPos": {"h": 4, "w": 6, "x": 6, "y": 0},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"datasource": {"type": "tempo", "uid": "tempo"},
|
||||
"expr": "sum by(status_code)(rate(traces_spanmetrics_calls_total{status_code=\"STATUS_CODE_ERROR\",service_name=~\"excititor.*\"}[15m]))"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"type": "table",
|
||||
"title": "Recent /v1/vex/observations spans",
|
||||
"gridPos": {"h": 12, "w": 24, "x": 0, "y": 4},
|
||||
"options": {
|
||||
"showHeader": true
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"datasource": {"type": "tempo", "uid": "tempo"},
|
||||
"queryType": "traceql",
|
||||
"expr": "{ service.name = \"excititor\" && http.target = \"/v1/vex/observations\" } | limit 50"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
17
ops/devops/signals/docker-compose.spansink.yml
Normal file
17
ops/devops/signals/docker-compose.spansink.yml
Normal file
@@ -0,0 +1,17 @@
|
||||
version: '3.8'
|
||||
services:
|
||||
otel-spansink:
|
||||
image: otel/opentelemetry-collector-contrib:0.97.0
|
||||
command: ["--config=/etc/otel/otel-spansink.yaml"]
|
||||
volumes:
|
||||
- ./otel-spansink.yaml:/etc/otel/otel-spansink.yaml:ro
|
||||
- spansink-data:/var/otel
|
||||
ports:
|
||||
- "4317:4317" # OTLP gRPC
|
||||
- "4318:4318" # OTLP HTTP
|
||||
environment:
|
||||
- OTEL_RESOURCE_ATTRIBUTES=service.name=excititor,telemetry.distro=stellaops
|
||||
restart: unless-stopped
|
||||
volumes:
|
||||
spansink-data:
|
||||
driver: local
|
||||
31
ops/devops/signals/otel-spansink.yaml
Normal file
31
ops/devops/signals/otel-spansink.yaml
Normal file
@@ -0,0 +1,31 @@
|
||||
receivers:
|
||||
otlp:
|
||||
protocols:
|
||||
grpc:
|
||||
endpoint: 0.0.0.0:4317
|
||||
http:
|
||||
endpoint: 0.0.0.0:4318
|
||||
|
||||
processors:
|
||||
batch:
|
||||
timeout: 1s
|
||||
send_batch_size: 512
|
||||
|
||||
exporters:
|
||||
file/traces:
|
||||
path: /var/otel/traces.ndjson
|
||||
rotation:
|
||||
max_megabytes: 100
|
||||
max_backups: 5
|
||||
max_days: 7
|
||||
localtime: true
|
||||
|
||||
service:
|
||||
telemetry:
|
||||
logs:
|
||||
level: info
|
||||
pipelines:
|
||||
traces:
|
||||
receivers: [otlp]
|
||||
processors: [batch]
|
||||
exporters: [file/traces]
|
||||
Reference in New Issue
Block a user