stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 1 Releases Wiki Activity

feat(docs): Add comprehensive documentation for Vexer, Vulnerability Explorer, and Zastava modules

Browse Source 
- Introduced AGENTS.md, README.md, TASKS.md, and implementation_plan.md for Vexer, detailing mission, responsibilities, key components, and operational notes.
- Established similar documentation structure for Vulnerability Explorer and Zastava modules, including their respective workflows, integrations, and observability notes.
- Created risk scoring profiles documentation outlining the core workflow, factor model, governance, and deliverables.
- Ensured all modules adhere to the Aggregation-Only Contract and maintain determinism and provenance in outputs.
This commit is contained in:
master
2025-10-30 00:09:39 +02:00
parent 3154c67978
commit 7b5bdcf4d3
503 changed files with 16136 additions and 54638 deletions
Download Patch File Download Diff File Expand all files Collapse all files

22
docs/modules/ci/AGENTS.md Normal file
View File

@@ -0,0 +1,22 @@
# CI Recipes agent guide
## Mission
CI module collects reproducible pipeline recipes for builds, tests, and release promotion across supported platforms.
## Key docs
- [Module README](./README.md)
- [Architecture](./architecture.md)
- [Implementation plan](./implementation_plan.md)
- [Task board](./TASKS.md)
## How to get started
1. Open ../../implplan/SPRINTS.md and locate the stories referencing this module.
2. Review ./TASKS.md for local follow-ups and confirm status transitions (TODO → DOING → DONE/BLOCKED).
3. Read the architecture and README for domain context before editing code or docs.
4. Coordinate cross-module changes in the main /AGENTS.md description and through the sprint plan.
## Guardrails
- Honour the Aggregation-Only Contract where applicable (see ../../ingestion/aggregation-only-contract.md).
- Preserve determinism: sort outputs, normalise timestamps (UTC ISO-8601), and avoid machine-specific artefacts.
- Keep Offline Kit parity in mind—document air-gapped workflows for any new feature.
- Update runbooks/observability assets when operational characteristics change.

29
docs/modules/ci/README.md Normal file
View File

@@ -0,0 +1,29 @@
# StellaOps CI Recipes
CI module collects reproducible pipeline recipes for builds, tests, and release promotion across supported platforms.
## Responsibilities
- Provide ready-to-use pipeline snippets for ingestion, scanning, policy evaluation, and exports.
- Document required secrets/scopes and deterministic build knobs.
- Highlight offline-compatible workflows and cache strategies.
## Key components
- Recipe catalogue in ./recipes.md.
## Integrations & dependencies
- DevOps release workflows.
- Module-specific test suites referenced in recipes.
## Operational notes
- Encourage reuse through templated YAML/JSON fragments.
## Related resources
- ./recipes.md
## Backlog references
- CI recipes refresh tracked in ../../TASKS.md under DOCS-CI stories.
## Epic alignment
- **Epic 1 – AOC enforcement:** bake ingestion/verifier guardrails into CI recipes.
- **Epic 10 – Export Center:** provide pipeline snippets for export packaging, signing, and Offline Kit publication.
- **Epic 11 – Notifications Studio:** offer CI hooks for notification previews/tests where relevant.

9
docs/modules/ci/TASKS.md Normal file
View File

@@ -0,0 +1,9 @@
# Task board — CI Recipes
> Local tasks should link back to ./AGENTS.md and mirror status updates into ../../TASKS.md when applicable.
| ID | Status | Owner(s) | Description | Notes |
|----|--------|----------|-------------|-------|
| CI RECIPES-DOCS-0001 | TODO | Docs Guild | Validate that ./README.md aligns with the latest release notes. | See ./AGENTS.md |
| CI RECIPES-OPS-0001 | TODO | Ops Guild | Review runbooks/observability assets after next sprint demo. | Sync outcomes back to ../../TASKS.md |
| CI RECIPES-ENG-0001 | TODO | Module Team | Cross-check implementation plan milestones against ../../implplan/SPRINTS.md. | Update status via ./AGENTS.md workflow |

7
docs/modules/ci/architecture.md Normal file
View File

@@ -0,0 +1,7 @@
# CI Recipes architecture
> Reference the AOC guardrails, export workflows, and notification patterns documented in the Authority, Export Center, and Notify module guides when designing CI templates.
This placeholder summarises the planned architecture for CI Recipes. Consolidate design details from implementation plans and upcoming epics before coding.
Refer to the module README and implementation plan for immediate context, and update this document once component boundaries and data flows are finalised.

21
docs/modules/ci/implementation_plan.md Normal file
View File

@@ -0,0 +1,21 @@
# Implementation plan — CI Recipes
## Current objectives
- Maintain deterministic behaviour and offline parity across releases.
- Keep documentation, telemetry, and runbooks aligned with the latest sprint outcomes.
## Workstreams
- Backlog grooming: reconcile open stories in ../../TASKS.md with this module's roadmap.
- Implementation: collaborate with service owners to land feature work defined in SPRINTS/EPIC docs.
- Validation: extend tests/fixtures to preserve determinism and provenance requirements.
## Epic milestones
- **Epic 1 – AOC enforcement:** ensure pipelines enforce schemas, provenance, and verifier jobs.
- **Epic 10 – Export Center:** add export/signing/Offline Kit automation templates.
- **Epic 11 – Notifications Studio:** document CI hooks for notification previews/tests.
- Track DOCS-CI stories in ../../TASKS.md.
## Coordination
- Review ./AGENTS.md before picking up new work.
- Sync with cross-cutting teams noted in ../../implplan/SPRINTS.md.
- Update this plan whenever scope, dependencies, or guardrails change.

353
docs/modules/ci/recipes.md Executable file
View File

@@ -0,0 +1,353 @@
# Stella Ops CI Recipes — (2025‑08‑04)
## 0 · Key variables (export these once)
| Variable | Meaning | Typical value |
| ------------- | --------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------- |
| `STELLA_URL` | Host that: ① stores the **CLI** & **SBOM‑builder** images under `/registry` **and** ② receives API calls at `https://$STELLA_URL` | `stella-ops.ci.acme.example` |
| `DOCKER_HOST` | How containers reach your Docker daemon (because we no longer mount `/var/run/docker.sock`) | `tcp://docker:2375` |
| `WORKSPACE` | Directory where the pipeline stores artefacts (SBOM file) | `$(pwd)` |
| `IMAGE` | The image you are building & scanning | `acme/backend:sha-${COMMIT_SHA}` |
| `SBOM_FILE` | Immutable SBOM name – `<image-ref>‑YYYYMMDDThhmmssZ.sbom.json` | `acme_backend_sha‑abc123‑20250804T153050Z.sbom.json` |
> **Authority graph scopes note (2025-10-27):** CI stages that spin up the Authority compose profile now rely on the checked-in `etc/authority.yaml`. Before running integration smoke jobs, inject real secrets for every `etc/secrets/*.secret` file (Cartographer, Graph API, Policy Engine, Concelier, Excititor). The repository defaults contain `*-change-me` placeholders and Authority will reject tokens if those secrets are not overridden. Reissue CI tokens that previously used `policy:write`/`policy:submit`/`policy:edit` scopes—new bundles must request `policy:read`, `policy:author`, `policy:review`, `policy:simulate`, and (`policy:approve`/`policy:operate`/`policy:activate` when pipelines promote policies).
```bash
export STELLA_URL="stella-ops.ci.acme.example"
export DOCKER_HOST="tcp://docker:2375" # Jenkins/Circle often expose it like this
export WORKSPACE="$(pwd)"
export IMAGE="acme/backend:sha-${COMMIT_SHA}"
export SBOM_FILE="$(echo "${IMAGE}" | tr '/:+' '__')-$(date -u +%Y%m%dT%H%M%SZ).sbom.json"
```
---
## 1 · SBOM creation strategies
### Option A – **Buildx attested SBOM** (preferred if you can use BuildKit)
You pass **two build args** so the Dockerfile can run the builder and copy the result out of the build context.
```bash
docker buildx build \
--build-arg STELLA_SBOM_BUILDER="$STELLA_URL/registry/stella-sbom-builder:latest" \
--provenance=true --sbom=true \
--build-arg SBOM_FILE="$SBOM_FILE" \
-t "$IMAGE" .
```
**If you **cannot** use Buildx, use Option B below.** The older “run a builder stage inside the Dockerfile” pattern is unreliable for producing an SBOM of the final image.
```Dockerfile
ARG STELLA_SBOM_BUILDER
ARG SBOM_FILE
FROM $STELLA_SBOM_BUILDER as sbom
ARG IMAGE
ARG SBOM_FILE
RUN $STELLA_SBOM_BUILDER build --image $IMAGE --output /out/$SBOM_FILE
# ---- actual build stages … ----
FROM alpine:3.20
COPY --from=sbom /out/$SBOM_FILE / # (optional) keep or discard
# (rest of your Dockerfile)
```
### Option B – **External builder step** (works everywhere; recommended baseline if Buildx isn’t available)
*(keep this block if your pipeline already has an image‑build step that you can’t modify)*
```bash
docker run --rm \
-e DOCKER_HOST="$DOCKER_HOST" \ # let builder reach the daemon remotely
-v "$WORKSPACE:/workspace" \ # place SBOM beside the source code
"$STELLA_URL/registry/stella-sbom-builder:latest" \
build --image "$IMAGE" --output "/workspace/${SBOM_FILE}"
```
---
## 2 · Scan the image & upload results
```bash
docker run --rm \
-e DOCKER_HOST="$DOCKER_HOST" \ # remote‑daemon pointer
-v "$WORKSPACE/${SBOM_FILE}:/${SBOM_FILE}:ro" \ # mount SBOM under same name at container root
-e STELLA_OPS_URL="https://${STELLA_URL}" \ # where the CLI posts findings
"$STELLA_URL/registry/stella-cli:latest" \
scan --sbom "/${SBOM_FILE}" "$IMAGE"
```
The CLI returns **exit 0** if policies pass, **>0** if blocked — perfect for failing the job.
---
## 3 · CI templates
Below are minimal, cut‑and‑paste snippets.
**Feel free to delete Option B** if you adopt Option A.
### 3.1 Jenkins (Declarative Pipeline)
```groovy
pipeline {
agent { docker { image 'docker:25' args '--privileged' } } // gives us /usr/bin/docker
environment {
STELLA_URL = 'stella-ops.ci.acme.example'
DOCKER_HOST = 'tcp://docker:2375'
IMAGE = "acme/backend:${env.BUILD_NUMBER}"
SBOM_FILE = "acme_backend_${env.BUILD_NUMBER}-${new Date().format('yyyyMMdd\'T\'HHmmss\'Z\'', TimeZone.getTimeZone('UTC'))}.sbom.json"
}
stages {
stage('Build image + SBOM (Option A)') {
steps {
sh '''
docker build \
--build-arg STELLA_SBOM_BUILDER="$STELLA_URL/registry/stella-sbom-builder:latest" \
--build-arg SBOM_FILE="$SBOM_FILE" \
-t "$IMAGE" .
'''
}
}
/* ---------- Option B fallback (when you must keep the existing build step as‑is) ----------
stage('SBOM builder (Option B)') {
steps {
sh '''
docker run --rm -e DOCKER_HOST="$DOCKER_HOST" \
-v "$WORKSPACE:/workspace" \
"$STELLA_URL/registry/stella-sbom-builder:latest" \
build --image "$IMAGE" --output "/workspace/${SBOM_FILE}"
'''
}
}
------------------------------------------------------------------------------------------ */
stage('Scan & upload') {
steps {
sh '''
docker run --rm -e DOCKER_HOST="$DOCKER_HOST" \
-v "$WORKSPACE/${SBOM_FILE}:/${SBOM_FILE}:ro" \
-e STELLA_OPS_URL="https://$STELLA_URL" \
"$STELLA_URL/registry/stella-cli:latest" \
scan --sbom "/${SBOM_FILE}" "$IMAGE"
'''
}
}
}
}
```
---
### 3.2 CircleCI `.circleci/config.yml`
```yaml
version: 2.1
jobs:
stella_scan:
docker:
- image: cimg/base:stable # baremetal image with Docker CLI
environment:
STELLA_URL: stella-ops.ci.acme.example
DOCKER_HOST: tcp://docker:2375 # Circle’s “remote Docker” socket
steps:
- checkout
- run:
name: Compute vars
command: |
echo 'export IMAGE="acme/backend:${CIRCLE_SHA1}"' >> $BASH_ENV
echo 'export SBOM_FILE="$(echo acme/backend:${CIRCLE_SHA1} | tr "/:+" "__")-$(date -u +%Y%m%dT%H%M%SZ).sbom.json"' >> $BASH_ENV
- run:
name: Build image + SBOM (Option A)
command: |
docker build \
--build-arg STELLA_SBOM_BUILDER="$STELLA_URL/registry/stella-sbom-builder:latest" \
--build-arg SBOM_FILE="$SBOM_FILE" \
-t "$IMAGE" .
# --- Option B fallback (when you must keep the existing build step as‑is) ---
#- run:
# name: SBOM builder (Option B)
# command: |
# docker run --rm -e DOCKER_HOST="$DOCKER_HOST" \
# -v "$PWD:/workspace" \
# "$STELLA_URL/registry/stella-sbom-builder:latest" \
# build --image "$IMAGE" --output "/workspace/${SBOM_FILE}"
- run:
name: Scan
command: |
docker run --rm -e DOCKER_HOST="$DOCKER_HOST" \
-v "$PWD/${SBOM_FILE}:/${SBOM_FILE}:ro" \
-e STELLA_OPS_URL="https://$STELLA_URL" \
"$STELLA_URL/registry/stella-cli:latest" \
scan --sbom "/${SBOM_FILE}" "$IMAGE"
workflows:
stella:
jobs: [stella_scan]
```
---
### 3.3 Gitea Actions `.gitea/workflows/stella.yml`
*(Gitea 1.22+ ships native Actions compatible with GitHub syntax)*
```yaml
name: Stella Scan
on: [push]
jobs:
stella:
runs-on: ubuntu-latest
env:
STELLA_URL: ${{ secrets.STELLA_URL }}
DOCKER_HOST: tcp://docker:2375 # provided by the docker:dind service
services:
docker:
image: docker:dind
options: >-
--privileged
steps:
- uses: actions/checkout@v4
- name: Compute vars
id: vars
run: |
echo "IMAGE=ghcr.io/${{ gitea.repository }}:${{ gitea.sha }}" >> $GITEA_OUTPUT
echo "SBOM_FILE=$(echo ghcr.io/${{ gitea.repository }}:${{ gitea.sha }} | tr '/:+' '__')-$(date -u +%Y%m%dT%H%M%SZ).sbom.json" >> $GITEA_OUTPUT
- name: Build image + SBOM (Option A)
run: |
docker build \
--build-arg STELLA_SBOM_BUILDER="${STELLA_URL}/registry/stella-sbom-builder:latest" \
--build-arg SBOM_FILE="${{ steps.vars.outputs.SBOM_FILE }}" \
-t "${{ steps.vars.outputs.IMAGE }}" .
# --- Option B fallback (when you must keep the existing build step as‑is) ---
#- name: SBOM builder (Option B)
# run: |
# docker run --rm -e DOCKER_HOST="$DOCKER_HOST" \
# -v "$(pwd):/workspace" \
# "${STELLA_URL}/registry/stella-sbom-builder:latest" \
# build --image "${{ steps.vars.outputs.IMAGE }}" --output "/workspace/${{ steps.vars.outputs.SBOM_FILE }}"
- name: Scan
run: |
docker run --rm -e DOCKER_HOST="$DOCKER_HOST" \
-v "$(pwd)/${{ steps.vars.outputs.SBOM_FILE }}:/${{ steps.vars.outputs.SBOM_FILE }}:ro" \
-e STELLA_OPS_URL="https://${STELLA_URL}" \
"${STELLA_URL}/registry/stella-cli:latest" \
scan --sbom "/${{ steps.vars.outputs.SBOM_FILE }}" "${{ steps.vars.outputs.IMAGE }}"
```
---
## 4 · Docs CI (Gitea Actions & Offline Mirror)
StellaOps ships a dedicated Docs workflow at `.gitea/workflows/docs.yml`. When mirroring the pipeline offline or running it locally, install the same toolchain so markdown linting, schema validation, and HTML preview stay deterministic.
### 4.1 Toolchain bootstrap
```bash
# Node.js 20.x is required; install once per runner
npm install --no-save \
markdown-link-check \
remark-cli \
remark-preset-lint-recommended \
ajv \
ajv-cli \
ajv-formats
# Python 3.11+ powers the preview renderer
python -m pip install --upgrade pip
python -m pip install markdown pygments
```
> **No `pip` available?** Some hardened Python builds (including the repo’s `tmp/docenv`
> interpreter) ship without `pip`/`ensurepip`. In that case download the pure‑Python
> sdists (e.g. `Markdown-3.x.tar.gz`, `pygments-2.x.tar.gz`) and extract their
> packages directly into the virtualenv’s `lib/python*/site-packages/` folder.
> This keeps the renderer working even when package managers are disabled.
**Offline tip.** Add the packages above to your artifact mirror (for example `ops/devops/offline-kit.json`) so runners can install them via `npm --offline` / `pip --no-index`.
### 4.2 Schema validation step
Ajv compiles every event schema to guard against syntax or format regressions. The workflow uses `ajv-formats` for UUID/date-time support.
```bash
for schema in docs/events/*.json; do
npx ajv compile -c ajv-formats -s "$schema"
done
```
Run this loop before committing schema changes. For new references, append `-r additional-file.json` so CI and local runs stay aligned.
### 4.3 Preview build
```bash
python scripts/render_docs.py --source docs --output artifacts/docs-preview --clean
```
Host the resulting bundle via any static file server for review (for example `python -m http.server`).
### 4.4 Publishing checklist
- [ ] Toolchain installs succeed without hitting the public internet (mirror or cached tarballs).
- [ ] Ajv validation passes for `scanner.report.ready@1`, `scheduler.rescan.delta@1`, `attestor.logged@1`.
- [ ] Markdown link check (`npx markdown-link-check`) reports no broken references.
- [ ] Preview bundle archived (or attached) for stakeholders.
### 4.5 Policy DSL lint stage
Policy Engine v2 pipelines now fail fast if policy documents are malformed. After checkout and dotnet restore, run:
```bash
dotnet run \
--project src/Tools/PolicyDslValidator/PolicyDslValidator.csproj \
-- \
--strict docs/examples/policies/*.yaml
```
- `--strict` treats warnings as errors so missing metadata doesn’t slip through.
- The validator accepts globs, so you can point it at tenant policy directories later (`policies/**/*.yaml`).
- Exit codes follow UNIX conventions: `0` success, `1` parse/errors, `2` warnings when `--strict` is set, `64` usage mistakes.
Capture the validator output as part of your build logs; Support uses it when triaging policy rollout issues.
### 4.6 Policy simulation smoke
Catch unexpected policy regressions by exercising a small set of golden SBOM findings via the simulation smoke tool:
```bash
dotnet run \
--project src/Tools/PolicySimulationSmoke/PolicySimulationSmoke.csproj \
-- \
--scenario-root samples/policy/simulations \
--output artifacts/policy-simulations
```
- The tool loads each `scenario.json` under `samples/policy/simulations`, evaluates the referenced policy, and fails the build if projected verdicts change.
- In CI the command runs twice (to `run1/` and `run2/`) and `diff -u` compares the summaries—any mismatch signals a determinism regression.
- Artifacts land in `artifacts/policy-simulations/policy-simulation-summary.json`; upload them for later inspection (see CI workflow).
- Expand scenarios by copying real-world findings into the samples directory—ensure expected statuses are recorded so regressions trip the pipeline.
---
## 5 · Troubleshooting cheat‑sheet
| Symptom | Root cause | First things to try |
| ------------------------------------- | --------------------------- | --------------------------------------------------------------- |
| `no such host $STELLA_URL` | DNS typo or VPN outage | `ping $STELLA_URL` from runner |
| `connection refused` when CLI uploads | Port 443 blocked | open firewall / check ingress |
| `failed to stat /<sbom>.json` | SBOM wasn’t produced | Did Option A actually run builder? If not, enable Option B |
| `registry unauthorized` | Runner lacks registry creds | `docker login $STELLA_URL/registry` (store creds in CI secrets) |
| Non‑zero scan exit | Blocking vuln/licence | Open project in Ops UI → triage or waive |
---
### Change log
* **2025‑10‑18** – Documented Docs CI toolchain (Ajv validation, static preview) and offline checklist.
* **2025‑08‑04** – Variable clean‑up, removed Docker‑socket & cache mounts, added Jenkins / CircleCI / Gitea examples, clarified Option B comment.