stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 2 Releases Wiki Activity

docs consolidation

Browse Source
This commit is contained in:
StellaOps Bot
2025-12-24 12:38:14 +02:00
parent 7503c19b8f
commit 9a08d10b89
215 changed files with 2188 additions and 9623 deletions
Download Patch File Download Diff File Expand all files Collapse all files

528
docs/10_CONCELIER_CLI_QUICKSTART.md
View File

@@ -1,426 +1,102 @@
# 10 · Concelier + CLI Quickstart
This guide walks through configuring the Concelier web service and the `stellaops-cli`
tool so an operator can ingest advisories, merge them, and publish exports from a
single workstation. It focuses on deployment-facing surfaces only (configuration,
runtime wiring, CLI usage) and leaves connector/internal customization for later.
---
## 0 · Prerequisites
- .NET SDK **10.0.100-preview** (matches `global.json`)
- PostgreSQL instance reachable from the host (local Docker or managed)
- `trivy-db` binary on `PATH` for Trivy exports (and `oras` if publishing to OCI)
- Plugin assemblies present in `StellaOps.Concelier.PluginBinaries/` (already included in the repo)
- Optional: Docker/Podman runtime if you plan to run scanners locally
> **Tip** – air-gapped installs should preload `trivy-db` and `oras` binaries into the
> runner image since Concelier never fetches them dynamically.
---
## 1 · Configure Concelier
1. Copy the sample config to the expected location (CI/CD pipelines can stamp values
into this file during deployment—see the “Deployment automation” note below):
```bash
mkdir -p etc
cp etc/concelier.yaml.sample etc/concelier.yaml
```
2. Edit `etc/concelier.yaml` and update the PostgreSQL DSN (and optional database name).
The default template configures plug-in discovery to look in `StellaOps.Concelier.PluginBinaries/`
and disables remote telemetry exporters by default.
3. (Optional) Override settings via environment variables. All keys are prefixed with
`CONCELIER_`. Example:
```bash
export CONCELIER_STORAGE__DSN="Host=localhost;Port=5432;Database=concelier;Username=user;Password=pass"
export CONCELIER_TELEMETRY__ENABLETRACING=false
```
4. Start the web service from the repository root:
```bash
dotnet run --project src/Concelier/StellaOps.Concelier.WebService
```
On startup Concelier validates the options, boots PostgreSQL indexes, loads plug-ins,
and exposes:
- `GET /health` – returns service status and telemetry settings
- `GET /ready` – performs a PostgreSQL `ping`
- `GET /jobs` + `POST /jobs/{kind}` – inspect and trigger connector/export jobs
> **Security note** – authentication now ships via StellaOps Authority. Keep
> `authority.allowAnonymousFallback: true` only during the staged rollout and
> disable it before **2025-12-31 UTC** so tokens become mandatory.
Rollout checkpoints for the two Authority toggles:
| Phase | `authority.enabled` | `authority.allowAnonymousFallback` | Goal | Observability focus |
| ----- | ------------------- | ---------------------------------- | ---- | ------------------- |
| **Validation (staging)** | `true` | `true` | Verify token issuance, CLI scopes, and audit log noise without breaking cron jobs. | Watch `Concelier.Authorization.Audit` for `bypass=True` events and scope gaps; confirm CLI `auth status` succeeds. |
| **Cutover rehearsal** | `true` | `false` | Exercise production-style enforcement before the deadline; ensure only approved maintenance ranges remain in `bypassNetworks`. | Expect some HTTP 401s; verify `web.jobs.triggered` metrics flatten for unauthenticated calls and audit logs highlight missing tokens. |
| **Enforced (steady state)** | `true` | `false` | Production baseline after the 2025-12-31 UTC cutoff. | Alert on new `bypass=True` entries and on repeated 401 bursts; correlate with Authority availability dashboards. |
### Authority companion configuration (preview)
1. Copy the Authority sample configuration:
```bash
cp etc/authority.yaml.sample etc/authority.yaml
```
2. Update the issuer URL, token lifetimes, and plug-in descriptors to match your
environment. Authority expects per-plugin manifests in `etc/authority.plugins/`;
sample `standard.yaml` and `ldap.yaml` files are provided as starting points.
For air-gapped installs keep the default plug-in binary directory
(`../StellaOps.Authority.PluginBinaries`) so packaged plug-ins load without outbound access.
3. Environment variables prefixed with `STELLAOPS_AUTHORITY_` override individual
fields. Example:
```bash
export STELLAOPS_AUTHORITY__ISSUER="https://authority.stella-ops.local"
export STELLAOPS_AUTHORITY__PLUGINDIRECTORIES__0="/srv/authority/plugins"
```
---
## 2 · Configure the CLI
The CLI reads configuration from JSON/YAML files *and* environment variables. The
defaults live in `src/Cli/StellaOps.Cli/appsettings.json` and expect overrides at runtime.
| Setting | Environment variable | Default | Purpose |
| ------- | -------------------- | ------- | ------- |
| `BackendUrl` | `STELLAOPS_BACKEND_URL` | _empty_ | Base URL of the Concelier web service |
| `ApiKey` | `API_KEY` | _empty_ | Reserved for legacy key auth; leave empty when using Authority |
| `ScannerCacheDirectory` | `STELLAOPS_SCANNER_CACHE_DIRECTORY` | `scanners` | Local cache folder |
| `ResultsDirectory` | `STELLAOPS_RESULTS_DIRECTORY` | `results` | Where scan outputs are written |
| `Authority.Url` | `STELLAOPS_AUTHORITY_URL` | _empty_ | StellaOps Authority issuer/token endpoint |
| `Authority.ClientId` | `STELLAOPS_AUTHORITY_CLIENT_ID` | _empty_ | Client identifier for the CLI |
| `Authority.ClientSecret` | `STELLAOPS_AUTHORITY_CLIENT_SECRET` | _empty_ | Client secret (omit when using username/password grant) |
| `Authority.Username` | `STELLAOPS_AUTHORITY_USERNAME` | _empty_ | Username for password grant flows |
| `Authority.Password` | `STELLAOPS_AUTHORITY_PASSWORD` | _empty_ | Password for password grant flows |
| `Authority.Scope` | `STELLAOPS_AUTHORITY_SCOPE` | `concelier.jobs.trigger advisory:ingest` | Space-separated OAuth scopes requested for backend operations |
| `Authority.TokenCacheDirectory` | `STELLAOPS_AUTHORITY_TOKEN_CACHE_DIR` | `~/.stellaops/tokens` | Directory that persists cached tokens |
| `Authority.Resilience.EnableRetries` | `STELLAOPS_AUTHORITY_ENABLE_RETRIES` | `true` | Toggle Polly retry handler for Authority HTTP calls |
| `Authority.Resilience.RetryDelays` | `STELLAOPS_AUTHORITY_RETRY_DELAYS` | `1s,2s,5s` | Comma- or space-separated backoff delays (hh:mm:ss) |
| `Authority.Resilience.AllowOfflineCacheFallback` | `STELLAOPS_AUTHORITY_ALLOW_OFFLINE_CACHE_FALLBACK` | `true` | Allow CLI to reuse cached discovery/JWKS metadata when Authority is offline |
| `Authority.Resilience.OfflineCacheTolerance` | `STELLAOPS_AUTHORITY_OFFLINE_CACHE_TOLERANCE` | `00:10:00` | Additional tolerance window applied to cached metadata |
Example bootstrap:
```bash
export STELLAOPS_BACKEND_URL="http://localhost:5000"
export STELLAOPS_RESULTS_DIRECTORY="$HOME/.stellaops/results"
export STELLAOPS_AUTHORITY_URL="https://authority.local"
export STELLAOPS_AUTHORITY_CLIENT_ID="concelier-cli"
export STELLAOPS_AUTHORITY_CLIENT_SECRET="s3cr3t"
export STELLAOPS_AUTHORITY_SCOPE="concelier.jobs.trigger advisory:ingest advisory:read"
dotnet run --project src/Cli/StellaOps.Cli -- db merge
# Acquire a bearer token and confirm cache state
dotnet run --project src/Cli/StellaOps.Cli -- auth login
dotnet run --project src/Cli/StellaOps.Cli -- auth status
dotnet run --project src/Cli/StellaOps.Cli -- auth whoami
```
Refer to `docs/dev/32_AUTH_CLIENT_GUIDE.md` for deeper guidance on tuning retry/offline settings and rollout checklists.
To persist configuration, you can create `stellaops-cli.yaml` next to the binary or
rely on environment variables for ephemeral runners.
---
## 3 · Operating Workflow
1. **Trigger connector fetch stages**
```bash
dotnet run --project src/Cli/StellaOps.Cli -- db fetch --source osv --stage fetch
dotnet run --project src/Cli/StellaOps.Cli -- db fetch --source osv --stage parse
dotnet run --project src/Cli/StellaOps.Cli -- db fetch --source osv --stage map
```
Use `--mode resume` when continuing from a previous window:
```bash
dotnet run --project src/Cli/StellaOps.Cli -- db fetch --source redhat --stage fetch --mode resume
```
2. **Merge canonical advisories**
```bash
dotnet run --project src/Cli/StellaOps.Cli -- db merge
```
3. **Produce exports**
```bash
# JSON tree (vuln-list style)
dotnet run --project src/Cli/StellaOps.Cli -- db export --format json
# Trivy DB (delta example)
dotnet run --project src/Cli/StellaOps.Cli -- db export --format trivy-db --delta
```
Concelier always produces a deterministic OCI layout. The first run after a clean
bootstrap emits a **full** baseline; subsequent `--delta` runs reuse the previous
baseline’s blobs when only JSON manifests change. If the exporter detects that a
prior delta is still active (i.e., `LastDeltaDigest` is recorded) it automatically
upgrades the next run to a full export and resets the baseline so operators never
chain deltas indefinitely. The CLI exposes `--publish-full/--publish-delta` (for
ORAS pushes) and `--include-full/--include-delta` (for offline bundles) should you
need to override the defaults interactively.
**Smoke-check delta reuse:** after the first baseline completes, run the export a
second time with `--delta` and verify that the new directory reports `mode=delta`
while reusing the previous layer blob.
```bash
export_root=${CONCELIER_EXPORT_ROOT:-exports/trivy}
base=$(ls -1d "$export_root"/* | sort | tail -n2 | head -n1)
delta=$(ls -1d "$export_root"/* | sort | tail -n1)
jq -r '.mode,.baseExportId' "$delta/metadata.json"
base_manifest=$(jq -r '.manifests[0].digest' "$base/index.json")
delta_manifest=$(jq -r '.manifests[0].digest' "$delta/index.json")
printf 'baseline manifest: %s\ndelta manifest: %s\n' "$base_manifest" "$delta_manifest"
layer_digest=$(jq -r '.layers[0].digest' "$base/blobs/sha256/${base_manifest#sha256:}")
cmp "$base/blobs/sha256/${layer_digest#sha256:}" \
"$delta/blobs/sha256/${layer_digest#sha256:}"
```
`cmp` returning exit code `0` confirms the delta export reuses the baseline’s
`db.tar.gz` layer instead of rebuilding it.
4. **Verify guard compliance**
```bash
export STELLA_TENANT="${STELLA_TENANT:-tenant-a}"
dotnet run --project src/Cli/StellaOps.Cli -- aoc verify \
--since 24h \
--format table \
--tenant "$STELLA_TENANT"
# Optional: capture JSON evidence for pipelines/audits
dotnet run --project src/Cli/StellaOps.Cli -- aoc verify \
--since 7d \
--limit 100 \
--format json \
--export artifacts/aoc-verify.json \
--tenant "$STELLA_TENANT"
```
The CLI exits with `0` when no violations are detected. Guard failures map
to `ERR_AOC_00x` codes (`11…17`), while truncated results return `18`. Use
`--sources`/`--codes` to focus on noisy connectors and feed the exported JSON
into dashboards or evidence lockers for compliance reviews.
5. **Pre-flight individual payloads**
```bash
stella sources ingest --dry-run \
--source redhat \
--input ./fixtures/redhat/RHSA-2025-9999.json \
--tenant "$STELLA_TENANT" \
--format json \
--output artifacts/redhat-dry-run.json
```
Exit code `0` confirms the candidate document is AOC compliant. Any guard
violation is emitted as deterministic `ERR_AOC_00x` exit codes (`11…17`);
reuse the exported JSON in PRs or incident timelines to show offending paths.
6. **Manage scanners (optional)**
```bash
dotnet run --project src/Cli/StellaOps.Cli -- scanner download --channel stable
dotnet run --project src/Cli/StellaOps.Cli -- scan run --entry scanners/latest/Scanner.dll --target ./sboms
dotnet run --project src/Cli/StellaOps.Cli -- scan upload --file results/scan-001.json
```
Add `--verbose` to any command for structured console logs. All commands honour
`Ctrl+C` cancellation and exit with non-zero status codes when the backend returns
a problem document.
---
## 4 · Verification Checklist
- Concelier `/health` returns `"status":"healthy"` and Storage bootstrap is marked
complete after startup.
- CLI commands return HTTP 202 with a `Location` header (job tracking URL) when
triggering Concelier jobs.
- Export artefacts are materialised under the configured output directories and
their manifests record digests.
- PostgreSQL contains the expected `document`, `dto`, `advisory`, and `export_state`
tables after a run.
---
## 5 · Deployment Automation
- Treat `etc/concelier.yaml.sample` as the canonical template. CI/CD should copy it to
the deployment artifact and replace placeholders (DSN, telemetry endpoints, cron
overrides) with environment-specific secrets.
- Keep secret material (PostgreSQL credentials, OTLP tokens) outside of the repository;
inject them via secret stores or pipeline variables at stamp time.
- When building container images, include `trivy-db` (and `oras` if used) so air-gapped
clusters do not need outbound downloads at runtime.
---
## 5 · Next Steps
- Enable authority-backed authentication in non-production first. Set
`authority.enabled: true` while keeping `authority.allowAnonymousFallback: true`
to observe logs, then flip it to `false` before 2025-12-31 UTC to enforce tokens.
- Automate the workflow above via CI/CD (compose stack or Kubernetes CronJobs).
- Pair with the Concelier connector teams when enabling additional sources so their
module-specific requirements are pulled in safely.
---
## 6 · Authority Integration
- Concelier now authenticates callers through StellaOps Authority using OAuth 2.0
resource server flows. Populate the `authority` block in `concelier.yaml`:
```yaml
authority:
enabled: true
allowAnonymousFallback: false # keep true only during the staged rollout window
issuer: "https://authority.example.org"
audiences:
- "api://concelier"
requiredScopes:
- "concelier.jobs.trigger"
- "advisory:read"
- "advisory:ingest"
requiredTenants:
- "tenant-default"
clientId: "concelier-jobs"
clientSecretFile: "../secrets/concelier-jobs.secret"
clientScopes:
- "concelier.jobs.trigger"
- "advisory:read"
- "advisory:ingest"
bypassNetworks:
- "127.0.0.1/32"
- "::1/128"
```
- Store the client secret outside of source control. Either provide it via
`authority.clientSecret` (environment variable `CONCELIER_AUTHORITY__CLIENTSECRET`)
or point `authority.clientSecretFile` to a file mounted at runtime.
- Cron jobs running on the same host can keep using the API thanks to the loopback
bypass mask. Add additional CIDR ranges as needed; every bypass is logged.
- Export the same configuration to Kubernetes or systemd by setting environment
variables such as:
```bash
export CONCELIER_AUTHORITY__ENABLED=true
export CONCELIER_AUTHORITY__ALLOWANONYMOUSFALLBACK=false
export CONCELIER_AUTHORITY__ISSUER="https://authority.example.org"
export CONCELIER_AUTHORITY__CLIENTID="concelier-jobs"
export CONCELIER_AUTHORITY__CLIENTSECRETFILE="/var/run/secrets/concelier/authority-client"
export CONCELIER_AUTHORITY__REQUIREDSCOPES__0="concelier.jobs.trigger"
export CONCELIER_AUTHORITY__REQUIREDSCOPES__1="advisory:read"
export CONCELIER_AUTHORITY__REQUIREDSCOPES__2="advisory:ingest"
export CONCELIER_AUTHORITY__CLIENTSCOPES__0="concelier.jobs.trigger"
export CONCELIER_AUTHORITY__CLIENTSCOPES__1="advisory:read"
export CONCELIER_AUTHORITY__CLIENTSCOPES__2="advisory:ingest"
export CONCELIER_AUTHORITY__REQUIREDTENANTS__0="tenant-default"
```
- CLI commands already pass `Authorization` headers when credentials are supplied.
Configure the CLI with matching Authority settings (`docs/09_API_CLI_REFERENCE.md`)
so that automation can obtain tokens with the same client credentials. Concelier
logs every job request with the client ID, subject (if present), scopes, and
a `bypass` flag so operators can audit cron traffic.
- **Rollout checklist.**
1. Stage the integration with fallback enabled (`allowAnonymousFallback=true`) and confirm CLI/token issuance using `stella auth status`.
2. Follow the rehearsal pattern (`allowAnonymousFallback=false`) while monitoring `Concelier.Authorization.Audit` and `web.jobs.triggered`/`web.jobs.trigger.failed` metrics.
3. Lock in enforcement, review the audit runbook (`docs/modules/concelier/operations/authority-audit-runbook.md`), and document the bypass CIDR approvals in your change log.
---
## 7 · Policy Starter Pack (Day 1)
StellaOps provides a production-ready starter policy that blocks reachable HIGH/CRITICAL
vulnerabilities while respecting VEX statements and enforcing metadata quality gates.
### Quick Installation
```bash
# Install the starter policy pack
stellaops policy install starter-day1
# Verify the installation
stellaops policy list-packs
```
### One-Liner for Scans with Policy
```bash
# Scan with policy evaluation
stellaops scan run --image myregistry/myapp:latest --policy starter-day1
# Check policy verdict
stellaops verdict verify myregistry/myapp:latest
```
### What Starter Policy Does
| Finding Type | Action | Notes |
|--------------|--------|-------|
| Reachable HIGH/CRITICAL | **Block** | Unless VEX says `not_affected` |
| Reachable MEDIUM | **Warn** | Review recommended |
| Unreachable vulnerabilities | **Allow** | Logged for awareness |
| Unknowns > 5% | **Block** | Quality gate for SBOM coverage |
| Unsigned SBOM (prod) | **Block** | Integrity requirement |
### Environment Overrides
Apply environment-specific behavior:
```bash
# Development - warnings only, no signing required
stellaops scan run --image myapp:dev --policy starter-day1 --env development
# Production - full enforcement (default)
stellaops scan run --image myapp:prod --policy starter-day1 --env production
```
### Simulate Before Deploying
Test policy impact on existing scans without blocking:
```bash
stellaops policy simulate --policy policies/starter-day1.yaml --scan <scan-id>
```
### Distribution (Air-Gapped Environments)
For air-gapped installations, export and import policy packs as offline bundles:
```bash
# Export to offline bundle
stellaops policy export-bundle --policy policies/starter-day1.yaml \
--output starter-day1-bundle.tar.gz
# Import in air-gapped environment
stellaops policy import-bundle --bundle starter-day1-bundle.tar.gz
```
See `docs/policy/starter-guide.md` for detailed customization and migration guidance.
# Concelier + CLI Quickstart
This quickstart gets an operator to a working advisory ingestion loop:
- Run Concelier (advisory ingestion + deterministic normalization).
- Trigger ingestion/export jobs.
- Inspect results via the `stella` CLI.
This document stays high level and defers detailed configuration and connector behavior to the Concelier module dossier.
## 1) Prerequisites
- Deployment: follow `docs/21_INSTALL_GUIDE.md` (Compose profiles under `deploy/compose/`).
- Offline/air-gap: follow `docs/24_OFFLINE_KIT.md` and `docs/airgap/overview.md`.
- Local dev (optional): .NET SDK version pinned by `global.json`.
## 2) Run Concelier
### Option A: Run via deployment bundles (recommended)
Use the deterministic Compose profiles under `deploy/compose/` and enable Concelier in the selected profile.
Start here:
- `docs/21_INSTALL_GUIDE.md`
- `docs/modules/concelier/operations/`
### Option B: Run the service from source (dev/debug)
```bash
dotnet run --project src/Concelier/StellaOps.Concelier.WebService
```
Concelier reads `etc/concelier.yaml` by default (and supports environment overrides). See:
- `docs/modules/concelier/architecture.md`
- `docs/modules/concelier/operations/`
## 3) Configure Concelier (minimum)
1. Copy the sample config:
```bash
mkdir -p etc
cp etc/concelier.yaml.sample etc/concelier.yaml
```
2. Update storage/DSN and any connector configuration needed for your sources.
3. Keep configuration deterministic and offline-friendly (no hidden outbound calls in air-gap profiles).
Connector deep dives and operational guidance live under:
- `docs/modules/concelier/operations/connectors/`
## 4) Harden the `/jobs*` surface with Authority (recommended)
Concelier job triggers are operationally sensitive. In production-style installs, require Authority-issued tokens.
Operator entry point:
- `docs/modules/concelier/operations/authority-audit-runbook.md`
At minimum, ensure:
- Authority enforcement is enabled.
- Anonymous fallback is disabled outside controlled rollout windows.
- Any bypass CIDRs are explicitly approved and monitored.
## 5) Use the CLI for ingestion and exports
This guide uses `stella` as the CLI command name. If your packaging uses a different filename, add a local shim/symlink.
### 5.1 Point the CLI at Concelier
Set the backend base URL (example):
```bash
export STELLAOPS_BACKEND_URL="https://concelier.example.internal"
```
Authenticate using the configured Authority credentials:
```bash
stella auth login
stella auth whoami
```
See: `docs/modules/cli/guides/commands/auth.md`.
### 5.2 Trigger connector stages
Trigger a connector stage (example):
```bash
stella db fetch --source osv --stage fetch
stella db fetch --source osv --stage parse
stella db fetch --source osv --stage map
```
### 5.3 Reconcile merges (when needed)
```bash
stella db merge
```
### 5.4 Produce exports
```bash
stella db export --format json
```
See: `docs/modules/cli/guides/commands/db.md`.
### 5.5 Inspect advisory results
For read-only inspection (list/get/export), use:
- `docs/modules/cli/guides/commands/advisory.md`
## 6) Next links
- Concelier module dossier: `docs/modules/concelier/README.md`
- Concelier operations: `docs/modules/concelier/operations/`
- CLI command guides: `docs/modules/cli/guides/commands/`
- API + CLI reference index: `docs/09_API_CLI_REFERENCE.md`