Resolve Concelier/Excititor merge conflicts

2025-10-20 14:19:25 +03:00
parent 67d581d2e8 5fd4032c7c
commit 09b6a28172
2687 changed files with 212646 additions and 85913 deletions
--- a/docs/10_CONCELIER_CLI_QUICKSTART.md
+++ b/docs/10_CONCELIER_CLI_QUICKSTART.md
@@ -0,0 +1,289 @@
+# 10 · Concelier + CLI Quickstart
+
+This guide walks through configuring the Concelier web service and the `stellaops-cli`
+tool so an operator can ingest advisories, merge them, and publish exports from a
+single workstation. It focuses on deployment-facing surfaces only (configuration,
+runtime wiring, CLI usage) and leaves connector/internal customization for later.
+
+---
+
+## 0 · Prerequisites
+
+- .NET SDK **10.0.100-preview** (matches `global.json`)
+- MongoDB instance reachable from the host (local Docker or managed)
+- `trivy-db` binary on `PATH` for Trivy exports (and `oras` if publishing to OCI)
+- Plugin assemblies present in `StellaOps.Concelier.PluginBinaries/` (already included in the repo)
+- Optional: Docker/Podman runtime if you plan to run scanners locally
+
+> **Tip** – air-gapped installs should preload `trivy-db` and `oras` binaries into the
+> runner image since Concelier never fetches them dynamically.
+
+---
+
+## 1 · Configure Concelier
+
+1. Copy the sample config to the expected location (CI/CD pipelines can stamp values
+   into this file during deployment—see the “Deployment automation” note below):
+
+   ```bash
+   mkdir -p etc
+   cp etc/concelier.yaml.sample etc/concelier.yaml
+   ```
+
+2. Edit `etc/concelier.yaml` and update the MongoDB DSN (and optional database name).
+   The default template configures plug-in discovery to look in `StellaOps.Concelier.PluginBinaries/`
+   and disables remote telemetry exporters by default.
+
+3. (Optional) Override settings via environment variables. All keys are prefixed with
+   `CONCELIER_`. Example:
+
+   ```bash
+   export CONCELIER_STORAGE__DSN="mongodb://user:pass@mongo:27017/concelier"
+   export CONCELIER_TELEMETRY__ENABLETRACING=false
+   ```
+
+4. Start the web service from the repository root:
+
+   ```bash
+ dotnet run --project src/StellaOps.Concelier.WebService
+  ```
+
+   On startup Concelier validates the options, boots MongoDB indexes, loads plug-ins,
+   and exposes:
+
+   - `GET /health` – returns service status and telemetry settings
+   - `GET /ready` – performs a MongoDB `ping`
+   - `GET /jobs` + `POST /jobs/{kind}` – inspect and trigger connector/export jobs
+
+  > **Security note** – authentication now ships via StellaOps Authority. Keep
+  > `authority.allowAnonymousFallback: true` only during the staged rollout and
+  > disable it before **2025-12-31 UTC** so tokens become mandatory.
+
+### Authority companion configuration (preview)
+
+1. Copy the Authority sample configuration:
+
+   ```bash
+   cp etc/authority.yaml.sample etc/authority.yaml
+   ```
+
+2. Update the issuer URL, token lifetimes, and plug-in descriptors to match your
+   environment. Authority expects per-plugin manifests in `etc/authority.plugins/`;
+   sample `standard.yaml` and `ldap.yaml` files are provided as starting points.
+   For air-gapped installs keep the default plug-in binary directory
+   (`../StellaOps.Authority.PluginBinaries`) so packaged plug-ins load without outbound access.
+
+3. Environment variables prefixed with `STELLAOPS_AUTHORITY_` override individual
+   fields. Example:
+
+   ```bash
+   export STELLAOPS_AUTHORITY__ISSUER="https://authority.stella-ops.local"
+   export STELLAOPS_AUTHORITY__PLUGINDIRECTORIES__0="/srv/authority/plugins"
+   ```
+
+---
+
+## 2 · Configure the CLI
+
+The CLI reads configuration from JSON/YAML files *and* environment variables. The
+defaults live in `src/StellaOps.Cli/appsettings.json` and expect overrides at runtime.
+
+| Setting | Environment variable | Default | Purpose |
+| ------- | -------------------- | ------- | ------- |
+| `BackendUrl` | `STELLAOPS_BACKEND_URL` | _empty_ | Base URL of the Concelier web service |
+| `ApiKey` | `API_KEY` | _empty_ | Reserved for legacy key auth; leave empty when using Authority |
+| `ScannerCacheDirectory` | `STELLAOPS_SCANNER_CACHE_DIRECTORY` | `scanners` | Local cache folder |
+| `ResultsDirectory` | `STELLAOPS_RESULTS_DIRECTORY` | `results` | Where scan outputs are written |
+| `Authority.Url` | `STELLAOPS_AUTHORITY_URL` | _empty_ | StellaOps Authority issuer/token endpoint |
+| `Authority.ClientId` | `STELLAOPS_AUTHORITY_CLIENT_ID` | _empty_ | Client identifier for the CLI |
+| `Authority.ClientSecret` | `STELLAOPS_AUTHORITY_CLIENT_SECRET` | _empty_ | Client secret (omit when using username/password grant) |
+| `Authority.Username` | `STELLAOPS_AUTHORITY_USERNAME` | _empty_ | Username for password grant flows |
+| `Authority.Password` | `STELLAOPS_AUTHORITY_PASSWORD` | _empty_ | Password for password grant flows |
+| `Authority.Scope` | `STELLAOPS_AUTHORITY_SCOPE` | `concelier.jobs.trigger` | OAuth scope requested for backend operations |
+| `Authority.TokenCacheDirectory` | `STELLAOPS_AUTHORITY_TOKEN_CACHE_DIR` | `~/.stellaops/tokens` | Directory that persists cached tokens |
+| `Authority.Resilience.EnableRetries` | `STELLAOPS_AUTHORITY_ENABLE_RETRIES` | `true` | Toggle Polly retry handler for Authority HTTP calls |
+| `Authority.Resilience.RetryDelays` | `STELLAOPS_AUTHORITY_RETRY_DELAYS` | `1s,2s,5s` | Comma- or space-separated backoff delays (hh:mm:ss) |
+| `Authority.Resilience.AllowOfflineCacheFallback` | `STELLAOPS_AUTHORITY_ALLOW_OFFLINE_CACHE_FALLBACK` | `true` | Allow CLI to reuse cached discovery/JWKS metadata when Authority is offline |
+| `Authority.Resilience.OfflineCacheTolerance` | `STELLAOPS_AUTHORITY_OFFLINE_CACHE_TOLERANCE` | `00:10:00` | Additional tolerance window applied to cached metadata |
+
+Example bootstrap:
+
+```bash
+export STELLAOPS_BACKEND_URL="http://localhost:5000"
+export STELLAOPS_RESULTS_DIRECTORY="$HOME/.stellaops/results"
+export STELLAOPS_AUTHORITY_URL="https://authority.local"
+export STELLAOPS_AUTHORITY_CLIENT_ID="concelier-cli"
+export STELLAOPS_AUTHORITY_CLIENT_SECRET="s3cr3t"
+dotnet run --project src/StellaOps.Cli -- db merge
+
+# Acquire a bearer token and confirm cache state
+dotnet run --project src/StellaOps.Cli -- auth login
+dotnet run --project src/StellaOps.Cli -- auth status
+dotnet run --project src/StellaOps.Cli -- auth whoami
+```
+
+Refer to `docs/dev/32_AUTH_CLIENT_GUIDE.md` for deeper guidance on tuning retry/offline settings and rollout checklists.
+
+To persist configuration, you can create `stellaops-cli.yaml` next to the binary or
+rely on environment variables for ephemeral runners.
+
+---
+
+## 3 · Operating Workflow
+
+1. **Trigger connector fetch stages**
+
+   ```bash
+   dotnet run --project src/StellaOps.Cli -- db fetch --source osv --stage fetch
+   dotnet run --project src/StellaOps.Cli -- db fetch --source osv --stage parse
+   dotnet run --project src/StellaOps.Cli -- db fetch --source osv --stage map
+   ```
+
+   Use `--mode resume` when continuing from a previous window:
+
+   ```bash
+   dotnet run --project src/StellaOps.Cli -- db fetch --source redhat --stage fetch --mode resume
+   ```
+
+2. **Merge canonical advisories**
+
+   ```bash
+   dotnet run --project src/StellaOps.Cli -- db merge
+   ```
+
+3. **Produce exports**
+
+   ```bash
+   # JSON tree (vuln-list style)
+   dotnet run --project src/StellaOps.Cli -- db export --format json
+
+   # Trivy DB (delta example)
+   dotnet run --project src/StellaOps.Cli -- db export --format trivy-db --delta
+   ```
+
+   Concelier always produces a deterministic OCI layout. The first run after a clean
+   bootstrap emits a **full** baseline; subsequent `--delta` runs reuse the previous
+   baseline’s blobs when only JSON manifests change. If the exporter detects that a
+   prior delta is still active (i.e., `LastDeltaDigest` is recorded) it automatically
+   upgrades the next run to a full export and resets the baseline so operators never
+   chain deltas indefinitely. The CLI exposes `--publish-full/--publish-delta` (for
+   ORAS pushes) and `--include-full/--include-delta` (for offline bundles) should you
+   need to override the defaults interactively.
+
+   **Smoke-check delta reuse:** after the first baseline completes, run the export a
+   second time with `--delta` and verify that the new directory reports `mode=delta`
+   while reusing the previous layer blob.
+
+   ```bash
+   export_root=${CONCELIER_EXPORT_ROOT:-exports/trivy}
+   base=$(ls -1d "$export_root"/* | sort | tail -n2 | head -n1)
+   delta=$(ls -1d "$export_root"/* | sort | tail -n1)
+
+   jq -r '.mode,.baseExportId' "$delta/metadata.json"
+
+   base_manifest=$(jq -r '.manifests[0].digest' "$base/index.json")
+   delta_manifest=$(jq -r '.manifests[0].digest' "$delta/index.json")
+   printf 'baseline manifest: %s\ndelta manifest:    %s\n' "$base_manifest" "$delta_manifest"
+
+   layer_digest=$(jq -r '.layers[0].digest' "$base/blobs/sha256/${base_manifest#sha256:}")
+   cmp "$base/blobs/sha256/${layer_digest#sha256:}" \
+       "$delta/blobs/sha256/${layer_digest#sha256:}"
+   ```
+
+   `cmp` returning exit code `0` confirms the delta export reuses the baseline’s
+   `db.tar.gz` layer instead of rebuilding it.
+
+4. **Manage scanners (optional)**
+
+   ```bash
+   dotnet run --project src/StellaOps.Cli -- scanner download --channel stable
+   dotnet run --project src/StellaOps.Cli -- scan run --entry scanners/latest/Scanner.dll --target ./sboms
+   dotnet run --project src/StellaOps.Cli -- scan upload --file results/scan-001.json
+   ```
+
+Add `--verbose` to any command for structured console logs. All commands honour
+`Ctrl+C` cancellation and exit with non-zero status codes when the backend returns
+a problem document.
+
+---
+
+## 4 · Verification Checklist
+
+- Concelier `/health` returns `"status":"healthy"` and Storage bootstrap is marked
+  complete after startup.
+- CLI commands return HTTP 202 with a `Location` header (job tracking URL) when
+  triggering Concelier jobs.
+- Export artefacts are materialised under the configured output directories and
+  their manifests record digests.
+- MongoDB contains the expected `document`, `dto`, `advisory`, and `export_state`
+  collections after a run.
+
+---
+
+## 5 · Deployment Automation
+
+- Treat `etc/concelier.yaml.sample` as the canonical template. CI/CD should copy it to
+  the deployment artifact and replace placeholders (DSN, telemetry endpoints, cron
+  overrides) with environment-specific secrets.
+- Keep secret material (Mongo credentials, OTLP tokens) outside of the repository;
+  inject them via secret stores or pipeline variables at stamp time.
+- When building container images, include `trivy-db` (and `oras` if used) so air-gapped
+  clusters do not need outbound downloads at runtime.
+
+---
+
+## 5 · Next Steps
+
+- Enable authority-backed authentication in non-production first. Set
+  `authority.enabled: true` while keeping `authority.allowAnonymousFallback: true`
+  to observe logs, then flip it to `false` before 2025-12-31 UTC to enforce tokens.
+- Automate the workflow above via CI/CD (compose stack or Kubernetes CronJobs).
+- Pair with the Concelier connector teams when enabling additional sources so their
+  module-specific requirements are pulled in safely.
+
+---
+
+## 6 · Authority Integration
+
+- Concelier now authenticates callers through StellaOps Authority using OAuth 2.0
+  resource server flows. Populate the `authority` block in `concelier.yaml`:
+
+  ```yaml
+  authority:
+    enabled: true
+    allowAnonymousFallback: false         # keep true only during the staged rollout window
+    issuer: "https://authority.example.org"
+    audiences:
+      - "api://concelier"
+    requiredScopes:
+      - "concelier.jobs.trigger"
+    clientId: "concelier-jobs"
+    clientSecretFile: "../secrets/concelier-jobs.secret"
+    clientScopes:
+      - "concelier.jobs.trigger"
+    bypassNetworks:
+      - "127.0.0.1/32"
+      - "::1/128"
+  ```
+
+- Store the client secret outside of source control. Either provide it via
+  `authority.clientSecret` (environment variable `CONCELIER_AUTHORITY__CLIENTSECRET`)
+  or point `authority.clientSecretFile` to a file mounted at runtime.
+- Cron jobs running on the same host can keep using the API thanks to the loopback
+  bypass mask. Add additional CIDR ranges as needed; every bypass is logged.
+- Export the same configuration to Kubernetes or systemd by setting environment
+  variables such as:
+
+  ```bash
+  export CONCELIER_AUTHORITY__ENABLED=true
+  export CONCELIER_AUTHORITY__ALLOWANONYMOUSFALLBACK=false
+  export CONCELIER_AUTHORITY__ISSUER="https://authority.example.org"
+  export CONCELIER_AUTHORITY__CLIENTID="concelier-jobs"
+  export CONCELIER_AUTHORITY__CLIENTSECRETFILE="/var/run/secrets/concelier/authority-client"
+  ```
+
+- CLI commands already pass `Authorization` headers when credentials are supplied.
+  Configure the CLI with matching Authority settings (`docs/09_API_CLI_REFERENCE.md`)
+  so that automation can obtain tokens with the same client credentials. Concelier
+  logs every job request with the client ID, subject (if present), scopes, and
+  a `bypass` flag so operators can audit cron traffic.