Align AOC tasks for Excititor and Concelier
This commit is contained in:
master
root
@@ -1,286 +1,286 @@
|
||||
# Offline Update Kit (OUK) — Air‑Gap Bundle
|
||||
|
||||
<!--
|
||||
Build‑time variable injection:
|
||||
{{ quota_anon }} = 33
|
||||
{{ quota_token }} = 333
|
||||
{{ dotnet }} = "10 LTS"
|
||||
-->
|
||||
|
||||
The **Offline Update Kit** packages everything Stella Ops needs to run on a
|
||||
completely isolated network:
|
||||
|
||||
| Component | Contents |
|
||||
|-----------|----------|
|
||||
| **Merged vulnerability feeds** | OSV, GHSA plus optional NVD 2.0, CNNVD, CNVD, ENISA, JVN and BDU |
|
||||
| **Container images** | `stella-ops`, *Zastava* sidecar (x86‑64 & arm64) |
|
||||
| **Provenance** | Cosign signature, SPDX 2.3 SBOM, in‑toto SLSA attestation |
|
||||
| **Attested manifest** | `offline-manifest.json` + detached JWS covering bundle metadata, signed during export. |
|
||||
| **Delta patches** | Daily diff bundles keep size \< 350 MB |
|
||||
| **Scanner plug-ins** | OS analyzers plus the Node.js, Go, .NET, and Python language analyzers packaged under `plugins/scanner/analyzers/**` with manifests so Workers load deterministically offline. |
|
||||
| **Debug store** | `.debug` artefacts laid out under `debug/.build-id/<aa>/<rest>.debug` with `debug/debug-manifest.json` mapping build-ids to originating images for symbol retrieval. |
|
||||
| **Telemetry collector bundle** | `telemetry/telemetry-offline-bundle.tar.gz` plus `.sha256`, containing OTLP collector config, Helm/Compose overlays, and operator instructions. |
|
||||
|
||||
**RU BDU note:** ship the official Russian Trusted Root/Sub CA bundle (`certificates/russian_trusted_bundle.pem`) inside the kit so `concelier:httpClients:source.bdu:trustedRootPaths` can resolve it when the service runs in an air‑gapped network. Drop the most recent `vulxml.zip` alongside the kit if operators need a cold-start cache.
|
||||
|
||||
**Language analyzers:** the kit now carries the restart-only Node.js, Go, .NET, and Python analyzer plug-ins (`plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Node/`, `...Lang.Go/`, `...Lang.DotNet/`, `...Lang.Python/`). Drop the directories alongside Worker binaries so the unified plug-in catalog can load them without outbound fetches; Rust remains on the Wave 4 roadmap.
|
||||
|
||||
*Scanner core:* C# 12 on **.NET {{ dotnet }}**.
|
||||
*Imports are idempotent and atomic — no service downtime.*
|
||||
|
||||
## 0 · Prepare the debug store
|
||||
|
||||
Before packaging the Offline Kit, mirror the release debug artefacts (GNU build-id `.debug` files and the associated manifest) into the staging directory:
|
||||
|
||||
```bash
|
||||
./ops/offline-kit/mirror_debug_store.py \
|
||||
--release-dir out/release \
|
||||
--offline-kit-dir out/offline-kit
|
||||
```
|
||||
|
||||
The helper copies `debug/.build-id/**`, validates `debug/debug-manifest.json` against its recorded SHA-256, and writes `out/offline-kit/metadata/debug-store.json` with a short summary (platforms, artefact counts, sample build-ids). The command exits non-zero if an artefact referenced by the manifest is missing or has the wrong digest, so run it as part of every kit build.
|
||||
|
||||
---
|
||||
|
||||
## 0.1 · Automated packaging
|
||||
|
||||
The packaging workflow is scripted via `ops/offline-kit/build_offline_kit.py`.
|
||||
It verifies the release artefacts, runs the Python analyzer smoke suite, mirrors the debug store, and emits a deterministic tarball + manifest set.
|
||||
|
||||
```bash
|
||||
python ops/offline-kit/build_offline_kit.py \
|
||||
--version 2025.10.0 \
|
||||
--channel edge \
|
||||
--release-dir out/release \
|
||||
--staging-dir out/offline-kit/staging \
|
||||
--output-dir out/offline-kit/dist
|
||||
|
||||
# Optional: regenerate the telemetry collector bundle prior to packaging.
|
||||
python ops/devops/telemetry/package_offline_bundle.py --output out/telemetry/telemetry-offline-bundle.tar.gz
|
||||
```
|
||||
|
||||
Outputs:
|
||||
|
||||
- `stella-ops-offline-kit-<version>-<channel>.tar.gz` — bundle (mtime/uid/gid forced to zero for reproducibility)
|
||||
- `stella-ops-offline-kit-<version>-<channel>.tar.gz.sha256` — bundle digest
|
||||
- `manifest/offline-manifest.json` + `.sha256` — inventories every file in the bundle
|
||||
- `<bundle>.metadata.json` — descriptor consumed by the CLI/Console import tooling
|
||||
- `telemetry/telemetry-offline-bundle.tar.gz` + `.sha256` — packaged OTLP collector assets for environments without upstream access
|
||||
- `plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Python/*.sig` (+ `.sha256`) — Cosign signatures for the Python analyzer DLL and manifest
|
||||
|
||||
### Policy Gateway configuration bundle
|
||||
|
||||
- Copy `etc/policy-gateway.yaml` (or the `*.sample` template if you expect operators to override values) into `config/policy-gateway/policy-gateway.yaml` within the staging tree.
|
||||
- Include the gateway DPoP private key under `secrets/policy-gateway/policy-gateway-dpop.pem` and reference the location inside the manifest notes. Set the permissions explicitly (`chmod 600 secrets/policy-gateway/policy-gateway-dpop.pem`) so only the kit importer can read it; the importer will refuse keys that are broader.
|
||||
- Document the gateway base URL and activation verification steps in `docs/policy/gateway.md` (bundled alongside the kit). Operators can use those curl snippets to smoke-test pack CRUD once the Offline Kit is imported.
|
||||
- Ensure the Prometheus snapshot captured during packaging contains `policy_gateway_activation_requests_total` so auditors can reconcile activation attempts performed via the gateway during the validation window.
|
||||
|
||||
Provide `--cosign-key` / `--cosign-identity-token` (and optional `--cosign-password`) to generate Cosign signatures for both the tarball and manifest.
|
||||
|
||||
---
|
||||
|
||||
## 1 · Download & verify
|
||||
|
||||
```bash
|
||||
curl -LO https://get.stella-ops.org/ouk/stella-ops-offline-kit-<DATE>.tgz
|
||||
curl -LO https://get.stella-ops.org/ouk/stella-ops-offline-kit-<DATE>.tgz.sig
|
||||
curl -LO https://get.stella-ops.org/ouk/offline-manifest-<DATE>.json
|
||||
curl -LO https://get.stella-ops.org/ouk/offline-manifest-<DATE>.json.jws
|
||||
|
||||
cosign verify-blob \
|
||||
--key https://stella-ops.org/keys/cosign.pub \
|
||||
--signature stella-ops-offline-kit-<DATE>.tgz.sig \
|
||||
stella-ops-offline-kit-<DATE>.tgz
|
||||
````
|
||||
|
||||
**CLI shortcut.** `stellaops-cli offline kit pull --destination ./offline-kit` downloads the bundle, manifest, and detached signatures in one step, resumes partial transfers, and writes a `.metadata.json` summary for later import.
|
||||
|
||||
Verification prints **OK** and the SHA‑256 digest; cross‑check against the
|
||||
[changelog](https://git.stella-ops.org/stella-ops/offline-kit/-/releases).
|
||||
|
||||
Validate the attested manifest before distribution:
|
||||
|
||||
```bash
|
||||
cosign verify-blob \
|
||||
--key https://stella-ops.org/keys/cosign.pub \
|
||||
--signature offline-manifest-<DATE>.json.jws \
|
||||
offline-manifest-<DATE>.json
|
||||
|
||||
jq '.artifacts[] | {name, sha256, size, capturedAt}' offline-manifest-<DATE>.json
|
||||
```
|
||||
|
||||
The manifest enumerates every artefact (`name`, `sha256`, `size`, `capturedAt`) and is signed with the same key registry as Authority revocation bundles. Operators can ship the manifest alongside the tarball so downstream mirrors can re-verify without unpacking the kit.
|
||||
|
||||
Example excerpt (2025-10-23 kit) showing the Go and .NET analyzer plug-in payloads:
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Go/StellaOps.Scanner.Analyzers.Lang.Go.dll",
|
||||
"sha256": "a6dc850fc51151c8967ef46a3c4730f08b549667e041079431f39a8a72d0b641",
|
||||
"size": 33792,
|
||||
"capturedAt": "2025-10-23T00:00:00Z"
|
||||
}
|
||||
{
|
||||
"name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Go/StellaOps.Scanner.Analyzers.Lang.Go.pdb",
|
||||
"sha256": "6cbdabf155282f458b89edf267e7f6bb2441a93029aad7aad45c8a9ec58b1b3b",
|
||||
"size": 32152,
|
||||
"capturedAt": "2025-10-23T00:00:00Z"
|
||||
}
|
||||
{
|
||||
"name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Go/manifest.json",
|
||||
"sha256": "c19bfca2fcbb7cb18f1082b5d0d5a8f15fc799c648b50e95fce8d8b109ce48c9",
|
||||
"size": 622,
|
||||
"capturedAt": "2025-10-23T00:00:00Z"
|
||||
}
|
||||
{
|
||||
"name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.DotNet/StellaOps.Scanner.Analyzers.Lang.DotNet.dll",
|
||||
"sha256": "0734d23e33277ce2ccb596782d2d42cfe394b3d372dc34da9cb28b59df9b9d22",
|
||||
"size": 70144,
|
||||
"capturedAt": "2025-10-23T00:00:00Z"
|
||||
}
|
||||
{
|
||||
"name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.DotNet/StellaOps.Scanner.Analyzers.Lang.DotNet.pdb",
|
||||
"sha256": "b853c1ff4b196715f5bd1447e1a13edeb4940917527ec9bf153b5048da49abaf",
|
||||
"size": 40400,
|
||||
"capturedAt": "2025-10-23T00:00:00Z"
|
||||
}
|
||||
{
|
||||
"name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.DotNet/manifest.json",
|
||||
"sha256": "5d483885f825f01bfd9943dcf2889ec2e0beba38ede92ecfe67d4f506cf14e37",
|
||||
"size": 647,
|
||||
"capturedAt": "2025-10-23T00:00:00Z"
|
||||
}
|
||||
{
|
||||
"name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Python/StellaOps.Scanner.Analyzers.Lang.Python.dll",
|
||||
"sha256": "a4f558f363394096e3dd6263f35b180b93b4112f9cf616c05872da8a8657d518",
|
||||
"size": 47104,
|
||||
"capturedAt": "2025-10-26T00:00:00Z"
|
||||
}
|
||||
{
|
||||
"name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Python/StellaOps.Scanner.Analyzers.Lang.Python.pdb",
|
||||
"sha256": "ef2ad78bc2cd1d7e99bae000b92357aa9a9c32938501899e9033d001096196d0",
|
||||
"size": 31896,
|
||||
"capturedAt": "2025-10-26T00:00:00Z"
|
||||
}
|
||||
{
|
||||
"name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Python/manifest.json",
|
||||
"sha256": "668ad9a1a35485628677b639db4d996d1e25f62021680a81a22482483800e557",
|
||||
"size": 648,
|
||||
"capturedAt": "2025-10-26T00:00:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2 · Import on the air‑gapped host
|
||||
|
||||
```bash
|
||||
docker compose --env-file .env \
|
||||
-f docker-compose.stella-ops.yml \
|
||||
exec stella-ops \
|
||||
stella admin import-offline-usage-kit stella-ops-offline-kit-<DATE>.tgz
|
||||
```
|
||||
|
||||
Alternatively, run
|
||||
|
||||
```bash
|
||||
stellaops-cli offline kit import stella-ops-offline-kit-<DATE>.tgz \
|
||||
--manifest offline-manifest-<DATE>.json \
|
||||
--bundle-signature stella-ops-offline-kit-<DATE>.tgz.sig \
|
||||
--manifest-signature offline-manifest-<DATE>.json.jws
|
||||
```
|
||||
|
||||
The CLI validates recorded digests (when `.metadata.json` is present) before streaming the multipart payload to `/api/offline-kit/import`.
|
||||
|
||||
* The CLI validates the Cosign signature **before** activation.
|
||||
* Old feeds are kept until the new bundle is fully verified.
|
||||
* Import time on a SATA SSD: ≈ 25 s for a 300 MB kit.
|
||||
|
||||
### 2.1 Validator + idempotency enablement (air-gap)
|
||||
|
||||
The Offline Kit carries the same helper scripts under `scripts/`:
|
||||
|
||||
1. **Duplicate audit:** run
|
||||
```bash
|
||||
mongo concelier ops/devops/scripts/check-advisory-raw-duplicates.js --eval 'var LIMIT=200;'
|
||||
```
|
||||
to verify no `(vendor, upstream_id, content_hash, tenant)` conflicts remain before enabling the idempotency index.
|
||||
2. **Apply validators:** execute `mongo concelier ops/devops/scripts/apply-aoc-validators.js` (and the Excititor equivalent) with `validationLevel: "moderate"` in maintenance mode.
|
||||
3. **Restart Concelier** so migrations `20251028_advisory_raw_idempotency_index` and `20251028_advisory_supersedes_backfill` run automatically. After the restart:
|
||||
- Confirm `db.advisory` resolves to a view on `advisory_backup_20251028`.
|
||||
- Spot-check a few `advisory_raw` entries to ensure `supersedes` chains are populated deterministically.
|
||||
4. **Smoke test:** run `stella sources ingest --dry-run --fixture advisory` (bundled fixtures) to confirm ingestion succeeds post-guard and the CLI reports zero violations.
|
||||
|
||||
### Authority scope sanity check
|
||||
|
||||
Offline installs rely on the bundled `etc/authority.yaml.sample`. Before promoting the kit, confirm the sample clients keep the Aggregation-Only guardrails:
|
||||
|
||||
- `aoc-verifier` requests `aoc:verify`, `advisory:read`, and `vex:read`.
|
||||
- `signals-uploader` requests `signals:write`, `signals:read`, and `aoc:verify`.
|
||||
|
||||
Authority now rejects tokens that request `advisory:read`, `vex:read`, or any `signals:*` scope without `aoc:verify`; the sample has been updated to match. If you maintain tenant-specific overlays, mirror the same pairing so air-gapped automation fails deterministically with `invalid_scope` when misconfigured.
|
||||
|
||||
**Quick smoke test:** before import, verify the tarball carries the Go analyzer plug-in:
|
||||
|
||||
```bash
|
||||
tar -tzf stella-ops-offline-kit-<DATE>.tgz 'plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Go/*' 'plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.DotNet/*' 'plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Python/*'
|
||||
```
|
||||
|
||||
The manifest lookup above and this `tar` listing should both surface the Go analyzer DLL, PDB, and manifest entries before the kit is promoted.
|
||||
|
||||
> **Release guardrail.** The automated release pipeline now publishes the Python plug-in from source and executes `dotnet run --project src/Tools/LanguageAnalyzerSmoke --configuration Release -- --repo-root <checkout>` to validate manifest integrity and cold/warm determinism within the < 30 s / < 5 s budgets (differences versus repository goldens are logged for triage). Run `ops/offline-kit/run-python-analyzer-smoke.sh` locally before shipping a refreshed kit if you rebuild artefacts outside CI or when preparing the air-gap bundle.
|
||||
|
||||
### Debug store mirror
|
||||
|
||||
Offline symbols (`debug/.build-id/**`) must accompany every Offline Kit to keep symbol lookup deterministic. The release workflow is expected to emit `out/release/debug/` containing the build-id tree plus `debug-manifest.json` and its `.sha256` companion. After a release completes:
|
||||
|
||||
```bash
|
||||
python ops/offline-kit/mirror_debug_store.py \
|
||||
--release-dir out/release \
|
||||
--offline-dir out/offline-kit \
|
||||
--summary out/offline-kit/metadata/debug-store.json
|
||||
```
|
||||
|
||||
The script mirrors the debug tree into the Offline Kit staging directory, verifies SHA-256 values against the manifest, and writes a summary under `metadata/debug-store.json` for audit logs. If the release pipeline does not populate `out/release/debug`, the tooling now logs a warning (`DEVOPS-REL-17-004`)—treat it as a build failure and re-run the release once symbol extraction is enabled.
|
||||
|
||||
---
|
||||
|
||||
## 3 · Delta patch workflow
|
||||
|
||||
1. **Connected site** fetches `stella-ouk-YYYY‑MM‑DD.delta.tgz`.
|
||||
2. Transfer via any medium (USB, portable disk).
|
||||
3. `stella admin import-offline-usage-kit <delta>` applies only changed CVE rows & images.
|
||||
|
||||
Daily deltas are **< 30 MB**; weekly roll‑up produces a fresh full kit.
|
||||
|
||||
---
|
||||
|
||||
## 4 · Quota behaviour offline
|
||||
|
||||
The scanner enforces the same fair‑use limits offline:
|
||||
|
||||
* **Anonymous:** {{ quota\_anon }} scans per UTC day
|
||||
* **Free JWT:** {{ quota\_token }} scans per UTC day
|
||||
|
||||
Soft reminder at 200 scans; throttle above the ceiling but **never block**.
|
||||
See the detailed rules in
|
||||
[`33_333_QUOTA_OVERVIEW.md`](33_333_QUOTA_OVERVIEW.md).
|
||||
|
||||
---
|
||||
|
||||
## 5 · Troubleshooting
|
||||
|
||||
| Symptom | Explanation | Fix |
|
||||
| -------------------------------------- | ---------------------------------------- | ------------------------------------- |
|
||||
| `could not verify SBOM hash` | Bundle corrupted in transit | Re‑download / re‑copy |
|
||||
| Import hangs at `Applying feeds…` | Low disk space in `/var/lib/stella` | Free ≥ 2 GiB before retry |
|
||||
| `quota exceeded` same day after import | Import resets counters at UTC 00:00 only | Wait until next UTC day or load a JWT |
|
||||
|
||||
---
|
||||
|
||||
## 6 · Related documentation
|
||||
|
||||
* **Install guide:** `/install/#air-gapped`
|
||||
* **Sovereign mode rationale:** `/sovereign/`
|
||||
* **Security policy:** `/security/#reporting-a-vulnerability`
|
||||
* **CERT-Bund snapshots:** `python src/Tools/certbund_offline_snapshot.py --help` (see `docs/modules/concelier/operations/connectors/certbund.md`)
|
||||
# Offline Update Kit (OUK) — Air‑Gap Bundle
|
||||
|
||||
<!--
|
||||
Build‑time variable injection:
|
||||
{{ quota_anon }} = 33
|
||||
{{ quota_token }} = 333
|
||||
{{ dotnet }} = "10 LTS"
|
||||
-->
|
||||
|
||||
The **Offline Update Kit** packages everything Stella Ops needs to run on a
|
||||
completely isolated network:
|
||||
|
||||
| Component | Contents |
|
||||
|-----------|----------|
|
||||
| **Merged vulnerability feeds** | OSV, GHSA plus optional NVD 2.0, CNNVD, CNVD, ENISA, JVN and BDU |
|
||||
| **Container images** | `stella-ops`, *Zastava* sidecar (x86‑64 & arm64) |
|
||||
| **Provenance** | Cosign signature, SPDX 2.3 SBOM, in‑toto SLSA attestation |
|
||||
| **Attested manifest** | `offline-manifest.json` + detached JWS covering bundle metadata, signed during export. |
|
||||
| **Delta patches** | Daily diff bundles keep size \< 350 MB |
|
||||
| **Scanner plug-ins** | OS analyzers plus the Node.js, Go, .NET, and Python language analyzers packaged under `plugins/scanner/analyzers/**` with manifests so Workers load deterministically offline. |
|
||||
| **Debug store** | `.debug` artefacts laid out under `debug/.build-id/<aa>/<rest>.debug` with `debug/debug-manifest.json` mapping build-ids to originating images for symbol retrieval. |
|
||||
| **Telemetry collector bundle** | `telemetry/telemetry-offline-bundle.tar.gz` plus `.sha256`, containing OTLP collector config, Helm/Compose overlays, and operator instructions. |
|
||||
|
||||
**RU BDU note:** ship the official Russian Trusted Root/Sub CA bundle (`certificates/russian_trusted_bundle.pem`) inside the kit so `concelier:httpClients:source.bdu:trustedRootPaths` can resolve it when the service runs in an air‑gapped network. Drop the most recent `vulxml.zip` alongside the kit if operators need a cold-start cache.
|
||||
|
||||
**Language analyzers:** the kit now carries the restart-only Node.js, Go, .NET, and Python analyzer plug-ins (`plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Node/`, `...Lang.Go/`, `...Lang.DotNet/`, `...Lang.Python/`). Drop the directories alongside Worker binaries so the unified plug-in catalog can load them without outbound fetches; Rust remains on the Wave 4 roadmap.
|
||||
|
||||
*Scanner core:* C# 12 on **.NET {{ dotnet }}**.
|
||||
*Imports are idempotent and atomic — no service downtime.*
|
||||
|
||||
## 0 · Prepare the debug store
|
||||
|
||||
Before packaging the Offline Kit, mirror the release debug artefacts (GNU build-id `.debug` files and the associated manifest) into the staging directory:
|
||||
|
||||
```bash
|
||||
./ops/offline-kit/mirror_debug_store.py \
|
||||
--release-dir out/release \
|
||||
--offline-kit-dir out/offline-kit
|
||||
```
|
||||
|
||||
The helper copies `debug/.build-id/**`, validates `debug/debug-manifest.json` against its recorded SHA-256, and writes `out/offline-kit/metadata/debug-store.json` with a short summary (platforms, artefact counts, sample build-ids). The command exits non-zero if an artefact referenced by the manifest is missing or has the wrong digest, so run it as part of every kit build.
|
||||
|
||||
---
|
||||
|
||||
## 0.1 · Automated packaging
|
||||
|
||||
The packaging workflow is scripted via `ops/offline-kit/build_offline_kit.py`.
|
||||
It verifies the release artefacts, runs the Python analyzer smoke suite, mirrors the debug store, and emits a deterministic tarball + manifest set.
|
||||
|
||||
```bash
|
||||
python ops/offline-kit/build_offline_kit.py \
|
||||
--version 2025.10.0 \
|
||||
--channel edge \
|
||||
--release-dir out/release \
|
||||
--staging-dir out/offline-kit/staging \
|
||||
--output-dir out/offline-kit/dist
|
||||
|
||||
# Optional: regenerate the telemetry collector bundle prior to packaging.
|
||||
python ops/devops/telemetry/package_offline_bundle.py --output out/telemetry/telemetry-offline-bundle.tar.gz
|
||||
```
|
||||
|
||||
Outputs:
|
||||
|
||||
- `stella-ops-offline-kit-<version>-<channel>.tar.gz` — bundle (mtime/uid/gid forced to zero for reproducibility)
|
||||
- `stella-ops-offline-kit-<version>-<channel>.tar.gz.sha256` — bundle digest
|
||||
- `manifest/offline-manifest.json` + `.sha256` — inventories every file in the bundle
|
||||
- `<bundle>.metadata.json` — descriptor consumed by the CLI/Console import tooling
|
||||
- `telemetry/telemetry-offline-bundle.tar.gz` + `.sha256` — packaged OTLP collector assets for environments without upstream access
|
||||
- `plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Python/*.sig` (+ `.sha256`) — Cosign signatures for the Python analyzer DLL and manifest
|
||||
|
||||
### Policy Gateway configuration bundle
|
||||
|
||||
- Copy `etc/policy-gateway.yaml` (or the `*.sample` template if you expect operators to override values) into `config/policy-gateway/policy-gateway.yaml` within the staging tree.
|
||||
- Include the gateway DPoP private key under `secrets/policy-gateway/policy-gateway-dpop.pem` and reference the location inside the manifest notes. Set the permissions explicitly (`chmod 600 secrets/policy-gateway/policy-gateway-dpop.pem`) so only the kit importer can read it; the importer will refuse keys that are broader.
|
||||
- Document the gateway base URL and activation verification steps in `docs/policy/gateway.md` (bundled alongside the kit). Operators can use those curl snippets to smoke-test pack CRUD once the Offline Kit is imported.
|
||||
- Ensure the Prometheus snapshot captured during packaging contains `policy_gateway_activation_requests_total` so auditors can reconcile activation attempts performed via the gateway during the validation window.
|
||||
|
||||
Provide `--cosign-key` / `--cosign-identity-token` (and optional `--cosign-password`) to generate Cosign signatures for both the tarball and manifest.
|
||||
|
||||
---
|
||||
|
||||
## 1 · Download & verify
|
||||
|
||||
```bash
|
||||
curl -LO https://get.stella-ops.org/ouk/stella-ops-offline-kit-<DATE>.tgz
|
||||
curl -LO https://get.stella-ops.org/ouk/stella-ops-offline-kit-<DATE>.tgz.sig
|
||||
curl -LO https://get.stella-ops.org/ouk/offline-manifest-<DATE>.json
|
||||
curl -LO https://get.stella-ops.org/ouk/offline-manifest-<DATE>.json.jws
|
||||
|
||||
cosign verify-blob \
|
||||
--key https://stella-ops.org/keys/cosign.pub \
|
||||
--signature stella-ops-offline-kit-<DATE>.tgz.sig \
|
||||
stella-ops-offline-kit-<DATE>.tgz
|
||||
````
|
||||
|
||||
**CLI shortcut.** `stellaops-cli offline kit pull --destination ./offline-kit` downloads the bundle, manifest, and detached signatures in one step, resumes partial transfers, and writes a `.metadata.json` summary for later import.
|
||||
|
||||
Verification prints **OK** and the SHA‑256 digest; cross‑check against the
|
||||
[changelog](https://git.stella-ops.org/stella-ops/offline-kit/-/releases).
|
||||
|
||||
Validate the attested manifest before distribution:
|
||||
|
||||
```bash
|
||||
cosign verify-blob \
|
||||
--key https://stella-ops.org/keys/cosign.pub \
|
||||
--signature offline-manifest-<DATE>.json.jws \
|
||||
offline-manifest-<DATE>.json
|
||||
|
||||
jq '.artifacts[] | {name, sha256, size, capturedAt}' offline-manifest-<DATE>.json
|
||||
```
|
||||
|
||||
The manifest enumerates every artefact (`name`, `sha256`, `size`, `capturedAt`) and is signed with the same key registry as Authority revocation bundles. Operators can ship the manifest alongside the tarball so downstream mirrors can re-verify without unpacking the kit.
|
||||
|
||||
Example excerpt (2025-10-23 kit) showing the Go and .NET analyzer plug-in payloads:
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Go/StellaOps.Scanner.Analyzers.Lang.Go.dll",
|
||||
"sha256": "a6dc850fc51151c8967ef46a3c4730f08b549667e041079431f39a8a72d0b641",
|
||||
"size": 33792,
|
||||
"capturedAt": "2025-10-23T00:00:00Z"
|
||||
}
|
||||
{
|
||||
"name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Go/StellaOps.Scanner.Analyzers.Lang.Go.pdb",
|
||||
"sha256": "6cbdabf155282f458b89edf267e7f6bb2441a93029aad7aad45c8a9ec58b1b3b",
|
||||
"size": 32152,
|
||||
"capturedAt": "2025-10-23T00:00:00Z"
|
||||
}
|
||||
{
|
||||
"name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Go/manifest.json",
|
||||
"sha256": "c19bfca2fcbb7cb18f1082b5d0d5a8f15fc799c648b50e95fce8d8b109ce48c9",
|
||||
"size": 622,
|
||||
"capturedAt": "2025-10-23T00:00:00Z"
|
||||
}
|
||||
{
|
||||
"name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.DotNet/StellaOps.Scanner.Analyzers.Lang.DotNet.dll",
|
||||
"sha256": "0734d23e33277ce2ccb596782d2d42cfe394b3d372dc34da9cb28b59df9b9d22",
|
||||
"size": 70144,
|
||||
"capturedAt": "2025-10-23T00:00:00Z"
|
||||
}
|
||||
{
|
||||
"name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.DotNet/StellaOps.Scanner.Analyzers.Lang.DotNet.pdb",
|
||||
"sha256": "b853c1ff4b196715f5bd1447e1a13edeb4940917527ec9bf153b5048da49abaf",
|
||||
"size": 40400,
|
||||
"capturedAt": "2025-10-23T00:00:00Z"
|
||||
}
|
||||
{
|
||||
"name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.DotNet/manifest.json",
|
||||
"sha256": "5d483885f825f01bfd9943dcf2889ec2e0beba38ede92ecfe67d4f506cf14e37",
|
||||
"size": 647,
|
||||
"capturedAt": "2025-10-23T00:00:00Z"
|
||||
}
|
||||
{
|
||||
"name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Python/StellaOps.Scanner.Analyzers.Lang.Python.dll",
|
||||
"sha256": "a4f558f363394096e3dd6263f35b180b93b4112f9cf616c05872da8a8657d518",
|
||||
"size": 47104,
|
||||
"capturedAt": "2025-10-26T00:00:00Z"
|
||||
}
|
||||
{
|
||||
"name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Python/StellaOps.Scanner.Analyzers.Lang.Python.pdb",
|
||||
"sha256": "ef2ad78bc2cd1d7e99bae000b92357aa9a9c32938501899e9033d001096196d0",
|
||||
"size": 31896,
|
||||
"capturedAt": "2025-10-26T00:00:00Z"
|
||||
}
|
||||
{
|
||||
"name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Python/manifest.json",
|
||||
"sha256": "668ad9a1a35485628677b639db4d996d1e25f62021680a81a22482483800e557",
|
||||
"size": 648,
|
||||
"capturedAt": "2025-10-26T00:00:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2 · Import on the air‑gapped host
|
||||
|
||||
```bash
|
||||
docker compose --env-file .env \
|
||||
-f docker-compose.stella-ops.yml \
|
||||
exec stella-ops \
|
||||
stella admin import-offline-usage-kit stella-ops-offline-kit-<DATE>.tgz
|
||||
```
|
||||
|
||||
Alternatively, run
|
||||
|
||||
```bash
|
||||
stellaops-cli offline kit import stella-ops-offline-kit-<DATE>.tgz \
|
||||
--manifest offline-manifest-<DATE>.json \
|
||||
--bundle-signature stella-ops-offline-kit-<DATE>.tgz.sig \
|
||||
--manifest-signature offline-manifest-<DATE>.json.jws
|
||||
```
|
||||
|
||||
The CLI validates recorded digests (when `.metadata.json` is present) before streaming the multipart payload to `/api/offline-kit/import`.
|
||||
|
||||
* The CLI validates the Cosign signature **before** activation.
|
||||
* Old feeds are kept until the new bundle is fully verified.
|
||||
* Import time on a SATA SSD: ≈ 25 s for a 300 MB kit.
|
||||
|
||||
### 2.1 Validator + idempotency enablement (air-gap)
|
||||
|
||||
The Offline Kit carries the same helper scripts under `scripts/`:
|
||||
|
||||
1. **Duplicate audit:** run
|
||||
```bash
|
||||
mongo concelier ops/devops/scripts/check-advisory-raw-duplicates.js --eval 'var LIMIT=200;'
|
||||
```
|
||||
to verify no `(vendor, upstream_id, content_hash, tenant)` conflicts remain before enabling the idempotency index.
|
||||
2. **Apply validators:** execute `mongo concelier ops/devops/scripts/apply-aoc-validators.js` (and the Excititor equivalent) with `validationLevel: "moderate"` in maintenance mode.
|
||||
3. **Restart Concelier** so migrations `20251028_advisory_raw_idempotency_index` and `20251028_advisory_supersedes_backfill` run automatically. After the restart:
|
||||
- Confirm `db.advisory` resolves to a view on `advisory_backup_20251028`.
|
||||
- Spot-check a few `advisory_raw` entries to ensure `supersedes` chains are populated deterministically.
|
||||
4. **Smoke test:** run `stella sources ingest --dry-run --fixture advisory` (bundled fixtures) to confirm ingestion succeeds post-guard and the CLI reports zero violations.
|
||||
|
||||
### Authority scope sanity check
|
||||
|
||||
Offline installs rely on the bundled `etc/authority.yaml.sample`. Before promoting the kit, confirm the sample clients keep the Aggregation-Only guardrails:
|
||||
|
||||
- `aoc-verifier` requests `aoc:verify`, `advisory:read`, and `vex:read`.
|
||||
- `signals-uploader` requests `signals:write`, `signals:read`, and `aoc:verify`.
|
||||
|
||||
Authority now rejects tokens that request `advisory:read`, `vex:read`, or any `signals:*` scope without `aoc:verify`; the sample has been updated to match. If you maintain tenant-specific overlays, mirror the same pairing so air-gapped automation fails deterministically with `invalid_scope` when misconfigured.
|
||||
|
||||
**Quick smoke test:** before import, verify the tarball carries the Go analyzer plug-in:
|
||||
|
||||
```bash
|
||||
tar -tzf stella-ops-offline-kit-<DATE>.tgz 'plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Go/*' 'plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.DotNet/*' 'plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Python/*'
|
||||
```
|
||||
|
||||
The manifest lookup above and this `tar` listing should both surface the Go analyzer DLL, PDB, and manifest entries before the kit is promoted.
|
||||
|
||||
> **Release guardrail.** The automated release pipeline now publishes the Python plug-in from source and executes `dotnet run --project src/Tools/LanguageAnalyzerSmoke --configuration Release -- --repo-root <checkout>` to validate manifest integrity and cold/warm determinism within the < 30 s / < 5 s budgets (differences versus repository goldens are logged for triage). Run `ops/offline-kit/run-python-analyzer-smoke.sh` locally before shipping a refreshed kit if you rebuild artefacts outside CI or when preparing the air-gap bundle.
|
||||
|
||||
### Debug store mirror
|
||||
|
||||
Offline symbols (`debug/.build-id/**`) must accompany every Offline Kit to keep symbol lookup deterministic. The release workflow is expected to emit `out/release/debug/` containing the build-id tree plus `debug-manifest.json` and its `.sha256` companion. After a release completes:
|
||||
|
||||
```bash
|
||||
python ops/offline-kit/mirror_debug_store.py \
|
||||
--release-dir out/release \
|
||||
--offline-dir out/offline-kit \
|
||||
--summary out/offline-kit/metadata/debug-store.json
|
||||
```
|
||||
|
||||
The script mirrors the debug tree into the Offline Kit staging directory, verifies SHA-256 values against the manifest, and writes a summary under `metadata/debug-store.json` for audit logs. If the release pipeline does not populate `out/release/debug`, the tooling now logs a warning (`DEVOPS-REL-17-004`)—treat it as a build failure and re-run the release once symbol extraction is enabled.
|
||||
|
||||
---
|
||||
|
||||
## 3 · Delta patch workflow
|
||||
|
||||
1. **Connected site** fetches `stella-ouk-YYYY‑MM‑DD.delta.tgz`.
|
||||
2. Transfer via any medium (USB, portable disk).
|
||||
3. `stella admin import-offline-usage-kit <delta>` applies only changed CVE rows & images.
|
||||
|
||||
Daily deltas are **< 30 MB**; weekly roll‑up produces a fresh full kit.
|
||||
|
||||
---
|
||||
|
||||
## 4 · Quota behaviour offline
|
||||
|
||||
The scanner enforces the same fair‑use limits offline:
|
||||
|
||||
* **Anonymous:** {{ quota\_anon }} scans per UTC day
|
||||
* **Free JWT:** {{ quota\_token }} scans per UTC day
|
||||
|
||||
Soft reminder at 200 scans; throttle above the ceiling but **never block**.
|
||||
See the detailed rules in
|
||||
[`33_333_QUOTA_OVERVIEW.md`](33_333_QUOTA_OVERVIEW.md).
|
||||
|
||||
---
|
||||
|
||||
## 5 · Troubleshooting
|
||||
|
||||
| Symptom | Explanation | Fix |
|
||||
| -------------------------------------- | ---------------------------------------- | ------------------------------------- |
|
||||
| `could not verify SBOM hash` | Bundle corrupted in transit | Re‑download / re‑copy |
|
||||
| Import hangs at `Applying feeds…` | Low disk space in `/var/lib/stella` | Free ≥ 2 GiB before retry |
|
||||
| `quota exceeded` same day after import | Import resets counters at UTC 00:00 only | Wait until next UTC day or load a JWT |
|
||||
|
||||
---
|
||||
|
||||
## 6 · Related documentation
|
||||
|
||||
* **Install guide:** `/install/#air-gapped`
|
||||
* **Sovereign mode rationale:** `/sovereign/`
|
||||
* **Security policy:** `/security/#reporting-a-vulnerability`
|
||||
* **CERT-Bund snapshots:** `python src/Tools/certbund_offline_snapshot.py --help` (see `docs/modules/concelier/operations/connectors/certbund.md`)
|
||||
|
||||
Reference in New Issue
Block a user