git.stella-ops.org/docs/24_OFFLINE_KIT.md

# Offline Update Kit (OUK) — Air‑Gap Bundle

<!--
  Build‑time variable injection:
    {{ quota_anon }}   = 33
    {{ quota_token }}  = 333
    {{ dotnet }}       = "10 LTS"
-->

The **Offline Update Kit** packages everything Stella Ops needs to run on a
completely isolated network:

| Component | Contents |
|-----------|----------|
| **Merged vulnerability feeds** | OSV, GHSA plus optional NVD 2.0, CNNVD, CNVD, ENISA, JVN and BDU |
| **Container images** | `stella-ops`, *Zastava* sidecar (x86‑64 & arm64) |
| **Provenance** | Cosign signature, SPDX 2.3 SBOM, in‑toto SLSA attestation |
| **Attested manifest** | `offline-manifest.json` + detached JWS covering bundle metadata, signed during export. |
| **Delta patches** | Daily diff bundles keep size \< 350 MB |
| **Scanner plug-ins** | OS analyzers plus the Node.js, Go, .NET, and Python language analyzers packaged under `plugins/scanner/analyzers/**` with manifests so Workers load deterministically offline. |
| **Debug store** | `.debug` artefacts laid out under `debug/.build-id/<aa>/<rest>.debug` with `debug/debug-manifest.json` mapping build-ids to originating images for symbol retrieval. |
| **Telemetry collector bundle** | `telemetry/telemetry-offline-bundle.tar.gz` plus `.sha256`, containing OTLP collector config, Helm/Compose overlays, and operator instructions. |

**RU BDU note:** ship the official Russian Trusted Root/Sub CA bundle (`certificates/russian_trusted_bundle.pem`) inside the kit so `concelier:httpClients:source.bdu:trustedRootPaths` can resolve it when the service runs in an air‑gapped network. Drop the most recent `vulxml.zip` alongside the kit if operators need a cold-start cache.

**Language analyzers:** the kit now carries the restart-only Node.js, Go, .NET, and Python analyzer plug-ins (`plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Node/`, `...Lang.Go/`, `...Lang.DotNet/`, `...Lang.Python/`). Drop the directories alongside Worker binaries so the unified plug-in catalog can load them without outbound fetches; Rust remains on the Wave 4 roadmap.

*Scanner core:* C# 12 on **.NET {{ dotnet }}**.  
*Imports are idempotent and atomic — no service downtime.*

## 0 · Prepare the debug store

Before packaging the Offline Kit, mirror the release debug artefacts (GNU build-id `.debug` files and the associated manifest) into the staging directory:

```bash
./ops/offline-kit/mirror_debug_store.py \
  --release-dir out/release \
  --offline-kit-dir out/offline-kit
```

The helper copies `debug/.build-id/**`, validates `debug/debug-manifest.json` against its recorded SHA-256, and writes `out/offline-kit/metadata/debug-store.json` with a short summary (platforms, artefact counts, sample build-ids). The command exits non-zero if an artefact referenced by the manifest is missing or has the wrong digest, so run it as part of every kit build.

---

## 0.1 · Automated packaging

The packaging workflow is scripted via `ops/offline-kit/build_offline_kit.py`.  
It verifies the release artefacts, runs the Python analyzer smoke suite, mirrors the debug store, and emits a deterministic tarball + manifest set.

```bash
python ops/offline-kit/build_offline_kit.py \
  --version 2025.10.0 \
  --channel edge \
  --release-dir out/release \
  --staging-dir out/offline-kit/staging \
  --output-dir out/offline-kit/dist

# Optional: regenerate the telemetry collector bundle prior to packaging.
python ops/devops/telemetry/package_offline_bundle.py --output out/telemetry/telemetry-offline-bundle.tar.gz
```

Outputs:

- `stella-ops-offline-kit-<version>-<channel>.tar.gz` — bundle (mtime/uid/gid forced to zero for reproducibility)  
- `stella-ops-offline-kit-<version>-<channel>.tar.gz.sha256` — bundle digest  
- `manifest/offline-manifest.json` + `.sha256` — inventories every file in the bundle  
- `<bundle>.metadata.json` — descriptor consumed by the CLI/Console import tooling
- `telemetry/telemetry-offline-bundle.tar.gz` + `.sha256` — packaged OTLP collector assets for environments without upstream access
- `plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Python/*.sig` (+ `.sha256`) — Cosign signatures for the Python analyzer DLL and manifest

### Policy Gateway configuration bundle

- Copy `etc/policy-gateway.yaml` (or the `*.sample` template if you expect operators to override values) into `config/policy-gateway/policy-gateway.yaml` within the staging tree.
- Include the gateway DPoP private key under `secrets/policy-gateway/policy-gateway-dpop.pem` and reference the location inside the manifest notes. Set the permissions explicitly (`chmod 600 secrets/policy-gateway/policy-gateway-dpop.pem`) so only the kit importer can read it; the importer will refuse keys that are broader.
- Document the gateway base URL and activation verification steps in `docs/policy/gateway.md` (bundled alongside the kit). Operators can use those curl snippets to smoke-test pack CRUD once the Offline Kit is imported.
- Ensure the Prometheus snapshot captured during packaging contains `policy_gateway_activation_requests_total` so auditors can reconcile activation attempts performed via the gateway during the validation window.

Provide `--cosign-key` / `--cosign-identity-token` (and optional `--cosign-password`) to generate Cosign signatures for both the tarball and manifest.

---

## 1 · Download & verify

```bash
curl -LO https://get.stella-ops.org/ouk/stella-ops-offline-kit-<DATE>.tgz
curl -LO https://get.stella-ops.org/ouk/stella-ops-offline-kit-<DATE>.tgz.sig
curl -LO https://get.stella-ops.org/ouk/offline-manifest-<DATE>.json
curl -LO https://get.stella-ops.org/ouk/offline-manifest-<DATE>.json.jws

cosign verify-blob \
  --key https://stella-ops.org/keys/cosign.pub \
  --signature stella-ops-offline-kit-<DATE>.tgz.sig \
  stella-ops-offline-kit-<DATE>.tgz
````

**CLI shortcut.** `stellaops-cli offline kit pull --destination ./offline-kit` downloads the bundle, manifest, and detached signatures in one step, resumes partial transfers, and writes a `.metadata.json` summary for later import.

Verification prints **OK** and the SHA‑256 digest; cross‑check against the
[changelog](https://git.stella-ops.org/stella-ops/offline-kit/-/releases).

Validate the attested manifest before distribution:

```bash
cosign verify-blob \
  --key https://stella-ops.org/keys/cosign.pub \
  --signature offline-manifest-<DATE>.json.jws \
  offline-manifest-<DATE>.json

jq '.artifacts[] | {name, sha256, size, capturedAt}' offline-manifest-<DATE>.json
```

The manifest enumerates every artefact (`name`, `sha256`, `size`, `capturedAt`) and is signed with the same key registry as Authority revocation bundles. Operators can ship the manifest alongside the tarball so downstream mirrors can re-verify without unpacking the kit.

Example excerpt (2025-10-23 kit) showing the Go and .NET analyzer plug-in payloads:

```json
{
  "name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Go/StellaOps.Scanner.Analyzers.Lang.Go.dll",
  "sha256": "a6dc850fc51151c8967ef46a3c4730f08b549667e041079431f39a8a72d0b641",
  "size": 33792,
  "capturedAt": "2025-10-23T00:00:00Z"
}
{
  "name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Go/StellaOps.Scanner.Analyzers.Lang.Go.pdb",
  "sha256": "6cbdabf155282f458b89edf267e7f6bb2441a93029aad7aad45c8a9ec58b1b3b",
  "size": 32152,
  "capturedAt": "2025-10-23T00:00:00Z"
}
{
  "name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Go/manifest.json",
  "sha256": "c19bfca2fcbb7cb18f1082b5d0d5a8f15fc799c648b50e95fce8d8b109ce48c9",
  "size": 622,
  "capturedAt": "2025-10-23T00:00:00Z"
}
{
  "name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.DotNet/StellaOps.Scanner.Analyzers.Lang.DotNet.dll",
  "sha256": "0734d23e33277ce2ccb596782d2d42cfe394b3d372dc34da9cb28b59df9b9d22",
  "size": 70144,
  "capturedAt": "2025-10-23T00:00:00Z"
}
{
  "name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.DotNet/StellaOps.Scanner.Analyzers.Lang.DotNet.pdb",
  "sha256": "b853c1ff4b196715f5bd1447e1a13edeb4940917527ec9bf153b5048da49abaf",
  "size": 40400,
  "capturedAt": "2025-10-23T00:00:00Z"
}
{
  "name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.DotNet/manifest.json",
  "sha256": "5d483885f825f01bfd9943dcf2889ec2e0beba38ede92ecfe67d4f506cf14e37",
  "size": 647,
  "capturedAt": "2025-10-23T00:00:00Z"
}
{
  "name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Python/StellaOps.Scanner.Analyzers.Lang.Python.dll",
  "sha256": "a4f558f363394096e3dd6263f35b180b93b4112f9cf616c05872da8a8657d518",
  "size": 47104,
  "capturedAt": "2025-10-26T00:00:00Z"
}
{
  "name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Python/StellaOps.Scanner.Analyzers.Lang.Python.pdb",
  "sha256": "ef2ad78bc2cd1d7e99bae000b92357aa9a9c32938501899e9033d001096196d0",
  "size": 31896,
  "capturedAt": "2025-10-26T00:00:00Z"
}
{
  "name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Python/manifest.json",
  "sha256": "668ad9a1a35485628677b639db4d996d1e25f62021680a81a22482483800e557",
  "size": 648,
  "capturedAt": "2025-10-26T00:00:00Z"
}
```

---

## 2 · Import on the air‑gapped host

```bash
docker compose --env-file .env \
  -f docker-compose.stella-ops.yml \
  exec stella-ops \
  stella admin import-offline-usage-kit stella-ops-offline-kit-<DATE>.tgz
```

Alternatively, run

```bash
stellaops-cli offline kit import stella-ops-offline-kit-<DATE>.tgz \
  --manifest offline-manifest-<DATE>.json \
  --bundle-signature stella-ops-offline-kit-<DATE>.tgz.sig \
  --manifest-signature offline-manifest-<DATE>.json.jws
```

The CLI validates recorded digests (when `.metadata.json` is present) before streaming the multipart payload to `/api/offline-kit/import`.

* The CLI validates the Cosign signature **before** activation.
* Old feeds are kept until the new bundle is fully verified.
* Import time on a SATA SSD: ≈ 25 s for a 300 MB kit.

### 2.1 Validator + idempotency enablement (air-gap)

The Offline Kit carries the same helper scripts under `scripts/`:

1. **Duplicate audit:** run
   ```bash
   mongo concelier ops/devops/scripts/check-advisory-raw-duplicates.js --eval 'var LIMIT=200;'
   ```
   to verify no `(vendor, upstream_id, content_hash, tenant)` conflicts remain before enabling the idempotency index.
2. **Apply validators:** execute `mongo concelier ops/devops/scripts/apply-aoc-validators.js` (and the Excititor equivalent) with `validationLevel: "moderate"` in maintenance mode.
3. **Restart Concelier** so migrations `20251028_advisory_raw_idempotency_index` and `20251028_advisory_supersedes_backfill` run automatically. After the restart:
   - Confirm `db.advisory` resolves to a view on `advisory_backup_20251028`.
   - Spot-check a few `advisory_raw` entries to ensure `supersedes` chains are populated deterministically.
4. **Smoke test:** run `stella sources ingest --dry-run --fixture advisory` (bundled fixtures) to confirm ingestion succeeds post-guard and the CLI reports zero violations.

### Authority scope sanity check

Offline installs rely on the bundled `etc/authority.yaml.sample`. Before promoting the kit, confirm the sample clients keep the Aggregation-Only guardrails:

- `aoc-verifier` requests `aoc:verify`, `advisory:read`, and `vex:read`.
- `signals-uploader` requests `signals:write`, `signals:read`, and `aoc:verify`.

Authority now rejects tokens that request `advisory:read`, `vex:read`, or any `signals:*` scope without `aoc:verify`; the sample has been updated to match. If you maintain tenant-specific overlays, mirror the same pairing so air-gapped automation fails deterministically with `invalid_scope` when misconfigured.

**Quick smoke test:** before import, verify the tarball carries the Go analyzer plug-in:

```bash
tar -tzf stella-ops-offline-kit-<DATE>.tgz 'plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Go/*' 'plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.DotNet/*' 'plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Python/*'
```

The manifest lookup above and this `tar` listing should both surface the Go analyzer DLL, PDB, and manifest entries before the kit is promoted.

> **Release guardrail.** The automated release pipeline now publishes the Python plug-in from source and executes `dotnet run --project src/Tools/LanguageAnalyzerSmoke --configuration Release -- --repo-root <checkout>` to validate manifest integrity and cold/warm determinism within the < 30 s / < 5 s budgets (differences versus repository goldens are logged for triage). Run `ops/offline-kit/run-python-analyzer-smoke.sh` locally before shipping a refreshed kit if you rebuild artefacts outside CI or when preparing the air-gap bundle.

### Debug store mirror

Offline symbols (`debug/.build-id/**`) must accompany every Offline Kit to keep symbol lookup deterministic. The release workflow is expected to emit `out/release/debug/` containing the build-id tree plus `debug-manifest.json` and its `.sha256` companion. After a release completes:

```bash
python ops/offline-kit/mirror_debug_store.py \
  --release-dir out/release \
  --offline-dir out/offline-kit \
  --summary out/offline-kit/metadata/debug-store.json
```

The script mirrors the debug tree into the Offline Kit staging directory, verifies SHA-256 values against the manifest, and writes a summary under `metadata/debug-store.json` for audit logs. If the release pipeline does not populate `out/release/debug`, the tooling now logs a warning (`DEVOPS-REL-17-004`)—treat it as a build failure and re-run the release once symbol extraction is enabled.

---

## 3 · Delta patch workflow

1. **Connected site** fetches `stella-ouk-YYYY‑MM‑DD.delta.tgz`.
2. Transfer via any medium (USB, portable disk).
3. `stella admin import-offline-usage-kit <delta>` applies only changed CVE rows & images.

Daily deltas are **< 30 MB**; weekly roll‑up produces a fresh full kit.

---

## 4 · Quota behaviour offline

The scanner enforces the same fair‑use limits offline:

* **Anonymous:** {{ quota\_anon }} scans per UTC day
* **Free JWT:** {{ quota\_token }} scans per UTC day

Soft reminder at 200 scans; throttle above the ceiling but **never block**.
See the detailed rules in
[`33_333_QUOTA_OVERVIEW.md`](33_333_QUOTA_OVERVIEW.md).

---

## 5 · Troubleshooting

| Symptom                                | Explanation                              | Fix                                   |
| -------------------------------------- | ---------------------------------------- | ------------------------------------- |
| `could not verify SBOM hash`           | Bundle corrupted in transit              | Re‑download / re‑copy                 |
| Import hangs at `Applying feeds…`      | Low disk space in `/var/lib/stella`      | Free ≥ 2 GiB before retry             |
| `quota exceeded` same day after import | Import resets counters at UTC 00:00 only | Wait until next UTC day or load a JWT |

---

## 6 · Related documentation

* **Install guide:** `/install/#air-gapped`
* **Sovereign mode rationale:** `/sovereign/`
* **Security policy:** `/security/#reporting-a-vulnerability`
* **CERT-Bund snapshots:** `python src/Tools/certbund_offline_snapshot.py --help` (see `docs/modules/concelier/operations/connectors/certbund.md`)