git.stella-ops.org/docs/24_OFFLINE_KIT.md

Offline Update Kit (OUK) — Air‑Gap Bundle

The Offline Update Kit packages everything Stella Ops needs to run on a completely isolated network:

Component	Contents
Merged vulnerability feeds	OSV, GHSA plus optional NVD 2.0, CNNVD, CNVD, ENISA, JVN and BDU
Container images	stella-ops, Zastava sidecar (x86‑64 & arm64)
Provenance	Cosign signature, SPDX 2.3 SBOM, in‑toto SLSA attestation
Attested manifest	offline-manifest.json + detached JWS covering bundle metadata, signed during export.
Delta patches	Daily diff bundles keep size < 350 MB
Scanner plug-ins	OS analyzers plus the Node.js, Go, .NET, and Python language analyzers packaged under plugins/scanner/analyzers/** with manifests so Workers load deterministically offline.
Debug store	.debug artefacts laid out under debug/.build-id/<aa>/<rest>.debug with debug/debug-manifest.json mapping build-ids to originating images for symbol retrieval.
Telemetry collector bundle	telemetry/telemetry-offline-bundle.tar.gz plus .sha256, containing OTLP collector config, Helm/Compose overlays, and operator instructions.

RU BDU note: ship the official Russian Trusted Root/Sub CA bundle (certificates/russian_trusted_bundle.pem) inside the kit so concelier:httpClients:source.bdu:trustedRootPaths can resolve it when the service runs in an air‑gapped network. Drop the most recent vulxml.zip alongside the kit if operators need a cold-start cache.

Language analyzers: the kit now carries the restart-only Node.js, Go, .NET, and Python analyzer plug-ins (plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Node/, ...Lang.Go/, ...Lang.DotNet/, ...Lang.Python/). Drop the directories alongside Worker binaries so the unified plug-in catalog can load them without outbound fetches; Rust remains on the Wave 4 roadmap.

Scanner core: C# 12 on .NET {{ dotnet }}.
Imports are idempotent and atomic — no service downtime.

0 · Prepare the debug store

Before packaging the Offline Kit, mirror the release debug artefacts (GNU build-id .debug files and the associated manifest) into the staging directory:

./ops/offline-kit/mirror_debug_store.py \
  --release-dir out/release \
  --offline-kit-dir out/offline-kit

The helper copies debug/.build-id/**, validates debug/debug-manifest.json against its recorded SHA-256, and writes out/offline-kit/metadata/debug-store.json with a short summary (platforms, artefact counts, sample build-ids). The command exits non-zero if an artefact referenced by the manifest is missing or has the wrong digest, so run it as part of every kit build.

---

0.1 · Automated packaging

The packaging workflow is scripted via ops/offline-kit/build_offline_kit.py.
It verifies the release artefacts, runs the Python analyzer smoke suite, mirrors the debug store, and emits a deterministic tarball + manifest set.

python ops/offline-kit/build_offline_kit.py \
  --version 2025.10.0 \
  --channel edge \
  --release-dir out/release \
  --staging-dir out/offline-kit/staging \
  --output-dir out/offline-kit/dist

# Optional: regenerate the telemetry collector bundle prior to packaging.
python ops/devops/telemetry/package_offline_bundle.py --output out/telemetry/telemetry-offline-bundle.tar.gz

Outputs:

• stella-ops-offline-kit-<version>-<channel>.tar.gz — bundle (mtime/uid/gid forced to zero for reproducibility)
• stella-ops-offline-kit-<version>-<channel>.tar.gz.sha256 — bundle digest
• manifest/offline-manifest.json + .sha256 — inventories every file in the bundle
• <bundle>.metadata.json — descriptor consumed by the CLI/Console import tooling
• telemetry/telemetry-offline-bundle.tar.gz + .sha256 — packaged OTLP collector assets for environments without upstream access
• plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Python/*.sig (+ .sha256) — Cosign signatures for the Python analyzer DLL and manifest

Provide --cosign-key / --cosign-identity-token (and optional --cosign-password) to generate Cosign signatures for both the tarball and manifest.

---

1 · Download & verify

curl -LO https://get.stella-ops.org/ouk/stella-ops-offline-kit-<DATE>.tgz
curl -LO https://get.stella-ops.org/ouk/stella-ops-offline-kit-<DATE>.tgz.sig
curl -LO https://get.stella-ops.org/ouk/offline-manifest-<DATE>.json
curl -LO https://get.stella-ops.org/ouk/offline-manifest-<DATE>.json.jws

cosign verify-blob \
  --key https://stella-ops.org/keys/cosign.pub \
  --signature stella-ops-offline-kit-<DATE>.tgz.sig \
  stella-ops-offline-kit-<DATE>.tgz

CLI shortcut. stellaops-cli offline kit pull --destination ./offline-kit downloads the bundle, manifest, and detached signatures in one step, resumes partial transfers, and writes a .metadata.json summary for later import.

Verification prints OK and the SHA‑256 digest; cross‑check against the changelog.

Validate the attested manifest before distribution:

cosign verify-blob \
  --key https://stella-ops.org/keys/cosign.pub \
  --signature offline-manifest-<DATE>.json.jws \
  offline-manifest-<DATE>.json

jq '.artifacts[] | {name, sha256, size, capturedAt}' offline-manifest-<DATE>.json

The manifest enumerates every artefact (name, sha256, size, capturedAt) and is signed with the same key registry as Authority revocation bundles. Operators can ship the manifest alongside the tarball so downstream mirrors can re-verify without unpacking the kit.

Example excerpt (2025-10-23 kit) showing the Go and .NET analyzer plug-in payloads:

{
  "name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Go/StellaOps.Scanner.Analyzers.Lang.Go.dll",
  "sha256": "a6dc850fc51151c8967ef46a3c4730f08b549667e041079431f39a8a72d0b641",
  "size": 33792,
  "capturedAt": "2025-10-23T00:00:00Z"
}
{
  "name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Go/StellaOps.Scanner.Analyzers.Lang.Go.pdb",
  "sha256": "6cbdabf155282f458b89edf267e7f6bb2441a93029aad7aad45c8a9ec58b1b3b",
  "size": 32152,
  "capturedAt": "2025-10-23T00:00:00Z"
}
{
  "name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Go/manifest.json",
  "sha256": "c19bfca2fcbb7cb18f1082b5d0d5a8f15fc799c648b50e95fce8d8b109ce48c9",
  "size": 622,
  "capturedAt": "2025-10-23T00:00:00Z"
}
{
  "name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.DotNet/StellaOps.Scanner.Analyzers.Lang.DotNet.dll",
  "sha256": "0734d23e33277ce2ccb596782d2d42cfe394b3d372dc34da9cb28b59df9b9d22",
  "size": 70144,
  "capturedAt": "2025-10-23T00:00:00Z"
}
{
  "name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.DotNet/StellaOps.Scanner.Analyzers.Lang.DotNet.pdb",
  "sha256": "b853c1ff4b196715f5bd1447e1a13edeb4940917527ec9bf153b5048da49abaf",
  "size": 40400,
  "capturedAt": "2025-10-23T00:00:00Z"
}
{
  "name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.DotNet/manifest.json",
  "sha256": "5d483885f825f01bfd9943dcf2889ec2e0beba38ede92ecfe67d4f506cf14e37",
  "size": 647,
  "capturedAt": "2025-10-23T00:00:00Z"
}
{
  "name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Python/StellaOps.Scanner.Analyzers.Lang.Python.dll",
  "sha256": "a4f558f363394096e3dd6263f35b180b93b4112f9cf616c05872da8a8657d518",
  "size": 47104,
  "capturedAt": "2025-10-26T00:00:00Z"
}
{
  "name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Python/StellaOps.Scanner.Analyzers.Lang.Python.pdb",
  "sha256": "ef2ad78bc2cd1d7e99bae000b92357aa9a9c32938501899e9033d001096196d0",
  "size": 31896,
  "capturedAt": "2025-10-26T00:00:00Z"
}
{
  "name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Python/manifest.json",
  "sha256": "668ad9a1a35485628677b639db4d996d1e25f62021680a81a22482483800e557",
  "size": 648,
  "capturedAt": "2025-10-26T00:00:00Z"
}

---

2 · Import on the air‑gapped host

docker compose --env-file .env \
  -f docker-compose.stella-ops.yml \
  exec stella-ops \
  stella admin import-offline-usage-kit stella-ops-offline-kit-<DATE>.tgz

Alternatively, run

stellaops-cli offline kit import stella-ops-offline-kit-<DATE>.tgz \
  --manifest offline-manifest-<DATE>.json \
  --bundle-signature stella-ops-offline-kit-<DATE>.tgz.sig \
  --manifest-signature offline-manifest-<DATE>.json.jws

The CLI validates recorded digests (when .metadata.json is present) before streaming the multipart payload to /api/offline-kit/import.

• The CLI validates the Cosign signature before activation.
• Old feeds are kept until the new bundle is fully verified.
• Import time on a SATA SSD: ≈ 25 s for a 300 MB kit.

Quick smoke test: before import, verify the tarball carries the Go analyzer plug-in:

tar -tzf stella-ops-offline-kit-<DATE>.tgz 'plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Go/*' 'plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.DotNet/*' 'plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Python/*'

The manifest lookup above and this tar listing should both surface the Go analyzer DLL, PDB, and manifest entries before the kit is promoted.

Release guardrail. The automated release pipeline now publishes the Python plug-in from source and executes dotnet run --project tools/LanguageAnalyzerSmoke --configuration Release -- --repo-root <checkout> to validate manifest integrity and cold/warm determinism within the < 30 s / < 5 s budgets (differences versus repository goldens are logged for triage). Run ops/offline-kit/run-python-analyzer-smoke.sh locally before shipping a refreshed kit if you rebuild artefacts outside CI or when preparing the air-gap bundle.

Debug store mirror

Offline symbols (debug/.build-id/**) must accompany every Offline Kit to keep symbol lookup deterministic. The release workflow is expected to emit out/release/debug/ containing the build-id tree plus debug-manifest.json and its .sha256 companion. After a release completes:

python ops/offline-kit/mirror_debug_store.py \
  --release-dir out/release \
  --offline-dir out/offline-kit \
  --summary out/offline-kit/metadata/debug-store.json

The script mirrors the debug tree into the Offline Kit staging directory, verifies SHA-256 values against the manifest, and writes a summary under metadata/debug-store.json for audit logs. If the release pipeline does not populate out/release/debug, the tooling now logs a warning (DEVOPS-REL-17-004)—treat it as a build failure and re-run the release once symbol extraction is enabled.

---

3 · Delta patch workflow

1. Connected site fetches stella-ouk-YYYY‑MM‑DD.delta.tgz.
2. Transfer via any medium (USB, portable disk).
3. stella admin import-offline-usage-kit <delta> applies only changed CVE rows & images.

Daily deltas are < 30 MB; weekly roll‑up produces a fresh full kit.

---

4 · Quota behaviour offline

The scanner enforces the same fair‑use limits offline:

• Anonymous: {{ quota_anon }} scans per UTC day
• Free JWT: {{ quota_token }} scans per UTC day

Soft reminder at 200 scans; throttle above the ceiling but never block. See the detailed rules in 33_333_QUOTA_OVERVIEW.md.

---

5 · Troubleshooting

Symptom	Explanation	Fix
could not verify SBOM hash	Bundle corrupted in transit	Re‑download / re‑copy
Import hangs at Applying feeds…	Low disk space in /var/lib/stella	Free ≥ 2 GiB before retry
quota exceeded same day after import	Import resets counters at UTC 00:00 only	Wait until next UTC day or load a JWT

---

6 · Related documentation

• Install guide: /install/#air-gapped
• Sovereign mode rationale: /sovereign/
• Security policy: /security/#reporting-a-vulnerability
• CERT-Bund snapshots: python tools/certbund_offline_snapshot.py --help (see docs/ops/concelier-certbund-operations.md)