git.stella-ops.org/docs/24_OFFLINE_KIT.md

Offline Update Kit (OUK) — Air‑Gap Bundle

The Offline Update Kit packages everything Stella Ops needs to run on a completely isolated network:

Component	Contents
Merged vulnerability feeds	OSV, GHSA plus optional NVD 2.0, CNNVD, CNVD, ENISA, JVN and BDU
Container images	stella-ops, Zastava sidecar (x86‑64 & arm64)
Provenance	Cosign signature, SPDX 2.3 SBOM, in‑toto SLSA attestation
Attested manifest	offline-manifest.json + detached JWS covering bundle metadata, signed during export.
Delta patches	Daily diff bundles keep size < 350 MB
Scanner plug-ins	OS analyzers plus the Node.js, Go, .NET, and Python language analyzers packaged under plugins/scanner/analyzers/** with manifests so Workers load deterministically offline.
Debug store	.debug artefacts laid out under debug/.build-id/<aa>/<rest>.debug with debug/debug-manifest.json mapping build-ids to originating images for symbol retrieval.
Telemetry collector bundle	telemetry/telemetry-offline-bundle.tar.gz plus .sha256, containing OTLP collector config, Helm/Compose overlays, and operator instructions.

RU BDU note: ship the official Russian Trusted Root/Sub CA bundle (certificates/russian_trusted_bundle.pem) inside the kit so concelier:httpClients:source.bdu:trustedRootPaths can resolve it when the service runs in an air‑gapped network. Drop the most recent vulxml.zip alongside the kit if operators need a cold-start cache.

Language analyzers: the kit now carries the restart-only Node.js, Go, .NET, and Python analyzer plug-ins (plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Node/, ...Lang.Go/, ...Lang.DotNet/, ...Lang.Python/). Drop the directories alongside Worker binaries so the unified plug-in catalog can load them without outbound fetches; Rust remains on the Wave 4 roadmap.

Scanner core: C# 12 on .NET {{ dotnet }}.
Imports are idempotent and atomic — no service downtime.

0 · Prepare the debug store

Before packaging the Offline Kit, mirror the release debug artefacts (GNU build-id .debug files and the associated manifest) into the staging directory:

./ops/offline-kit/mirror_debug_store.py \
  --release-dir out/release \
  --offline-kit-dir out/offline-kit

The helper copies debug/.build-id/**, validates debug/debug-manifest.json against its recorded SHA-256, and writes out/offline-kit/metadata/debug-store.json with a short summary (platforms, artefact counts, sample build-ids). The command exits non-zero if an artefact referenced by the manifest is missing or has the wrong digest, so run it as part of every kit build.

---

0.1 · Automated packaging

The packaging workflow is scripted via ops/offline-kit/build_offline_kit.py.
It verifies the release artefacts, runs the Python analyzer smoke suite, mirrors the debug store, and emits a deterministic tarball + manifest set.

python ops/offline-kit/build_offline_kit.py \
  --version 2025.10.0 \
  --channel edge \
  --release-dir out/release \
  --staging-dir out/offline-kit/staging \
  --output-dir out/offline-kit/dist

# Optional: regenerate the telemetry collector bundle prior to packaging.
python ops/devops/telemetry/package_offline_bundle.py --output out/telemetry/telemetry-offline-bundle.tar.gz

Outputs:

• stella-ops-offline-kit-<version>-<channel>.tar.gz — bundle (mtime/uid/gid forced to zero for reproducibility)
• stella-ops-offline-kit-<version>-<channel>.tar.gz.sha256 — bundle digest
• manifest/offline-manifest.json + .sha256 — inventories every file in the bundle
• <bundle>.metadata.json — descriptor consumed by the CLI/Console import tooling
• telemetry/telemetry-offline-bundle.tar.gz + .sha256 — packaged OTLP collector assets for environments without upstream access
• plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Python/*.sig (+ .sha256) — Cosign signatures for the Python analyzer DLL and manifest

Policy Gateway configuration bundle

• Copy etc/policy-gateway.yaml (or the *.sample template if you expect operators to override values) into config/policy-gateway/policy-gateway.yaml within the staging tree.
• Include the gateway DPoP private key under secrets/policy-gateway/policy-gateway-dpop.pem and reference the location inside the manifest notes. Set the permissions explicitly (chmod 600 secrets/policy-gateway/policy-gateway-dpop.pem) so only the kit importer can read it; the importer will refuse keys that are broader.
• Document the gateway base URL and activation verification steps in docs/policy/gateway.md (bundled alongside the kit). Operators can use those curl snippets to smoke-test pack CRUD once the Offline Kit is imported.
• Ensure the Prometheus snapshot captured during packaging contains policy_gateway_activation_requests_total so auditors can reconcile activation attempts performed via the gateway during the validation window.

Provide --cosign-key / --cosign-identity-token (and optional --cosign-password) to generate Cosign signatures for both the tarball and manifest.

---

1 · Download & verify

curl -LO https://get.stella-ops.org/ouk/stella-ops-offline-kit-<DATE>.tgz
curl -LO https://get.stella-ops.org/ouk/stella-ops-offline-kit-<DATE>.tgz.sig
curl -LO https://get.stella-ops.org/ouk/offline-manifest-<DATE>.json
curl -LO https://get.stella-ops.org/ouk/offline-manifest-<DATE>.json.jws

cosign verify-blob \
  --key https://stella-ops.org/keys/cosign.pub \
  --signature stella-ops-offline-kit-<DATE>.tgz.sig \
  stella-ops-offline-kit-<DATE>.tgz

CLI shortcut. stellaops-cli offline kit pull --destination ./offline-kit downloads the bundle, manifest, and detached signatures in one step, resumes partial transfers, and writes a .metadata.json summary for later import.

Verification prints OK and the SHA‑256 digest; cross‑check against the changelog.

Validate the attested manifest before distribution:

cosign verify-blob \
  --key https://stella-ops.org/keys/cosign.pub \
  --signature offline-manifest-<DATE>.json.jws \
  offline-manifest-<DATE>.json

jq '.artifacts[] | {name, sha256, size, capturedAt}' offline-manifest-<DATE>.json

The manifest enumerates every artefact (name, sha256, size, capturedAt) and is signed with the same key registry as Authority revocation bundles. Operators can ship the manifest alongside the tarball so downstream mirrors can re-verify without unpacking the kit.

Example excerpt (2025-10-23 kit) showing the Go and .NET analyzer plug-in payloads:

{
  "name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Go/StellaOps.Scanner.Analyzers.Lang.Go.dll",
  "sha256": "a6dc850fc51151c8967ef46a3c4730f08b549667e041079431f39a8a72d0b641",
  "size": 33792,
  "capturedAt": "2025-10-23T00:00:00Z"
}
{
  "name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Go/StellaOps.Scanner.Analyzers.Lang.Go.pdb",
  "sha256": "6cbdabf155282f458b89edf267e7f6bb2441a93029aad7aad45c8a9ec58b1b3b",
  "size": 32152,
  "capturedAt": "2025-10-23T00:00:00Z"
}
{
  "name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Go/manifest.json",
  "sha256": "c19bfca2fcbb7cb18f1082b5d0d5a8f15fc799c648b50e95fce8d8b109ce48c9",
  "size": 622,
  "capturedAt": "2025-10-23T00:00:00Z"
}
{
  "name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.DotNet/StellaOps.Scanner.Analyzers.Lang.DotNet.dll",
  "sha256": "0734d23e33277ce2ccb596782d2d42cfe394b3d372dc34da9cb28b59df9b9d22",
  "size": 70144,
  "capturedAt": "2025-10-23T00:00:00Z"
}
{
  "name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.DotNet/StellaOps.Scanner.Analyzers.Lang.DotNet.pdb",
  "sha256": "b853c1ff4b196715f5bd1447e1a13edeb4940917527ec9bf153b5048da49abaf",
  "size": 40400,
  "capturedAt": "2025-10-23T00:00:00Z"
}
{
  "name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.DotNet/manifest.json",
  "sha256": "5d483885f825f01bfd9943dcf2889ec2e0beba38ede92ecfe67d4f506cf14e37",
  "size": 647,
  "capturedAt": "2025-10-23T00:00:00Z"
}
{
  "name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Python/StellaOps.Scanner.Analyzers.Lang.Python.dll",
  "sha256": "a4f558f363394096e3dd6263f35b180b93b4112f9cf616c05872da8a8657d518",
  "size": 47104,
  "capturedAt": "2025-10-26T00:00:00Z"
}
{
  "name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Python/StellaOps.Scanner.Analyzers.Lang.Python.pdb",
  "sha256": "ef2ad78bc2cd1d7e99bae000b92357aa9a9c32938501899e9033d001096196d0",
  "size": 31896,
  "capturedAt": "2025-10-26T00:00:00Z"
}
{
  "name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Python/manifest.json",
  "sha256": "668ad9a1a35485628677b639db4d996d1e25f62021680a81a22482483800e557",
  "size": 648,
  "capturedAt": "2025-10-26T00:00:00Z"
}

---

2 · Import on the air‑gapped host

docker compose --env-file .env \
  -f docker-compose.stella-ops.yml \
  exec stella-ops \
  stella admin import-offline-usage-kit stella-ops-offline-kit-<DATE>.tgz

Alternatively, run

stellaops-cli offline kit import stella-ops-offline-kit-<DATE>.tgz \
  --manifest offline-manifest-<DATE>.json \
  --bundle-signature stella-ops-offline-kit-<DATE>.tgz.sig \
  --manifest-signature offline-manifest-<DATE>.json.jws

The CLI validates recorded digests (when .metadata.json is present) before streaming the multipart payload to /api/offline-kit/import.

• The CLI validates the Cosign signature before activation.
• Old feeds are kept until the new bundle is fully verified.
• Import time on a SATA SSD: ≈ 25 s for a 300 MB kit.

2.1 Validator + idempotency enablement (air-gap)

The Offline Kit carries the same helper scripts under scripts/:

1. Duplicate audit: run

   mongo concelier ops/devops/scripts/check-advisory-raw-duplicates.js --eval 'var LIMIT=200;'
   

   to verify no (vendor, upstream_id, content_hash, tenant) conflicts remain before enabling the idempotency index.
2. Apply validators: execute mongo concelier ops/devops/scripts/apply-aoc-validators.js (and the Excititor equivalent) with validationLevel: "moderate" in maintenance mode.
3. Restart Concelier so migrations 20251028_advisory_raw_idempotency_index and 20251028_advisory_supersedes_backfill run automatically. After the restart:
   • Confirm db.advisory resolves to a view on advisory_backup_20251028.
   • Spot-check a few advisory_raw entries to ensure supersedes chains are populated deterministically.
4. Smoke test: run stella sources ingest --dry-run --fixture advisory (bundled fixtures) to confirm ingestion succeeds post-guard and the CLI reports zero violations.

Authority scope sanity check

Offline installs rely on the bundled etc/authority.yaml.sample. Before promoting the kit, confirm the sample clients keep the Aggregation-Only guardrails:

• aoc-verifier requests aoc:verify, advisory:read, and vex:read.
• signals-uploader requests signals:write, signals:read, and aoc:verify.

Authority now rejects tokens that request advisory:read, vex:read, or any signals:* scope without aoc:verify; the sample has been updated to match. If you maintain tenant-specific overlays, mirror the same pairing so air-gapped automation fails deterministically with invalid_scope when misconfigured.

Quick smoke test: before import, verify the tarball carries the Go analyzer plug-in:

tar -tzf stella-ops-offline-kit-<DATE>.tgz 'plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Go/*' 'plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.DotNet/*' 'plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Python/*'

The manifest lookup above and this tar listing should both surface the Go analyzer DLL, PDB, and manifest entries before the kit is promoted.

Release guardrail. The automated release pipeline now publishes the Python plug-in from source and executes dotnet run --project tools/LanguageAnalyzerSmoke --configuration Release -- --repo-root <checkout> to validate manifest integrity and cold/warm determinism within the < 30 s / < 5 s budgets (differences versus repository goldens are logged for triage). Run ops/offline-kit/run-python-analyzer-smoke.sh locally before shipping a refreshed kit if you rebuild artefacts outside CI or when preparing the air-gap bundle.

Debug store mirror

Offline symbols (debug/.build-id/**) must accompany every Offline Kit to keep symbol lookup deterministic. The release workflow is expected to emit out/release/debug/ containing the build-id tree plus debug-manifest.json and its .sha256 companion. After a release completes:

python ops/offline-kit/mirror_debug_store.py \
  --release-dir out/release \
  --offline-dir out/offline-kit \
  --summary out/offline-kit/metadata/debug-store.json

The script mirrors the debug tree into the Offline Kit staging directory, verifies SHA-256 values against the manifest, and writes a summary under metadata/debug-store.json for audit logs. If the release pipeline does not populate out/release/debug, the tooling now logs a warning (DEVOPS-REL-17-004)—treat it as a build failure and re-run the release once symbol extraction is enabled.

---

3 · Delta patch workflow

1. Connected site fetches stella-ouk-YYYY‑MM‑DD.delta.tgz.
2. Transfer via any medium (USB, portable disk).
3. stella admin import-offline-usage-kit <delta> applies only changed CVE rows & images.

Daily deltas are < 30 MB; weekly roll‑up produces a fresh full kit.

---

4 · Quota behaviour offline

The scanner enforces the same fair‑use limits offline:

• Anonymous: {{ quota_anon }} scans per UTC day
• Free JWT: {{ quota_token }} scans per UTC day

Soft reminder at 200 scans; throttle above the ceiling but never block. See the detailed rules in 33_333_QUOTA_OVERVIEW.md.

---

5 · Troubleshooting

Symptom	Explanation	Fix
could not verify SBOM hash	Bundle corrupted in transit	Re‑download / re‑copy
Import hangs at Applying feeds…	Low disk space in /var/lib/stella	Free ≥ 2 GiB before retry
quota exceeded same day after import	Import resets counters at UTC 00:00 only	Wait until next UTC day or load a JWT

---

6 · Related documentation

• Install guide: /install/#air-gapped
• Sovereign mode rationale: /sovereign/
• Security policy: /security/#reporting-a-vulnerability
• CERT-Bund snapshots: python tools/certbund_offline_snapshot.py --help (see docs/ops/concelier-certbund-operations.md)