git.stella-ops.org/docs/24_OFFLINE_KIT.md

Offline Update Kit (OUK) — Air‑Gap Bundle

The Offline Update Kit packages everything Stella Ops needs to run on a completely isolated network:

Component	Contents
Merged vulnerability feeds	OSV, GHSA plus optional NVD 2.0, CNNVD, CNVD, ENISA, JVN and BDU
Container images	stella-ops, Zastava sidecar, advisory-ai-web, and advisory-ai-worker (x86‑64 & arm64)
Provenance	Cosign signature, SPDX 2.3 SBOM, in‑toto SLSA attestation
Attested manifest	offline-manifest.json + detached JWS covering bundle metadata, signed during export.
Delta patches	Daily diff bundles keep size < 350 MB
Scanner plug-ins	OS analyzers plus the Node.js, Go, .NET, Python, Ruby, Rust, and PHP language analyzers packaged under plugins/scanner/analyzers/** with manifests so Workers load deterministically offline.
Debug store	.debug artefacts laid out under debug/.build-id/<aa>/<rest>.debug with debug/debug-manifest.json mapping build-ids to originating images for symbol retrieval.
Telemetry collector bundle	telemetry/telemetry-offline-bundle.tar.gz plus .sha256, containing OTLP collector config, Helm/Compose overlays, and operator instructions.
CLI + Task Packs	cli/ binaries from release/cli, Task Runner bootstrap (bootstrap/task-runner/task-runner.yaml.sample), and task-pack docs under docs/task-packs/** + docs/modules/taskrunner/**.
Orchestrator/Export/Notifier kits	Orchestrator service, worker SDK, Postgres snapshot, dashboards (orchestrator/**), Export Center bundles (export-center/**), Notifier offline packs (notifier/**).
Container air-gap bundles	Any tar/tgz under containers/ or images/ (mirrored registries) plus docs/airgap/mirror-bundles.md.
Surface.Secrets	Encrypted secrets bundles and manifests (surface-secrets/**) for sealed-mode bootstrap.

RU BDU note: ship the official Russian Trusted Root/Sub CA bundle (certificates/russian_trusted_bundle.pem) inside the kit so concelier:httpClients:source.bdu:trustedRootPaths can resolve it when the service runs in an air‑gapped network. Drop the most recent vulxml.zip alongside the kit if operators need a cold-start cache.

Language analyzers: the kit now carries the restart-only Node.js, Go, .NET, Python, Ruby, Rust, and PHP plug-ins (plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Node/, ...Lang.Go/, ...Lang.DotNet/, ...Lang.Python/, ...Lang.Ruby/, ...Lang.Rust/, ...Lang.Php/). Drop the directories alongside Worker binaries so the unified plug-in catalog can load them without outbound fetches.

Ruby analyzer features:

• Gemfile/Gemfile.lock parsing with dependency edges (version constraints, PURLs)
• OCI container layer support (layers/, .layers/, layer/) for VFS/container workspace discovery
• Ruby version detection via .ruby-version, .tool-versions, Gemfile ruby directive, and binary paths
• Native extension detection for .so, .bundle, .dll files in gem paths
• Web server config parsing for Puma, Unicorn, and Passenger configurations
• AOC-compliant observations: entrypoints (script/rack/rackup), dependency edges, runtime edges, jobs, configs, warnings
• Optional runtime evidence via TracePoint; set STELLA_RUBY_ENTRYPOINT to enable runtime capture with SHA-256 path hashing for secure evidence correlation
• CLI inspection: run stella ruby inspect --root /path/to/app to analyze a Ruby workspace locally

The PHP analyzer parses composer.lock for Composer dependencies and supports optional runtime evidence via the stella-trace.php shim; set STELLA_PHP_OPCACHE=1 to enable opcache statistics collection.

Python analyzer features:

• Wheel/sdist/editable parsing with dependency edges from METADATA, PKG-INFO, requirements.txt, and pyproject.toml
• Virtual environment support for virtualenv, venv, and conda prefix layouts
• PEP 420 namespace packages with proper importlib resolution semantics across sys.path
• Python version detection via pyproject.toml, runtime.txt, Dockerfile FROM python:*, .python-version
• Native extension detection for .so, .pyd, CFFI modules, ctypes loaders, and embedded WASM
• Framework/config heuristics for Django, Flask, FastAPI, Celery, AWS Lambda, Gunicorn, Click/Typer CLIs
• AOC-compliant observations: entrypoints (module __main__, console_scripts, zipapp), components (modules/packages/native), edges (import, namespace, dynamic-hint, native-extension) with resolver traces
• Optional runtime evidence via import hook; the bundled stellaops_trace.py module captures module load events with SHA-256 path hashing for secure evidence correlation
• CLI inspection: run stella python inspect --root /path/to/app to analyze a Python workspace locally

Surface.Env configuration: Scanner Worker and WebService components use the Surface.Env library for configuration discovery. In air-gapped deployments, configure the following environment variables (see docs/modules/scanner/design/surface-env.md for details):

Variable	Description	Air-gap Default
SCANNER_SURFACE_FS_ENDPOINT	Base URI for Surface.FS / RustFS storage	http://rustfs:8080
SCANNER_SURFACE_FS_BUCKET	Bucket for manifests/artefacts	surface-cache
SCANNER_SURFACE_CACHE_ROOT	Local cache directory	/var/lib/stellaops/surface
SCANNER_SURFACE_CACHE_QUOTA_MB	Cache quota in MB (64-262144)	4096
SCANNER_SURFACE_PREFETCH_ENABLED	Enable manifest prefetch	false
SCANNER_SURFACE_TENANT	Tenant namespace	default
SCANNER_SURFACE_SECRETS_PROVIDER	Secrets provider (file, kubernetes)	file
SCANNER_SURFACE_SECRETS_ROOT	Root path for file provider	/etc/stellaops/secrets
SCANNER_SURFACE_SECRETS_ALLOW_INLINE	Allow inline secrets	false

For Helm deployments, configure via values.yaml:

surface:
  fs:
    endpoint: "http://rustfs:8080"
    bucket: "surface-cache"
  cache:
    root: "/var/lib/stellaops/surface"
    quotaMb: 4096
  tenant: "default"
  secrets:
    provider: "file"
    root: "/etc/stellaops/secrets"

For Docker Compose, these variables are pre-configured in docker-compose.airgap.yaml with sensible defaults.

Advisory AI volume primer: ship a tarball containing empty queue/, plans/, and outputs/ directories plus their ownership metadata. During import, extract it onto the RWX volume used by advisory-ai-web and advisory-ai-worker so pods start with the expected directory tree even on air-gapped nodes.

Scanner core: C# 12 on .NET {{ dotnet }}.
Imports are idempotent and atomic — no service downtime.

0 · Prepare the debug store

Before packaging the Offline Kit, mirror the release debug artefacts (GNU build-id .debug files and the associated manifest) into the staging directory:

./ops/offline-kit/mirror_debug_store.py \
  --release-dir out/release \
  --offline-kit-dir out/offline-kit

The helper copies debug/.build-id/**, validates debug/debug-manifest.json against its recorded SHA-256, and writes out/offline-kit/metadata/debug-store.json with a short summary (platforms, artefact counts, sample build-ids). The command exits non-zero if an artefact referenced by the manifest is missing or has the wrong digest, so run it as part of every kit build.

---

0.1 · Automated packaging

The packaging workflow is scripted via ops/offline-kit/build_offline_kit.py.
It verifies the release artefacts, runs the Python analyzer smoke suite, mirrors the debug store, and emits a deterministic tarball + manifest set.

What it picks up automatically (if present under --release-dir):

• cli/** → CLI binaries and installers.
• containers/** or images/** → air-gap container bundles.
• orchestrator/{service,worker-sdk,postgres,dashboards}/**.
• export-center/**, notifier/**, surface-secrets/**.
• Docs: docs/task-packs/**, docs/modules/taskrunner/**, docs/airgap/mirror-bundles.md.

python ops/offline-kit/build_offline_kit.py \
  --version 2025.10.0 \
  --channel edge \
  --release-dir out/release \
  --staging-dir out/offline-kit/staging \
  --output-dir out/offline-kit/dist

# Optional: regenerate the telemetry collector bundle prior to packaging.
python ops/devops/telemetry/package_offline_bundle.py --output out/telemetry/telemetry-offline-bundle.tar.gz

Outputs:

• stella-ops-offline-kit-<version>-<channel>.tar.gz — bundle (mtime/uid/gid forced to zero for reproducibility)
• stella-ops-offline-kit-<version>-<channel>.tar.gz.sha256 — bundle digest
• manifest/offline-manifest.json + .sha256 — inventories every file in the bundle
• <bundle>.metadata.json — descriptor consumed by the CLI/Console import tooling; includes counts for cli, taskPacksDocs, containers, orchestrator, exportCenter, notifier, surfaceSecrets so operators can sanity-check bundle composition without unpacking
• telemetry/telemetry-offline-bundle.tar.gz + .sha256 — packaged OTLP collector assets for environments without upstream access
• plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Python/*.sig (+ .sha256) — Cosign signatures for the Python analyzer DLL and manifest

Policy Gateway configuration bundle

• Copy etc/policy-gateway.yaml (or the *.sample template if you expect operators to override values) into config/policy-gateway/policy-gateway.yaml within the staging tree.
• Include the gateway DPoP private key under secrets/policy-gateway/policy-gateway-dpop.pem and reference the location inside the manifest notes. Set the permissions explicitly (chmod 600 secrets/policy-gateway/policy-gateway-dpop.pem) so only the kit importer can read it; the importer will refuse keys that are broader.
• Document the gateway base URL and activation verification steps in docs/policy/gateway.md (bundled alongside the kit). Operators can use those curl snippets to smoke-test pack CRUD once the Offline Kit is imported.
• Ensure the Prometheus snapshot captured during packaging contains policy_gateway_activation_requests_total so auditors can reconcile activation attempts performed via the gateway during the validation window.

Provide --cosign-key / --cosign-identity-token (and optional --cosign-password) to generate Cosign signatures for both the tarball and manifest.

---

1 · Download & verify

curl -LO https://get.stella-ops.org/ouk/stella-ops-offline-kit-<DATE>.tgz
curl -LO https://get.stella-ops.org/ouk/stella-ops-offline-kit-<DATE>.tgz.sig
curl -LO https://get.stella-ops.org/ouk/offline-manifest-<DATE>.json
curl -LO https://get.stella-ops.org/ouk/offline-manifest-<DATE>.json.jws

cosign verify-blob \
  --key https://stella-ops.org/keys/cosign.pub \
  --signature stella-ops-offline-kit-<DATE>.tgz.sig \
  stella-ops-offline-kit-<DATE>.tgz

CLI shortcut. stellaops-cli offline kit pull --destination ./offline-kit downloads the bundle, manifest, and detached signatures in one step, resumes partial transfers, and writes a .metadata.json summary for later import.

Verification prints OK and the SHA‑256 digest; cross‑check against the changelog.

Validate the attested manifest before distribution:

cosign verify-blob \
  --key https://stella-ops.org/keys/cosign.pub \
  --signature offline-manifest-<DATE>.json.jws \
  offline-manifest-<DATE>.json

jq '.artifacts[] | {name, sha256, size, capturedAt}' offline-manifest-<DATE>.json

The manifest enumerates every artefact (name, sha256, size, capturedAt) and is signed with the same key registry as Authority revocation bundles. Operators can ship the manifest alongside the tarball so downstream mirrors can re-verify without unpacking the kit.

Example excerpt (2025-10-23 kit) showing the Go and .NET analyzer plug-in payloads:

{
  "name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Go/StellaOps.Scanner.Analyzers.Lang.Go.dll",
  "sha256": "a6dc850fc51151c8967ef46a3c4730f08b549667e041079431f39a8a72d0b641",
  "size": 33792,
  "capturedAt": "2025-10-23T00:00:00Z"
}
{
  "name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Go/StellaOps.Scanner.Analyzers.Lang.Go.pdb",
  "sha256": "6cbdabf155282f458b89edf267e7f6bb2441a93029aad7aad45c8a9ec58b1b3b",
  "size": 32152,
  "capturedAt": "2025-10-23T00:00:00Z"
}
{
  "name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Go/manifest.json",
  "sha256": "c19bfca2fcbb7cb18f1082b5d0d5a8f15fc799c648b50e95fce8d8b109ce48c9",
  "size": 622,
  "capturedAt": "2025-10-23T00:00:00Z"
}
{
  "name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.DotNet/StellaOps.Scanner.Analyzers.Lang.DotNet.dll",
  "sha256": "0734d23e33277ce2ccb596782d2d42cfe394b3d372dc34da9cb28b59df9b9d22",
  "size": 70144,
  "capturedAt": "2025-10-23T00:00:00Z"
}
{
  "name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.DotNet/StellaOps.Scanner.Analyzers.Lang.DotNet.pdb",
  "sha256": "b853c1ff4b196715f5bd1447e1a13edeb4940917527ec9bf153b5048da49abaf",
  "size": 40400,
  "capturedAt": "2025-10-23T00:00:00Z"
}
{
  "name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.DotNet/manifest.json",
  "sha256": "5d483885f825f01bfd9943dcf2889ec2e0beba38ede92ecfe67d4f506cf14e37",
  "size": 647,
  "capturedAt": "2025-10-23T00:00:00Z"
}
{
  "name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Python/StellaOps.Scanner.Analyzers.Lang.Python.dll",
  "sha256": "a4f558f363394096e3dd6263f35b180b93b4112f9cf616c05872da8a8657d518",
  "size": 47104,
  "capturedAt": "2025-10-26T00:00:00Z"
}
{
  "name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Python/StellaOps.Scanner.Analyzers.Lang.Python.pdb",
  "sha256": "ef2ad78bc2cd1d7e99bae000b92357aa9a9c32938501899e9033d001096196d0",
  "size": 31896,
  "capturedAt": "2025-10-26T00:00:00Z"
}
{
  "name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Python/manifest.json",
  "sha256": "668ad9a1a35485628677b639db4d996d1e25f62021680a81a22482483800e557",
  "size": 648,
  "capturedAt": "2025-10-26T00:00:00Z"
}
{
  "name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Ruby/StellaOps.Scanner.Analyzers.Lang.Ruby.dll",
  "sha256": "<computed-at-release>",
  "size": 0,
  "capturedAt": "2025-11-27T00:00:00Z"
}
{
  "name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Ruby/StellaOps.Scanner.Analyzers.Lang.Ruby.pdb",
  "sha256": "<computed-at-release>",
  "size": 0,
  "capturedAt": "2025-11-27T00:00:00Z"
}
{
  "name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Ruby/manifest.json",
  "sha256": "<computed-at-release>",
  "size": 0,
  "capturedAt": "2025-11-27T00:00:00Z"
}
{
  "name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Rust/StellaOps.Scanner.Analyzers.Lang.Rust.dll",
  "sha256": "d90ba8b6ace7d98db563b1dec178d57ac09df474e1342fa1daa38bd55e17b185",
  "size": 54784,
  "capturedAt": "2025-11-01T00:00:00Z"
}
{
  "name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Rust/StellaOps.Scanner.Analyzers.Lang.Rust.pdb",
  "sha256": "6fac88640a4980d2bb8f7ea2dd2f3d0a521b90fd30ae3a84981575d5f76fa3df",
  "size": 36636,
  "capturedAt": "2025-11-01T00:00:00Z"
}
{
  "name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Rust/manifest.json",
  "sha256": "1ec47d1a2103ad5eff23e903532cb76b1ed7ded85d301c1a6631ff21aa966ed4",
  "size": 658,
  "capturedAt": "2025-11-01T00:00:00Z"
}

---

2 · Import on the air‑gapped host

docker compose --env-file .env \
  -f docker-compose.stella-ops.yml \
  exec stella-ops \
  stella admin import-offline-usage-kit stella-ops-offline-kit-<DATE>.tgz

Alternatively, run

stellaops-cli offline kit import stella-ops-offline-kit-<DATE>.tgz \
  --manifest offline-manifest-<DATE>.json \
  --bundle-signature stella-ops-offline-kit-<DATE>.tgz.sig \
  --manifest-signature offline-manifest-<DATE>.json.jws

The CLI validates recorded digests (when .metadata.json is present) before streaming the multipart payload to /api/offline-kit/import.

• The CLI validates the Cosign signature before activation.
• Old feeds are kept until the new bundle is fully verified.
• Import time on a SATA SSD: ≈ 25 s for a 300 MB kit.

2.1 Validator + idempotency enablement (air-gap)

The Offline Kit carries the same helper scripts under scripts/:

1. Duplicate audit: run

   mongo concelier ops/devops/scripts/check-advisory-raw-duplicates.js --eval 'var LIMIT=200;'
   

   to verify no (vendor, upstream_id, content_hash, tenant) conflicts remain before enabling the idempotency index.
2. Apply validators: execute mongo concelier ops/devops/scripts/apply-aoc-validators.js (and the Excititor equivalent) with validationLevel: "moderate" in maintenance mode.
3. Restart Concelier so migrations 20251028_advisory_raw_idempotency_index and 20251028_advisory_supersedes_backfill run automatically. After the restart:
   • Confirm db.advisory resolves to a view on advisory_backup_20251028.
   • Spot-check a few advisory_raw entries to ensure supersedes chains are populated deterministically.
4. Smoke test: run stella sources ingest --dry-run --fixture advisory (bundled fixtures) to confirm ingestion succeeds post-guard and the CLI reports zero violations.

Authority scope sanity check

Offline installs rely on the bundled etc/authority.yaml.sample. Before promoting the kit, confirm the sample clients keep the Aggregation-Only guardrails:

• aoc-verifier requests aoc:verify, advisory:read, and vex:read.
• signals-uploader requests signals:write, signals:read, and aoc:verify.
• airgap-operator requests airgap:status:read, airgap:import, and airgap:seal.
• task-runner requests packs.run and packs.read for execution flows.
• pack-approver requests packs.approve (plus packs.read) for automation that resumes runs after approvals.
• packs-registry requests packs.write and packs.read for publishing bundles.

Authority now rejects tokens that request advisory:read, vex:read, or any signals:* scope without aoc:verify; the sample has been updated to match. Air-gap scopes (airgap:*) also require an explicit tenant assignment—match the updated roles (airgap-viewer, airgap-operator, airgap-admin) so automation fails closed when misconfigured.

Quick smoke test: before import, verify the tarball carries the Go analyzer plug-in:

tar -tzf stella-ops-offline-kit-<DATE>.tgz 'plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Go/*' 'plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.DotNet/*' 'plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Python/*' 'plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Ruby/*' 'plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Php/*'

The manifest lookup above and this tar listing should both surface the Go analyzer DLL, PDB, and manifest entries before the kit is promoted.

Release guardrail. The automated release pipeline now publishes the Python, Ruby, Rust, and PHP plug-ins from source and executes dotnet run --project src/Tools/LanguageAnalyzerSmoke --configuration Release -- --repo-root <checkout> --analyzer <id> to validate manifest integrity and cold/warm determinism within the < 30 s / < 5 s budgets (differences versus repository goldens are logged for triage). Run ops/offline-kit/run-python-analyzer-smoke.sh, ops/offline-kit/run-ruby-analyzer-smoke.sh, ops/offline-kit/run-rust-analyzer-smoke.sh, and ops/offline-kit/run-php-analyzer-smoke.sh locally before shipping a refreshed kit if you rebuild artefacts outside CI or when preparing the air-gap bundle.

Debug store mirror

Offline symbols (debug/.build-id/**) must accompany every Offline Kit to keep symbol lookup deterministic. The release workflow is expected to emit out/release/debug/ containing the build-id tree plus debug-manifest.json and its .sha256 companion. After a release completes:

python ops/offline-kit/mirror_debug_store.py \
  --release-dir out/release \
  --offline-dir out/offline-kit \
  --summary out/offline-kit/metadata/debug-store.json

The script mirrors the debug tree into the Offline Kit staging directory, verifies SHA-256 values against the manifest, and writes a summary under metadata/debug-store.json for audit logs. If the release pipeline does not populate out/release/debug, the tooling now logs a warning (DEVOPS-REL-17-004)—treat it as a build failure and re-run the release once symbol extraction is enabled.

---

3 · Delta patch workflow

1. Connected site fetches stella-ouk-YYYY‑MM‑DD.delta.tgz.
2. Transfer via any medium (USB, portable disk).
3. stella admin import-offline-usage-kit <delta> applies only changed CVE rows & images.

Daily deltas are < 30 MB; weekly roll‑up produces a fresh full kit.

---

4 · Quota behaviour offline

The scanner enforces the same fair‑use limits offline:

• Anonymous: {{ quota_anon }} scans per UTC day
• Free JWT: {{ quota_token }} scans per UTC day

Soft reminder at 200 scans; throttle above the ceiling but never block. See the detailed rules in 33_333_QUOTA_OVERVIEW.md.

---

5 · Troubleshooting

Symptom	Explanation	Fix
could not verify SBOM hash	Bundle corrupted in transit	Re‑download / re‑copy
Import hangs at Applying feeds…	Low disk space in /var/lib/stella	Free ≥ 2 GiB before retry
quota exceeded same day after import	Import resets counters at UTC 00:00 only	Wait until next UTC day or load a JWT

---

6 · Related documentation

• Install guide: /install/#air-gapped
• Sovereign mode rationale: /sovereign/
• Security policy: /security/#reporting-a-vulnerability
• CERT-Bund snapshots: python src/Tools/certbund_offline_snapshot.py --help (see docs/modules/concelier/operations/connectors/certbund.md)