git.stella-ops.org/docs/airgap/symbol-bundles.md

Symbol Bundles for Air-Gapped Installations

Reference: SYMS-BUNDLE-401-014

This document describes how to create, verify, and deploy deterministic symbol bundles for air-gapped StellaOps installations.

Overview

Symbol bundles package debug symbols (PDBs, DWARF, etc.) into a single archive with:

• Deterministic ordering for reproducible builds
• BLAKE3 hashes for content verification
• DSSE signatures for authenticity
• Rekor checkpoints for transparency log integration
• Merkle inclusion proofs for offline verification

Bundle Structure

bundle-name-1.0.0.symbols.zip
├── manifest.json              # Bundle manifest with all metadata
├── symbols/
│   ├── {debug-id-1}/
│   │   ├── myapp.exe.symbols  # Symbol blob
│   │   └── myapp.exe.symbols.json  # Symbol manifest
│   ├── {debug-id-2}/
│   │   ├── libcrypto.so.symbols
│   │   └── libcrypto.so.symbols.json
│   └── ...

Creating a Bundle

Prerequisites

1. Collect symbol manifests from CI builds or ingest tools
2. Ensure all manifests follow the *.symbols.json naming convention
3. Have signing keys available (if signing is required)

Build Command

# Basic bundle creation
stella symbols bundle \
  --name "product-symbols" \
  --version "1.0.0" \
  --source ./symbols-dir \
  --output ./bundles

# With signing and Rekor submission
stella symbols bundle \
  --name "product-symbols" \
  --version "1.0.0" \
  --source ./symbols-dir \
  --output ./bundles \
  --sign \
  --key ./signing-key.pem \
  --key-id "release-key-2025" \
  --rekor \
  --rekor-url https://rekor.sigstore.dev

# Filter by platform
stella symbols bundle \
  --name "linux-symbols" \
  --version "1.0.0" \
  --source ./symbols-dir \
  --output ./bundles \
  --platform linux-x64

Bundle Options

Option	Description
--name	Bundle name (required)
--version	Bundle version in SemVer format (required)
--source	Source directory containing symbol manifests (required)
--output	Output directory for bundle archive (required)
--platform	Filter symbols by platform (e.g., linux-x64, win-x64)
--tenant	Filter symbols by tenant ID
--sign	Sign bundle with DSSE
--key	Path to signing key (PEM-encoded private key)
--key-id	Key ID for DSSE signature
--algorithm	Signing algorithm (ecdsa-p256, ed25519, rsa-pss-sha256)
--rekor	Submit to Rekor transparency log
--rekor-url	Rekor server URL
--format	Archive format: zip (default) or tar.gz
--compression	Compression level (0-9, default: 6)

Verifying a Bundle

Online Verification

stella symbols verify --bundle ./product-symbols-1.0.0.symbols.zip

Offline Verification

For air-gapped environments, include the Rekor public key:

stella symbols verify \
  --bundle ./product-symbols-1.0.0.symbols.zip \
  --public-key ./signing-public-key.pem \
  --rekor-offline \
  --rekor-key ./rekor-public-key.pem

Verification Output

Bundle verification successful!
  Bundle ID: a1b2c3d4e5f6g7h8
  Name: product-symbols-1.0.0.symbols
  Version: 1.0.0
  Signature: valid (ecdsa-p256)
  Hash verification: 42/42 valid

Extracting Symbols

Full Extraction

stella symbols extract \
  --bundle ./product-symbols-1.0.0.symbols.zip \
  --output ./extracted-symbols

Platform-Filtered Extraction

stella symbols extract \
  --bundle ./product-symbols-1.0.0.symbols.zip \
  --output ./linux-symbols \
  --platform linux-x64

Manifests Only

stella symbols extract \
  --bundle ./product-symbols-1.0.0.symbols.zip \
  --output ./manifests-only \
  --manifests-only

Inspecting Bundles

# Basic info
stella symbols inspect --bundle ./product-symbols-1.0.0.symbols.zip

# With entry listing
stella symbols inspect --bundle ./product-symbols-1.0.0.symbols.zip --entries

Bundle Manifest Schema

The bundle manifest (manifest.json) follows this schema:

{
  "schemaVersion": "stellaops.symbols.bundle/v1",
  "bundleId": "blake3-hash-of-content",
  "name": "product-symbols",
  "version": "1.0.0",
  "createdAt": "2025-12-14T10:30:00Z",
  "platform": null,
  "tenantId": null,
  "entries": [
    {
      "debugId": "abc123def456",
      "codeId": "...",
      "binaryName": "myapp.exe",
      "platform": "win-x64",
      "format": "pe",
      "manifestHash": "blake3...",
      "blobHash": "blake3...",
      "blobSizeBytes": 102400,
      "archivePath": "symbols/abc123def456/myapp.exe.symbols",
      "symbolCount": 5000
    }
  ],
  "totalSizeBytes": 10485760,
  "signature": {
    "signed": true,
    "algorithm": "ecdsa-p256",
    "keyId": "release-key-2025",
    "dsseDigest": "sha256:...",
    "signedAt": "2025-12-14T10:30:00Z",
    "publicKey": "-----BEGIN PUBLIC KEY-----..."
  },
  "rekorCheckpoint": {
    "rekorUrl": "https://rekor.sigstore.dev",
    "logEntryId": "...",
    "logIndex": 12345678,
    "integratedTime": "2025-12-14T10:30:01Z",
    "rootHash": "sha256:...",
    "treeSize": 987654321,
    "inclusionProof": {
      "logIndex": 12345678,
      "rootHash": "sha256:...",
      "treeSize": 987654321,
      "hashes": ["sha256:...", "sha256:..."]
    },
    "logPublicKey": "-----BEGIN PUBLIC KEY-----..."
  },
  "hashAlgorithm": "blake3"
}

Air-Gap Deployment Workflow

1. Create Bundle (Online Environment)

# On the online build server
stella symbols bundle \
  --name "release-v2.0.0-symbols" \
  --version "2.0.0" \
  --source /build/symbols \
  --output /export \
  --sign --key /keys/release.pem \
  --rekor

2. Transfer to Air-Gapped Environment

Copy the following files to the air-gapped environment:

• release-v2.0.0-symbols-2.0.0.symbols.zip
• release-v2.0.0-symbols-2.0.0.manifest.json
• signing-public-key.pem (if not already present)
• rekor-public-key.pem (for Rekor offline verification)

3. Verify (Air-Gapped Environment)

# On the air-gapped server
stella symbols verify \
  --bundle ./release-v2.0.0-symbols-2.0.0.symbols.zip \
  --public-key ./signing-public-key.pem \
  --rekor-offline \
  --rekor-key ./rekor-public-key.pem

4. Extract and Deploy

# Extract to symbols server directory
stella symbols extract \
  --bundle ./release-v2.0.0-symbols-2.0.0.symbols.zip \
  --output /var/stellaops/symbols \
  --verify

Determinism Guarantees

Symbol bundles are deterministic:

1. Entry ordering: Entries sorted by debug ID, then binary name (lexicographic)
2. Hash algorithm: BLAKE3 for all content hashes
3. Timestamps: UTC ISO-8601 format
4. JSON serialization: Canonical form (no whitespace, sorted keys)
5. Archive entries: Sorted by path within archive

This ensures that given the same input manifests, the same bundle (excluding signatures) is produced.

CI Integration

GitHub Actions Example

- name: Build symbol bundle
  run: |
    stella symbols bundle \
      --name "${{ github.repository }}-symbols" \
      --version "${{ github.ref_name }}" \
      --source ./build/symbols \
      --output ./dist \
      --sign --key ${{ secrets.SIGNING_KEY }} \
      --rekor

- name: Upload bundle artifact
  uses: actions/upload-artifact@v4
  with:
    name: symbol-bundle
    path: ./dist/*.symbols.zip

Troubleshooting

"No symbol manifests found"

Ensure manifests follow the *.symbols.json naming convention and are not DSSE envelopes (*.dsse.json).

"Signature verification failed"

Check that:

1. The public key matches the signing key
2. The bundle has not been modified after signing
3. The key ID matches what was used during signing

"Rekor inclusion proof invalid"

For offline verification:

1. Ensure the Rekor public key is current
2. The checkpoint was created when the log was online
3. The tree size hasn't changed since the checkpoint

Related Documentation

• Offline Kit Guide
• Symbol Server Architecture
• DSSE Signing Guide
• Rekor Integration