git.stella-ops.org/docs/airgap/symbol-bundles.md

# Symbol Bundles for Air-Gapped Installations

**Reference:** SYMS-BUNDLE-401-014

This document describes how to create, verify, and deploy deterministic symbol bundles for air-gapped StellaOps installations.

## Overview

Symbol bundles package debug symbols (PDBs, DWARF, etc.) into a single archive with:
- **Deterministic ordering** for reproducible builds
- **BLAKE3 hashes** for content verification
- **DSSE signatures** for authenticity
- **Rekor checkpoints** for transparency log integration
- **Merkle inclusion proofs** for offline verification

## Bundle Structure

```
bundle-name-1.0.0.symbols.zip
├── manifest.json              # Bundle manifest with all metadata
├── symbols/
│   ├── {debug-id-1}/
│   │   ├── myapp.exe.symbols  # Symbol blob
│   │   └── myapp.exe.symbols.json  # Symbol manifest
│   ├── {debug-id-2}/
│   │   ├── libcrypto.so.symbols
│   │   └── libcrypto.so.symbols.json
│   └── ...
```

## Creating a Bundle

### Prerequisites

1. Collect symbol manifests from CI builds or ingest tools
2. Ensure all manifests follow the `*.symbols.json` naming convention
3. Have signing keys available (if signing is required)

### Build Command

```bash
# Basic bundle creation
stella symbols bundle \
  --name "product-symbols" \
  --version "1.0.0" \
  --source ./symbols-dir \
  --output ./bundles

# With signing and Rekor submission
stella symbols bundle \
  --name "product-symbols" \
  --version "1.0.0" \
  --source ./symbols-dir \
  --output ./bundles \
  --sign \
  --key ./signing-key.pem \
  --key-id "release-key-2025" \
  --rekor \
  --rekor-url https://rekor.sigstore.dev

# Filter by platform
stella symbols bundle \
  --name "linux-symbols" \
  --version "1.0.0" \
  --source ./symbols-dir \
  --output ./bundles \
  --platform linux-x64
```

### Bundle Options

| Option | Description |
|--------|-------------|
| `--name` | Bundle name (required) |
| `--version` | Bundle version in SemVer format (required) |
| `--source` | Source directory containing symbol manifests (required) |
| `--output` | Output directory for bundle archive (required) |
| `--platform` | Filter symbols by platform (e.g., linux-x64, win-x64) |
| `--tenant` | Filter symbols by tenant ID |
| `--sign` | Sign bundle with DSSE |
| `--key` | Path to signing key (PEM-encoded private key) |
| `--key-id` | Key ID for DSSE signature |
| `--algorithm` | Signing algorithm (ecdsa-p256, ed25519, rsa-pss-sha256) |
| `--rekor` | Submit to Rekor transparency log |
| `--rekor-url` | Rekor server URL |
| `--format` | Archive format: zip (default) or tar.gz |
| `--compression` | Compression level (0-9, default: 6) |

## Verifying a Bundle

### Online Verification

```bash
stella symbols verify --bundle ./product-symbols-1.0.0.symbols.zip
```

### Offline Verification

For air-gapped environments, include the Rekor public key:

```bash
stella symbols verify \
  --bundle ./product-symbols-1.0.0.symbols.zip \
  --public-key ./signing-public-key.pem \
  --rekor-offline \
  --rekor-key ./rekor-public-key.pem
```

### Verification Output

```
Bundle verification successful!
  Bundle ID: a1b2c3d4e5f6g7h8
  Name: product-symbols-1.0.0.symbols
  Version: 1.0.0
  Signature: valid (ecdsa-p256)
  Hash verification: 42/42 valid
```

## Extracting Symbols

### Full Extraction

```bash
stella symbols extract \
  --bundle ./product-symbols-1.0.0.symbols.zip \
  --output ./extracted-symbols
```

### Platform-Filtered Extraction

```bash
stella symbols extract \
  --bundle ./product-symbols-1.0.0.symbols.zip \
  --output ./linux-symbols \
  --platform linux-x64
```

### Manifests Only

```bash
stella symbols extract \
  --bundle ./product-symbols-1.0.0.symbols.zip \
  --output ./manifests-only \
  --manifests-only
```

## Inspecting Bundles

```bash
# Basic info
stella symbols inspect --bundle ./product-symbols-1.0.0.symbols.zip

# With entry listing
stella symbols inspect --bundle ./product-symbols-1.0.0.symbols.zip --entries
```

## Bundle Manifest Schema

The bundle manifest (`manifest.json`) follows this schema:

```json
{
  "schemaVersion": "stellaops.symbols.bundle/v1",
  "bundleId": "blake3-hash-of-content",
  "name": "product-symbols",
  "version": "1.0.0",
  "createdAt": "2025-12-14T10:30:00Z",
  "platform": null,
  "tenantId": null,
  "entries": [
    {
      "debugId": "abc123def456",
      "codeId": "...",
      "binaryName": "myapp.exe",
      "platform": "win-x64",
      "format": "pe",
      "manifestHash": "blake3...",
      "blobHash": "blake3...",
      "blobSizeBytes": 102400,
      "archivePath": "symbols/abc123def456/myapp.exe.symbols",
      "symbolCount": 5000
    }
  ],
  "totalSizeBytes": 10485760,
  "signature": {
    "signed": true,
    "algorithm": "ecdsa-p256",
    "keyId": "release-key-2025",
    "dsseDigest": "sha256:...",
    "signedAt": "2025-12-14T10:30:00Z",
    "publicKey": "-----BEGIN PUBLIC KEY-----..."
  },
  "rekorCheckpoint": {
    "rekorUrl": "https://rekor.sigstore.dev",
    "logEntryId": "...",
    "logIndex": 12345678,
    "integratedTime": "2025-12-14T10:30:01Z",
    "rootHash": "sha256:...",
    "treeSize": 987654321,
    "inclusionProof": {
      "logIndex": 12345678,
      "rootHash": "sha256:...",
      "treeSize": 987654321,
      "hashes": ["sha256:...", "sha256:..."]
    },
    "logPublicKey": "-----BEGIN PUBLIC KEY-----..."
  },
  "hashAlgorithm": "blake3"
}
```

## Air-Gap Deployment Workflow

### 1. Create Bundle (Online Environment)

```bash
# On the online build server
stella symbols bundle \
  --name "release-v2.0.0-symbols" \
  --version "2.0.0" \
  --source /build/symbols \
  --output /export \
  --sign --key /keys/release.pem \
  --rekor
```

### 2. Transfer to Air-Gapped Environment

Copy the following files to the air-gapped environment:
- `release-v2.0.0-symbols-2.0.0.symbols.zip`
- `release-v2.0.0-symbols-2.0.0.manifest.json`
- `signing-public-key.pem` (if not already present)
- `rekor-public-key.pem` (for Rekor offline verification)

### 3. Verify (Air-Gapped Environment)

```bash
# On the air-gapped server
stella symbols verify \
  --bundle ./release-v2.0.0-symbols-2.0.0.symbols.zip \
  --public-key ./signing-public-key.pem \
  --rekor-offline \
  --rekor-key ./rekor-public-key.pem
```

### 4. Extract and Deploy

```bash
# Extract to symbols server directory
stella symbols extract \
  --bundle ./release-v2.0.0-symbols-2.0.0.symbols.zip \
  --output /var/stellaops/symbols \
  --verify
```

## Determinism Guarantees

Symbol bundles are deterministic:

1. **Entry ordering**: Entries sorted by debug ID, then binary name (lexicographic)
2. **Hash algorithm**: BLAKE3 for all content hashes
3. **Timestamps**: UTC ISO-8601 format
4. **JSON serialization**: Canonical form (no whitespace, sorted keys)
5. **Archive entries**: Sorted by path within archive

This ensures that given the same input manifests, the same bundle (excluding signatures) is produced.

## CI Integration

### GitHub Actions Example

```yaml
- name: Build symbol bundle
  run: |
    stella symbols bundle \
      --name "${{ github.repository }}-symbols" \
      --version "${{ github.ref_name }}" \
      --source ./build/symbols \
      --output ./dist \
      --sign --key ${{ secrets.SIGNING_KEY }} \
      --rekor

- name: Upload bundle artifact
  uses: actions/upload-artifact@v4
  with:
    name: symbol-bundle
    path: ./dist/*.symbols.zip
```

## Troubleshooting

### "No symbol manifests found"

Ensure manifests follow the `*.symbols.json` naming convention and are not DSSE envelopes (`*.dsse.json`).

### "Signature verification failed"

Check that:
1. The public key matches the signing key
2. The bundle has not been modified after signing
3. The key ID matches what was used during signing

### "Rekor inclusion proof invalid"

For offline verification:
1. Ensure the Rekor public key is current
2. The checkpoint was created when the log was online
3. The tree size hasn't changed since the checkpoint

## Related Documentation

- [Offline Kit Guide](../24_OFFLINE_KIT.md)
- [Symbol Server Architecture](../modules/scanner/architecture.md)
- [DSSE Signing Guide](../modules/signer/architecture.md)
- [Rekor Integration](../modules/attestor/architecture.md)