git.stella-ops.org/docs/modules/concelier/federation-setup.md

# Federation Setup and Operations Guide

This guide covers the setup and operation of StellaOps federation for multi-site vulnerability data synchronization.

## Overview

Federation enables secure, cursor-based synchronization of canonical vulnerability advisories between StellaOps sites. It supports:

- **Delta exports**: Only changed records since the last cursor are included
- **Air-gap transfers**: Bundles can be written to files for offline transfer
- **Multi-site topology**: Multiple sites can synchronize independently
- **Cryptographic verification**: DSSE signatures ensure bundle authenticity

## Architecture

```
┌─────────────┐     ┌─────────────┐     ┌─────────────┐
│   Site A    │────▶│   Bundle    │────▶│   Site B    │
│  (Export)   │     │   (.zst)    │     │  (Import)   │
└─────────────┘     └─────────────┘     └─────────────┘
                          │
                          ▼
                    ┌───────────┐
                    │  Site C   │
                    │ (Import)  │
                    └───────────┘
```

## Bundle Format

Federation bundles are ZST-compressed TAR archives containing:

| File | Description |
|------|-------------|
| `MANIFEST.json` | Bundle metadata, cursor, counts, hash |
| `canonicals.ndjson` | Canonical advisories (one per line) |
| `edges.ndjson` | Source edges linking advisories to sources |
| `deletions.ndjson` | Withdrawn/deleted advisory IDs |
| `SIGNATURE.json` | Optional DSSE signature envelope |

## Configuration

### Export Site Configuration

```yaml
# concelier.yaml
federation:
  enabled: true
  site_id: "us-west-1"                    # Unique site identifier
  export:
    enabled: true
    default_compression_level: 3           # ZST level (1-19)
    sign_bundles: true                     # Sign exported bundles
    max_items_per_bundle: 10000           # Maximum items per export
```

### Import Site Configuration

```yaml
# concelier.yaml
federation:
  enabled: true
  site_id: "eu-central-1"
  import:
    enabled: true
    skip_signature_verification: false     # NEVER set true in production
    allowed_sites:                         # Trusted site IDs
      - "us-west-1"
      - "ap-south-1"
    conflict_resolution: "prefer_remote"   # prefer_remote | prefer_local | fail
    force_cursor_validation: true          # Reject out-of-order imports
```

## API Endpoints

### Export Endpoints

```bash
# Export delta bundle since cursor
GET /api/v1/federation/export?since_cursor={cursor}

# Preview export (counts only)
GET /api/v1/federation/export/preview?since_cursor={cursor}

# Get federation status
GET /api/v1/federation/status
```

### Import Endpoints

```bash
# Import bundle
POST /api/v1/federation/import
Content-Type: application/zstd

# Validate bundle without importing
POST /api/v1/federation/validate
Content-Type: application/zstd

# List federated sites
GET /api/v1/federation/sites

# Update site policy
PUT /api/v1/federation/sites/{site_id}/policy
```

## CLI Commands

### Export Operations

```bash
# Export full bundle (no cursor = all data)
feedser bundle export --output bundle.zst

# Export delta since last cursor
feedser bundle export --since-cursor "2025-01-15T10:00:00Z#0001" --output delta.zst

# Preview export without creating bundle
feedser bundle preview --since-cursor "2025-01-15T10:00:00Z#0001"

# Export without signing (testing only)
feedser bundle export --no-sign --output unsigned.zst
```

### Import Operations

```bash
# Import bundle
feedser bundle import bundle.zst

# Dry run (validate without importing)
feedser bundle import bundle.zst --dry-run

# Import from stdin (pipe)
cat bundle.zst | feedser bundle import -

# Force import (skip cursor validation)
feedser bundle import bundle.zst --force
```

### Site Management

```bash
# List federated sites
feedser sites list

# Show site details
feedser sites show us-west-1

# Enable/disable site
feedser sites enable ap-south-1
feedser sites disable ap-south-1
```

## Cursor Format

Cursors use ISO-8601 timestamp with sequence number:

```
{ISO-8601 timestamp}#{sequence number}

Examples:
2025-01-15T10:00:00.000Z#0001
2025-01-15T10:00:00.000Z#0002
```

- Cursors are site-specific (each site maintains independent cursors)
- Sequence numbers distinguish concurrent exports
- Cursors are monotonically increasing within a site

## Air-Gap Transfer Workflow

For environments without network connectivity:

```bash
# On Source Site (connected to authority)
feedser bundle export --since-cursor "$LAST_CURSOR" --output /media/usb/bundle.zst
feedser bundle preview --since-cursor "$LAST_CURSOR" > /media/usb/manifest.txt

# Transfer media to target site...

# On Target Site (air-gapped)
feedser bundle import /media/usb/bundle.zst --dry-run  # Validate first
feedser bundle import /media/usb/bundle.zst            # Import
```

## Multi-Site Synchronization

### Hub-and-Spoke Topology

```
           ┌─────────────┐
           │   Hub Site  │
           │  (Primary)  │
           └──────┬──────┘
                  │
       ┌──────────┼──────────┐
       ▼          ▼          ▼
 ┌──────────┐ ┌──────────┐ ┌──────────┐
 │  Site A  │ │  Site B  │ │  Site C  │
 │ (Spoke)  │ │ (Spoke)  │ │ (Spoke)  │
 └──────────┘ └──────────┘ └──────────┘
```

### Mesh Topology

Each site can import from multiple sources:

```yaml
federation:
  import:
    allowed_sites:
      - "hub-primary"
      - "hub-secondary"  # Redundancy
```

## Merge Behavior

### Conflict Resolution

When importing, conflicts are resolved based on configuration:

| Strategy | Behavior |
|----------|----------|
| `prefer_remote` | Remote (bundle) value wins (default) |
| `prefer_local` | Local value preserved |
| `fail` | Import aborts on any conflict |

### Merge Actions

| Action | Description |
|--------|-------------|
| `Created` | New canonical added |
| `Updated` | Existing canonical updated |
| `Skipped` | No change needed (identical) |

## Verification

### Hash Verification

Bundle hash is computed over compressed content:

```
SHA256(compressed bundle content)
```

### Signature Verification

DSSE envelope contains:

```json
{
  "payloadType": "application/stellaops.federation.bundle+json",
  "payload": "base64(bundle_hash + site_id + cursor)",
  "signatures": [
    {
      "keyId": "signing-key-001",
      "algorithm": "ES256",
      "signature": "base64(signature)"
    }
  ]
}
```

## Monitoring

### Key Metrics

- `federation_export_duration_seconds` - Export time
- `federation_import_duration_seconds` - Import time
- `federation_bundle_size_bytes` - Bundle sizes
- `federation_items_processed_total` - Items processed by type
- `federation_conflicts_total` - Merge conflicts encountered

### Health Checks

```bash
# Check federation status
curl http://localhost:5000/api/v1/federation/status

# Response
{
  "site_id": "us-west-1",
  "export_enabled": true,
  "import_enabled": true,
  "last_export": "2025-01-15T10:00:00Z",
  "last_import": "2025-01-15T09:30:00Z",
  "sites_synced": 2
}
```

## Troubleshooting

### Common Issues

**Import fails with "cursor validation failed"**
- Bundle cursor is not after current site cursor
- Use `--force` to override (not recommended)
- Check if bundle was already imported

**Signature verification failed**
- Signing key not trusted on target site
- Key expired or revoked
- Use `--skip-signature` for testing only

**Large bundle timeout**
- Increase `federation.export.timeout`
- Use smaller `max_items_per_bundle`
- Stream directly to file

### Debug Logging

```yaml
logging:
  level:
    StellaOps.Concelier.Federation: Debug
```

## Security Considerations

1. **Never skip signature verification in production**
2. **Validate allowed_sites whitelist**
3. **Use TLS for API endpoints**
4. **Rotate signing keys periodically**
5. **Audit import events**
6. **Monitor for duplicate bundle imports**

## Related Documentation

- [Bundle Export Format](federation-bundle-export.md)
- [Sync Ledger Schema](../db/sync-ledger.md)
- [Signing Configuration](../security/signing.md)