279 lines
7.0 KiB
Markdown
279 lines
7.0 KiB
Markdown
# Federation Bundle Export
|
|
|
|
Per SPRINT_8200_0014_0002.
|
|
|
|
## Overview
|
|
|
|
Federation bundles enable multi-site synchronization of canonical advisory data. Each bundle contains a delta of changes since a specified cursor position, allowing incremental sync between federated Concelier instances.
|
|
|
|
## Bundle Format
|
|
|
|
Bundles use a TAR archive compressed with ZStandard (ZST):
|
|
|
|
```
|
|
feedser-bundle-v1.zst
|
|
├── MANIFEST.json # Bundle metadata
|
|
├── canonicals.ndjson # Canonical advisories (one per line)
|
|
├── edges.ndjson # Source edges (one per line)
|
|
├── deletions.ndjson # Withdrawn/deleted canonical IDs
|
|
└── SIGNATURE.json # DSSE envelope (optional)
|
|
```
|
|
|
|
### MANIFEST.json
|
|
|
|
```json
|
|
{
|
|
"version": "feedser-bundle/1.0",
|
|
"site_id": "site-us-west-1",
|
|
"export_cursor": "2025-01-15T10:30:00.000Z#0042",
|
|
"since_cursor": "2025-01-14T00:00:00.000Z#0000",
|
|
"exported_at": "2025-01-15T10:30:15.123Z",
|
|
"counts": {
|
|
"canonicals": 1234,
|
|
"edges": 3456,
|
|
"deletions": 12,
|
|
"total": 4702
|
|
},
|
|
"bundle_hash": "sha256:a1b2c3d4..."
|
|
}
|
|
```
|
|
|
|
| Field | Type | Description |
|
|
|-------|------|-------------|
|
|
| `version` | string | Bundle format version identifier |
|
|
| `site_id` | string | Identifier of the exporting site |
|
|
| `export_cursor` | string | Cursor position after this export |
|
|
| `since_cursor` | string? | Cursor position from which changes were exported (null for full export) |
|
|
| `exported_at` | ISO8601 | Timestamp when bundle was created |
|
|
| `counts` | object | Item counts by type |
|
|
| `bundle_hash` | string | SHA256 hash of compressed bundle content |
|
|
|
|
### canonicals.ndjson
|
|
|
|
Each line contains a canonical advisory record:
|
|
|
|
```json
|
|
{"id":"uuid","cve":"CVE-2024-1234","affects_key":"pkg:npm/express@4.0.0","merge_hash":"a1b2c3...","status":"active","severity":"high","title":"..."}
|
|
```
|
|
|
|
### edges.ndjson
|
|
|
|
Each line contains a source edge linking a canonical to its source advisory:
|
|
|
|
```json
|
|
{"id":"uuid","canonical_id":"uuid","source":"nvd","source_advisory_id":"CVE-2024-1234","vendor_status":"affected"}
|
|
```
|
|
|
|
### deletions.ndjson
|
|
|
|
Each line contains a deletion record for withdrawn or deleted canonicals:
|
|
|
|
```json
|
|
{"canonical_id":"uuid","deleted_at":"2025-01-15T10:00:00Z","reason":"withdrawn"}
|
|
```
|
|
|
|
### SIGNATURE.json
|
|
|
|
When signing is enabled, contains a DSSE envelope over the bundle hash:
|
|
|
|
```json
|
|
{
|
|
"payload_type": "application/vnd.stellaops.bundle-hash+json",
|
|
"payload": "eyJidW5kbGVfaGFzaCI6InNoYTI1NjphMWIy..."}",
|
|
"signatures": [
|
|
{
|
|
"keyid": "sha256:xyz...",
|
|
"sig": "MEUCIQD..."
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
## API Endpoints
|
|
|
|
### Export Bundle
|
|
|
|
```
|
|
GET /api/v1/federation/export
|
|
```
|
|
|
|
Exports a delta bundle for federation sync.
|
|
|
|
**Query Parameters:**
|
|
|
|
| Parameter | Type | Default | Description |
|
|
|-----------|------|---------|-------------|
|
|
| `since_cursor` | string | null | Export changes since this cursor (null = full export) |
|
|
| `sign` | bool | true | Sign the bundle with Authority key |
|
|
| `max_items` | int | 10000 | Maximum items per bundle (1-100000) |
|
|
| `compress_level` | int | 3 | ZST compression level (1-19) |
|
|
|
|
**Response Headers:**
|
|
|
|
| Header | Description |
|
|
|--------|-------------|
|
|
| `Content-Type` | `application/zstd` |
|
|
| `Content-Disposition` | `attachment; filename="feedser-bundle-{timestamp}.zst"` |
|
|
| `X-Bundle-Hash` | SHA256 hash of bundle content |
|
|
| `X-Export-Cursor` | Cursor position after this export |
|
|
| `X-Items-Count` | Total items in bundle |
|
|
|
|
**Response:** Streaming ZST-compressed TAR archive.
|
|
|
|
**Errors:**
|
|
|
|
| Status | Code | Description |
|
|
|--------|------|-------------|
|
|
| 400 | `VALIDATION_FAILED` | Invalid parameter values |
|
|
| 503 | `FEDERATION_DISABLED` | Federation is not enabled |
|
|
|
|
### Preview Export
|
|
|
|
```
|
|
GET /api/v1/federation/export/preview
|
|
```
|
|
|
|
Preview export statistics without creating a bundle.
|
|
|
|
**Query Parameters:**
|
|
|
|
| Parameter | Type | Default | Description |
|
|
|-----------|------|---------|-------------|
|
|
| `since_cursor` | string | null | Preview changes since this cursor |
|
|
|
|
**Response:**
|
|
|
|
```json
|
|
{
|
|
"since_cursor": "2025-01-14T00:00:00Z#0000",
|
|
"estimated_canonicals": 1234,
|
|
"estimated_edges": 3456,
|
|
"estimated_deletions": 12,
|
|
"estimated_size_bytes": 5242880,
|
|
"estimated_size_mb": 5.0
|
|
}
|
|
```
|
|
|
|
### Federation Status
|
|
|
|
```
|
|
GET /api/v1/federation/status
|
|
```
|
|
|
|
Get federation configuration status.
|
|
|
|
**Response:**
|
|
|
|
```json
|
|
{
|
|
"enabled": true,
|
|
"site_id": "site-us-west-1",
|
|
"default_compression_level": 3,
|
|
"default_max_items": 10000
|
|
}
|
|
```
|
|
|
|
## CLI Commands
|
|
|
|
### Export Bundle
|
|
|
|
```bash
|
|
stella feedser bundle export [options]
|
|
```
|
|
|
|
**Options:**
|
|
|
|
| Option | Short | Default | Description |
|
|
|--------|-------|---------|-------------|
|
|
| `--since-cursor` | `-c` | null | Export changes since cursor |
|
|
| `--output` | `-o` | stdout | Output file path |
|
|
| `--sign` | `-s` | true | Sign bundle with Authority key |
|
|
| `--compress-level` | `-l` | 3 | ZST compression level (1-19) |
|
|
| `--max-items` | `-m` | 10000 | Maximum items per bundle |
|
|
| `--json` | | false | Output metadata as JSON |
|
|
|
|
**Examples:**
|
|
|
|
```bash
|
|
# Full export to file
|
|
stella feedser bundle export -o ./bundle.zst
|
|
|
|
# Delta export since cursor
|
|
stella feedser bundle export -c "2025-01-14T00:00:00Z#0000" -o ./delta.zst
|
|
|
|
# Export without signing (for testing)
|
|
stella feedser bundle export --sign=false -o ./unsigned.zst
|
|
|
|
# High compression for archival
|
|
stella feedser bundle export -l 19 -o ./archived.zst
|
|
```
|
|
|
|
### Preview Export
|
|
|
|
```bash
|
|
stella feedser bundle preview [options]
|
|
```
|
|
|
|
**Options:**
|
|
|
|
| Option | Short | Description |
|
|
|--------|-------|-------------|
|
|
| `--since-cursor` | `-c` | Preview changes since cursor |
|
|
| `--json` | | Output as JSON |
|
|
|
|
**Example:**
|
|
|
|
```bash
|
|
stella feedser bundle preview -c "2025-01-14T00:00:00Z#0000"
|
|
```
|
|
|
|
## Configuration
|
|
|
|
Federation is configured in `concelier.yaml`:
|
|
|
|
```yaml
|
|
Federation:
|
|
Enabled: true
|
|
SiteId: "site-us-west-1"
|
|
DefaultCompressionLevel: 3
|
|
DefaultMaxItems: 10000
|
|
RequireSignature: true
|
|
```
|
|
|
|
| Setting | Type | Default | Description |
|
|
|---------|------|---------|-------------|
|
|
| `Enabled` | bool | false | Enable federation endpoints |
|
|
| `SiteId` | string | "default" | Identifier for this site |
|
|
| `DefaultCompressionLevel` | int | 3 | Default ZST compression level |
|
|
| `DefaultMaxItems` | int | 10000 | Default max items per bundle |
|
|
| `RequireSignature` | bool | true | Require bundle signatures |
|
|
|
|
## Cursor Format
|
|
|
|
Cursors encode a timestamp and sequence number:
|
|
|
|
```
|
|
{ISO8601}#{sequence}
|
|
```
|
|
|
|
Example: `2025-01-15T10:30:00.000Z#0042`
|
|
|
|
- Timestamp: When the change was recorded
|
|
- Sequence: Monotonically increasing within timestamp
|
|
|
|
Cursors are opaque to consumers and should be passed through unchanged.
|
|
|
|
## Determinism
|
|
|
|
Bundles are deterministic:
|
|
- Same cursor range produces identical bundle content
|
|
- Same content produces identical bundle hash
|
|
- Suitable for caching and deduplication
|
|
|
|
## Security
|
|
|
|
- Bundles can be signed with DSSE for integrity verification
|
|
- Signatures use Authority keys for cross-site trust
|
|
- Bundle hash prevents tampering during transit
|
|
- ZST compression is not encryption - bundles should be transferred over TLS
|