save development progress
This commit is contained in:
StellaOps Bot
278
docs/modules/concelier/federation-bundle-export.md
Normal file
278
docs/modules/concelier/federation-bundle-export.md
Normal file
@@ -0,0 +1,278 @@
|
||||
# Federation Bundle Export
|
||||
|
||||
Per SPRINT_8200_0014_0002.
|
||||
|
||||
## Overview
|
||||
|
||||
Federation bundles enable multi-site synchronization of canonical advisory data. Each bundle contains a delta of changes since a specified cursor position, allowing incremental sync between federated Concelier instances.
|
||||
|
||||
## Bundle Format
|
||||
|
||||
Bundles use a TAR archive compressed with ZStandard (ZST):
|
||||
|
||||
```
|
||||
feedser-bundle-v1.zst
|
||||
├── MANIFEST.json # Bundle metadata
|
||||
├── canonicals.ndjson # Canonical advisories (one per line)
|
||||
├── edges.ndjson # Source edges (one per line)
|
||||
├── deletions.ndjson # Withdrawn/deleted canonical IDs
|
||||
└── SIGNATURE.json # DSSE envelope (optional)
|
||||
```
|
||||
|
||||
### MANIFEST.json
|
||||
|
||||
```json
|
||||
{
|
||||
"version": "feedser-bundle/1.0",
|
||||
"site_id": "site-us-west-1",
|
||||
"export_cursor": "2025-01-15T10:30:00.000Z#0042",
|
||||
"since_cursor": "2025-01-14T00:00:00.000Z#0000",
|
||||
"exported_at": "2025-01-15T10:30:15.123Z",
|
||||
"counts": {
|
||||
"canonicals": 1234,
|
||||
"edges": 3456,
|
||||
"deletions": 12,
|
||||
"total": 4702
|
||||
},
|
||||
"bundle_hash": "sha256:a1b2c3d4..."
|
||||
}
|
||||
```
|
||||
|
||||
| Field | Type | Description |
|
||||
|-------|------|-------------|
|
||||
| `version` | string | Bundle format version identifier |
|
||||
| `site_id` | string | Identifier of the exporting site |
|
||||
| `export_cursor` | string | Cursor position after this export |
|
||||
| `since_cursor` | string? | Cursor position from which changes were exported (null for full export) |
|
||||
| `exported_at` | ISO8601 | Timestamp when bundle was created |
|
||||
| `counts` | object | Item counts by type |
|
||||
| `bundle_hash` | string | SHA256 hash of compressed bundle content |
|
||||
|
||||
### canonicals.ndjson
|
||||
|
||||
Each line contains a canonical advisory record:
|
||||
|
||||
```json
|
||||
{"id":"uuid","cve":"CVE-2024-1234","affects_key":"pkg:npm/express@4.0.0","merge_hash":"a1b2c3...","status":"active","severity":"high","title":"..."}
|
||||
```
|
||||
|
||||
### edges.ndjson
|
||||
|
||||
Each line contains a source edge linking a canonical to its source advisory:
|
||||
|
||||
```json
|
||||
{"id":"uuid","canonical_id":"uuid","source":"nvd","source_advisory_id":"CVE-2024-1234","vendor_status":"affected"}
|
||||
```
|
||||
|
||||
### deletions.ndjson
|
||||
|
||||
Each line contains a deletion record for withdrawn or deleted canonicals:
|
||||
|
||||
```json
|
||||
{"canonical_id":"uuid","deleted_at":"2025-01-15T10:00:00Z","reason":"withdrawn"}
|
||||
```
|
||||
|
||||
### SIGNATURE.json
|
||||
|
||||
When signing is enabled, contains a DSSE envelope over the bundle hash:
|
||||
|
||||
```json
|
||||
{
|
||||
"payload_type": "application/vnd.stellaops.bundle-hash+json",
|
||||
"payload": "eyJidW5kbGVfaGFzaCI6InNoYTI1NjphMWIy..."}",
|
||||
"signatures": [
|
||||
{
|
||||
"keyid": "sha256:xyz...",
|
||||
"sig": "MEUCIQD..."
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## API Endpoints
|
||||
|
||||
### Export Bundle
|
||||
|
||||
```
|
||||
GET /api/v1/federation/export
|
||||
```
|
||||
|
||||
Exports a delta bundle for federation sync.
|
||||
|
||||
**Query Parameters:**
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
|-----------|------|---------|-------------|
|
||||
| `since_cursor` | string | null | Export changes since this cursor (null = full export) |
|
||||
| `sign` | bool | true | Sign the bundle with Authority key |
|
||||
| `max_items` | int | 10000 | Maximum items per bundle (1-100000) |
|
||||
| `compress_level` | int | 3 | ZST compression level (1-19) |
|
||||
|
||||
**Response Headers:**
|
||||
|
||||
| Header | Description |
|
||||
|--------|-------------|
|
||||
| `Content-Type` | `application/zstd` |
|
||||
| `Content-Disposition` | `attachment; filename="feedser-bundle-{timestamp}.zst"` |
|
||||
| `X-Bundle-Hash` | SHA256 hash of bundle content |
|
||||
| `X-Export-Cursor` | Cursor position after this export |
|
||||
| `X-Items-Count` | Total items in bundle |
|
||||
|
||||
**Response:** Streaming ZST-compressed TAR archive.
|
||||
|
||||
**Errors:**
|
||||
|
||||
| Status | Code | Description |
|
||||
|--------|------|-------------|
|
||||
| 400 | `VALIDATION_FAILED` | Invalid parameter values |
|
||||
| 503 | `FEDERATION_DISABLED` | Federation is not enabled |
|
||||
|
||||
### Preview Export
|
||||
|
||||
```
|
||||
GET /api/v1/federation/export/preview
|
||||
```
|
||||
|
||||
Preview export statistics without creating a bundle.
|
||||
|
||||
**Query Parameters:**
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
|-----------|------|---------|-------------|
|
||||
| `since_cursor` | string | null | Preview changes since this cursor |
|
||||
|
||||
**Response:**
|
||||
|
||||
```json
|
||||
{
|
||||
"since_cursor": "2025-01-14T00:00:00Z#0000",
|
||||
"estimated_canonicals": 1234,
|
||||
"estimated_edges": 3456,
|
||||
"estimated_deletions": 12,
|
||||
"estimated_size_bytes": 5242880,
|
||||
"estimated_size_mb": 5.0
|
||||
}
|
||||
```
|
||||
|
||||
### Federation Status
|
||||
|
||||
```
|
||||
GET /api/v1/federation/status
|
||||
```
|
||||
|
||||
Get federation configuration status.
|
||||
|
||||
**Response:**
|
||||
|
||||
```json
|
||||
{
|
||||
"enabled": true,
|
||||
"site_id": "site-us-west-1",
|
||||
"default_compression_level": 3,
|
||||
"default_max_items": 10000
|
||||
}
|
||||
```
|
||||
|
||||
## CLI Commands
|
||||
|
||||
### Export Bundle
|
||||
|
||||
```bash
|
||||
stella feedser bundle export [options]
|
||||
```
|
||||
|
||||
**Options:**
|
||||
|
||||
| Option | Short | Default | Description |
|
||||
|--------|-------|---------|-------------|
|
||||
| `--since-cursor` | `-c` | null | Export changes since cursor |
|
||||
| `--output` | `-o` | stdout | Output file path |
|
||||
| `--sign` | `-s` | true | Sign bundle with Authority key |
|
||||
| `--compress-level` | `-l` | 3 | ZST compression level (1-19) |
|
||||
| `--max-items` | `-m` | 10000 | Maximum items per bundle |
|
||||
| `--json` | | false | Output metadata as JSON |
|
||||
|
||||
**Examples:**
|
||||
|
||||
```bash
|
||||
# Full export to file
|
||||
stella feedser bundle export -o ./bundle.zst
|
||||
|
||||
# Delta export since cursor
|
||||
stella feedser bundle export -c "2025-01-14T00:00:00Z#0000" -o ./delta.zst
|
||||
|
||||
# Export without signing (for testing)
|
||||
stella feedser bundle export --sign=false -o ./unsigned.zst
|
||||
|
||||
# High compression for archival
|
||||
stella feedser bundle export -l 19 -o ./archived.zst
|
||||
```
|
||||
|
||||
### Preview Export
|
||||
|
||||
```bash
|
||||
stella feedser bundle preview [options]
|
||||
```
|
||||
|
||||
**Options:**
|
||||
|
||||
| Option | Short | Description |
|
||||
|--------|-------|-------------|
|
||||
| `--since-cursor` | `-c` | Preview changes since cursor |
|
||||
| `--json` | | Output as JSON |
|
||||
|
||||
**Example:**
|
||||
|
||||
```bash
|
||||
stella feedser bundle preview -c "2025-01-14T00:00:00Z#0000"
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
Federation is configured in `concelier.yaml`:
|
||||
|
||||
```yaml
|
||||
Federation:
|
||||
Enabled: true
|
||||
SiteId: "site-us-west-1"
|
||||
DefaultCompressionLevel: 3
|
||||
DefaultMaxItems: 10000
|
||||
RequireSignature: true
|
||||
```
|
||||
|
||||
| Setting | Type | Default | Description |
|
||||
|---------|------|---------|-------------|
|
||||
| `Enabled` | bool | false | Enable federation endpoints |
|
||||
| `SiteId` | string | "default" | Identifier for this site |
|
||||
| `DefaultCompressionLevel` | int | 3 | Default ZST compression level |
|
||||
| `DefaultMaxItems` | int | 10000 | Default max items per bundle |
|
||||
| `RequireSignature` | bool | true | Require bundle signatures |
|
||||
|
||||
## Cursor Format
|
||||
|
||||
Cursors encode a timestamp and sequence number:
|
||||
|
||||
```
|
||||
{ISO8601}#{sequence}
|
||||
```
|
||||
|
||||
Example: `2025-01-15T10:30:00.000Z#0042`
|
||||
|
||||
- Timestamp: When the change was recorded
|
||||
- Sequence: Monotonically increasing within timestamp
|
||||
|
||||
Cursors are opaque to consumers and should be passed through unchanged.
|
||||
|
||||
## Determinism
|
||||
|
||||
Bundles are deterministic:
|
||||
- Same cursor range produces identical bundle content
|
||||
- Same content produces identical bundle hash
|
||||
- Suitable for caching and deduplication
|
||||
|
||||
## Security
|
||||
|
||||
- Bundles can be signed with DSSE for integrity verification
|
||||
- Signatures use Authority keys for cross-site trust
|
||||
- Bundle hash prevents tampering during transit
|
||||
- ZST compression is not encryption - bundles should be transferred over TLS
|
||||
Reference in New Issue
Block a user