458 lines
16 KiB
Markdown
458 lines
16 KiB
Markdown
# Offline Sync Flow
|
|
|
|
## Overview
|
|
|
|
The Offline Sync Flow describes how StellaOps supports air-gapped and disconnected environments through the Offline Kit. This flow covers advisory bundle generation, secure transfer, verification, and import in environments with no external network connectivity.
|
|
|
|
**Business Value**: Enable full vulnerability scanning and policy evaluation capabilities in highly secure, air-gapped environments while maintaining audit trails and cryptographic verification.
|
|
|
|
## Actors
|
|
|
|
| Actor | Type | Role |
|
|
|-------|------|------|
|
|
| Online Admin | Human | Generates and exports offline bundles |
|
|
| Offline Admin | Human | Imports and verifies bundles |
|
|
| Mirror | Service | Creates advisory snapshots |
|
|
| EvidenceLocker | Service | Seals bundles for transfer |
|
|
| AirGap Importer | Service | Validates and imports bundles |
|
|
| Signer | Service | Signs bundle manifests |
|
|
|
|
## Prerequisites
|
|
|
|
### Online Environment
|
|
- Access to vulnerability feeds (NVD, GHSA, etc.)
|
|
- Signing keys configured
|
|
- Bundle generation scheduled
|
|
|
|
### Offline Environment
|
|
- AirGap services deployed
|
|
- Trust anchors configured (public keys)
|
|
- Secure transfer mechanism available
|
|
|
|
## Bundle Types
|
|
|
|
| Bundle Type | Contents | Frequency |
|
|
|-------------|----------|-----------|
|
|
| Advisory Bundle | CVE data, CVSS scores, affected versions | Daily/Weekly |
|
|
| VEX Bundle | VEX statements from trusted issuers | Daily |
|
|
| Policy Bundle | Policy sets, rules, exceptions | On-demand |
|
|
| Trust Bundle | Signing keys, certificates, CRLs | Monthly |
|
|
| Time Anchor | Roughtime proofs, NTP alternatives | Daily |
|
|
|
|
## Flow Diagram
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────────────────────┐
|
|
│ Offline Sync Flow │
|
|
└─────────────────────────────────────────────────────────────────────────────────┘
|
|
|
|
ONLINE ENVIRONMENT OFFLINE ENVIRONMENT
|
|
┌────────────────────────────────┐ ┌────────────────────────────────┐
|
|
│ │ │ │
|
|
│ ┌────────┐ ┌────────┐ ┌────┐│ │┌────┐ ┌─────────┐ ┌────────┐│
|
|
│ │ Mirror │ │Evidence│ │Sign││ ││Verif│ │ AirGap │ │Concelier││
|
|
│ │ │ │ Locker │ │ ││ ││ │ │Importer │ │ ││
|
|
│ └───┬────┘ └───┬────┘ └──┬─┘│ │└──┬──┘ └────┬────┘ └───┬────┘│
|
|
│ │ │ │ │ │ │ │ │ │
|
|
│ │ Snapshot │ │ │ │ │ │ │ │
|
|
│ │ advisories│ │ │ │ │ │ │ │
|
|
│ │───┐ │ │ │ │ │ │ │ │
|
|
│ │ │ │ │ │ │ │ │ │ │
|
|
│ │<──┘ │ │ │ │ │ │ │ │
|
|
│ │ │ │ │ │ │ │ │ │
|
|
│ │ Create │ │ │ │ │ │ │ │
|
|
│ │ bundle │ │ │ │ │ │ │ │
|
|
│ │──────────>│ │ │ │ │ │ │ │
|
|
│ │ │ │ │ │ │ │ │ │
|
|
│ │ │ Seal │ │ │ │ │ │ │
|
|
│ │ │──────────> │ │ │ │ │ │
|
|
│ │ │ │ │ │ │ │ │ │
|
|
│ │ │ Signed │ │ │ │ │ │ │
|
|
│ │ │ bundle │ │ │ │ │ │ │
|
|
│ │ │<────────── │ │ │ │ │ │
|
|
│ │ │ │ │ │ │ │ │ │
|
|
│ │ │ │ │ │ │ │ │ │
|
|
│ │ [Export to ] │ │ │ │ │ │
|
|
│ │ [removable media ] │ =========> [Import from│ │ │
|
|
│ │ │ │ │ │ │removable]│ │ │
|
|
│ │ │ │ │ │ │ │ │ │
|
|
│ │ │ │ │ │ │ Verify │ │ │
|
|
│ │ │ │ │ │ │ signature│ │ │
|
|
│ │ │ │ │ │ │───┐ │ │ │
|
|
│ │ │ │ │ │ │ │ │ │ │
|
|
│ │ │ │ │ │ │<──┘ │ │ │
|
|
│ │ │ │ │ │ │ │ │ │
|
|
│ │ │ │ │ │ │ Verify │ │ │
|
|
│ │ │ │ │ │ │ Merkle │ │ │
|
|
│ │ │ │ │ │ │───┐ │ │ │
|
|
│ │ │ │ │ │ │ │ │ │ │
|
|
│ │ │ │ │ │ │<──┘ │ │ │
|
|
│ │ │ │ │ │ │ │ │ │
|
|
│ │ │ │ │ │ │ Import │ │ │
|
|
│ │ │ │ │ │ │──────────> │ │
|
|
│ │ │ │ │ │ │ │ │ │
|
|
│ │ │ │ │ │ │ │ Unpack │ │
|
|
│ │ │ │ │ │ │ │ advisories│ │
|
|
│ │ │ │ │ │ │ │───┐ │ │
|
|
│ │ │ │ │ │ │ │ │ │ │
|
|
│ │ │ │ │ │ │ │<──┘ │ │
|
|
│ │ │ │ │ │ │ │ │ │
|
|
│ │ │ │ │ │ │ │ Merge to │ │
|
|
│ │ │ │ │ │ │ │ Concelier │ │
|
|
│ │ │ │ │ │ │ │──────────>│ │
|
|
│ │ │ │ │ │ │ │ │ │
|
|
│ │ │ │ │ │ │ │ │ │
|
|
└────────────────────────────────┘ └────────────────────────────────┘
|
|
```
|
|
|
|
## Step-by-Step
|
|
|
|
### 1. Advisory Snapshot (Online)
|
|
|
|
Mirror service creates point-in-time snapshot:
|
|
|
|
```json
|
|
{
|
|
"snapshot_id": "snap-20241229",
|
|
"created_at": "2024-12-29T00:00:00Z",
|
|
"sources": [
|
|
{"name": "nvd", "last_sync": "2024-12-29T00:00:00Z", "count": 245678},
|
|
{"name": "ghsa", "last_sync": "2024-12-28T23:45:00Z", "count": 45678},
|
|
{"name": "osv", "last_sync": "2024-12-28T23:30:00Z", "count": 89012}
|
|
],
|
|
"delta_from": "snap-20241228",
|
|
"advisories": {
|
|
"new": 127,
|
|
"updated": 456,
|
|
"unchanged": 245095
|
|
}
|
|
}
|
|
```
|
|
|
|
### 2. Bundle Generation (Online)
|
|
|
|
EvidenceLocker creates sealed bundle:
|
|
|
|
```json
|
|
{
|
|
"bundle_id": "offline-adv-20241229",
|
|
"bundle_type": "advisory",
|
|
"created_at": "2024-12-29T01:00:00Z",
|
|
"contents": {
|
|
"advisories": {
|
|
"full_count": 245678,
|
|
"delta_count": 583,
|
|
"format": "ndjson.gz"
|
|
},
|
|
"metadata": {
|
|
"sources": ["nvd", "ghsa", "osv"],
|
|
"schema_version": "1.0.0"
|
|
}
|
|
},
|
|
"integrity": {
|
|
"merkle_root": "sha256:abc123...",
|
|
"file_count": 15,
|
|
"total_size": "125 MB"
|
|
}
|
|
}
|
|
```
|
|
|
|
### 3. Bundle Signing (Online)
|
|
|
|
Signer creates detached signature:
|
|
|
|
```json
|
|
{
|
|
"bundle_id": "offline-adv-20241229",
|
|
"signature": {
|
|
"algorithm": "ecdsa-p256",
|
|
"keyid": "sha256:offline-signing-key",
|
|
"sig": "base64:signature...",
|
|
"timestamp": "2024-12-29T01:00:00Z"
|
|
},
|
|
"certificate_chain": [
|
|
"base64:signing-cert...",
|
|
"base64:intermediate-ca...",
|
|
"base64:root-ca..."
|
|
],
|
|
"timestamping": {
|
|
"tsa_url": "https://timestamp.stellaops.io",
|
|
"timestamp_token": "base64:tst..."
|
|
}
|
|
}
|
|
```
|
|
|
|
### 4. Export Package
|
|
|
|
Final export package structure:
|
|
|
|
```
|
|
offline-kit-20241229/
|
|
├── manifest.json # Bundle manifest
|
|
├── manifest.sig # Detached signature
|
|
├── advisories/
|
|
│ ├── nvd-full.ndjson.gz
|
|
│ ├── nvd-delta.ndjson.gz
|
|
│ ├── ghsa-full.ndjson.gz
|
|
│ └── osv-full.ndjson.gz
|
|
├── vex/
|
|
│ └── vex-statements.ndjson.gz
|
|
├── policies/
|
|
│ └── policy-sets.json
|
|
├── trust/
|
|
│ ├── root-ca.pem
|
|
│ └── signing-keys.json
|
|
├── merkle/
|
|
│ └── tree.json
|
|
├── verify.sh # Verification script
|
|
└── README.md
|
|
```
|
|
|
|
### 5. Secure Transfer
|
|
|
|
Transfer via approved mechanism:
|
|
- USB drive (encrypted)
|
|
- Optical media (write-once)
|
|
- Data diode (one-way network)
|
|
- Secure courier
|
|
|
|
### 6. Import Verification (Offline)
|
|
|
|
AirGap Importer verifies bundle:
|
|
|
|
```bash
|
|
# Run verification
|
|
stellaops-airgap verify /media/usb/offline-kit-20241229/
|
|
|
|
# Verification steps:
|
|
✓ Manifest signature valid
|
|
✓ Certificate chain verified (trust anchor: sha256:root-ca)
|
|
✓ Timestamp verified (within 7 day window)
|
|
✓ Merkle root matches: sha256:abc123...
|
|
✓ All 15 files verified against Merkle tree
|
|
✓ No tamper detected
|
|
|
|
Bundle verified successfully. Ready for import.
|
|
```
|
|
|
|
### 7. Time Anchor Verification (Offline)
|
|
|
|
Verify bundle freshness without network time:
|
|
|
|
```json
|
|
{
|
|
"time_verification": {
|
|
"bundle_timestamp": "2024-12-29T01:00:00Z",
|
|
"tsa_timestamp": "2024-12-29T01:00:05Z",
|
|
"local_time_anchor": "2024-12-29T10:00:00Z",
|
|
"max_age_policy": "7d",
|
|
"age": "9h",
|
|
"status": "FRESH"
|
|
}
|
|
}
|
|
```
|
|
|
|
### 8. Advisory Import (Offline)
|
|
|
|
AirGap Importer loads advisories into Concelier:
|
|
|
|
```json
|
|
{
|
|
"import_id": "import-20241229-001",
|
|
"bundle_id": "offline-adv-20241229",
|
|
"started_at": "2024-12-29T10:30:00Z",
|
|
"completed_at": "2024-12-29T10:35:00Z",
|
|
"results": {
|
|
"advisories_imported": 583,
|
|
"advisories_updated": 456,
|
|
"advisories_new": 127,
|
|
"conflicts_resolved": 0,
|
|
"errors": 0
|
|
},
|
|
"state": {
|
|
"previous_snapshot": "snap-20241228",
|
|
"current_snapshot": "snap-20241229"
|
|
}
|
|
}
|
|
```
|
|
|
|
## Bundle Freshness Policies
|
|
|
|
### Strict (High Security)
|
|
|
|
```yaml
|
|
freshness_policy:
|
|
mode: strict
|
|
max_age:
|
|
advisory: 24h
|
|
vex: 24h
|
|
policy: 7d
|
|
trust: 30d
|
|
require_tsa: true
|
|
reject_stale: true
|
|
```
|
|
|
|
### Standard
|
|
|
|
```yaml
|
|
freshness_policy:
|
|
mode: standard
|
|
max_age:
|
|
advisory: 7d
|
|
vex: 7d
|
|
policy: 30d
|
|
trust: 90d
|
|
require_tsa: false
|
|
warn_stale: true
|
|
```
|
|
|
|
### Permissive (Disconnected Operations)
|
|
|
|
```yaml
|
|
freshness_policy:
|
|
mode: permissive
|
|
max_age:
|
|
advisory: 30d
|
|
vex: 30d
|
|
policy: 90d
|
|
trust: 365d
|
|
require_tsa: false
|
|
warn_stale: true
|
|
allow_manual_override: true
|
|
```
|
|
|
|
## Data Contracts
|
|
|
|
### Bundle Manifest Schema
|
|
|
|
```typescript
|
|
interface OfflineBundle {
|
|
bundle_id: string;
|
|
bundle_type: 'advisory' | 'vex' | 'policy' | 'trust' | 'time_anchor';
|
|
version: string;
|
|
created_at: string;
|
|
created_by: string;
|
|
contents: {
|
|
files: Array<{
|
|
path: string;
|
|
size: number;
|
|
sha256: string;
|
|
}>;
|
|
metadata: Record<string, unknown>;
|
|
};
|
|
integrity: {
|
|
merkle_root: string;
|
|
algorithm: 'sha256';
|
|
tree_path: string;
|
|
};
|
|
signature: {
|
|
keyid: string;
|
|
algorithm: string;
|
|
sig: string;
|
|
};
|
|
freshness: {
|
|
timestamp: string;
|
|
tsa_timestamp?: string;
|
|
valid_until?: string;
|
|
};
|
|
}
|
|
```
|
|
|
|
### Import Result Schema
|
|
|
|
```typescript
|
|
interface ImportResult {
|
|
import_id: string;
|
|
bundle_id: string;
|
|
status: 'success' | 'partial' | 'failed';
|
|
started_at: string;
|
|
completed_at: string;
|
|
results: {
|
|
records_imported: number;
|
|
records_updated: number;
|
|
records_new: number;
|
|
conflicts: number;
|
|
errors: number;
|
|
};
|
|
verification: {
|
|
signature_valid: boolean;
|
|
merkle_verified: boolean;
|
|
freshness_check: 'FRESH' | 'STALE' | 'EXPIRED';
|
|
};
|
|
audit_log_entry: string;
|
|
}
|
|
```
|
|
|
|
## Scheduling Strategies
|
|
|
|
### Daily Sync
|
|
|
|
```yaml
|
|
offline_sync:
|
|
advisory_bundle:
|
|
schedule: "0 1 * * *" # Daily at 1 AM
|
|
type: delta
|
|
retention: 7
|
|
vex_bundle:
|
|
schedule: "0 2 * * *" # Daily at 2 AM
|
|
type: delta
|
|
retention: 7
|
|
```
|
|
|
|
### Weekly Full + Daily Delta
|
|
|
|
```yaml
|
|
offline_sync:
|
|
advisory_bundle:
|
|
full:
|
|
schedule: "0 0 * * SUN" # Weekly full on Sunday
|
|
retention: 4
|
|
delta:
|
|
schedule: "0 1 * * MON-SAT" # Daily delta Mon-Sat
|
|
retention: 7
|
|
```
|
|
|
|
## Error Handling
|
|
|
|
| Error | Recovery |
|
|
|-------|----------|
|
|
| Signature invalid | Reject bundle, alert admin |
|
|
| Merkle verification failed | Reject bundle, request retransfer |
|
|
| Bundle too old | Warn user, require override |
|
|
| Import conflict | Log conflict, apply latest |
|
|
| Disk space insufficient | Cleanup old imports, retry |
|
|
|
|
## Observability
|
|
|
|
### Metrics (Online)
|
|
|
|
| Metric | Type | Labels |
|
|
|--------|------|--------|
|
|
| `offline_bundle_created_total` | Counter | `type` |
|
|
| `offline_bundle_size_bytes` | Histogram | `type` |
|
|
| `offline_bundle_advisory_count` | Gauge | `bundle_id` |
|
|
|
|
### Metrics (Offline)
|
|
|
|
| Metric | Type | Labels |
|
|
|--------|------|--------|
|
|
| `offline_import_total` | Counter | `status`, `type` |
|
|
| `offline_bundle_age_hours` | Gauge | `bundle_id` |
|
|
| `offline_advisory_freshness_hours` | Gauge | - |
|
|
|
|
### Key Log Events
|
|
|
|
| Event | Level | Fields |
|
|
|-------|-------|--------|
|
|
| `offline.bundle.created` | INFO | `bundle_id`, `type`, `size` |
|
|
| `offline.bundle.verified` | INFO | `bundle_id`, `verifier` |
|
|
| `offline.import.started` | INFO | `import_id`, `bundle_id` |
|
|
| `offline.import.complete` | INFO | `import_id`, `records` |
|
|
| `offline.freshness.warning` | WARN | `bundle_id`, `age` |
|
|
|
|
## Related Flows
|
|
|
|
- [Evidence Bundle Export Flow](13-evidence-bundle-export-flow.md) - Similar sealing mechanics
|
|
- [Advisory Drift Re-scan Flow](11-advisory-drift-rescan-flow.md) - Advisory consumption
|
|
- [Scan Submission Flow](02-scan-submission-flow.md) - Uses imported advisories
|