feat: add bulk triage view component and related stories
- Exported BulkTriageViewComponent and its related types from findings module. - Created a new accessibility test suite for score components using axe-core. - Introduced design tokens for score components to standardize styling. - Enhanced score breakdown popover for mobile responsiveness with drag handle. - Added date range selector functionality to score history chart component. - Implemented unit tests for date range selector in score history chart. - Created Storybook stories for bulk triage view and score history chart with date range selector.
This commit is contained in:
StellaOps Bot
386
docs/modules/concelier/federation-operations.md
Normal file
386
docs/modules/concelier/federation-operations.md
Normal file
@@ -0,0 +1,386 @@
|
||||
# Federation Setup and Operations
|
||||
|
||||
Per SPRINT_8200_0014_0003.
|
||||
|
||||
## Overview
|
||||
|
||||
Federation enables multi-site synchronization of canonical advisory data between Concelier instances. Sites can export bundles containing delta changes and import bundles from other sites to maintain synchronized vulnerability intelligence.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────────┐
|
||||
│ Federation Topology │
|
||||
├─────────────────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ ┌──────────────────┐ ┌──────────────────┐ │
|
||||
│ │ Site A (HQ) │ │ Site B (Branch) │ │
|
||||
│ │ │ │ │ │
|
||||
│ │ ┌────────────┐ │ Export │ ┌────────────┐ │ │
|
||||
│ │ │ Concelier │──┼──────────►│ │ Concelier │ │ │
|
||||
│ │ │ │ │ Bundle │ │ │ │ │
|
||||
│ │ └────────────┘ │ │ └────────────┘ │ │
|
||||
│ │ │ │ │ │ │ │
|
||||
│ │ ▼ │ │ ▼ │ │
|
||||
│ │ ┌────────────┐ │ │ ┌────────────┐ │ │
|
||||
│ │ │ PostgreSQL │ │ │ │ PostgreSQL │ │ │
|
||||
│ │ └────────────┘ │ │ └────────────┘ │ │
|
||||
│ └──────────────────┘ └──────────────────┘ │
|
||||
│ │
|
||||
│ ┌──────────────────┐ │
|
||||
│ │ Site C (Air-Gap)│ │
|
||||
│ │ │ │
|
||||
│ │ ┌────────────┐ │ USB/Secure │
|
||||
│ │ │ Concelier │◄─┼───Transfer │
|
||||
│ │ │ │ │ │
|
||||
│ │ └────────────┘ │ │
|
||||
│ └──────────────────┘ │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Setup
|
||||
|
||||
### 1. Enable Federation
|
||||
|
||||
Configure federation in `concelier.yaml`:
|
||||
|
||||
```yaml
|
||||
Federation:
|
||||
Enabled: true
|
||||
SiteId: "site-us-west-1" # Unique identifier for this site
|
||||
DefaultCompressionLevel: 3
|
||||
DefaultMaxItems: 10000
|
||||
RequireSignature: true
|
||||
|
||||
FederationImport:
|
||||
AllowedSites:
|
||||
- "site-us-east-1"
|
||||
- "site-eu-central-1"
|
||||
MaxBundleSizeBytes: 104857600 # 100 MB
|
||||
SkipSignatureOnTrustedSites: false
|
||||
```
|
||||
|
||||
### 2. Configure Site Policies
|
||||
|
||||
Create site policies for each trusted federation partner:
|
||||
|
||||
```bash
|
||||
# Add trusted site
|
||||
stella feedser sites add site-us-east-1 \
|
||||
--display-name "US East Production" \
|
||||
--enabled
|
||||
|
||||
# Configure policy
|
||||
stella feedser sites policy site-us-east-1 \
|
||||
--max-bundle-size 100MB \
|
||||
--allowed-sources nvd,ghsa,debian
|
||||
```
|
||||
|
||||
### 3. Generate Signing Keys
|
||||
|
||||
For signed bundles, configure Authority keys:
|
||||
|
||||
```bash
|
||||
# Generate federation signing key
|
||||
stella authority keys generate \
|
||||
--name federation-signer \
|
||||
--algorithm ES256 \
|
||||
--purpose federation
|
||||
|
||||
# Export public key for distribution
|
||||
stella authority keys export federation-signer --public
|
||||
```
|
||||
|
||||
## Import Operations
|
||||
|
||||
### API Import
|
||||
|
||||
```
|
||||
POST /api/v1/federation/import
|
||||
Content-Type: application/zstd
|
||||
```
|
||||
|
||||
**Query Parameters:**
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
|-----------|------|---------|-------------|
|
||||
| `dry_run` | bool | false | Validate without importing |
|
||||
| `skip_signature` | bool | false | Skip signature verification (requires trust) |
|
||||
| `on_conflict` | enum | prefer_remote | `prefer_remote`, `prefer_local`, `fail` |
|
||||
| `force` | bool | false | Import even if cursor is not after current |
|
||||
|
||||
**Response:**
|
||||
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"bundle_hash": "sha256:a1b2c3...",
|
||||
"imported_cursor": "2025-01-15T10:30:00.000Z#0042",
|
||||
"counts": {
|
||||
"canonical_created": 100,
|
||||
"canonical_updated": 25,
|
||||
"canonical_skipped": 10,
|
||||
"edges_added": 200,
|
||||
"deletions_processed": 5
|
||||
},
|
||||
"conflicts": [],
|
||||
"duration_ms": 1234
|
||||
}
|
||||
```
|
||||
|
||||
### CLI Import
|
||||
|
||||
```bash
|
||||
# Import from file
|
||||
stella feedser bundle import ./bundle.zst
|
||||
|
||||
# Import with dry run
|
||||
stella feedser bundle import ./bundle.zst --dry-run
|
||||
|
||||
# Import from stdin (for pipes)
|
||||
cat bundle.zst | stella feedser bundle import -
|
||||
|
||||
# Import without signature verification (testing only)
|
||||
stella feedser bundle import ./bundle.zst --skip-signature
|
||||
|
||||
# Force import (override cursor check)
|
||||
stella feedser bundle import ./bundle.zst --force
|
||||
```
|
||||
|
||||
### Conflict Resolution
|
||||
|
||||
When conflicts occur between local and remote values:
|
||||
|
||||
| Strategy | Behavior |
|
||||
|----------|----------|
|
||||
| `prefer_remote` | Remote value wins (default) |
|
||||
| `prefer_local` | Local value preserved |
|
||||
| `fail` | Abort import on first conflict |
|
||||
|
||||
Conflicts are logged with full details:
|
||||
|
||||
```json
|
||||
{
|
||||
"merge_hash": "sha256:abc...",
|
||||
"field": "severity",
|
||||
"local_value": "high",
|
||||
"remote_value": "critical",
|
||||
"resolution": "prefer_remote"
|
||||
}
|
||||
```
|
||||
|
||||
## Site Management
|
||||
|
||||
### List Sites
|
||||
|
||||
```bash
|
||||
stella feedser sites list
|
||||
```
|
||||
|
||||
Output:
|
||||
```
|
||||
SITE ID STATUS LAST SYNC CURSOR
|
||||
───────────────────────── ──────── ─────────────────── ──────────────────────────
|
||||
site-us-east-1 enabled 2025-01-15 10:30 2025-01-15T10:30:00Z#0042
|
||||
site-eu-central-1 enabled 2025-01-15 09:15 2025-01-15T09:15:00Z#0038
|
||||
site-asia-pacific-1 disabled never -
|
||||
```
|
||||
|
||||
### View Site History
|
||||
|
||||
```bash
|
||||
stella feedser sites history site-us-east-1 --limit 10
|
||||
```
|
||||
|
||||
### Update Site Policy
|
||||
|
||||
```bash
|
||||
stella feedser sites policy site-us-east-1 \
|
||||
--enabled false # Disable imports from this site
|
||||
```
|
||||
|
||||
## Air-Gap Operations
|
||||
|
||||
For sites without network connectivity:
|
||||
|
||||
### Export for Transfer
|
||||
|
||||
```bash
|
||||
# On connected site
|
||||
stella feedser bundle export \
|
||||
-c "2025-01-14T00:00:00Z#0000" \
|
||||
-o ./delta-2025-01-15.zst
|
||||
|
||||
# Transfer via USB/secure media
|
||||
```
|
||||
|
||||
### Import on Air-Gap Site
|
||||
|
||||
```bash
|
||||
# On air-gapped site
|
||||
stella feedser bundle import ./delta-2025-01-15.zst
|
||||
|
||||
# Verify import
|
||||
stella feedser sites list
|
||||
```
|
||||
|
||||
### Full Sync Workflow
|
||||
|
||||
1. **Initial Sync:**
|
||||
```bash
|
||||
# Export full dataset
|
||||
stella feedser bundle export -o ./full-sync.zst
|
||||
```
|
||||
|
||||
2. **Transfer to air-gap site**
|
||||
|
||||
3. **Import on air-gap:**
|
||||
```bash
|
||||
stella feedser bundle import ./full-sync.zst
|
||||
```
|
||||
|
||||
4. **Subsequent Delta Syncs:**
|
||||
```bash
|
||||
# Get current cursor from air-gap site
|
||||
stella feedser sites list # Note the cursor
|
||||
|
||||
# On connected site, export delta
|
||||
stella feedser bundle export -c "{cursor}" -o ./delta.zst
|
||||
|
||||
# Transfer and import on air-gap
|
||||
```
|
||||
|
||||
## Verification
|
||||
|
||||
### Validate Bundle Without Import
|
||||
|
||||
```bash
|
||||
stella feedser bundle validate ./bundle.zst
|
||||
```
|
||||
|
||||
Output:
|
||||
```
|
||||
Bundle: bundle.zst
|
||||
Version: feedser-bundle/1.0
|
||||
Site: site-us-east-1
|
||||
Cursor: 2025-01-15T10:30:00.000Z#0042
|
||||
|
||||
Counts:
|
||||
Canonicals: 1,234
|
||||
Edges: 3,456
|
||||
Deletions: 12
|
||||
Total: 4,702
|
||||
|
||||
Verification:
|
||||
Hash: ✓ Valid
|
||||
Signature: ✓ Valid (key: sha256:abc...)
|
||||
Format: ✓ Valid
|
||||
|
||||
Ready for import.
|
||||
```
|
||||
|
||||
### Preview Import Impact
|
||||
|
||||
```bash
|
||||
stella feedser bundle import ./bundle.zst --dry-run --json
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
### Sync Status Endpoint
|
||||
|
||||
```
|
||||
GET /api/v1/federation/sync/status
|
||||
```
|
||||
|
||||
Response:
|
||||
```json
|
||||
{
|
||||
"sites": [
|
||||
{
|
||||
"site_id": "site-us-east-1",
|
||||
"enabled": true,
|
||||
"last_sync_at": "2025-01-15T10:30:00Z",
|
||||
"last_cursor": "2025-01-15T10:30:00.000Z#0042",
|
||||
"bundles_imported": 156,
|
||||
"total_items_imported": 45678
|
||||
}
|
||||
],
|
||||
"local_cursor": "2025-01-15T10:35:00.000Z#0044"
|
||||
}
|
||||
```
|
||||
|
||||
### Event Stream
|
||||
|
||||
Import events are published to the `canonical-imported` stream:
|
||||
|
||||
```json
|
||||
{
|
||||
"canonical_id": "uuid",
|
||||
"cve": "CVE-2024-1234",
|
||||
"affects_key": "pkg:npm/express@4.0.0",
|
||||
"merge_hash": "sha256:...",
|
||||
"action": "Created",
|
||||
"bundle_hash": "sha256:...",
|
||||
"site_id": "site-us-east-1",
|
||||
"imported_at": "2025-01-15T10:30:15Z"
|
||||
}
|
||||
```
|
||||
|
||||
### Cache Invalidation
|
||||
|
||||
After import, cache indexes are automatically updated:
|
||||
- PURL index updated for affected packages
|
||||
- CVE index updated for vulnerability lookups
|
||||
- Existing cache entries invalidated for refresh
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
| Issue | Cause | Solution |
|
||||
|-------|-------|----------|
|
||||
| "Cursor not after current" | Bundle is stale | Use `--force` or export newer bundle |
|
||||
| "Signature verification failed" | Key mismatch | Verify signing key is trusted |
|
||||
| "Site not allowed" | Policy restriction | Add site to `AllowedSites` config |
|
||||
| "Bundle too large" | Size limit exceeded | Increase `MaxBundleSizeBytes` or export smaller delta |
|
||||
|
||||
### Debug Logging
|
||||
|
||||
Enable verbose logging for federation operations:
|
||||
|
||||
```yaml
|
||||
Logging:
|
||||
LogLevel:
|
||||
StellaOps.Concelier.Federation: Debug
|
||||
```
|
||||
|
||||
### Verify Sync State
|
||||
|
||||
```bash
|
||||
# Check local vs remote cursor
|
||||
stella feedser sites status site-us-east-1
|
||||
|
||||
# List recent imports
|
||||
stella feedser sites history site-us-east-1 --limit 5
|
||||
|
||||
# Verify specific canonical was imported
|
||||
stella feedser canonical get sha256:mergehash...
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Regular Sync Schedule:** Configure automated delta exports/imports on a schedule (e.g., hourly)
|
||||
|
||||
2. **Monitor Cursor Drift:** Alert if cursor falls too far behind
|
||||
|
||||
3. **Verify Signatures:** Only disable signature verification in development
|
||||
|
||||
4. **Size Bundles Appropriately:** For large deltas, split into multiple bundles
|
||||
|
||||
5. **Test Import Before Production:** Use `--dry-run` to validate bundles
|
||||
|
||||
6. **Maintain Key Trust:** Regularly rotate and verify federation signing keys
|
||||
|
||||
7. **Document Site Policies:** Keep a registry of trusted sites and their policies
|
||||
332
docs/modules/concelier/federation-setup.md
Normal file
332
docs/modules/concelier/federation-setup.md
Normal file
@@ -0,0 +1,332 @@
|
||||
# Federation Setup and Operations Guide
|
||||
|
||||
This guide covers the setup and operation of StellaOps federation for multi-site vulnerability data synchronization.
|
||||
|
||||
## Overview
|
||||
|
||||
Federation enables secure, cursor-based synchronization of canonical vulnerability advisories between StellaOps sites. It supports:
|
||||
|
||||
- **Delta exports**: Only changed records since the last cursor are included
|
||||
- **Air-gap transfers**: Bundles can be written to files for offline transfer
|
||||
- **Multi-site topology**: Multiple sites can synchronize independently
|
||||
- **Cryptographic verification**: DSSE signatures ensure bundle authenticity
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
|
||||
│ Site A │────▶│ Bundle │────▶│ Site B │
|
||||
│ (Export) │ │ (.zst) │ │ (Import) │
|
||||
└─────────────┘ └─────────────┘ └─────────────┘
|
||||
│
|
||||
▼
|
||||
┌───────────┐
|
||||
│ Site C │
|
||||
│ (Import) │
|
||||
└───────────┘
|
||||
```
|
||||
|
||||
## Bundle Format
|
||||
|
||||
Federation bundles are ZST-compressed TAR archives containing:
|
||||
|
||||
| File | Description |
|
||||
|------|-------------|
|
||||
| `MANIFEST.json` | Bundle metadata, cursor, counts, hash |
|
||||
| `canonicals.ndjson` | Canonical advisories (one per line) |
|
||||
| `edges.ndjson` | Source edges linking advisories to sources |
|
||||
| `deletions.ndjson` | Withdrawn/deleted advisory IDs |
|
||||
| `SIGNATURE.json` | Optional DSSE signature envelope |
|
||||
|
||||
## Configuration
|
||||
|
||||
### Export Site Configuration
|
||||
|
||||
```yaml
|
||||
# concelier.yaml
|
||||
federation:
|
||||
enabled: true
|
||||
site_id: "us-west-1" # Unique site identifier
|
||||
export:
|
||||
enabled: true
|
||||
default_compression_level: 3 # ZST level (1-19)
|
||||
sign_bundles: true # Sign exported bundles
|
||||
max_items_per_bundle: 10000 # Maximum items per export
|
||||
```
|
||||
|
||||
### Import Site Configuration
|
||||
|
||||
```yaml
|
||||
# concelier.yaml
|
||||
federation:
|
||||
enabled: true
|
||||
site_id: "eu-central-1"
|
||||
import:
|
||||
enabled: true
|
||||
skip_signature_verification: false # NEVER set true in production
|
||||
allowed_sites: # Trusted site IDs
|
||||
- "us-west-1"
|
||||
- "ap-south-1"
|
||||
conflict_resolution: "prefer_remote" # prefer_remote | prefer_local | fail
|
||||
force_cursor_validation: true # Reject out-of-order imports
|
||||
```
|
||||
|
||||
## API Endpoints
|
||||
|
||||
### Export Endpoints
|
||||
|
||||
```bash
|
||||
# Export delta bundle since cursor
|
||||
GET /api/v1/federation/export?since_cursor={cursor}
|
||||
|
||||
# Preview export (counts only)
|
||||
GET /api/v1/federation/export/preview?since_cursor={cursor}
|
||||
|
||||
# Get federation status
|
||||
GET /api/v1/federation/status
|
||||
```
|
||||
|
||||
### Import Endpoints
|
||||
|
||||
```bash
|
||||
# Import bundle
|
||||
POST /api/v1/federation/import
|
||||
Content-Type: application/zstd
|
||||
|
||||
# Validate bundle without importing
|
||||
POST /api/v1/federation/validate
|
||||
Content-Type: application/zstd
|
||||
|
||||
# List federated sites
|
||||
GET /api/v1/federation/sites
|
||||
|
||||
# Update site policy
|
||||
PUT /api/v1/federation/sites/{site_id}/policy
|
||||
```
|
||||
|
||||
## CLI Commands
|
||||
|
||||
### Export Operations
|
||||
|
||||
```bash
|
||||
# Export full bundle (no cursor = all data)
|
||||
feedser bundle export --output bundle.zst
|
||||
|
||||
# Export delta since last cursor
|
||||
feedser bundle export --since-cursor "2025-01-15T10:00:00Z#0001" --output delta.zst
|
||||
|
||||
# Preview export without creating bundle
|
||||
feedser bundle preview --since-cursor "2025-01-15T10:00:00Z#0001"
|
||||
|
||||
# Export without signing (testing only)
|
||||
feedser bundle export --no-sign --output unsigned.zst
|
||||
```
|
||||
|
||||
### Import Operations
|
||||
|
||||
```bash
|
||||
# Import bundle
|
||||
feedser bundle import bundle.zst
|
||||
|
||||
# Dry run (validate without importing)
|
||||
feedser bundle import bundle.zst --dry-run
|
||||
|
||||
# Import from stdin (pipe)
|
||||
cat bundle.zst | feedser bundle import -
|
||||
|
||||
# Force import (skip cursor validation)
|
||||
feedser bundle import bundle.zst --force
|
||||
```
|
||||
|
||||
### Site Management
|
||||
|
||||
```bash
|
||||
# List federated sites
|
||||
feedser sites list
|
||||
|
||||
# Show site details
|
||||
feedser sites show us-west-1
|
||||
|
||||
# Enable/disable site
|
||||
feedser sites enable ap-south-1
|
||||
feedser sites disable ap-south-1
|
||||
```
|
||||
|
||||
## Cursor Format
|
||||
|
||||
Cursors use ISO-8601 timestamp with sequence number:
|
||||
|
||||
```
|
||||
{ISO-8601 timestamp}#{sequence number}
|
||||
|
||||
Examples:
|
||||
2025-01-15T10:00:00.000Z#0001
|
||||
2025-01-15T10:00:00.000Z#0002
|
||||
```
|
||||
|
||||
- Cursors are site-specific (each site maintains independent cursors)
|
||||
- Sequence numbers distinguish concurrent exports
|
||||
- Cursors are monotonically increasing within a site
|
||||
|
||||
## Air-Gap Transfer Workflow
|
||||
|
||||
For environments without network connectivity:
|
||||
|
||||
```bash
|
||||
# On Source Site (connected to authority)
|
||||
feedser bundle export --since-cursor "$LAST_CURSOR" --output /media/usb/bundle.zst
|
||||
feedser bundle preview --since-cursor "$LAST_CURSOR" > /media/usb/manifest.txt
|
||||
|
||||
# Transfer media to target site...
|
||||
|
||||
# On Target Site (air-gapped)
|
||||
feedser bundle import /media/usb/bundle.zst --dry-run # Validate first
|
||||
feedser bundle import /media/usb/bundle.zst # Import
|
||||
```
|
||||
|
||||
## Multi-Site Synchronization
|
||||
|
||||
### Hub-and-Spoke Topology
|
||||
|
||||
```
|
||||
┌─────────────┐
|
||||
│ Hub Site │
|
||||
│ (Primary) │
|
||||
└──────┬──────┘
|
||||
│
|
||||
┌──────────┼──────────┐
|
||||
▼ ▼ ▼
|
||||
┌──────────┐ ┌──────────┐ ┌──────────┐
|
||||
│ Site A │ │ Site B │ │ Site C │
|
||||
│ (Spoke) │ │ (Spoke) │ │ (Spoke) │
|
||||
└──────────┘ └──────────┘ └──────────┘
|
||||
```
|
||||
|
||||
### Mesh Topology
|
||||
|
||||
Each site can import from multiple sources:
|
||||
|
||||
```yaml
|
||||
federation:
|
||||
import:
|
||||
allowed_sites:
|
||||
- "hub-primary"
|
||||
- "hub-secondary" # Redundancy
|
||||
```
|
||||
|
||||
## Merge Behavior
|
||||
|
||||
### Conflict Resolution
|
||||
|
||||
When importing, conflicts are resolved based on configuration:
|
||||
|
||||
| Strategy | Behavior |
|
||||
|----------|----------|
|
||||
| `prefer_remote` | Remote (bundle) value wins (default) |
|
||||
| `prefer_local` | Local value preserved |
|
||||
| `fail` | Import aborts on any conflict |
|
||||
|
||||
### Merge Actions
|
||||
|
||||
| Action | Description |
|
||||
|--------|-------------|
|
||||
| `Created` | New canonical added |
|
||||
| `Updated` | Existing canonical updated |
|
||||
| `Skipped` | No change needed (identical) |
|
||||
|
||||
## Verification
|
||||
|
||||
### Hash Verification
|
||||
|
||||
Bundle hash is computed over compressed content:
|
||||
|
||||
```
|
||||
SHA256(compressed bundle content)
|
||||
```
|
||||
|
||||
### Signature Verification
|
||||
|
||||
DSSE envelope contains:
|
||||
|
||||
```json
|
||||
{
|
||||
"payloadType": "application/stellaops.federation.bundle+json",
|
||||
"payload": "base64(bundle_hash + site_id + cursor)",
|
||||
"signatures": [
|
||||
{
|
||||
"keyId": "signing-key-001",
|
||||
"algorithm": "ES256",
|
||||
"signature": "base64(signature)"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
### Key Metrics
|
||||
|
||||
- `federation_export_duration_seconds` - Export time
|
||||
- `federation_import_duration_seconds` - Import time
|
||||
- `federation_bundle_size_bytes` - Bundle sizes
|
||||
- `federation_items_processed_total` - Items processed by type
|
||||
- `federation_conflicts_total` - Merge conflicts encountered
|
||||
|
||||
### Health Checks
|
||||
|
||||
```bash
|
||||
# Check federation status
|
||||
curl http://localhost:5000/api/v1/federation/status
|
||||
|
||||
# Response
|
||||
{
|
||||
"site_id": "us-west-1",
|
||||
"export_enabled": true,
|
||||
"import_enabled": true,
|
||||
"last_export": "2025-01-15T10:00:00Z",
|
||||
"last_import": "2025-01-15T09:30:00Z",
|
||||
"sites_synced": 2
|
||||
}
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
**Import fails with "cursor validation failed"**
|
||||
- Bundle cursor is not after current site cursor
|
||||
- Use `--force` to override (not recommended)
|
||||
- Check if bundle was already imported
|
||||
|
||||
**Signature verification failed**
|
||||
- Signing key not trusted on target site
|
||||
- Key expired or revoked
|
||||
- Use `--skip-signature` for testing only
|
||||
|
||||
**Large bundle timeout**
|
||||
- Increase `federation.export.timeout`
|
||||
- Use smaller `max_items_per_bundle`
|
||||
- Stream directly to file
|
||||
|
||||
### Debug Logging
|
||||
|
||||
```yaml
|
||||
logging:
|
||||
level:
|
||||
StellaOps.Concelier.Federation: Debug
|
||||
```
|
||||
|
||||
## Security Considerations
|
||||
|
||||
1. **Never skip signature verification in production**
|
||||
2. **Validate allowed_sites whitelist**
|
||||
3. **Use TLS for API endpoints**
|
||||
4. **Rotate signing keys periodically**
|
||||
5. **Audit import events**
|
||||
6. **Monitor for duplicate bundle imports**
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Bundle Export Format](federation-bundle-export.md)
|
||||
- [Sync Ledger Schema](../db/sync-ledger.md)
|
||||
- [Signing Configuration](../security/signing.md)
|
||||
247
docs/modules/concelier/interest-scoring.md
Normal file
247
docs/modules/concelier/interest-scoring.md
Normal file
@@ -0,0 +1,247 @@
|
||||
# Interest Scoring
|
||||
|
||||
Per SPRINT_8200_0013_0002.
|
||||
|
||||
## Overview
|
||||
|
||||
Interest scoring learns which advisories matter to your organization by analyzing SBOM intersections, reachability data, VEX status, and deployment signals. High-interest advisories are prioritized and cached, while low-interest ones are degraded to lightweight stubs to save resources.
|
||||
|
||||
## Score Components
|
||||
|
||||
The interest score is computed from weighted factors:
|
||||
|
||||
| Factor | Weight | Description |
|
||||
|--------|--------|-------------|
|
||||
| `in_sbom` | 0.4 | Advisory affects a component in registered SBOMs |
|
||||
| `reachable` | 0.25 | Affected code is reachable in call graph |
|
||||
| `deployed` | 0.15 | Component is deployed in production |
|
||||
| `no_vex_na` | 0.1 | No VEX "not_affected" statement exists |
|
||||
| `recent` | 0.1 | Advisory is recent (age decay applied) |
|
||||
|
||||
### Score Calculation
|
||||
|
||||
```
|
||||
score = (in_sbom × 0.4) + (reachable × 0.25) + (deployed × 0.15)
|
||||
+ (no_vex_na × 0.1) + (recent × 0.1)
|
||||
```
|
||||
|
||||
Each factor is binary (0.0 or 1.0), except `recent` which decays over time.
|
||||
|
||||
## Configuration
|
||||
|
||||
Configure in `concelier.yaml`:
|
||||
|
||||
```yaml
|
||||
InterestScoring:
|
||||
Enabled: true
|
||||
|
||||
# Factor weights (must sum to 1.0)
|
||||
Weights:
|
||||
InSbom: 0.4
|
||||
Reachable: 0.25
|
||||
Deployed: 0.15
|
||||
NoVexNa: 0.1
|
||||
Recent: 0.1
|
||||
|
||||
# Age decay for recent factor
|
||||
RecentDecay:
|
||||
HalfLifeDays: 90
|
||||
MinScore: 0.1
|
||||
|
||||
# Stub degradation policy
|
||||
Degradation:
|
||||
Enabled: true
|
||||
ThresholdScore: 0.1
|
||||
GracePeriodDays: 30
|
||||
RetainCve: true # Keep CVE ID even in stub form
|
||||
|
||||
# Recalculation job
|
||||
RecalculationJob:
|
||||
Enabled: true
|
||||
IncrementalIntervalMinutes: 15
|
||||
FullRecalculationHour: 3 # 3 AM daily
|
||||
```
|
||||
|
||||
## API Endpoints
|
||||
|
||||
### Get Score
|
||||
|
||||
```
|
||||
GET /api/v1/canonical/{id}/score
|
||||
```
|
||||
|
||||
Response:
|
||||
```json
|
||||
{
|
||||
"canonical_id": "uuid",
|
||||
"score": 0.85,
|
||||
"tier": "high",
|
||||
"reasons": [
|
||||
{ "factor": "in_sbom", "value": 1.0, "weight": 0.4, "contribution": 0.4 },
|
||||
{ "factor": "reachable", "value": 1.0, "weight": 0.25, "contribution": 0.25 },
|
||||
{ "factor": "deployed", "value": 1.0, "weight": 0.15, "contribution": 0.15 },
|
||||
{ "factor": "no_vex_na", "value": 0.0, "weight": 0.1, "contribution": 0.0 },
|
||||
{ "factor": "recent", "value": 0.5, "weight": 0.1, "contribution": 0.05 }
|
||||
],
|
||||
"computed_at": "2025-01-15T10:30:00Z",
|
||||
"last_seen_in_build": "uuid"
|
||||
}
|
||||
```
|
||||
|
||||
### Trigger Recalculation (Admin)
|
||||
|
||||
```
|
||||
POST /api/v1/scores/recalculate
|
||||
```
|
||||
|
||||
Request:
|
||||
```json
|
||||
{
|
||||
"mode": "incremental", // or "full"
|
||||
"canonical_ids": null // null = all, or specific IDs
|
||||
}
|
||||
```
|
||||
|
||||
Response:
|
||||
```json
|
||||
{
|
||||
"job_id": "uuid",
|
||||
"mode": "incremental",
|
||||
"canonicals_queued": 1234,
|
||||
"started_at": "2025-01-15T10:30:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
## Score Tiers
|
||||
|
||||
| Tier | Score Range | Cache TTL | Behavior |
|
||||
|------|-------------|-----------|----------|
|
||||
| High | >= 0.7 | 24 hours | Full advisory cached, prioritized in queries |
|
||||
| Medium | 0.3 - 0.7 | 4 hours | Full advisory cached |
|
||||
| Low | 0.1 - 0.3 | 1 hour | Full advisory, eligible for degradation |
|
||||
| Negligible | < 0.1 | none | Degraded to stub after grace period |
|
||||
|
||||
## Stub Degradation
|
||||
|
||||
Low-interest advisories are degraded to lightweight stubs to save storage:
|
||||
|
||||
### Full Advisory vs Stub
|
||||
|
||||
| Field | Full | Stub |
|
||||
|-------|------|------|
|
||||
| id | ✓ | ✓ |
|
||||
| cve | ✓ | ✓ |
|
||||
| affects_key | ✓ | ✓ |
|
||||
| merge_hash | ✓ | ✓ |
|
||||
| severity | ✓ | ✓ |
|
||||
| title | ✓ | ✗ |
|
||||
| summary | ✓ | ✗ |
|
||||
| references | ✓ | ✗ |
|
||||
| weaknesses | ✓ | ✗ |
|
||||
| source_edges | ✓ | ✗ |
|
||||
|
||||
### Restoration
|
||||
|
||||
Stubs are restored to full advisories when:
|
||||
- Interest score increases above threshold
|
||||
- Advisory is directly queried by ID
|
||||
- Advisory appears in scan results
|
||||
|
||||
```csharp
|
||||
// Automatic restoration
|
||||
await scoringService.RestoreFromStubAsync(canonicalId, ct);
|
||||
```
|
||||
|
||||
## Integration Points
|
||||
|
||||
### SBOM Registration
|
||||
|
||||
When an SBOM is registered, interest scores are updated:
|
||||
|
||||
```
|
||||
POST /api/v1/learn/sbom → Triggers score recalculation
|
||||
```
|
||||
|
||||
### Scan Events
|
||||
|
||||
Subscribe to `ScanCompleted` events:
|
||||
|
||||
```csharp
|
||||
// In event handler
|
||||
public async Task HandleScanCompletedAsync(ScanCompletedEvent evt)
|
||||
{
|
||||
await sbomService.LearnFromScanAsync(evt.SbomDigest, ct);
|
||||
// This triggers interest score updates for affected advisories
|
||||
}
|
||||
```
|
||||
|
||||
### VEX Updates
|
||||
|
||||
VEX imports trigger score recalculation:
|
||||
|
||||
```csharp
|
||||
// In VEX ingestion pipeline
|
||||
await interestService.RecalculateForCanonicalAsync(canonicalId, ct);
|
||||
```
|
||||
|
||||
## CLI Commands
|
||||
|
||||
```bash
|
||||
# View score for advisory
|
||||
stella scores get sha256:mergehash...
|
||||
|
||||
# Trigger recalculation
|
||||
stella scores recalculate --mode incremental
|
||||
|
||||
# Full recalculation
|
||||
stella scores recalculate --mode full
|
||||
|
||||
# List high-interest advisories
|
||||
stella scores list --tier high --limit 100
|
||||
|
||||
# Force restore from stub
|
||||
stella scores restore sha256:mergehash...
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
### Metrics
|
||||
|
||||
| Metric | Type | Description |
|
||||
|--------|------|-------------|
|
||||
| `interest_score_computed_total` | Counter | Scores computed |
|
||||
| `interest_score_distribution` | Histogram | Score distribution |
|
||||
| `interest_stubs_created_total` | Counter | Advisories degraded to stubs |
|
||||
| `interest_stubs_restored_total` | Counter | Stubs restored to full |
|
||||
| `interest_job_duration_seconds` | Histogram | Recalculation job duration |
|
||||
|
||||
### Alerts
|
||||
|
||||
```yaml
|
||||
# Example Prometheus alert
|
||||
- alert: InterestScoreJobStale
|
||||
expr: time() - interest_score_last_full_recalc > 172800
|
||||
for: 1h
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Interest score full recalculation hasn't run in 2 days"
|
||||
```
|
||||
|
||||
## Database Schema
|
||||
|
||||
```sql
|
||||
CREATE TABLE vuln.interest_score (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
canonical_id UUID NOT NULL REFERENCES vuln.advisory_canonical(id),
|
||||
score NUMERIC(3,2) NOT NULL CHECK (score >= 0 AND score <= 1),
|
||||
reasons JSONB NOT NULL DEFAULT '[]',
|
||||
last_seen_in_build UUID,
|
||||
computed_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
CONSTRAINT uq_interest_score_canonical UNIQUE (canonical_id)
|
||||
);
|
||||
|
||||
CREATE INDEX idx_interest_score_score ON vuln.interest_score(score DESC);
|
||||
CREATE INDEX idx_interest_score_high ON vuln.interest_score(canonical_id)
|
||||
WHERE score >= 0.7;
|
||||
```
|
||||
315
docs/modules/concelier/operations/valkey-advisory-cache.md
Normal file
315
docs/modules/concelier/operations/valkey-advisory-cache.md
Normal file
@@ -0,0 +1,315 @@
|
||||
# Valkey Advisory Cache
|
||||
|
||||
Per SPRINT_8200_0013_0001.
|
||||
|
||||
## Overview
|
||||
|
||||
The Valkey Advisory Cache provides sub-20ms read latency for canonical advisory lookups by caching advisory data and maintaining fast-path indexes. The cache integrates with the Concelier CanonicalAdvisoryService as a read-through cache with automatic population and invalidation.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────────┐
|
||||
│ Advisory Cache Flow │
|
||||
├─────────────────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ ┌───────────┐ miss ┌───────────┐ fetch ┌───────────────┐ │
|
||||
│ │ Client │ ─────────► │ Valkey │ ──────────► │ PostgreSQL │ │
|
||||
│ │ Request │ │ Cache │ │ Canonical │ │
|
||||
│ └───────────┘ └───────────┘ │ Store │ │
|
||||
│ ▲ │ └───────────────┘ │
|
||||
│ │ │ hit │ │
|
||||
│ │ ▼ │ │
|
||||
│ └──────────────── (< 20ms) ◄────────────────────────┘ │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
Configure in `concelier.yaml`:
|
||||
|
||||
```yaml
|
||||
ConcelierCache:
|
||||
# Valkey connection settings
|
||||
ConnectionString: "localhost:6379,abortConnect=false,connectTimeout=5000"
|
||||
Database: 0
|
||||
InstanceName: "concelier:"
|
||||
|
||||
# TTL settings by interest score tier
|
||||
TtlPolicy:
|
||||
HighInterestTtlHours: 24 # Interest score >= 0.7
|
||||
MediumInterestTtlHours: 4 # Interest score 0.3 - 0.7
|
||||
LowInterestTtlHours: 1 # Interest score < 0.3
|
||||
DefaultTtlHours: 2
|
||||
|
||||
# Index settings
|
||||
HotSetMaxSize: 10000 # Max entries in rank:hot
|
||||
EnablePurlIndex: true
|
||||
EnableCveIndex: true
|
||||
|
||||
# Connection pool settings
|
||||
PoolSize: 20
|
||||
ConnectRetryCount: 3
|
||||
ReconnectRetryDelayMs: 1000
|
||||
|
||||
# Fallback behavior when Valkey unavailable
|
||||
FallbackToPostgres: true
|
||||
HealthCheckIntervalSeconds: 30
|
||||
```
|
||||
|
||||
## Key Schema
|
||||
|
||||
### Advisory Entry
|
||||
|
||||
**Key:** `advisory:{merge_hash}`
|
||||
|
||||
**Value:** JSON-serialized `CanonicalAdvisory`
|
||||
|
||||
**TTL:** Based on interest score tier
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "uuid",
|
||||
"cve": "CVE-2024-1234",
|
||||
"affects_key": "pkg:npm/express@4.0.0",
|
||||
"merge_hash": "sha256:a1b2c3...",
|
||||
"severity": "high",
|
||||
"interest_score": 0.85,
|
||||
"title": "...",
|
||||
"updated_at": "2025-01-15T10:30:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
### Hot Set (Sorted Set)
|
||||
|
||||
**Key:** `rank:hot`
|
||||
|
||||
**Score:** Interest score (0.0 - 1.0)
|
||||
|
||||
**Member:** merge_hash
|
||||
|
||||
Stores top advisories by interest score for quick access.
|
||||
|
||||
### PURL Index (Set)
|
||||
|
||||
**Key:** `by:purl:{normalized_purl}`
|
||||
|
||||
**Members:** Set of merge_hash values
|
||||
|
||||
Maps package URLs to affected advisories.
|
||||
|
||||
Example: `by:purl:pkg:npm/express@4.0.0` → `{sha256:a1b2c3..., sha256:d4e5f6...}`
|
||||
|
||||
### CVE Index (Set)
|
||||
|
||||
**Key:** `by:cve:{cve_id}`
|
||||
|
||||
**Members:** Set of merge_hash values
|
||||
|
||||
Maps CVE IDs to canonical advisories.
|
||||
|
||||
Example: `by:cve:cve-2024-1234` → `{sha256:a1b2c3...}`
|
||||
|
||||
## Operations
|
||||
|
||||
### Get Advisory
|
||||
|
||||
```csharp
|
||||
// Service interface
|
||||
public interface IAdvisoryCacheService
|
||||
{
|
||||
Task<CanonicalAdvisory?> GetAsync(string mergeHash, CancellationToken ct = default);
|
||||
Task SetAsync(CanonicalAdvisory advisory, CancellationToken ct = default);
|
||||
Task InvalidateAsync(string mergeHash, CancellationToken ct = default);
|
||||
Task<IReadOnlyList<CanonicalAdvisory>> GetByPurlAsync(string purl, CancellationToken ct = default);
|
||||
Task<IReadOnlyList<CanonicalAdvisory>> GetHotAsync(int count = 100, CancellationToken ct = default);
|
||||
Task IndexPurlAsync(string purl, string mergeHash, CancellationToken ct = default);
|
||||
Task IndexCveAsync(string cve, string mergeHash, CancellationToken ct = default);
|
||||
}
|
||||
```
|
||||
|
||||
### Read-Through Cache
|
||||
|
||||
```
|
||||
1. GetAsync(mergeHash) called
|
||||
2. Check Valkey: GET advisory:{mergeHash}
|
||||
└─ Hit: deserialize and return
|
||||
└─ Miss: fetch from PostgreSQL, cache result, return
|
||||
```
|
||||
|
||||
### Cache Population
|
||||
|
||||
Advisories are cached when:
|
||||
- First read (read-through)
|
||||
- Ingested from source connectors
|
||||
- Imported from federation bundles
|
||||
- Updated by merge operations
|
||||
|
||||
### Cache Invalidation
|
||||
|
||||
Invalidation occurs when:
|
||||
- Advisory is updated (re-merge with new data)
|
||||
- Advisory is withdrawn
|
||||
- Manual cache flush requested
|
||||
|
||||
```csharp
|
||||
// Invalidate single advisory
|
||||
await cacheService.InvalidateAsync(mergeHash, ct);
|
||||
|
||||
// Invalidate multiple (e.g., after bulk import)
|
||||
await cacheService.InvalidateManyAsync(mergeHashes, ct);
|
||||
```
|
||||
|
||||
## TTL Policy
|
||||
|
||||
Interest score determines TTL tier:
|
||||
|
||||
| Interest Score | TTL | Rationale |
|
||||
|----------------|-----|-----------|
|
||||
| >= 0.7 (High) | 24 hours | Hot advisories: likely to be queried frequently |
|
||||
| 0.3 - 0.7 (Medium) | 4 hours | Moderate interest: balance between freshness and cache hits |
|
||||
| < 0.3 (Low) | 1 hour | Low interest: evict quickly to save memory |
|
||||
|
||||
TTL is set when advisory is cached:
|
||||
|
||||
```csharp
|
||||
var ttl = ttlPolicy.GetTtl(advisory.InterestScore);
|
||||
await cache.SetAsync(key, advisory, ttl, ct);
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
### Metrics
|
||||
|
||||
| Metric | Type | Description |
|
||||
|--------|------|-------------|
|
||||
| `concelier_cache_hits_total` | Counter | Total cache hits |
|
||||
| `concelier_cache_misses_total` | Counter | Total cache misses |
|
||||
| `concelier_cache_hit_rate` | Gauge | Hit rate (hits / total) |
|
||||
| `concelier_cache_latency_ms` | Histogram | Cache operation latency |
|
||||
| `concelier_cache_size_bytes` | Gauge | Estimated cache memory usage |
|
||||
| `concelier_cache_hot_set_size` | Gauge | Entries in rank:hot |
|
||||
|
||||
### OpenTelemetry Spans
|
||||
|
||||
Cache operations emit spans:
|
||||
|
||||
```
|
||||
concelier.cache.get
|
||||
├── cache.key: "advisory:sha256:..."
|
||||
├── cache.hit: true/false
|
||||
└── cache.latency_ms: 2.5
|
||||
|
||||
concelier.cache.set
|
||||
├── cache.key: "advisory:sha256:..."
|
||||
└── cache.ttl_hours: 24
|
||||
```
|
||||
|
||||
### Health Check
|
||||
|
||||
```
|
||||
GET /health/cache
|
||||
```
|
||||
|
||||
Response:
|
||||
```json
|
||||
{
|
||||
"status": "healthy",
|
||||
"valkey_connected": true,
|
||||
"latency_ms": 1.2,
|
||||
"hot_set_size": 8542,
|
||||
"hit_rate_1h": 0.87
|
||||
}
|
||||
```
|
||||
|
||||
## Performance
|
||||
|
||||
### Benchmarks
|
||||
|
||||
| Operation | p50 | p95 | p99 |
|
||||
|-----------|-----|-----|-----|
|
||||
| GetAsync (hit) | 1.2ms | 3.5ms | 8.0ms |
|
||||
| GetAsync (miss + populate) | 12ms | 25ms | 45ms |
|
||||
| SetAsync | 1.5ms | 4.0ms | 9.0ms |
|
||||
| GetByPurlAsync | 2.5ms | 6.0ms | 15ms |
|
||||
| GetHotAsync(100) | 3.0ms | 8.0ms | 18ms |
|
||||
|
||||
### Optimization Tips
|
||||
|
||||
1. **Connection Pooling:** Use shared multiplexer with `PoolSize: 20`
|
||||
|
||||
2. **Pipeline Reads:** For bulk operations, use pipelining:
|
||||
```csharp
|
||||
var batch = cache.CreateBatch();
|
||||
foreach (var hash in mergeHashes)
|
||||
tasks.Add(batch.GetAsync(hash));
|
||||
batch.Execute();
|
||||
```
|
||||
|
||||
3. **Hot Set Preload:** Run warmup job on startup to preload hot set
|
||||
|
||||
4. **Compression:** Enable Valkey LZF compression for large advisories
|
||||
|
||||
## Fallback Behavior
|
||||
|
||||
When Valkey is unavailable:
|
||||
|
||||
1. **FallbackToPostgres: true** (default)
|
||||
- All reads go directly to PostgreSQL
|
||||
- Performance degrades but system remains operational
|
||||
- Reconnection attempts continue in background
|
||||
|
||||
2. **FallbackToPostgres: false**
|
||||
- Cache misses return null/empty
|
||||
- Only cached data is accessible
|
||||
- Use for strict latency requirements
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
| Issue | Cause | Solution |
|
||||
|-------|-------|----------|
|
||||
| High miss rate | Cache cold / insufficient TTL | Run warmup job, increase TTLs |
|
||||
| Latency spikes | Connection exhaustion | Increase PoolSize |
|
||||
| Memory pressure | Too many cached advisories | Reduce HotSetMaxSize, lower TTLs |
|
||||
| Index stale | Invalidation not triggered | Check event handlers, verify IndexPurlAsync calls |
|
||||
|
||||
### Debug Commands
|
||||
|
||||
```bash
|
||||
# Check cache stats
|
||||
stella cache stats
|
||||
|
||||
# View hot set
|
||||
stella cache list-hot --limit 10
|
||||
|
||||
# Check specific advisory
|
||||
stella cache get sha256:mergehash...
|
||||
|
||||
# Flush cache
|
||||
stella cache flush --confirm
|
||||
|
||||
# Check PURL index
|
||||
stella cache lookup-purl pkg:npm/express@4.0.0
|
||||
```
|
||||
|
||||
### Valkey CLI
|
||||
|
||||
```bash
|
||||
# Connect to Valkey
|
||||
redis-cli -h localhost -p 6379
|
||||
|
||||
# Check memory usage
|
||||
INFO memory
|
||||
|
||||
# List hot set entries
|
||||
ZRANGE rank:hot 0 9 WITHSCORES
|
||||
|
||||
# Check PURL index
|
||||
SMEMBERS by:purl:pkg:npm/express@4.0.0
|
||||
|
||||
# Get advisory
|
||||
GET advisory:sha256:a1b2c3...
|
||||
```
|
||||
345
docs/modules/concelier/sbom-learning-api.md
Normal file
345
docs/modules/concelier/sbom-learning-api.md
Normal file
@@ -0,0 +1,345 @@
|
||||
# SBOM Learning API
|
||||
|
||||
Per SPRINT_8200_0013_0003.
|
||||
|
||||
## Overview
|
||||
|
||||
The SBOM Learning API enables Concelier to learn which advisories are relevant to your organization by registering SBOMs from scanned images. When an SBOM is registered, Concelier matches its components against the canonical advisory database and updates interest scores accordingly.
|
||||
|
||||
## Flow
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────────┐
|
||||
│ SBOM Learning Flow │
|
||||
├─────────────────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ ┌─────────┐ scan ┌─────────┐ SBOM ┌───────────┐ │
|
||||
│ │ Image │ ──────────► │ Scanner │ ─────────► │ Concelier │ │
|
||||
│ │ │ │ │ │ │ │
|
||||
│ └─────────┘ └─────────┘ └─────┬─────┘ │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌─────────────────────┐ │
|
||||
│ │ SBOM Registration │ │
|
||||
│ │ ┌───────────────┐ │ │
|
||||
│ │ │ Extract PURLs │ │ │
|
||||
│ │ └───────┬───────┘ │ │
|
||||
│ │ │ │ │
|
||||
│ │ ▼ │ │
|
||||
│ │ ┌───────────────┐ │ │
|
||||
│ │ │ Match Advs │ │ │
|
||||
│ │ └───────┬───────┘ │ │
|
||||
│ │ │ │ │
|
||||
│ │ ▼ │ │
|
||||
│ │ ┌───────────────┐ │ │
|
||||
│ │ │ Update Scores │ │ │
|
||||
│ │ └───────────────┘ │ │
|
||||
│ └─────────────────────┘ │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## API Endpoints
|
||||
|
||||
### Register SBOM
|
||||
|
||||
```
|
||||
POST /api/v1/learn/sbom
|
||||
Content-Type: application/vnd.cyclonedx+json
|
||||
```
|
||||
|
||||
or
|
||||
|
||||
```
|
||||
POST /api/v1/learn/sbom
|
||||
Content-Type: application/spdx+json
|
||||
```
|
||||
|
||||
**Request Body:** CycloneDX or SPDX SBOM document
|
||||
|
||||
**Query Parameters:**
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
|-----------|------|---------|-------------|
|
||||
| `artifact_id` | string | required | Image digest or artifact identifier |
|
||||
| `update_scores` | bool | true | Trigger immediate score recalculation |
|
||||
| `include_reachability` | bool | true | Include reachability data in matching |
|
||||
|
||||
**Response:**
|
||||
|
||||
```json
|
||||
{
|
||||
"sbom_id": "uuid",
|
||||
"sbom_digest": "sha256:abc123...",
|
||||
"artifact_id": "sha256:image...",
|
||||
"component_count": 234,
|
||||
"matched_advisories": 15,
|
||||
"scores_updated": true,
|
||||
"registered_at": "2025-01-15T10:30:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
### Get Affected Advisories
|
||||
|
||||
```
|
||||
GET /api/v1/sboms/{digest}/affected
|
||||
```
|
||||
|
||||
**Response:**
|
||||
|
||||
```json
|
||||
{
|
||||
"sbom_digest": "sha256:abc123...",
|
||||
"artifact_id": "sha256:image...",
|
||||
"matched_advisories": [
|
||||
{
|
||||
"canonical_id": "uuid",
|
||||
"cve": "CVE-2024-1234",
|
||||
"severity": "high",
|
||||
"interest_score": 0.85,
|
||||
"matched_component": "pkg:npm/express@4.17.1",
|
||||
"is_reachable": true
|
||||
},
|
||||
{
|
||||
"canonical_id": "uuid",
|
||||
"cve": "CVE-2024-5678",
|
||||
"severity": "medium",
|
||||
"interest_score": 0.65,
|
||||
"matched_component": "pkg:npm/lodash@4.17.20",
|
||||
"is_reachable": false
|
||||
}
|
||||
],
|
||||
"total_count": 15,
|
||||
"last_matched_at": "2025-01-15T10:30:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
### List Registered SBOMs
|
||||
|
||||
```
|
||||
GET /api/v1/sboms
|
||||
```
|
||||
|
||||
**Query Parameters:**
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
|-----------|------|---------|-------------|
|
||||
| `artifact_id` | string | null | Filter by artifact |
|
||||
| `since` | datetime | null | Only SBOMs registered after this time |
|
||||
| `limit` | int | 100 | Max results |
|
||||
| `cursor` | string | null | Pagination cursor |
|
||||
|
||||
**Response:**
|
||||
|
||||
```json
|
||||
{
|
||||
"sboms": [
|
||||
{
|
||||
"id": "uuid",
|
||||
"artifact_id": "sha256:image...",
|
||||
"sbom_digest": "sha256:abc123...",
|
||||
"sbom_format": "cyclonedx",
|
||||
"component_count": 234,
|
||||
"matched_advisory_count": 15,
|
||||
"registered_at": "2025-01-15T10:30:00Z"
|
||||
}
|
||||
],
|
||||
"total_count": 42,
|
||||
"next_cursor": "cursor..."
|
||||
}
|
||||
```
|
||||
|
||||
### Unregister SBOM
|
||||
|
||||
```
|
||||
DELETE /api/v1/sboms/{digest}
|
||||
```
|
||||
|
||||
**Query Parameters:**
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
|-----------|------|---------|-------------|
|
||||
| `update_scores` | bool | true | Recalculate scores after removal |
|
||||
|
||||
## Matching Algorithm
|
||||
|
||||
### PURL Matching
|
||||
|
||||
1. **Exact Match:** `pkg:npm/express@4.17.1` matches advisories affecting exactly that version
|
||||
2. **Range Match:** Uses semantic version ranges from advisory affects_key
|
||||
3. **Namespace Normalization:** `@scope/pkg` normalized for comparison
|
||||
|
||||
### CPE Matching
|
||||
|
||||
For OS packages (rpm, deb):
|
||||
|
||||
1. Extract CPE from SBOM
|
||||
2. Match against advisory CPE patterns
|
||||
3. Apply distro-specific version logic (NEVRA/EVR)
|
||||
|
||||
### Reachability Integration
|
||||
|
||||
When `include_reachability=true`:
|
||||
|
||||
1. Query Scanner call graph data for matched components
|
||||
2. Mark `is_reachable` based on path from entry point
|
||||
3. Factor into interest score calculation
|
||||
|
||||
## Events
|
||||
|
||||
### SbomLearned
|
||||
|
||||
Published when SBOM is registered:
|
||||
|
||||
```json
|
||||
{
|
||||
"event_type": "sbom_learned",
|
||||
"sbom_id": "uuid",
|
||||
"sbom_digest": "sha256:...",
|
||||
"artifact_id": "sha256:...",
|
||||
"component_count": 234,
|
||||
"matched_advisory_count": 15,
|
||||
"timestamp": "2025-01-15T10:30:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
### ScoresUpdated
|
||||
|
||||
Published after batch score update:
|
||||
|
||||
```json
|
||||
{
|
||||
"event_type": "scores_updated",
|
||||
"trigger": "sbom_registration",
|
||||
"sbom_digest": "sha256:...",
|
||||
"advisories_updated": 15,
|
||||
"timestamp": "2025-01-15T10:30:05Z"
|
||||
}
|
||||
```
|
||||
|
||||
## Auto-Learning
|
||||
|
||||
Subscribe to Scanner events for automatic SBOM registration:
|
||||
|
||||
### Configuration
|
||||
|
||||
```yaml
|
||||
SbomIntegration:
|
||||
AutoLearn:
|
||||
Enabled: true
|
||||
SubscribeToScanEvents: true
|
||||
EventSource: "scanner:scan_completed"
|
||||
|
||||
Matching:
|
||||
EnablePurl: true
|
||||
EnableCpe: true
|
||||
IncludeReachability: true
|
||||
|
||||
ScoreUpdate:
|
||||
BatchSize: 1000
|
||||
DelaySeconds: 5 # Debounce rapid updates
|
||||
```
|
||||
|
||||
### Event Handler
|
||||
|
||||
```csharp
|
||||
// Automatic registration on scan completion
|
||||
public class ScanCompletedHandler : IEventHandler<ScanCompletedEvent>
|
||||
{
|
||||
public async Task HandleAsync(ScanCompletedEvent evt, CancellationToken ct)
|
||||
{
|
||||
await _sbomService.LearnFromScanAsync(
|
||||
artifactId: evt.ImageDigest,
|
||||
sbomDigest: evt.SbomDigest,
|
||||
sbomContent: evt.SbomContent,
|
||||
cancellationToken: ct);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## CLI Commands
|
||||
|
||||
```bash
|
||||
# Register SBOM from file
|
||||
stella learn sbom --file ./sbom.json --artifact sha256:image...
|
||||
|
||||
# Register from stdin
|
||||
cat sbom.json | stella learn sbom --artifact sha256:image...
|
||||
|
||||
# List affected advisories
|
||||
stella sbom affected sha256:sbomdigest...
|
||||
|
||||
# List registered SBOMs
|
||||
stella sbom list --limit 20
|
||||
|
||||
# Unregister SBOM
|
||||
stella sbom unregister sha256:sbomdigest...
|
||||
```
|
||||
|
||||
## Integration Examples
|
||||
|
||||
### CI/CD Pipeline
|
||||
|
||||
```yaml
|
||||
# Example GitHub Actions workflow
|
||||
- name: Scan image
|
||||
run: stella scan image myapp:latest -o sbom.json
|
||||
|
||||
- name: Register SBOM
|
||||
run: stella learn sbom --file sbom.json --artifact ${{ steps.build.outputs.digest }}
|
||||
|
||||
- name: Check for critical advisories
|
||||
run: |
|
||||
AFFECTED=$(stella sbom affected ${{ steps.sbom.outputs.digest }} --severity critical --count)
|
||||
if [ "$AFFECTED" -gt 0 ]; then
|
||||
echo "::error::Found $AFFECTED critical advisories"
|
||||
exit 1
|
||||
fi
|
||||
```
|
||||
|
||||
### Programmatic Registration
|
||||
|
||||
```csharp
|
||||
// Register SBOM from code
|
||||
var result = await sbomService.RegisterSbomAsync(
|
||||
artifactId: imageDigest,
|
||||
sbomContent: sbomJson,
|
||||
format: SbomFormat.CycloneDX,
|
||||
options: new RegistrationOptions
|
||||
{
|
||||
UpdateScores = true,
|
||||
IncludeReachability = true
|
||||
},
|
||||
cancellationToken);
|
||||
|
||||
// Get affected advisories
|
||||
var affected = await sbomService.GetAffectedAdvisoriesAsync(
|
||||
sbomDigest: result.SbomDigest,
|
||||
cancellationToken);
|
||||
```
|
||||
|
||||
## Database Schema
|
||||
|
||||
```sql
|
||||
CREATE TABLE vuln.sbom_registry (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
tenant_id UUID NOT NULL,
|
||||
artifact_id TEXT NOT NULL,
|
||||
sbom_digest TEXT NOT NULL,
|
||||
sbom_format TEXT NOT NULL,
|
||||
component_count INT NOT NULL DEFAULT 0,
|
||||
registered_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
last_matched_at TIMESTAMPTZ,
|
||||
CONSTRAINT uq_sbom_registry_digest UNIQUE (tenant_id, sbom_digest)
|
||||
);
|
||||
|
||||
CREATE TABLE vuln.sbom_canonical_match (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
sbom_id UUID NOT NULL REFERENCES vuln.sbom_registry(id),
|
||||
canonical_id UUID NOT NULL REFERENCES vuln.advisory_canonical(id),
|
||||
matched_purl TEXT NOT NULL,
|
||||
is_reachable BOOLEAN NOT NULL DEFAULT false,
|
||||
matched_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
CONSTRAINT uq_sbom_canonical_match UNIQUE (sbom_id, canonical_id)
|
||||
);
|
||||
```
|
||||
Reference in New Issue
Block a user