334 lines
22 KiB
Markdown
334 lines
22 KiB
Markdown
# SBOM Sources Architecture
|
|
|
|
> Sprint: SPRINT_20251229_000_PLATFORM_sbom_sources_overview
|
|
> Last Updated: 2025-12-29
|
|
|
|
## 1. Overview
|
|
|
|
The SBOM Sources subsystem provides a unified configuration and management layer for all SBOM ingestion pathways: Zastava (registry webhooks), Docker (direct image scans), CLI (external submissions), and Git (repository scans).
|
|
|
|
### 1.1 Problem Statement
|
|
|
|
**Current State:**
|
|
- Fragmented source management across Orchestrator, Concelier, and Scanner
|
|
- No unified UI for configuring ingestion sources
|
|
- Credentials scattered without centralized management
|
|
- No visibility into source health and scan history
|
|
|
|
**Target State:**
|
|
- Single "Sources" module for all SBOM ingestion configuration
|
|
- Unified UI with type-specific wizards
|
|
- Centralized credential management via AuthRef pattern
|
|
- Real-time status monitoring and run history
|
|
|
|
## 2. Source Types Taxonomy
|
|
|
|
| Type | Trigger Mode | Target | Auth Pattern | Discovery |
|
|
|------|--------------|--------|--------------|-----------|
|
|
| **Zastava** | Webhook (push) | Registry images | Registry creds + webhook secret | From webhook payload |
|
|
| **Docker** | Scheduled/Manual | Specific images | Registry credentials | Tag patterns |
|
|
| **CLI** | External submission | Any SBOM | API token | N/A (receives SBOMs) |
|
|
| **Git** | Webhook/Scheduled | Repository code | PAT/SSH + webhook secret | Branch patterns |
|
|
|
|
## 3. Health Signals
|
|
|
|
Each source maintains the following health signals:
|
|
|
|
| Signal | Description | Metric |
|
|
|--------|-------------|--------|
|
|
| `status` | Current operational state | `active`, `paused`, `error`, `disabled`, `pending` |
|
|
| `lastRunAt` | Timestamp of most recent run | ISO-8601 UTC |
|
|
| `lastRunStatus` | Status of most recent run | `succeeded`, `failed`, `partial-success` |
|
|
| `consecutiveFailures` | Count of consecutive failed runs | Integer, resets on success |
|
|
| `currentHourScans` | Scans executed in rate-limit window | Integer, resets hourly |
|
|
|
|
## 4. API Contract
|
|
|
|
### 4.1 Source CRUD Endpoints
|
|
|
|
```
|
|
GET /api/v1/sources List sources with filtering/pagination
|
|
POST /api/v1/sources Create new source
|
|
GET /api/v1/sources/{sourceId} Get source details
|
|
PUT /api/v1/sources/{sourceId} Update source configuration
|
|
DELETE /api/v1/sources/{sourceId} Delete source
|
|
```
|
|
|
|
### 4.2 Operational Endpoints
|
|
|
|
```
|
|
POST /api/v1/sources/{sourceId}/test Test source connection
|
|
POST /api/v1/sources/{sourceId}/trigger Manual scan trigger
|
|
POST /api/v1/sources/{sourceId}/pause Pause source (with reason)
|
|
POST /api/v1/sources/{sourceId}/resume Resume paused source
|
|
```
|
|
|
|
### 4.3 Run History Endpoints
|
|
|
|
```
|
|
GET /api/v1/sources/{sourceId}/runs List run history (paginated)
|
|
GET /api/v1/sources/{sourceId}/runs/{runId} Get run details
|
|
```
|
|
|
|
### 4.4 Webhook Endpoints
|
|
|
|
```
|
|
POST /api/v1/webhooks/zastava/{sourceId} Zastava registry webhook
|
|
POST /api/v1/webhooks/git/{sourceId} Git repository webhook
|
|
```
|
|
|
|
## 5. Domain Events
|
|
|
|
| Event | Payload | When Emitted |
|
|
|-------|---------|--------------|
|
|
| `source.created` | `{ sourceId, sourceType, tenantId }` | New source registered |
|
|
| `source.updated` | `{ sourceId, changedFields[] }` | Source configuration updated |
|
|
| `source.deleted` | `{ sourceId }` | Source removed |
|
|
| `source.paused` | `{ sourceId, reason, pausedBy }` | Source paused |
|
|
| `source.resumed` | `{ sourceId, resumedBy }` | Source resumed |
|
|
| `source.run.started` | `{ runId, sourceId, trigger }` | Run initiated |
|
|
| `source.run.completed` | `{ runId, sourceId, status, metrics }` | Run finished |
|
|
|
|
## 6. Credential Lifecycle (AuthRef Pattern)
|
|
|
|
All source credentials use the AuthRef pattern for secure storage:
|
|
|
|
1. **Storage**: Credentials stored in Authority vault, never inline in source configs
|
|
2. **Reference**: Sources hold `authRef` identifiers pointing to vault entries
|
|
3. **Rotation**: Rotate-on-demand API; old refs invalidated, new ref issued
|
|
4. **Audit**: All credential access logged with source context
|
|
|
|
### 6.1 Supported Credential Types
|
|
|
|
| Source Type | Credential Types |
|
|
|-------------|------------------|
|
|
| Zastava | Registry auth (basic/token), webhook secret |
|
|
| Docker | Registry auth (basic/token/ECR/GCR/ACR) |
|
|
| CLI | API token, OIDC identity |
|
|
| Git | PAT, SSH key, webhook secret |
|
|
|
|
## 7. Telemetry Schema
|
|
|
|
### 7.1 Metrics
|
|
|
|
| Metric | Type | Labels | Description |
|
|
|--------|------|--------|-------------|
|
|
| `sbom_source_runs_total` | Counter | `source_type`, `status`, `trigger` | Total runs by type and outcome |
|
|
| `sbom_source_run_duration_seconds` | Histogram | `source_type` | Run execution time |
|
|
| `sbom_source_items_scanned_total` | Counter | `source_type` | Items processed per run |
|
|
| `sbom_source_connection_test_duration_seconds` | Histogram | `source_type` | Connection test latency |
|
|
| `sbom_source_active_count` | Gauge | `source_type`, `status` | Active sources by type |
|
|
|
|
### 7.2 Structured Logs
|
|
|
|
All source operations emit structured logs including:
|
|
- `tenantId`, `sourceId`, `sourceType`
|
|
- `runId` (when applicable)
|
|
- `correlationId` for cross-service tracing
|
|
|
|
## 8. Module Ownership Matrix
|
|
|
|
| Component | Owner Module | Interface |
|
|
|-----------|--------------|-----------|
|
|
| Source entity persistence | SbomService | PostgreSQL + Repository |
|
|
| Credential storage | Authority | AuthRef vault API |
|
|
| Webhook signature verification | SbomService | HMAC validation |
|
|
| Scheduled trigger dispatch | Scheduler | Cron job registration |
|
|
| Scan job execution | Scanner | Job queue interface |
|
|
| UI configuration | Web | Sources Manager feature |
|
|
|
|
## 9. UI Information Architecture
|
|
|
|
### 9.1 Navigation Placement
|
|
|
|
| Section | Route | Purpose |
|
|
|---------|-------|---------|
|
|
| Sources List | `/sources` | Primary view with filtering, status overview, bulk actions |
|
|
| Source Detail | `/sources/{id}` | Configuration view, run history, health metrics |
|
|
| Add Source Wizard | `/sources/new` | Multi-step creation flow |
|
|
| Edit Source | `/sources/{id}/edit` | Modify existing source configuration |
|
|
|
|
### 9.2 Wizard Flow Architecture
|
|
|
|
The Add/Edit Source Wizard follows a 6-step progressive disclosure pattern:
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────────────────┐
|
|
│ Step 1: Source Type Selection │
|
|
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
|
|
│ │ Zastava │ │ Docker │ │ CLI │ │ Git │ │
|
|
│ │ (Webhook) │ │ (Scanner) │ │ (Receiver) │ │ (Repository)│ │
|
|
│ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘ │
|
|
└─────────────────────────────────────────────────────────────────────────────┘
|
|
↓
|
|
┌─────────────────────────────────────────────────────────────────────────────┐
|
|
│ Step 2: Basic Information │
|
|
│ • Name (required, unique per tenant) │
|
|
│ • Description (optional) │
|
|
│ • Tags (multi-select, for filtering) │
|
|
│ • Metadata key-value pairs (optional) │
|
|
└─────────────────────────────────────────────────────────────────────────────┘
|
|
↓
|
|
┌─────────────────────────────────────────────────────────────────────────────┐
|
|
│ Step 3: Type-Specific Configuration (varies by source type) │
|
|
│ │
|
|
│ Zastava: Docker: │
|
|
│ • Registry URL • Registry URL │
|
|
│ • Registry type dropdown • Image references (multi-add) │
|
|
│ • Repository filters • Tag patterns (include/exclude) │
|
|
│ • Tag patterns • Platform selection │
|
|
│ • Webhook path (generated) • Scan options (analyzers, reachability) │
|
|
│ │
|
|
│ CLI: Git: │
|
|
│ • Allowed tools list • Provider (GitHub/GitLab/Gitea/etc.) │
|
|
│ • Allowed CI systems • Repository URL │
|
|
│ • Validation rules • Branch patterns (include/exclude) │
|
|
│ • Attribution requirements • Trigger modes (push/PR/tag/scheduled) │
|
|
│ • Max SBOM size limit • Scan paths and exclusions │
|
|
└─────────────────────────────────────────────────────────────────────────────┘
|
|
↓
|
|
┌─────────────────────────────────────────────────────────────────────────────┐
|
|
│ Step 4: Credentials (AuthRef Pattern) │
|
|
│ • Credential type selector (per source type) │
|
|
│ • Create new credential vs. select existing │
|
|
│ • Inline credential entry (stored via Authority vault) │
|
|
│ • Webhook secret generation (for Zastava/Git) │
|
|
│ • Test credential validity button │
|
|
└─────────────────────────────────────────────────────────────────────────────┘
|
|
↓
|
|
┌─────────────────────────────────────────────────────────────────────────────┐
|
|
│ Step 5: Schedule (Optional - for Docker/Git scheduled triggers) │
|
|
│ • Enable scheduled scans toggle │
|
|
│ • Cron expression builder or preset selector │
|
|
│ • Timezone picker │
|
|
│ • Rate limit (max scans per hour) │
|
|
│ • Next run preview │
|
|
└─────────────────────────────────────────────────────────────────────────────┘
|
|
↓
|
|
┌─────────────────────────────────────────────────────────────────────────────┐
|
|
│ Step 6: Review & Test Connection │
|
|
│ • Configuration summary (read-only) │
|
|
│ • Test Connection button with status indicator │
|
|
│ • Error details expansion if test fails │
|
|
│ • Create Source / Save Changes button │
|
|
│ • Webhook endpoint display (for Zastava/Git - copy to clipboard) │
|
|
└─────────────────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
### 9.3 Wizard Step Validation
|
|
|
|
| Step | Required Fields | Validation |
|
|
|------|-----------------|------------|
|
|
| 1 - Type | `sourceType` | Must select one type |
|
|
| 2 - Basic Info | `name` | Unique name, 3-100 chars |
|
|
| 3 - Config | Varies by type | Type-specific required fields |
|
|
| 4 - Credentials | `authRef` (if required) | Valid credential for source type |
|
|
| 5 - Schedule | None (optional step) | Valid cron if enabled |
|
|
| 6 - Review | None | Connection test recommended |
|
|
|
|
### 9.4 Source List Page Components
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────────────────┐
|
|
│ SBOM Sources [+ Add Source] │
|
|
├─────────────────────────────────────────────────────────────────────────────┤
|
|
│ Status Cards: │
|
|
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
|
|
│ │ Active │ │ Paused │ │ Error │ │ Pending │ │
|
|
│ │ 12 │ │ 3 │ │ 2 │ │ 1 │ │
|
|
│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │
|
|
├─────────────────────────────────────────────────────────────────────────────┤
|
|
│ Filters: [Type ▼] [Status ▼] [Tags ▼] [Search: ________] │
|
|
├─────────────────────────────────────────────────────────────────────────────┤
|
|
│ □ │ Name │ Type │ Status │ Last Run │ Next Run │ Actions │
|
|
│ ──┼───────────────┼─────────┼──────────┼────────────┼──────────┼─────────│
|
|
│ □ │ DockerHub Prd │ Zastava │ ● Active │ 5 min ago │ - │ ⋮ │
|
|
│ □ │ Harbor Dev │ Zastava │ ● Active │ 12 min ago │ - │ ⋮ │
|
|
│ □ │ Nightly Scans │ Docker │ ● Active │ 2h ago │ 02:00 AM │ ⋮ │
|
|
│ □ │ CI Pipeline │ CLI │ ⏸ Paused │ 1 day ago │ - │ ⋮ │
|
|
│ □ │ Monorepo │ Git │ ⚠ Error │ Failed │ - │ ⋮ │
|
|
├─────────────────────────────────────────────────────────────────────────────┤
|
|
│ Showing 1-5 of 18 [< Prev] [1] [2] [3] [4] [Next >] │
|
|
└─────────────────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
### 9.5 Row Actions Menu
|
|
|
|
| Action | Icon | Availability |
|
|
|--------|------|--------------|
|
|
| View Details | 👁 | Always |
|
|
| Edit | ✏️ | Always |
|
|
| Trigger Scan | ▶ | Active sources |
|
|
| Test Connection | 🔌 | Always |
|
|
| Pause | ⏸ | Active sources |
|
|
| Resume | ▶ | Paused sources |
|
|
| Delete | 🗑 | Always (with confirmation) |
|
|
|
|
### 9.6 Source Detail Page Layout
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────────────────┐
|
|
│ ← Back to Sources │
|
|
├─────────────────────────────────────────────────────────────────────────────┤
|
|
│ Docker Hub Production [Trigger] [Test] [Edit] [⋮] │
|
|
│ Type: Zastava │ Status: ● Active │ Created: Dec 15, 2025 │
|
|
├─────────────────────────────────────────────────────────────────────────────┤
|
|
│ [Overview] [Configuration] [Run History] [Metrics] │
|
|
├─────────────────────────────────────────────────────────────────────────────┤
|
|
│ │
|
|
│ Configuration: │
|
|
│ ┌───────────────────────────────────────────────────────────────────────┐ │
|
|
│ │ Registry URL: https://registry-1.docker.io │ │
|
|
│ │ Registry Type: Docker Hub │ │
|
|
│ │ Webhook Path: /api/v1/webhooks/zastava/src-abc123 │ │
|
|
│ │ Repository Filter: myorg/*, prod-* │ │
|
|
│ │ Tag Filter: v*, latest (excluding: *-dev, *-test) │ │
|
|
│ │ Analyzers: OS, Node.js, Python, Go │ │
|
|
│ │ Reachability: Enabled │ │
|
|
│ │ VEX Lookup: Enabled │ │
|
|
│ └───────────────────────────────────────────────────────────────────────┘ │
|
|
│ │
|
|
│ Run History: │
|
|
│ ┌───────────────────────────────────────────────────────────────────────┐ │
|
|
│ │ Run ID │ Trigger │ Status │ Started │ Duration │ Items │ │
|
|
│ │ run-456 │ Webhook │ ✓ Success │ 5 min ago │ 45s │ 3/3 │ │
|
|
│ │ run-455 │ Webhook │ ✓ Success │ 12 min ago │ 38s │ 1/1 │ │
|
|
│ │ run-454 │ Manual │ ✓ Success │ 1h ago │ 2m 15s │ 12/12 │ │
|
|
│ │ run-453 │ Webhook │ ⚠ Partial │ 2h ago │ 1m 30s │ 4/5 │ │
|
|
│ │ run-452 │ Schedule │ ✗ Failed │ 3h ago │ 12s │ 0/0 │ │
|
|
│ └───────────────────────────────────────────────────────────────────────┘ │
|
|
│ [Load More] │
|
|
│ │
|
|
└─────────────────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
## 10. Security Considerations
|
|
|
|
| Concern | Mitigation |
|
|
|---------|------------|
|
|
| Credential exposure | AuthRef pattern - never inline, always reference vault |
|
|
| Webhook forgery | HMAC signature verification for all webhooks |
|
|
| Unauthorized access | Scope-based authorization (`sources:read`, `sources:write`, `sources:admin`) |
|
|
| Secret logging | Audit logging excludes credential values |
|
|
| Webhook secret rotation | Rotate-on-demand API endpoint |
|
|
|
|
## 10. Configuration
|
|
|
|
Environment variables (prefix `SBOM_SbomService__Sources__`):
|
|
|
|
| Variable | Default | Description |
|
|
|----------|---------|-------------|
|
|
| `MaxSourcesPerTenant` | 100 | Maximum sources per tenant |
|
|
| `DefaultMaxScansPerHour` | 60 | Default rate limit per source |
|
|
| `RunHistoryRetentionDays` | 90 | Run history retention period |
|
|
| `WebhookSignatureAlgorithm` | `HMAC-SHA256` | Webhook signature algorithm |
|
|
| `ConnectionTestTimeoutSeconds` | 30 | Connection test timeout |
|
|
|
|
## 11. Related Documentation
|
|
|
|
- [SbomService Architecture](../architecture.md)
|
|
- [Lineage Ledger](../lineage-ledger.md)
|
|
- [Authority Architecture](../../authority/architecture.md)
|
|
- [Scanner Architecture](../../scanner/architecture.md)
|
|
- [UI Architecture](../../ui/architecture.md)
|