Frontend gaps fill work. Testing fixes work. Auditing in progress.

2025-12-30 01:22:58 +02:00
parent 1dc4bcbf10
commit 7a5210e2aa
928 changed files with 183942 additions and 3941 deletions
--- a/docs/modules/sbomservice/sources/architecture.md
+++ b/docs/modules/sbomservice/sources/architecture.md
@@ -0,0 +1,333 @@
+# SBOM Sources Architecture
+
+> Sprint: SPRINT_20251229_000_PLATFORM_sbom_sources_overview
+> Last Updated: 2025-12-29
+
+## 1. Overview
+
+The SBOM Sources subsystem provides a unified configuration and management layer for all SBOM ingestion pathways: Zastava (registry webhooks), Docker (direct image scans), CLI (external submissions), and Git (repository scans).
+
+### 1.1 Problem Statement
+
+**Current State:**
+- Fragmented source management across Orchestrator, Concelier, and Scanner
+- No unified UI for configuring ingestion sources
+- Credentials scattered without centralized management
+- No visibility into source health and scan history
+
+**Target State:**
+- Single "Sources" module for all SBOM ingestion configuration
+- Unified UI with type-specific wizards
+- Centralized credential management via AuthRef pattern
+- Real-time status monitoring and run history
+
+## 2. Source Types Taxonomy
+
+| Type | Trigger Mode | Target | Auth Pattern | Discovery |
+|------|--------------|--------|--------------|-----------|
+| **Zastava** | Webhook (push) | Registry images | Registry creds + webhook secret | From webhook payload |
+| **Docker** | Scheduled/Manual | Specific images | Registry credentials | Tag patterns |
+| **CLI** | External submission | Any SBOM | API token | N/A (receives SBOMs) |
+| **Git** | Webhook/Scheduled | Repository code | PAT/SSH + webhook secret | Branch patterns |
+
+## 3. Health Signals
+
+Each source maintains the following health signals:
+
+| Signal | Description | Metric |
+|--------|-------------|--------|
+| `status` | Current operational state | `active`, `paused`, `error`, `disabled`, `pending` |
+| `lastRunAt` | Timestamp of most recent run | ISO-8601 UTC |
+| `lastRunStatus` | Status of most recent run | `succeeded`, `failed`, `partial-success` |
+| `consecutiveFailures` | Count of consecutive failed runs | Integer, resets on success |
+| `currentHourScans` | Scans executed in rate-limit window | Integer, resets hourly |
+
+## 4. API Contract
+
+### 4.1 Source CRUD Endpoints
+
+```
+GET    /api/v1/sources                    List sources with filtering/pagination
+POST   /api/v1/sources                    Create new source
+GET    /api/v1/sources/{sourceId}         Get source details
+PUT    /api/v1/sources/{sourceId}         Update source configuration
+DELETE /api/v1/sources/{sourceId}         Delete source
+```
+
+### 4.2 Operational Endpoints
+
+```
+POST   /api/v1/sources/{sourceId}/test    Test source connection
+POST   /api/v1/sources/{sourceId}/trigger Manual scan trigger
+POST   /api/v1/sources/{sourceId}/pause   Pause source (with reason)
+POST   /api/v1/sources/{sourceId}/resume  Resume paused source
+```
+
+### 4.3 Run History Endpoints
+
+```
+GET    /api/v1/sources/{sourceId}/runs              List run history (paginated)
+GET    /api/v1/sources/{sourceId}/runs/{runId}      Get run details
+```
+
+### 4.4 Webhook Endpoints
+
+```
+POST   /api/v1/webhooks/zastava/{sourceId}   Zastava registry webhook
+POST   /api/v1/webhooks/git/{sourceId}       Git repository webhook
+```
+
+## 5. Domain Events
+
+| Event | Payload | When Emitted |
+|-------|---------|--------------|
+| `source.created` | `{ sourceId, sourceType, tenantId }` | New source registered |
+| `source.updated` | `{ sourceId, changedFields[] }` | Source configuration updated |
+| `source.deleted` | `{ sourceId }` | Source removed |
+| `source.paused` | `{ sourceId, reason, pausedBy }` | Source paused |
+| `source.resumed` | `{ sourceId, resumedBy }` | Source resumed |
+| `source.run.started` | `{ runId, sourceId, trigger }` | Run initiated |
+| `source.run.completed` | `{ runId, sourceId, status, metrics }` | Run finished |
+
+## 6. Credential Lifecycle (AuthRef Pattern)
+
+All source credentials use the AuthRef pattern for secure storage:
+
+1. **Storage**: Credentials stored in Authority vault, never inline in source configs
+2. **Reference**: Sources hold `authRef` identifiers pointing to vault entries
+3. **Rotation**: Rotate-on-demand API; old refs invalidated, new ref issued
+4. **Audit**: All credential access logged with source context
+
+### 6.1 Supported Credential Types
+
+| Source Type | Credential Types |
+|-------------|------------------|
+| Zastava | Registry auth (basic/token), webhook secret |
+| Docker | Registry auth (basic/token/ECR/GCR/ACR) |
+| CLI | API token, OIDC identity |
+| Git | PAT, SSH key, webhook secret |
+
+## 7. Telemetry Schema
+
+### 7.1 Metrics
+
+| Metric | Type | Labels | Description |
+|--------|------|--------|-------------|
+| `sbom_source_runs_total` | Counter | `source_type`, `status`, `trigger` | Total runs by type and outcome |
+| `sbom_source_run_duration_seconds` | Histogram | `source_type` | Run execution time |
+| `sbom_source_items_scanned_total` | Counter | `source_type` | Items processed per run |
+| `sbom_source_connection_test_duration_seconds` | Histogram | `source_type` | Connection test latency |
+| `sbom_source_active_count` | Gauge | `source_type`, `status` | Active sources by type |
+
+### 7.2 Structured Logs
+
+All source operations emit structured logs including:
+- `tenantId`, `sourceId`, `sourceType`
+- `runId` (when applicable)
+- `correlationId` for cross-service tracing
+
+## 8. Module Ownership Matrix
+
+| Component | Owner Module | Interface |
+|-----------|--------------|-----------|
+| Source entity persistence | SbomService | PostgreSQL + Repository |
+| Credential storage | Authority | AuthRef vault API |
+| Webhook signature verification | SbomService | HMAC validation |
+| Scheduled trigger dispatch | Scheduler | Cron job registration |
+| Scan job execution | Scanner | Job queue interface |
+| UI configuration | Web | Sources Manager feature |
+
+## 9. UI Information Architecture
+
+### 9.1 Navigation Placement
+
+| Section | Route | Purpose |
+|---------|-------|---------|
+| Sources List | `/sources` | Primary view with filtering, status overview, bulk actions |
+| Source Detail | `/sources/{id}` | Configuration view, run history, health metrics |
+| Add Source Wizard | `/sources/new` | Multi-step creation flow |
+| Edit Source | `/sources/{id}/edit` | Modify existing source configuration |
+
+### 9.2 Wizard Flow Architecture
+
+The Add/Edit Source Wizard follows a 6-step progressive disclosure pattern:
+
+```
+┌─────────────────────────────────────────────────────────────────────────────┐
+│  Step 1: Source Type Selection                                              │
+│  ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐          │
+│  │  Zastava    │ │   Docker    │ │    CLI      │ │    Git      │          │
+│  │  (Webhook)  │ │  (Scanner)  │ │ (Receiver)  │ │ (Repository)│          │
+│  └─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘          │
+└─────────────────────────────────────────────────────────────────────────────┘
+                                    ↓
+┌─────────────────────────────────────────────────────────────────────────────┐
+│  Step 2: Basic Information                                                  │
+│  • Name (required, unique per tenant)                                       │
+│  • Description (optional)                                                   │
+│  • Tags (multi-select, for filtering)                                       │
+│  • Metadata key-value pairs (optional)                                      │
+└─────────────────────────────────────────────────────────────────────────────┘
+                                    ↓
+┌─────────────────────────────────────────────────────────────────────────────┐
+│  Step 3: Type-Specific Configuration (varies by source type)                │
+│                                                                             │
+│  Zastava:                      Docker:                                      │
+│  • Registry URL                • Registry URL                               │
+│  • Registry type dropdown      • Image references (multi-add)               │
+│  • Repository filters          • Tag patterns (include/exclude)             │
+│  • Tag patterns                • Platform selection                         │
+│  • Webhook path (generated)    • Scan options (analyzers, reachability)     │
+│                                                                             │
+│  CLI:                          Git:                                         │
+│  • Allowed tools list          • Provider (GitHub/GitLab/Gitea/etc.)        │
+│  • Allowed CI systems          • Repository URL                             │
+│  • Validation rules            • Branch patterns (include/exclude)          │
+│  • Attribution requirements    • Trigger modes (push/PR/tag/scheduled)      │
+│  • Max SBOM size limit         • Scan paths and exclusions                  │
+└─────────────────────────────────────────────────────────────────────────────┘
+                                    ↓
+┌─────────────────────────────────────────────────────────────────────────────┐
+│  Step 4: Credentials (AuthRef Pattern)                                      │
+│  • Credential type selector (per source type)                               │
+│  • Create new credential vs. select existing                                │
+│  • Inline credential entry (stored via Authority vault)                     │
+│  • Webhook secret generation (for Zastava/Git)                              │
+│  • Test credential validity button                                          │
+└─────────────────────────────────────────────────────────────────────────────┘
+                                    ↓
+┌─────────────────────────────────────────────────────────────────────────────┐
+│  Step 5: Schedule (Optional - for Docker/Git scheduled triggers)            │
+│  • Enable scheduled scans toggle                                            │
+│  • Cron expression builder or preset selector                               │
+│  • Timezone picker                                                          │
+│  • Rate limit (max scans per hour)                                          │
+│  • Next run preview                                                         │
+└─────────────────────────────────────────────────────────────────────────────┘
+                                    ↓
+┌─────────────────────────────────────────────────────────────────────────────┐
+│  Step 6: Review & Test Connection                                           │
+│  • Configuration summary (read-only)                                        │
+│  • Test Connection button with status indicator                             │
+│  • Error details expansion if test fails                                    │
+│  • Create Source / Save Changes button                                      │
+│  • Webhook endpoint display (for Zastava/Git - copy to clipboard)           │
+└─────────────────────────────────────────────────────────────────────────────┘
+```
+
+### 9.3 Wizard Step Validation
+
+| Step | Required Fields | Validation |
+|------|-----------------|------------|
+| 1 - Type | `sourceType` | Must select one type |
+| 2 - Basic Info | `name` | Unique name, 3-100 chars |
+| 3 - Config | Varies by type | Type-specific required fields |
+| 4 - Credentials | `authRef` (if required) | Valid credential for source type |
+| 5 - Schedule | None (optional step) | Valid cron if enabled |
+| 6 - Review | None | Connection test recommended |
+
+### 9.4 Source List Page Components
+
+```
+┌─────────────────────────────────────────────────────────────────────────────┐
+│  SBOM Sources                                               [+ Add Source]  │
+├─────────────────────────────────────────────────────────────────────────────┤
+│  Status Cards:                                                              │
+│  ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐                       │
+│  │ Active   │ │ Paused   │ │ Error    │ │ Pending  │                       │
+│  │   12     │ │    3     │ │    2     │ │    1     │                       │
+│  └──────────┘ └──────────┘ └──────────┘ └──────────┘                       │
+├─────────────────────────────────────────────────────────────────────────────┤
+│  Filters: [Type ▼] [Status ▼] [Tags ▼] [Search: ________]                  │
+├─────────────────────────────────────────────────────────────────────────────┤
+│  □ │ Name          │ Type    │ Status   │ Last Run   │ Next Run │ Actions │
+│  ──┼───────────────┼─────────┼──────────┼────────────┼──────────┼─────────│
+│  □ │ DockerHub Prd │ Zastava │ ● Active │ 5 min ago  │ -        │ ⋮       │
+│  □ │ Harbor Dev    │ Zastava │ ● Active │ 12 min ago │ -        │ ⋮       │
+│  □ │ Nightly Scans │ Docker  │ ● Active │ 2h ago     │ 02:00 AM │ ⋮       │
+│  □ │ CI Pipeline   │ CLI     │ ⏸ Paused │ 1 day ago  │ -        │ ⋮       │
+│  □ │ Monorepo      │ Git     │ ⚠ Error  │ Failed     │ -        │ ⋮       │
+├─────────────────────────────────────────────────────────────────────────────┤
+│  Showing 1-5 of 18                      [< Prev] [1] [2] [3] [4] [Next >]   │
+└─────────────────────────────────────────────────────────────────────────────┘
+```
+
+### 9.5 Row Actions Menu
+
+| Action | Icon | Availability |
+|--------|------|--------------|
+| View Details | 👁 | Always |
+| Edit | ✏️ | Always |
+| Trigger Scan | ▶ | Active sources |
+| Test Connection | 🔌 | Always |
+| Pause | ⏸ | Active sources |
+| Resume | ▶ | Paused sources |
+| Delete | 🗑 | Always (with confirmation) |
+
+### 9.6 Source Detail Page Layout
+
+```
+┌─────────────────────────────────────────────────────────────────────────────┐
+│  ← Back to Sources                                                          │
+├─────────────────────────────────────────────────────────────────────────────┤
+│  Docker Hub Production                      [Trigger] [Test] [Edit] [⋮]    │
+│  Type: Zastava │ Status: ● Active │ Created: Dec 15, 2025                   │
+├─────────────────────────────────────────────────────────────────────────────┤
+│  [Overview] [Configuration] [Run History] [Metrics]                         │
+├─────────────────────────────────────────────────────────────────────────────┤
+│                                                                             │
+│  Configuration:                                                             │
+│  ┌───────────────────────────────────────────────────────────────────────┐ │
+│  │ Registry URL:     https://registry-1.docker.io                        │ │
+│  │ Registry Type:    Docker Hub                                          │ │
+│  │ Webhook Path:     /api/v1/webhooks/zastava/src-abc123                 │ │
+│  │ Repository Filter: myorg/*, prod-*                                    │ │
+│  │ Tag Filter:        v*, latest (excluding: *-dev, *-test)              │ │
+│  │ Analyzers:         OS, Node.js, Python, Go                            │ │
+│  │ Reachability:      Enabled                                            │ │
+│  │ VEX Lookup:        Enabled                                            │ │
+│  └───────────────────────────────────────────────────────────────────────┘ │
+│                                                                             │
+│  Run History:                                                               │
+│  ┌───────────────────────────────────────────────────────────────────────┐ │
+│  │ Run ID  │ Trigger  │ Status    │ Started    │ Duration │ Items       │ │
+│  │ run-456 │ Webhook  │ ✓ Success │ 5 min ago  │ 45s      │ 3/3         │ │
+│  │ run-455 │ Webhook  │ ✓ Success │ 12 min ago │ 38s      │ 1/1         │ │
+│  │ run-454 │ Manual   │ ✓ Success │ 1h ago     │ 2m 15s   │ 12/12       │ │
+│  │ run-453 │ Webhook  │ ⚠ Partial │ 2h ago     │ 1m 30s   │ 4/5         │ │
+│  │ run-452 │ Schedule │ ✗ Failed  │ 3h ago     │ 12s      │ 0/0         │ │
+│  └───────────────────────────────────────────────────────────────────────┘ │
+│  [Load More]                                                                │
+│                                                                             │
+└─────────────────────────────────────────────────────────────────────────────┘
+```
+
+## 10. Security Considerations
+
+| Concern | Mitigation |
+|---------|------------|
+| Credential exposure | AuthRef pattern - never inline, always reference vault |
+| Webhook forgery | HMAC signature verification for all webhooks |
+| Unauthorized access | Scope-based authorization (`sources:read`, `sources:write`, `sources:admin`) |
+| Secret logging | Audit logging excludes credential values |
+| Webhook secret rotation | Rotate-on-demand API endpoint |
+
+## 10. Configuration
+
+Environment variables (prefix `SBOM_SbomService__Sources__`):
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `MaxSourcesPerTenant` | 100 | Maximum sources per tenant |
+| `DefaultMaxScansPerHour` | 60 | Default rate limit per source |
+| `RunHistoryRetentionDays` | 90 | Run history retention period |
+| `WebhookSignatureAlgorithm` | `HMAC-SHA256` | Webhook signature algorithm |
+| `ConnectionTestTimeoutSeconds` | 30 | Connection test timeout |
+
+## 11. Related Documentation
+
+- [SbomService Architecture](../architecture.md)
+- [Lineage Ledger](../lineage-ledger.md)
+- [Authority Architecture](../../authority/architecture.md)
+- [Scanner Architecture](../../scanner/architecture.md)
+- [UI Architecture](../../ui/architecture.md)