master
b97fc7685a
Some checks failed
Build Test Deploy / authority-container (push) Has been cancelled
Build Test Deploy / docs (push) Has been cancelled
Build Test Deploy / deploy (push) Has been cancelled
Build Test Deploy / build-test (push) Has been cancelled
Docs CI / lint-and-preview (push) Has been cancelled
209 lines
10 KiB
Markdown
Executable File
209 lines
10 KiB
Markdown
Executable File
# 8 · Detailed Module Specifications — **Stella Ops Feedser**
|
||
_This document describes the Feedser service, its supporting libraries, connectors, exporters, and test assets that live in the OSS repository._
|
||
|
||
---
|
||
|
||
## 0 Scope
|
||
|
||
Feedser is the vulnerability ingest/merge/export subsystem of Stella Ops. It
|
||
fetches primary advisories, normalizes and deduplicates them into MongoDB, and
|
||
produces deterministic JSON and Trivy DB exports. This document lists the
|
||
projects that make up that workflow, the extension points they expose, and the
|
||
artefacts they ship.
|
||
|
||
---
|
||
|
||
## 1 Repository layout (current)
|
||
|
||
```text
|
||
src/
|
||
├─ Directory.Build.props / Directory.Build.targets
|
||
├─ StellaOps.Plugin/
|
||
├─ StellaOps.Feedser.Core/
|
||
├─ StellaOps.Feedser.Core.Tests/
|
||
├─ StellaOps.Feedser.Models/ (+ .Tests/)
|
||
├─ StellaOps.Feedser.Normalization/ (+ .Tests/)
|
||
├─ StellaOps.Feedser.Merge/ (+ .Tests/)
|
||
├─ StellaOps.Feedser.Storage.Mongo/ (+ .Tests/)
|
||
├─ StellaOps.Feedser.Exporter.Json/ (+ .Tests/)
|
||
├─ StellaOps.Feedser.Exporter.TrivyDb/ (+ .Tests/)
|
||
├─ StellaOps.Feedser.Source.* / StellaOps.Feedser.Source.*.Tests/
|
||
├─ StellaOps.Feedser.Testing/
|
||
├─ StellaOps.Feedser.Tests.Shared/
|
||
├─ StellaOps.Feedser.WebService/ (+ .Tests/)
|
||
├─ PluginBinaries/
|
||
└─ StellaOps.Feedser.sln
|
||
```
|
||
|
||
Each folder is a .NET project (or set of projects) referenced by
|
||
`StellaOps.Feedser.sln`. Build assets are shared through the root
|
||
`Directory.Build.props/targets` so conventions stay consistent.
|
||
|
||
---
|
||
|
||
## 2 Shared libraries
|
||
|
||
| Project | Purpose | Key extension points |
|
||
|---------|---------|----------------------|
|
||
| `StellaOps.Plugin` | Base contracts for connectors, exporters, and DI routines plus Cosign validation helpers. | `IFeedConnector`, `IExporterPlugin`, `IDependencyInjectionRoutine` |
|
||
| `StellaOps.DependencyInjection` | Composable service registrations for Feedser and plug-ins. | `IDependencyInjectionRoutine` discovery |
|
||
| `StellaOps.Feedser.Testing` | Common fixtures, builders, and harnesses for integration/unit tests. | `FeedserMongoFixture`, test builders |
|
||
| `StellaOps.Feedser.Tests.Shared` | Shared assembly metadata and fixtures wired in via `Directory.Build.props`. | Test assembly references |
|
||
|
||
---
|
||
|
||
## 3 Core projects
|
||
|
||
| Project | Responsibility | Extensibility |
|
||
|---------|----------------|---------------|
|
||
| `StellaOps.Feedser.WebService` | ASP.NET Core minimal API hosting Feedser jobs, status endpoints, and scheduler. | DI-based plug-in discovery; configuration binding |
|
||
| `StellaOps.Feedser.Core` | Job orchestration, connector pipelines, merge workflows, export coordination. | `IFeedConnector`, `IExportJob`, deterministic merge policies |
|
||
| `StellaOps.Feedser.Models` | Canonical advisory DTOs and enums persisted in MongoDB and exported artefacts. | Partial classes for source-specific metadata |
|
||
| `StellaOps.Feedser.Normalization` | Version comparison, CVSS normalization, text utilities for canonicalization. | Helpers consumed by connectors/merge |
|
||
| `StellaOps.Feedser.Merge` | Precedence evaluation, alias graph maintenance, merge-event hashing. | Policy extensions via DI |
|
||
| `StellaOps.Feedser.Storage.Mongo` | Repository layer for documents, DTOs, advisories, merge events, export state. | Connection string/config via options |
|
||
| `StellaOps.Feedser.Exporter.Json` | Deterministic vuln-list JSON export pipeline. | Dependency injection for storage + plugin to host |
|
||
| `StellaOps.Feedser.Exporter.TrivyDb` | Builds Trivy DB artefacts from canonical advisories. | Optional ORAS push routines |
|
||
|
||
### 3.1 StellaOps.Feedser.WebService
|
||
|
||
* Hosts minimal API endpoints (`/health`, `/status`, `/jobs`).
|
||
* Runs the scheduler that triggers connectors and exporters according to
|
||
configured windows.
|
||
* Applies dependency-injection routines from `PluginBinaries/` at startup only
|
||
(restart-time plug-ins).
|
||
|
||
### 3.2 StellaOps.Feedser.Core
|
||
|
||
* Defines job primitives (fetch, parse, map, merge, export) used by connectors.
|
||
* Coordinates deterministic merge flows and writes `merge_event` documents.
|
||
* Provides telemetry/log scopes consumed by WebService and exporters.
|
||
|
||
### 3.3 StellaOps.Feedser.Storage.Mongo
|
||
|
||
* Persists raw documents, DTO records, canonical advisories, aliases, affected
|
||
packages, references, merge events, export state, and job leases.
|
||
* Exposes repository helpers for exporters to stream full/delta snapshots.
|
||
|
||
### 3.4 StellaOps.Feedser.Exporter.*
|
||
|
||
* `Exporter.Json` mirrors the Aqua vuln-list tree with canonical ordering.
|
||
* `Exporter.TrivyDb` builds Trivy DB Bolt archives and optional OCI bundles.
|
||
* Both exporters honour deterministic hashing and respect export cursors.
|
||
|
||
---
|
||
|
||
## 4 Source connectors
|
||
|
||
Connectors live under `StellaOps.Feedser.Source.*` and conform to the interfaces
|
||
in `StellaOps.Plugin`.
|
||
|
||
| Family | Project(s) | Notes |
|
||
|--------|------------|-------|
|
||
| Distro PSIRTs | `StellaOps.Feedser.Source.Distro.*` | Debian, Red Hat, SUSE, Ubuntu connectors with NEVRA/EVR helpers. |
|
||
| Vendor PSIRTs | `StellaOps.Feedser.Source.Vndr.*` | Adobe, Apple, Cisco, Chromium, Microsoft, Oracle, VMware. |
|
||
| Regional CERTs | `StellaOps.Feedser.Source.Cert*`, `Source.Ru.*`, `Source.Ics.*`, `Source.Kisa` | Provide enrichment metadata while preserving vendor precedence. |
|
||
| OSS ecosystems | `StellaOps.Feedser.Source.Ghsa`, `Source.Osv`, `Source.Cve`, `Source.Kev`, `Source.Acsc`, `Source.Cccs`, `Source.Jvn` | Emit SemVer/alias-rich advisories. |
|
||
|
||
Each connector ships fixtures/tests under the matching `*.Tests` project.
|
||
|
||
---
|
||
|
||
## 5 · Module Details
|
||
|
||
> _Focus on the Feedser-specific services that replace the legacy FeedMerge cron._
|
||
|
||
### 5.1 Feedser.Core
|
||
|
||
* Owns the fetch → parse → merge → export job pipeline and enforces deterministic
|
||
merge hashes (`merge_event`).
|
||
* Provides `JobSchedulerBuilder`, job coordinator, and telemetry scopes consumed
|
||
by the WebService and exporters.
|
||
|
||
### 5.2 Feedser.Storage.Mongo
|
||
|
||
* Bootstrapper creates collections/indexes (documents, dto, advisory, alias,
|
||
affected, merge_event, export_state, jobs, locks).
|
||
* Repository APIs surface full/delta advisory reads for exporters, plus
|
||
SourceState and job lease persistence.
|
||
|
||
### 5.3 Feedser.Exporter.Json / Feedser.Exporter.TrivyDb
|
||
|
||
* JSON exporter mirrors vuln-list layout with per-file digests and manifest.
|
||
* Trivy DB exporter shells or native-builds Bolt archives, optionally pushes OCI
|
||
layers, and records export cursors. Delta runs reuse unchanged blobs from the
|
||
previous full baseline, annotating `metadata.json` with `mode`, `baseExportId`,
|
||
`baseManifestDigest`, `resetBaseline`, and `delta.changedFiles[]`/`delta.removedPaths[]`.
|
||
ORAS pushes honour `publishFull` / `publishDelta`, and offline bundles respect
|
||
`includeFull` / `includeDelta` for air-gapped syncs.
|
||
|
||
### 5.4 Feedser.WebService
|
||
|
||
* Minimal API host exposing `/health`, `/ready`, `/jobs` and wiring telemetry.
|
||
* Loads restart-time plug-ins from `PluginBinaries/`, executes Mongo bootstrap,
|
||
and registers built-in connectors/exporters with the scheduler.
|
||
|
||
### 5.5 Plugin host & DI bridge
|
||
|
||
* `StellaOps.Plugin` + `StellaOps.DependencyInjection` provide the contracts and
|
||
helper routines for connectors/exporters to integrate with the WebService.
|
||
|
||
---
|
||
|
||
## 6 · Plug-ins & Agents
|
||
|
||
* **Plug-in discovery** – restart-only; the WebService enumerates
|
||
`PluginBinaries/` (or configured directories) and executes the contained
|
||
`IDependencyInjectionRoutine` implementations.
|
||
* **Connector/exporter packages** – each source/exporter can ship as a plug-in
|
||
assembly with its own options and HttpClient configuration, keeping the core
|
||
image minimal.
|
||
* **StellaOps CLI (agent)** – new `StellaOps.Cli` module that exposes
|
||
`scanner`, `scan`, and `db` verbs (via System.CommandLine 2.0) to download
|
||
scanner container bundles, install them locally, execute scans against target
|
||
directories, automatically upload results, and trigger Feedser jobs (`db
|
||
fetch/merge/export`) aligned with the SBOM-first workflow described in
|
||
`AGENTS.md`.
|
||
* **Offline Kit** – bundles Feedser plug-ins, JSON tree, Trivy DB, and export
|
||
manifests so air-gapped sites can load the latest vulnerability data without
|
||
outbound connectivity.
|
||
|
||
---
|
||
|
||
## 7 · Docker & Distribution Artefacts
|
||
|
||
| Artefact | Path / Identifier | Notes |
|
||
|----------|-------------------|-------|
|
||
| Feedser WebService image | `containers/feedser/Dockerfile` (built via CI) | Self-contained ASP.NET runtime hosting scheduler/endpoints. |
|
||
| Plugin bundle | `PluginBinaries/` | Mounted or baked-in assemblies for connectors/exporters. |
|
||
| Offline Kit tarball | Produced by CI release pipeline | Contains JSON tree, Trivy DB OCI layout, export manifest, and plug-ins. |
|
||
| Local dev compose | `scripts/` + future compose overlays | Developers can run MongoDB, Redis (optional), and WebService locally. |
|
||
|
||
---
|
||
|
||
## 8 · Performance Budget
|
||
|
||
| Scenario | Budget | Source |
|
||
|----------|--------|--------|
|
||
| Advisory upsert (large advisory) | ≤ 500 ms/advisory | `AdvisoryStorePerformanceTests` (Mongo) |
|
||
| Advisory fetch (`GetRecent`) | ≤ 200 ms/advisory | Same performance test harness |
|
||
| Advisory point lookup (`Find`) | ≤ 200 ms/advisory | Same performance test harness |
|
||
| Bulk upsert/fetch cycle | ≤ 28 s total for 30 large advisories | Same performance test harness |
|
||
| Feedser job scheduling | Deterministic cron execution via `JobSchedulerHostedService` | `StellaOps.Feedser.Core` tests |
|
||
| Trivy DB export | Deterministic digests across runs (ongoing TODO for end-to-end test) | `Exporter.TrivyDb` backlog |
|
||
|
||
Budgets are enforced in automated tests where available; outstanding TODO/DOING
|
||
items (see task boards) continue tracking gaps such as exporter determinism.
|
||
|
||
---
|
||
|
||
## 9 Testing
|
||
|
||
* Unit and integration tests live alongside each component (`*.Tests`).
|
||
* Shared fixtures come from `StellaOps.Feedser.Testing` and
|
||
`StellaOps.Feedser.Tests.Shared` (linked via `Directory.Build.props`).
|
||
* Integration suites use ephemeral MongoDB and Redis via Testcontainers to
|
||
validate end-to-end flow without external dependencies.
|
||
|
||
---
|