Initial commit (history squashed)
This commit is contained in:
Vladimir Moushkov
201
docs/08_MODULE_SPECIFICATIONS.md
Executable file
201
docs/08_MODULE_SPECIFICATIONS.md
Executable file
@@ -0,0 +1,201 @@
|
||||
# 8 · Detailed Module Specifications — **Stella Ops Feedser**
|
||||
_This document describes the Feedser service, its supporting libraries, connectors, exporters, and test assets that live in the OSS repository._
|
||||
|
||||
---
|
||||
|
||||
## 0 Scope
|
||||
|
||||
Feedser is the vulnerability ingest/merge/export subsystem of Stella Ops. It
|
||||
fetches primary advisories, normalizes and deduplicates them into MongoDB, and
|
||||
produces deterministic JSON and Trivy DB exports. This document lists the
|
||||
projects that make up that workflow, the extension points they expose, and the
|
||||
artefacts they ship.
|
||||
|
||||
---
|
||||
|
||||
## 1 Repository layout (current)
|
||||
|
||||
```text
|
||||
src/
|
||||
├─ Directory.Build.props / Directory.Build.targets
|
||||
├─ StellaOps.Plugin/
|
||||
├─ StellaOps.Feedser.Core/
|
||||
├─ StellaOps.Feedser.Core.Tests/
|
||||
├─ StellaOps.Feedser.Models/ (+ .Tests/)
|
||||
├─ StellaOps.Feedser.Normalization/ (+ .Tests/)
|
||||
├─ StellaOps.Feedser.Merge/ (+ .Tests/)
|
||||
├─ StellaOps.Feedser.Storage.Mongo/ (+ .Tests/)
|
||||
├─ StellaOps.Feedser.Exporter.Json/ (+ .Tests/)
|
||||
├─ StellaOps.Feedser.Exporter.TrivyDb/ (+ .Tests/)
|
||||
├─ StellaOps.Feedser.Source.* / StellaOps.Feedser.Source.*.Tests/
|
||||
├─ StellaOps.Feedser.Testing/
|
||||
├─ StellaOps.Feedser.Tests.Shared/
|
||||
├─ StellaOps.Feedser.WebService/ (+ .Tests/)
|
||||
├─ PluginBinaries/
|
||||
└─ StellaOps.Feedser.sln
|
||||
```
|
||||
|
||||
Each folder is a .NET project (or set of projects) referenced by
|
||||
`StellaOps.Feedser.sln`. Build assets are shared through the root
|
||||
`Directory.Build.props/targets` so conventions stay consistent.
|
||||
|
||||
---
|
||||
|
||||
## 2 Shared libraries
|
||||
|
||||
| Project | Purpose | Key extension points |
|
||||
|---------|---------|----------------------|
|
||||
| `StellaOps.Plugin` | Base contracts for connectors, exporters, and DI routines plus Cosign validation helpers. | `IFeedConnector`, `IExporterPlugin`, `IDependencyInjectionRoutine` |
|
||||
| `StellaOps.DependencyInjection` | Composable service registrations for Feedser and plug-ins. | `IDependencyInjectionRoutine` discovery |
|
||||
| `StellaOps.Feedser.Testing` | Common fixtures, builders, and harnesses for integration/unit tests. | `FeedserMongoFixture`, test builders |
|
||||
| `StellaOps.Feedser.Tests.Shared` | Shared assembly metadata and fixtures wired in via `Directory.Build.props`. | Test assembly references |
|
||||
|
||||
---
|
||||
|
||||
## 3 Core projects
|
||||
|
||||
| Project | Responsibility | Extensibility |
|
||||
|---------|----------------|---------------|
|
||||
| `StellaOps.Feedser.WebService` | ASP.NET Core minimal API hosting Feedser jobs, status endpoints, and scheduler. | DI-based plug-in discovery; configuration binding |
|
||||
| `StellaOps.Feedser.Core` | Job orchestration, connector pipelines, merge workflows, export coordination. | `IFeedConnector`, `IExportJob`, deterministic merge policies |
|
||||
| `StellaOps.Feedser.Models` | Canonical advisory DTOs and enums persisted in MongoDB and exported artefacts. | Partial classes for source-specific metadata |
|
||||
| `StellaOps.Feedser.Normalization` | Version comparison, CVSS normalization, text utilities for canonicalization. | Helpers consumed by connectors/merge |
|
||||
| `StellaOps.Feedser.Merge` | Precedence evaluation, alias graph maintenance, merge-event hashing. | Policy extensions via DI |
|
||||
| `StellaOps.Feedser.Storage.Mongo` | Repository layer for documents, DTOs, advisories, merge events, export state. | Connection string/config via options |
|
||||
| `StellaOps.Feedser.Exporter.Json` | Deterministic vuln-list JSON export pipeline. | Dependency injection for storage + plugin to host |
|
||||
| `StellaOps.Feedser.Exporter.TrivyDb` | Builds Trivy DB artefacts from canonical advisories. | Optional ORAS push routines |
|
||||
|
||||
### 3.1 StellaOps.Feedser.WebService
|
||||
|
||||
* Hosts minimal API endpoints (`/health`, `/status`, `/jobs`).
|
||||
* Runs the scheduler that triggers connectors and exporters according to
|
||||
configured windows.
|
||||
* Applies dependency-injection routines from `PluginBinaries/` at startup only
|
||||
(restart-time plug-ins).
|
||||
|
||||
### 3.2 StellaOps.Feedser.Core
|
||||
|
||||
* Defines job primitives (fetch, parse, map, merge, export) used by connectors.
|
||||
* Coordinates deterministic merge flows and writes `merge_event` documents.
|
||||
* Provides telemetry/log scopes consumed by WebService and exporters.
|
||||
|
||||
### 3.3 StellaOps.Feedser.Storage.Mongo
|
||||
|
||||
* Persists raw documents, DTO records, canonical advisories, aliases, affected
|
||||
packages, references, merge events, export state, and job leases.
|
||||
* Exposes repository helpers for exporters to stream full/delta snapshots.
|
||||
|
||||
### 3.4 StellaOps.Feedser.Exporter.*
|
||||
|
||||
* `Exporter.Json` mirrors the Aqua vuln-list tree with canonical ordering.
|
||||
* `Exporter.TrivyDb` builds Trivy DB Bolt archives and optional OCI bundles.
|
||||
* Both exporters honour deterministic hashing and respect export cursors.
|
||||
|
||||
---
|
||||
|
||||
## 4 Source connectors
|
||||
|
||||
Connectors live under `StellaOps.Feedser.Source.*` and conform to the interfaces
|
||||
in `StellaOps.Plugin`.
|
||||
|
||||
| Family | Project(s) | Notes |
|
||||
|--------|------------|-------|
|
||||
| Distro PSIRTs | `StellaOps.Feedser.Source.Distro.*` | Debian, Red Hat, SUSE, Ubuntu connectors with NEVRA/EVR helpers. |
|
||||
| Vendor PSIRTs | `StellaOps.Feedser.Source.Vndr.*` | Adobe, Apple, Cisco, Chromium, Microsoft, Oracle, VMware. |
|
||||
| Regional CERTs | `StellaOps.Feedser.Source.Cert*`, `Source.Ru.*`, `Source.Ics.*`, `Source.Kisa` | Provide enrichment metadata while preserving vendor precedence. |
|
||||
| OSS ecosystems | `StellaOps.Feedser.Source.Ghsa`, `Source.Osv`, `Source.Cve`, `Source.Kev`, `Source.Acsc`, `Source.Cccs`, `Source.Jvn` | Emit SemVer/alias-rich advisories. |
|
||||
|
||||
Each connector ships fixtures/tests under the matching `*.Tests` project.
|
||||
|
||||
---
|
||||
|
||||
## 5 · Module Details
|
||||
|
||||
> _Focus on the Feedser-specific services that replace the legacy FeedMerge cron._
|
||||
|
||||
### 5.1 Feedser.Core
|
||||
|
||||
* Owns the fetch → parse → merge → export job pipeline and enforces deterministic
|
||||
merge hashes (`merge_event`).
|
||||
* Provides `JobSchedulerBuilder`, job coordinator, and telemetry scopes consumed
|
||||
by the WebService and exporters.
|
||||
|
||||
### 5.2 Feedser.Storage.Mongo
|
||||
|
||||
* Bootstrapper creates collections/indexes (documents, dto, advisory, alias,
|
||||
affected, merge_event, export_state, jobs, locks).
|
||||
* Repository APIs surface full/delta advisory reads for exporters, plus
|
||||
SourceState and job lease persistence.
|
||||
|
||||
### 5.3 Feedser.Exporter.Json / Feedser.Exporter.TrivyDb
|
||||
|
||||
* JSON exporter mirrors vuln-list layout with per-file digests and manifest.
|
||||
* Trivy DB exporter shells or native-builds Bolt archives, optionally pushes OCI
|
||||
layers, and records export cursors.
|
||||
|
||||
### 5.4 Feedser.WebService
|
||||
|
||||
* Minimal API host exposing `/health`, `/ready`, `/jobs` and wiring telemetry.
|
||||
* Loads restart-time plug-ins from `PluginBinaries/`, executes Mongo bootstrap,
|
||||
and registers built-in connectors/exporters with the scheduler.
|
||||
|
||||
### 5.5 Plugin host & DI bridge
|
||||
|
||||
* `StellaOps.Plugin` + `StellaOps.DependencyInjection` provide the contracts and
|
||||
helper routines for connectors/exporters to integrate with the WebService.
|
||||
|
||||
---
|
||||
|
||||
## 6 · Plug-ins & Agents
|
||||
|
||||
* **Plug-in discovery** – restart-only; the WebService enumerates
|
||||
`PluginBinaries/` (or configured directories) and executes the contained
|
||||
`IDependencyInjectionRoutine` implementations.
|
||||
* **Connector/exporter packages** – each source/exporter can ship as a plug-in
|
||||
assembly with its own options and HttpClient configuration, keeping the core
|
||||
image minimal.
|
||||
* **Stella CLI (agent)** – triggers feed-related jobs (`stella db fetch/merge/export`)
|
||||
and consumes the exported JSON/Trivy DB artefacts, aligning with the SBOM-first
|
||||
workflow described in `AGENTS.md`.
|
||||
* **Offline Kit** – bundles Feedser plug-ins, JSON tree, Trivy DB, and export
|
||||
manifests so air-gapped sites can load the latest vulnerability data without
|
||||
outbound connectivity.
|
||||
|
||||
---
|
||||
|
||||
## 7 · Docker & Distribution Artefacts
|
||||
|
||||
| Artefact | Path / Identifier | Notes |
|
||||
|----------|-------------------|-------|
|
||||
| Feedser WebService image | `containers/feedser/Dockerfile` (built via CI) | Self-contained ASP.NET runtime hosting scheduler/endpoints. |
|
||||
| Plugin bundle | `PluginBinaries/` | Mounted or baked-in assemblies for connectors/exporters. |
|
||||
| Offline Kit tarball | Produced by CI release pipeline | Contains JSON tree, Trivy DB OCI layout, export manifest, and plug-ins. |
|
||||
| Local dev compose | `scripts/` + future compose overlays | Developers can run MongoDB, Redis (optional), and WebService locally. |
|
||||
|
||||
---
|
||||
|
||||
## 8 · Performance Budget
|
||||
|
||||
| Scenario | Budget | Source |
|
||||
|----------|--------|--------|
|
||||
| Advisory upsert (large advisory) | ≤ 500 ms/advisory | `AdvisoryStorePerformanceTests` (Mongo) |
|
||||
| Advisory fetch (`GetRecent`) | ≤ 200 ms/advisory | Same performance test harness |
|
||||
| Advisory point lookup (`Find`) | ≤ 200 ms/advisory | Same performance test harness |
|
||||
| Bulk upsert/fetch cycle | ≤ 28 s total for 30 large advisories | Same performance test harness |
|
||||
| Feedser job scheduling | Deterministic cron execution via `JobSchedulerHostedService` | `StellaOps.Feedser.Core` tests |
|
||||
| Trivy DB export | Deterministic digests across runs (ongoing TODO for end-to-end test) | `Exporter.TrivyDb` backlog |
|
||||
|
||||
Budgets are enforced in automated tests where available; outstanding TODO/DOING
|
||||
items (see task boards) continue tracking gaps such as exporter determinism.
|
||||
|
||||
---
|
||||
|
||||
## 9 Testing
|
||||
|
||||
* Unit and integration tests live alongside each component (`*.Tests`).
|
||||
* Shared fixtures come from `StellaOps.Feedser.Testing` and
|
||||
`StellaOps.Feedser.Tests.Shared` (linked via `Directory.Build.props`).
|
||||
* Integration suites use ephemeral MongoDB and Redis via Testcontainers to
|
||||
validate end-to-end flow without external dependencies.
|
||||
|
||||
---
|
||||
Reference in New Issue
Block a user