# 8 · Detailed Module Specifications — **Stella Ops Feedser**
_This document describes the Feedser service, its supporting libraries, connectors, exporters, and test assets that live in the OSS repository._

---

## 0 Scope  

Feedser is the vulnerability ingest/merge/export subsystem of Stella Ops. It
fetches primary advisories, normalizes and deduplicates them into MongoDB, and
produces deterministic JSON and Trivy DB exports. This document lists the
projects that make up that workflow, the extension points they expose, and the
artefacts they ship.

---

## 1 Repository layout (current)  

```text
src/
 ├─ Directory.Build.props / Directory.Build.targets
 ├─ StellaOps.Plugin/
 ├─ StellaOps.Feedser.Core/
 ├─ StellaOps.Feedser.Core.Tests/
 ├─ StellaOps.Feedser.Models/ (+ .Tests/)
 ├─ StellaOps.Feedser.Normalization/ (+ .Tests/)
 ├─ StellaOps.Feedser.Merge/ (+ .Tests/)
 ├─ StellaOps.Feedser.Storage.Mongo/ (+ .Tests/)
 ├─ StellaOps.Feedser.Exporter.Json/ (+ .Tests/)
 ├─ StellaOps.Feedser.Exporter.TrivyDb/ (+ .Tests/)
 ├─ StellaOps.Feedser.Source.* / StellaOps.Feedser.Source.*.Tests/
 ├─ StellaOps.Feedser.Testing/
 ├─ StellaOps.Feedser.Tests.Shared/
 ├─ StellaOps.Feedser.WebService/ (+ .Tests/)
 ├─ PluginBinaries/
 └─ StellaOps.Feedser.sln
```

Each folder is a .NET project (or set of projects) referenced by
`StellaOps.Feedser.sln`. Build assets are shared through the root
`Directory.Build.props/targets` so conventions stay consistent.

---

## 2 Shared libraries  

| Project | Purpose | Key extension points |
|---------|---------|----------------------|
| `StellaOps.Plugin` | Base contracts for connectors, exporters, and DI routines plus Cosign validation helpers. | `IFeedConnector`, `IExporterPlugin`, `IDependencyInjectionRoutine` |
| `StellaOps.DependencyInjection` | Composable service registrations for Feedser and plug-ins. | `IDependencyInjectionRoutine` discovery |
| `StellaOps.Feedser.Testing` | Common fixtures, builders, and harnesses for integration/unit tests. | `FeedserMongoFixture`, test builders |
| `StellaOps.Feedser.Tests.Shared` | Shared assembly metadata and fixtures wired in via `Directory.Build.props`. | Test assembly references |

---

## 3 Core projects  

| Project | Responsibility | Extensibility |
|---------|----------------|---------------|
| `StellaOps.Feedser.WebService` | ASP.NET Core minimal API hosting Feedser jobs, status endpoints, and scheduler. | DI-based plug-in discovery; configuration binding | 
| `StellaOps.Feedser.Core` | Job orchestration, connector pipelines, merge workflows, export coordination. | `IFeedConnector`, `IExportJob`, deterministic merge policies |
| `StellaOps.Feedser.Models` | Canonical advisory DTOs and enums persisted in MongoDB and exported artefacts. | Partial classes for source-specific metadata |
| `StellaOps.Feedser.Normalization` | Version comparison, CVSS normalization, text utilities for canonicalization. | Helpers consumed by connectors/merge |
| `StellaOps.Feedser.Merge` | Precedence evaluation, alias graph maintenance, merge-event hashing. | Policy extensions via DI |
| `StellaOps.Feedser.Storage.Mongo` | Repository layer for documents, DTOs, advisories, merge events, export state. | Connection string/config via options |
| `StellaOps.Feedser.Exporter.Json` | Deterministic vuln-list JSON export pipeline. | Dependency injection for storage + plugin to host | 
| `StellaOps.Feedser.Exporter.TrivyDb` | Builds Trivy DB artefacts from canonical advisories. | Optional ORAS push routines |

### 3.1 StellaOps.Feedser.WebService  

* Hosts minimal API endpoints (`/health`, `/status`, `/jobs`).
* Runs the scheduler that triggers connectors and exporters according to
  configured windows.
* Applies dependency-injection routines from `PluginBinaries/` at startup only
  (restart-time plug-ins).

### 3.2 StellaOps.Feedser.Core  

* Defines job primitives (fetch, parse, map, merge, export) used by connectors.
* Coordinates deterministic merge flows and writes `merge_event` documents.
* Provides telemetry/log scopes consumed by WebService and exporters.

### 3.3 StellaOps.Feedser.Storage.Mongo  

* Persists raw documents, DTO records, canonical advisories, aliases, affected
  packages, references, merge events, export state, and job leases.
* Exposes repository helpers for exporters to stream full/delta snapshots.

### 3.4 StellaOps.Feedser.Exporter.*  

* `Exporter.Json` mirrors the Aqua vuln-list tree with canonical ordering.
* `Exporter.TrivyDb` builds Trivy DB Bolt archives and optional OCI bundles.
* Both exporters honour deterministic hashing and respect export cursors.

---

## 4 Source connectors  

Connectors live under `StellaOps.Feedser.Source.*` and conform to the interfaces
in `StellaOps.Plugin`.

| Family | Project(s) | Notes |
|--------|------------|-------|
| Distro PSIRTs | `StellaOps.Feedser.Source.Distro.*` | Debian, Red Hat, SUSE, Ubuntu connectors with NEVRA/EVR helpers. |
| Vendor PSIRTs | `StellaOps.Feedser.Source.Vndr.*` | Adobe, Apple, Cisco, Chromium, Microsoft, Oracle, VMware. |
| Regional CERTs | `StellaOps.Feedser.Source.Cert*`, `Source.Ru.*`, `Source.Ics.*`, `Source.Kisa` | Provide enrichment metadata while preserving vendor precedence. |
| OSS ecosystems | `StellaOps.Feedser.Source.Ghsa`, `Source.Osv`, `Source.Cve`, `Source.Kev`, `Source.Acsc`, `Source.Cccs`, `Source.Jvn` | Emit SemVer/alias-rich advisories. |

Each connector ships fixtures/tests under the matching `*.Tests` project.

---

## 5 · Module Details  

> _Focus on the Feedser-specific services that replace the legacy FeedMerge cron._

### 5.1 Feedser.Core  

* Owns the fetch → parse → merge → export job pipeline and enforces deterministic
  merge hashes (`merge_event`).
* Provides `JobSchedulerBuilder`, job coordinator, and telemetry scopes consumed
  by the WebService and exporters.

### 5.2 Feedser.Storage.Mongo  

* Bootstrapper creates collections/indexes (documents, dto, advisory, alias,
  affected, merge_event, export_state, jobs, locks).
* Repository APIs surface full/delta advisory reads for exporters, plus
  SourceState and job lease persistence.

### 5.3 Feedser.Exporter.Json / Feedser.Exporter.TrivyDb  

* JSON exporter mirrors vuln-list layout with per-file digests and manifest.
* Trivy DB exporter shells or native-builds Bolt archives, optionally pushes OCI
  layers, and records export cursors. Delta runs reuse unchanged blobs from the
  previous full baseline, annotating `metadata.json` with `mode`, `baseExportId`,
  `baseManifestDigest`, `resetBaseline`, and `delta.changedFiles[]`/`delta.removedPaths[]`.
  ORAS pushes honour `publishFull` / `publishDelta`, and offline bundles respect
  `includeFull` / `includeDelta` for air-gapped syncs.

### 5.4 Feedser.WebService  

* Minimal API host exposing `/health`, `/ready`, `/jobs` and wiring telemetry.
* Loads restart-time plug-ins from `PluginBinaries/`, executes Mongo bootstrap,
  and registers built-in connectors/exporters with the scheduler.

### 5.5 Plugin host & DI bridge  

* `StellaOps.Plugin` + `StellaOps.DependencyInjection` provide the contracts and
  helper routines for connectors/exporters to integrate with the WebService.

---

## 6 · Plug-ins & Agents  

* **Plug-in discovery** – restart-only; the WebService enumerates
  `PluginBinaries/` (or configured directories) and executes the contained
  `IDependencyInjectionRoutine` implementations.
* **Connector/exporter packages** – each source/exporter can ship as a plug-in
  assembly with its own options and HttpClient configuration, keeping the core
  image minimal.
* **StellaOps CLI (agent)** – new `StellaOps.Cli` module that exposes
  `scanner`, `scan`, and `db` verbs (via System.CommandLine 2.0) to download
  scanner container bundles, install them locally, execute scans against target
  directories, automatically upload results, and trigger Feedser jobs (`db
  fetch/merge/export`) aligned with the SBOM-first workflow described in
  `AGENTS.md`.
* **Offline Kit** – bundles Feedser plug-ins, JSON tree, Trivy DB, and export
  manifests so air-gapped sites can load the latest vulnerability data without
  outbound connectivity.

---

## 7 · Docker & Distribution Artefacts  

| Artefact | Path / Identifier | Notes |
|----------|-------------------|-------|
| Feedser WebService image | `containers/feedser/Dockerfile` (built via CI) | Self-contained ASP.NET runtime hosting scheduler/endpoints. |
| Plugin bundle | `PluginBinaries/` | Mounted or baked-in assemblies for connectors/exporters. |
| Offline Kit tarball | Produced by CI release pipeline | Contains JSON tree, Trivy DB OCI layout, export manifest, and plug-ins. |
| Local dev compose | `scripts/` + future compose overlays | Developers can run MongoDB, Redis (optional), and WebService locally. |

---

## 8 · Performance Budget  

| Scenario | Budget | Source |
|----------|--------|--------|
| Advisory upsert (large advisory) | ≤ 500 ms/advisory | `AdvisoryStorePerformanceTests` (Mongo) |
| Advisory fetch (`GetRecent`) | ≤ 200 ms/advisory | Same performance test harness |
| Advisory point lookup (`Find`) | ≤ 200 ms/advisory | Same performance test harness |
| Bulk upsert/fetch cycle | ≤ 28 s total for 30 large advisories | Same performance test harness |
| Feedser job scheduling | Deterministic cron execution via `JobSchedulerHostedService` | `StellaOps.Feedser.Core` tests |
| Trivy DB export | Deterministic digests across runs (ongoing TODO for end-to-end test) | `Exporter.TrivyDb` backlog |

Budgets are enforced in automated tests where available; outstanding TODO/DOING
items (see task boards) continue tracking gaps such as exporter determinism.

---

## 9 Testing  

* Unit and integration tests live alongside each component (`*.Tests`).
* Shared fixtures come from `StellaOps.Feedser.Testing` and
  `StellaOps.Feedser.Tests.Shared` (linked via `Directory.Build.props`).
* Integration suites use ephemeral MongoDB and Redis via Testcontainers to
  validate end-to-end flow without external dependencies.

---
