commit and up
This commit is contained in:
@@ -1,371 +1,201 @@
|
||||
# 8 · Detailed Module Specifications — **Stella Ops**
|
||||
_This document defines every backend/agent module that composes Stella Ops,
|
||||
their public contracts, configuration keys and extension points._
|
||||
|
||||
---
|
||||
|
||||
## 0 Scope
|
||||
|
||||
Describes **every .NET, and Angular project** that ships in the OSS Core, the plug‑in contracts they expose, and the runtime artefacts (Dockerfiles, Compose files) used to build and operate them. Commercial capabilities appear *only* as extension points.
|
||||
|
||||
---
|
||||
|
||||
## 1 Repository Layout (flat)
|
||||
|
||||
~~~text
|
||||
src/
|
||||
│ docker-compose.yml
|
||||
└─ docker-compose-library/
|
||||
│ ├─ docker-compose.no-deps.yml
|
||||
│ ├─ docker-compose.dep.redis.yml
|
||||
│ ├─ docker-compose.dep.mongo.yml
|
||||
│ ├─ docker-compose.dep.proxy.yml
|
||||
│ ├─ docker-compose.dep.repository.yml
|
||||
│ └─ docker-compose.local.yml
|
||||
└─ backend/
|
||||
│ ├─ Dockerfile
|
||||
│ ├─ StellaOps.Web/
|
||||
│ ├─ StellaOps.Common/
|
||||
│ ├─ StellaOps.Plugins/
|
||||
│ ├─ StellaOps.Configuration/
|
||||
│ ├─ StellaOps.Localization/
|
||||
│ ├─ StellaOps.TlsProvider.OpenSSL/
|
||||
│ ├─ StellaOps.TlsProvider.OpenSSL.LegacyRegional/
|
||||
│ ├─ StellaOps.TlsProvider.Plugin.CustomTlsVendor/
|
||||
│ ├─ StellaOps.VulnerabilityDatabase/
|
||||
│ ├─ StellaOps.Scheduling/
|
||||
│ ├─ StellaOps.Scheduling.SbomsRescan/
|
||||
│ ├─ StellaOps.Scheduling.MutesExpire/
|
||||
│ ├─ StellaOps.Scheduling.Plugin.CommonCveFeed/
|
||||
│ ├─ StellaOps.Scheduling.Plugin.RegionalCatalogueFeed/
|
||||
│ ├─ StellaOps.Scanners.Trivy/
|
||||
│ ├─ StellaOps.Quota/
|
||||
│ ├─ StellaOps.Reporting/
|
||||
│ ├─ StellaOps.Notifications/
|
||||
│ ├─ StellaOps.Notifications.Email/
|
||||
│ ├─ StellaOps.Notifications.Plugin.MsTeams/
|
||||
│ ├─ StellaOps.Authority/
|
||||
│ ├─ StellaOps.Authority.AD/
|
||||
│ ├─ StellaOps.Cli/
|
||||
│ └─ StellaOps.Agent.Zastava/
|
||||
└─ frontend/
|
||||
├─ Dockerfile
|
||||
├─ angular.json
|
||||
├─ stella-ops-ui/
|
||||
└─ libs/
|
||||
├─ dashboard/
|
||||
├─ scans/
|
||||
├─ settings/
|
||||
├─ core-ui/
|
||||
└─ i18n/
|
||||
~~~
|
||||
|
||||
All projects are referenced by **`StellaOps.sln`**; `dotnet publish -c Release -p:PublishSingleFile=true` builds a self‑contained **`StellaOps.Api`** binary (plug‑ins load at runtime).
|
||||
|
||||
---
|
||||
|
||||
## 2 Shared Libraries
|
||||
|
||||
| Project | Purpose | Key Interfaces |
|
||||
|---------|---------|----------------|
|
||||
| `StellaOps.Common` | Serilog sinks, Redis key helpers, DTO primitives. | `RedisKeys`, `Result<T>` |
|
||||
| `StellaOps.Plugins` | Plug‑in contracts + Cosign verification. | `IStellaPlugin`, `IScannerRunner`, `ITlsProvider`, `IScheduleJob` |
|
||||
| `StellaOps.Localization` | Loads JSON locale bundles (backend & Angular). | `ILocaleProvider`, `CultureMiddleware` |
|
||||
|
||||
Angular JSON‑bundle workflow matches the official i18n guide .
|
||||
|
||||
---
|
||||
|
||||
## 3 Core Back‑end Projects
|
||||
|
||||
| Project | Responsibility | Extensibility |
|
||||
|---------|----------------|---------------|
|
||||
| **`StellaOps.Api`** | ASP.NET host; source‑gen auto‑wires module endpoints. | Attributes `[MapRestController]`, `[MapHealth]`. |
|
||||
| **`StellaOps.Configuration`** | Bind `appsettings.json` → typed options; `/health`. | `IConfigValidator`. |
|
||||
| **`StellaOps.Quota`** | Enforces **Free‑tier quota** ({{ quota_token }}s scans/day) with early‑warning banner, 5 s soft back‑off, 60 s wait‑wall. | Swappable via `IQuotaStore` (e.g., Postgres). |
|
||||
| **`StellaOps.JwtIssuer` *(new)* | Issues, refreshes and validates **Client‑JWTs**. For offline sites it produces a 30‑day token during OUK build and again on every OUK import. | `ITokenSigner` (e.g., HSM) |
|
||||
| **`StellaOps.TlsProvider.OpenSSL`** | Default TLS suites. | New suites via `ITlsProvider` plug‑in. |
|
||||
| **`StellaOps.TlsProvider.OpenSSL.LegacyRegional`** | . | — |
|
||||
| **`StellaOps.VulnerabilityDatabase`** | Feed‑merge CLI writing Redis. | `IAdditionalFeedSource` (OSV, GHSA, regional catalogues). |
|
||||
| **`StellaOps.Scheduling`** | Hangfire host inside API . | Jobs via `IScheduleJob`. |
|
||||
| **`StellaOps.Scheduling.SbomsRescan`** | Nightly SBOM re‑scan (`0 2 * * *`). | — |
|
||||
| **`StellaOps.Scheduling.MutesExpire`** | Daily mute expiry cleanup. | — |
|
||||
| **`StellaOps.Scanners.Trivy`** | Trivy CLI for SBOM & image scans . | Other engines implement `IScannerRunner`. |
|
||||
| **`StellaOps.Reporting`** | RazorLight HTML reports. | `IReportRenderer` for SARIF, CycloneDX. |
|
||||
| **`StellaOps.Notifications`** | DI contracts for alerts. | `INotifier`. |
|
||||
| **`StellaOps.Notifications.Email`** | SMTP channel. | — |
|
||||
| **`StellaOps.Authority`** | OAuth2 / OIDC via OpenIddict 4 . | External IdPs via plug‑in. |
|
||||
| **`StellaOps.Registry`** | read‑only Docker registry for agents + SBOM‑builder | Registry v2 (nginx‑hardened) | `IRegistryProvider` |
|
||||
| **`StellaOps.MutePolicies`** | store YAML / Rego policies, validate & version | MongoDB + Redis | `IPolicyStore` |
|
||||
| **`StellaOps.Attestor`** *(TODO)*| SLSA provenance + Rekor verification | Sigstore Rekor | `IAttestor` |
|
||||
|
||||
## 3 · Module Details
|
||||
|
||||
> _Only contracts and configuration that may change in the next two quarters are shown; for stable, unchanging keys see the inline XML‑doc in the codebase._
|
||||
|
||||
### 3.1. StellaOps.Configuration
|
||||
|
||||
* **Responsibility** – parse environment variables or `appsettings.json`; expose `/health`, `/metrics`.
|
||||
* **Key extension point** – `IConfigValidator` → validate & normalise custom settings before DI builds.
|
||||
|
||||
### 3.2. StellaOps.Authority
|
||||
|
||||
* **Responsibility** – ship with OpenIddict 6, supporting *client‑credentials* and *password* grants.
|
||||
* `IIdentityProvider` plug‑in can delegate token issuance to LDAP, SAML, Keycloak …
|
||||
|
||||
|
||||
### 3.3. StellaOps.Scanners
|
||||
|
||||
* **Primary flow** – SBOM‑first; falls back to image‑unpack if SBOM absent.
|
||||
* **Multi‑Format Support** – side‑car `.sbom.type` file; auto‑detects (`SPDXID:` or `bomFormat` heuristics).
|
||||
* **Delta Layer Workflow** – `POST /layers/missing` (`SET DIFF` on Redis) responds < 20 ms; Stella CLI passes only new layers.
|
||||
* **Plug‑in contract evolution**
|
||||
|
||||
```csharp
|
||||
// current
|
||||
Task<ScanResult> RunAsync(Stream sbomJson, CancellationToken ct);
|
||||
|
||||
// v2 (preferred)
|
||||
Task<ScanResult> RunAsync(Stream sbom, SbomFormat fmt, CancellationToken ct);
|
||||
```
|
||||
|
||||
### 3.5 StellOps.Registry
|
||||
|
||||
* **Purpose** – internal, anonymous **read‑only** Docker registry to avoid GHCR / Docker Hub pulls.
|
||||
* **Deployment** – container `stellops.registry:2`; mounted volume `/var/lib/registry`; optional TLS via env vars.
|
||||
|
||||
| Key | Default | Notes |
|
||||
|----------------------------------|---------|---------------------------------|
|
||||
| `REGISTRY_READONLY` | `true` | Forces 403 on PUT, 405 on DELETE |
|
||||
| `REGISTRY_STORAGE_DELETE_ENABLED`| `false` | Immutable tags |
|
||||
|
||||
**Plug‑in contract** — `IRegistryProvider.PullAsync(string imageRef)` for mapping to Artifactory, Harbor, etc.
|
||||
|
||||
---
|
||||
|
||||
### 3.6 StellaOps.MutePolicies
|
||||
|
||||
* **Purpose** – central Policy‑as‑Code store (YAML v1 now, Rego soon).
|
||||
* **Persistence** – current live rules in Redis (`policies:active`); immutable commits in Mongo `policies_history`.
|
||||
|
||||
| REST verb | Path | Description |
|
||||
|-----------|---------------------|---------------------------|
|
||||
| `GET` | `/policy/export` | download active YAML |
|
||||
| `POST` | `/policy/import` | upload YAML / Rego file |
|
||||
| `POST` | `/policy/validate` | lint without persisting |
|
||||
|
||||
**CLI** – Stella CLI gains `--policy-file scan-policy.yaml`.
|
||||
|
||||
**Plug‑in contract** — `IPolicyStore` for GitOps back‑ends, Vault, etc.
|
||||
|
||||
---
|
||||
|
||||
### 3.7. StellaOps.Attestor *(Planned – Q1‑2026)*
|
||||
|
||||
Handles SLSA provenance docs and Rekor log verification.
|
||||
|
||||
```csharp
|
||||
public interface IAttestor {
|
||||
Task<ProvenanceDoc> CreateAsync(ImageRef img, Sbom sbom);
|
||||
Task<bool> VerifyAsync(ProvenanceDoc doc);
|
||||
}
|
||||
```
|
||||
|
||||
### 3.7. StellaOps.FeedMerge.Service
|
||||
|
||||
Nightly Hangfire job (01:00) merges NVD JSON; plug‑ins can provide ISourceFeed for OSV, GHSA, NVD, CNNVD, CNVD, ENISA and BDU feeds.
|
||||
|
||||
### 3.8. StellOps.Tls
|
||||
|
||||
Abstracts TLS stack; default OpenSSL; `ITlsProvider` lets enterprises swap in custom suites—**including SM2, where law or security requires it**.
|
||||
|
||||
### 3.9. StellaOps.Reporting
|
||||
|
||||
HTML / PDF generation via RazorLight; custom renderers via IReportRenderer.
|
||||
|
||||
### 3.10 UI
|
||||
|
||||
Angular 17 SPA; lazy‑loaded feature modules, standalone component routes for UI plug‑ins.
|
||||
|
||||
Static Go daemon / k8s DaemonSet; watches Docker/CRI‑O events; uploads SBOMs; optional enforce mode via policy plug‑in.
|
||||
|
||||
### 3.11 StellaOps.Quota — **Free‑Tier Daily Quota Service**
|
||||
|
||||
**Responsibility**
|
||||
|
||||
* Track per‑token scan count (`quota:<token>` key in Redis).
|
||||
* Reset counters at **00:00 UTC** with key TTL.
|
||||
* Inject HTTP headers
|
||||
* `X‑Stella‑Quota‑Remaining`
|
||||
* `X‑Stella‑Reset`
|
||||
* Apply adaptive throttling:
|
||||
* scans 90% of {{ quota_token }};
|
||||
* scans 10% of the max daily → UI banner flag `X‑Stella‑Quota‑Warn:true`;
|
||||
* scans ≥ {{ quota_token }} → reply is slower.
|
||||
* **Offline token awareness** — if `token.valid == false` and
|
||||
`OfflineMode == true`, return HTTP *451 ComplianceBlock* so that CLI gives a
|
||||
clear actionable error.
|
||||
* New config:
|
||||
|
||||
```json
|
||||
"Quota": {
|
||||
"OfflineGraceDays": 7 // show banner this many days before token expiry
|
||||
}
|
||||
```
|
||||
|
||||
**Interface**
|
||||
|
||||
```csharp
|
||||
public interface IQuotaService
|
||||
{
|
||||
/// <summary>Returns true when the call is allowed.</summary>
|
||||
Task<QuotaVerdict> CheckAsync(string token, CancellationToken ct);
|
||||
}
|
||||
|
||||
public readonly record struct QuotaVerdict(
|
||||
bool IsAllowed,
|
||||
int Remaining,
|
||||
DateTimeOffset ResetUtc,
|
||||
TimeSpan RetryAfter);
|
||||
```
|
||||
|
||||
**Configuration** (`appsettings.json` keys)
|
||||
|
||||
```json
|
||||
"Quota": {
|
||||
"FreeTierDailyLimit": {{ quota_token }} ,
|
||||
"WarnThreshold": 200,
|
||||
"SoftRetrySeconds": 5,
|
||||
"HardRetrySeconds": 60
|
||||
}
|
||||
```
|
||||
|
||||
**Extensibility**
|
||||
|
||||
* Override storage by providing an `IQuotaStore` plug‑in for Postgres or Mongo.
|
||||
* UI plug‑ins can subscribe to SSE `/quota/events` for custom dashboards.
|
||||
|
||||
### 3.12 StellaOps.JwtIssuer — new section
|
||||
|
||||
|API |Path| Notes|
|
||||
|-----|----|-------|
|
||||
|`POST /token/offline` | Admin‑only. | Generates a 30 d Client‑JWT for air‑gapped clusters; returns ZIP that the admin can copy to the target host.
|
||||
|
||||
*OUK hook*
|
||||
|
||||
* OUK builder calls JwtIssuer.SignOfflineToken(exp=+30d).
|
||||
* Drops client.jwt into ouk/root/.
|
||||
* Backend OUK importer places file under /var/lib/stella/tokens/.
|
||||
|
||||
---
|
||||
|
||||
## 4 · Compose / Helm Snippet (reference)
|
||||
|
||||
```yaml
|
||||
services:
|
||||
registry:
|
||||
image: stellops.registry:2
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
REGISTRY_READONLY: "true"
|
||||
volumes:
|
||||
- ./_registry:/var/lib/registry
|
||||
ports:
|
||||
- "5000:5000"
|
||||
|
||||
backend:
|
||||
image: registry.local/stellops/backend:${TAG}
|
||||
depends_on: [registry, redis]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4 Plug‑ins (sign‑required)
|
||||
|
||||
| Plug‑in | Contract | Notes |
|
||||
|---------|----------|-------|
|
||||
| `StellaOps.Notifications.Plugin.MsTeams` | `INotifier` | Sends cards to Teams webhooks. |
|
||||
| `StellaOps.Authority.AD` | `IIdentityProvider` | LDAP/Active‑Directory token issue. |
|
||||
| `StellaOps.Scheduling.Plugin.CommonCveFeed` | `IScheduleJob` | Merges OSV & NVD JSON hourly . |
|
||||
| `StellaOps.Scheduling.Plugin.RegionalCatalogueFeed` | `IScheduleJob` | Imports NVD 2.0, CNNVD, CNVD, ENISA, JVN and BDU XML daily. |
|
||||
| `StellaOps.TlsProvider.Plugin.CustomTlsVendor` | `ITlsProvider` | Binds regional specific shared libs. |
|
||||
|
||||
Cosign signatures are mandatory; loader rejects unsigned DLLs when `DisableUnsigned=false`.
|
||||
|
||||
---
|
||||
|
||||
## 5 Agents
|
||||
|
||||
### 5.1 `StellaOps.Cli`
|
||||
|
||||
Distroless CLI;
|
||||
Returns exit‑code 1 on policy violation, enabling CI blocking.
|
||||
* **Role** – CI helper: Build SBOM, call `/scan`, exit non‑zero on high severity.
|
||||
* **Flags** – `--engine`, `--threshold`, `--registry-pull-token`, `--pdf-out`, `--delta`, `--sbom-type`, `--policy-file.`.
|
||||
* **Auth** – OAuth2 *scanner* scope.
|
||||
|
||||
### 5.2 `StellaOps.Agent.Zastava`
|
||||
|
||||
* **Role** – Passive container inventory → uploads SBOMs via `/agent/sbom`.
|
||||
* **Modes** – `off`, `inventory` (Core default).
|
||||
* No kernel driver (unlike Falco) .
|
||||
|
||||
---
|
||||
|
||||
## 6 Angular Front‑end
|
||||
|
||||
| Package | Path | Feature | Lazy |
|
||||
|---------|------|---------|------|
|
||||
| **App** | `frontend/stella-ops-ui/` | Shell, auth guards. | — |
|
||||
| `dashboard` | `libs/dashboard/` | Live metrics tiles. | ✔ |
|
||||
| `scans` | `libs/scans/` | List, detail, mute, diff. | ✔ |
|
||||
| `settings` | `libs/settings/` | Feed cron, workers, TLS switch. | ✔ |
|
||||
| `core-ui` | `libs/core-ui/` | Tailwind components. | — |
|
||||
| `i18n` | `libs/i18n/` | Runtime locale switch, pipe. | — |
|
||||
|
||||
Lazy loading of workspace libs follows Nx/Angular guidance .
|
||||
|
||||
---
|
||||
|
||||
## 7 Docker Artefacts
|
||||
|
||||
### 7.1 Dockerfiles
|
||||
|
||||
* **`backend/Dockerfile`** – multi‑stage .NET {{ dotnet }}; single‑file publish; distroless runtime .
|
||||
* **`frontend/Dockerfile`** – Node 20 build → Nginx static serve.
|
||||
* Every plug‑in repo may include its own Dockerfile when shipping side‑cars (e.g., custom scanner).
|
||||
|
||||
### 7.2 Compose Stacks
|
||||
|
||||
* **`docker-compose.yml`**
|
||||
* Extends above with Redis 7 and Mongo 7 for small on‑prem installs.
|
||||
|
||||
* **`docker-compose.no-deps.yml`**
|
||||
* backend, frontend, Trivy, Maven proxy.
|
||||
* Assumes external Redis & Mongo.
|
||||
|
||||
* **`docker-compose.local.yml`**
|
||||
* Build images from local source and bring up backend, frontend, Redis, Mongo, Trivy, Maven proxy for dev‑loop.
|
||||
|
||||
Docker Compose override precedence matches official docs .
|
||||
|
||||
---
|
||||
|
||||
## 8 Performance Budget
|
||||
|
||||
| Flow | P95 target | Bottleneck |
|
||||
|------|-----------:|-----------|
|
||||
| SBOM fast‑path | ≤ 5 s | Redis queue depth (keep P99 < 1 ms) |
|
||||
| Image‑unpack | ≤ 10 s | Trivy layer unpack. |
|
||||
| Nightly re‑scan | 80 SBOM/s | Runner CPU. |
|
||||
|
||||
---
|
||||
|
||||
## Change Log
|
||||
|
||||
| Version | Date | Notes |
|
||||
|---------|------|-------|
|
||||
| **v2.2** | 2025‑07‑11 | Flat layout; stella‑ops‑ui naming; Dockerfiles & 3 Compose stacks; agents and localisation library. |
|
||||
| v2.1 | 2025‑07‑11 | First flat‑structure draft. |
|
||||
|
||||
*(End of Module Specifications v2.2‑core)*
|
||||
# 8 · Detailed Module Specifications — **Stella Ops Feedser**
|
||||
_This document describes the Feedser service, its supporting libraries, connectors, exporters, and test assets that live in the OSS repository._
|
||||
|
||||
---
|
||||
|
||||
## 0 Scope
|
||||
|
||||
Feedser is the vulnerability ingest/merge/export subsystem of Stella Ops. It
|
||||
fetches primary advisories, normalizes and deduplicates them into MongoDB, and
|
||||
produces deterministic JSON and Trivy DB exports. This document lists the
|
||||
projects that make up that workflow, the extension points they expose, and the
|
||||
artefacts they ship.
|
||||
|
||||
---
|
||||
|
||||
## 1 Repository layout (current)
|
||||
|
||||
```text
|
||||
src/
|
||||
├─ Directory.Build.props / Directory.Build.targets
|
||||
├─ StellaOps.Plugin/
|
||||
├─ StellaOps.Feedser.Core/
|
||||
├─ StellaOps.Feedser.Core.Tests/
|
||||
├─ StellaOps.Feedser.Models/ (+ .Tests/)
|
||||
├─ StellaOps.Feedser.Normalization/ (+ .Tests/)
|
||||
├─ StellaOps.Feedser.Merge/ (+ .Tests/)
|
||||
├─ StellaOps.Feedser.Storage.Mongo/ (+ .Tests/)
|
||||
├─ StellaOps.Feedser.Exporter.Json/ (+ .Tests/)
|
||||
├─ StellaOps.Feedser.Exporter.TrivyDb/ (+ .Tests/)
|
||||
├─ StellaOps.Feedser.Source.* / StellaOps.Feedser.Source.*.Tests/
|
||||
├─ StellaOps.Feedser.Testing/
|
||||
├─ StellaOps.Feedser.Tests.Shared/
|
||||
├─ StellaOps.Feedser.WebService/ (+ .Tests/)
|
||||
├─ PluginBinaries/
|
||||
└─ StellaOps.Feedser.sln
|
||||
```
|
||||
|
||||
Each folder is a .NET project (or set of projects) referenced by
|
||||
`StellaOps.Feedser.sln`. Build assets are shared through the root
|
||||
`Directory.Build.props/targets` so conventions stay consistent.
|
||||
|
||||
---
|
||||
|
||||
## 2 Shared libraries
|
||||
|
||||
| Project | Purpose | Key extension points |
|
||||
|---------|---------|----------------------|
|
||||
| `StellaOps.Plugin` | Base contracts for connectors, exporters, and DI routines plus Cosign validation helpers. | `IFeedConnector`, `IExporterPlugin`, `IDependencyInjectionRoutine` |
|
||||
| `StellaOps.DependencyInjection` | Composable service registrations for Feedser and plug-ins. | `IDependencyInjectionRoutine` discovery |
|
||||
| `StellaOps.Feedser.Testing` | Common fixtures, builders, and harnesses for integration/unit tests. | `FeedserMongoFixture`, test builders |
|
||||
| `StellaOps.Feedser.Tests.Shared` | Shared assembly metadata and fixtures wired in via `Directory.Build.props`. | Test assembly references |
|
||||
|
||||
---
|
||||
|
||||
## 3 Core projects
|
||||
|
||||
| Project | Responsibility | Extensibility |
|
||||
|---------|----------------|---------------|
|
||||
| `StellaOps.Feedser.WebService` | ASP.NET Core minimal API hosting Feedser jobs, status endpoints, and scheduler. | DI-based plug-in discovery; configuration binding |
|
||||
| `StellaOps.Feedser.Core` | Job orchestration, connector pipelines, merge workflows, export coordination. | `IFeedConnector`, `IExportJob`, deterministic merge policies |
|
||||
| `StellaOps.Feedser.Models` | Canonical advisory DTOs and enums persisted in MongoDB and exported artefacts. | Partial classes for source-specific metadata |
|
||||
| `StellaOps.Feedser.Normalization` | Version comparison, CVSS normalization, text utilities for canonicalization. | Helpers consumed by connectors/merge |
|
||||
| `StellaOps.Feedser.Merge` | Precedence evaluation, alias graph maintenance, merge-event hashing. | Policy extensions via DI |
|
||||
| `StellaOps.Feedser.Storage.Mongo` | Repository layer for documents, DTOs, advisories, merge events, export state. | Connection string/config via options |
|
||||
| `StellaOps.Feedser.Exporter.Json` | Deterministic vuln-list JSON export pipeline. | Dependency injection for storage + plugin to host |
|
||||
| `StellaOps.Feedser.Exporter.TrivyDb` | Builds Trivy DB artefacts from canonical advisories. | Optional ORAS push routines |
|
||||
|
||||
### 3.1 StellaOps.Feedser.WebService
|
||||
|
||||
* Hosts minimal API endpoints (`/health`, `/status`, `/jobs`).
|
||||
* Runs the scheduler that triggers connectors and exporters according to
|
||||
configured windows.
|
||||
* Applies dependency-injection routines from `PluginBinaries/` at startup only
|
||||
(restart-time plug-ins).
|
||||
|
||||
### 3.2 StellaOps.Feedser.Core
|
||||
|
||||
* Defines job primitives (fetch, parse, map, merge, export) used by connectors.
|
||||
* Coordinates deterministic merge flows and writes `merge_event` documents.
|
||||
* Provides telemetry/log scopes consumed by WebService and exporters.
|
||||
|
||||
### 3.3 StellaOps.Feedser.Storage.Mongo
|
||||
|
||||
* Persists raw documents, DTO records, canonical advisories, aliases, affected
|
||||
packages, references, merge events, export state, and job leases.
|
||||
* Exposes repository helpers for exporters to stream full/delta snapshots.
|
||||
|
||||
### 3.4 StellaOps.Feedser.Exporter.*
|
||||
|
||||
* `Exporter.Json` mirrors the Aqua vuln-list tree with canonical ordering.
|
||||
* `Exporter.TrivyDb` builds Trivy DB Bolt archives and optional OCI bundles.
|
||||
* Both exporters honour deterministic hashing and respect export cursors.
|
||||
|
||||
---
|
||||
|
||||
## 4 Source connectors
|
||||
|
||||
Connectors live under `StellaOps.Feedser.Source.*` and conform to the interfaces
|
||||
in `StellaOps.Plugin`.
|
||||
|
||||
| Family | Project(s) | Notes |
|
||||
|--------|------------|-------|
|
||||
| Distro PSIRTs | `StellaOps.Feedser.Source.Distro.*` | Debian, Red Hat, SUSE, Ubuntu connectors with NEVRA/EVR helpers. |
|
||||
| Vendor PSIRTs | `StellaOps.Feedser.Source.Vndr.*` | Adobe, Apple, Cisco, Chromium, Microsoft, Oracle, VMware. |
|
||||
| Regional CERTs | `StellaOps.Feedser.Source.Cert*`, `Source.Ru.*`, `Source.Ics.*`, `Source.Kisa` | Provide enrichment metadata while preserving vendor precedence. |
|
||||
| OSS ecosystems | `StellaOps.Feedser.Source.Ghsa`, `Source.Osv`, `Source.Cve`, `Source.Kev`, `Source.Acsc`, `Source.Cccs`, `Source.Jvn` | Emit SemVer/alias-rich advisories. |
|
||||
|
||||
Each connector ships fixtures/tests under the matching `*.Tests` project.
|
||||
|
||||
---
|
||||
|
||||
## 5 · Module Details
|
||||
|
||||
> _Focus on the Feedser-specific services that replace the legacy FeedMerge cron._
|
||||
|
||||
### 5.1 Feedser.Core
|
||||
|
||||
* Owns the fetch → parse → merge → export job pipeline and enforces deterministic
|
||||
merge hashes (`merge_event`).
|
||||
* Provides `JobSchedulerBuilder`, job coordinator, and telemetry scopes consumed
|
||||
by the WebService and exporters.
|
||||
|
||||
### 5.2 Feedser.Storage.Mongo
|
||||
|
||||
* Bootstrapper creates collections/indexes (documents, dto, advisory, alias,
|
||||
affected, merge_event, export_state, jobs, locks).
|
||||
* Repository APIs surface full/delta advisory reads for exporters, plus
|
||||
SourceState and job lease persistence.
|
||||
|
||||
### 5.3 Feedser.Exporter.Json / Feedser.Exporter.TrivyDb
|
||||
|
||||
* JSON exporter mirrors vuln-list layout with per-file digests and manifest.
|
||||
* Trivy DB exporter shells or native-builds Bolt archives, optionally pushes OCI
|
||||
layers, and records export cursors.
|
||||
|
||||
### 5.4 Feedser.WebService
|
||||
|
||||
* Minimal API host exposing `/health`, `/ready`, `/jobs` and wiring telemetry.
|
||||
* Loads restart-time plug-ins from `PluginBinaries/`, executes Mongo bootstrap,
|
||||
and registers built-in connectors/exporters with the scheduler.
|
||||
|
||||
### 5.5 Plugin host & DI bridge
|
||||
|
||||
* `StellaOps.Plugin` + `StellaOps.DependencyInjection` provide the contracts and
|
||||
helper routines for connectors/exporters to integrate with the WebService.
|
||||
|
||||
---
|
||||
|
||||
## 6 · Plug-ins & Agents
|
||||
|
||||
* **Plug-in discovery** – restart-only; the WebService enumerates
|
||||
`PluginBinaries/` (or configured directories) and executes the contained
|
||||
`IDependencyInjectionRoutine` implementations.
|
||||
* **Connector/exporter packages** – each source/exporter can ship as a plug-in
|
||||
assembly with its own options and HttpClient configuration, keeping the core
|
||||
image minimal.
|
||||
* **Stella CLI (agent)** – triggers feed-related jobs (`stella db fetch/merge/export`)
|
||||
and consumes the exported JSON/Trivy DB artefacts, aligning with the SBOM-first
|
||||
workflow described in `AGENTS.md`.
|
||||
* **Offline Kit** – bundles Feedser plug-ins, JSON tree, Trivy DB, and export
|
||||
manifests so air-gapped sites can load the latest vulnerability data without
|
||||
outbound connectivity.
|
||||
|
||||
---
|
||||
|
||||
## 7 · Docker & Distribution Artefacts
|
||||
|
||||
| Artefact | Path / Identifier | Notes |
|
||||
|----------|-------------------|-------|
|
||||
| Feedser WebService image | `containers/feedser/Dockerfile` (built via CI) | Self-contained ASP.NET runtime hosting scheduler/endpoints. |
|
||||
| Plugin bundle | `PluginBinaries/` | Mounted or baked-in assemblies for connectors/exporters. |
|
||||
| Offline Kit tarball | Produced by CI release pipeline | Contains JSON tree, Trivy DB OCI layout, export manifest, and plug-ins. |
|
||||
| Local dev compose | `scripts/` + future compose overlays | Developers can run MongoDB, Redis (optional), and WebService locally. |
|
||||
|
||||
---
|
||||
|
||||
## 8 · Performance Budget
|
||||
|
||||
| Scenario | Budget | Source |
|
||||
|----------|--------|--------|
|
||||
| Advisory upsert (large advisory) | ≤ 500 ms/advisory | `AdvisoryStorePerformanceTests` (Mongo) |
|
||||
| Advisory fetch (`GetRecent`) | ≤ 200 ms/advisory | Same performance test harness |
|
||||
| Advisory point lookup (`Find`) | ≤ 200 ms/advisory | Same performance test harness |
|
||||
| Bulk upsert/fetch cycle | ≤ 28 s total for 30 large advisories | Same performance test harness |
|
||||
| Feedser job scheduling | Deterministic cron execution via `JobSchedulerHostedService` | `StellaOps.Feedser.Core` tests |
|
||||
| Trivy DB export | Deterministic digests across runs (ongoing TODO for end-to-end test) | `Exporter.TrivyDb` backlog |
|
||||
|
||||
Budgets are enforced in automated tests where available; outstanding TODO/DOING
|
||||
items (see task boards) continue tracking gaps such as exporter determinism.
|
||||
|
||||
---
|
||||
|
||||
## 9 Testing
|
||||
|
||||
* Unit and integration tests live alongside each component (`*.Tests`).
|
||||
* Shared fixtures come from `StellaOps.Feedser.Testing` and
|
||||
`StellaOps.Feedser.Tests.Shared` (linked via `Directory.Build.props`).
|
||||
* Integration suites use ephemeral MongoDB and Redis via Testcontainers to
|
||||
validate end-to-end flow without external dependencies.
|
||||
|
||||
---
|
||||
|
||||
Reference in New Issue
Block a user