Add tests for SBOM generation determinism across multiple formats
- Created `StellaOps.TestKit.Tests` project for unit tests related to determinism. - Implemented `DeterminismManifestTests` to validate deterministic output for canonical bytes and strings, file read/write operations, and error handling for invalid schema versions. - Added `SbomDeterminismTests` to ensure identical inputs produce consistent SBOMs across SPDX 3.0.1 and CycloneDX 1.6/1.7 formats, including parallel execution tests. - Updated project references in `StellaOps.Integration.Determinism` to include the new determinism testing library.
This commit is contained in:
master
StellaOps Bot
@@ -1,133 +1,62 @@
|
||||
# Stella Ops — High‑Level Architecture
|
||||
# Architecture Overview (High-Level)
|
||||
|
||||
<!--
|
||||
Use constants injected at build:
|
||||
{{ dotnet }} = "10 LTS"
|
||||
{{ angular }} = "20"
|
||||
-->
|
||||
This document is a high-level orientation to StellaOps: what components exist, how they fit together, and what "offline-first + deterministic + evidence-linked decisions" means in practice.
|
||||
|
||||
This document offers a birds‑eye view of how the major components interact,
|
||||
why the system leans *monolith‑plus‑plug‑ins*, and where extension points live.
|
||||
For the full reference map (services, boundaries, detailed flows), see `docs/07_HIGH_LEVEL_ARCHITECTURE.md`.
|
||||
|
||||
> For a *timeline* of when features arrive, see the public
|
||||
> [road‑map](/roadmap/) — no version details are repeated here.
|
||||
## Guiding Principles
|
||||
|
||||
---
|
||||
- **SBOM-first:** scan and reason over SBOMs; fall back to unpacking only when needed.
|
||||
- **Deterministic replay:** the same inputs yield the same outputs (stable ordering, canonical hashing, UTC timestamps).
|
||||
- **Evidence-linked decisions:** policy decisions link back to specific evidence artifacts (SBOM slices, advisory/VEX observations, reachability proofs, attestations).
|
||||
- **Aggregation-not-merge:** upstream advisories and VEX are stored and exposed with provenance; conflicts are visible, not silently collapsed.
|
||||
- **Offline-first:** the same workflow runs connected or air-gapped via Offline Kit snapshots and signed bundles.
|
||||
|
||||
## 0 · Guiding principles
|
||||
## System Map (What Runs)
|
||||
|
||||
| Principle | Rationale |
|
||||
|-----------|-----------|
|
||||
| **SBOM‑first** | Scan existing CycloneDX/SPDX if present; fall back to layer unpack. |
|
||||
| **Δ‑processing** | Re‑analyse only changed layers; reduces P95 warm path to \< 5 s. |
|
||||
| **All‑managed code** | Entire stack is 100 % managed (.NET / TypeScript); no `unsafe` blocks or native extensions — eases review and reproducible builds. |
|
||||
| **Restart‑time plug‑ins** | Avoids the attack surface of runtime DLL injection; still allows custom scanners & exporters. |
|
||||
| **Sovereign‑by‑design** | No mandatory outbound traffic; Offline Kit distributes feeds. |
|
||||
At a high level, StellaOps is a set of services grouped by responsibility:
|
||||
|
||||
---
|
||||
- **Identity and authorization:** Authority (OIDC/OAuth2, scopes/tenancy)
|
||||
- **Scanning and SBOM:** Scanner WebService + Worker (facts generation)
|
||||
- **Advisories:** Concelier (ingest/normalize/export vulnerability sources)
|
||||
- **VEX:** Excititor + VEX Lens (VEX observations/linksets and exploration)
|
||||
- **Decisioning:** Policy Engine surfaces (lattice-style explainable policy)
|
||||
- **Signing and transparency:** Signer + Attestor (DSSE/in-toto and optional transparency)
|
||||
- **Orchestration and delivery:** Scheduler, Notify, Export Center
|
||||
- **Console:** Web UI for operators and auditors
|
||||
|
||||
## 1 · Module graph
|
||||
## Infrastructure (What Is Required)
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
A(API Gateway)
|
||||
B1(Scanner Core<br/>.NET latest LTS)
|
||||
B2(Concelier service\n(vuln ingest/merge/export))
|
||||
B3(Policy Engine OPA)
|
||||
C1(Redis 7)
|
||||
C2(PostgreSQL 16)
|
||||
D(UI SPA<br/>Angular latest version)
|
||||
A -->|gRPC| B1
|
||||
B1 -->|async| B2
|
||||
B1 -->|OPA| B3
|
||||
B1 --> C1
|
||||
B1 --> C2
|
||||
A -->|REST/WS| D
|
||||
```
|
||||
**Required**
|
||||
|
||||
---
|
||||
- **PostgreSQL:** canonical persistent store for module schemas.
|
||||
- **Valkey:** Redis-compatible cache/streams and DPoP nonce store.
|
||||
- **RustFS (or equivalent S3-compatible store):** object storage for artifacts, bundles, and evidence.
|
||||
|
||||
## 2 · Key components
|
||||
**Optional (deployment-dependent)**
|
||||
|
||||
| Component | Language / tech | Responsibility |
|
||||
| ---------------------------- | --------------------- | ---------------------------------------------------- |
|
||||
| **API Gateway** | ASP.NET Minimal API | Auth (JWT), quotas, request routing |
|
||||
| **Scanner Core** | C# 12, Polly | Layer diffing, SBOM generation, vuln correlation |
|
||||
| **Concelier (vulnerability ingest/merge/export service)** | C# source-gen workers | Consolidate NVD + regional CVE feeds into the canonical PostgreSQL store and drive JSON / Trivy DB exports |
|
||||
| **Policy Engine** | OPA (Rego) | admission decisions, custom org rules |
|
||||
| **Redis 7** | Key‑DB compatible | LRU cache, quota counters |
|
||||
| **PostgreSQL 16** | JSONB storage | SBOM & findings storage |
|
||||
| **Angular {{ angular }} UI** | RxJS, Tailwind | Dashboard, reports, admin UX |
|
||||
- **NATS JetStream:** optional messaging transport in some deployments.
|
||||
- **Transparency log services:** Rekor mirror (and CA services) when transparency is enabled.
|
||||
|
||||
---
|
||||
## End-to-End Flow (Typical)
|
||||
|
||||
## 3 · Plug‑in system
|
||||
1. **Ingest evidence sources:** Concelier and Excititor ingest upstream advisories/VEX into immutable observations with provenance.
|
||||
2. **Scan:** Scanner accepts an SBOM or image reference, produces scan facts and evidence artifacts.
|
||||
3. **Decide:** Policy evaluation merges scan facts with advisory/VEX evidence to produce an explainable verdict.
|
||||
4. **Seal:** Signer/Attestor wrap outputs into signed bundles (DSSE/in-toto) and optionally anchor in transparency logs.
|
||||
5. **Export and notify:** Export Center produces portable evidence bundles and Offline Kit material; Notify delivers digests/incidents.
|
||||
6. **Operate:** Console exposes triage, explainability, verification, and governance workflows.
|
||||
|
||||
* Discovered once at start‑up from `/opt/stella/plugins/**`.
|
||||
* Runs under Linux user `stella‑plugin` (UID 1001).
|
||||
* Extension points:
|
||||
## Extension Points (Where You Customize)
|
||||
|
||||
* `ISbomMutator`
|
||||
* `IVulnerabilityProvider`
|
||||
* `IResultSink`
|
||||
* Policy files (`*.rego`)
|
||||
* Each DLL is SHA‑256 hashed; digest embedded in the run report for provenance.
|
||||
- **Scanner analyzers** (restart-time plug-ins) for ecosystem-specific parsing and facts extraction.
|
||||
- **Concelier connectors** for new advisory sources (preserving aggregation-only guardrails).
|
||||
- **Policy packs** for organization-specific gating and waivers/justifications.
|
||||
- **Export profiles** for output formats and offline bundle shapes.
|
||||
|
||||
Hot‑plugging is deferred until after v 1.0 for security review.
|
||||
## References
|
||||
|
||||
---
|
||||
|
||||
## 4 · Data & control flow
|
||||
|
||||
1. **Client** calls `/api/scan` with image reference.
|
||||
2. **Gateway** enforces quota, forwards to **Scanner Core** via gRPC.
|
||||
3. **Core**:
|
||||
|
||||
* Queries Redis for cached SBOM.
|
||||
* If miss → pulls layers, generates SBOM.
|
||||
* Executes plug‑ins (mutators, additional scanners).
|
||||
4. **Policy Engine** evaluates `scanResult` document.
|
||||
5. **Findings** stored in PostgreSQL; WebSocket event notifies UI.
|
||||
6. **ResultSink plug‑ins** export to Slack, Splunk, JSON file, etc.
|
||||
|
||||
---
|
||||
|
||||
## 5 · Security hardening
|
||||
|
||||
| Surface | Mitigation |
|
||||
| ----------------- | ------------------------------------------------------------ |
|
||||
| Container runtime | Distroless base, non‑root UID, seccomp + AppArmor |
|
||||
| Plug‑in sandbox | Separate UID, SELinux profile, cgroup 1 CPU / 256 MiB |
|
||||
| Supply chain | Cosign signatures, in‑toto SLSA Level 3 (target) |
|
||||
| Secrets | `Docker secrets` or K8s `Secret` mounts; never hard‑coded |
|
||||
| Quota abuse | Redis rate‑limit gates (see `30_QUOTA_ENFORCEMENT_FLOW1.md`) |
|
||||
|
||||
---
|
||||
|
||||
## 6 · Build & release pipeline (TL;DR)
|
||||
|
||||
* **Git commits** trigger CI → unit / integration / E2E tests.
|
||||
* Successful merge to `main`:
|
||||
|
||||
* Build `.NET {{ dotnet }}` trimmed self‑contained binary.
|
||||
* `docker build --sbom=spdx-json`.
|
||||
* Sign image and tarball with Cosign.
|
||||
* Attach SBOM + provenance; push to registry and download portal.
|
||||
|
||||
---
|
||||
|
||||
## 7 · Future extraction path
|
||||
|
||||
Although the default deployment is a single container, each sub‑service can be
|
||||
extracted:
|
||||
|
||||
* Concelier → standalone cron pod.
|
||||
* Policy Engine → side‑car (OPA) with gRPC contract.
|
||||
* ResultSink → queue worker (RabbitMQ or Azure Service Bus).
|
||||
|
||||
Interfaces are stable **as of v0.2 β**; extraction requires a recompilation
|
||||
only, not a fork of the core.
|
||||
|
||||
---
|
||||
|
||||
*Last updated {{ "now" | date: "%Y‑%m‑%d" }} – constants auto‑injected.*
|
||||
- `docs/07_HIGH_LEVEL_ARCHITECTURE.md`
|
||||
- `docs/24_OFFLINE_KIT.md`
|
||||
- `docs/09_API_CLI_REFERENCE.md`
|
||||
- `docs/modules/platform/architecture-overview.md`
|
||||
|
||||
Reference in New Issue
Block a user