stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 2 Releases Wiki Activity

Add determinism tests for verdict artifact generation and update SHA256 sums script

Browse Source 
- Implemented comprehensive tests for verdict artifact generation to ensure deterministic outputs across various scenarios, including identical inputs, parallel execution, and change ordering.
- Created helper methods for generating sample verdict inputs and computing canonical hashes.
- Added tests to validate the stability of canonical hashes, proof spine ordering, and summary statistics.
- Introduced a new PowerShell script to update SHA256 sums for files, ensuring accurate hash generation and file integrity checks.
This commit is contained in:
StellaOps Bot
2025-12-24 02:17:34 +02:00
parent e59921374e
commit 7503c19b8f
390 changed files with 37389 additions and 5380 deletions
Download Patch File Download Diff File Expand all files Collapse all files

71
docs/technical/architecture/README.md
View File

@@ -1,44 +1,51 @@
# Platform Architecture & Module Dossiers
# Platform architecture & module dossiers
Use this index to locate architecture narratives, boundaries, and implementation plans for every Stella Ops component.
Use this index to locate platform-level architecture references and per-module dossiers.
## Core Views
- [../high-level-architecture.md](../../high-level-architecture.md) – 10-minute overview of the end-to-end flow.
- [../07_HIGH_LEVEL_ARCHITECTURE.md](../../07_HIGH_LEVEL_ARCHITECTURE.md) – exhaustive reference (data flows, trust boundaries, operational traits).
- [../40_ARCHITECTURE_OVERVIEW.md](../../40_ARCHITECTURE_OVERVIEW.md) – design principles applied across modules.
- [../scanner-core-contracts.md](../../scanner-core-contracts.md) – canonical DTOs shared by Scanner services and consumers.
- Legacy service dossier: [../11_AUTHORITY.md](../../11_AUTHORITY.md) – Authority overview before module split.
- UI documentation set: [../../ui/](../../ui/) (navigation, policies, findings, runs, tours).
- Component map: [component-map.md](component-map.md) – quick descriptions of every `src/` module and how they interact.
## Core views
- [Architecture overview (10-minute tour)](../../40_ARCHITECTURE_OVERVIEW.md)
- [High-level architecture (reference map)](../../07_HIGH_LEVEL_ARCHITECTURE.md)
- [Scanner core contracts](../../scanner-core-contracts.md)
- [Authority (legacy overview)](../../11_AUTHORITY.md)
- [Console operator guide](../../15_UI_GUIDE.md) and deep dives under [console](../../console/) and [ux](../../ux/)
- [Component map](component-map.md) (quick descriptions of every module under `src/`)
## Module Catalogue
Each module directory bundles an ownership charter (`AGENTS.md`), current work (`TASKS.md`), architecture dossier, and implementation plan. Operations guides live under `operations/` where applicable.
## Detailed references
- [Platform topology](platform-topology.md)
- [Infrastructure dependencies](infrastructure-dependencies.md)
- [Request and data flows](request-flows.md)
- [Data isolation model](data-isolation.md)
- [Security boundaries](security-boundaries.md)
| Module | Architecture | Implementation Plan | Operations / Extras |
|--------|--------------|---------------------|---------------------|
| Authority | [architecture.md](../../modules/authority/architecture.md) | [implementation_plan.md](../../modules/authority/implementation_plan.md) | [operations](../../modules/authority/operations/) |
| Advisory AI | [architecture.md](../../modules/advisory-ai/architecture.md) | [implementation_plan.md](../../modules/advisory-ai/implementation_plan.md) | — |
| Attestor | [architecture.md](../../modules/attestor/architecture.md) | [implementation_plan.md](../../modules/attestor/implementation_plan.md) | — |
## Module catalogue
Each module directory bundles an ownership charter (`AGENTS.md`), current work (`TASKS.md`), an architecture dossier, and an implementation plan. Operations guides live under `operations/` where applicable.
| Module | Architecture | Implementation plan | Operations / extras |
| --- | --- | --- | --- |
| Authority | [architecture.md](../../modules/authority/architecture.md) | [implementation_plan.md](../../modules/authority/implementation_plan.md) | [operations/](../../modules/authority/operations/) |
| Advisory AI | [architecture.md](../../modules/advisory-ai/architecture.md) | [implementation_plan.md](../../modules/advisory-ai/implementation_plan.md) | - |
| Attestor | [architecture.md](../../modules/attestor/architecture.md) | [implementation_plan.md](../../modules/attestor/implementation_plan.md) | - |
| CLI | [architecture.md](../../modules/cli/architecture.md) | [implementation_plan.md](../../modules/cli/implementation_plan.md) | [operations/release-and-packaging.md](../../modules/cli/operations/release-and-packaging.md) |
| CI Recipes | [architecture.md](../../modules/ci/architecture.md) | [implementation_plan.md](../../modules/ci/implementation_plan.md) | [recipes.md](../../modules/ci/recipes.md) |
| CI recipes | [architecture.md](../../modules/ci/architecture.md) | [implementation_plan.md](../../modules/ci/implementation_plan.md) | [recipes.md](../../modules/ci/recipes.md) |
| Concelier | [architecture.md](../../modules/concelier/architecture.md) | [implementation_plan.md](../../modules/concelier/implementation_plan.md) | [operations/](../../modules/concelier/operations/) |
| DevOps / Release | [architecture.md](../../modules/devops/architecture.md) | [implementation_plan.md](../../modules/devops/implementation_plan.md) | [runbooks](../../modules/devops/runbooks/) |
| DevOps / release | [architecture.md](../../modules/devops/architecture.md) | [implementation_plan.md](../../modules/devops/implementation_plan.md) | [runbooks/](../../modules/devops/runbooks/) |
| Excititor | [architecture.md](../../modules/excititor/architecture.md) | [implementation_plan.md](../../modules/excititor/implementation_plan.md) | [mirrors.md](../../modules/excititor/mirrors.md) |
| Export Center | [architecture.md](../../modules/export-center/architecture.md) | [implementation_plan.md](../../modules/export-center/implementation_plan.md) | [operations/runbook.md](../../modules/export-center/operations/runbook.md) |
| Graph | [architecture.md](../../modules/graph/architecture.md) | [implementation_plan.md](../../modules/graph/implementation_plan.md) | — |
| Notify | [architecture.md](../../modules/notify/architecture.md) | [implementation_plan.md](../../modules/notify/implementation_plan.md) | — |
| Orchestrator | [architecture.md](../../modules/orchestrator/architecture.md) | [implementation_plan.md](../../modules/orchestrator/implementation_plan.md) | — |
| Platform | [architecture-overview.md](../../modules/platform/architecture-overview.md) + [architecture.md](../../modules/platform/architecture.md) | [implementation_plan.md](../../modules/platform/implementation_plan.md) | — |
| Policy Engine | [architecture.md](../../modules/policy/architecture.md) | [implementation_plan.md](../../modules/policy/implementation_plan.md) | — |
| Registry Token Service | [architecture.md](../../modules/registry/architecture.md) | [implementation_plan.md](../../modules/registry/implementation_plan.md) | [operations/token-service.md](../../modules/registry/operations/token-service.md) |
| Graph | [architecture.md](../../modules/graph/architecture.md) | [implementation_plan.md](../../modules/graph/implementation_plan.md) | - |
| Notify | [architecture.md](../../modules/notify/architecture.md) | [implementation_plan.md](../../modules/notify/implementation_plan.md) | - |
| Orchestrator | [architecture.md](../../modules/orchestrator/architecture.md) | [implementation_plan.md](../../modules/orchestrator/implementation_plan.md) | - |
| Platform | [architecture-overview.md](../../modules/platform/architecture-overview.md) + [architecture.md](../../modules/platform/architecture.md) | [implementation_plan.md](../../modules/platform/implementation_plan.md) | - |
| Policy engine | [architecture.md](../../modules/policy/architecture.md) | [implementation_plan.md](../../modules/policy/implementation_plan.md) | - |
| Registry token service | [architecture.md](../../modules/registry/architecture.md) | [implementation_plan.md](../../modules/registry/implementation_plan.md) | [operations/token-service.md](../../modules/registry/operations/token-service.md) |
| Scanner | [architecture.md](../../modules/scanner/architecture.md) | [implementation_plan.md](../../modules/scanner/implementation_plan.md) | [operations/](../../modules/scanner/operations/) |
| Scheduler | [architecture.md](../../modules/scheduler/architecture.md) | [implementation_plan.md](../../modules/scheduler/implementation_plan.md) | [operations/](../../modules/scheduler/operations/) |
| Signer | [architecture.md](../../modules/signer/architecture.md) | [implementation_plan.md](../../modules/signer/implementation_plan.md) | — |
| Telemetry Stack | [architecture.md](../../modules/telemetry/architecture.md) | [implementation_plan.md](../../modules/telemetry/implementation_plan.md) | [operations/collector.md](../../modules/telemetry/operations/collector.md), [operations/storage.md](../../modules/telemetry/operations/storage.md) |
| UI / Console | [architecture.md](../../modules/ui/architecture.md), [console-architecture.md](../../modules/ui/console-architecture.md) | [implementation_plan.md](../../modules/ui/implementation_plan.md) | — |
| Vuln Explorer | [architecture.md](../../modules/vuln-explorer/architecture.md) | [implementation_plan.md](../../modules/vuln-explorer/implementation_plan.md) | — |
| VEX Lens | [architecture.md](../../modules/vex-lens/architecture.md) | [implementation_plan.md](../../modules/vex-lens/implementation_plan.md) | — |
| Signer | [architecture.md](../../modules/signer/architecture.md) | [implementation_plan.md](../../modules/signer/implementation_plan.md) | - |
| Telemetry stack | [architecture.md](../../modules/telemetry/architecture.md) | [implementation_plan.md](../../modules/telemetry/implementation_plan.md) | [operations/collector.md](../../modules/telemetry/operations/collector.md), [operations/storage.md](../../modules/telemetry/operations/storage.md) |
| UI / Console | [architecture.md](../../modules/ui/architecture.md), [console-architecture.md](../../modules/ui/console-architecture.md) | [implementation_plan.md](../../modules/ui/implementation_plan.md) | - |
| Vuln Explorer | [architecture.md](../../modules/vuln-explorer/architecture.md) | [implementation_plan.md](../../modules/vuln-explorer/implementation_plan.md) | - |
| VEX Lens | [architecture.md](../../modules/vex-lens/architecture.md) | [implementation_plan.md](../../modules/vex-lens/implementation_plan.md) | - |
| Excitor | [architecture.md](../../modules/excitor/architecture.md) | [implementation_plan.md](../../modules/excitor/implementation_plan.md) | [scoring.md](../../modules/excitor/scoring.md) |
| Zastava | [architecture.md](../../modules/zastava/architecture.md) | [implementation_plan.md](../../modules/zastava/implementation_plan.md) | — |
| Zastava | [architecture.md](../../modules/zastava/architecture.md) | [implementation_plan.md](../../modules/zastava/implementation_plan.md) | - |
> **Tip:** Every module directory also exposes `README.md`, `AGENTS.md`, and `TASKS.md` for roles, current backlog, and ownership responsibilities.
> Tip: every module directory also exposes `README.md`, `AGENTS.md`, and `TASKS.md` for roles, current backlog, and ownership responsibilities.

2
docs/technical/architecture/component-map.md
View File

@@ -67,7 +67,7 @@ Concise descriptions of every top-level component under `src/`, summarising the
- **Aoc** library (mentioned above) is reused by ingestion components and verification tooling to enforce the Aggregation-Only Contract.
## How It All Connects
High-level flows (see `docs/high-level-architecture.md` for diagrams):
High-level flows (see `docs/40_ARCHITECTURE_OVERVIEW.md` for the 10-minute tour):
1. **Ingest** — Concelier and Excititor use AOC to ingest advisories/VEX; Scheduler observes deltas.
2. **Scan & Evaluate** — Scanner generates SBOM evidence and hands to Signer/Attestor; Policy Engine merges SBOM, advisory, VEX, runtime signals; RiskEngine prioritises.
3. **Store & Export** — EvidenceLocker and Export Center package results; Registry serves artefacts; AirGap bundles offline editions.

25
docs/technical/architecture/data-isolation.md Normal file
View File

@@ -0,0 +1,25 @@
# Data isolation model (PostgreSQL)
StellaOps uses PostgreSQL as the canonical durable store. Isolation is achieved by:
- One schema per service (clear ownership boundaries).
- Tenant identifiers on all tenant-scoped records (enabling row-level strategies where required).
- Append-only patterns for specific evidence stores to preserve replayability.
## Schema ownership map
| Schema | Owner (primary) | Data class |
| --- | --- | --- |
| `authority` | Authority | Identity, clients, keys, auth audit trails. |
| `scanner` | Scanner | Scan manifests, triage, scan result metadata. |
| `vuln` | Concelier | Advisory raw documents, linksets, observations. |
| `vex` | Excititor | VEX raw statements and consensus state. |
| `scheduler` | Scheduler | Job orchestration state. |
| `notify` | Notify | Notifications state and delivery history. |
| `policy` | Policy | Exceptions, policy snapshots, unknown tracking. |
| `orchestrator` | Orchestrator | Workflow orchestration state. |
## Where to find authoritative schemas
This document is descriptive. The authoritative contract is:
- Module dossiers and migration notes under `docs/modules/<module>/`
- Database schema reference: `docs/11_DATA_SCHEMAS.md`

42
docs/technical/architecture/infrastructure-dependencies.md Normal file
View File

@@ -0,0 +1,42 @@
# Infrastructure dependencies (detailed)
StellaOps is designed to run with a small set of required infrastructure components. Everything else is optional and must not be a hidden dependency for core workflows.
## PostgreSQL (required)
Primary store for durable state. Each service owns a schema to keep boundaries clear and enable tenant isolation strategies.
| Schema | Owner (primary) | Purpose |
| --- | --- | --- |
| `authority` | Authority | Users, clients, tenants, keys, audit trails. |
| `scanner` | Scanner | Scan manifests, triage, scan results metadata. |
| `vuln` | Concelier | Advisory raw documents, linksets, observations. |
| `vex` | Excititor | VEX raw documents, consensus, provider state. |
| `scheduler` | Scheduler | Jobs, runs, schedules, impact snapshots. |
| `notify` | Notify | Channels, templates, delivery history, digests. |
| `policy` | Policy | Exception objects, snapshots, unknowns. |
| `orchestrator` | Orchestrator | Sources, runs, jobs, DAGs, pack runs. |
## Valkey (required)
Redis-compatible cache + coordination substrate.
| Pattern | Typical services | Purpose |
| --- | --- | --- |
| DPoP nonces | Authority | RFC 9449 nonce storage (short TTL). |
| Streams / events | Scanner, Notify, Scheduler | Event emission and fan-out (deterministic ordering per stream). |
| Queues | Scanner, Notify | Worker coordination (consumer groups). |
| Cache | All services | Tenant-prefixed caching with explicit TTLs. |
| Rate limiting | Gateway, Authority | Token bucket counters. |
## RustFS / S3-compatible object storage (required)
Artifact store for SBOMs, evidence bundles, and replayable outputs. The exact bucket layout depends on the deployment profile; treat deployment manifests as authoritative.
## NATS JetStream (optional)
Alternative messaging transport for environments that require persistent streams or specific operational characteristics. NATS must be explicitly configured and must not be required for core workflows.
## Deployment references
- Compose profiles: `deploy/compose/README.md`
- Deployment bundles overview: `deploy/README.md`

17
docs/technical/architecture/platform-topology.md Normal file
View File

@@ -0,0 +1,17 @@
# Platform topology (detailed)
This document provides a clean, audit-friendly view of StellaOps platform topology without relying on fragile ASCII diagrams. For module-specific details (APIs, schemas, operations), use `docs/modules/`.
## Layers
| Layer | Primary components | Responsibility |
| --- | --- | --- |
| Client | CLI, Web UI, CI/CD pipelines, runtime observers | Submit scan requests, query results, manage policy/tenancy. |
| Gateway | Gateway.WebService | Auth enforcement, tenant routing, rate limiting, request correlation, API routing. |
| Auth & crypto | Authority, Signer, Attestor, IssuerDirectory | Token issuance, signing, transparency/attestation workflows, issuer trust registry. |
| Core engines | Scanner, Concelier, Excititor, Policy, Scheduler, Notify, Orchestrator | Scanning, ingestion, verdicts, orchestration, notifications, exports. |
| Data plane | PostgreSQL, Valkey, RustFS (S3), optional NATS | Persistent state, queues/streams, artifact storage, optional alternative messaging. |
## Notes
- Module dossiers live under `docs/modules/<module>/architecture.md`.
- Deployment defaults (ports, profile overlays, pinned digests) live under `deploy/` (`deploy/compose/`, `deploy/helm/`, `deploy/releases/`).

48
docs/technical/architecture/request-flows.md Normal file
View File

@@ -0,0 +1,48 @@
# Request and data flows (detailed)
This document describes the canonical end-to-end flows at a level useful for debugging and auditing. Exact endpoints and payloads are defined by each module dossier under `docs/modules/`.
## 1) Scan execution (happy path)
1. **Client -> Gateway**: submit scan request (authenticated; tenant-scoped).
2. **Gateway -> Scanner.WebService**: route request after auth/rate-limit checks.
3. **Scanner.WebService -> PostgreSQL**: persist scan manifest and initial status.
4. **Scanner.WebService -> queue/stream**: enqueue a scan job (Valkey streams by default; optional alternative transports exist).
5. **Scanner.Worker -> queue/stream**: claim job, pull image, extract layers, run analyzers.
6. **Scanner.Worker -> RustFS/S3**: write SBOM fragments, composed SBOMs, and other scan artifacts.
7. **Scanner.Worker -> Concelier**: query linksets / observations needed for evaluation (deployment-dependent).
8. **Scanner.Worker -> Scanner.WebService**: heartbeat and completion callbacks.
9. **Scanner.WebService -> Policy**: request verdict evaluation using SBOM + advisory + VEX + policy inputs.
10. **Scanner.WebService -> Signer / Attestor (optional)**: create DSSE/in-toto evidence bundles and (optionally) attach transparency receipts.
11. **Scanner.WebService -> events stream**: publish completion events for notifications and downstream consumers.
12. **Notify.WebService/Worker -> channels**: render and deliver notifications with idempotency tracking.
Offline note: for air-gapped deployments, step 6 writes to local object storage and step 7 relies on offline mirrors/bundles rather than public feeds. See `docs/24_OFFLINE_KIT.md` and `docs/airgap/overview.md`.
## 2) Advisory ingestion (delta-driven)
1. **Concelier.Worker** fetches advisories from configured sources (mirrors first; no hidden outbound calls in air-gap profiles).
2. **Concelier** validates and normalizes advisories, producing canonical observations and linksets.
3. **Concelier -> PostgreSQL (`vuln`)** persists immutable raw documents (append-only patterns where required) plus derived linksets.
4. **Concelier -> Scheduler** notifies about deltas (new/updated advisories) via webhook/event.
5. **Scheduler** schedules impacted re-scans or evaluations based on the delta.
## 3) VEX ingestion and consensus
1. **Excititor.Worker** fetches VEX statements from configured sources (mirrors/bundles for offline).
2. **Excititor** verifies signatures where required and normalizes statements into a canonical shape.
3. **Excititor -> PostgreSQL (`vex`)** persists immutable raw statements and consensus outcomes.
4. **Excititor -> Scheduler / Policy** emits deltas so verdicts can be recomputed deterministically.
## 4) Policy evaluation (decision trace)
1. **Caller (Scanner/UI/CLI) -> Policy.Gateway** submits evaluation request.
2. **Policy.Gateway** loads exception objects and policy snapshots from its own store.
3. **Policy Engine** consumes advisory/VEX observations (by read model, replication, or API depending on deployment) and applies deterministic precedence/lattice rules.
4. **Policy.Gateway -> caller** returns a verdict plus a trace/explain payload suitable for audits.
## 5) Notification delivery
1. **Notify.WebService** consumes platform events (scan completed, advisory delta, etc.).
2. **Notify.WebService -> queue/stream** enqueues delivery tasks with idempotency keys.
3. **Notify.Worker -> channels** delivers (email/chat/webhook), records results, and retries with deterministic backoff rules.

33
docs/technical/architecture/security-boundaries.md Normal file
View File

@@ -0,0 +1,33 @@
# Security boundaries (platform-level)
This document describes the baseline security boundaries expected across StellaOps modules. Module dossiers may impose stricter requirements.
## Authentication and authorization
All externally reachable services are expected to enforce:
1. Token validation (short-lived, tenant-scoped access tokens).
2. Sender constraints where configured (DPoP / mTLS).
3. Scope-based authorization (least privilege).
4. Tenant isolation: requests and data access are filtered by tenant context.
## Network segmentation (typical deployment)
- **Front door / ingress**: TLS termination, rate limiting, and WAF controls.
- **Gateway layer**: authenticated routing and tenant resolution; exposes only required service surfaces.
- **Private service network**: internal service-to-service traffic (least privilege, explicit allowlists).
- **Stateful infrastructure**: PostgreSQL, Valkey, object storage, message brokers (not directly internet-exposed).
Deployment bundles under `deploy/` are the authoritative source of concrete network layouts.
## Data protection
- TLS for in-transit protection (including internal traffic where required by the profile).
- Secrets must not be embedded in documentation examples; use a secrets provider (file, Docker secrets, Kubernetes secrets, vault).
- Evidence artifacts should be written to content-addressed or immutable stores where replay/audit requires it.
## Auditability
The platform is designed for audits:
- Deterministic outputs for the same inputs.
- Evidence artifacts (SBOM slices, advisories/VEX observations, explain traces) linkable to digests.
- Structured logs and correlation IDs for tracing request paths across services.