stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 2 Releases Wiki Activity

wip: doctor/cli/docs/api to vector db consolidation; api hardening for descriptions, tenant, and scopes; migrations and conversions of all DALs to EF v10

Browse Source
This commit is contained in:
master
2026-02-23 15:30:50 +02:00
parent bd8fee6ed8
commit e746577380
1424 changed files with 81225 additions and 25251 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
docs/modules/timeline-indexer/README.md
View File

@@ -16,7 +16,7 @@ TimelineIndexer provides fast, indexed access to timeline events across all Stel
| Attribute | Value |
|-----------|-------|
| **Maturity** | Production |
| **Last Reviewed** | 2025-12-29 |
| **Last Reviewed** | 2026-02-22 |
| **Maintainer** | Platform Guild |
## Key Features
@@ -25,11 +25,14 @@ TimelineIndexer provides fast, indexed access to timeline events across all Stel
- **Time-Range Queries**: Efficient time-series queries with filtering
- **Event Stream Integration**: Consume from NATS/Valkey event streams
- **PostgreSQL Storage**: Time-series indexes for fast retrieval
- **EF Core DAL**: Database-first scaffolded model baseline with regeneration-safe partial overlays
- **Compiled EF Model**: Static compiled model module is used at runtime for context initialization
## Dependencies
### Upstream (this module depends on)
- **PostgreSQL** - Event storage with time-series indexes
- **EF Core 10 + Npgsql provider** - DAL and schema mapping
- **NATS/Valkey** - Event stream consumption
- **Authority** - Authentication

27
docs/modules/timeline-indexer/architecture.md
View File

@@ -1,4 +1,4 @@
# component_architecture_timelineindexer.md - **Stella Ops TimelineIndexer** (2025Q4)
# component_architecture_timelineindexer.md - **Stella Ops TimelineIndexer** (2026Q1)
> Timeline event indexing and query service.
@@ -22,18 +22,31 @@
```
src/TimelineIndexer/StellaOps.TimelineIndexer/
├─ StellaOps.TimelineIndexer.Core/ # Event models, indexing logic
├─ StellaOps.TimelineIndexer.Infrastructure/ # Storage adapters
├─ StellaOps.TimelineIndexer.WebService/ # Query API
├─ StellaOps.TimelineIndexer.Worker/ # Event consumer
└─ StellaOps.TimelineIndexer.Tests/
|- StellaOps.TimelineIndexer.Core/ # Event models, indexing logic
|- StellaOps.TimelineIndexer.Infrastructure/ # Storage adapters and DAL
|- StellaOps.TimelineIndexer.WebService/ # Query API
|- StellaOps.TimelineIndexer.Worker/ # Event consumer
`- StellaOps.TimelineIndexer.Tests/
```
### 1.1 Persistence implementation (2026-02-22)
* TimelineIndexer persistence uses **EF Core 10** with database-first scaffolded models.
* Generated artifacts are stored in:
* `src/TimelineIndexer/StellaOps.TimelineIndexer/StellaOps.TimelineIndexer.Infrastructure/EfCore/Context`
* `src/TimelineIndexer/StellaOps.TimelineIndexer/StellaOps.TimelineIndexer.Infrastructure/EfCore/Models`
* `src/TimelineIndexer/StellaOps.TimelineIndexer/StellaOps.TimelineIndexer.Infrastructure/EfCore/CompiledModels`
* Store adapters (`TimelineEventStore`, `TimelineQueryStore`) run through `TimelineIndexerDataSource` tenant-scoped sessions, preserving `app.current_tenant` and RLS behavior.
* Manual model corrections (enum mapping and FK relationship configuration) are implemented in partial files, so scaffolded files remain regeneratable.
* Runtime context initialization uses the static compiled model module:
* `options.UseModel(TimelineIndexerDbContextModel.Instance)`
---
## 2) External dependencies
* **PostgreSQL** - Event storage with time-series indexes
* **EF Core 10 + Npgsql provider** - DAL and model mapping for timeline schema
* **NATS/Valkey** - Event stream consumption
* **Authority** - Authentication
@@ -79,5 +92,3 @@ GET /healthz | /readyz | /metrics
* Signals: `../signals/architecture.md`
* Scanner: `../scanner/architecture.md`

76
docs/modules/timeline-indexer/guides/timeline.md
View File

@@ -2,12 +2,12 @@
> **Imposed rule:** Timeline is append-only; events may not be deleted or rewritten. Redactions require creating a compensating `redaction_notice` event that references the original ULID.
The Timeline Indexer service aggregates structured events (scanner runs, policy verdicts, runtime posture, evidence locker activity) so operators can audit changes over time. This guide summarises the event schema, query surfaces, and integration points.
The Timeline Indexer service aggregates structured events (scanner runs, policy verdicts, runtime posture, evidence locker activity) so operators can audit changes over time. This guide summarizes the event schema, query surfaces, persistence implementation, and integration points.
## 1. Event Model
| Field | Description |
|-------|-------------|
| --- | --- |
| `event_id` | ULID identifying the event. |
| `tenant` | Tenant scope. |
| `timestamp` | UTC ISO-8601 time the event occurred. |
@@ -19,22 +19,23 @@ Events are stored append-only with tenant-specific partitions. Producers include
### Event kinds (normative)
- `scan.completed` – scanner job finished; includes SBOM digest, findings counts, determinism score.
- `policy.verdict` – policy engine decision with overlay cache version and allow/deny rationale.
- `attestation.verified` – attestation verification result with Rekor UUID and bundle digest.
- `evidence.ingested` – Evidence Locker write with WORM vault identifier.
- `notify.sent` – outbound notification with target channel and template id.
- `runtime.alert` – runtime enforcement or observation event from Zastava Observer.
- `redaction_notice` – compensating entry when data is logically withdrawn; must include `supersedes` ULID.
- `scan.completed` - scanner job finished; includes SBOM digest, findings counts, determinism score.
- `policy.verdict` - policy engine decision with overlay cache version and allow/deny rationale.
- `attestation.verified` - attestation verification result with Rekor UUID and bundle digest.
- `evidence.ingested` - Evidence Locker write with WORM vault identifier.
- `notify.sent` - outbound notification with target channel and template id.
- `runtime.alert` - runtime enforcement or observation event from Zastava Observer.
- `redaction_notice` - compensating entry when data is logically withdrawn; must include `supersedes` ULID.
## 2. APIs
- Native endpoints:
Native endpoints:
- `GET /timeline` - query timeline entries with filter parameters.
- `GET /timeline/{eventId}` - fetch a single timeline entry.
- `GET /timeline/{eventId}/evidence` - fetch evidence linked to a timeline entry.
- `POST /timeline/events` - ingestion ack endpoint.
- Router/Gateway aliases for microservice transport routing:
Router/Gateway aliases for microservice transport routing:
- `GET /api/v1/timeline`
- `GET /api/v1/timeline/{eventId}`
- `GET /api/v1/timeline/{eventId}/evidence`
@@ -50,10 +51,10 @@ API headers: `X-Stella-Tenant`, optional `X-Stella-TraceId`, and `If-None-Match`
- For WORM verification, filter `category=evidence` and join on Evidence Locker bundle digest.
- Use `category=attestation.verified` and `details.rekor_uuid` to reconcile transparency proofs.
Example queries
Example queries:
```sh
# Recent scan → policy → notify chain for a digest
# Recent scan -> policy -> notify chain for a digest
stella timeline list --tenant acme --category scan.completed --subject sha256:abc... --limit 5
stella timeline list --tenant acme --category policy.verdict --trace-id <trace>
stella timeline list --tenant acme --category notify.sent --trace-id <trace>
@@ -64,7 +65,51 @@ curl -H "X-Stella-Tenant: acme" \
-o timeline-2025-11-01.ndjson
```
## 4. Integration
## 4. Persistence (EF Core)
- TimelineIndexer DAL is EF Core 10 based (`StellaOps.TimelineIndexer.Infrastructure/Db`).
- Database-first scaffolded context/models are under:
- `src/TimelineIndexer/StellaOps.TimelineIndexer/StellaOps.TimelineIndexer.Infrastructure/EfCore/Context`
- `src/TimelineIndexer/StellaOps.TimelineIndexer/StellaOps.TimelineIndexer.Infrastructure/EfCore/Models`
- Keep generated files regeneration-safe. Place manual fixes in partial classes/files.
Scaffold command used for baseline generation:
```sh
dotnet ef dbcontext scaffold "Host=localhost;Port=55433;Database=postgres;Username=postgres;Password=postgres" \
Npgsql.EntityFrameworkCore.PostgreSQL \
--project src/TimelineIndexer/StellaOps.TimelineIndexer/StellaOps.TimelineIndexer.Infrastructure/StellaOps.TimelineIndexer.Infrastructure.csproj \
--startup-project src/TimelineIndexer/StellaOps.TimelineIndexer/StellaOps.TimelineIndexer.WebService/StellaOps.TimelineIndexer.WebService.csproj \
--context TimelineIndexerDbContext \
--context-dir EfCore/Context \
--output-dir EfCore/Models \
--schema timeline \
--table timeline_events \
--table timeline_event_details \
--table timeline_event_digests \
--no-onconfiguring \
--use-database-names \
--force
```
Expected scaffold caveat: enum `timeline.event_severity` and composite FKs on `(event_id, tenant_id)` are not fully inferred by the scaffold command; these are restored in partial mapping files under `EfCore/Context` and `EfCore/Models`.
Generate/refresh compiled EF model artifacts:
```sh
dotnet ef dbcontext optimize \
--project src/TimelineIndexer/StellaOps.TimelineIndexer/StellaOps.TimelineIndexer.Infrastructure/StellaOps.TimelineIndexer.Infrastructure.csproj \
--startup-project src/TimelineIndexer/StellaOps.TimelineIndexer/StellaOps.TimelineIndexer.WebService/StellaOps.TimelineIndexer.WebService.csproj \
--context TimelineIndexerDbContext \
--output-dir EfCore/CompiledModels \
--namespace StellaOps.TimelineIndexer.Infrastructure.EfCore.CompiledModels
```
Design-time connection can be overridden with `STELLAOPS_TIMELINEINDEXER_EF_CONNECTION`.
Runtime usage is explicit in `TimelineIndexerDbContextFactory` via `UseModel(TimelineIndexerDbContextModel.Instance)` so the static compiled model module is always used by DAL stores.
## 5. Integration
- Evidence Locker attaches evidence bundle digests; the console links from timeline to evidence viewer.
- Notifier creates acknowledgement events for incident workflows.
@@ -74,10 +119,9 @@ Retention: events are retained per-tenant for at least `timeline.retention_days`
Privacy/PII: producers must redact PII before emission; once emitted, redactions occur via `redaction_notice` only.
## 5. References
## 6. References
- `docs/modules/telemetry/architecture.md`
- `docs/modules/zastava/architecture.md`
- `docs/modules/export-center/architecture.md`
- `src/TimelineIndexer/StellaOps.TimelineIndexer`