wip: doctor/cli/docs/api to vector db consolidation; api hardening for descriptions, tenant, and scopes; migrations and conversions of all DALs to EF v10
This commit is contained in:
master
@@ -2,12 +2,12 @@
|
||||
|
||||
> **Imposed rule:** Timeline is append-only; events may not be deleted or rewritten. Redactions require creating a compensating `redaction_notice` event that references the original ULID.
|
||||
|
||||
The Timeline Indexer service aggregates structured events (scanner runs, policy verdicts, runtime posture, evidence locker activity) so operators can audit changes over time. This guide summarises the event schema, query surfaces, and integration points.
|
||||
The Timeline Indexer service aggregates structured events (scanner runs, policy verdicts, runtime posture, evidence locker activity) so operators can audit changes over time. This guide summarizes the event schema, query surfaces, persistence implementation, and integration points.
|
||||
|
||||
## 1. Event Model
|
||||
|
||||
| Field | Description |
|
||||
|-------|-------------|
|
||||
| --- | --- |
|
||||
| `event_id` | ULID identifying the event. |
|
||||
| `tenant` | Tenant scope. |
|
||||
| `timestamp` | UTC ISO-8601 time the event occurred. |
|
||||
@@ -19,22 +19,23 @@ Events are stored append-only with tenant-specific partitions. Producers include
|
||||
|
||||
### Event kinds (normative)
|
||||
|
||||
- `scan.completed` – scanner job finished; includes SBOM digest, findings counts, determinism score.
|
||||
- `policy.verdict` – policy engine decision with overlay cache version and allow/deny rationale.
|
||||
- `attestation.verified` – attestation verification result with Rekor UUID and bundle digest.
|
||||
- `evidence.ingested` – Evidence Locker write with WORM vault identifier.
|
||||
- `notify.sent` – outbound notification with target channel and template id.
|
||||
- `runtime.alert` – runtime enforcement or observation event from Zastava Observer.
|
||||
- `redaction_notice` – compensating entry when data is logically withdrawn; must include `supersedes` ULID.
|
||||
- `scan.completed` - scanner job finished; includes SBOM digest, findings counts, determinism score.
|
||||
- `policy.verdict` - policy engine decision with overlay cache version and allow/deny rationale.
|
||||
- `attestation.verified` - attestation verification result with Rekor UUID and bundle digest.
|
||||
- `evidence.ingested` - Evidence Locker write with WORM vault identifier.
|
||||
- `notify.sent` - outbound notification with target channel and template id.
|
||||
- `runtime.alert` - runtime enforcement or observation event from Zastava Observer.
|
||||
- `redaction_notice` - compensating entry when data is logically withdrawn; must include `supersedes` ULID.
|
||||
|
||||
## 2. APIs
|
||||
|
||||
- Native endpoints:
|
||||
Native endpoints:
|
||||
- `GET /timeline` - query timeline entries with filter parameters.
|
||||
- `GET /timeline/{eventId}` - fetch a single timeline entry.
|
||||
- `GET /timeline/{eventId}/evidence` - fetch evidence linked to a timeline entry.
|
||||
- `POST /timeline/events` - ingestion ack endpoint.
|
||||
- Router/Gateway aliases for microservice transport routing:
|
||||
|
||||
Router/Gateway aliases for microservice transport routing:
|
||||
- `GET /api/v1/timeline`
|
||||
- `GET /api/v1/timeline/{eventId}`
|
||||
- `GET /api/v1/timeline/{eventId}/evidence`
|
||||
@@ -50,10 +51,10 @@ API headers: `X-Stella-Tenant`, optional `X-Stella-TraceId`, and `If-None-Match`
|
||||
- For WORM verification, filter `category=evidence` and join on Evidence Locker bundle digest.
|
||||
- Use `category=attestation.verified` and `details.rekor_uuid` to reconcile transparency proofs.
|
||||
|
||||
Example queries
|
||||
Example queries:
|
||||
|
||||
```sh
|
||||
# Recent scan → policy → notify chain for a digest
|
||||
# Recent scan -> policy -> notify chain for a digest
|
||||
stella timeline list --tenant acme --category scan.completed --subject sha256:abc... --limit 5
|
||||
stella timeline list --tenant acme --category policy.verdict --trace-id <trace>
|
||||
stella timeline list --tenant acme --category notify.sent --trace-id <trace>
|
||||
@@ -64,7 +65,51 @@ curl -H "X-Stella-Tenant: acme" \
|
||||
-o timeline-2025-11-01.ndjson
|
||||
```
|
||||
|
||||
## 4. Integration
|
||||
## 4. Persistence (EF Core)
|
||||
|
||||
- TimelineIndexer DAL is EF Core 10 based (`StellaOps.TimelineIndexer.Infrastructure/Db`).
|
||||
- Database-first scaffolded context/models are under:
|
||||
- `src/TimelineIndexer/StellaOps.TimelineIndexer/StellaOps.TimelineIndexer.Infrastructure/EfCore/Context`
|
||||
- `src/TimelineIndexer/StellaOps.TimelineIndexer/StellaOps.TimelineIndexer.Infrastructure/EfCore/Models`
|
||||
- Keep generated files regeneration-safe. Place manual fixes in partial classes/files.
|
||||
|
||||
Scaffold command used for baseline generation:
|
||||
|
||||
```sh
|
||||
dotnet ef dbcontext scaffold "Host=localhost;Port=55433;Database=postgres;Username=postgres;Password=postgres" \
|
||||
Npgsql.EntityFrameworkCore.PostgreSQL \
|
||||
--project src/TimelineIndexer/StellaOps.TimelineIndexer/StellaOps.TimelineIndexer.Infrastructure/StellaOps.TimelineIndexer.Infrastructure.csproj \
|
||||
--startup-project src/TimelineIndexer/StellaOps.TimelineIndexer/StellaOps.TimelineIndexer.WebService/StellaOps.TimelineIndexer.WebService.csproj \
|
||||
--context TimelineIndexerDbContext \
|
||||
--context-dir EfCore/Context \
|
||||
--output-dir EfCore/Models \
|
||||
--schema timeline \
|
||||
--table timeline_events \
|
||||
--table timeline_event_details \
|
||||
--table timeline_event_digests \
|
||||
--no-onconfiguring \
|
||||
--use-database-names \
|
||||
--force
|
||||
```
|
||||
|
||||
Expected scaffold caveat: enum `timeline.event_severity` and composite FKs on `(event_id, tenant_id)` are not fully inferred by the scaffold command; these are restored in partial mapping files under `EfCore/Context` and `EfCore/Models`.
|
||||
|
||||
Generate/refresh compiled EF model artifacts:
|
||||
|
||||
```sh
|
||||
dotnet ef dbcontext optimize \
|
||||
--project src/TimelineIndexer/StellaOps.TimelineIndexer/StellaOps.TimelineIndexer.Infrastructure/StellaOps.TimelineIndexer.Infrastructure.csproj \
|
||||
--startup-project src/TimelineIndexer/StellaOps.TimelineIndexer/StellaOps.TimelineIndexer.WebService/StellaOps.TimelineIndexer.WebService.csproj \
|
||||
--context TimelineIndexerDbContext \
|
||||
--output-dir EfCore/CompiledModels \
|
||||
--namespace StellaOps.TimelineIndexer.Infrastructure.EfCore.CompiledModels
|
||||
```
|
||||
|
||||
Design-time connection can be overridden with `STELLAOPS_TIMELINEINDEXER_EF_CONNECTION`.
|
||||
|
||||
Runtime usage is explicit in `TimelineIndexerDbContextFactory` via `UseModel(TimelineIndexerDbContextModel.Instance)` so the static compiled model module is always used by DAL stores.
|
||||
|
||||
## 5. Integration
|
||||
|
||||
- Evidence Locker attaches evidence bundle digests; the console links from timeline to evidence viewer.
|
||||
- Notifier creates acknowledgement events for incident workflows.
|
||||
@@ -74,10 +119,9 @@ Retention: events are retained per-tenant for at least `timeline.retention_days`
|
||||
|
||||
Privacy/PII: producers must redact PII before emission; once emitted, redactions occur via `redaction_notice` only.
|
||||
|
||||
## 5. References
|
||||
## 6. References
|
||||
|
||||
- `docs/modules/telemetry/architecture.md`
|
||||
- `docs/modules/zastava/architecture.md`
|
||||
- `docs/modules/export-center/architecture.md`
|
||||
- `src/TimelineIndexer/StellaOps.TimelineIndexer`
|
||||
|
||||
|
||||
Reference in New Issue
Block a user