consolidation of some of the modules, localization fixes, product advisories work, qa work
This commit is contained in:
master
59
docs-archived/modules/timeline-indexer/README.md
Normal file
59
docs-archived/modules/timeline-indexer/README.md
Normal file
@@ -0,0 +1,59 @@
|
||||
# Timeline Indexer
|
||||
|
||||
> Timeline event indexing and query service.
|
||||
|
||||
## Purpose
|
||||
|
||||
TimelineIndexer provides fast, indexed access to timeline events across all StellaOps services. It enables efficient querying of vulnerability history, scan timelines, and policy evaluation trails.
|
||||
|
||||
## Quick Links
|
||||
|
||||
- [Architecture](./architecture.md) - Technical design and implementation details
|
||||
- [Guides](./guides/) - Query and configuration guides
|
||||
|
||||
## Status
|
||||
|
||||
| Attribute | Value |
|
||||
|-----------|-------|
|
||||
| **Maturity** | Production |
|
||||
| **Last Reviewed** | 2026-02-22 |
|
||||
| **Maintainer** | Platform Guild |
|
||||
|
||||
## Key Features
|
||||
|
||||
- **Event Indexing**: Index events from multiple StellaOps services
|
||||
- **Time-Range Queries**: Efficient time-series queries with filtering
|
||||
- **Event Stream Integration**: Consume from NATS/Valkey event streams
|
||||
- **PostgreSQL Storage**: Time-series indexes for fast retrieval
|
||||
- **EF Core DAL**: Database-first scaffolded model baseline with regeneration-safe partial overlays
|
||||
- **Compiled EF Model**: Static compiled model module is used at runtime for context initialization
|
||||
|
||||
## Dependencies
|
||||
|
||||
### Upstream (this module depends on)
|
||||
- **PostgreSQL** - Event storage with time-series indexes
|
||||
- **EF Core 10 + Npgsql provider** - DAL and schema mapping
|
||||
- **NATS/Valkey** - Event stream consumption
|
||||
- **Authority** - Authentication
|
||||
|
||||
### Downstream (modules that depend on this)
|
||||
- **Web Console** - Timeline visualization
|
||||
- **CLI** - Timeline queries
|
||||
- **ExportCenter** - Timeline data exports
|
||||
|
||||
## Configuration
|
||||
|
||||
```yaml
|
||||
timeline_indexer:
|
||||
event_sources:
|
||||
- nats://events.stellaops.local
|
||||
retention_days: 365
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
TimelineIndexer indexes events; it does not generate them. Events are received from event streams published by other services.
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Telemetry Architecture](../telemetry/architecture.md)
|
||||
94
docs-archived/modules/timeline-indexer/architecture.md
Normal file
94
docs-archived/modules/timeline-indexer/architecture.md
Normal file
@@ -0,0 +1,94 @@
|
||||
# component_architecture_timelineindexer.md - **Stella Ops TimelineIndexer** (2026Q1)
|
||||
|
||||
> Timeline event indexing and query service.
|
||||
|
||||
> **Scope.** Implementation-ready architecture for **TimelineIndexer**: indexing and querying timeline events for vulnerability findings, scans, and policy evaluations.
|
||||
|
||||
---
|
||||
|
||||
## 0) Mission & boundaries
|
||||
|
||||
**Mission.** Provide **fast, indexed access** to timeline events across all StellaOps services. Enable efficient querying of vulnerability history, scan timelines, and policy evaluation trails.
|
||||
|
||||
**Boundaries.**
|
||||
|
||||
* TimelineIndexer **indexes events**; it does not generate them.
|
||||
* Events are received from **event streams** (NATS, Valkey).
|
||||
* Supports **time-range queries** with filtering.
|
||||
|
||||
---
|
||||
|
||||
## 1) Solution & project layout
|
||||
|
||||
```
|
||||
src/TimelineIndexer/StellaOps.TimelineIndexer/
|
||||
|- StellaOps.TimelineIndexer.Core/ # Event models, indexing logic
|
||||
|- StellaOps.TimelineIndexer.Infrastructure/ # Storage adapters and DAL
|
||||
|- StellaOps.TimelineIndexer.WebService/ # Query API
|
||||
|- StellaOps.TimelineIndexer.Worker/ # Event consumer
|
||||
`- StellaOps.TimelineIndexer.Tests/
|
||||
```
|
||||
|
||||
### 1.1 Persistence implementation (2026-02-22)
|
||||
|
||||
* TimelineIndexer persistence uses **EF Core 10** with database-first scaffolded models.
|
||||
* Generated artifacts are stored in:
|
||||
* `src/TimelineIndexer/StellaOps.TimelineIndexer/StellaOps.TimelineIndexer.Infrastructure/EfCore/Context`
|
||||
* `src/TimelineIndexer/StellaOps.TimelineIndexer/StellaOps.TimelineIndexer.Infrastructure/EfCore/Models`
|
||||
* `src/TimelineIndexer/StellaOps.TimelineIndexer/StellaOps.TimelineIndexer.Infrastructure/EfCore/CompiledModels`
|
||||
* Store adapters (`TimelineEventStore`, `TimelineQueryStore`) run through `TimelineIndexerDataSource` tenant-scoped sessions, preserving `app.current_tenant` and RLS behavior.
|
||||
* Manual model corrections (enum mapping and FK relationship configuration) are implemented in partial files, so scaffolded files remain regeneratable.
|
||||
* Runtime context initialization uses the static compiled model module:
|
||||
* `options.UseModel(TimelineIndexerDbContextModel.Instance)`
|
||||
|
||||
---
|
||||
|
||||
## 2) External dependencies
|
||||
|
||||
* **PostgreSQL** - Event storage with time-series indexes
|
||||
* **EF Core 10 + Npgsql provider** - DAL and model mapping for timeline schema
|
||||
* **NATS/Valkey** - Event stream consumption
|
||||
* **Authority** - Authentication
|
||||
|
||||
---
|
||||
|
||||
## 3) Contracts & data model
|
||||
|
||||
### 3.1 TimelineEvent
|
||||
|
||||
```json
|
||||
{
|
||||
"eventId": "evt-2025-01-15-abc123",
|
||||
"eventType": "scan.completed",
|
||||
"timestamp": "2025-01-15T10:30:00Z",
|
||||
"tenantId": "tenant-xyz",
|
||||
"subjectId": "image:sha256:abc123",
|
||||
"payload": { /* event-specific data */ }
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4) REST API
|
||||
|
||||
```
|
||||
GET /timeline?eventType=&source=&correlationId=&traceId=&severity=&since=&after=&limit=
|
||||
GET /timeline/{eventId}
|
||||
GET /timeline/{eventId}/evidence
|
||||
POST /timeline/events
|
||||
|
||||
# Gateway microservice aliases
|
||||
GET /api/v1/timeline
|
||||
GET /api/v1/timeline/{eventId}
|
||||
GET /api/v1/timeline/{eventId}/evidence
|
||||
POST /api/v1/timeline/events
|
||||
|
||||
GET /healthz | /readyz | /metrics
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Related Documentation
|
||||
|
||||
* Signals: `../signals/architecture.md`
|
||||
* Scanner: `../scanner/architecture.md`
|
||||
127
docs-archived/modules/timeline-indexer/guides/timeline.md
Normal file
127
docs-archived/modules/timeline-indexer/guides/timeline.md
Normal file
@@ -0,0 +1,127 @@
|
||||
# Timeline Forensics Guide
|
||||
|
||||
> **Imposed rule:** Timeline is append-only; events may not be deleted or rewritten. Redactions require creating a compensating `redaction_notice` event that references the original ULID.
|
||||
|
||||
The Timeline Indexer service aggregates structured events (scanner runs, policy verdicts, runtime posture, evidence locker activity) so operators can audit changes over time. This guide summarizes the event schema, query surfaces, persistence implementation, and integration points.
|
||||
|
||||
## 1. Event Model
|
||||
|
||||
| Field | Description |
|
||||
| --- | --- |
|
||||
| `event_id` | ULID identifying the event. |
|
||||
| `tenant` | Tenant scope. |
|
||||
| `timestamp` | UTC ISO-8601 time the event occurred. |
|
||||
| `category` | Logical grouping (scanner, policy, runtime, evidence). |
|
||||
| `details` | JSON payload describing the event; contract defined per producer. |
|
||||
| `trace_id` | Optional distributed trace correlation ID. |
|
||||
|
||||
Events are stored append-only with tenant-specific partitions. Producers include Scanner WebService, Policy Engine, Zastava Observer, Evidence Locker, and Notify.
|
||||
|
||||
### Event kinds (normative)
|
||||
|
||||
- `scan.completed` - scanner job finished; includes SBOM digest, findings counts, determinism score.
|
||||
- `policy.verdict` - policy engine decision with overlay cache version and allow/deny rationale.
|
||||
- `attestation.verified` - attestation verification result with Rekor UUID and bundle digest.
|
||||
- `evidence.ingested` - Evidence Locker write with WORM vault identifier.
|
||||
- `notify.sent` - outbound notification with target channel and template id.
|
||||
- `runtime.alert` - runtime enforcement or observation event from Zastava Observer.
|
||||
- `redaction_notice` - compensating entry when data is logically withdrawn; must include `supersedes` ULID.
|
||||
|
||||
## 2. APIs
|
||||
|
||||
Native endpoints:
|
||||
- `GET /timeline` - query timeline entries with filter parameters.
|
||||
- `GET /timeline/{eventId}` - fetch a single timeline entry.
|
||||
- `GET /timeline/{eventId}/evidence` - fetch evidence linked to a timeline entry.
|
||||
- `POST /timeline/events` - ingestion ack endpoint.
|
||||
|
||||
Router/Gateway aliases for microservice transport routing:
|
||||
- `GET /api/v1/timeline`
|
||||
- `GET /api/v1/timeline/{eventId}`
|
||||
- `GET /api/v1/timeline/{eventId}/evidence`
|
||||
- `POST /api/v1/timeline/events`
|
||||
|
||||
API headers: `X-Stella-Tenant`, optional `X-Stella-TraceId`, and `If-None-Match` for cache revalidation.
|
||||
|
||||
## 3. Query Tips
|
||||
|
||||
- Use `category` + `trace_id` to follow a scan-to-policy-to-notification flow.
|
||||
- Combine `tenant` and `timestamp` filters for SLA audits.
|
||||
- CLI command `stella timeline list` mirrors the API for automation.
|
||||
- For WORM verification, filter `category=evidence` and join on Evidence Locker bundle digest.
|
||||
- Use `category=attestation.verified` and `details.rekor_uuid` to reconcile transparency proofs.
|
||||
|
||||
Example queries:
|
||||
|
||||
```sh
|
||||
# Recent scan -> policy -> notify chain for a digest
|
||||
stella timeline list --tenant acme --category scan.completed --subject sha256:abc... --limit 5
|
||||
stella timeline list --tenant acme --category policy.verdict --trace-id <trace>
|
||||
stella timeline list --tenant acme --category notify.sent --trace-id <trace>
|
||||
|
||||
# Export window for audit
|
||||
curl -H "X-Stella-Tenant: acme" \
|
||||
"https://console.example/api/v1/timeline/export?from=2025-11-01T00:00:00Z&to=2025-11-02T00:00:00Z" \
|
||||
-o timeline-2025-11-01.ndjson
|
||||
```
|
||||
|
||||
## 4. Persistence (EF Core)
|
||||
|
||||
- TimelineIndexer DAL is EF Core 10 based (`StellaOps.TimelineIndexer.Infrastructure/Db`).
|
||||
- Database-first scaffolded context/models are under:
|
||||
- `src/TimelineIndexer/StellaOps.TimelineIndexer/StellaOps.TimelineIndexer.Infrastructure/EfCore/Context`
|
||||
- `src/TimelineIndexer/StellaOps.TimelineIndexer/StellaOps.TimelineIndexer.Infrastructure/EfCore/Models`
|
||||
- Keep generated files regeneration-safe. Place manual fixes in partial classes/files.
|
||||
|
||||
Scaffold command used for baseline generation:
|
||||
|
||||
```sh
|
||||
dotnet ef dbcontext scaffold "Host=localhost;Port=55433;Database=postgres;Username=postgres;Password=postgres" \
|
||||
Npgsql.EntityFrameworkCore.PostgreSQL \
|
||||
--project src/TimelineIndexer/StellaOps.TimelineIndexer/StellaOps.TimelineIndexer.Infrastructure/StellaOps.TimelineIndexer.Infrastructure.csproj \
|
||||
--startup-project src/TimelineIndexer/StellaOps.TimelineIndexer/StellaOps.TimelineIndexer.WebService/StellaOps.TimelineIndexer.WebService.csproj \
|
||||
--context TimelineIndexerDbContext \
|
||||
--context-dir EfCore/Context \
|
||||
--output-dir EfCore/Models \
|
||||
--schema timeline \
|
||||
--table timeline_events \
|
||||
--table timeline_event_details \
|
||||
--table timeline_event_digests \
|
||||
--no-onconfiguring \
|
||||
--use-database-names \
|
||||
--force
|
||||
```
|
||||
|
||||
Expected scaffold caveat: enum `timeline.event_severity` and composite FKs on `(event_id, tenant_id)` are not fully inferred by the scaffold command; these are restored in partial mapping files under `EfCore/Context` and `EfCore/Models`.
|
||||
|
||||
Generate/refresh compiled EF model artifacts:
|
||||
|
||||
```sh
|
||||
dotnet ef dbcontext optimize \
|
||||
--project src/TimelineIndexer/StellaOps.TimelineIndexer/StellaOps.TimelineIndexer.Infrastructure/StellaOps.TimelineIndexer.Infrastructure.csproj \
|
||||
--startup-project src/TimelineIndexer/StellaOps.TimelineIndexer/StellaOps.TimelineIndexer.WebService/StellaOps.TimelineIndexer.WebService.csproj \
|
||||
--context TimelineIndexerDbContext \
|
||||
--output-dir EfCore/CompiledModels \
|
||||
--namespace StellaOps.TimelineIndexer.Infrastructure.EfCore.CompiledModels
|
||||
```
|
||||
|
||||
Design-time connection can be overridden with `STELLAOPS_TIMELINEINDEXER_EF_CONNECTION`.
|
||||
|
||||
Runtime usage is explicit in `TimelineIndexerDbContextFactory` via `UseModel(TimelineIndexerDbContextModel.Instance)` so the static compiled model module is always used by DAL stores.
|
||||
|
||||
## 5. Integration
|
||||
|
||||
- Evidence Locker attaches evidence bundle digests; the console links from timeline to evidence viewer.
|
||||
- Notifier creates acknowledgement events for incident workflows.
|
||||
- Offline kits package timeline exports for compliance reviews.
|
||||
|
||||
Retention: events are retained per-tenant for at least `timeline.retention_days` (default 400 days) and replicated to cold storage weekly. Index rebuilds must preserve ordering and ULIDs.
|
||||
|
||||
Privacy/PII: producers must redact PII before emission; once emitted, redactions occur via `redaction_notice` only.
|
||||
|
||||
## 6. References
|
||||
|
||||
- `docs/modules/telemetry/architecture.md`
|
||||
- `docs/modules/zastava/architecture.md`
|
||||
- `docs/modules/export-center/architecture.md`
|
||||
- `src/TimelineIndexer/StellaOps.TimelineIndexer`
|
||||
Reference in New Issue
Block a user