themesd advisories enhanced

This commit is contained in:
StellaOps Bot
2025-12-14 21:29:44 +02:00
parent 9202cd7da8
commit 3411e825cd
10 changed files with 359 additions and 20 deletions

View File

@@ -8,6 +8,19 @@
---
## 0. POSTGRESQL VS MONGODB (DECISION RULE)
Default posture:
- **System of record**: PostgreSQL (JSONB-first, per-module schema isolation).
- **Queues & coordination**: PostgreSQL (`SKIP LOCKED`, advisory locks when needed).
- **Cache/acceleration only**: Valkey/Redis (ephemeral).
- **MongoDB**: only when you have a *clear* need for very large, read-optimized snapshot workloads (e.g., extremely large historical graphs), and you can regenerate those snapshots deterministically from the Postgres source-of-truth.
When MongoDB is justified:
- Interactive exploration over hundreds of millions of nodes/edges where denormalized reads beat relational joins.
- Snapshot cadence is batchy (hourly/daily) and you can re-emit snapshots deterministically.
- You need to isolate read spikes from transactional control-plane writes.
## 1. MODULE-SCHEMA MAPPING
| Module | Schema | Primary Tables |
@@ -132,6 +145,23 @@ where j.id = cte.id
returning j.*;
```
### 3.3.1 Advisory Locks (coordination / idempotency guards)
Use advisory locks for per-tenant singleton work or "at-most-once" critical sections (do not hold them while doing long-running work):
```sql
-- Acquire (per tenant, per artifact) for the duration of the transaction
select pg_try_advisory_xact_lock(hashtextextended('recalc:' || $1 || ':' || $2, 0));
```
### 3.3.2 LISTEN/NOTIFY (nudge, not a durable queue)
Use `LISTEN/NOTIFY` to wake workers quickly after inserting work into a durable table:
```sql
notify stella_scan, json_build_object('purl', $1, 'priority', 5)::text;
```
### 3.4 Temporal Pattern (Unknowns)
```sql
@@ -281,6 +311,50 @@ create index brin_scan_events_time
- Derived data modeled as projection tables or materialized views
- Idempotency enforced in DB: unique keys for imports/jobs/results
### 8.4 Materialized Views vs Projection Tables
Materialized views are acceptable when:
- You can refresh them deterministically at a defined cadence (owned by a specific worker/job).
- You can afford full refresh cost, or the dataset is bounded.
- You provide a unique index to enable `REFRESH MATERIALIZED VIEW CONCURRENTLY`.
Prefer projection tables when:
- You need incremental updates (on import/scan completion).
- You need deterministic point-in-time snapshots per scan manifest (replay/audit).
- Refresh cost would scale with the entire dataset on every change.
Checklist:
- Every derived read model declares: owner, refresh cadence/trigger, retention, and idempotency key.
- No UI/API endpoint depends on a heavy non-materialized view for hot paths.
### 8.5 Queue + Outbox Rules (avoid deadlocks)
Queue claim rules:
- Claim in a short transaction (commit immediately after lock acquisition).
- Do work outside the transaction.
- On failure: increment attempts, compute backoff into `run_after`, and release locks.
- Define a DLQ condition (`attempts > N`) that is queryable and observable.
Outbox dispatch rules:
- Dispatch is idempotent (consumer must tolerate duplicates).
- The dispatcher writes a stable delivery attempt record (`dispatched_at`, `dispatch_attempts`, `error`).
### 8.6 Migration Safety Rules
- Create/drop indexes concurrently on large tables (`CREATE INDEX CONCURRENTLY`, `DROP INDEX CONCURRENTLY`).
- Add `NOT NULL` in stages: add nullable column → backfill in batches → enforce constraint → then add default (if needed).
- Avoid long-running `ALTER TABLE` on high-volume tables without a lock plan.
### 8.7 Definition of Done (new table/view)
A PR adding a table/view is incomplete unless it includes:
- Table classification (SoR / projection / queue / event).
- Primary key + idempotency unique key.
- Tenant scoping strategy (and RLS policy when applicable).
- Index plan mapped to the top 13 query patterns (include `EXPLAIN (ANALYZE, BUFFERS)` output).
- Retention plan (partitioning and drop policy for high-volume tables).
- Refresh/update plan for derived models (owner + cadence).
## 9. FEATURE FLAG SCHEMA
```sql