themesd advisories enhanced
This commit is contained in:
@@ -8,6 +8,19 @@
|
||||
|
||||
---
|
||||
|
||||
## 0. POSTGRESQL VS MONGODB (DECISION RULE)
|
||||
|
||||
Default posture:
|
||||
- **System of record**: PostgreSQL (JSONB-first, per-module schema isolation).
|
||||
- **Queues & coordination**: PostgreSQL (`SKIP LOCKED`, advisory locks when needed).
|
||||
- **Cache/acceleration only**: Valkey/Redis (ephemeral).
|
||||
- **MongoDB**: only when you have a *clear* need for very large, read-optimized snapshot workloads (e.g., extremely large historical graphs), and you can regenerate those snapshots deterministically from the Postgres source-of-truth.
|
||||
|
||||
When MongoDB is justified:
|
||||
- Interactive exploration over hundreds of millions of nodes/edges where denormalized reads beat relational joins.
|
||||
- Snapshot cadence is batchy (hourly/daily) and you can re-emit snapshots deterministically.
|
||||
- You need to isolate read spikes from transactional control-plane writes.
|
||||
|
||||
## 1. MODULE-SCHEMA MAPPING
|
||||
|
||||
| Module | Schema | Primary Tables |
|
||||
@@ -132,6 +145,23 @@ where j.id = cte.id
|
||||
returning j.*;
|
||||
```
|
||||
|
||||
### 3.3.1 Advisory Locks (coordination / idempotency guards)
|
||||
|
||||
Use advisory locks for per-tenant singleton work or "at-most-once" critical sections (do not hold them while doing long-running work):
|
||||
|
||||
```sql
|
||||
-- Acquire (per tenant, per artifact) for the duration of the transaction
|
||||
select pg_try_advisory_xact_lock(hashtextextended('recalc:' || $1 || ':' || $2, 0));
|
||||
```
|
||||
|
||||
### 3.3.2 LISTEN/NOTIFY (nudge, not a durable queue)
|
||||
|
||||
Use `LISTEN/NOTIFY` to wake workers quickly after inserting work into a durable table:
|
||||
|
||||
```sql
|
||||
notify stella_scan, json_build_object('purl', $1, 'priority', 5)::text;
|
||||
```
|
||||
|
||||
### 3.4 Temporal Pattern (Unknowns)
|
||||
|
||||
```sql
|
||||
@@ -281,6 +311,50 @@ create index brin_scan_events_time
|
||||
- Derived data modeled as projection tables or materialized views
|
||||
- Idempotency enforced in DB: unique keys for imports/jobs/results
|
||||
|
||||
### 8.4 Materialized Views vs Projection Tables
|
||||
|
||||
Materialized views are acceptable when:
|
||||
- You can refresh them deterministically at a defined cadence (owned by a specific worker/job).
|
||||
- You can afford full refresh cost, or the dataset is bounded.
|
||||
- You provide a unique index to enable `REFRESH MATERIALIZED VIEW CONCURRENTLY`.
|
||||
|
||||
Prefer projection tables when:
|
||||
- You need incremental updates (on import/scan completion).
|
||||
- You need deterministic point-in-time snapshots per scan manifest (replay/audit).
|
||||
- Refresh cost would scale with the entire dataset on every change.
|
||||
|
||||
Checklist:
|
||||
- Every derived read model declares: owner, refresh cadence/trigger, retention, and idempotency key.
|
||||
- No UI/API endpoint depends on a heavy non-materialized view for hot paths.
|
||||
|
||||
### 8.5 Queue + Outbox Rules (avoid deadlocks)
|
||||
|
||||
Queue claim rules:
|
||||
- Claim in a short transaction (commit immediately after lock acquisition).
|
||||
- Do work outside the transaction.
|
||||
- On failure: increment attempts, compute backoff into `run_after`, and release locks.
|
||||
- Define a DLQ condition (`attempts > N`) that is queryable and observable.
|
||||
|
||||
Outbox dispatch rules:
|
||||
- Dispatch is idempotent (consumer must tolerate duplicates).
|
||||
- The dispatcher writes a stable delivery attempt record (`dispatched_at`, `dispatch_attempts`, `error`).
|
||||
|
||||
### 8.6 Migration Safety Rules
|
||||
|
||||
- Create/drop indexes concurrently on large tables (`CREATE INDEX CONCURRENTLY`, `DROP INDEX CONCURRENTLY`).
|
||||
- Add `NOT NULL` in stages: add nullable column → backfill in batches → enforce constraint → then add default (if needed).
|
||||
- Avoid long-running `ALTER TABLE` on high-volume tables without a lock plan.
|
||||
|
||||
### 8.7 Definition of Done (new table/view)
|
||||
|
||||
A PR adding a table/view is incomplete unless it includes:
|
||||
- Table classification (SoR / projection / queue / event).
|
||||
- Primary key + idempotency unique key.
|
||||
- Tenant scoping strategy (and RLS policy when applicable).
|
||||
- Index plan mapped to the top 1–3 query patterns (include `EXPLAIN (ANALYZE, BUFFERS)` output).
|
||||
- Retention plan (partitioning and drop policy for high-volume tables).
|
||||
- Refresh/update plan for derived models (owner + cadence).
|
||||
|
||||
## 9. FEATURE FLAG SCHEMA
|
||||
|
||||
```sql
|
||||
|
||||
Reference in New Issue
Block a user