docs consolidation work

2025-12-25 18:48:11 +02:00
parent 2a06f780cf
commit 0103defcff
114 changed files with 4143 additions and 2395 deletions
--- a/docs/implplan/archived/adr/0000-template.md
+++ b/docs/implplan/archived/adr/0000-template.md
@@ -0,0 +1,34 @@
+# ADR-0000: Title
+
+## Status
+Proposed
+
+## Date
+YYYY-MM-DD
+
+## Authors
+- Name (team)
+
+## Deciders
+- Names of approvers / reviewers
+
+## Context
+- What decision needs to be made?
+- What are the forces (requirements, constraints, stakeholders)?
+- Why now? What triggers the ADR?
+
+## Decision
+- Summary of the chosen option.
+- Key rationale points.
+
+## Consequences
+- Positive/negative consequences.
+- Follow-up actions or tasks.
+- Rollback plan or re-evaluation criteria.
+
+## Alternatives Considered
+- Option A — pros/cons.
+- Option B — pros/cons.
+
+## References
+- Links to related ADRs, issues, documents.
--- a/docs/implplan/archived/adr/0001-postgresql-for-control-plane.md
+++ b/docs/implplan/archived/adr/0001-postgresql-for-control-plane.md
@@ -0,0 +1,211 @@
+# ADR-0001: PostgreSQL for Control-Plane Storage
+
+## Status
+Accepted
+
+## Date
+2025-12-04
+
+## Authors
+- Platform Team
+
+## Deciders
+- Architecture Guild
+- Platform Team
+
+## Context
+
+StellaOps control-plane services (Authority, Scheduler, Notify, Concelier/Excititor, Policy) require persistent storage for:
+
+- Identity and authorization data (users, roles, tokens, sessions)
+- Job scheduling and execution state
+- Notification rules, templates, and delivery tracking
+- Vulnerability advisories and VEX statements
+- Policy packs, rules, and evaluation history
+
+**Triggers for this decision:**
+
+1. **Licensing trust & ecosystem stability** — PostgreSQL is licensed under the permissive PostgreSQL License (similar to MIT/BSD), OSI-approved, with no vendor lock-in concerns. MongoDB's SSPL license (2018) is not OSI-approved and creates uncertainty for self-hosted/sovereign deployments. For a platform emphasizing sovereignty and auditability, database licensing must be beyond reproach.
+2. **Schema complexity** — Control-plane domains have well-defined, relational schemas with referential integrity requirements (foreign keys, cascading deletes, constraints).
+3. **Query patterns** — Complex joins, aggregations, and window functions are common (e.g., finding all images affected by a newly published CVE).
+4. **ACID requirements** — Job scheduling, token issuance, and notification delivery require strong transactional guarantees.
+5. **Multi-tenancy** — Row-level security (RLS) needed for tenant isolation without schema-per-tenant overhead.
+6. **Migration tooling** — Need deterministic, forward-only migrations with advisory lock coordination for multi-instance deployments.
+7. **Air-gap operation** — All schema and data must be embeddable in assemblies without external network dependencies.
+8. **Auditability** — PostgreSQL's mature ecosystem includes proven audit logging, compliance tooling, and forensic capabilities trusted by regulated industries.
+
+## Decision
+
+**Adopt PostgreSQL (≥16) as the primary database for all StellaOps control-plane domains.**
+
+Key architectural choices:
+
+### 1. Per-Module Schema Isolation
+
+Each module owns exactly one PostgreSQL schema:
+
+| Schema | Owner | Description |
+|--------|-------|-------------|
+| `auth` | Authority | Identity, authentication, authorization, licensing |
+| `vuln` | Concelier | Vulnerability advisories, sources, affected packages |
+| `vex` | Excititor | VEX statements, graphs, observations, consensus |
+| `scheduler` | Scheduler | Jobs, triggers, workers, execution history |
+| `notify` | Notify | Channels, templates, rules, deliveries |
+| `policy` | Policy | Policy packs, rules, risk profiles |
+| `audit` | Shared | Cross-cutting audit log (optional) |
+
+**Rationale:**
+- Clear ownership boundaries
+- Independent migration lifecycles
+- Schema-level access control
+- Simplified testing and development
+
+### 2. Multi-Tenancy via tenant_id Column
+
+Single database, single schema set, `tenant_id` column on all tenant-scoped tables.
+
+```sql
+-- Session-level tenant context
+SET app.tenant_id = '<tenant-uuid>';
+
+-- Row-level security (defense in depth)
+CREATE POLICY tenant_isolation ON <table>
+    USING (tenant_id = current_setting('app.tenant_id')::uuid);
+```
+
+**Rationale:**
+- Simplest operational model
+- Shared connection pooling
+- Easy cross-tenant queries for admin operations
+- Composite indexes on `(tenant_id, ...)` for query performance
+
+### 3. Forward-Only Migrations with Advisory Locks
+
+Migrations are embedded in assemblies and executed at startup with PostgreSQL advisory locks:
+
+```sql
+SELECT pg_try_advisory_lock(hashtext('auth'));  -- Per-schema lock
+```
+
+**Migration categories:**
+- **Startup (001-099)**: Automatic, non-breaking DDL
+- **Release (100-199)**: Manual CLI, breaking changes
+- **Seed (S001-S999)**: Reference data
+- **Data (DM001-DM999)**: Batched background jobs
+
+**Rationale:**
+- No down migrations needed (forward-only with fix-forward)
+- Advisory locks prevent concurrent migrations across instances
+- Checksum validation catches unauthorized modifications
+- Air-gap compatible (no external migration service needed)
+
+### 4. RustFS for Binary Artifacts
+
+PostgreSQL stores metadata and indexes; RustFS stores binary artifacts (SBOMs, attestations, reports):
+
+```
+PostgreSQL: Schema definitions, relationships, indexes, audit trails
+RustFS:     sbom.cdx.json.zst, inventory.cdx.pb, bom-index.bin, *.dsse.json
+```
+
+**Rationale:**
+- Right tool for each job
+- PostgreSQL excellent for structured queries
+- Object storage better for large binary blobs
+- Clear separation of concerns
+
+## Consequences
+
+### Positive
+
+1. **Licensing trust** — PostgreSQL License is permissive, OSI-approved, and universally accepted. No vendor lock-in, no license ambiguity for sovereign deployments. Trusted by governments, regulated industries, and security-conscious organizations.
+2. **Ecosystem stability** — 30+ years of development, included in all major distributions, no license rug-pulls. Community governance ensures long-term trust.
+3. **Relational integrity** — Foreign keys, constraints, and transactions ensure data consistency.
+4. **Query flexibility** — Complex joins, CTEs, window functions, and full-text search available natively.
+5. **Operational maturity** — Well-understood backup, replication, and monitoring ecosystem.
+6. **Row-level security** — Built-in multi-tenancy support without application-layer hacks.
+7. **Schema evolution** — Mature migration tooling with online DDL capabilities.
+8. **Performance** — Excellent query planning, connection pooling (PgBouncer), and indexing options.
+9. **Auditability** — Proven audit logging extensions (pgAudit), compliance certifications, forensic tooling.
+
+### Negative
+
+1. **Schema rigidity** — Changes require migrations; less flexible than document stores for rapidly evolving schemas.
+2. **Operational overhead** — Requires PostgreSQL expertise for tuning, vacuuming, and monitoring.
+3. **Connection limits** — Need PgBouncer for high-concurrency workloads.
+
+### Follow-up Actions
+
+- [x] Create `docs/db/` documentation directory with specification, rules, and conversion plan
+- [x] Define migration infrastructure in `StellaOps.Infrastructure.Postgres`
+- [x] Complete phased conversion from MongoDB per `docs/db/tasks/PHASE_*.md` — **COMPLETED in Sprint 4400 (November 2024)**
+- [x] Update deployment guides for PostgreSQL requirements — **COMPLETED in Sprint 4400**
+- [x] Add PostgreSQL health checks to all control-plane services — **COMPLETED in Sprint 4400**
+
+### Completion Status
+
+**MongoDB removal completed in Sprint 4400 (November 2024).** All control-plane services now use PostgreSQL 16+. MongoDB dependencies removed from all modules. This ADR is now a historical record of the architectural decision.
+
+### Rollback Criteria
+
+Revert to MongoDB (or hybrid) if:
+- Migration performance unacceptable (> 60s startup time)
+- Query complexity exceeds PostgreSQL capabilities
+- Operational burden exceeds team capacity
+
+## Alternatives Considered
+
+### Option A: Continue with MongoDB
+
+**Pros:**
+- Already in use for some components
+- Flexible schema
+- Good for document-centric workloads
+
+**Cons:**
+- **Licensing uncertainty** — MongoDB's SSPL (Server Side Public License, 2018) is not OSI-approved. Creates legal ambiguity for sovereign/self-hosted deployments, especially in regulated industries and government contexts where license provenance matters.
+- **Ecosystem trust erosion** — SSPL switch caused major distributions (Debian, Fedora, RHEL) to drop MongoDB packages. Sovereign customers may have policies against non-OSI licenses.
+- No referential integrity (app-enforced)
+- Limited join capabilities
+- Multi-tenancy requires additional logic
+- No row-level security
+- Less mature migration tooling
+
+**Rejected because:** Licensing uncertainty is incompatible with StellaOps' sovereign-first positioning. Control-plane domains are also fundamentally relational with strong consistency requirements.
+
+### Option B: Hybrid (PostgreSQL + MongoDB)
+
+**Pros:**
+- Use each database for appropriate workloads
+- Gradual migration possible
+
+**Cons:**
+- Two databases to operate and monitor
+- Complex deployment
+- Cross-database consistency challenges
+- Higher operational burden
+
+**Rejected because:** Unified PostgreSQL approach is simpler and sufficient for all control-plane needs.
+
+### Option C: CockroachDB / YugabyteDB
+
+**Pros:**
+- PostgreSQL-compatible
+- Built-in horizontal scaling
+- Multi-region capabilities
+
+**Cons:**
+- Additional operational complexity
+- Less mature than PostgreSQL
+- Overkill for current scale
+- Air-gap deployment challenges
+
+**Rejected because:** PostgreSQL provides sufficient scale and simpler operations for current requirements. Can revisit if horizontal scaling becomes necessary.
+
+## References
+
+- [`docs/db/README.md`](../db/README.md) — Database documentation index
+- [`docs/db/SPECIFICATION.md`](../db/SPECIFICATION.md) — Schema design specification
+- [`docs/db/MIGRATION_STRATEGY.md`](../db/MIGRATION_STRATEGY.md) — Migration execution strategy
+- [`docs/db/RULES.md`](../db/RULES.md) — Database coding rules
+- [`docs/07_HIGH_LEVEL_ARCHITECTURE.md`](../07_HIGH_LEVEL_ARCHITECTURE.md) — High-level architecture overview
--- a/docs/implplan/archived/adr/index.md
+++ b/docs/implplan/archived/adr/index.md
@@ -0,0 +1,48 @@
+# Architecture Decision Records (ADRs)
+
+Architecture Decision Records document long-lived choices that shape StellaOps architecture, security posture, and operator experience. They complement RFCs by capturing the final call and the context that led to it.
+
+## When to file an ADR
+- Decisions that affect cross-module contracts, persistence models, or external interfaces.
+- Security or compliance controls with on-going operational ownership.
+- Rollout strategies that require coordination across guilds or sprints.
+- Reversals or deprecations of previously accepted ADRs.
+
+Small, module-local refactors that do not modify public behaviour can live in commit messages instead.
+
+## Workflow at a glance
+1. Copy `docs/adr/0000-template.md` to `docs/adr/NNNN-short-slug.md` with a zero-padded sequence (see **Numbering**).
+2. Fill in context, decision, consequences, and alternatives. Include links to RFCs, issues, benchmarks, or experiments.
+3. Request async review from the impacted guilds. Capture sign-offs in the **Deciders** field.
+4. Merge the ADR with the code/config changes (or in a preparatory PR).
+5. Announce the accepted ADR in the Docs Guild channel or sprint notes so downstream teams can consume it.
+
+## Numbering and status
+- Use zero-padded integers (e.g., `0001`, `0002`) in file names and the document header. Increment from the highest existing number.
+- Valid statuses: `Proposed`, `Accepted`, `Rejected`, `Deprecated`, `Superseded`. Update the status when follow-up work lands.
+- When an ADR supersedes another, link them in both documents’ **References** sections.
+
+## Review expectations
+- Highlight edge-case handling, trade-offs, and determinism requirements.
+- Include operational checklists for any new runtime path (quota updates, schema migrations, credential rotation, etc.).
+- Attach diagrams under `docs/adr/assets/` when visuals improve comprehension.
+- Add TODO tasks for follow-up work in the relevant module’s `TASKS.md` and link them from the ADR.
+
+## Verification checklist
+- [ ] `Status`, `Date`, `Authors`, and `Deciders` populated.
+- [ ] Links to code/config PRs or experiments recorded under **References**.
+- [ ] Consequences call out migration or rollback steps.
+- [ ] Announcement posted to Docs Guild updates (or sprint log).
+
+## ADR Index
+
+| ADR | Title | Status | Date |
+|-----|-------|--------|------|
+| [0001](./0001-postgresql-for-control-plane.md) | PostgreSQL for Control-Plane Storage | Accepted | 2025-12-04 |
+
+## Related resources
+- [Docs Guild Task Board](../TASKS.md)
+- [High-Level Architecture Overview](../07_HIGH_LEVEL_ARCHITECTURE.md)
+- [Database Documentation](../db/README.md)
+- [Coding Standards](../18_CODING_STANDARDS.md)
+- [Release Engineering Playbook](../13_RELEASE_ENGINEERING_PLAYBOOK.md)