chore: remove outdated documentation and prep notes

- Deleted several draft and prep documents related to benchmarks, authority DPoP & mTLS implementation, Java analyzer observation, link-not-merge determinism tests, replay operations, and crypto provider registry.
- Updated the merge semver playbook to reflect current database schema usage.
- Cleaned up the technical development README to remove references to obsolete documents and streamline guidance for contributors.
This commit is contained in:
StellaOps Bot
2025-12-24 12:47:50 +02:00
parent 02772c7a27
commit 40362de568
20 changed files with 6 additions and 758 deletions

View File

@@ -1,146 +0,0 @@
# Authority DPoP & mTLS Implementation Plan (2025-10-19)
## Purpose
- Provide the implementation blueprint for AUTH-DPOP-11-001 and AUTH-MTLS-11-002.
- Unify sender-constraint validation across Authority, downstream services, and clients.
- Capture deterministic, testable steps that unblock UI/Signer guilds depending on DPoP/mTLS hardening.
## Scope
- Token endpoint validation, issuance, and storage changes inside `StellaOps.Authority`.
- Shared security primitives consumed by Authority, Scanner, Signer, CLI, and UI.
- Operator-facing configuration, auditing, and observability.
- Out of scope: PoE enforcement (Signer) and CLI/UI client UX; those teams consume the new capabilities.
> **Status update (2025-10-19):** `ValidateDpopProofHandler`, `AuthorityClientCertificateValidator`, and the supporting storage/audit plumbing now live in `src/Authority/StellaOps.Authority`. DPoP proofs populate `cnf.jkt`, mTLS bindings enforce certificate thumbprints via `cnf.x5t#S256`, and token documents persist the sender constraint metadata. In-memory nonce issuance is wired (Redis implementation to follow). Documentation and configuration references were updated (`docs/11_AUTHORITY.md`). Targeted unit/integration tests were added; running the broader test suite is currently blocked by pre-existing `StellaOps.Concelier.Storage.Mongo` build errors.
>
> **Status update (2025-10-20):** Redis-backed nonce configuration is exposed through `security.senderConstraints.dpop.nonce` with sample YAML (`etc/authority.yaml.sample`) and architecture docs refreshed. Operator guide now includes concrete Redis/required audiences snippet; nonce challenge regression remains covered by `ValidateDpopProof_IssuesNonceChallenge_WhenNonceMissing`.
>
> **Status update (2025-10-23):** mTLS enforcement now honours `security.senderConstraints.mtls.enforceForAudiences`, automatically rejecting non-mTLS clients targeting audiences such as `signer`. Certificate bindings validate thumbprint, issuer, subject, serial number, and SAN values, producing deterministic error codes for operators. Introspection responses include `cnf.x5t#S256`, and new unit tests cover audience enforcement, binding mismatches, and bootstrap storage. Docs/sample config updated accordingly.
## Design Summary
- Extract the existing Scanner `DpopProofValidator` stack into a shared `StellaOps.Auth.Security` library used by Authority and resource servers.
- Extend Authority configuration (`authority.yaml`) with strongly-typed `senderConstraints.dpop` and `senderConstraints.mtls` sections (map to sample already shown in architecture doc).
- Require DPoP proofs on `/token` when the registered client policy is `senderConstraint=dpop`; bind issued access tokens via `cnf.jkt`.
- Introduce Authority-managed nonce issuance for “high value” audiences (default: `signer`, `attestor`) with Redis-backed persistence and deterministic auditing.
- Enable OAuth 2.0 mTLS (RFC 8705) by storing certificate bindings per client, requesting client certificates at TLS termination, and stamping `cnf.x5t#S256` into issued tokens plus introspection output.
- Surface structured logs and counters for both DPoP and mTLS flows; provide integration tests that cover success, replay, invalid proof, and certificate mismatch cases.
## AUTH-DPOP-11-001 — Proof Validation & Nonce Handling
**Shared validator**
- Move `DpopProofValidator`, option types, and replay cache interfaces from `StellaOps.Scanner.Core` into a new assembly `StellaOps.Auth.Security`.
- Provide pluggable caches: `InMemoryDpopReplayCache` (existing) and new `RedisDpopReplayCache` (leveraging the Authority Redis connection).
- Ensure the validator exposes the validated `SecurityKey`, `jti`, and `iat` so Authority can construct the `cnf` claim and compute nonce expiry.
**Configuration model**
- Extend `StellaOpsAuthorityOptions.Security` with a `SenderConstraints` property containing:
- `Dpop` (`enabled`, `allowedAlgorithms`, `maxAgeSeconds`, `clockSkewSeconds`, `replayWindowSeconds`, `nonce` settings with `enabled`, `ttlSeconds`, `requiredAudiences`, `maxIssuancePerMinute`).
- `Mtls` (`enabled`, `requireChainValidation`, `clientCaBundle`, `allowedSubjectPatterns`, `allowedSanTypes`).
- Bind from YAML (`authority.security.senderConstraints.*`) while preserving backwards compatibility (defaults keep both disabled).
**Token endpoint pipeline**
- Introduce a scoped OpenIddict handler `ValidateDpopProofHandler` inserted before `ValidateClientCredentialsHandler`.
- Determine the required sender constraint from client metadata:
- Add `AuthorityClientMetadataKeys.SenderConstraint` storing `dpop` or `mtls`.
- Optionally allow per-client overrides for nonce requirement.
- When `dpop` is required:
- Read the `DPoP` header from the ASP.NET request, reject with `invalid_token` + `WWW-Authenticate: DPoP error="invalid_dpop_proof"` if absent.
- Call the shared validator with method/URI. Enforce algorithm allowlist and `iat` window from options.
- Persist the `jkt` thumbprint plus replay cache state in the OpenIddict transaction (`AuthorityOpenIddictConstants.DpopKeyThumbprintProperty`, `DpopIssuedAtProperty`).
- When the requested audience intersects `SenderConstraints.Dpop.Nonce.RequiredAudiences`, require `nonce` in the proof; on first failure respond with HTTP 401, `error="use_dpop_nonce"`, and include `DPoP-Nonce` header (see nonce note below). Cache the rejection reason for audit logging.
**Nonce service**
- Add `IDpopNonceStore` with methods `IssueAsync(audience, clientId, jkt)` and `TryConsumeAsync(nonce, audience, clientId, jkt)`.
- Default implementation `RedisDpopNonceStore` storing SHA-256 hashes of nonces keyed by `audience:clientId:jkt`. TTL comes from `SenderConstraints.Dpop.Nonce.Ttl`.
- Create helper `DpopNonceIssuer` used by `ValidateDpopProofHandler` to issue nonces when missing/expired, enforcing issuance rate limits (per options) and tagging audit/log records.
- On successful validation (nonce supplied and consumed) stamp metadata into the transaction for auditing.
- Update `ClientCredentialsHandlers` to observe nonce enforcement: when a nonce challenge was sent, emit structured audit with `nonce_issued`, `audiences`, and `retry`.
**Token issuance**
- In `HandleClientCredentialsHandler`, if the transaction contains a validated DPoP key:
- Build `cnf.jkt` using thumbprint from validator.
- Include `auth_time`/`dpop_jti` as needed for diagnostics.
- Persist the thumbprint alongside token metadata in Mongo (extend `AuthorityTokenDocument` with `SenderConstraint`, `KeyThumbprint`, `Nonce` fields).
**Auditing & observability**
- Emit new audit events:
- `authority.dpop.proof.validated` (success/failure, clientId, audience, thumbprint, nonce status, jti).
- `authority.dpop.nonce.issued` and `authority.dpop.nonce.consumed`.
- Metrics (Prometheus style):
- `authority_dpop_validations_total{result,reason}`.
- `authority_dpop_nonce_issued_total{audience}` and `authority_dpop_nonce_fails_total{reason}`.
- Structured logs include `authority.sender_constraint=dpop`, `authority.dpop_thumbprint`, `authority.dpop_nonce`.
**Testing**
- Unit tests for the handler pipeline using fake OpenIddict transactions.
- Replay/nonce tests with in-memory and Redis stores.
- Integration tests in `StellaOps.Authority.Tests` covering:
- Valid DPoP proof issuing `cnf.jkt`.
- Missing header → challenge with nonce.
- Replayed `jti` rejected.
- Invalid nonce rejected even after issuance.
- Contract tests to ensure `/.well-known/openid-configuration` advertises `dpop_signing_alg_values_supported` and `dpop_nonce_supported` when enabled.
## AUTH-MTLS-11-002 — Certificate-Bound Tokens
**Configuration model**
- Reuse `SenderConstraints.Mtls` described above; include:
- `enforceForAudiences` list (defaults `signer`, `attestor`, `scheduler`).
- `certificateRotationGraceSeconds` for overlap.
- `allowedClientCertificateAuthorities` absolute paths.
**Kestrel/TLS pipeline**
- Configure Kestrel with `ClientCertificateMode.AllowCertificate` globally and implement middleware that enforces certificate presence only when the resolved client requires mTLS.
- Add `IAuthorityClientCertificateValidator` that validates presented certificate chain, SANs (`dns`, `uri`, optional SPIFFE), and thumbprint matches one of the stored bindings.
- Cache validation results per connection id to avoid rehashing on every request.
**Client registration & storage**
- Extend `AuthorityClientDocument` with `List<AuthorityClientCertificateBinding>` containing:
- `Thumbprint`, `SerialNumber`, `Subject`, `NotBefore`, `NotAfter`, `Sans`, `CreatedAt`, `UpdatedAt`, `Label`.
- Provide admin API mutations (`/admin/clients/{id}/certificates`) for ops tooling (deferred implementation but schema ready).
- Update plugin provisioning store (`StandardClientProvisioningStore`) to map descriptors with certificate bindings and `senderConstraint`.
- Persist binding state in Mongo migrations (index on `{clientId, thumbprint}`).
**Token issuance & introspection**
- Add a transaction property capturing the validated client certificate thumbprint.
- `HandleClientCredentialsHandler`:
- When mTLS required, ensure certificate info present; reject otherwise.
- Stamp `cnf` claim: `principal.SetClaim("cnf", JsonSerializer.Serialize(new { x5t#S256 = thumbprint }))`.
- Store binding metadata in issued token document for audit.
- Update `ValidateAccessTokenHandler` and introspection responses to surface `cnf.x5t#S256`.
- Ensure refresh tokens (if ever enabled) copy the binding data.
**Auditing & observability**
- Audit events:
- `authority.mtls.handshake` (success/failure, clientId, thumbprint, issuer, subject).
- `authority.mtls.binding.missing` when a required client posts without a cert.
- Metrics:
- `authority_mtls_handshakes_total{result}`.
- `authority_mtls_certificate_rotations_total`.
- Logs include `authority.sender_constraint=mtls`, `authority.mtls_thumbprint`, `authority.mtls_subject`.
**Testing**
- Unit tests for certificate validation rules (SAN mismatches, expiry, CA trust).
- Integration tests running Kestrel with test certificates:
- Successful token issuance with bound certificate.
- Request without certificate → `invalid_client`.
- Token introspection reveals `cnf.x5t#S256`.
- Rotation scenario (old + new cert allowed during grace window).
## Implementation Checklist
**DPoP work-stream**
1. Extract shared validator into `StellaOps.Auth.Security`; update Scanner references.
2. Introduce configuration classes and bind from YAML/environment.
3. Implement nonce store (Redis + in-memory), handler integration, and OpenIddict transaction plumbing.
4. Stamp `cnf.jkt`, audit events, and metrics; update Mongo documents and migrations.
5. Extend docs: `docs/modules/authority/architecture.md`, `docs/security/audit-events.md`, `docs/security/rate-limits.md`, CLI/UI references.
**mTLS work-stream**
1. Extend client document/schema and provisioning stores with certificate bindings + sender constraint flag.
2. Configure Kestrel/middleware for optional client certificates and validation service.
3. Update token issuance/introspection to honour certificate bindings and emit `cnf.x5t#S256`.
4. Add auditing/metrics and integration tests (happy path + failure).
5. Refresh operator documentation (`docs/modules/authority/operations/backup-restore.md`, `docs/modules/authority/operations/monitoring.md`, sample `authority.yaml`) to cover certificate lifecycle.
Both streams should conclude with `dotnet test src/Authority/StellaOps.Authority/StellaOps.Authority.sln` and documentation cross-links so dependent guilds can unblock UI/Signer work.

View File

@@ -1,37 +0,0 @@
# Java Analyzer Observation Writer Plan
_Status: 2025-10-29_
SCANNER-ANALYZERS-JAVA-21-008 (resolver + AOC writer) is blocked by upstream heuristics that need to settle before we can emit observation JSON. This note itemises the remaining work so the analyzer guild can sequence delivery without re-opening design discussions in every stand-up.
## Prerequisite summary
- **SCANNER-ANALYZERS-JAVA-21-004** (reflection / dynamic loader heuristics) must emit normalized reflection edges with confidence + call-site metadata. Outstanding items: TCCL coverage for servlet containers and resource-based plugin hints. Owners: Java Analyzer Guild.
- **SCANNER-ANALYZERS-JAVA-21-005** (framework config extraction) required to surface Spring/Jakarta entrypoints that feed observation entrypoint metadata. Add YAML/property parsing fixtures and document reason codes (`config-spring`, `config-jaxrs`, etc.).
- **SCANNER-ANALYZERS-JAVA-21-006** (JNI/native hints) optional but highly recommended before observation writer so JNI edges land alongside static ones. Coordinate with native analyzer on reason codes.
- **Advisory core** ensure AOC writer schema (`JavaObservation.json`) is frozen before we serialise to avoid churn downstream.
## Deliverables for SCANNER-ANALYZERS-JAVA-21-008
1. **Observation projection (`JavaObservationWriter`)**
- Inputs: normalised workspace + analyzer outputs (classpath graph, SPI table, reflection edges, config hints, JNI hints).
- Outputs: deterministic JSON containing entrypoints, components, edges, warnings, provenance. Align with `docs/aoc/java-observation-schema.md` once published.
2. **AOC guard integration**
- Serialize observation documents through `Scanner.Aoc` guard pipeline; add unit tests covering required fields and forbidden derived data.
3. **Fixture updates**
- Expand `fixtures/lang/java/` set to include reflection-heavy app, Spring Boot sample, JNI sample, modular app. Record golden outputs with `UPDATE_JAVA_FIXTURES=1`.
4. **Metrics & logging**
- Emit counters (`scanner.java.observation.edges_total`, etc.) to trace observation completeness during CI runs.
5. **Documentation**
- Update `docs/scanner/java-analyzer.md` with reason code matrix and observation field definitions.
## Action items
| Owner | Task | Due | Notes |
|-------|------|-----|-------|
| Java Analyzer Guild | Land reflection TODOs (TCCL + resource plugin hints) | 2025-11-01 | Required for reliable dynamic edges. |
| Java Analyzer Guild | Finish config extractor for Spring/Jakarta | 2025-11-02 | Use sample apps in `fixtures/lang/java/config-*`. |
| Java Analyzer Guild | Draft observation writer spike PR using new schema | 2025-11-04 | PR can be draft but should include JSON schema + sample. |
| Scanner AOC Owners | Validate observation JSON against AOC guard + schema | 2025-11-05 | Blocker for marking 21-008 as DOING. |
| QA Guild | Prepare regression harness + performance gate (<300ms per fat jar) | 2025-11-06 | Align with SCANNER-ANALYZERS-JAVA-21-009. |
## Reporting
- Track these checkpoints in the Java analyzer weekly sync; once prerequisites are green, flip SCANNER-ANALYZERS-JAVA-21-008 to **DOING**.
- Store schema and sample output under `docs/scanner/java-observations/` so AOC reviewers have a stable reference.

View File

@@ -1,53 +0,0 @@
# Link-Not-Merge Determinism Test Plan
**Task:** MERGE-LNM-21-003 — replace legacy merge determinism suites with observation/linkset regressions now that `NoMergeEnabled` is defaulted to `true`.
## Objectives
- Validate raw advisory documents remain byte-stable through observation/linkset materialisation.
- Ensure conflicts detected during linkset building surface in telemetry and persisted artifacts without merge-side mutation.
- Keep canonical hash output stable for exports/evidence bundles after repeated runs.
## Test Coverage Outline
1. **Raw → Observation determinism**
- Feed canonical advisory raw fixtures containing mixed casing, duplicate aliases, and provenance metadata.
- Assert repeated runs of `AdvisoryObservationFactory` emit identical observations (structural equality + canonical JSON hash).
- Verify raw linkset payload retains original ordering/whitespace while canonical linkset stays normalised.
- Initial coverage implemented via `AdvisoryObservationFactoryTests.Create_IsDeterministicAcrossRuns` (core tests).
2. **Linkset conflict surfacing**
- Build linksets from conflicting advisory observations (e.g., differing severity or status flags).
- Confirm conflict markers propagate to `AdvisoryLinkset` outputs and associated metrics/log records.
- Capture deterministic ordering of conflict explanations for evidence exports.
- Coverage landed via `AdvisoryObservationFactoryTests.Create_PreservesRawReferencesForConflictAudits` (raw linkset + attribute parity) and `AdvisoryEventLogTests.AppendAsync_SortsConflictStatementIds` (canonical conflict JSON + stable hashes).
3. **Evidence/export parity**
- Re-run observation/linkset pipelines against identical fixtures and assert resulting evidence manifests hash-identically.
- Track monotonic `supersedes` chains and ensure canonical link records include `PRIMARY` schemes.
- `JsonExportSnapshotBuilderTests.WriteAsync_DifferentInputOrderProducesSameDigest` now proves export bundles remain byte-identical regardless of advisory enumeration order; digest sampling extends `ProducesIdenticalBytesAcrossRuns`.
## Mongo2Go/OpenSSL toolchain
Concelier solution tests (and most connector suites) depend on Mongo2Gos embedded `mongod`, which is linked against OpenSSL 1.1. The repo already ships the required libraries in `tests/native/openssl-1.1/linux-x64/{libcrypto.so.1.1,libssl.so.1.1}`; use them instead of installing global packages so offline runners stay deterministic.
1. Add the shim to your shell before executing any Mongo-backed suite:
```bash
export LD_LIBRARY_PATH="$(git rev-parse --show-toplevel)/tests/native/openssl-1.1/linux-x64:${LD_LIBRARY_PATH:-}"
```
2. For single commands you can prefix the invocation (handy for CI copy/paste):
```bash
LD_LIBRARY_PATH="$(pwd)/tests/native/openssl-1.1/linux-x64" \
dotnet test src/Concelier/StellaOps.Concelier.sln --nologo
```
3. The shims provenance and troubleshooting notes live in `tests/native/openssl-1.1/README.md`; reference it when mirroring the toolchain into air-gapped runners.
## Migration Steps
- [x] Retire `StellaOps.Concelier.Merge.Tests` determinism suites once observation/linkset equivalents land.
- [x] Introduce new regression fixtures under `StellaOps.Concelier.Core.Tests` (shared via `StellaOps.Concelier.Testing`).
- [ ] Wire test helpers to Mongo in-memory harness for end-to-end parity runs.
- [ ] Update documentation (`docs/migration/no-merge.md`) with validation checklist once new tests are green.
_Pending_: execute suites on a workstation with the .NET 10 preview SDK; local environment lacks a functioning CLI, so validation runs must happen downstream.

View File

@@ -69,7 +69,7 @@ db.advisories.aggregate([
]);
```
Pair this query with the indexes listed in [Normalized Versions Query Guide](mongo_indices.md).
**Note:** The above MongoDB query syntax is for reference only. The current system uses PostgreSQL. See `docs/db/SPECIFICATION.md` for current database schema.
## 5. Recommended indexes
@@ -98,7 +98,7 @@ Follow the operational checklist in `docs/modules/devops/migrations/semver-style
- [ ] Capture merge decisions via `decisionReason`.
- [ ] Confirm integration tests include fixtures with normalized rules and SemVer styles.
For deeper query examples and maintenance tasks, continue with [Normalized Versions Query Guide](mongo_indices.md).
For current database schema and query patterns, see `docs/db/SPECIFICATION.md`.
## 8. Storage projection reference