This commit is contained in:
master
2025-10-12 20:37:18 +03:00
parent 016c5a3fe7
commit d3a98326d1
306 changed files with 21409 additions and 4449 deletions

View File

@@ -1,6 +1,6 @@
# Authority Plug-in Developer Guide
> **Status:** Ready for Docs/DOC4 editorial review as of 2025-10-10. Content aligns with PLG6 acceptance criteria and references stable Authority primitives.
> **Status:** Updated 2025-10-11 (AUTHPLUG-DOCS-01-001) with lifecycle + limiter diagrams and refreshed rate-limit guidance aligned to PLG6 acceptance criteria.
## 1. Overview
Authority plug-ins extend the **StellaOps Authority** service with custom identity providers, credential stores, and client-management logic. Unlike Feedser plug-ins (which ingest or export advisories), Authority plug-ins participate directly in authentication flows:
@@ -17,6 +17,10 @@ Authority hosts follow a deterministic plug-in lifecycle. The flow below can be
3. **Registrar execution** each assembly is searched for `IAuthorityPluginRegistrar` implementations. Registrars bind options, register services, and optionally queue bootstrap tasks.
4. **Runtime** the host resolves `IIdentityProviderPlugin` instances, uses capability metadata to decide which OAuth grants to expose, and invokes health checks for readiness endpoints.
![Authority plug-in lifecycle diagram](../assets/authority/authority-plugin-lifecycle.svg)
_Source:_ `docs/assets/authority/authority-plugin-lifecycle.mmd`
**Data persistence primer:** the standard Mongo-backed plugin stores users in collections named `authority_users_<pluginName>` and lockout metadata in embedded documents. Additional plugins must document their storage layout and provide deterministic collection naming to honour the Offline Kit replication process.
## 3. Capability Metadata
@@ -100,7 +104,7 @@ Capability flags let the host reason about what your plug-in supports:
- Optional `IClientProvisioningStore` for machine-to-machine clients.
- `AuthorityIdentityProviderCapabilities` to advertise supported flows.
- Password guidance:
- Prefer Argon2 (Security Guild upcoming recommendation); Standard plug-in currently ships PBKDF2 with easy swap via `IPasswordHasher`.
- Standard plug-in hashes via `ICryptoProvider` using Argon2id by default and emits PHC-compliant strings. Successful PBKDF2 logins trigger automatic rehashes so migrations complete gradually. See `docs/security/password-hashing.md` for tuning advice.
- Enforce password policies before hashing to avoid storing weak credentials.
- Health checks should probe backing stores (e.g., Mongo `ping`) and return `AuthorityPluginHealthResult` so `/ready` can surface issues.
- When supporting additional factors (e.g., TOTP), implement `SupportsMfa` and document the enrolment flow for resource servers.
@@ -111,14 +115,64 @@ Capability flags let the host reason about what your plug-in supports:
- Never store raw secrets in git: allow operators to supply them via `.local.yaml`, environment variables, or injected secret files. Document which keys are mandatory.
- Validate configuration as soon as the registrar runs; use explicit error messages to guide operators. The Standard plug-in now enforces complete bootstrap credentials (username + password) and positive lockout windows via `StandardPluginOptions.Validate`.
- Cross-reference bootstrap workflows with `docs/ops/authority_bootstrap.md` (to be published alongside CORE6) so operators can reuse the same payload formats for manual provisioning.
- `passwordHashing` inherits defaults from `authority.security.passwordHashing`. Override only when hardware constraints differ per plug-in:
```yaml
passwordHashing:
algorithm: Argon2id
memorySizeInKib: 19456
iterations: 2
parallelism: 1
```
Invalid values (≤0) fail fast during startup, and legacy PBKDF2 hashes rehash automatically once the new algorithm succeeds.
## 8. Logging, Metrics, and Diagnostics
### 7.1 Token Persistence Contract
- The host automatically persists every issued principal (access, refresh, device, authorization code) in `authority_tokens`. Plug-in code **must not** bypass this store; use the provided `IAuthorityTokenStore` helpers when implementing custom flows.
- When a plug-in disables a subject or client outside the standard handlers, call `IAuthorityTokenStore.UpdateStatusAsync(...)` for each affected token so revocation bundles stay consistent.
- Supply machine-friendly `revokedReason` codes (`compromised`, `rotation`, `policy`, `lifecycle`, etc.) and optional `revokedMetadata` entries when invalidating credentials. These flow straight into `revocation-bundle.json` and should remain deterministic.
- Token scopes should be normalised (trimmed, unique, ordinal sort) before returning from plug-in verification paths. `TokenPersistenceHandlers` will keep that ordering for downstream consumers.
### 7.2 Claims & Enrichment Checklist
- Authority always sets the OpenID Connect basics: `sub`, `client_id`, `preferred_username`, optional `name`, and `role` (for password flows). Plug-ins must use `IClaimsEnricher` to append additional claims in a **deterministic** order (sort arrays, normalise casing) so resource servers can rely on stable shapes.
- Recommended enrichment keys:
- `stellaops.realm` plug-in/tenant identifier so services can scope policies.
- `stellaops.subject.type` values such as `human`, `service`, `bootstrap`.
- `groups` / `projects` sorted arrays describing operator entitlements.
- Claims visible in tokens should mirror what `/token` and `/userinfo` emit. Avoid injecting sensitive PII directly; mark values with `ClassifiedString.Personal` inside the plug-in so audit sinks can tag them appropriately.
- For client-credential flows, remember to enrich both the client principal and the validation path (`TokenValidationHandlers`) so refresh flows keep the same metadata.
### 7.3 Revocation Bundles & Reasons
- Use `IAuthorityRevocationStore` to record subject/client/token revocations when credentials are deleted or rotated. Stick to the standard categories (`token`, `subject`, `client`, `key`).
- Include a deterministic `reason` string and optional `reasonDescription` so operators understand *why* a subject was revoked when inspecting bundles offline.
- Plug-ins should populate `metadata` with stable keys (e.g., `revokedBy`, `sourcePlugin`, `ticketId`) to simplify SOC correlation. The keys must be lowercase, ASCII, and free of secrets—bundles are mirrored to air-gapped agents.
## 8. Rate Limiting & Lockout Interplay
Rate limiting and account lockouts are complementary controls. Plug-ins must surface both deterministically so operators can correlate limiter hits with credential rejections.
**Baseline quotas** (from `docs/dev/authority-rate-limit-tuning-outline.md`):
| Endpoint | Default policy | Notes |
|----------|----------------|-------|
| `/token` | 30 requests / 60s, queue 0 | Drop to 10/60s for untrusted ranges; raise only with WAF + monitoring. |
| `/authorize` | 60 requests / 60s, queue 10 | Reduce carefully; interactive UX depends on headroom. |
| `/internal/*` | Disabled by default; recommended 5/60s when enabled | Keep queue 0 for bootstrap APIs. |
**Retry metadata:** The middleware stamps `Retry-After` plus tags `authority.client_id`, `authority.remote_ip`, and `authority.endpoint`. Plug-ins should keep these tags intact when crafting responses or telemetry so dashboards remain consistent.
**Lockout counters:** Treat lockouts as **subject-scoped** decisions. When multiple instances update counters, reuse the deterministic tie-breakers documented in `src/DEDUP_CONFLICTS_RESOLUTION_ALGO.md` (freshness overrides, precedence, and stable hashes) to avoid divergent lockout states across replicas.
**Alerting hooks:** Emit structured logs/metrics when either the limiter or credential store rejects access. Suggested gauges include `aspnetcore_rate_limiting_rejections_total{limiter="authority-token"}` and any custom `auth.plugins.<pluginName>.lockouts_total` counter.
![Authority rate limit and lockout flow](../assets/authority/authority-rate-limit-flow.svg)
_Source:_ `docs/assets/authority/authority-rate-limit-flow.mmd`
## 9. Logging, Metrics, and Diagnostics
- Always log via the injected `ILogger<T>`; include `pluginName` and correlation IDs where available.
- Activity/metric names should align with `AuthorityTelemetry` constants (`service.name=stellaops-authority`).
- Expose additional diagnostics via structured logging rather than writing custom HTTP endpoints; the host will integrate these into `/health` and `/ready`.
- Emit metrics with stable names (`auth.plugins.<pluginName>.*`) when introducing custom instrumentation; coordinate with the Observability guild to reserve prefixes.
## 9. Testing & Tooling
## 10. Testing & Tooling
- Unit tests: use Mongo2Go (or similar) to exercise credential stores without hitting production infrastructure (`StandardUserCredentialStoreTests` is a template).
- Determinism: fix timestamps to UTC and sort outputs consistently; avoid random GUIDs unless stable.
- Smoke tests: launch `dotnet run --project src/StellaOps.Authority/StellaOps.Authority` with your plug-in under `PluginBinaries/Authority` and verify `/ready`.
@@ -137,13 +191,13 @@ Capability flags let the host reason about what your plug-in supports:
}
```
## 10. Packaging & Delivery
## 11. Packaging & Delivery
- Output assembly should follow `StellaOps.Authority.Plugin.<Name>.dll` so the hosts search pattern picks it up.
- Place the compiled DLL plus dependencies under `PluginBinaries/Authority` for offline deployments; include hashes/signatures in release notes (Security Guild guidance forthcoming).
- Document any external prerequisites (e.g., CA cert bundle) in your plug-in README.
- Update `etc/authority.plugins/<plugin>.yaml` samples and include deterministic SHA256 hashes for optional bootstrap payloads when distributing Offline Kit artefacts.
## 11. Checklist & Handoff
## 12. Checklist & Handoff
- ✅ Capabilities declared and validated in automated tests.
- ✅ Bootstrap workflows documented (if `bootstrap` capability used) and repeatable.
- ✅ Local smoke test + unit/integration suites green (`dotnet test`).
@@ -151,7 +205,4 @@ Capability flags let the host reason about what your plug-in supports:
- Submit the developer guide update referencing PLG6/DOC4 and tag DevEx + Docs reviewers for sign-off.
---
**Next documentation actions:**
- Add rendered architectural diagram (PlantUML/mermaid) reflecting the lifecycle above once the Docs toolkit pipeline is ready.
- Reference the LDAP RFC (`docs/rfcs/authority-plugin-ldap.md`) in the capability section once review completes.
- Sync terminology with `docs/11_AUTHORITY.md` when that chapter is published to keep glossary terms consistent.
Mermaid sources for the embedded diagrams live under `docs/assets/authority/`. Regenerate the SVG assets with your preferred renderer before committing future updates so the visuals stay in sync with the `.mmd` definitions.

34
docs/dev/fixtures.md Normal file
View File

@@ -0,0 +1,34 @@
# Feedser Fixture Maintenance
Feedser uses a handful of deterministic fixtures to keep connector regressions in check. This guide lists the
fixture sets, where they live, and how to regenerate them safely.
---
## GHSA ↔ OSV parity fixtures
- **Location:** `src/StellaOps.Feedser.Source.Osv.Tests/Fixtures/osv-ghsa.*.json`
- **Purpose:** Exercised by `OsvGhsaParityRegressionTests` to ensure OSV + GHSA outputs stay aligned on aliases,
ranges, references, and credits.
- **Regeneration:** Either run the test harness with online regeneration (`UPDATE_PARITY_FIXTURES=1 dotnet test src/StellaOps.Feedser.Source.Osv.Tests/StellaOps.Feedser.Source.Osv.Tests.csproj`)
or execute the fixture updater (`dotnet run --project tools/FixtureUpdater/FixtureUpdater.csproj`). Both paths
normalise timestamps and canonical ordering.
- **Verification:** Inspect the diff, then re-run `dotnet test src/StellaOps.Feedser.Source.Osv.Tests/StellaOps.Feedser.Source.Osv.Tests.csproj` to confirm parity.
## GHSA credit parity fixtures
- **Location:** `src/StellaOps.Feedser.Source.Ghsa.Tests/Fixtures/credit-parity.{ghsa,osv,nvd}.json`
- **Purpose:** Exercised by `GhsaCreditParityRegressionTests` to guarantee GHSA/NVD/OSV acknowledgements remain in lockstep.
- **Regeneration:** `dotnet run --project tools/FixtureUpdater/FixtureUpdater.csproj` rewrites all three canonical snapshots.
- **Verification:** `dotnet test src/StellaOps.Feedser.Source.Ghsa.Tests/StellaOps.Feedser.Source.Ghsa.Tests.csproj`.
> Always commit fixture changes together with the code that motivated them and reference the regression test that guards the behaviour.
## Apple security update fixtures
- **Location:** `src/StellaOps.Feedser.Source.Vndr.Apple.Tests/Apple/Fixtures/*.html` and `.expected.json`.
- **Purpose:** Exercised by `AppleLiveRegressionTests` to guarantee the Apple HTML parser and mapper stay deterministic while covering Rapid Security Responses and multi-device advisories.
- **Regeneration:** Use the helper scripts (`scripts/update-apple-fixtures.sh` or `scripts/update-apple-fixtures.ps1`). They export `UPDATE_APPLE_FIXTURES=1`, propagate the flag through `WSLENV`, touch `.update-apple-fixtures`, and then run the Apple test project. This keeps WSL/VSCode test invocations in sync while the refresh workflow fetches live Apple support pages, sanitises them, and rewrites both the HTML and expected DTO snapshots with normalised ordering.
- **Verification:** Inspect the generated diffs and re-run `dotnet test src/StellaOps.Feedser.Source.Vndr.Apple.Tests/StellaOps.Feedser.Source.Vndr.Apple.Tests.csproj` without the env var to confirm determinism.
> **Tip for other connector owners:** mirror the sentinel + `WSLENV` pattern (`touch .update-<connector>-fixtures`, append the env var via `WSLENV`) when you add fixture refresh scripts so contributors running under WSL inherit the regeneration flag automatically.

View File

@@ -29,6 +29,8 @@ var rule = primitive.ToNormalizedVersionRule(notes: "nvd:CVE-2025-1234");
// rule => scheme=semver, type=range, min=1.2.3, minInclusive=true, max=2.0.0, maxInclusive=false
```
If you omit the optional `notes` argument, `ToNormalizedVersionRule` now falls back to the primitives `ConstraintExpression`, ensuring the original comparator expression is preserved for provenance/audit queries.
Emit the resulting rule inside `AffectedPackage.NormalizedVersions` while continuing to populate `AffectedVersionRange.RangeExpression` for backward compatibility.
## 3. Merge dedupe flow
@@ -97,3 +99,56 @@ Follow the operational checklist in `docs/ops/migrations/SEMVER_STYLE.md`. The s
- [ ] Confirm integration tests include fixtures with normalized rules and SemVer styles.
For deeper query examples and maintenance tasks, continue with [Normalized Versions Query Guide](mongo_indices.md).
## 8. Storage projection reference
`NormalizedVersionDocumentFactory` copies each normalized rule into MongoDB using the shape below. Use this as a contract when reviewing connector fixtures or diagnosing merge/storage diffs:
```json
{
"packageId": "pkg:npm/example",
"packageType": "npm",
"scheme": "semver",
"type": "range",
"style": "range",
"min": "1.2.3",
"minInclusive": true,
"max": "2.0.0",
"maxInclusive": false,
"value": null,
"notes": "ghsa:GHSA-xxxx-yyyy",
"decisionReason": "ghsa-precedence-over-nvd",
"constraint": ">= 1.2.3 < 2.0.0",
"source": "ghsa",
"recordedAt": "2025-10-11T00:00:00Z"
}
```
For distro-specific ranges (`nevra`, `evr`) the same envelope applies with `scheme` switched accordingly. Example:
```json
{
"packageId": "bash",
"packageType": "rpm",
"scheme": "nevra",
"type": "range",
"style": "range",
"min": "0:4.4.18-2.el7",
"minInclusive": true,
"max": "0:4.4.20-1.el7",
"maxInclusive": false,
"value": null,
"notes": "redhat:RHSA-2025:1234",
"decisionReason": "rhel-priority-over-nvd",
"constraint": "<= 0:4.4.20-1.el7",
"source": "redhat",
"recordedAt": "2025-10-11T00:00:00Z"
}
```
If a new scheme is required (for example, `apple.build` or `ios.semver`), raise it with the Models team before emitting documents so merge comparers and hashing logic can incorporate the change deterministically.
## 9. Observability signals
- `feedser.merge.normalized_rules` (counter, tags: `package_type`, `scheme`) increments once per normalized rule retained after precedence merge.
- `feedser.merge.normalized_rules_missing` (counter, tags: `package_type`) increments when a merged package still carries version ranges but no normalized rules; watch for spikes to catch connectors that have not emitted normalized arrays yet.

View File

@@ -2,6 +2,8 @@
This guide complements the Sprint12 normalized versions rollout. It documents recommended indexes and aggregation patterns for querying `AffectedPackage.normalizedVersions`.
For a field-by-field look at how normalized rules persist in MongoDB (including provenance metadata), see Section8 of the [Feedser SemVer Merge Playbook](merge_semver_playbook.md).
## 1. Recommended indexes
When `feedser.storage.enableSemVerStyle` is enabled, advisories expose a flattened

View File

@@ -0,0 +1,49 @@
# Normalized Versions Rollout Dashboard (Sprint 2 Feedser)
_Status date: 2025-10-12 17:05 UTC_
This dashboard tracks connector readiness for emitting `AffectedPackage.NormalizedVersions` arrays and highlights upcoming coordination checkpoints. Use it alongside:
- [`src/StellaOps.Feedser.Merge/RANGE_PRIMITIVES_COORDINATION.md`](../../src/StellaOps.Feedser.Merge/RANGE_PRIMITIVES_COORDINATION.md) for detailed guidance and timelines.
- [Feedser SemVer Merge Playbook](merge_semver_playbook.md) §8 for persisted Mongo document shapes.
- [Normalized Versions Query Guide](mongo_indices.md) for index/query validation steps.
## Key milestones
- **2025-10-13** Normalization to finalize `SemVerRangeRuleBuilder` API contract for review.
- **2025-10-17** Connector owners to post fixture PRs showing `NormalizedVersions` arrays (even if feature-flagged).
- **2025-10-18** Merge cross-connector review to validate consistent field usage before enabling union logic.
## Connector readiness matrix
| Connector | Owner team | Normalized versions status | Last update | Next action / link |
|-----------|------------|---------------------------|-------------|--------------------|
| Acsc | BE-Conn-ACSC | ❌ Not started mapper pending | 2025-10-11 | Design DTOs + mapper with normalized rule array; see `src/StellaOps.Feedser.Source.Acsc/TASKS.md`. |
| Cccs | BE-Conn-CCCS | ❌ Not started mapper pending | 2025-10-11 | Add normalized SemVer array in canonical mapper; coordinate fixtures per `TASKS.md`. |
| CertBund | BE-Conn-CERTBUND | ❌ Not started mapper pending | 2025-10-11 | Capture firmware-style ranges; emit normalized payload; `src/StellaOps.Feedser.Source.CertBund/TASKS.md`. |
| CertCc | BE-Conn-CERTCC | ⚠️ In progress fetch pipeline DOING | 2025-10-11 | Implement VINCE mapper with SemVer/NEVRA rules; unblock snapshot regeneration; `src/StellaOps.Feedser.Source.CertCc/TASKS.md`. |
| Kev | BE-Conn-KEV | ✅ Normalized catalog/due-date rules verified | 2025-10-12 | Fixtures reconfirmed via `dotnet test src/StellaOps.Feedser.Source.Kev.Tests`; `src/StellaOps.Feedser.Source.Kev/TASKS.md`. |
| Cve | BE-Conn-CVE | ✅ Normalized SemVer rules verified | 2025-10-12 | Snapshot parity green (`dotnet test src/StellaOps.Feedser.Source.Cve.Tests`); `src/StellaOps.Feedser.Source.Cve/TASKS.md`. |
| Ghsa | BE-Conn-GHSA | ⚠️ DOING normalized rollout task active | 2025-10-11 18:45 UTC | Wire `SemVerRangeRuleBuilder` + refresh fixtures; `src/StellaOps.Feedser.Source.Ghsa/TASKS.md`. |
| Osv | BE-Conn-OSV | ✅ SemVer mapper & parity fixtures verified | 2025-10-12 | GHSA parity regression passing (`dotnet test src/StellaOps.Feedser.Source.Osv.Tests`); `src/StellaOps.Feedser.Source.Osv/TASKS.md`. |
| Ics.Cisa | BE-Conn-ICS-CISA | ❌ Not started mapper TODO | 2025-10-11 | Plan SemVer/firmware scheme selection; `src/StellaOps.Feedser.Source.Ics.Cisa/TASKS.md`. |
| Kisa | BE-Conn-KISA | ❌ Not started mapper TODO | 2025-10-11 | Localisation-aware mapper with normalized rules; `src/StellaOps.Feedser.Source.Kisa/TASKS.md`. |
| Ru.Bdu | BE-Conn-BDU | ❌ Not started mapper TODO | 2025-10-11 | Emit normalized ranges, capture provenance; `src/StellaOps.Feedser.Source.Ru.Bdu/TASKS.md`. |
| Ru.Nkcki | BE-Conn-Nkcki | ❌ Not started mapper TODO | 2025-10-11 | Similar to BDU; ensure Cyrillic provenance preserved; `src/StellaOps.Feedser.Source.Ru.Nkcki/TASKS.md`. |
| Vndr.Apple | BE-Conn-Apple | ✅ Shipped emitting normalized arrays | 2025-10-11 | Continue fixture/tooling work; `src/StellaOps.Feedser.Source.Vndr.Apple/TASKS.md`. |
| Vndr.Cisco | BE-Conn-Cisco | ❌ Not started mapper TODO | 2025-10-11 | Decide on scheme (`semver` vs custom) before emitting rules; `src/StellaOps.Feedser.Source.Vndr.Cisco/TASKS.md`. |
| Vndr.Msrc | BE-Conn-MSRC | ❌ Not started mapper TODO | 2025-10-11 | Gather samples, define scheme, emit normalized rules; `src/StellaOps.Feedser.Source.Vndr.Msrc/TASKS.md`. |
| Nvd | BE-Conn-NVD | ⚠️ Needs follow-up mapper complete but normalized array MR pending | 2025-10-11 | Align CVE notes + normalized payload flag; `src/StellaOps.Feedser.Source.Nvd/TASKS.md`. |
Legend: ✅ complete, ⚠️ in progress/partial, ❌ not started.
## Monitoring
- Merge now emits `feedser.merge.normalized_rules` (tags: `package_type`, `scheme`) and `feedser.merge.normalized_rules_missing` (tags: `package_type`). Track these counters to confirm normalized arrays land as connectors roll out.
- Expect `normalized_rules_missing` to trend toward zero as each connector flips on normalized output. Investigate any sustained counts by checking the corresponding module `TASKS.md`.
## How to use this dashboard
1. Before opening a connector PR, update the module `TASKS.md` entry and drop a short bullet here (status + timestamp).
2. When a connector lands normalized outputs, flip the status to ✅ and note any rollout toggles (feature flags, fixture regenerations).
3. If a dependency or blocker emerges, add it both in the module `TASKS.md` and in this matrix so merge/storage can escalate quickly.