stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 1 Releases Wiki Activity

Initial commit (history squashed)

Browse Source
This commit is contained in:
master
2025-10-07 10:14:21 +03:00
commit 016c5a3fe7
1132 changed files with 117842 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

146
docs/dev/30_PLUGIN_DEV_GUIDE.md Executable file
View File

@@ -0,0 +1,146 @@
# Writing Plug‑ins for Stella Ops SDK *Preview 3*
> **SDK status:** *Preview 3* is compatible with the **v0.1 α** runtime.
> Interfaces freeze at **v0.2 β**; binary‑breaking changes are still possible
> until then.
| SDK NuGet | Runtime compat | Notes |
|-----------|---------------|-------|
| `StellaOps.SDK 0.2.0-preview3` | `stella-ops >= 0.1.0-alpha` | Current preview |
| `StellaOps.SDK 0.2.x‑beta` | v0.2 β (Q1 2026) | Interface **freeze** |
| `StellaOps.SDK 1.0.0` | v1.0 GA (Q4 2026) | Semantic Ver from here |
---
## 0 · Extension points
| Area | Interface / format | Example |
|------|--------------------|---------|
| SBOM mutator | `ISbomMutator` | Inject SPDX licences |
| Additional scanner | `IVulnerabilityProvider` | Rust Crates ecosystem |
| Policy engine | **OPA Rego** file | Custom pass/fail rule |
| Result exporter | `IResultSink` | Slack webhook notifier |
*Hot‑plugging (live reload) is **post‑1.0**; modules are discovered once
during service start‑up.*
---
## 1 · Five‑minute quick‑start (C# /.NET {{ dotnet }})
```bash
dotnet new classlib -n SlackExporter
cd SlackExporter
dotnet add package StellaOps.SDK --version 0.2.0-preview3
````
```csharp
using System.Net.Http.Json;
using StellaOps.Plugin;
public sealed class SlackSink : IResultSink
{
private readonly string _webhook =
Environment.GetEnvironmentVariable("SLACK_WEBHOOK")
?? throw new InvalidOperationException("Missing SLACK_WEBHOOK");
public string Name => "Slack Notifier";
public async Task ExportAsync(ScanResult result, CancellationToken ct)
{
var payload = new
{
text = $":rotating_light: *{result.Image}* " +
$"→ {result.Findings.Count} findings (max {result.MaxSeverity})"
};
using var client = new HttpClient();
await client.PostAsJsonAsync(_webhook, payload, ct);
}
}
```
```bash
dotnet publish -c Release -o out
sudo mkdir -p /opt/stella/plugins/Slack
sudo cp out/SlackExporter.dll /opt/stella/plugins/Slack/
sudo systemctl restart stella-ops
```
Start‑up log:
```
[PluginLoader] Loaded 1 plug‑in:
• Slack Notifier
```
---
## 2 · Packaging rules
| Item | Rule |
| ------ | ----------------------------------------- |
| Folder | `/opt/stella/plugins/<NiceName>/` |
| DLLs | Your plug‑in + non‑GAC deps |
| Config | Env‑vars or `settings.yaml` |
| SBOM | Optional `addon.spdx.json` for provenance |
---
## 3 · Security sandbox
* Runs as Linux user **`stella‑plugin` (UID 1001)**.
* SELinux/AppArmor profile blocks inbound traffic; outbound :80/443 only.
* cgroup default: **1 CPU / 256 MiB** (adjustable).
* SHA‑256 of every DLL is embedded in the run report.
---
## 4 · Debugging
| Technique | Command |
| ----------------- | ---------------------------------- |
| Verbose core log | `STELLA_LOG=debug` |
| Per‑plug‑in log | Inject `ILogger<YourClass>` |
| Dry‑run (no fail) | `--plugin-mode warn` |
| Hot reload | *Not supported* (planned post‑1.0) |
Logs: `/var/log/stella-ops/plugins/YYYY‑MM‑DD.log`.
---
## 5 · Interface reference (Preview 3)
```csharp
namespace StellaOps.Plugin
{
public interface ISbomMutator
{
string Name { get; }
Task<SoftwareBillOfMaterials> MutateAsync(
SoftwareBillOfMaterials sbom,
CancellationToken ct = default);
}
public interface IVulnerabilityProvider
{
string Ecosystem { get; }
Task<IReadOnlyList<Vulnerability>> QueryAsync(
PackageReference p, CancellationToken ct = default);
}
public interface IResultSink
{
string Name { get; }
Task ExportAsync(
ScanResult result, CancellationToken ct = default);
}
}
```
Full POCO docs: [https://git.stella-ops.org/stella-ops/sdk/-/tree/main/docs/api](https://git.stella-ops.org/stella-ops/sdk/-/tree/main/docs/api).
---
*Last updated {{ "now" | date: "%Y‑%m‑%d" }} – constants auto‑injected.*

157
docs/dev/31_AUTHORITY_PLUGIN_DEVELOPER_GUIDE.md Normal file
View File

@@ -0,0 +1,157 @@
# Authority Plug-in Developer Guide
> **Status:** Ready for Docs/DOC4 editorial review as of 2025-10-10. Content aligns with PLG6 acceptance criteria and references stable Authority primitives.
## 1. Overview
Authority plug-ins extend the **StellaOps Authority** service with custom identity providers, credential stores, and client-management logic. Unlike Feedser plug-ins (which ingest or export advisories), Authority plug-ins participate directly in authentication flows:
- **Use cases:** integrate corporate directories (LDAP/AD), delegate to external IDPs, enforce bespoke password/lockout policies, or add client provisioning automation.
- **Constraints:** plug-ins load only during service start (no hot-reload), must function without outbound internet access, and must emit deterministic results for identical configuration and input data.
- **Ship targets:** target the same .NET 10 preview as the host, honour offline-first requirements, and provide clear diagnostics so operators can triage issues from `/ready`.
## 2. Architecture Snapshot
Authority hosts follow a deterministic plug-in lifecycle. The flow below can be rendered as a sequence diagram in the final authored documentation, but all touchpoints are described here for offline viewers:
1. **Configuration load** – `AuthorityPluginConfigurationLoader` resolves YAML manifests under `etc/authority.plugins/`.
2. **Assembly discovery** – the shared `PluginHost` scans `PluginBinaries/Authority` for `StellaOps.Authority.Plugin.*.dll` assemblies.
3. **Registrar execution** – each assembly is searched for `IAuthorityPluginRegistrar` implementations. Registrars bind options, register services, and optionally queue bootstrap tasks.
4. **Runtime** – the host resolves `IIdentityProviderPlugin` instances, uses capability metadata to decide which OAuth grants to expose, and invokes health checks for readiness endpoints.
**Data persistence primer:** the standard Mongo-backed plugin stores users in collections named `authority_users_<pluginName>` and lockout metadata in embedded documents. Additional plugins must document their storage layout and provide deterministic collection naming to honour the Offline Kit replication process.
## 3. Capability Metadata
Capability flags let the host reason about what your plug-in supports:
- Declare capabilities in your descriptor using the string constants from `AuthorityPluginCapabilities` (`password`, `mfa`, `clientProvisioning`, `bootstrap`). The configuration loader now validates these tokens and rejects unknown values at startup.
- `AuthorityIdentityProviderCapabilities.FromCapabilities` projects those strings into strongly typed booleans (`SupportsPassword`, etc.). Authority Core will use these flags when wiring flows such as the password grant. Built-in plugins (e.g., Standard) will fail fast or force-enable required capabilities if the descriptor is misconfigured, so keep manifests accurate.
- Typical configuration (`etc/authority.plugins/standard.yaml`):
```yaml
plugins:
descriptors:
standard:
assemblyName: "StellaOps.Authority.Plugin.Standard"
capabilities:
- password
- bootstrap
```
- Only declare a capability if the plug-in genuinely implements it. For example, if `SupportsClientProvisioning` is `true`, the plug-in must supply a working `IClientProvisioningStore`.
**Operational reminder:** the Authority host surfaces capability summaries during startup (see `AuthorityIdentityProviderRegistry` log lines). Use those logs during smoke tests to ensure manifests align with expectations.
**Configuration path normalisation:** Manifest-relative paths (e.g., `tokenSigning.keyDirectory: "../keys"`) are resolved against the YAML file location and environment variables are expanded before validation. Plug-ins should expect to receive an absolute, canonical path when options are injected.
## 4. Project Scaffold
- Target **.NET 10 preview**, enable nullable, treat warnings as errors, and mark Authority plug-ins with `<IsAuthorityPlugin>true</IsAuthorityPlugin>`.
- Minimum references:
- `StellaOps.Authority.Plugins.Abstractions` (contracts & capability helpers)
- `StellaOps.Plugin` (hosting/DI helpers)
- `StellaOps.Auth.*` libraries as needed for shared token utilities (optional today).
- Example `.csproj` (trimmed from `StellaOps.Authority.Plugin.Standard`):
```xml
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>net10.0</TargetFramework>
<Nullable>enable</Nullable>
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
<IsAuthorityPlugin>true</IsAuthorityPlugin>
</PropertyGroup>
<ItemGroup>
<ProjectReference Include="..\StellaOps.Authority.Plugins.Abstractions\StellaOps.Authority.Plugins.Abstractions.csproj" />
<ProjectReference Include="..\..\StellaOps.Plugin\StellaOps.Plugin.csproj" />
</ItemGroup>
</Project>
```
(Add other references—e.g., MongoDB driver, shared auth libraries—according to your implementation.)
## 5. Implementing `IAuthorityPluginRegistrar`
- Create a parameterless registrar class that returns your plug-in type name via `PluginType`.
- Use `AuthorityPluginRegistrationContext` to:
- Bind options (`AddOptions<T>(pluginName).Bind(...)`).
- Register singletons for stores/enrichers using manifest metadata.
- Register any hosted bootstrap tasks (e.g., seed admin users).
- Always validate configuration inside `PostConfigure` and throw meaningful `InvalidOperationException` to fail fast during startup.
- Use the provided `ILoggerFactory` from DI; avoid static loggers or console writes.
- Example skeleton:
```csharp
internal sealed class MyPluginRegistrar : IAuthorityPluginRegistrar
{
public string PluginType => "my-custom";
public void Register(AuthorityPluginRegistrationContext context)
{
var name = context.Plugin.Manifest.Name;
context.Services.AddOptions<MyPluginOptions>(name)
.Bind(context.Plugin.Configuration)
.PostConfigure(opts => opts.Validate(name));
context.Services.AddSingleton<IIdentityProviderPlugin>(sp =>
new MyIdentityProvider(context.Plugin, sp.GetRequiredService<MyCredentialStore>(),
sp.GetRequiredService<MyClaimsEnricher>(),
sp.GetRequiredService<ILogger<MyIdentityProvider>>()));
}
}
```
## 6. Identity Provider Surface
- Implement `IIdentityProviderPlugin` to expose:
- `IUserCredentialStore` for password validation and user CRUD.
- `IClaimsEnricher` to append roles/attributes onto issued principals.
- Optional `IClientProvisioningStore` for machine-to-machine clients.
- `AuthorityIdentityProviderCapabilities` to advertise supported flows.
- Password guidance:
- Prefer Argon2 (Security Guild upcoming recommendation); Standard plug-in currently ships PBKDF2 with easy swap via `IPasswordHasher`.
- Enforce password policies before hashing to avoid storing weak credentials.
- Health checks should probe backing stores (e.g., Mongo `ping`) and return `AuthorityPluginHealthResult` so `/ready` can surface issues.
- When supporting additional factors (e.g., TOTP), implement `SupportsMfa` and document the enrolment flow for resource servers.
## 7. Configuration & Secrets
- Authority looks for manifests under `etc/authority.plugins/`. Each YAML file maps directly to a plug-in name.
- Support environment overrides using `STELLAOPS_AUTHORITY_PLUGINS__DESCRIPTORS__<NAME>__...`.
- Never store raw secrets in git: allow operators to supply them via `.local.yaml`, environment variables, or injected secret files. Document which keys are mandatory.
- Validate configuration as soon as the registrar runs; use explicit error messages to guide operators. The Standard plug-in now enforces complete bootstrap credentials (username + password) and positive lockout windows via `StandardPluginOptions.Validate`.
- Cross-reference bootstrap workflows with `docs/ops/authority_bootstrap.md` (to be published alongside CORE6) so operators can reuse the same payload formats for manual provisioning.
## 8. Logging, Metrics, and Diagnostics
- Always log via the injected `ILogger<T>`; include `pluginName` and correlation IDs where available.
- Activity/metric names should align with `AuthorityTelemetry` constants (`service.name=stellaops-authority`).
- Expose additional diagnostics via structured logging rather than writing custom HTTP endpoints; the host will integrate these into `/health` and `/ready`.
- Emit metrics with stable names (`auth.plugins.<pluginName>.*`) when introducing custom instrumentation; coordinate with the Observability guild to reserve prefixes.
## 9. Testing & Tooling
- Unit tests: use Mongo2Go (or similar) to exercise credential stores without hitting production infrastructure (`StandardUserCredentialStoreTests` is a template).
- Determinism: fix timestamps to UTC and sort outputs consistently; avoid random GUIDs unless stable.
- Smoke tests: launch `dotnet run --project src/StellaOps.Authority/StellaOps.Authority` with your plug-in under `PluginBinaries/Authority` and verify `/ready`.
- Example verification snippet:
```csharp
[Fact]
public async Task VerifyPasswordAsync_ReturnsSuccess()
{
var store = CreateCredentialStore();
await store.UpsertUserAsync(new AuthorityUserRegistration("alice", "Pa55!", null, null, false,
Array.Empty<string>(), new Dictionary<string, string?>()), CancellationToken.None);
var result = await store.VerifyPasswordAsync("alice", "Pa55!", CancellationToken.None);
Assert.True(result.Succeeded);
Assert.True(result.User?.Roles.Count == 0);
}
```
## 10. Packaging & Delivery
- Output assembly should follow `StellaOps.Authority.Plugin.<Name>.dll` so the host’s search pattern picks it up.
- Place the compiled DLL plus dependencies under `PluginBinaries/Authority` for offline deployments; include hashes/signatures in release notes (Security Guild guidance forthcoming).
- Document any external prerequisites (e.g., CA cert bundle) in your plug-in README.
- Update `etc/authority.plugins/<plugin>.yaml` samples and include deterministic SHA256 hashes for optional bootstrap payloads when distributing Offline Kit artefacts.
## 11. Checklist & Handoff
- ✅ Capabilities declared and validated in automated tests.
- ✅ Bootstrap workflows documented (if `bootstrap` capability used) and repeatable.
- ✅ Local smoke test + unit/integration suites green (`dotnet test`).
- ✅ Operational docs updated: configuration keys, secrets guidance, troubleshooting.
- Submit the developer guide update referencing PLG6/DOC4 and tag DevEx + Docs reviewers for sign-off.
---
**Next documentation actions:**
- Add rendered architectural diagram (PlantUML/mermaid) reflecting the lifecycle above once the Docs toolkit pipeline is ready.
- Reference the LDAP RFC (`docs/rfcs/authority-plugin-ldap.md`) in the capability section once review completes.
- Sync terminology with `docs/11_AUTHORITY.md` when that chapter is published to keep glossary terms consistent.

91
docs/dev/32_AUTH_CLIENT_GUIDE.md Normal file
View File

@@ -0,0 +1,91 @@
# StellaOps Auth Client — Integration Guide
> **Status:** Drafted 2025-10-10 as part of LIB5. Consumer teams (Feedser, CLI, Agent) should review before wiring the new options into their configuration surfaces.
The `StellaOps.Auth.Client` library provides a resilient OpenID Connect client for services and tools that talk to **StellaOps Authority**. LIB5 introduced configurable HTTP retry/backoff policies and an offline-fallback window so downstream components stay deterministic even when Authority is briefly unavailable.
This guide explains how to consume the new settings, when to toggle them, and how to test your integration.
## 1. Registering the client
```csharp
services.AddStellaOpsAuthClient(options =>
{
options.Authority = configuration["StellaOps:Authority:Url"]!;
options.ClientId = configuration["StellaOps:Authority:ClientId"]!;
options.ClientSecret = configuration["StellaOps:Authority:ClientSecret"];
options.DefaultScopes.Add("feedser.jobs.trigger");
options.EnableRetries = true;
options.RetryDelays.Clear();
options.RetryDelays.Add(TimeSpan.FromMilliseconds(500));
options.RetryDelays.Add(TimeSpan.FromSeconds(2));
options.AllowOfflineCacheFallback = true;
options.OfflineCacheTolerance = TimeSpan.FromMinutes(5);
});
```
> **Reminder:** `AddStellaOpsAuthClient` binds the options via `IOptionsMonitor<T>` so changes picked up from configuration reloads will be applied to future HTTP calls without restarting the host.
## 2. Resilience options
| Option | Default | Notes |
|--------|---------|-------|
| `EnableRetries` | `true` | When disabled, the shared Polly policy is a no-op and HTTP calls will fail fast. |
| `RetryDelays` | `1s, 2s, 5s` | Edit in ascending order; zero/negative entries are ignored. Clearing the list and leaving it empty keeps the defaults. |
| `AllowOfflineCacheFallback` | `true` | When `true`, stale discovery/JWKS responses are reused within the tolerance window if Authority is unreachable. |
| `OfflineCacheTolerance` | `00:10:00` | Added to the normal cache lifetime. E.g. a 10 minute JWKS cache plus 5 minute tolerance keeps keys for 15 minutes if Authority is offline. |
The HTTP retry policy handles:
- 5xx responses
- 429 responses
- Transient transport failures (`HttpRequestException`, timeouts, aborted sockets)
Retries emit warnings via the `StellaOps.Auth.Client.HttpRetry` logger. Tune the delay values to honour your deployment’s SLOs.
## 3. Configuration mapping
Suggested configuration keys (coordinate with consuming teams before finalising):
```yaml
StellaOps:
Authority:
Url: "https://authority.stella-ops.local"
ClientId: "feedser"
ClientSecret: "change-me"
AuthClient:
EnableRetries: true
RetryDelays:
- "00:00:01"
- "00:00:02"
- "00:00:05"
AllowOfflineCacheFallback: true
OfflineCacheTolerance: "00:10:00"
```
Environment variable binding follows the usual double-underscore rules, e.g.
```
STELLAOPS__AUTHORITY__AUTHCLIENT__RETRYDELAYS__0=00:00:02
STELLAOPS__AUTHORITY__AUTHCLIENT__OFFLINECACHETOLERANCE=00:05:00
```
CLI and Feedser teams should expose these knobs once they adopt the auth client.
## 4. Testing recommendations
1. **Unit tests:** assert option binding by configuring `StellaOpsAuthClientOptions` via a `ConfigurationBuilder` and ensuring `Validate()` normalises the retry delays and scope list.
2. **Offline fallback:** simulate an unreachable Authority by swapping `HttpMessageHandler` to throw `HttpRequestException` after priming the discovery/JWKS caches. Verify that tokens are still issued until the tolerance expires.
3. **Observability:** watch for `StellaOps.Auth.Client.HttpRetry` warnings in your logs. Excessive retries mean the upstream Authority cluster needs attention.
4. **Determinism:** keep retry delays deterministic. Avoid random jitter—operators can introduce jitter at the infrastructure layer if desired.
## 5. Rollout checklist
- [ ] Update consuming service/CLI configuration schema to include the new settings.
- [ ] Document recommended defaults for offline (air-gapped) versus connected deployments.
- [ ] Extend smoke tests to cover Authority outage scenarios.
- [ ] Coordinate with Docs Guild so user-facing quickstarts reference the new knobs.
Once Feedser and CLI integrate these changes, we can mark LIB5 **DONE**; further packaging work is deferred until the backlog reintroduces it.

33
docs/dev/authority-rate-limit-tuning-outline.md Normal file
View File

@@ -0,0 +1,33 @@
# Authority Rate Limit Tuning Outline (2025-10-11)
## Purpose
- Drive the remaining work on SEC3.B (Security Guild) and PLG6.DOC (Docs Guild) by capturing the agreed baseline for Authority rate limits and related documentation deliverables.
- Provide a single reference for lockout + rate limit interplay so Docs can lift accurate copy into `docs/security/rate-limits.md` and `docs/dev/31_AUTHORITY_PLUGIN_DEVELOPER_GUIDE.md`.
## Baseline Configuration
- `/token`: fixed window, permitLimit 30, window 60s, queueLimit 0. Reduce to 10/60s for untrusted IP ranges; raise to 60/60s only with compensating controls (WAF + active monitoring).
- `/authorize`: permitLimit 60, window 60s, queueLimit 10. Intended for interactive browser flows; lowering below 30 requires UX review.
- `/internal/*`: disabled by default; recommended 5/60s with queueLimit 0 when bootstrap API exposed.
- Configuration path: `authority.security.rateLimiting.<endpoint>` (e.g., `token.permitLimit`). YAML/ENV bindings follow the standard options hierarchy.
- Retry metadata: middleware stamps `Retry-After` along with tags `authority.client_id`, `authority.remote_ip`, `authority.endpoint`. Docs should highlight these for operator dashboards.
## Parameter Matrix
| Scenario | permitLimit | window | queueLimit | Notes |
|----------|-------------|--------|------------|-------|
| Default production | 30 | 60s | 0 | Works with anonymous quota (33 scans/day). |
| High-trust clustered IPs | 60 | 60s | 5 | Requires `authorize_rate_limit_hits` alert ≤ 1% sustained. |
| Air-gapped lab | 10 | 120s | 0 | Emphasise reduced concurrency + manual queue draining. |
| Incident lockdown | 5 | 300s | 0 | Pair with lockout lowering to 3 attempts. |
## Lockout Interplay
- Ensure Docs explain difference between rate limit (per IP/client) vs lockout (per subject). Provide table mapping retry-after headers to recommended support scripts.
- Security Guild to define alert thresholds: trigger SOC ticket when 429 rate > 25% for 5 minutes or when limiter emits >100 events/hour per client.
## Observability
- Surface metrics: `aspnetcore_rate_limiting_rejections_total{limiter="authority-token"}` and custom log tags from `AuthorityRateLimiterMetadataMiddleware`.
- Recommend dashboard sections: request volume vs. rejections, top offending clientIds, per-endpoint heatmap.
## Action Items
1. Security Guild (SEC3.B): incorporate matrix + alert rules into `docs/security/rate-limits.md`, add YAML examples for override blocks, and cross-link lockout policy doc.
2. Docs Guild (PLG6.DOC): update developer guide section 9 with the middleware sequence and reference this outline for retry metadata + tuning guidance.
3. Authority Core: validate appsettings sample includes the `security.rateLimiting` block with comments and link back to published doc once ready.

99
docs/dev/merge_semver_playbook.md Normal file
View File

@@ -0,0 +1,99 @@
# Feedser SemVer Merge Playbook (Sprint 1–2)
This playbook describes how the merge layer and connector teams should emit the new SemVer primitives introduced in Sprint 1–2, how those primitives become normalized version rules, and how downstream jobs query them deterministically.
## 1. What landed in Sprint 1–2
- `RangePrimitives.SemVer` now infers a canonical `style` (`range`, `exact`, `lt`, `lte`, `gt`, `gte`) and captures `exactValue` when the constraint is a single version.
- `NormalizedVersionRule` documents the analytics-friendly projection of each `AffectedPackage` coverage entry and is persisted alongside legacy `versionRanges`.
- `AdvisoryProvenance.decisionReason` records whether merge resolution favored precedence, freshness, or a tie-breaker comparison.
See `src/StellaOps.Feedser.Models/CANONICAL_RECORDS.md` for the full schema and field descriptions.
## 2. Mapper pattern
Connectors should emit SemVer primitives as soon as they can normalize a vendor constraint. The helper `SemVerPrimitiveExtensions.ToNormalizedVersionRule` turns those primitives into the persisted rules:
```csharp
var primitive = new SemVerPrimitive(
introduced: "1.2.3",
introducedInclusive: true,
fixed: "2.0.0",
fixedInclusive: false,
lastAffected: null,
lastAffectedInclusive: false,
constraintExpression: ">=1.2.3 <2.0.0",
exactValue: null);
var rule = primitive.ToNormalizedVersionRule(notes: "nvd:CVE-2025-1234");
// rule => scheme=semver, type=range, min=1.2.3, minInclusive=true, max=2.0.0, maxInclusive=false
```
Emit the resulting rule inside `AffectedPackage.NormalizedVersions` while continuing to populate `AffectedVersionRange.RangeExpression` for backward compatibility.
## 3. Merge dedupe flow
During merge, feed all package candidates through `NormalizedVersionRuleComparer.Instance` prior to persistence. The comparer orders by scheme → type → min → minInclusive → max → maxInclusive → value → notes, guaranteeing consistent document layout and making `$unwind` pipelines deterministic.
If multiple connectors emit identical constraints, the merge layer should:
1. Combine provenance entries (preserving one per source).
2. Preserve a single normalized rule instance (thanks to `NormalizedVersionRuleEqualityComparer.Instance`).
3. Attach `decisionReason="precedence"` if one source overrides another.
## 4. Example Mongo pipeline
Use the following aggregation to locate advisories that affect a specific SemVer:
```javascript
db.advisories.aggregate([
{ $match: { "affectedPackages.type": "semver", "affectedPackages.identifier": "pkg:npm/lodash" } },
{ $unwind: "$affectedPackages" },
{ $unwind: "$affectedPackages.normalizedVersions" },
{ $match: {
$or: [
{ "affectedPackages.normalizedVersions.type": "exact",
"affectedPackages.normalizedVersions.value": "4.17.21" },
{ "affectedPackages.normalizedVersions.type": "range",
"affectedPackages.normalizedVersions.min": { $lte: "4.17.21" },
"affectedPackages.normalizedVersions.max": { $gt: "4.17.21" } },
{ "affectedPackages.normalizedVersions.type": "gte",
"affectedPackages.normalizedVersions.min": { $lte: "4.17.21" } },
{ "affectedPackages.normalizedVersions.type": "lte",
"affectedPackages.normalizedVersions.max": { $gte: "4.17.21" } }
]
}},
{ $project: { advisoryKey: 1, title: 1, "affectedPackages.identifier": 1 } }
]);
```
Pair this query with the indexes listed in [Normalized Versions Query Guide](mongo_indices.md).
## 5. Recommended indexes
| Collection | Index | Purpose |
|------------|-------|---------|
| `advisory` | `{ "affectedPackages.identifier": 1, "affectedPackages.normalizedVersions.scheme": 1, "affectedPackages.normalizedVersions.type": 1 }` (compound, multikey) | Speeds up `$match` on identifier + rule style. |
| `advisory` | `{ "affectedPackages.normalizedVersions.value": 1 }` (sparse) | Optimizes lookups for exact version hits. |
Coordinate with the Storage team when enabling these indexes so deployment windows account for collection size.
## 6. Dual-write rollout
Follow the operational checklist in `docs/ops/migrations/SEMVER_STYLE.md`. The summary:
1. **Dual write (now)** – emit both legacy `versionRanges` and the new `normalizedVersions`.
2. **Backfill** – follow the storage migration in `docs/ops/migrations/SEMVER_STYLE.md` to rewrite historical advisories before switching consumers.
3. **Verify** – run the aggregation above (with `explain("executionStats")`) to ensure the new indexes are used.
4. **Cutover** – after consumers switch to normalized rules, mark the old `rangeExpression` as deprecated.
## 7. Checklist for connectors & merge
- [ ] Populate `SemVerPrimitive` for every SemVer-friendly constraint.
- [ ] Call `ToNormalizedVersionRule` and store the result.
- [ ] Emit provenance masks covering both `versionRanges[].primitives.semver` and `normalizedVersions[]`.
- [ ] Ensure merge deduping relies on the canonical comparer.
- [ ] Capture merge decisions via `decisionReason`.
- [ ] Confirm integration tests include fixtures with normalized rules and SemVer styles.
For deeper query examples and maintenance tasks, continue with [Normalized Versions Query Guide](mongo_indices.md).

106
docs/dev/mongo_indices.md Normal file
View File

@@ -0,0 +1,106 @@
# Normalized Versions Query Guide
This guide complements the Sprint 1–2 normalized versions rollout. It documents recommended indexes and aggregation patterns for querying `AffectedPackage.normalizedVersions`.
## 1. Recommended indexes
When `feedser.storage.enableSemVerStyle` is enabled, advisories expose a flattened
`normalizedVersions` array at the document root. Create these indexes in `mongosh`
after the migration completes (adjust collection name if you use a prefix):
```javascript
db.advisories.createIndex(
{
"normalizedVersions.packageId": 1,
"normalizedVersions.scheme": 1,
"normalizedVersions.type": 1
},
{ name: "advisory_normalizedVersions_pkg_scheme_type" }
);
db.advisories.createIndex(
{ "normalizedVersions.value": 1 },
{ name: "advisory_normalizedVersions_value", sparse: true }
);
```
- The compound index accelerates `$match` stages that filter by package identifier and rule style without unwinding `affectedPackages`.
- The sparse index keeps storage costs low while supporting pure exact-version lookups (type `exact`).
The storage bootstrapper creates the same indexes automatically when the feature flag is enabled.
## 2. Query patterns
### 2.1 Determine if a specific version is affected
```javascript
db.advisories.aggregate([
{ $match: { "normalizedVersions.packageId": "pkg:npm/lodash" } },
{ $unwind: "$normalizedVersions" },
{ $match: {
$or: [
{ "normalizedVersions.type": "exact",
"normalizedVersions.value": "4.17.21" },
{ "normalizedVersions.type": "range",
"normalizedVersions.min": { $lte: "4.17.21" },
"normalizedVersions.max": { $gt: "4.17.21" } },
{ "normalizedVersions.type": "gte",
"normalizedVersions.min": { $lte: "4.17.21" } },
{ "normalizedVersions.type": "lte",
"normalizedVersions.max": { $gte: "4.17.21" } }
]
}},
{ $project: { advisoryKey: 1, title: 1, "normalizedVersions.packageId": 1 } }
]);
```
Use this pipeline during Sprint 2 staging validation runs. Invoke `explain("executionStats")` to confirm the compound index is selected.
### 2.2 Locate advisories missing normalized rules
```javascript
db.advisories.aggregate([
{ $match: { $or: [
{ "normalizedVersions": { $exists: false } },
{ "normalizedVersions": { $size: 0 } }
] } },
{ $project: { advisoryKey: 1, affectedPackages: 1 } }
]);
```
Run this query after backfill jobs to identify gaps that still rely solely on `rangeExpression`.
### 2.3 Deduplicate overlapping rules
```javascript
db.advisories.aggregate([
{ $unwind: "$normalizedVersions" },
{ $group: {
_id: {
identifier: "$normalizedVersions.packageId",
scheme: "$normalizedVersions.scheme",
type: "$normalizedVersions.type",
min: "$normalizedVersions.min",
minInclusive: "$normalizedVersions.minInclusive",
max: "$normalizedVersions.max",
maxInclusive: "$normalizedVersions.maxInclusive",
value: "$normalizedVersions.value"
},
advisories: { $addToSet: "$advisoryKey" },
notes: { $addToSet: "$normalizedVersions.notes" }
}},
{ $match: { "advisories.1": { $exists: true } } },
{ $sort: { "_id.identifier": 1, "_id.type": 1 } }
]);
```
Use this to confirm the merge dedupe logic keeps only one normalized rule per unique constraint.
## 3. Operational checklist
- [ ] Create the indexes in staging before toggling dual-write in production.
- [ ] Capture explain plans and attach them to the release notes.
- [ ] Notify downstream services that consume advisory snapshots about the new `normalizedVersions` array.
- [ ] Update export fixtures once dedupe verification passes.
Additional background and mapper examples live in [Feedser SemVer Merge Playbook](merge_semver_playbook.md).