docs consolidation
This commit is contained in:
StellaOps Bot
@@ -1,220 +0,0 @@
|
||||
# Excitor Connector Packaging Guide
|
||||
|
||||
> **Audience:** teams implementing new Excitor provider plug‑ins (CSAF feeds,
|
||||
> OpenVEX attestations, etc.)
|
||||
> **Prerequisites:** read `docs/modules/excitor/architecture.md` and the module
|
||||
> `AGENTS.md` in `src/Excititor/__Libraries/StellaOps.Excititor.Connectors.Abstractions/`.
|
||||
|
||||
The Excitor connector SDK gives you:
|
||||
|
||||
- `VexConnectorBase` – deterministic logging, SHA‑256 helpers, time provider.
|
||||
- `VexConnectorOptionsBinder` – strongly typed YAML/JSON configuration binding.
|
||||
- `IVexConnectorOptionsValidator<T>` – custom validation hooks (offline defaults, auth invariants).
|
||||
- `VexConnectorDescriptor` & metadata helpers for consistent telemetry.
|
||||
|
||||
This guide explains how to package a connector so the Excitor Worker/WebService
|
||||
can load it via the plugin host.
|
||||
|
||||
---
|
||||
|
||||
## 1. Project layout
|
||||
|
||||
Start from the template under
|
||||
`docs/dev/templates/excitor-connector/`. It contains:
|
||||
|
||||
```
|
||||
Excitor.MyConnector/
|
||||
├── src/
|
||||
│ ├── Excitor.MyConnector.csproj
|
||||
│ ├── MyConnectorOptions.cs
|
||||
│ ├── MyConnector.cs
|
||||
│ └── MyConnectorPlugin.cs
|
||||
└── manifest/
|
||||
└── connector.manifest.yaml
|
||||
```
|
||||
|
||||
Key points:
|
||||
|
||||
- Target `net10.0`, enable `TreatWarningsAsErrors`, reference the
|
||||
`StellaOps.Excitor.Connectors.Abstractions` project (or NuGet once published).
|
||||
- Keep project ID prefix `StellaOps.Excitor.Connectors.<Provider>` so the
|
||||
plugin loader can discover it with the default search pattern.
|
||||
|
||||
### 1.1 csproj snippet
|
||||
|
||||
```xml
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
<PropertyGroup>
|
||||
<TargetFramework>net10.0</TargetFramework>
|
||||
<Nullable>enable</Nullable>
|
||||
<ImplicitUsings>enable</ImplicitUsings>
|
||||
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
|
||||
</PropertyGroup>
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="..\..\..\src\StellaOps.Excitor.Connectors.Abstractions\StellaOps.Excitor.Connectors.Abstractions.csproj" />
|
||||
</ItemGroup>
|
||||
</Project>
|
||||
```
|
||||
|
||||
Adjust the `ProjectReference` for your checkout (or switch to a NuGet package
|
||||
once published).
|
||||
|
||||
---
|
||||
|
||||
## 2. Implement the connector
|
||||
|
||||
1. **Options model** – create an options POCO with data-annotation attributes.
|
||||
Bind it via `VexConnectorOptionsBinder.Bind<TOptions>` in your connector
|
||||
constructor or `ValidateAsync`.
|
||||
2. **Validator** – implement `IVexConnectorOptionsValidator<TOptions>` to add
|
||||
complex checks (e.g., ensure both `clientId` and `clientSecret` are present).
|
||||
3. **Connector** – inherit from `VexConnectorBase`. Implement:
|
||||
- `ValidateAsync` – run binder/validators, log configuration summary.
|
||||
- `FetchAsync` – stream raw documents to `context.RawSink`.
|
||||
- `NormalizeAsync` – convert raw documents into `VexClaimBatch` via
|
||||
format-specific normalizers (`context.Normalizers`).
|
||||
4. **Plugin adapter** – expose the connector via a plugin entry point so the
|
||||
host can instantiate it.
|
||||
|
||||
### 2.1 Options binding example
|
||||
|
||||
```csharp
|
||||
public sealed class MyConnectorOptions
|
||||
{
|
||||
[Required]
|
||||
[Url]
|
||||
public string CatalogUri { get; set; } = default!;
|
||||
|
||||
[Required]
|
||||
public string ApiKey { get; set; } = default!;
|
||||
|
||||
[Range(1, 64)]
|
||||
public int MaxParallelRequests { get; set; } = 4;
|
||||
}
|
||||
|
||||
public sealed class MyConnectorOptionsValidator : IVexConnectorOptionsValidator<MyConnectorOptions>
|
||||
{
|
||||
public void Validate(VexConnectorDescriptor descriptor, MyConnectorOptions options, IList<string> errors)
|
||||
{
|
||||
if (!options.CatalogUri.StartsWith("https://", StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
errors.Add("CatalogUri must use HTTPS.");
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Bind inside the connector:
|
||||
|
||||
```csharp
|
||||
private readonly MyConnectorOptions _options;
|
||||
|
||||
public MyConnector(VexConnectorDescriptor descriptor, ILogger<MyConnector> logger, TimeProvider timeProvider)
|
||||
: base(descriptor, logger, timeProvider)
|
||||
{
|
||||
// `settings` comes from the orchestrator; validators registered via DI.
|
||||
_options = VexConnectorOptionsBinder.Bind<MyConnectorOptions>(
|
||||
descriptor,
|
||||
VexConnectorSettings.Empty,
|
||||
validators: new[] { new MyConnectorOptionsValidator() });
|
||||
}
|
||||
```
|
||||
|
||||
Replace `VexConnectorSettings.Empty` with the actual settings from context
|
||||
inside `ValidateAsync`.
|
||||
|
||||
---
|
||||
|
||||
## 3. Plugin adapter & manifest
|
||||
|
||||
Create a simple plugin class that implements
|
||||
`StellaOps.Plugin.IConnectorPlugin`. The Worker/WebService plugin host uses
|
||||
this contract today.
|
||||
|
||||
```csharp
|
||||
public sealed class MyConnectorPlugin : IConnectorPlugin
|
||||
{
|
||||
private static readonly VexConnectorDescriptor Descriptor =
|
||||
new("excitor:my-provider", VexProviderKind.Vendor, "My Provider VEX");
|
||||
|
||||
public string Name => Descriptor.DisplayName;
|
||||
|
||||
public bool IsAvailable(IServiceProvider services) => true; // inject feature flags if needed
|
||||
|
||||
public IFeedConnector Create(IServiceProvider services)
|
||||
{
|
||||
var logger = services.GetRequiredService<ILogger<MyConnector>>();
|
||||
var timeProvider = services.GetRequiredService<TimeProvider>();
|
||||
return new MyConnector(Descriptor, logger, timeProvider);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
> **Note:** the Excitor Worker currently instantiates connectors through the
|
||||
> shared `IConnectorPlugin` contract. Once a dedicated Excitor plugin interface
|
||||
> lands you simply swap the base interface; the descriptor/connector code
|
||||
> remains unchanged.
|
||||
|
||||
Provide a manifest describing the assembly for operational tooling:
|
||||
|
||||
```yaml
|
||||
# manifest/connector.manifest.yaml
|
||||
id: excitor-my-provider
|
||||
assembly: StellaOps.Excitor.Connectors.MyProvider.dll
|
||||
entryPoint: StellaOps.Excitor.Connectors.MyProvider.MyConnectorPlugin
|
||||
description: >
|
||||
Official VEX feed for ExampleCorp products (CSAF JSON, daily updates).
|
||||
tags:
|
||||
- excitor
|
||||
- csaf
|
||||
- vendor
|
||||
```
|
||||
|
||||
Store manifests under `/opt/stella/excitor/plugins/<connector>/manifest/` in
|
||||
production so the deployment tooling can inventory and verify plug‑ins.
|
||||
|
||||
---
|
||||
|
||||
## 4. Packaging workflow
|
||||
|
||||
1. `dotnet publish -c Release` → copy the published DLLs to
|
||||
`/opt/stella/excitor/plugins/<Provider>/`.
|
||||
2. Place `connector.manifest.yaml` next to the binaries.
|
||||
3. Restart the Excitor Worker or WebService (hot reload not supported yet).
|
||||
4. Verify logs: `VEX-ConnectorLoader` should list the connector descriptor.
|
||||
|
||||
### 4.1 Offline kits
|
||||
|
||||
- Add the connector folder (binaries + manifest) to the Offline Kit bundle.
|
||||
- Include a `settings.sample.yaml` demonstrating offline-friendly defaults.
|
||||
- Document any external dependencies (e.g., SHA mirrors) in the manifest `notes`
|
||||
field.
|
||||
|
||||
---
|
||||
|
||||
## 5. Testing checklist
|
||||
|
||||
- Unit tests around options binding & validators.
|
||||
- Integration tests (future `StellaOps.Excitor.Connectors.Abstractions.Tests`)
|
||||
verifying deterministic logging scopes:
|
||||
`logger.BeginScope` should produce `vex.connector.id`, `vex.connector.kind`,
|
||||
and `vex.connector.operation`.
|
||||
- Deterministic SHA tests: repeated `CreateRawDocument` calls with identical
|
||||
content must return the same digest.
|
||||
|
||||
---
|
||||
|
||||
## 6. Reference template
|
||||
|
||||
See `docs/dev/templates/excitor-connector/` for the full quick‑start including:
|
||||
|
||||
- Sample options class + validator.
|
||||
- Connector implementation inheriting from `VexConnectorBase`.
|
||||
- Plugin adapter + manifest.
|
||||
|
||||
Copy the directory, rename namespaces/IDs, then iterate on provider-specific
|
||||
logic.
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2025-10-17*
|
||||
@@ -1,25 +0,0 @@
|
||||
# AOC Normalization Removal Notes
|
||||
|
||||
_Last updated: 2025-10-29_
|
||||
|
||||
## Goal
|
||||
|
||||
Document follow-up actions for CONCELIER-CORE-AOC-19-004 as we unwind the final pieces of normalization from the ingestion/runtime path.
|
||||
|
||||
## Current Findings
|
||||
|
||||
- `AdvisoryRawService` and `MongoAdvisoryRawRepository` already preserve upstream ordering and duplicate aliases (trim-only). No additional code changes required there.
|
||||
- Observation layers (`AdvisoryObservationFactory`, `AdvisoryObservationQueryService`) still canonicalise aliases, PURLs, CPEs, and references. These need to be relaxed so Policy/overlays receive raw linksets and can own dedupe logic.
|
||||
- Linkset mapper continues to emit deterministic hints. We will keep the mapper but ensure observation output can surface both raw and canonical views for downstream services.
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. Introduce a raw linkset projection alongside the existing canonical mapper so Policy Engine can choose which flavour to consume. ✅ 2025-10-31: `AdvisoryObservation` now surfaces `RawLinkset`; Mongo documents store both canonical & raw shapes; tests/goldens updated.
|
||||
2. Update observation factory/query tests to assert duplicate handling and ordering with the relaxed projection. ✅ 2025-10-31.
|
||||
3. Refresh docs (`docs/ingestion/aggregation-only-contract.md`) once behaviour lands to explain the “raw vs canonical linkset” split. ✅ 2025-11-06: Added invariant notes and rollout guidance, linked to `docs/migration/no-merge.md` and `docs/dev/raw-linkset-backfill-plan.md`.
|
||||
4. Coordinate with Policy Guild to validate consumers against the new raw projection before flipping defaults. ↺ Ongoing — see action items in `docs/dev/raw-linkset-backfill-plan.md` (2025-10-31 handshake with POLICY-ENGINE-20-003 owners).
|
||||
|
||||
- 2025-11-05: Catalogued residual normalization paths tied to the legacy Merge service and outlined `noMergeEnabled` feature-toggle work to keep AOC ingestion fully merge-free.
|
||||
- 2025-11-05 19:20Z: Observation factory/linkset now preserve upstream ordering and duplicates; canonicalisation shifts to downstream services.
|
||||
- 2025-11-06: Documented post-merge rollout plan and annotated sprint trackers with analyzer gating updates.
|
||||
- 2025-11-06 23:30Z: Concelier core/linkset query paths now keep alias/reference casing & whitespace intact; alias filters switched to case-insensitive regex so raw data and lookups remain compatible.
|
||||
@@ -1,77 +0,0 @@
|
||||
# Authority Plug-in Scoped Service Coordination
|
||||
|
||||
> Created: 2025-10-19 — Plugin Platform Guild & Authority Core
|
||||
> Status: Completed (workshop held 2025-10-20 15:00–16:05 UTC)
|
||||
|
||||
This document tracks preparation, agenda, and outcomes for the scoped-service workshop required before implementing PLUGIN-DI-08-002.
|
||||
|
||||
## Objectives
|
||||
|
||||
- Inventory Authority plug-in surfaces that need scoped service lifetimes.
|
||||
- Confirm session/scope handling for identity-provider registrars and background jobs.
|
||||
- Assign follow-up tasks/actions with owners and due dates.
|
||||
|
||||
## Scheduling Snapshot
|
||||
|
||||
- **Meeting time:** 2025-10-20 15:00–16:00 UTC (10:00–11:00 CDT / 08:00–09:00 PDT).
|
||||
- **Facilitator:** Plugin Platform Guild — Alicia Rivera.
|
||||
- **Attendees (confirmed):** Authority Core — Jasmin Patel; Authority Security Guild — Mohan Singh; Plugin Platform — Alicia Rivera, Leah Chen.
|
||||
- **Optional invitees:** DevOps liaison — Sofia Ortega (accepted).
|
||||
- **Logistics:** Invites sent via shared calendar on 2025-10-19 15:30 UTC with Teams bridge + offline dial-in. Meeting notes will be captured here.
|
||||
- **Preparation deadline:** 2025-10-20 12:00 UTC — complete checklist below.
|
||||
|
||||
## Pre-work Checklist
|
||||
|
||||
- Review `ServiceBindingAttribute` contract introduced by PLUGIN-DI-08-001.
|
||||
- Collect existing Authority plug-in registration code paths to evaluate.
|
||||
- Audit background jobs that assume singleton lifetimes.
|
||||
- Identify plug-in health checks/telemetry surfaces impacted by scoped lifetimes.
|
||||
|
||||
### Pre-work References
|
||||
|
||||
| Focus | Path | Notes |
|
||||
|-------|------|-------|
|
||||
| Host DI wiring | `src/Authority/StellaOps.Authority/StellaOps.Authority/Program.cs:159` | Startup registers `IAuthorityIdentityProviderRegistry` as a singleton and invokes `AuthorityPluginLoader.RegisterPlugins(...)` before the container is built. Any scoped plugin services will currently be captured in the singleton registry context. |
|
||||
| Registrar discovery | `src/Authority/StellaOps.Authority/StellaOps.Authority/Plugins/AuthorityPluginLoader.cs:46` | Loader instantiates `IAuthorityPluginRegistrar` implementations via `Activator.CreateInstance`, so registrars cannot depend on host services yet. Need agreement on whether to move discovery post-build or introduce `ActivatorUtilities`. |
|
||||
| Registry aggregation | `src/Authority/StellaOps.Authority/StellaOps.Authority/AuthorityIdentityProviderRegistry.cs:16` | Registry caches `IIdentityProviderPlugin` instances at construction time. With scoped lifetimes we must revisit how providers are resolved (factory vs accessor). |
|
||||
| Standard registrar services | `src/Authority/StellaOps.Authority/StellaOps.Authority.Plugin.Standard/StandardPluginRegistrar.cs:21` | All plugin services are registered as singletons today (`StandardUserCredentialStore`, `StandardClientProvisioningStore`, hosted bootstrapper). This registrar is our baseline for migrating to scoped bindings. |
|
||||
| Hosted bootstrapper | `src/Authority/StellaOps.Authority/StellaOps.Authority.Plugin.Standard/Bootstrap/StandardPluginBootstrapper.cs:17` | Background job directly consumes `StandardUserCredentialStore`. If the store becomes scoped we will need an `IServiceScopeFactory` bridge. |
|
||||
| Password grant handler | `src/Authority/StellaOps.Authority/StellaOps.Authority/OpenIddict/Handlers/PasswordGrantHandlers.cs:26` | Password flow resolves `IIdentityProviderPlugin` during scoped requests. Scope semantics must ensure credential stores stay cancellation-aware. |
|
||||
| Client credential handler | `src/Authority/StellaOps.Authority/StellaOps.Authority/OpenIddict/Handlers/ClientCredentialsHandlers.cs:21` | Handler fetches provider + `ClientProvisioning` store; confirms need for consistent scoping in both user and client flows. |
|
||||
|
||||
## Preliminary Findings — 2025-10-20
|
||||
|
||||
- `IAuthorityIdentityProviderRegistry` must stop materialising provider singletons when scoped lifetimes land. Options to evaluate: make the registry itself scoped, convert it to a factory over `IServiceProvider`, or cache lightweight descriptors and resolve implementations on-demand.
|
||||
- `AuthorityPluginLoader` instantiates registrars without DI support. To let registrars request scoped helpers (e.g. `IServiceScopeFactory`) we may need a two-phase registration: discover types at build time, defer execution until the container is available.
|
||||
- Hosted bootstrap tasks (e.g. `StandardPluginBootstrapper`) will break if their dependencies become scoped. Workshop should align on using scoped pipelines inside `StartAsync` or shifting bootstrap work to queued jobs.
|
||||
- Standard plugin stores assume singleton access to Mongo collections and password hashing utilities. If we embrace scoped stores, document thread-safety expectations and reuse of Mongo clients across scopes.
|
||||
- OpenIddict handlers already run as scoped services; once providers move to scoped lifetimes we must ensure the new resolution path stays cancellation-aware and avoids redundant service resolution per request.
|
||||
- 2025-10-20 (PLUGIN-DI-08-003): Registry implementation updated to expose metadata + scoped handles; OpenIddict flows, bootstrap endpoints, and `/health` now resolve providers via scoped leases with accompanying test coverage.
|
||||
- 2025-10-20 (PLUGIN-DI-08-004): Authority plugin loader now instantiates registrars via scoped DI activations and honours `[ServiceBinding]` metadata in plugin assemblies.
|
||||
- 2025-10-20 (PLUGIN-DI-08-005): `StandardPluginBootstrapper` shifted to scope-per-run execution using `IServiceScopeFactory`, enabling future scoped stores without singleton leaks.
|
||||
|
||||
## Draft Agenda
|
||||
|
||||
1. Context recap (5 min) — why scoped DI is needed; summary of PLUGIN-DI-08-001 changes.
|
||||
2. Authority plug-in surfaces (15 min) — registrars, background services, telemetry.
|
||||
3. Session handling strategy (10 min) — scope creation semantics, cancellation propagation.
|
||||
4. Action items & owners (10 min) — capture code/docs/test tasks with due dates.
|
||||
5. Risks & follow-ups (5 min) — dependencies, rollout sequencing.
|
||||
|
||||
## Notes
|
||||
|
||||
- Session opened with recap of scoped-service goals and PLUGIN-DI-08-001 changes, confirming Authority readiness to adopt `[ServiceBinding]` metadata.
|
||||
- Agreed to treat `IAuthorityIdentityProviderRegistry` as a scoped-factory facade rather than a singleton cache; registry will track descriptors and resolve implementations on-demand per request/worker scope.
|
||||
- Standard plug-in bootstrap will create scopes via `IServiceScopeFactory` and pass cancellation tokens through to avoid lingering singleton references.
|
||||
- Authority Plugin Loader will enumerate plug-in assemblies at startup but defer registrar activation until a scoped service provider is available, aligning with PLUGIN-DI-08-004 implementation.
|
||||
- Follow-up engineering tasks assigned to land PLUGIN-DI-08-002 code path adjustments and Authority host updates before 2025-10-24.
|
||||
|
||||
## Action Item Log
|
||||
|
||||
| Item | Owner | Due | Status | Notes |
|
||||
|------|-------|-----|--------|-------|
|
||||
| Confirm meeting time | Alicia Rivera | 2025-10-19 15:30 UTC | DONE | Calendar invite sent; all required attendees accepted |
|
||||
| Compile Authority plug-in DI entry points | Jasmin Patel | 2025-10-20 | DONE (2025-10-20) | Scoped-service touchpoints summarised in **Pre-work References** and **Preliminary Findings** ahead of the workshop. |
|
||||
| Outline scoped-session pattern for background jobs | Leah Chen | 2025-10-21 | DONE (2025-10-20) | Pattern agreed: bootstrap services must open transient scopes per execution via `IServiceScopeFactory`; document update to follow in PLUGIN-DI-08-002 patch. |
|
||||
| Update PLUGIN-DI-08-002 implementation plan | Alicia Rivera | 2025-10-21 | DONE (2025-10-20) | Task board + correspoding sprint file `../implplan/SPRINT_*.md` updated with scoped-integration delivery notes and test references. |
|
||||
| Sync Authority host backlog | Mohan Singh | 2025-10-21 | DONE (2025-10-20) | Authority/Plugin TASKS.md and correspoding sprint file `../implplan/SPRINT_*.md` entries reflect scoped-service completion. |
|
||||
@@ -1,53 +0,0 @@
|
||||
# Cartographer Graph Handshake Plan
|
||||
|
||||
> **Archived (2025-10-30).** Cartographer has been retired in favour of the Graph Indexer + Graph API platform (see `docs/devops/contracts-and-rules.md`). Keep this document only for historical reference; new work must reference `GRAPH-*` tasks and the Graph module docs.
|
||||
>
|
||||
> **2025-12-04 update:** The inspector contract requested here now lives at `docs/modules/graph/contracts/graph.inspect.v1.md` with schema `graph.inspect.v1.schema.json` and sample payloads under `docs/modules/graph/contracts/examples/`. Treat this file as historical background only.
|
||||
|
||||
_Status: 2025-10-29_
|
||||
|
||||
## Why this exists
|
||||
The Concelier/Excititor graph enrichment work (CONCELIER-GRAPH-21-001/002, EXCITITOR-GRAPH-21-001/002/005) and the merge-side coordination tasks (FEEDMERGE-COORD-02-901/902) are blocked on a clear contract with Cartographer and the Policy Engine. This document captures the minimum artefacts each guild owes so we can unblock the graph pipeline and resume implementation without re-scoping every stand-up.
|
||||
|
||||
## Deliverables by guild
|
||||
|
||||
### Cartographer Guild
|
||||
- **CARTO-GRAPH-21-002** (Inspector contract): publish the inspector payload schema (`graph.inspect.v1`) including the fields Cartographer needs from Concelier/Excititor (SBOM relationships, advisory/VEX linkouts, justification summaries). Target format: shared Proto/JSON schema stored under `src/Cartographer/Contracts/`.
|
||||
- **CARTO-GRAPH-21-005** (Inspector access patterns): document the query shapes Cartographer will execute (PURL → advisory, PURL → VEX statement, policy scope filters) so storage can project the right indexes/materialized views. Include sample `mongosh` queries and desired TTL/limit behaviour.
|
||||
- Provide a test harness (e.g., Postman collection or integration fixture) Cartographer will use to validate the Concelier/Excititor endpoints once they land.
|
||||
|
||||
### Concelier Core Guild
|
||||
- Derive adjacency data from SBOM normalization as described in CONCELIER-GRAPH-21-001 (depends on `CONCELIER-POLICY-20-002`). Once Cartographer publishes the schema above, implement:
|
||||
- Node payloads: component metadata, scopes, entrypoint annotations.
|
||||
- Edge payloads: `contains`, `depends_on`, `provides`, provenance array.
|
||||
- **Change events (CONCELIER-GRAPH-21-002)**: define `sbom.relationship.changed` event contract with tenant + context metadata, referencing Cartographer’s filter requirements. Include event samples and replay instructions in `docs/graph/concelier-events.md`.
|
||||
- Coordinate with Cartographer on pagination/streaming expectations (page size, continuation token, retention window).
|
||||
|
||||
### Excititor Core & Storage Guilds
|
||||
- **Inspector linkouts (EXCITITOR-GRAPH-21-001)**: expose Batched VEX/advisory lookup endpoint that accepts graph node PURLs and responds with raw document slices + justification metadata. Ensure Policy Engine scope enrichment (EXCITITOR-POLICY-20-002) feeds this response so Cartographer does not need to call multiple services.
|
||||
- **Overlay enrichment (EXCITITOR-GRAPH-21-002)**: align the overlay metadata with Cartographer’s schema once it lands (include justification summaries, document versions, and provenance).
|
||||
- **Indexes/materialized views (EXCITITOR-GRAPH-21-005)**: after Cartographer publishes query shapes, create the necessary indexes (PURL + tenant, policy scope) and document migrations in storage runbooks. Provide load testing evidence before enabling in production.
|
||||
|
||||
### Policy Guild
|
||||
- **CONCELIER-POLICY-20-002**: publish the enriched linkset schema that powers both Concelier and Excititor payloads. Include enumerations for relationship types and scope tags.
|
||||
- Share the Policy Engine timeline for policy overlay metadata (`POLICY-ENGINE-30-001`) so Excititor can plan the overlay enrichment delivery.
|
||||
|
||||
## Shared action items
|
||||
|
||||
| Owner | Task | Deadline | Notes |
|
||||
|-------|------|----------|-------|
|
||||
| Cartographer | Publish inspector schema + query patterns (`CARTO-GRAPH-21-002`/`21-005`) | 2025-11-04 | Attach schema files + examples to this doc once merged. |
|
||||
| Concelier Core | Draft change-event payload with sample JSON | 2025-11-06 | Blocked until Cartographer schema lands; prepare skeleton PR in `docs/graph/concelier-events.md`. |
|
||||
| Excititor Core/Storage | Prototype batch linkout API + index design doc | 2025-11-07 | Leverage Cartographer query patterns to size indexes; include perf targets. |
|
||||
| Policy Guild | Confirm linkset enrichment fields + overlay timeline | 2025-11-05 | Needed to unblock both Concelier enrichment and Excititor overlay tasks. |
|
||||
|
||||
## Reporting
|
||||
- Track progress in the `#cartographer-handshake` Slack thread (create once Cartographer posts the schema MR).
|
||||
- During the twice-weekly graph sync, review outstanding checklist items above and update the task notes (`TASKS.md`) so the backlog reflects real-time status.
|
||||
- Once the schema and query contracts are merged, the Concelier/Excititor teams can flip their tasks from **BLOCKED** to **DOING** and attach implementation plans referencing this document.
|
||||
|
||||
## Appendix: references
|
||||
- `CONCELIER-GRAPH-21-001`, `CONCELIER-GRAPH-21-002` (Concelier Core task board)
|
||||
- `EXCITITOR-GRAPH-21-001`, `EXCITITOR-GRAPH-21-002`, `EXCITITOR-GRAPH-21-005` (Excititor Core/Storage task boards)
|
||||
- `CARTO-GRAPH-21-002`, `CARTO-GRAPH-21-005` (Cartographer task board)
|
||||
- `POLICY-ENGINE-30-001`, `CONCELIER-POLICY-20-002`, `EXCITITOR-POLICY-20-002` (Policy Engine roadmap)
|
||||
@@ -1,108 +0,0 @@
|
||||
# Normalized Versions Query Guide
|
||||
|
||||
This guide complements the Sprint 1–2 normalized versions rollout. It documents recommended indexes and aggregation patterns for querying `AffectedPackage.normalizedVersions`.
|
||||
|
||||
For a field-by-field look at how normalized rules persist in MongoDB (including provenance metadata), see Section 8 of the [Concelier SemVer Merge Playbook](merge_semver_playbook.md).
|
||||
|
||||
## 1. Recommended indexes
|
||||
|
||||
When `concelier.storage.enableSemVerStyle` is enabled, advisories expose a flattened
|
||||
`normalizedVersions` array at the document root. Create these indexes in `mongosh`
|
||||
after the migration completes (adjust collection name if you use a prefix):
|
||||
|
||||
```javascript
|
||||
db.advisories.createIndex(
|
||||
{
|
||||
"normalizedVersions.packageId": 1,
|
||||
"normalizedVersions.scheme": 1,
|
||||
"normalizedVersions.type": 1
|
||||
},
|
||||
{ name: "advisory_normalizedVersions_pkg_scheme_type" }
|
||||
);
|
||||
|
||||
db.advisories.createIndex(
|
||||
{ "normalizedVersions.value": 1 },
|
||||
{ name: "advisory_normalizedVersions_value", sparse: true }
|
||||
);
|
||||
```
|
||||
|
||||
- The compound index accelerates `$match` stages that filter by package identifier and rule style without unwinding `affectedPackages`.
|
||||
- The sparse index keeps storage costs low while supporting pure exact-version lookups (type `exact`).
|
||||
|
||||
The storage bootstrapper creates the same indexes automatically when the feature flag is enabled.
|
||||
|
||||
## 2. Query patterns
|
||||
|
||||
### 2.1 Determine if a specific version is affected
|
||||
|
||||
```javascript
|
||||
db.advisories.aggregate([
|
||||
{ $match: { "normalizedVersions.packageId": "pkg:npm/lodash" } },
|
||||
{ $unwind: "$normalizedVersions" },
|
||||
{ $match: {
|
||||
$or: [
|
||||
{ "normalizedVersions.type": "exact",
|
||||
"normalizedVersions.value": "4.17.21" },
|
||||
{ "normalizedVersions.type": "range",
|
||||
"normalizedVersions.min": { $lte: "4.17.21" },
|
||||
"normalizedVersions.max": { $gt: "4.17.21" } },
|
||||
{ "normalizedVersions.type": "gte",
|
||||
"normalizedVersions.min": { $lte: "4.17.21" } },
|
||||
{ "normalizedVersions.type": "lte",
|
||||
"normalizedVersions.max": { $gte: "4.17.21" } }
|
||||
]
|
||||
}},
|
||||
{ $project: { advisoryKey: 1, title: 1, "normalizedVersions.packageId": 1 } }
|
||||
]);
|
||||
```
|
||||
|
||||
Use this pipeline during Sprint 2 staging validation runs. Invoke `explain("executionStats")` to confirm the compound index is selected.
|
||||
|
||||
### 2.2 Locate advisories missing normalized rules
|
||||
|
||||
```javascript
|
||||
db.advisories.aggregate([
|
||||
{ $match: { $or: [
|
||||
{ "normalizedVersions": { $exists: false } },
|
||||
{ "normalizedVersions": { $size: 0 } }
|
||||
] } },
|
||||
{ $project: { advisoryKey: 1, affectedPackages: 1 } }
|
||||
]);
|
||||
```
|
||||
|
||||
Run this query after backfill jobs to identify gaps that still rely solely on `rangeExpression`.
|
||||
|
||||
### 2.3 Deduplicate overlapping rules
|
||||
|
||||
```javascript
|
||||
db.advisories.aggregate([
|
||||
{ $unwind: "$normalizedVersions" },
|
||||
{ $group: {
|
||||
_id: {
|
||||
identifier: "$normalizedVersions.packageId",
|
||||
scheme: "$normalizedVersions.scheme",
|
||||
type: "$normalizedVersions.type",
|
||||
min: "$normalizedVersions.min",
|
||||
minInclusive: "$normalizedVersions.minInclusive",
|
||||
max: "$normalizedVersions.max",
|
||||
maxInclusive: "$normalizedVersions.maxInclusive",
|
||||
value: "$normalizedVersions.value"
|
||||
},
|
||||
advisories: { $addToSet: "$advisoryKey" },
|
||||
notes: { $addToSet: "$normalizedVersions.notes" }
|
||||
}},
|
||||
{ $match: { "advisories.1": { $exists: true } } },
|
||||
{ $sort: { "_id.identifier": 1, "_id.type": 1 } }
|
||||
]);
|
||||
```
|
||||
|
||||
Use this to confirm the merge dedupe logic keeps only one normalized rule per unique constraint.
|
||||
|
||||
## 3. Operational checklist
|
||||
|
||||
- [ ] Create the indexes in staging before toggling dual-write in production.
|
||||
- [ ] Capture explain plans and attach them to the release notes.
|
||||
- [ ] Notify downstream services that consume advisory snapshots about the new `normalizedVersions` array.
|
||||
- [ ] Update export fixtures once dedupe verification passes.
|
||||
|
||||
Additional background and mapper examples live in [Concelier SemVer Merge Playbook](merge_semver_playbook.md).
|
||||
@@ -1,55 +0,0 @@
|
||||
# Normalized Versions Rollout Dashboard (Sprint 2 – Concelier)
|
||||
|
||||
_Status date: 2025-10-20 19:10 UTC_
|
||||
|
||||
This dashboard tracks connector readiness for emitting `AffectedPackage.NormalizedVersions` arrays and highlights upcoming coordination checkpoints. Use it alongside:
|
||||
|
||||
- [`src/Concelier/__Libraries/StellaOps.Concelier.Merge/RANGE_PRIMITIVES_COORDINATION.md`](../../src/Concelier/__Libraries/StellaOps.Concelier.Merge/RANGE_PRIMITIVES_COORDINATION.md) for detailed guidance and timelines.
|
||||
- [Concelier SemVer Merge Playbook](merge_semver_playbook.md) §8 for persisted Mongo document shapes.
|
||||
- [Normalized Versions Query Guide](mongo_indices.md) for index/query validation steps.
|
||||
|
||||
## Key milestones
|
||||
|
||||
- **2025-10-21** – Cccs and Cisco connectors finalize normalized rule emission and share merge-counter screenshots.
|
||||
- **2025-10-22** – CertBund localisation translator reviewed; blockers escalated if localisation guidance slips.
|
||||
- **2025-10-23** – ICS-CISA confirms SemVer reuse vs new firmware scheme and files Models ticket if needed.
|
||||
- **2025-10-24** – KISA firmware scheme proposal due; Merge provides same-day review.
|
||||
- **2025-10-25** – Merge runs cross-connector validation before enabling normalized-rule union logic by default.
|
||||
|
||||
## Connector readiness matrix
|
||||
|
||||
| Connector | Owner team | Normalized versions status | Last update | Next action / link |
|
||||
|-----------|------------|---------------------------|-------------|--------------------|
|
||||
| Acsc | BE-Conn-ACSC | ❌ Not started – normalized helper pending relay stability | 2025-10-20 | Prepare builder integration plan for 2025-10-24 kickoff; update `src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Acsc/TASKS.md` once branch opens. |
|
||||
| Cccs | BE-Conn-CCCS | ⚠️ DOING – trailing-version helper MR reviewing (due 2025-10-21) | 2025-10-20 | Land helper + fixture refresh, post merge-counter screenshot; `src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Cccs/TASKS.md`. |
|
||||
| CertBund | BE-Conn-CERTBUND | ⚠️ In progress – localisation translator WIP (due 2025-10-22) | 2025-10-20 | Finish translator + provenance notes, regenerate fixtures; `src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.CertBund/TASKS.md`. |
|
||||
| CertCc | BE-Conn-CERTCC | ✅ Complete – `certcc.vendor` rules emitting | 2025-10-20 | Monitor VINCE payload changes; no action. |
|
||||
| Kev | BE-Conn-KEV | ✅ Complete – catalog/due-date rules verified | 2025-10-20 | Routine monitoring only. |
|
||||
| Cve | BE-Conn-CVE | ✅ Complete – SemVer normalized rules live | 2025-10-20 | Keep fixtures in sync as CVE schema evolves. |
|
||||
| Ghsa | BE-Conn-GHSA | ✅ Complete – rollout merged 2025-10-11 | 2025-10-20 | Maintain parity with OSV ecosystems; no action. |
|
||||
| Osv | BE-Conn-OSV | ✅ Complete – normalized rules shipping | 2025-10-20 | Watch for new ecosystems; refresh fixtures as needed. |
|
||||
| Ics.Cisa | BE-Conn-ICS-CISA | ⚠️ Decision pending – exact SemVer promotion due 2025-10-23 | 2025-10-20 | Promote primitives or request new scheme; `src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ics.Cisa/TASKS.md`. |
|
||||
| Kisa | BE-Conn-KISA | ⚠️ Proposal drafting – firmware scheme due 2025-10-24 | 2025-10-20 | Finalise `kisa.build` proposal with Models; update mapper/tests; `src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Kisa/TASKS.md`. |
|
||||
| Ru.Bdu | BE-Conn-BDU | ✅ Complete – `ru-bdu.raw` rules live | 2025-10-20 | Continue monitoring UTF-8 handling; no action. |
|
||||
| Ru.Nkcki | BE-Conn-Nkcki | ✅ Complete – normalized rules emitted | 2025-10-20 | Maintain transliteration guidance; no action. |
|
||||
| Vndr.Apple | BE-Conn-Apple | ✅ Complete – normalized arrays emitting | 2025-10-20 | Add beta-channel coverage follow-up; see module README. |
|
||||
| Vndr.Cisco | BE-Conn-Cisco | ⚠️ DOING – normalized promotion branch open (due 2025-10-21) | 2025-10-20 | Merge helper branch, refresh fixtures, post counters; `src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Vndr.Cisco/TASKS.md`. |
|
||||
| Vndr.Msrc | BE-Conn-MSRC | ✅ Complete – `msrc.build` rules emitting | 2025-10-20 | Monitor monthly rollups; no action. |
|
||||
| Nvd | BE-Conn-NVD | ✅ Complete – normalized SemVer output live | 2025-10-20 | Keep provenance aligned with CVE IDs; monitor export parity toggle. |
|
||||
|
||||
Legend: ✅ complete, ⚠️ in progress/partial, ❌ not started.
|
||||
|
||||
## Monitoring
|
||||
|
||||
- Merge now emits `concelier.merge.normalized_rules` (tags: `package_type`, `scheme`) and `concelier.merge.normalized_rules_missing` (tags: `package_type`). Track these counters to confirm normalized arrays land as connectors roll out.
|
||||
- Expect `normalized_rules_missing` to trend toward zero as each connector flips on normalized output. Investigate any sustained counts by checking the corresponding module `TASKS.md`.
|
||||
|
||||
## Implementation tips
|
||||
|
||||
- When a connector only needs to populate `AffectedPackage.NormalizedVersions` (without reusing range primitives), call `SemVerRangeRuleBuilder.BuildNormalizedRules(rawRange, patchedVersion, note)` to project the normalized rule list directly. This avoids re-wrapping `SemVerRangeBuildResult` instances and keeps provenance notes consistent with the shared builder.
|
||||
|
||||
## How to use this dashboard
|
||||
|
||||
1. Before opening a connector PR, update the module `TASKS.md` entry and drop a short bullet here (status + timestamp).
|
||||
2. When a connector lands normalized outputs, flip the status to ✅ and note any rollout toggles (feature flags, fixture regenerations).
|
||||
3. If a dependency or blocker emerges, add it both in the module `TASKS.md` and in this matrix so merge/storage can escalate quickly.
|
||||
@@ -1,56 +0,0 @@
|
||||
# Raw Linkset Backfill & Adoption Plan
|
||||
|
||||
_Last updated: 2025-11-06_
|
||||
Owners: Concelier Storage Guild, DevOps Guild, Policy Guild
|
||||
|
||||
## Context
|
||||
|
||||
- Concelier observations now emit both a **canonical linkset** (deduped, normalised identifiers) and a **raw linkset** (`rawLinkset`) that preserves upstream ordering, duplicates, and original pointer metadata.
|
||||
- Existing `concelier.advisory_observations` documents created before 2025-10-31 do **not** contain the `rawLinkset` field.
|
||||
- Policy Engine selection joiners (`POLICY-ENGINE-20-003`) will switch to the raw projection once backfill completes and consumers validate fixtures.
|
||||
|
||||
## Objectives
|
||||
|
||||
1. Populate `rawLinkset` for historical observations across online clusters and Offline Kit bundles without breaking append-only guarantees.
|
||||
2. Provide migration scripts + runbook so operators can rehearse in staging (and air-gapped deployments) before production rollout.
|
||||
3. Unblock Policy Engine adoption by guaranteeing dual projections exist for all tenants.
|
||||
|
||||
## Deliverables
|
||||
|
||||
- [ ] **Migration script** (`20251104_advisory_observations_raw_linkset_backfill.csx`)
|
||||
- Iterates observations lacking `rawLinkset`
|
||||
- Rehydrates raw document via existing snapshot (or cached DTO)
|
||||
- Reuses `AdvisoryObservationFactory.CreateRawLinkset`
|
||||
- Writes using `$set` with optimistic retry; preserves `updatedAt` via `setOnInsert`
|
||||
- [ ] **Offline Kit updater** (extend `ops/offline-kit/scripts/export_offline_bundle.py`) to patch bundles in-place
|
||||
- [ ] **Runbook** covering:
|
||||
- Pre-check query: `db.concelier.advisory_observations.countDocuments({ rawLinkset: { $exists: false } })`
|
||||
- Backup procedure (`mongodump` or snapshot requirement)
|
||||
- Dry-run mode limiting batches by tenant
|
||||
- Metrics/telemetry expectations (`concelier.migrations.documents_processed_total`)
|
||||
- Rollback (no-op because field addition; note to retain snapshot for verification)
|
||||
- [ ] **Fixture updates** ensuring storage/CLI/Policy tests include `rawLinkset`
|
||||
- [ ] **Policy Engine follow-up** to flip joiners once `rawLinkset` population reaches 100% (tracked via metrics).
|
||||
|
||||
## Timeline
|
||||
|
||||
| Date (UTC) | Milestone | Notes |
|
||||
|------------|-----------|-------|
|
||||
| 2025-10-31 | Handshake w/ Policy | Agreement to consume `rawLinkset`; this document created. |
|
||||
| 2025-11-01 | Draft migration script | Validate against staging dataset snapshots. |
|
||||
| 2025-11-04 | Storage task CONCELIER-STORE-AOC-19-005 due | Deliver script + runbook for review. |
|
||||
| 2025-11-06 | Staging backfill rehearsal | Target < 30 min runtime on 5M observations; docs refreshed to highlight raw vs canonical invariants and analyzer guardrails. |
|
||||
| 2025-11-08 | Policy fixtures updated | POL engine branch consumes `rawLinkset`. |
|
||||
| 2025-11-11 | Production rollout window | Pending DevOps sign-off after rehearsals. |
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Do we need archival of the canonical-only projection for backwards compatibility exports? (Policy to confirm.)
|
||||
- Offline Kit delta: should we regenerate entire bundle or ship incremental patch? (DevOps reviewing.)
|
||||
- Metrics: add `raw_linkset_missing_total` counter to detect regressions post-backfill?
|
||||
|
||||
## Next Actions
|
||||
|
||||
- [ ] Concelier Storage Guild: prototype migration script, share for review (`2025-11-01`).
|
||||
- [ ] DevOps Guild: schedule staging rehearsal + update `docs/deploy/containers.md` with new runbook section.
|
||||
- [ ] Policy Guild: prepare feature flag/branch to switch joiners once metrics show zero missing `rawLinkset`.
|
||||
Reference in New Issue
Block a user