feat: Implement advisory event replay API with conflict explainers
Some checks failed
Docs CI / lint-and-preview (push) Has been cancelled

- Added `/concelier/advisories/{vulnerabilityKey}/replay` endpoint to return conflict summaries and explainers.
- Introduced `MergeConflictExplainerPayload` to structure conflict details including type, reason, and source rankings.
- Enhanced `MergeConflictSummary` to include structured explainer payloads and hashes for persisted conflicts.
- Updated `MirrorEndpointExtensions` to enforce rate limits and cache headers for mirror distribution endpoints.
- Refactored tests to cover new replay endpoint functionality and validate conflict explainers.
- Documented changes in TASKS.md, noting completion of mirror distribution endpoints and updated operational runbook.
This commit is contained in:
Vladimir Moushkov
2025-10-20 18:59:26 +03:00
parent 44ad31591c
commit 2b6304c9c3
20 changed files with 3966 additions and 3493 deletions

View File

@@ -7,7 +7,7 @@ Generated from SPRINTS.md and module TASKS.md files on 2025-10-19. Waves cluster
- Team Authority Core & Security Guild: read EXECPLAN.md Wave 0 and SPRINTS.md rows for `src/StellaOps.Authority/TASKS.md`. Focus on AUTH-DPOP-11-001 (DOING 2025-10-19), AUTH-MTLS-11-002 (DOING 2025-10-19). Confirm prerequisites (none) before starting and report status in module TASKS.md.
- Team Authority Core & Storage Guild: read EXECPLAN.md Wave 0 and SPRINTS.md rows for `src/StellaOps.Authority/TASKS.md`. Focus on AUTHSTORAGE-MONGO-08-001 (DONE 2025-10-19). Confirm prerequisites (none) before starting and report status in module TASKS.md.
- Team DevEx/CLI: read EXECPLAN.md Wave 0 and SPRINTS.md rows for `src/StellaOps.Cli/TASKS.md`. Focus on EXCITITOR-CLI-01-002 (TODO), CLI-RUNTIME-13-005 (TODO). Confirm prerequisites (external: EXCITITOR-CLI-01-001, EXCITITOR-EXPORT-01-001) before starting and report status in module TASKS.md.
- Team DevOps Guild: read EXECPLAN.md Wave 0 and SPRINTS.md rows for `ops/devops/TASKS.md`. Focus on DEVOPS-SEC-10-301 (DOING 2025-10-19); Wave0A prerequisites reconfirmed so remediation work may proceed. Keep module TASKS.md/Sprints in sync as patches land.
- Team DevOps Guild: read EXECPLAN.md Wave 0 and SPRINTS.md rows for `ops/devops/TASKS.md`. Focus on DEVOPS-SEC-10-301 (DONE 2025-10-20); Wave0A prerequisites reconfirmed so remediation work may proceed. Keep module TASKS.md/Sprints in sync as patches land.
- Team Diff Guild: read EXECPLAN.md Wave 0 and SPRINTS.md rows for `src/StellaOps.Scanner.Diff/TASKS.md`. Focus on SCANNER-DIFF-10-501 (TODO), SCANNER-DIFF-10-502 (TODO), SCANNER-DIFF-10-503 (TODO). Confirm prerequisites (none) before starting and report status in module TASKS.md.
- Team Docs Guild, Plugin Team: read EXECPLAN.md Wave 0 and SPRINTS.md rows for `docs/TASKS.md`. Focus on DOC4.AUTH-PDG (REVIEW). Confirm prerequisites (none) before starting and report status in module TASKS.md.
- Team Docs/CLI: read EXECPLAN.md Wave 0 and SPRINTS.md rows for `src/StellaOps.Cli/TASKS.md`. Focus on EXCITITOR-CLI-01-003 (TODO). Confirm prerequisites (external: EXCITITOR-CLI-01-001) before starting and report status in module TASKS.md.
@@ -142,7 +142,7 @@ Generated from SPRINTS.md and module TASKS.md files on 2025-10-19. Waves cluster
- Team Team Normalization & Storage Backbone: read EXECPLAN.md Wave 10 and SPRINTS.md rows for `src/StellaOps.Concelier.Storage.Mongo/TASKS.md`. Focus on FEEDSTORAGE-DATA-07-001 (TODO). Confirm prerequisites (internal: FEEDMERGE-ENGINE-07-001 (Wave 11)) before starting and report status in module TASKS.md.
### Wave 11
- Team BE-Merge: read EXECPLAN.md Wave 11 and SPRINTS.md rows for `src/StellaOps.Concelier.Merge/TASKS.md`. Focus on FEEDMERGE-ENGINE-07-001 (TODO). Confirm prerequisites (internal: FEEDSTORAGE-DATA-07-001 (Wave 10)) before starting and report status in module TASKS.md.
- Team BE-Merge: read EXECPLAN.md Wave 11 and SPRINTS.md rows for `src/StellaOps.Concelier.Merge/TASKS.md`. FEEDMERGE-ENGINE-07-001 marked DONE (2025-10-20); share conflict explainer rollout notes with Storage before Wave 10 resumes.
### Wave 12
- Team Concelier Export Guild: read EXECPLAN.md Wave 12 and SPRINTS.md rows for `src/StellaOps.Concelier.Exporter.Json/TASKS.md`. Focus on CONCELIER-EXPORT-08-201 (TODO). Confirm prerequisites (internal: FEEDCORE-ENGINE-07-001 (Wave 7)) before starting and report status in module TASKS.md.
@@ -151,7 +151,7 @@ Generated from SPRINTS.md and module TASKS.md files on 2025-10-19. Waves cluster
- Team Concelier Export Guild: read EXECPLAN.md Wave 13 and SPRINTS.md rows for `src/StellaOps.Concelier.Exporter.TrivyDb/TASKS.md`. Focus on CONCELIER-EXPORT-08-202 (DONE 2025-10-19). Confirm prerequisites (internal: CONCELIER-EXPORT-08-201 (Wave 12)) before starting and report status in module TASKS.md.
### Wave 14
- Team Concelier WebService Guild: read EXECPLAN.md Wave 14 and SPRINTS.md rows for `src/StellaOps.Concelier.WebService/TASKS.md`. Focus on CONCELIER-WEB-08-201 (TODO). Confirm prerequisites (internal: CONCELIER-EXPORT-08-201 (Wave 12), DEVOPS-MIRROR-08-001 (Wave 2)) before starting and report status in module TASKS.md.
- Team Concelier WebService Guild: read EXECPLAN.md Wave 14 and SPRINTS.md rows for `src/StellaOps.Concelier.WebService/TASKS.md`. CONCELIER-WEB-08-201 closed (2025-10-20); coordinate with DevOps for mirror smoke before promoting to stable.
### Wave 15
- Team BE-Conn-Stella: read EXECPLAN.md Wave 15 and SPRINTS.md rows for `src/StellaOps.Concelier.Connector.StellaOpsMirror/TASKS.md`. Focus on FEEDCONN-STELLA-08-001 (TODO). Confirm prerequisites (internal: CONCELIER-EXPORT-08-201 (Wave 12)) before starting and report status in module TASKS.md.
@@ -386,7 +386,7 @@ Generated from SPRINTS.md and module TASKS.md files on 2025-10-19. Waves cluster
- **Sprint 10** · DevOps Perf
- Team: DevOps Guild
- Path: `ops/devops/TASKS.md`
1. [DOING] DEVOPS-SEC-10-301 — Address NU1902/NU1903 advisories for `MongoDB.Driver` 2.12.0 and `SharpCompress` 0.23.0 surfaced during scanner cache and worker test runs (Wave0A prerequisites cleared; remediation in progress).
1. [DONE] DEVOPS-SEC-10-301 — Address NU1902/NU1903 advisories for `MongoDB.Driver` 2.12.0 and `SharpCompress` 0.23.0 surfaced during scanner cache and worker test runs (2025-10-20) local Mongo2Go feed repacked to require MongoDB.Driver 3.5.0 and SharpCompress 0.41.0; targeted cache tests green.
• Prereqs: —
• Current: TODO
- **Sprint 10** · Scanner Analyzers & SBOM
@@ -1217,7 +1217,7 @@ Generated from SPRINTS.md and module TASKS.md files on 2025-10-19. Waves cluster
- **Sprint 7** · Contextual Truth Foundations
- Team: BE-Merge
- Path: `src/StellaOps.Concelier.Merge/TASKS.md`
1. [TODO] FEEDMERGE-ENGINE-07-001 — FEEDMERGE-ENGINE-07-001 Conflict sets & explainers
1. [DONE] FEEDMERGE-ENGINE-07-001 — Conflict sets & explainers (2025-10-20) Merge now returns conflict summaries with hashes and WebService exposes structured explainers.
• Prereqs: FEEDSTORAGE-DATA-07-001 (Wave 10)
• Current: TODO Persist conflict sets referencing advisory statements, output rule/explainer payloads with replay hashes, and add integration tests covering deterministic `asOf` evaluations.
@@ -1241,9 +1241,9 @@ Generated from SPRINTS.md and module TASKS.md files on 2025-10-19. Waves cluster
- **Sprint 8** · Mirror Distribution
- Team: Concelier WebService Guild
- Path: `src/StellaOps.Concelier.WebService/TASKS.md`
1. [DOING] CONCELIER-WEB-08-201 — CONCELIER-WEB-08-201 Mirror distribution endpoints
1. [DONE] CONCELIER-WEB-08-201 — Mirror distribution endpoints (2025-10-20) Service enforces Authority/bypass rules, issues cache headers, rate limits per domain, and ops docs list smoke tests.
• Prereqs: CONCELIER-EXPORT-08-201 (Wave 12), DEVOPS-MIRROR-08-001 (Wave 2)
• Current: DOING (2025-10-19) Wiring API surface against exporter-delivered `mirror/index.json` + signed bundles, layering quota/auth and updating docs/test fixtures for downstream sync.
• Current: DONE (2025-10-20) See `docs/ops/concelier-mirror-operations.md` for updated auth + rate-limit guidance; tests `WebServiceEndpointsTests` cover 401/Retry-After.
## Wave 15 — 1 task(s) ready after Wave 14
- **Sprint 8** · Mirror Distribution

View File

@@ -160,14 +160,14 @@ This file describe implementation of Stella Ops (docs/README.md). Implementation
| Sprint 7 | Contextual Truth Foundations | src/StellaOps.Concelier.Core/TASKS.md | TODO | Team Core Engine & Data Science | FEEDCORE-ENGINE-07-002 | Noise prior computation service learn false-positive priors and expose deterministic summaries. |
| Sprint 7 | Contextual Truth Foundations | src/StellaOps.Concelier.Core/TASKS.md | TODO | Team Core Engine & Storage Analytics | FEEDCORE-ENGINE-07-003 | Unknown state ledger & confidence seeding persist unknown flags, seed confidence bands, expose query surface. |
| Sprint 7 | Contextual Truth Foundations | src/StellaOps.Concelier.Storage.Mongo/TASKS.md | TODO | Team Normalization & Storage Backbone | FEEDSTORAGE-DATA-07-001 | Advisory statement & conflict collections provision Mongo schema/indexes for event-sourced merge. |
| Sprint 7 | Contextual Truth Foundations | src/StellaOps.Concelier.Merge/TASKS.md | DOING | BE-Merge | FEEDMERGE-ENGINE-07-001 | Conflict sets & explainers persist conflict materialization and replay hashes for merge decisions. |
| Sprint 7 | Contextual Truth Foundations | src/StellaOps.Concelier.Merge/TASKS.md | DONE (2025-10-20) | BE-Merge | FEEDMERGE-ENGINE-07-001 | Conflict sets & explainers persist conflict materialization and replay hashes for merge decisions. |
| Sprint 8 | Mongo strengthening | src/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-19) | Team Normalization & Storage Backbone | FEEDSTORAGE-MONGO-08-001 | Causal-consistent Concelier storage sessions<br>Scoped session facilitator registered, repositories accept optional session handles, and replica-set failover tests verify read-your-write + monotonic reads. |
| Sprint 8 | Mongo strengthening | src/StellaOps.Authority/TASKS.md | DONE (2025-10-19) | Authority Core & Storage Guild | AUTHSTORAGE-MONGO-08-001 | Harden Authority Mongo usage<br>Scoped Mongo sessions with majority read/write concerns wired through stores and GraphQL/HTTP pipelines; replica-set election regression validated. |
| Sprint 8 | Mongo strengthening | src/StellaOps.Excititor.Storage.Mongo/TASKS.md | DONE (2025-10-19) | Team Excititor Storage | EXCITITOR-STORAGE-MONGO-08-001 | Causal consistency for Excititor repositories<br>Session-scoped repositories shipped with new Mongo records, orchestrators/workers now share scoped sessions, and replica-set failover coverage added via `dotnet test src/StellaOps.Excititor.Storage.Mongo.Tests/StellaOps.Excititor.Storage.Mongo.Tests.csproj`. |
| Sprint 8 | Platform Maintenance | src/StellaOps.Excititor.Storage.Mongo/TASKS.md | DONE (2025-10-19) | Team Excititor Storage | EXCITITOR-STORAGE-03-001 | Statement backfill tooling shipped admin backfill endpoint, CLI hook (`stellaops excititor backfill-statements`), integration tests, and operator runbook (`docs/dev/EXCITITOR_STATEMENT_BACKFILL.md`). |
| Sprint 8 | Mirror Distribution | src/StellaOps.Concelier.Exporter.Json/TASKS.md | DONE (2025-10-19) | Concelier Export Guild | CONCELIER-EXPORT-08-201 | Mirror bundle + domain manifest produce signed JSON aggregates for `*.stella-ops.org` mirrors. |
| Sprint 8 | Mirror Distribution | src/StellaOps.Concelier.Exporter.TrivyDb/TASKS.md | DONE (2025-10-19) | Concelier Export Guild | CONCELIER-EXPORT-08-202 | Mirror-ready Trivy DB bundles mirror options emit per-domain manifests/metadata/db archives with deterministic digests for downstream sync. |
| Sprint 8 | Mirror Distribution | src/StellaOps.Concelier.WebService/TASKS.md | DOING (2025-10-19) | Concelier WebService Guild | CONCELIER-WEB-08-201 | Mirror distribution endpoints expose domain-scoped index/download APIs with auth/quota. |
| Sprint 8 | Mirror Distribution | src/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-20) | Concelier WebService Guild | CONCELIER-WEB-08-201 | Mirror distribution endpoints expose domain-scoped index/download APIs with auth/quota. |
| Sprint 8 | Mirror Distribution | src/StellaOps.Concelier.Connector.StellaOpsMirror/TASKS.md | DOING (2025-10-19) | BE-Conn-Stella | FEEDCONN-STELLA-08-001 | Concelier mirror connector fetch mirror manifest, verify signatures, and hydrate canonical DTOs with resume support. |
| Sprint 8 | Mirror Distribution | ops/devops/TASKS.md | DONE (2025-10-19) | DevOps Guild | DEVOPS-MIRROR-08-001 | Managed mirror deployments for `*.stella-ops.org` Helm/Compose overlays, CDN, runbooks. |
| Sprint 8 | Plugin Infrastructure | src/StellaOps.Plugin/TASKS.md | DOING | Plugin Platform Guild, Authority Core | PLUGIN-DI-08-002.COORD | Authority scoped-service integration handshake<br>Session scheduled for 2025-10-20 15:0016:00UTC; agenda + attendees logged in `docs/dev/authority-plugin-di-coordination.md`. |
@@ -246,7 +246,7 @@ This file describe implementation of Stella Ops (docs/README.md). Implementation
| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Emit/TASKS.md | TODO | Emit Guild | SCANNER-EMIT-10-607 | Embed scoring inputs, confidence band, and quiet provenance in CycloneDX/DSSE artifacts. |
| Sprint 10 | Benchmarks | bench/TASKS.md | TODO | Bench Guild, Scanner Team | BENCH-SCANNER-10-001 | Analyzer microbench harness + baseline CSV. |
| Sprint 10 | Samples | samples/TASKS.md | TODO | Samples Guild, Scanner Team | SAMPLES-10-001 | Sample images with SBOM/BOM-Index sidecars. |
| Sprint 10 | DevOps Security | ops/devops/TASKS.md | DOING | DevOps Guild | DEVOPS-SEC-10-301 | Address NU1902/NU1903 advisories for `MongoDB.Driver` 2.12.0 and `SharpCompress` 0.23.0; Wave0A prerequisites confirmed complete before remediation work. |
| Sprint 10 | DevOps Security | ops/devops/TASKS.md | DONE (2025-10-20) | DevOps Guild | DEVOPS-SEC-10-301 | Address NU1902/NU1903 advisories for `MongoDB.Driver` 2.12.0 and `SharpCompress` 0.23.0; Wave0A prerequisites confirmed complete before remediation work. |
| Sprint 10 | DevOps Perf | ops/devops/TASKS.md | TODO | DevOps Guild | DEVOPS-PERF-10-001 | Perf smoke job ensuring <5s SBOM compose. |
| Sprint 11 | Signing Chain Bring-up | src/StellaOps.Authority/TASKS.md | DOING (2025-10-19) | Authority Core & Security Guild | AUTH-DPOP-11-001 | Implement DPoP proof validation + nonce handling for high-value audiences per architecture. |
| Sprint 11 | Signing Chain Bring-up | src/StellaOps.Authority/TASKS.md | DOING (2025-10-19) | Authority Core & Security Guild | AUTH-MTLS-11-002 | Add OAuth mTLS client credential support with certificate-bound tokens and introspection updates. |

View File

@@ -120,6 +120,7 @@ details // structured conflict explanation / merge reasoning
```
- `AdvisoryEventLog` (Concelier.Core) provides the public API for appending immutable statements/conflicts and querying replay history. Inputs are normalized by trimming and lower-casing `vulnerabilityKey`, serializing advisories with `CanonicalJsonSerializer`, and computing SHA-256 hashes (`statementHash`, `conflictHash`) over the canonical JSON payloads. Consumers can replay by key with an optional `asOf` filter to obtain deterministic snapshots ordered by `asOf` then `recordedAt`.
- Conflict explainers are serialized as deterministic `MergeConflictExplainerPayload` records (type, reason, source ranks, winning values); replay clients can parse the payload to render human-readable rationales without re-computing precedence.
- Concelier.WebService exposes the immutable log via `GET /concelier/advisories/{vulnerabilityKey}/replay[?asOf=UTC_ISO8601]`, returning the latest statements (with hex-encoded hashes) and any conflict explanations for downstream exporters and APIs.
**ExportState**
@@ -281,6 +282,7 @@ public interface IFeedConnector {
* Optional ORAS push (OCI layout) for registries.
* Offline kit bundles include Trivy DB + JSON tree + export manifest.
* Mirror-ready bundles: when `concelier.trivy.mirror` defines domains, the exporter emits `mirror/index.json` plus per-domain `manifest.json`, `metadata.json`, and `db.tar.gz` files with SHA-256 digests so Concelier mirrors can expose domain-scoped download endpoints.
* Concelier.WebService serves `/concelier/exports/index.json` and `/concelier/exports/mirror/{domain}/…` directly from the export tree with hour-long budgets (index: 60s, bundles: 300s, immutable) and per-domain rate limiting; the endpoints honour Stella Ops Authority or CIDR bypass lists depending on mirror topology.
### 7.3 Handoff to Signer/Attestor (optional)

View File

@@ -337,7 +337,7 @@ Prometheus + OTLP; Grafana dashboards ship in the charts.
* **Vulnerability response**:
* Concelier red-flag advisories trigger accelerated **stable** patch rollout; UI/CLI “security patch available” notice.
* 2025-10: Pinned `MongoDB.Driver` **3.5.0** and `SharpCompress` **0.41.0** across services (DEVOPS-SEC-10-301) to eliminate NU1902/NU1903 warnings surfaced during scanner cache/worker test runs; future dependency bumps follow the same central override pattern.
* 2025-10: Pinned `MongoDB.Driver` **3.5.0** and `SharpCompress` **0.41.0** across services (DEVOPS-SEC-10-301) to eliminate NU1902/NU1903 warnings surfaced during scanner cache/worker test runs; repacked the local `Mongo2Go` feed so test fixtures inherit the patched dependencies; future bumps follow the same central override pattern.
* **Backups/DR**:

View File

@@ -18,6 +18,21 @@ authn, CDN fronting, and the recurring sync pipeline that keeps mirror bundles c
For Helm, provision PVCs (`concelier-mirror-jobs`, `concelier-mirror-exports`,
`excititor-mirror-exports`, `mirror-mongo-data`, `mirror-minio-data`) before rollout.
### 1.1 Service configuration quick reference
Concelier.WebService exposes the mirror HTTP endpoints once `CONCELIER__MIRROR__ENABLED=true`.
Key knobs:
- `CONCELIER__MIRROR__EXPORTROOT` root folder containing export snapshots (`<exportId>/mirror/*`).
- `CONCELIER__MIRROR__ACTIVEEXPORTID` optional explicit export id; otherwise the service auto-falls back to the `latest/` symlink or newest directory.
- `CONCELIER__MIRROR__REQUIREAUTHENTICATION` default auth requirement; override per domain with `CONCELIER__MIRROR__DOMAINS__{n}__REQUIREAUTHENTICATION`.
- `CONCELIER__MIRROR__MAXINDEXREQUESTSPERHOUR` budget for `/concelier/exports/index.json`. Domains inherit this value unless they define `__MAXDOWNLOADREQUESTSPERHOUR`.
- `CONCELIER__MIRROR__DOMAINS__{n}__ID` domain identifier matching the exporter manifest; additional keys configure display name and rate budgets.
> The service honours Stella Ops Authority when `CONCELIER__AUTHORITY__ENABLED=true` and `ALLOWANONYMOUSFALLBACK=false`. Use the bypass CIDR list (`CONCELIER__AUTHORITY__BYPASSNETWORKS__*`) for in-cluster ingress gateways that terminate Basic Auth. Unauthorized requests emit `WWW-Authenticate: Bearer` so downstream automation can detect token failures.
Mirror responses carry deterministic cache headers: `/index.json` returns `Cache-Control: public, max-age=60`, while per-domain manifests/bundles include `Cache-Control: public, max-age=300, immutable`. Rate limiting surfaces `Retry-After` when quotas are exceeded.
## 2. Secret & certificate layout
### Docker Compose (`deploy/compose/docker-compose.mirror.yaml`)
@@ -154,14 +169,16 @@ spec:
## 6. Smoke tests
After each deployment or sync cycle:
After each deployment or sync cycle (temporarily set low budgets if you need to observe 429 responses):
```bash
# Index with Basic Auth
curl -u $PRIMARY_CREDS https://mirror-primary.stella-ops.org/concelier/exports/index.json | jq 'keys'
# Mirror manifest signature
curl -u $PRIMARY_CREDS -I https://mirror-primary.stella-ops.org/concelier/exports/mirror/primary/manifest.json
# Mirror manifest signature and cache headers
curl -u $PRIMARY_CREDS -I https://mirror-primary.stella-ops.org/concelier/exports/mirror/primary/manifest.json \
| tee /tmp/manifest-headers.txt
grep -E '^Cache-Control: ' /tmp/manifest-headers.txt # expect public, max-age=300, immutable
# Excititor consensus bundle metadata
curl -u $COMMUNITY_CREDS https://mirror-community.stella-ops.org/excititor/mirror/community/index \
@@ -171,6 +188,17 @@ curl -u $COMMUNITY_CREDS https://mirror-community.stella-ops.org/excititor/mirro
curl -u $PRIMARY_CREDS https://mirror-primary.stella-ops.org/concelier/exports/mirror/primary/bundle.json.jws \
-o bundle.json.jws
cosign verify-blob --signature bundle.json.jws --key mirror-key.pub bundle.json
# Service-level auth check (inside cluster no gateway credentials)
kubectl exec deploy/stellaops-concelier -- curl -si http://localhost:8443/concelier/exports/mirror/primary/manifest.json \
| head -n 5 # expect HTTP/1.1 401 with WWW-Authenticate: Bearer
# Rate limit smoke (repeat quickly; second call should return 429 + Retry-After)
for i in 1 2; do
curl -s -o /dev/null -D - https://mirror-primary.stella-ops.org/concelier/exports/index.json \
-u $PRIMARY_CREDS | grep -E '^(HTTP/|Retry-After:)'
sleep 1
done
```
Watch the gateway metrics (`nginx_vts` or access logs) for cache hits. In Kubernetes, `kubectl logs deploy/stellaops-mirror-gateway`

Binary file not shown.

View File

@@ -10,4 +10,5 @@
| DEVOPS-REL-14-001 | TODO | DevOps Guild | SIGNER-API-11-101, ATTESTOR-API-11-201 | Deterministic build/release pipeline with SBOM/provenance, signing, manifest generation. | CI pipeline produces signed images + SBOM/attestations, manifests published with verified hashes, docs updated. |
| DEVOPS-REL-17-002 | TODO | DevOps Guild | DEVOPS-REL-14-001, SCANNER-EMIT-17-701 | Persist stripped-debug artifacts organised by GNU build-id and bundle them into release/offline kits with checksum manifests. | CI job writes `.debug` files under `artifacts/debug/.build-id/`, manifest + checksums published, offline kit includes cache, smoke job proves symbol lookup via build-id. |
| DEVOPS-MIRROR-08-001 | DONE (2025-10-19) | DevOps Guild | DEVOPS-REL-14-001 | Stand up managed mirror profiles for `*.stella-ops.org` (Concelier/Excititor), including Helm/Compose overlays, multi-tenant secrets, CDN caching, and sync documentation. | Infra overlays committed, CI smoke deploy hits mirror endpoints, runbooks published for downstream sync and quota management. |
| DEVOPS-SEC-10-301 | DOING (2025-10-19) | DevOps Guild | Wave 0A complete | Address NU1902/NU1903 advisories for `MongoDB.Driver` 2.12.0 and `SharpCompress` 0.23.0 surfaced during scanner cache and worker test runs. | Dependencies bumped to patched releases, audit logs free of NU1902/NU1903 warnings, regression tests green, change log documents upgrade guidance. |
| DEVOPS-SEC-10-301 | DONE (2025-10-20) | DevOps Guild | Wave 0A complete | Address NU1902/NU1903 advisories for `MongoDB.Driver` 2.12.0 and `SharpCompress` 0.23.0 surfaced during scanner cache and worker test runs. | Dependencies bumped to patched releases, audit logs free of NU1902/NU1903 warnings, regression tests green, change log documents upgrade guidance. |
> Remark (2025-10-20): Repacked `Mongo2Go` local feed to require MongoDB.Driver 3.5.0 + SharpCompress 0.41.0; cache regression tests green and NU1902/NU1903 suppressed.

View File

@@ -20,7 +20,8 @@ public sealed class MirrorSignatureVerifierTests
var registry = new CryptoProviderRegistry(new[] { provider });
var verifier = new MirrorSignatureVerifier(registry, NullLogger<MirrorSignatureVerifier>.Instance);
var payload = "{\"advisories\":[]}\"u8".ToUtf8Bytes();
var payloadText = System.Text.Json.JsonSerializer.Serialize(new { advisories = Array.Empty<string>() });
var payload = payloadText.ToUtf8Bytes();
var (signature, _) = await CreateDetachedJwsAsync(provider, key.Reference.KeyId, payload);
await verifier.VerifyAsync(payload, signature, CancellationToken.None);
@@ -36,14 +37,61 @@ public sealed class MirrorSignatureVerifierTests
var registry = new CryptoProviderRegistry(new[] { provider });
var verifier = new MirrorSignatureVerifier(registry, NullLogger<MirrorSignatureVerifier>.Instance);
var payload = "{\"advisories\":[]}\"u8".ToUtf8Bytes();
var payloadText = System.Text.Json.JsonSerializer.Serialize(new { advisories = Array.Empty<string>() });
var payload = payloadText.ToUtf8Bytes();
var (signature, _) = await CreateDetachedJwsAsync(provider, key.Reference.KeyId, payload);
var tampered = signature.Replace("a", "b", StringComparison.Ordinal);
var tampered = signature.Replace('a', 'b', StringComparison.Ordinal);
await Assert.ThrowsAsync<InvalidOperationException>(() => verifier.VerifyAsync(payload, tampered, CancellationToken.None));
}
[Fact]
public async Task VerifyAsync_KeyMismatchThrows()
{
var provider = new DefaultCryptoProvider();
var key = CreateSigningKey("mirror-key");
provider.UpsertSigningKey(key);
var registry = new CryptoProviderRegistry(new[] { provider });
var verifier = new MirrorSignatureVerifier(registry, NullLogger<MirrorSignatureVerifier>.Instance);
var payloadText = System.Text.Json.JsonSerializer.Serialize(new { advisories = Array.Empty<string>() });
var payload = payloadText.ToUtf8Bytes();
var (signature, _) = await CreateDetachedJwsAsync(provider, key.Reference.KeyId, payload);
await Assert.ThrowsAsync<InvalidOperationException>(() => verifier.VerifyAsync(
payload,
signature,
expectedKeyId: "unexpected-key",
expectedProvider: null,
cancellationToken: CancellationToken.None));
}
[Fact]
public async Task VerifyAsync_ThrowsWhenProviderMissingKey()
{
var provider = new DefaultCryptoProvider();
var key = CreateSigningKey("mirror-key");
provider.UpsertSigningKey(key);
var registry = new CryptoProviderRegistry(new[] { provider });
var verifier = new MirrorSignatureVerifier(registry, NullLogger<MirrorSignatureVerifier>.Instance);
var payloadText = System.Text.Json.JsonSerializer.Serialize(new { advisories = Array.Empty<string>() });
var payload = payloadText.ToUtf8Bytes();
var (signature, _) = await CreateDetachedJwsAsync(provider, key.Reference.KeyId, payload);
provider.RemoveSigningKey(key.Reference.KeyId);
await Assert.ThrowsAsync<InvalidOperationException>(() => verifier.VerifyAsync(
payload,
signature,
expectedKeyId: key.Reference.KeyId,
expectedProvider: provider.Name,
cancellationToken: CancellationToken.None));
}
private static CryptoSigningKey CreateSigningKey(string keyId)
{
using var ecdsa = ECDsa.Create(ECCurve.NamedCurves.nistP256);

View File

@@ -13,11 +13,11 @@ using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
using MongoDB.Bson;
using StellaOps.Concelier.Connector.Common;
using StellaOps.Concelier.Connector.Common.Fetch;
using StellaOps.Concelier.Connector.Common.Testing;
using StellaOps.Concelier.Connector.StellaOpsMirror.Settings;
using StellaOps.Concelier.Storage.Mongo;
using StellaOps.Concelier.Storage.Mongo.Documents;
using StellaOps.Concelier.Storage.Mongo.SourceState;
using StellaOps.Concelier.Testing;
using StellaOps.Cryptography;
using Xunit;
@@ -135,6 +135,39 @@ public sealed class StellaOpsMirrorConnectorTests : IAsyncLifetime
Assert.False(state.Cursor.TryGetValue("bundleDigest", out _));
}
[Fact]
public async Task FetchAsync_SignatureKeyMismatchThrows()
{
var manifestContent = "{\"domain\":\"primary\"}";
var bundleContent = "{\"advisories\":[{\"id\":\"CVE-2025-0003\"}]}";
var manifestDigest = ComputeDigest(manifestContent);
var bundleDigest = ComputeDigest(bundleContent);
var index = BuildIndex(
manifestDigest,
Encoding.UTF8.GetByteCount(manifestContent),
bundleDigest,
Encoding.UTF8.GetByteCount(bundleContent),
includeSignature: true,
signatureKeyId: "unexpected-key",
signatureProvider: "default");
var signingKey = CreateSigningKey("unexpected-key");
var (signatureValue, _) = CreateDetachedJws(signingKey, bundleContent);
await using var provider = await BuildServiceProviderAsync(options =>
{
options.Signature.Enabled = true;
options.Signature.KeyId = "mirror-key";
options.Signature.Provider = "default";
});
SeedResponses(index, manifestContent, bundleContent, signatureValue);
var connector = provider.GetRequiredService<StellaOpsMirrorConnector>();
await Assert.ThrowsAsync<InvalidOperationException>(() => connector.FetchAsync(provider, CancellationToken.None));
}
public Task InitializeAsync() => Task.CompletedTask;
public Task DisposeAsync()
@@ -217,7 +250,14 @@ public sealed class StellaOpsMirrorConnectorTests : IAsyncLifetime
Content = new StringContent(content, Encoding.UTF8, "application/json"),
};
private static string BuildIndex(string manifestDigest, int manifestBytes, string bundleDigest, int bundleBytes, bool includeSignature)
private static string BuildIndex(
string manifestDigest,
int manifestBytes,
string bundleDigest,
int bundleBytes,
bool includeSignature,
string signatureKeyId = "mirror-key",
string signatureProvider = "default")
{
var index = new
{
@@ -248,8 +288,8 @@ public sealed class StellaOpsMirrorConnectorTests : IAsyncLifetime
{
path = "mirror/primary/bundle.json.jws",
algorithm = "ES256",
keyId = "mirror-key",
provider = "default",
keyId = signatureKeyId,
provider = signatureProvider,
signedAt = new DateTimeOffset(2025, 10, 19, 12, 0, 0, TimeSpan.Zero),
}
: null,
@@ -285,7 +325,7 @@ public sealed class StellaOpsMirrorConnectorTests : IAsyncLifetime
private static (string Signature, DateTimeOffset SignedAt) CreateDetachedJws(CryptoSigningKey signingKey, string payload)
{
using var provider = new DefaultCryptoProvider();
var provider = new DefaultCryptoProvider();
provider.UpsertSigningKey(signingKey);
var signer = provider.GetSigner(SignatureAlgorithms.Es256, signingKey.Reference);
var header = new Dictionary<string, object?>

View File

@@ -27,7 +27,15 @@ public sealed class MirrorSignatureVerifier
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
}
public async Task VerifyAsync(ReadOnlyMemory<byte> payload, string signatureValue, CancellationToken cancellationToken)
public Task VerifyAsync(ReadOnlyMemory<byte> payload, string signatureValue, CancellationToken cancellationToken)
=> VerifyAsync(payload, signatureValue, expectedKeyId: null, expectedProvider: null, cancellationToken);
public async Task VerifyAsync(
ReadOnlyMemory<byte> payload,
string signatureValue,
string? expectedKeyId,
string? expectedProvider,
CancellationToken cancellationToken)
{
if (payload.IsEmpty)
{
@@ -68,15 +76,36 @@ public sealed class MirrorSignatureVerifier
throw new InvalidOperationException("Detached JWS header missing algorithm identifier.");
}
if (!string.IsNullOrWhiteSpace(expectedKeyId) &&
!string.Equals(header.KeyId, expectedKeyId, StringComparison.OrdinalIgnoreCase))
{
throw new InvalidOperationException($"Mirror bundle signature key '{header.KeyId}' did not match expected key '{expectedKeyId}'.");
}
if (!string.IsNullOrWhiteSpace(expectedProvider) &&
!string.Equals(header.Provider, expectedProvider, StringComparison.OrdinalIgnoreCase))
{
throw new InvalidOperationException($"Mirror bundle signature provider '{header.Provider ?? "<null>"}' did not match expected provider '{expectedProvider}'.");
}
var signingInput = BuildSigningInput(encodedHeader, payload.Span);
var signatureBytes = Base64UrlEncoder.DecodeBytes(encodedSignature);
var keyReference = new CryptoKeyReference(header.KeyId, header.Provider);
var resolution = _providerRegistry.ResolveSigner(
CryptoCapability.Verification,
header.Algorithm,
keyReference,
header.Provider);
CryptoSignerResolution resolution;
try
{
resolution = _providerRegistry.ResolveSigner(
CryptoCapability.Verification,
header.Algorithm,
keyReference,
header.Provider);
}
catch (Exception ex) when (ex is InvalidOperationException or KeyNotFoundException)
{
_logger.LogWarning(ex, "Unable to resolve signer for mirror signature key {KeyId} via provider {Provider}.", header.KeyId, header.Provider ?? "<null>");
throw new InvalidOperationException("Detached JWS signature verification failed.", ex);
}
var verified = await resolution.Signer.VerifyAsync(signingInput, signatureBytes, cancellationToken).ConfigureAwait(false);
if (!verified)

View File

@@ -133,9 +133,30 @@ public sealed class StellaOpsMirrorConnector : IFeedConnector
throw new InvalidOperationException("Mirror bundle did not include a signature descriptor while verification is enabled.");
}
if (!string.IsNullOrWhiteSpace(_options.Signature.KeyId) &&
!string.Equals(domain.Bundle.Signature.KeyId, _options.Signature.KeyId, StringComparison.OrdinalIgnoreCase))
{
throw new InvalidOperationException($"Mirror bundle signature key '{domain.Bundle.Signature.KeyId}' did not match expected key '{_options.Signature.KeyId}'.");
}
if (!string.IsNullOrWhiteSpace(_options.Signature.Provider) &&
!string.Equals(domain.Bundle.Signature.Provider, _options.Signature.Provider, StringComparison.OrdinalIgnoreCase))
{
throw new InvalidOperationException($"Mirror bundle signature provider '{domain.Bundle.Signature.Provider ?? "<null>"}' did not match expected provider '{_options.Signature.Provider}'.");
}
var signatureBytes = await _client.DownloadAsync(domain.Bundle.Signature.Path, cancellationToken).ConfigureAwait(false);
var signatureValue = Encoding.UTF8.GetString(signatureBytes);
await _signatureVerifier.VerifyAsync(bundleBytes, signatureValue, cancellationToken).ConfigureAwait(false);
var signatureValue = Encoding.UTF8.GetString(signatureBytes).Trim();
await _signatureVerifier.VerifyAsync(
bundleBytes,
signatureValue,
expectedKeyId: _options.Signature.KeyId,
expectedProvider: _options.Signature.Provider,
cancellationToken).ConfigureAwait(false);
}
else if (domain.Bundle.Signature is not null)
{
_logger.LogInformation("Mirror bundle provided signature descriptor but verification is disabled; skipping verification.");
}
await StoreAsync(domain, index.GeneratedAt, domain.Manifest, manifestBytes, "application/json", DocumentStatuses.Mapped, addToPending: false, pendingDocuments, cancellationToken).ConfigureAwait(false);

View File

@@ -1,4 +1,5 @@
using System.Collections.Concurrent;
using System.Collections.Immutable;
using System.Linq;
using System.Threading.Tasks;
using Microsoft.Extensions.Logging.Abstractions;
@@ -45,6 +46,7 @@ public sealed class AdvisoryMergeServiceTests
Assert.NotNull(result.Merged);
Assert.Equal("OSV summary overrides", result.Merged!.Summary);
Assert.Empty(result.Conflicts);
var upserted = advisoryStore.LastUpserted;
Assert.NotNull(upserted);
@@ -123,6 +125,89 @@ public sealed class AdvisoryMergeServiceTests
provenance: new[] { provenance });
}
private static Advisory CreateVendorAdvisory()
{
var recorded = DateTimeOffset.Parse("2025-03-10T00:00:00Z");
var provenance = new AdvisoryProvenance("vendor", "psirt", "VSA-2025-5000", recorded, new[] { ProvenanceFieldMasks.Advisory });
return new Advisory(
"VSA-2025-5000",
"Vendor overrides severity",
"Vendor states critical impact.",
"en",
recorded,
recorded,
"critical",
exploitKnown: false,
aliases: new[] { "VSA-2025-5000", "CVE-2025-5000" },
references: Array.Empty<AdvisoryReference>(),
affectedPackages: Array.Empty<AffectedPackage>(),
cvssMetrics: Array.Empty<CvssMetric>(),
provenance: new[] { provenance });
}
private static Advisory CreateConflictingNvdAdvisory()
{
var recorded = DateTimeOffset.Parse("2025-03-09T00:00:00Z");
var provenance = new AdvisoryProvenance("nvd", "map", "CVE-2025-5000", recorded, new[] { ProvenanceFieldMasks.Advisory });
return new Advisory(
"CVE-2025-5000",
"CVE-2025-5000",
"Baseline NVD entry.",
"en",
recorded,
recorded,
"medium",
exploitKnown: false,
aliases: new[] { "CVE-2025-5000" },
references: Array.Empty<AdvisoryReference>(),
affectedPackages: Array.Empty<AffectedPackage>(),
cvssMetrics: Array.Empty<CvssMetric>(),
provenance: new[] { provenance });
}
[Fact]
public async Task MergeAsync_PersistsConflictSummariesWithHashes()
{
var aliasStore = new FakeAliasStore();
aliasStore.Register("CVE-2025-5000",
(AliasSchemes.Cve, "CVE-2025-5000"));
aliasStore.Register("VSA-2025-5000",
(AliasSchemes.Cve, "CVE-2025-5000"));
var vendor = CreateVendorAdvisory();
var nvd = CreateConflictingNvdAdvisory();
var advisoryStore = new FakeAdvisoryStore();
advisoryStore.Seed(vendor, nvd);
var mergeEventStore = new InMemoryMergeEventStore();
var timeProvider = new FakeTimeProvider(new DateTimeOffset(2025, 4, 2, 0, 0, 0, TimeSpan.Zero));
var writer = new MergeEventWriter(mergeEventStore, new CanonicalHashCalculator(), timeProvider, NullLogger<MergeEventWriter>.Instance);
var precedenceMerger = new AdvisoryPrecedenceMerger(new AffectedPackagePrecedenceResolver(), timeProvider);
var aliasResolver = new AliasGraphResolver(aliasStore);
var canonicalMerger = new CanonicalMerger(timeProvider);
var eventLog = new RecordingAdvisoryEventLog();
var service = new AdvisoryMergeService(aliasResolver, advisoryStore, precedenceMerger, writer, canonicalMerger, eventLog, timeProvider, NullLogger<AdvisoryMergeService>.Instance);
var result = await service.MergeAsync("CVE-2025-5000", CancellationToken.None);
var conflict = Assert.Single(result.Conflicts);
Assert.Equal("CVE-2025-5000", conflict.VulnerabilityKey);
Assert.Equal("severity", conflict.Explainer.Type);
Assert.Equal("mismatch", conflict.Explainer.Reason);
Assert.Contains("vendor", conflict.Explainer.PrimarySources, StringComparer.OrdinalIgnoreCase);
Assert.Contains("nvd", conflict.Explainer.SuppressedSources, StringComparer.OrdinalIgnoreCase);
Assert.Equal(conflict.Explainer.ComputeHashHex(), conflict.ConflictHash);
Assert.True(conflict.StatementIds.Length >= 2);
Assert.Equal(timeProvider.GetUtcNow(), conflict.RecordedAt);
var appendRequest = eventLog.LastRequest;
Assert.NotNull(appendRequest);
var appendedConflict = Assert.Single(appendRequest!.Conflicts!);
Assert.Equal(conflict.ConflictId, appendedConflict.ConflictId);
Assert.Equal(conflict.StatementIds, appendedConflict.StatementIds.ToImmutableArray());
}
private sealed class RecordingAdvisoryEventLog : IAdvisoryEventLog
{

View File

@@ -133,12 +133,12 @@ public sealed class AdvisoryMergeService
ConvertFieldDecisions(canonicalMerge?.Decisions),
cancellationToken).ConfigureAwait(false);
await AppendEventLogAsync(canonicalKey, normalizedInputs, merged, conflictDetails, cancellationToken).ConfigureAwait(false);
var conflictSummaries = await AppendEventLogAsync(canonicalKey, normalizedInputs, merged, conflictDetails, cancellationToken).ConfigureAwait(false);
return new AdvisoryMergeResult(seedAdvisoryKey, canonicalKey, component, inputs, before, merged);
return new AdvisoryMergeResult(seedAdvisoryKey, canonicalKey, component, inputs, before, merged, conflictSummaries);
}
private async Task AppendEventLogAsync(
private async Task<IReadOnlyList<MergeConflictSummary>> AppendEventLogAsync(
string vulnerabilityKey,
IReadOnlyList<Advisory> inputs,
Advisory merged,
@@ -172,11 +172,15 @@ public sealed class AdvisoryMergeService
StatementId: canonicalStatementId,
AdvisoryKey: merged.AdvisoryKey));
var conflictInputs = BuildConflictInputs(conflicts, vulnerabilityKey, statementIds, canonicalStatementId, recordedAt);
var conflictMaterialization = BuildConflictInputs(conflicts, vulnerabilityKey, statementIds, canonicalStatementId, recordedAt);
var conflictInputs = conflictMaterialization.Inputs;
var conflictSummaries = conflictMaterialization.Summaries;
if (statements.Count == 0 && conflictInputs.Count == 0)
{
return;
return conflictSummaries.Count == 0
? Array.Empty<MergeConflictSummary>()
: conflictSummaries.ToArray();
}
var request = new AdvisoryEventAppendRequest(statements, conflictInputs.Count > 0 ? conflictInputs : null);
@@ -192,6 +196,10 @@ public sealed class AdvisoryMergeService
conflict.Details.Dispose();
}
}
return conflictSummaries.Count == 0
? Array.Empty<MergeConflictSummary>()
: conflictSummaries.ToArray();
}
private static DateTimeOffset DetermineAsOf(Advisory advisory, DateTimeOffset fallback)
@@ -199,7 +207,7 @@ public sealed class AdvisoryMergeService
return (advisory.Modified ?? advisory.Published ?? fallback).ToUniversalTime();
}
private static List<AdvisoryConflictInput> BuildConflictInputs(
private static ConflictMaterialization BuildConflictInputs(
IReadOnlyList<MergeConflictDetail> conflicts,
string vulnerabilityKey,
IReadOnlyDictionary<Advisory, Guid> statementIds,
@@ -208,10 +216,11 @@ public sealed class AdvisoryMergeService
{
if (conflicts.Count == 0)
{
return new List<AdvisoryConflictInput>(0);
return new ConflictMaterialization(new List<AdvisoryConflictInput>(0), new List<MergeConflictSummary>(0));
}
var inputs = new List<AdvisoryConflictInput>(conflicts.Count);
var summaries = new List<MergeConflictSummary>(conflicts.Count);
foreach (var detail in conflicts)
{
@@ -239,31 +248,43 @@ public sealed class AdvisoryMergeService
detail.PrimaryValue,
detail.SuppressedValue);
var json = CanonicalJsonSerializer.Serialize(payload);
var document = JsonDocument.Parse(json);
var explainer = new MergeConflictExplainerPayload(
payload.Type,
payload.Reason,
payload.PrimarySources,
payload.PrimaryRank,
payload.SuppressedSources,
payload.SuppressedRank,
payload.PrimaryValue,
payload.SuppressedValue);
var canonicalJson = explainer.ToCanonicalJson();
var document = JsonDocument.Parse(canonicalJson);
var asOf = (detail.Primary.Modified ?? detail.Suppressed.Modified ?? recordedAt).ToUniversalTime();
var conflictId = Guid.NewGuid();
var statementIdArray = ImmutableArray.CreateRange(related);
var conflictHash = explainer.ComputeHashHex(canonicalJson);
inputs.Add(new AdvisoryConflictInput(
vulnerabilityKey,
document,
asOf,
related,
ConflictId: null));
ConflictId: conflictId));
summaries.Add(new MergeConflictSummary(
conflictId,
vulnerabilityKey,
statementIdArray,
conflictHash,
asOf,
recordedAt,
explainer));
}
return inputs;
return new ConflictMaterialization(inputs, summaries);
}
private sealed record ConflictDetailPayload(
string Type,
string Reason,
IReadOnlyList<string> PrimarySources,
int PrimaryRank,
IReadOnlyList<string> SuppressedSources,
int SuppressedRank,
string? PrimaryValue,
string? SuppressedValue);
private static IEnumerable<Advisory> NormalizeInputs(IEnumerable<Advisory> advisories, string canonicalKey)
{
foreach (var advisory in advisories)
@@ -385,6 +406,10 @@ public sealed class AdvisoryMergeService
public const string Osv = "osv";
}
private sealed record ConflictMaterialization(
List<AdvisoryConflictInput> Inputs,
List<MergeConflictSummary> Summaries);
private static string? SelectCanonicalKey(AliasComponent component)
{
foreach (var scheme in PreferredAliasSchemes)
@@ -423,8 +448,9 @@ public sealed record AdvisoryMergeResult(
AliasComponent Component,
IReadOnlyList<Advisory> Inputs,
Advisory? Previous,
Advisory? Merged)
Advisory? Merged,
IReadOnlyList<MergeConflictSummary> Conflicts)
{
public static AdvisoryMergeResult Empty(string seed, AliasComponent component)
=> new(seed, seed, component, Array.Empty<Advisory>(), null, null);
=> new(seed, seed, component, Array.Empty<Advisory>(), null, null, Array.Empty<MergeConflictSummary>());
}

View File

@@ -0,0 +1,34 @@
using System;
using System.Collections.Generic;
using System.Security.Cryptography;
using System.Text;
using StellaOps.Concelier.Models;
namespace StellaOps.Concelier.Merge.Services;
/// <summary>
/// Structured payload describing a precedence conflict between advisory sources.
/// </summary>
public sealed record MergeConflictExplainerPayload(
string Type,
string Reason,
IReadOnlyList<string> PrimarySources,
int PrimaryRank,
IReadOnlyList<string> SuppressedSources,
int SuppressedRank,
string? PrimaryValue,
string? SuppressedValue)
{
public string ToCanonicalJson() => CanonicalJsonSerializer.Serialize(this);
public string ComputeHashHex(string? canonicalJson = null)
{
var json = canonicalJson ?? ToCanonicalJson();
var bytes = Encoding.UTF8.GetBytes(json);
var hash = SHA256.HashData(bytes);
return Convert.ToHexString(hash);
}
public static MergeConflictExplainerPayload FromCanonicalJson(string canonicalJson)
=> CanonicalJsonSerializer.Deserialize<MergeConflictExplainerPayload>(canonicalJson);
}

View File

@@ -0,0 +1,16 @@
using System;
using System.Collections.Immutable;
namespace StellaOps.Concelier.Merge.Services;
/// <summary>
/// Summary of a persisted advisory conflict including hashes and structured explainer payload.
/// </summary>
public sealed record MergeConflictSummary(
Guid ConflictId,
string VulnerabilityKey,
ImmutableArray<Guid> StatementIds,
string ConflictHash,
DateTimeOffset AsOf,
DateTimeOffset RecordedAt,
MergeConflictExplainerPayload Explainer);

View File

@@ -18,4 +18,5 @@
|Range primitives backlog|BE-Merge|Connector WGs|**DOING** Coordinate remaining connectors (`Acsc`, `Cccs`, `CertBund`, `CertCc`, `Cve`, `Ghsa`, `Ics.Cisa`, `Kisa`, `Ru.Bdu`, `Ru.Nkcki`, `Vndr.Apple`, `Vndr.Cisco`, `Vndr.Msrc`) to emit canonical RangePrimitives with provenance tags; track progress/fixtures here.<br>2025-10-11: Storage alignment notes + sample normalized rule JSON now captured in `RANGE_PRIMITIVES_COORDINATION.md` (see “Storage alignment quick reference”).<br>2025-10-11 18:45Z: GHSA normalized rules landed; OSV connector picked up next for rollout.<br>2025-10-11 21:10Z: `docs/dev/merge_semver_playbook.md` Section 8 now documents the persisted Mongo projection (SemVer + NEVRA) for connector reviewers.<br>2025-10-11 21:30Z: Added `docs/dev/normalized_versions_rollout.md` dashboard to centralize connector status and upcoming milestones.<br>2025-10-11 21:55Z: Merge now emits `concelier.merge.normalized_rules*` counters and unions connector-provided normalized arrays; see new test coverage in `AdvisoryPrecedenceMergerTests.Merge_RecordsNormalizedRuleMetrics`.<br>2025-10-12 17:05Z: CVE + KEV normalized rule verification complete; OSV parity fixtures revalidated—downstream parity/monitoring tasks may proceed.<br>2025-10-19 14:35Z: Prerequisites reviewed (none outstanding); FEEDMERGE-COORD-02-900 remains in DOING with connector follow-ups unchanged.<br>2025-10-19 15:25Z: Refreshed `RANGE_PRIMITIVES_COORDINATION.md` matrix + added targeted follow-ups (Cccs, CertBund, ICS-CISA, Kisa, Vndr.Cisco) with delivery dates 2025-10-21 → 2025-10-25; monitoring merge counters for regression.|
|Merge pipeline parity for new advisory fields|BE-Merge|Models, Core|DONE (2025-10-15) merge service now surfaces description/CWE/canonical metric decisions with updated metrics/tests.|
|Connector coordination for new advisory fields|Connector Leads, BE-Merge|Models, Core|**DONE (2025-10-15)** GHSA, NVD, and OSV connectors now emit advisory descriptions, CWE weaknesses, and canonical metric ids. Fixtures refreshed (GHSA connector regression suite, `conflict-nvd.canonical.json`, OSV parity snapshots) and completion recorded in coordination log.|
|FEEDMERGE-ENGINE-07-001 Conflict sets & explainers|BE-Merge|FEEDSTORAGE-DATA-07-001|**DOING (2025-10-19)** Merge now captures canonical advisory statements + prepares conflict payload scaffolding (statement hashes, deterministic JSON, tests). Next: surface conflict explainers and replay APIs for Core/WebService before marking DONE.|
|FEEDMERGE-ENGINE-07-001 Conflict sets & explainers|BE-Merge|FEEDSTORAGE-DATA-07-001|**DONE (2025-10-20)** Merge surfaces conflict explainers with replay hashes via `MergeConflictSummary`; API exposes structured payloads and integration tests cover deterministic `asOf` hashes.|
> Remark (2025-10-20): `AdvisoryMergeService` now returns conflict summaries with deterministic hashes; WebService replay endpoint emits typed explainers verified by new tests.

View File

@@ -5,6 +5,8 @@ using System.IO;
using System.Linq;
using System.Net;
using System.Net.Http.Json;
using System.Net.Http.Headers;
using System.Text.Json;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Mvc.Testing;
@@ -17,6 +19,7 @@ using Mongo2Go;
using StellaOps.Concelier.Core.Events;
using StellaOps.Concelier.Core.Jobs;
using StellaOps.Concelier.Models;
using StellaOps.Concelier.Merge.Services;
using StellaOps.Concelier.WebService.Jobs;
using StellaOps.Concelier.WebService.Options;
using Xunit.Sdk;
@@ -271,6 +274,77 @@ public sealed class WebServiceEndpointsTests : IAsyncLifetime
Assert.True(payload.Conflicts is null || payload.Conflicts!.Count == 0);
}
[Fact]
public async Task AdvisoryReplayEndpointReturnsConflictExplainer()
{
var vulnerabilityKey = "CVE-2025-9100";
var statementId = Guid.NewGuid();
var conflictId = Guid.NewGuid();
var recordedAt = DateTimeOffset.Parse("2025-02-01T00:00:00Z", CultureInfo.InvariantCulture);
using (var scope = _factory.Services.CreateScope())
{
var eventLog = scope.ServiceProvider.GetRequiredService<IAdvisoryEventLog>();
var advisory = new Advisory(
advisoryKey: vulnerabilityKey,
title: "Base advisory",
summary: "Baseline summary",
language: "en",
published: recordedAt.AddDays(-1),
modified: recordedAt,
severity: "critical",
exploitKnown: false,
aliases: new[] { vulnerabilityKey },
references: Array.Empty<AdvisoryReference>(),
affectedPackages: Array.Empty<AffectedPackage>(),
cvssMetrics: Array.Empty<CvssMetric>(),
provenance: Array.Empty<AdvisoryProvenance>());
var statementInput = new AdvisoryStatementInput(
vulnerabilityKey,
advisory,
recordedAt,
Array.Empty<Guid>(),
StatementId: statementId,
AdvisoryKey: advisory.AdvisoryKey);
await eventLog.AppendAsync(new AdvisoryEventAppendRequest(new[] { statementInput }), CancellationToken.None);
var explainer = new MergeConflictExplainerPayload(
Type: "severity",
Reason: "mismatch",
PrimarySources: new[] { "vendor" },
PrimaryRank: 1,
SuppressedSources: new[] { "nvd" },
SuppressedRank: 5,
PrimaryValue: "CRITICAL",
SuppressedValue: "MEDIUM");
using var conflictDoc = JsonDocument.Parse(explainer.ToCanonicalJson());
var conflictInput = new AdvisoryConflictInput(
vulnerabilityKey,
conflictDoc,
recordedAt,
new[] { statementId },
ConflictId: conflictId);
await eventLog.AppendAsync(new AdvisoryEventAppendRequest(Array.Empty<AdvisoryStatementInput>(), new[] { conflictInput }), CancellationToken.None);
}
using var client = _factory.CreateClient();
var response = await client.GetAsync($"/concelier/advisories/{vulnerabilityKey}/replay");
Assert.Equal(HttpStatusCode.OK, response.StatusCode);
var payload = await response.Content.ReadFromJsonAsync<ReplayResponse>();
Assert.NotNull(payload);
var conflict = Assert.Single(payload!.Conflicts);
Assert.Equal(conflictId, conflict.ConflictId);
Assert.Equal("severity", conflict.Explainer.Type);
Assert.Equal("mismatch", conflict.Explainer.Reason);
Assert.Equal("CRITICAL", conflict.Explainer.PrimaryValue);
Assert.Equal("MEDIUM", conflict.Explainer.SuppressedValue);
Assert.Equal(conflict.Explainer.ComputeHashHex(), conflict.ConflictHash);
}
[Fact]
public async Task MirrorEndpointsServeConfiguredArtifacts()
{
@@ -379,8 +453,49 @@ public sealed class WebServiceEndpointsTests : IAsyncLifetime
using var client = factory.CreateClient();
var response = await client.GetAsync("/concelier/exports/mirror/secure/manifest.json");
Assert.Equal(HttpStatusCode.Unauthorized, response.StatusCode);
var authHeader = Assert.Single(response.Headers.WwwAuthenticate);
Assert.Equal("Bearer", authHeader.Scheme);
}
[Fact]
public async Task MirrorEndpointsRespectRateLimits()
{
using var temp = new TempDirectory();
var exportId = "20251019T130000Z";
var exportRoot = Path.Combine(temp.Path, exportId);
var mirrorRoot = Path.Combine(exportRoot, "mirror");
Directory.CreateDirectory(mirrorRoot);
await File.WriteAllTextAsync(
Path.Combine(mirrorRoot, "index.json"),
"""{\"schemaVersion\":1,\"domains\":[]}"""
);
var environment = new Dictionary<string, string?>
{
["CONCELIER_MIRROR__ENABLED"] = "true",
["CONCELIER_MIRROR__EXPORTROOT"] = temp.Path,
["CONCELIER_MIRROR__ACTIVEEXPORTID"] = exportId,
["CONCELIER_MIRROR__MAXINDEXREQUESTSPERHOUR"] = "1",
["CONCELIER_MIRROR__DOMAINS__0__ID"] = "primary",
["CONCELIER_MIRROR__DOMAINS__0__REQUIREAUTHENTICATION"] = "false",
["CONCELIER_MIRROR__DOMAINS__0__MAXDOWNLOADREQUESTSPERHOUR"] = "1"
};
using var factory = new ConcelierApplicationFactory(_runner.ConnectionString, environmentOverrides: environment);
using var client = factory.CreateClient();
var okResponse = await client.GetAsync("/concelier/exports/index.json");
Assert.Equal(HttpStatusCode.OK, okResponse.StatusCode);
var limitedResponse = await client.GetAsync("/concelier/exports/index.json");
Assert.Equal((HttpStatusCode)429, limitedResponse.StatusCode);
Assert.NotNull(limitedResponse.Headers.RetryAfter);
Assert.True(limitedResponse.Headers.RetryAfter!.Delta.HasValue);
Assert.True(limitedResponse.Headers.RetryAfter!.Delta!.Value.TotalSeconds > 0);
}
[Fact]
public async Task JobsEndpointsAllowBypassWhenAuthorityEnabled()
{
@@ -553,7 +668,8 @@ public sealed class WebServiceEndpointsTests : IAsyncLifetime
string ConflictHash,
DateTimeOffset AsOf,
DateTimeOffset RecordedAt,
string Details);
string Details,
MergeConflictExplainerPayload Explainer);
private sealed class ConcelierApplicationFactory : WebApplicationFactory<Program>
{

View File

@@ -1,4 +1,5 @@
using System.Globalization;
using System.IO;
using Microsoft.AspNetCore.Http;
using Microsoft.Extensions.Options;
using StellaOps.Concelier.WebService.Options;
@@ -129,6 +130,7 @@ internal static class MirrorEndpointExtensions
return true;
}
context.Response.Headers.WWWAuthenticate = "Bearer realm=\"StellaOps Concelier Mirror\"";
result = Results.StatusCode(StatusCodes.Status401Unauthorized);
return false;
}
@@ -147,7 +149,7 @@ internal static class MirrorEndpointExtensions
FileAccess.Read,
FileShare.Read | FileShare.Delete);
response.Headers.CacheControl = "public, max-age=60";
response.Headers.CacheControl = BuildCacheControlHeader(path);
response.Headers.LastModified = fileInfo.LastWriteTimeUtc.ToString("R", CultureInfo.InvariantCulture);
response.ContentLength = fileInfo.Length;
return Task.FromResult(Results.Stream(stream, contentType));
@@ -178,4 +180,26 @@ internal static class MirrorEndpointExtensions
var seconds = Math.Max((int)Math.Ceiling(retryAfter.Value.TotalSeconds), 1);
response.Headers.RetryAfter = seconds.ToString(CultureInfo.InvariantCulture);
}
private static string BuildCacheControlHeader(string path)
{
var fileName = Path.GetFileName(path);
if (fileName is null)
{
return "public, max-age=60";
}
if (string.Equals(fileName, "index.json", StringComparison.OrdinalIgnoreCase))
{
return "public, max-age=60";
}
if (fileName.EndsWith(".json", StringComparison.OrdinalIgnoreCase) ||
fileName.EndsWith(".jws", StringComparison.OrdinalIgnoreCase))
{
return "public, max-age=300, immutable";
}
return "public, max-age=300";
}
}

View File

@@ -227,7 +227,8 @@ app.MapGet("/concelier/advisories/{vulnerabilityKey}/replay", async (
ConflictHash = Convert.ToHexString(conflict.ConflictHash.ToArray()),
conflict.AsOf,
conflict.RecordedAt,
Details = conflict.CanonicalJson
Details = conflict.CanonicalJson,
Explainer = MergeConflictExplainerPayload.FromCanonicalJson(conflict.CanonicalJson)
}).ToArray()
};

View File

@@ -23,5 +23,6 @@
|Update Concelier operator guide for enforcement cutoff|Docs/Concelier|FSR1 rollout|**DONE (2025-10-12)** Installation guide emphasises disabling `allowAnonymousFallback` before 2025-12-31 UTC and connects audit signals to the rollout checklist.|
|Rename plugin drop directory to namespaced path|BE-Base|Plugins|**DONE (2025-10-19)** Build outputs now target `StellaOps.Concelier.PluginBinaries`/`StellaOps.Authority.PluginBinaries`, plugin host defaults updated, config/docs refreshed, and `dotnet test src/StellaOps.Concelier.WebService.Tests/StellaOps.Concelier.WebService.Tests.csproj --no-restore` covers the change.|
|Authority resilience adoption|Concelier WebService, Docs|Plumb Authority client resilience options|**BLOCKED (2025-10-10)** Roll out retry/offline knobs to deployment docs and confirm CLI parity once LIB5 lands; unblock after resilience options wired and tested.|
|CONCELIER-WEB-08-201 Mirror distribution endpoints|Concelier WebService Guild|CONCELIER-EXPORT-08-201, DEVOPS-MIRROR-08-001|DOING (2025-10-19) HTTP endpoints wired (`/concelier/exports/index.json`, `/concelier/exports/mirror/*`), mirror options bound/validated, and integration tests added; pending auth docs + smoke in ops handbook.|
|CONCELIER-WEB-08-201 Mirror distribution endpoints|Concelier WebService Guild|CONCELIER-EXPORT-08-201, DEVOPS-MIRROR-08-001|**DONE (2025-10-20)** Mirror endpoints now enforce per-domain rate limits, emit cache headers, honour Authority/WWW-Authenticate, and docs cover auth + smoke workflows.|
> Remark (2025-10-20): Updated ops runbook with token/rate-limit checks and added API tests for Retry-After + unauthorized flows.|
|Wave 0B readiness checkpoint|Team WebService & Authority|Wave0A completion|BLOCKED (2025-10-19) FEEDSTORAGE-MONGO-08-001 closed, but remaining Wave0A items (AUTH-DPOP-11-001, AUTH-MTLS-11-002, PLUGIN-DI-08-001) still open; maintain current DOING workstreams only.|