feat: Implement approvals workflow and notifications integration

- Added approvals orchestration with persistence and workflow scaffolding.
- Integrated notifications insights and staged resume hooks.
- Introduced approval coordinator and policy notification bridge with unit tests.
- Added approval decision API with resume requeue and persisted plan snapshots.
- Documented the Excitor consensus API beta and provided JSON sample payload.
- Created analyzers to flag usage of deprecated merge service APIs.
- Implemented logging for artifact uploads and approval decision service.
- Added tests for PackRunApprovalDecisionService and related components.

src/Excititor/__Libraries/StellaOps.Excititor.Attestation/EXCITITOR-ATTEST-01-003-plan.md

@@ -13,16 +13,18 @@
 3. Emit observability signals (logs, metrics, optional tracing) that can run offline and degrade gracefully when transparency services are unreachable.
 4. Add regression tests (unit + integration) covering positive path, negative path, and offline fallback scenarios.
 
-## 2. Deliverables
-
-- `IVexAttestationVerifier` abstraction + `VexAttestationVerifier` implementation inside `StellaOps.Excititor.Attestation`, encapsulating DSSE validation, predicate checks, artifact digest confirmation, Rekor inclusion verification, and deterministic diagnostics.
-- DI wiring (extension method) for registering verifier + instrumentation dependencies alongside the existing signer/rekor client.
-- Shared `VexAttestationDiagnostics` record describing normalized diagnostic keys consumed by Worker/WebService logging.
-- Metrics utility (`AttestationMetrics`) exposing counters/histograms via `System.Diagnostics.Metrics`, exported under `StellaOps.Excititor.Attestation` meter.
-- Activity source (`AttestationActivitySource`) for optional tracing spans around sign/verify operations.
-- Documentation updates (`EXCITITOR-ATTEST-01-003-plan.md`, `TASKS.md` notes) describing instrumentation + test expectations.
-- Test coverage in `StellaOps.Excititor.Attestation.Tests` (unit) and scaffolding notes for WebService/Worker integration tests.
-
+## 2. Deliverables
+
+- `IVexAttestationVerifier` abstraction + `VexAttestationVerifier` implementation inside `StellaOps.Excititor.Attestation`, encapsulating DSSE validation, predicate checks, artifact digest confirmation, Rekor inclusion verification, and deterministic diagnostics.
+- DI wiring (extension method) for registering verifier + instrumentation dependencies alongside the existing signer/rekor client.
+- Shared `VexAttestationDiagnostics` record describing normalized diagnostic keys consumed by Worker/WebService logging.
+- Metrics utility (`AttestationMetrics`) exposing counters/histograms via `System.Diagnostics.Metrics`, exported under `StellaOps.Excititor.Attestation` meter.
+- Activity source (`AttestationActivitySource`) for optional tracing spans around sign/verify operations.
+- 2025-11-05: Implemented `VexAttestationDiagnostics`, activity tagging via `VexAttestationActivitySource`, and updated verifier/tests to emit structured failure reasons.
+- 2025-11-05 (pm): Worker attestation verifier now records structured diagnostics/metrics and logs result/failure reasons using `VexAttestationDiagnostics`; attestation success/failure labels propagate to verification counters.
+- Documentation updates (`EXCITITOR-ATTEST-01-003-plan.md`, `TASKS.md` notes) describing instrumentation + test expectations.
+- Test coverage in `StellaOps.Excititor.Attestation.Tests` (unit) and scaffolding notes for WebService/Worker integration tests.
+
 ## 3. Verification Flow
 
 ### 3.1 Inputs

src/Excititor/__Libraries/StellaOps.Excititor.Attestation/TASKS.md

@@ -1,6 +1,7 @@
-If you are working on this file you need to read docs/modules/excititor/ARCHITECTURE.md and ./AGENTS.md).
-# TASKS
-| Task | Owner(s) | Depends on | Notes |
-|---|---|---|---|
-|EXCITITOR-ATTEST-01-003 – Verification suite & observability|Team Excititor Attestation|EXCITITOR-ATTEST-01-002|DOING (2025-10-22) – Continuing implementation: build `IVexAttestationVerifier`, wire metrics/logging, and add regression tests. Draft plan in `EXCITITOR-ATTEST-01-003-plan.md` (2025-10-19) guides scope; updating with worknotes as progress lands.<br>2025-10-31: Verifier now tolerates duplicate source providers from AOC raw projections, downgrades offline Rekor verification to a degraded result, and enforces trusted signer registry checks with detailed diagnostics/tests.|
-
+If you are working on this file you need to read docs/modules/excititor/ARCHITECTURE.md and ./AGENTS.md).
+# TASKS
+| Task | Owner(s) | Depends on | Notes |
+|---|---|---|---|
+|EXCITITOR-ATTEST-01-003 – Verification suite & observability|Team Excititor Attestation|EXCITITOR-ATTEST-01-002|DOING (2025-10-22) – Continuing implementation: build `IVexAttestationVerifier`, wire metrics/logging, and add regression tests. Draft plan in `EXCITITOR-ATTEST-01-003-plan.md` (2025-10-19) guides scope; updating with worknotes as progress lands.<br>2025-10-31: Verifier now tolerates duplicate source providers from AOC raw projections, downgrades offline Rekor verification to a degraded result, and enforces trusted signer registry checks with detailed diagnostics/tests.<br>2025-11-05 14:35Z: Picking up diagnostics record/ActivitySource work and aligning metrics dimensions before wiring verifier into WebService/Worker paths.|
+> 2025-11-05 19:10Z: Worker signature verifier now emits structured diagnostics/metrics via `VexAttestationDiagnostics`; attestation verification results flow into metric labels and logs.
+> Remark (2025-10-22): Added verifier implementation + metrics/tests; next steps include wiring into WebService/Worker flows and expanding negative-path coverage.

src/Excititor/__Libraries/StellaOps.Excititor.Attestation/Verification/VexAttestationActivitySource.cs

@@ -0,0 +1,10 @@
+using System.Diagnostics;
+
+namespace StellaOps.Excititor.Attestation.Verification;
+
+public static class VexAttestationActivitySource
+{
+    public const string Name = "StellaOps.Excititor.Attestation";
+
+    public static readonly ActivitySource Value = new(Name);
+}

src/Excititor/__Libraries/StellaOps.Excititor.Attestation/Verification/VexAttestationVerifier.cs

@@ -1,8 +1,8 @@
 using System;
-using System.Collections.Generic;
-using System.Collections.Immutable;
-using System.Diagnostics;
-using System.Linq;
+using System.Collections.Generic;
+using System.Collections.Immutable;
+using System.Diagnostics;
+using System.Linq;
 using System.Text;
 using System.Text.Json;
 using System.Text.Json.Serialization;
@@ -59,104 +59,120 @@ internal sealed class VexAttestationVerifier : IVexAttestationVerifier
     public async ValueTask<VexAttestationVerification> VerifyAsync(
         VexAttestationVerificationRequest request,
         CancellationToken cancellationToken)
-    {
-        ArgumentNullException.ThrowIfNull(request);
-
-        var stopwatch = Stopwatch.StartNew();
-        var diagnostics = ImmutableDictionary.CreateBuilder<string, string>(StringComparer.Ordinal);
+    {
+        ArgumentNullException.ThrowIfNull(request);
+
+        var stopwatch = Stopwatch.StartNew();
+        var diagnostics = ImmutableDictionary.CreateBuilder<string, string>(StringComparer.Ordinal);
         var resultLabel = "valid";
         var rekorState = "skipped";
         var component = request.IsReverify ? "worker" : "webservice";
+        void SetFailure(string reason) => diagnostics["failure_reason"] = reason;
+        using var activity = VexAttestationActivitySource.Value.StartActivity("Verify", ActivityKind.Internal);
+        activity?.SetTag("attestation.component", component);
+        activity?.SetTag("attestation.export_id", request.Attestation.ExportId);
 
         try
         {
-            if (string.IsNullOrWhiteSpace(request.Envelope))
-            {
-                diagnostics["envelope.state"] = "missing";
-                _logger.LogWarning("Attestation envelope is missing for export {ExportId}", request.Attestation.ExportId);
-                resultLabel = "invalid";
-                return BuildResult(false);
-            }
-
-            if (!TryDeserializeEnvelope(request.Envelope, out var envelope, diagnostics))
-            {
-                _logger.LogWarning("Failed to deserialize attestation envelope for export {ExportId}", request.Attestation.ExportId);
-                resultLabel = "invalid";
-                return BuildResult(false);
-            }
-
-            if (!string.Equals(envelope.PayloadType, VexDsseBuilder.PayloadType, StringComparison.OrdinalIgnoreCase))
-            {
-                diagnostics["payload.type"] = envelope.PayloadType ?? string.Empty;
-                _logger.LogWarning(
-                    "Unexpected DSSE payload type {PayloadType} for export {ExportId}",
-                    envelope.PayloadType,
-                    request.Attestation.ExportId);
-                resultLabel = "invalid";
-                return BuildResult(false);
-            }
-
-            if (envelope.Signatures is null || envelope.Signatures.Count == 0)
-            {
-                diagnostics["signature.state"] = "missing";
-                _logger.LogWarning("Attestation envelope for export {ExportId} does not contain signatures.", request.Attestation.ExportId);
-                resultLabel = "invalid";
-                return BuildResult(false);
-            }
-
-            var payloadBase64 = envelope.Payload ?? string.Empty;
-            if (!TryDecodePayload(payloadBase64, out var payloadBytes, diagnostics))
-            {
-                _logger.LogWarning("Failed to decode attestation payload for export {ExportId}", request.Attestation.ExportId);
-                resultLabel = "invalid";
-                return BuildResult(false);
-            }
-
-            if (!TryDeserializeStatement(payloadBytes, out var statement, diagnostics))
-            {
-                _logger.LogWarning("Failed to deserialize DSSE statement for export {ExportId}", request.Attestation.ExportId);
-                resultLabel = "invalid";
-                return BuildResult(false);
-            }
-
-            if (!ValidatePredicateType(statement, request, diagnostics))
-            {
-                _logger.LogWarning("Predicate type mismatch for export {ExportId}", request.Attestation.ExportId);
-                resultLabel = "invalid";
-                return BuildResult(false);
-            }
-
-            if (!ValidateSubject(statement, request, diagnostics))
-            {
-                _logger.LogWarning("Subject mismatch for export {ExportId}", request.Attestation.ExportId);
-                resultLabel = "invalid";
-                return BuildResult(false);
-            }
-
-            if (!ValidatePredicate(statement, request, diagnostics))
-            {
-                _logger.LogWarning("Predicate payload mismatch for export {ExportId}", request.Attestation.ExportId);
-                resultLabel = "invalid";
-                return BuildResult(false);
-            }
-
-            if (!ValidateMetadataDigest(envelope, request.Metadata, diagnostics))
-            {
-                _logger.LogWarning("Attestation digest mismatch for export {ExportId}", request.Attestation.ExportId);
-                resultLabel = "invalid";
-                return BuildResult(false);
-            }
-
-            if (!ValidateSignedAt(request.Metadata, request.Attestation.CreatedAt, diagnostics))
-            {
-                _logger.LogWarning("SignedAt validation failed for export {ExportId}", request.Attestation.ExportId);
-                resultLabel = "invalid";
-                return BuildResult(false);
-            }
-
+            if (string.IsNullOrWhiteSpace(request.Envelope))
+            {
+                diagnostics["envelope.state"] = "missing";
+                SetFailure("missing_envelope");
+                _logger.LogWarning("Attestation envelope is missing for export {ExportId}", request.Attestation.ExportId);
+                resultLabel = "invalid";
+                return BuildResult(false);
+            }
+
+            if (!TryDeserializeEnvelope(request.Envelope, out var envelope, diagnostics))
+            {
+                SetFailure("invalid_envelope");
+                _logger.LogWarning("Failed to deserialize attestation envelope for export {ExportId}", request.Attestation.ExportId);
+                resultLabel = "invalid";
+                return BuildResult(false);
+            }
+
+            if (!string.Equals(envelope.PayloadType, VexDsseBuilder.PayloadType, StringComparison.OrdinalIgnoreCase))
+            {
+                diagnostics["payload.type"] = envelope.PayloadType ?? string.Empty;
+                SetFailure("unexpected_payload_type");
+                _logger.LogWarning(
+                    "Unexpected DSSE payload type {PayloadType} for export {ExportId}",
+                    envelope.PayloadType,
+                    request.Attestation.ExportId);
+                resultLabel = "invalid";
+                return BuildResult(false);
+            }
+
+            if (envelope.Signatures is null || envelope.Signatures.Count == 0)
+            {
+                diagnostics["signature.state"] = "missing";
+                SetFailure("missing_signature");
+                _logger.LogWarning("Attestation envelope for export {ExportId} does not contain signatures.", request.Attestation.ExportId);
+                resultLabel = "invalid";
+                return BuildResult(false);
+            }
+
+            var payloadBase64 = envelope.Payload ?? string.Empty;
+            if (!TryDecodePayload(payloadBase64, out var payloadBytes, diagnostics))
+            {
+                SetFailure("payload_decode_failed");
+                _logger.LogWarning("Failed to decode attestation payload for export {ExportId}", request.Attestation.ExportId);
+                resultLabel = "invalid";
+                return BuildResult(false);
+            }
+
+            if (!TryDeserializeStatement(payloadBytes, out var statement, diagnostics))
+            {
+                SetFailure("invalid_statement");
+                _logger.LogWarning("Failed to deserialize DSSE statement for export {ExportId}", request.Attestation.ExportId);
+                resultLabel = "invalid";
+                return BuildResult(false);
+            }
+
+            if (!ValidatePredicateType(statement, request, diagnostics))
+            {
+                SetFailure("predicate_type_mismatch");
+                _logger.LogWarning("Predicate type mismatch for export {ExportId}", request.Attestation.ExportId);
+                resultLabel = "invalid";
+                return BuildResult(false);
+            }
+
+            if (!ValidateSubject(statement, request, diagnostics))
+            {
+                SetFailure("subject_mismatch");
+                _logger.LogWarning("Subject mismatch for export {ExportId}", request.Attestation.ExportId);
+                resultLabel = "invalid";
+                return BuildResult(false);
+            }
+
+            if (!ValidatePredicate(statement, request, diagnostics))
+            {
+                SetFailure("predicate_mismatch");
+                _logger.LogWarning("Predicate payload mismatch for export {ExportId}", request.Attestation.ExportId);
+                resultLabel = "invalid";
+                return BuildResult(false);
+            }
+
+            if (!ValidateMetadataDigest(envelope, request.Metadata, diagnostics))
+            {
+                SetFailure("envelope_digest_mismatch");
+                _logger.LogWarning("Attestation digest mismatch for export {ExportId}", request.Attestation.ExportId);
+                resultLabel = "invalid";
+                return BuildResult(false);
+            }
+
+            if (!ValidateSignedAt(request.Metadata, request.Attestation.CreatedAt, diagnostics))
+            {
+                SetFailure("signedat_out_of_range");
+                _logger.LogWarning("SignedAt validation failed for export {ExportId}", request.Attestation.ExportId);
+                resultLabel = "invalid";
+                return BuildResult(false);
+            }
+
             rekorState = await VerifyTransparencyAsync(request.Metadata, diagnostics, cancellationToken).ConfigureAwait(false);
             if (rekorState is "missing" or "unverified" or "client_unavailable")
             {
+                SetFailure(rekorState);
                 resultLabel = "invalid";
                 return BuildResult(false);
             }
@@ -164,6 +180,9 @@ internal sealed class VexAttestationVerifier : IVexAttestationVerifier
             var signaturesVerified = await VerifySignaturesAsync(payloadBytes, envelope.Signatures, diagnostics, cancellationToken).ConfigureAwait(false);
             if (!signaturesVerified)
             {
+                diagnostics["failure_reason"] = diagnostics.TryGetValue("signature.reason", out var reason)
+                    ? reason
+                    : "signature_verification_failed";
                 if (_options.RequireSignatureVerification)
                 {
                     resultLabel = "invalid";
@@ -183,13 +202,16 @@ internal sealed class VexAttestationVerifier : IVexAttestationVerifier
         catch (Exception ex)
         {
             diagnostics["error"] = ex.GetType().Name;
-            diagnostics["error.message"] = ex.Message;            resultLabel = "error";
-            _logger.LogError(ex, "Unexpected exception verifying attestation for export {ExportId}", request.Attestation.ExportId);
-            return BuildResult(false);
-        }
-        finally
-        {
-            stopwatch.Stop();
+            diagnostics["error.message"] = ex.Message;            resultLabel = "error";
+            _logger.LogError(ex, "Unexpected exception verifying attestation for export {ExportId}", request.Attestation.ExportId);
+            diagnostics["failure_reason"] = diagnostics.TryGetValue("error", out var errorCode)
+                ? errorCode
+                : ex.GetType().Name;
+            return BuildResult(false);
+        }
+        finally
+        {
+            stopwatch.Stop();
             var tags = new KeyValuePair<string, object?>[]
             {
                 new("result", resultLabel),
@@ -200,12 +222,32 @@ internal sealed class VexAttestationVerifier : IVexAttestationVerifier
             _metrics.VerifyDuration.Record(stopwatch.Elapsed.TotalSeconds, tags);
         }
 
-        VexAttestationVerification BuildResult(bool isValid)
-        {
-            diagnostics["result"] = resultLabel;
-            diagnostics["component"] = component;
+        VexAttestationVerification BuildResult(bool isValid)
+        {
+            diagnostics["result"] = resultLabel;
+            diagnostics["component"] = component;
             diagnostics["rekor.state"] = rekorState;
-            return new VexAttestationVerification(isValid, diagnostics.ToImmutable());
+            var snapshot = VexAttestationDiagnostics.FromBuilder(diagnostics);
+
+            if (activity is { } currentActivity)
+            {
+                currentActivity.SetTag("attestation.result", resultLabel);
+                currentActivity.SetTag("attestation.rekor", rekorState);
+                if (!isValid)
+                {
+                    var failure = snapshot.FailureReason ?? "verification_failed";
+                    currentActivity.SetStatus(ActivityStatusCode.Error, failure);
+                    currentActivity.SetTag("attestation.failure_reason", failure);
+                }
+                else
+                {
+                    currentActivity.SetStatus(resultLabel is "degraded"
+                        ? ActivityStatusCode.Ok
+                        : ActivityStatusCode.Ok);
+                }
+            }
+
+            return new VexAttestationVerification(isValid, snapshot);
         }
     }
 

src/Excititor/__Libraries/StellaOps.Excititor.Core/StellaOps.Excititor.Core.csproj

@@ -14,4 +14,4 @@
     <ProjectReference Include="../../../Aoc/__Libraries/StellaOps.Aoc/StellaOps.Aoc.csproj" />
     <ProjectReference Include="../../../Concelier/__Libraries/StellaOps.Concelier.RawModels/StellaOps.Concelier.RawModels.csproj" />
   </ItemGroup>
-</Project>
+</Project>

src/Excititor/__Libraries/StellaOps.Excititor.Core/VexAttestationAbstractions.cs

@@ -1,7 +1,8 @@
 using System;
 using System.Collections.Immutable;
 using System.Threading;
-using System.Threading.Tasks;
+using System.Threading.Tasks;
+using StellaOps.Excititor.Attestation.Verification;
 
 namespace StellaOps.Excititor.Core;
 
@@ -33,4 +34,4 @@ public sealed record VexAttestationVerificationRequest(
 
 public sealed record VexAttestationVerification(
     bool IsValid,
-    ImmutableDictionary<string, string> Diagnostics);
+    VexAttestationDiagnostics Diagnostics);

src/Excititor/__Libraries/StellaOps.Excititor.Core/VexAttestationDiagnostics.cs

@@ -0,0 +1,57 @@
+using System.Collections;
+using System.Collections.Generic;
+using System.Collections.Immutable;
+
+namespace StellaOps.Excititor.Attestation.Verification;
+
+public sealed class VexAttestationDiagnostics : IReadOnlyDictionary<string, string>
+{
+    private readonly ImmutableDictionary<string, string> _values;
+
+    private VexAttestationDiagnostics(ImmutableDictionary<string, string> values)
+    {
+        _values = values ?? ImmutableDictionary<string, string>.Empty;
+    }
+
+    public static VexAttestationDiagnostics FromBuilder(ImmutableDictionary<string, string>.Builder builder)
+    {
+        ArgumentNullException.ThrowIfNull(builder);
+        return new(builder.ToImmutable());
+    }
+
+    public static VexAttestationDiagnostics Empty { get; } = new(ImmutableDictionary<string, string>.Empty);
+
+    public string? Result => TryGetValue("result", out var value) ? value : null;
+
+    public string? Component => TryGetValue("component", out var value) ? value : null;
+
+    public string? RekorState => TryGetValue("rekor.state", out var value) ? value : null;
+
+    public string? FailureReason => TryGetValue("failure_reason", out var value) ? value : null;
+
+    public string this[string key] => _values[key];
+
+    public IEnumerable<string> Keys => _values.Keys;
+
+    public IEnumerable<string> Values => _values.Values;
+
+    public int Count => _values.Count;
+
+    public bool ContainsKey(string key) => _values.ContainsKey(key);
+
+    public bool TryGetValue(string key, out string value)
+    {
+        if (_values.TryGetValue(key, out var stored))
+        {
+            value = stored;
+            return true;
+        }
+
+        value = string.Empty;
+        return false;
+    }
+
+    public IEnumerator<KeyValuePair<string, string>> GetEnumerator() => _values.GetEnumerator();
+
+    IEnumerator IEnumerable.GetEnumerator() => GetEnumerator();
+}