tests fixes and some product advisories tunes ups

docs/implplan/SPRINT_0127_001_0001_oci_referrer_bundle_export.md

@@ -1,378 +0,0 @@
-# Sprint 0127 · OCI Referrer Bundle Export (Critical Gap Closure)
-
-## Topic & Scope
-- **Critical gap**: Mirror bundle and offline kit exports do NOT discover or include OCI referrer artifacts (SBOMs, attestations, signatures) linked to images via the OCI 1.1 referrers API.
-- Integrate existing `OciReferrerDiscovery` infrastructure into `MirrorAdapter`, `MirrorBundleBuilder`, and `OfflineKitPackager` flows.
-- Ensure `ImportValidator` verifies referrer artifacts are present for each subject image.
-- Support fallback tag-based discovery for registries without OCI 1.1 API (e.g., GHCR).
-- **Working directory:** `src/ExportCenter/`, `src/AirGap/`
-- **Expected evidence:** Unit tests, integration tests with Testcontainers, deterministic bundle output verification.
-
-## Dependencies & Concurrency
-- Upstream: `OciReferrerDiscovery` and `OciReferrerFallback` already implemented in `src/ExportCenter/.../Distribution/Oci/`.
-- No blocking dependencies; can proceed immediately.
-- Concurrency: Tasks 1-3 can proceed in parallel; Task 4-6 depend on 1-3.
-
-## Documentation Prerequisites
-- `docs/modules/export-center/architecture.md` (update with referrer discovery flow)
-- `docs/modules/airgap/guides/offline-bundle-format.md` (update bundle structure)
-- Advisory source: OCI v1.1 referrers API specification and registry compatibility matrix.
-
----
-
-## Delivery Tracker
-
-### REF-EXPORT-01 - Add Referrer Discovery to MirrorAdapter
-Status: DONE
-Dependency: None
-Owners: ExportCenter Guild
-
-Task description:
-Modify `MirrorAdapter.CollectDataSourcesAsync()` to detect image references in items and automatically discover their OCI referrer artifacts.
-
-For each item that represents a container image (identifiable by digest pattern `sha256:*` or image reference format):
-1. Call `OciReferrerDiscovery.ListReferrersAsync()` with the image digest
-2. If referrers API returns 404/empty, call `OciReferrerFallback.DiscoverViaTagsAsync()` to check for `sha256-{digest}.*` tags
-3. For each discovered referrer (SBOM, attestation, signature, VEX), fetch the artifact content
-4. Add discovered artifacts to the data sources list with appropriate `MirrorBundleDataCategory`
-
-Inject `IOciReferrerDiscovery` and `IOciReferrerFallback` via DI into `MirrorAdapter`.
-
-Handle errors gracefully: if referrer discovery fails for a single image, log warning and continue with other images.
-
-Implementation completed:
-- Created `IReferrerDiscoveryService` interface in Core with `DiscoverReferrersAsync` and `GetReferrerContentAsync`
-- Created `ReferrerDiscoveryResult`, `DiscoveredReferrer`, `ReferrerLayer` models
-- Added `NullReferrerDiscoveryService` for when discovery is disabled
-- Modified `MirrorAdapter` to inject `IReferrerDiscoveryService` (optional)
-- Added `IsImageReference()` detection and `DiscoverAndCollectReferrersAsync()` method
-- Added artifact type to category mapping (SBOM, VEX, Attestation, DSSE, SLSA, etc.)
-- Created `OciReferrerDiscoveryService` wrapper in WebService to implement `IReferrerDiscoveryService`
-- Updated DI registration in `ExportAdapterRegistry`
-- Added 21 unit tests for MirrorAdapter referrer discovery
-- Added 15 unit tests for OciReferrerDiscoveryService
-
-Completion criteria:
-- [x] `MirrorAdapter.CollectDataSourcesAsync()` calls `OciReferrerDiscovery.ListReferrersAsync()` for image items
-- [x] Fallback tag discovery is invoked when native API returns 404 (via OciReferrerDiscovery)
-- [x] Discovered SBOMs are added with category `Sbom`
-- [x] Discovered attestations are added with category `Attestation`
-- [x] Discovered VEX statements are added with category `Vex`
-- [x] Unit tests verify discovery flow with mocked HTTP handlers (36 tests passing)
-- [ ] Integration test with Testcontainers `registry:2` verifies end-to-end flow (deferred)
-
----
-
-### REF-EXPORT-02 - Extend MirrorBundleBuilder for Referrer Metadata
-Status: DONE
-Dependency: None
-Owners: ExportCenter Guild
-
-Task description:
-Update `MirrorBundleBuilder` to track the relationship between subject images and their referrer artifacts in the bundle manifest.
-
-Add to `manifest.yaml`:
-```yaml
-referrers:
-  - subject: "sha256:abc123..."
-    artifacts:
-      - digest: "sha256:def456..."
-        artifactType: "application/vnd.cyclonedx+json"
-        mediaType: "application/vnd.oci.image.manifest.v1+json"
-        size: 12345
-        annotations:
-          org.opencontainers.image.created: "2026-01-27T10:00:00Z"
-      - digest: "sha256:ghi789..."
-        artifactType: "application/vnd.in-toto+json"
-        ...
-```
-
-Update bundle structure to include referrer artifacts under `referrers/` directory:
-```
-bundle.tgz
-├── manifest.yaml          # Updated with referrers section
-├── images/
-│   └── sha256-abc123/
-│       └── manifest.json
-├── referrers/
-│   └── sha256-abc123/     # Keyed by subject digest
-│       ├── sha256-def456.json  # SBOM
-│       └── sha256-ghi789.json  # Attestation
-└── checksums.txt
-```
-
-Implementation completed:
-- Added `Attestation = 8` and `Referrer = 9` to `MirrorBundleDataCategory` enum
-- Updated `MirrorBundleManifestCounts` to include `Attestations` and `Referrers` fields
-- Updated `MirrorBundleBuilder.ComputeBundlePath()` to handle referrer categories under `referrers/{subject-digest}/`
-- Updated `SerializeManifestToYaml()` to include attestation and referrer counts
-- Updated `BuildReadme()` to include attestation and referrer counts
-- Added `indexes/attestations.index.json` and `indexes/referrers.index.json` placeholder files
-- Created referrer metadata models in `MirrorBundleModels.cs`:
-  - `MirrorBundleReferrersSection`, `MirrorBundleSubjectReferrers`, `MirrorBundleReferrerArtifact`
-  - `MirrorBundleReferrerCounts`, `MirrorBundleReferrerDataSource`
-- All 13 existing MirrorBundleBuilder tests continue to pass
-
-Completion criteria:
-- [x] `MirrorBundleBuilder` accepts referrer metadata in build request
-- [x] `manifest.yaml` includes counts for attestations and referrers
-- [x] Referrer artifacts stored under `referrers/{subject-digest}/` directory
-- [x] `checksums.txt` includes referrer artifact hashes (existing behavior)
-- [x] Bundle structure is deterministic (sorted by digest)
-- [x] Unit tests verify manifest structure (existing tests pass)
-- [x] Existing tests continue to pass (13/13 pass)
-
----
-
-### REF-EXPORT-03 - Extend OfflineKitPackager for Referrer Artifacts
-Status: DONE
-Dependency: None
-Owners: ExportCenter Guild · AirGap Guild
-
-Task description:
-Update `OfflineKitPackager` to propagate referrer artifacts from mirror bundles into offline kits.
-
-When packaging an offline kit from mirror bundles:
-1. Detect `referrers/` directory in source mirror bundle
-2. Copy referrer artifacts to offline kit with same structure
-3. Update offline kit manifest to include referrer metadata
-4. Add verification for referrer presence in `verify-offline-kit.sh`
-
-Update `OfflineKitManifest` to include:
-```csharp
-public IReadOnlyList<OfflineKitReferrerEntry> Referrers { get; init; }
-
-public record OfflineKitReferrerEntry
-{
-    public required string SubjectDigest { get; init; }
-    public required IReadOnlyList<OfflineKitReferrerArtifact> Artifacts { get; init; }
-}
-
-public record OfflineKitReferrerArtifact
-{
-    public required string Digest { get; init; }
-    public required string ArtifactType { get; init; }
-    public required string MediaType { get; init; }
-    public required long SizeBytes { get; init; }
-    public required string RelativePath { get; init; }
-}
-```
-
-Implementation completed:
-- Added `OfflineKitReferrersSummary` record with counts for subjects, artifacts, SBOMs, attestations, VEX, other
-- Updated `OfflineKitMirrorEntry` to include optional `Referrers` summary field
-- Updated `OfflineKitMirrorRequest` to accept optional `Referrers` parameter
-- Updated `OfflineKitPackager.CreateMirrorEntry()` to include referrer summary in manifest entry
-- Note: Referrer artifacts are already inside the mirror bundle (tar.gz), so no separate copying needed
-- All 27 existing OfflineKitPackager tests continue to pass
-
-Completion criteria:
-- [x] `OfflineKitPackager` propagates referrer summary from request to manifest
-- [x] Offline kit manifest includes referrer metadata summary (counts, API support)
-- [ ] `verify-offline-kit.sh` validates referrer artifact presence (deferred - inside bundle)
-- [x] Unit tests verify referrer handling (existing tests pass)
-- [ ] Integration test packages kit with referrers and verifies structure (deferred)
-
----
-
-### REF-EXPORT-04 - Add Referrer Verification to ImportValidator
-Status: DONE
-Dependency: REF-EXPORT-02, REF-EXPORT-03
-Owners: AirGap Guild
-
-Task description:
-Update `ImportValidator` to verify that all referrer artifacts declared in the manifest are present in the bundle.
-
-In `ImportValidator.ValidateAsync()`:
-1. Parse `referrers` section from manifest
-2. For each subject image:
-   - Verify all declared referrer artifacts exist at expected paths
-   - Verify artifact checksums match declared values
-   - Verify artifact sizes match declared values
-3. Add validation result entries for:
-   - `ReferrerMissing`: Declared artifact not found in bundle
-   - `ReferrerChecksumMismatch`: Artifact checksum doesn't match
-   - `ReferrerSizeMismatch`: Artifact size doesn't match
-   - `OrphanedReferrer`: Artifact exists but not declared (warning only)
-
-Update `BundleValidationResult` to include referrer validation summary:
-```csharp
-public record ReferrerValidationSummary
-{
-    public int TotalSubjects { get; init; }
-    public int TotalReferrers { get; init; }
-    public int ValidReferrers { get; init; }
-    public int MissingReferrers { get; init; }
-    public int ChecksumMismatches { get; init; }
-    public IReadOnlyList<ReferrerValidationIssue> Issues { get; init; }
-}
-```
-
-Implementation completed:
-- Created `ReferrerValidator` class with `Validate()` method that parses referrers section from manifest JSON
-- Created `ReferrerValidationSummary`, `ReferrerValidationIssue`, `ReferrerValidationIssueType`, `ReferrerValidationSeverity` types
-- Updated `BundleValidationResult` to include optional `ReferrerSummary` property
-- Integrated `ReferrerValidator` into `ImportValidator` as optional dependency
-- Added validation for missing artifacts, checksum mismatches, size mismatches
-- Orphaned referrers (files in referrers/ not declared in manifest) produce warnings only
-- Added `IsBundleTypeWithReferrers()` to enable validation only for mirror-bundle and offline-kit types
-- Created 17 unit tests for ReferrerValidator
-- Created 2 integration tests for ImportValidator with referrer validation
-
-Completion criteria:
-- [x] `ImportValidator` parses and validates referrer section
-- [x] Missing referrer artifacts fail validation
-- [x] Checksum mismatches fail validation
-- [x] Orphaned referrers produce warnings (not failures)
-- [x] `BundleValidationResult` includes referrer summary
-- [x] Unit tests cover all validation scenarios (17 tests in ReferrerValidatorTests.cs + 2 in ImportValidatorTests.cs)
-- [ ] Integration test imports bundle with intentional errors and verifies detection (deferred)
-
----
-
-### REF-EXPORT-05 - Add Registry Capability Probing to Export Flow
-Status: DONE
-Dependency: REF-EXPORT-01
-Owners: ExportCenter Guild
-
-Task description:
-Before discovering referrers for an image, probe the registry to determine the best discovery strategy.
-
-Use `OciReferrerFallback.ProbeCapabilitiesAsync()` to detect:
-- `SupportsReferrersApi`: Native OCI 1.1 referrers API available
-- `DistributionVersion`: OCI Distribution spec version
-- `SupportsArtifactType`: Registry supports artifactType field
-
-Cache capabilities per registry host (already implemented with 1-hour TTL).
-
-Log registry capabilities at start of export:
-```
-[INFO] Registry registry.example.com: OCI 1.1 (referrers API supported)
-[WARN] Registry ghcr.io: OCI 1.0 (using fallback tag discovery)
-```
-
-Add export metrics:
-- `export_registry_capabilities_probed_total{registry,api_supported}`
-- `export_referrer_discovery_method_total{method=native|fallback}`
-
-Implementation completed:
-- Added `ProbeRegistryCapabilitiesAsync` to `IReferrerDiscoveryService` interface and `RegistryCapabilitiesInfo` record
-- Updated `OciReferrerDiscoveryService` to probe capabilities using `IOciReferrerFallback.ProbeCapabilitiesAsync()` with caching
-- Updated `MirrorAdapter.DiscoverAndCollectReferrersAsync()` to probe all unique registries before starting discovery
-- Added logging at export start: "Probing {RegistryCount} registries for OCI referrer capabilities before export"
-- Added capability logging: "Registry {Registry}: OCI 1.1 (referrers API supported, version={Version}, probe_ms={ProbeMs})" or warning for fallback
-- Using existing `ExportTelemetry` metrics: `RegistryCapabilitiesProbedTotal`, `ReferrerDiscoveryMethodTotal`, `ReferrersDiscoveredTotal`, `ReferrerDiscoveryFailuresTotal`
-- Added 3 new unit tests for probe-then-discover flow in `MirrorAdapterReferrerDiscoveryTests.cs`
-
-Completion criteria:
-- [x] Export flow probes registry capabilities before discovery
-- [x] Capabilities are logged at export start
-- [x] Metrics track probe results and discovery methods
-- [x] Fallback is automatically used for registries without API support
-- [x] Unit tests verify probe-then-discover flow
-- [ ] Integration test with `registry:2` verifies native API path (deferred)
-
----
-
-### REF-EXPORT-06 - Update Documentation and Architecture Docs
-Status: DONE
-Dependency: REF-EXPORT-01, REF-EXPORT-02, REF-EXPORT-03, REF-EXPORT-04
-Owners: Documentation Guild
-
-Task description:
-Update documentation to reflect new referrer discovery and bundle handling.
-
-Files to update:
-1. `docs/modules/export-center/architecture.md`:
-   - Add section on OCI referrer discovery
-   - Document fallback mechanism for non-OCI-1.1 registries
-   - Add sequence diagram for referrer discovery flow
-
-2. `docs/modules/airgap/guides/offline-bundle-format.md`:
-   - Update bundle structure to show `referrers/` directory
-   - Document referrer manifest format
-   - Add example with SBOM and attestation referrers
-
-3. `docs/runbooks/registry-referrer-troubleshooting.md` (new):
-   - How to diagnose referrer discovery issues
-   - Registry compatibility matrix (brief, links to detailed doc)
-   - Common issues and solutions
-
-4. `docs/modules/export-center/registry-compatibility.md` (new):
-   - Detailed registry compatibility matrix
-   - Per-registry quirks and workarounds
-   - Includes: GHCR, ACR, ECR, GCR, Harbor, Quay, JFrog
-
-Implementation completed:
-- Updated `architecture.md` with "OCI Referrer Discovery" section including:
-  - Discovery flow diagram (ASCII)
-  - Capability probing explanation
-  - Telemetry metrics table
-  - Artifact type mapping table
-  - Error handling notes
-  - Links to related docs
-- Updated `offline-bundle-format.md` with "OCI Referrer Artifacts" section including:
-  - Referrer directory structure
-  - Manifest referrers section YAML example
-  - Referrer validation table
-  - Artifact types table
-  - Registry compatibility note
-- Created `registry-referrer-troubleshooting.md` runbook with:
-  - Quick reference table
-  - Registry compatibility quick reference
-  - Diagnostic steps (logs, metrics, connectivity tests)
-  - Common issues and solutions
-  - Validation commands
-  - Escalation process
-- Created `registry-compatibility.md` with:
-  - Compatibility summary table
-  - Detection behavior explanation
-  - Per-registry details (Docker Hub, GHCR, GCR, ECR, ACR, Harbor, Quay, JFrog)
-  - Fallback tag discovery documentation
-  - Testing instructions
-
-Completion criteria:
-- [x] Architecture doc updated with referrer discovery flow
-- [x] Bundle format doc updated with referrer structure
-- [x] New runbook created for troubleshooting
-- [x] New compatibility matrix doc created
-- [x] All docs link to each other appropriately
-- [x] Code comments reference relevant docs (via doc links)
-
----
-
-## Execution Log
-| Date (UTC) | Update | Owner |
-| --- | --- | --- |
-| 2026-01-27 | Sprint created from OCI v1.1 referrers advisory review; critical gap identified in mirror bundle export. | Planning |
-| 2026-01-27 | REF-EXPORT-01 DONE: Created IReferrerDiscoveryService interface, integrated into MirrorAdapter, added OciReferrerDiscoveryService wrapper, 36 tests passing. | Implementation |
-| 2026-01-27 | REF-EXPORT-02 DONE: Added attestation/referrer counts to manifest YAML and README, added index placeholders, all 13 existing tests pass. | Implementation |
-| 2026-01-27 | REF-EXPORT-03 DONE: Added OfflineKitReferrersSummary, updated OfflineKitMirrorEntry/Request, all 27 existing tests pass. | Implementation |
-| 2026-01-27 | Core implementation complete (01, 02, 03). REF-EXPORT-04, 05, 06 deferred for follow-up. Total: 76 tests passing across 10 new/modified files. | Implementation |
-| 2026-01-27 | REF-EXPORT-04 DONE: Created ReferrerValidator with validation logic, integrated into ImportValidator, updated BundleValidationResult with ReferrerSummary. 19 new tests (17 ReferrerValidator + 2 ImportValidator). | Implementation |
-| 2026-01-27 | REF-EXPORT-05 verified TODO: ProbeCapabilitiesAsync infrastructure exists in OciReferrerFallback.cs with 1-hour cache, but MirrorAdapter does not call it before discovery. No metrics implemented. Fallback works automatically via OciReferrerDiscovery.ListReferrersAsync(). | Verification |
-| 2026-01-27 | REF-EXPORT-06 verified TODO: Checked architecture.md and offline-bundle-format.md - no referrer mentions. registry-compatibility.md and registry-referrer-troubleshooting.md do not exist. | Verification |
-| 2026-01-27 | REF-EXPORT-05 DONE: Added ProbeRegistryCapabilitiesAsync to IReferrerDiscoveryService, updated OciReferrerDiscoveryService with probing and metrics, updated MirrorAdapter to probe before discovery. 3 new tests. | Implementation |
-| 2026-01-27 | REF-EXPORT-06 DONE: Updated architecture.md and offline-bundle-format.md with OCI referrer sections. Created registry-referrer-troubleshooting.md runbook and registry-compatibility.md with detailed per-registry info. All docs cross-linked. | Documentation |
-| 2026-01-27 | Sprint COMPLETE: All 6 tasks DONE. Core implementation (01-04) + capability probing (05) + documentation (06). Integration tests deferred as noted in criteria. | Milestone |
-
-## Decisions & Risks
-| Item | Status / Decision | Notes |
-| --- | --- | --- |
-| Critical gap confirmation | CONFIRMED | `MirrorAdapter` does not call `OciReferrerDiscovery`; artifacts silently dropped. |
-| Referrer storage structure | PROPOSED | `referrers/{subject-digest}/` hierarchy; to be confirmed during implementation. |
-| Fallback tag pattern | USE EXISTING | `sha256-{digest}.*` pattern already in `OciReferrerFallback`. |
-
-### Risk table
-| Risk | Severity | Mitigation / Owner |
-| --- | --- | --- |
-| Referrer discovery significantly increases export time | Medium | Add parallelism, cache registry probes; measure in integration tests. |
-| Large referrer artifacts bloat bundles | Medium | Add size limits/warnings; document recommended max sizes. |
-| Fallback tag discovery misses artifacts | Low | Comprehensive testing with GHCR-like behavior. |
-
-## Next Checkpoints
-| Date (UTC) | Session / Owner | Target outcome | Fallback / Escalation |
-| --- | --- | --- | --- |
-| 2026-02-03 | REF-EXPORT-01/02/03 completion | Core referrer discovery and bundle integration complete. | If blocked, escalate registry access issues. |
-| 2026-02-07 | REF-EXPORT-04/05 completion | Validation and capability probing complete. | Defer non-critical enhancements if needed. |
-| 2026-02-10 | Sprint completion + docs | All tasks DONE, documentation updated. | Archive sprint; carry forward any blockers. |