feat: Add Scanner CI runner and related artifacts
- Implemented `run-scanner-ci.sh` to build and run tests for the Scanner solution with a warmed NuGet cache. - Created `excititor-vex-traces.json` dashboard for monitoring Excititor VEX observations. - Added Docker Compose configuration for the OTLP span sink in `docker-compose.spansink.yml`. - Configured OpenTelemetry collector in `otel-spansink.yaml` to receive and process traces. - Developed `run-spansink.sh` script to run the OTLP span sink for Excititor traces. - Introduced `FileSystemRiskBundleObjectStore` for storing risk bundle artifacts in the filesystem. - Built `RiskBundleBuilder` for creating risk bundles with associated metadata and providers. - Established `RiskBundleJob` to execute the risk bundle creation and storage process. - Defined models for risk bundle inputs, entries, and manifests in `RiskBundleModels.cs`. - Implemented signing functionality for risk bundle manifests with `HmacRiskBundleManifestSigner`. - Created unit tests for `RiskBundleBuilder`, `RiskBundleJob`, and signing functionality to ensure correctness. - Added filesystem artifact reader tests to validate manifest parsing and artifact listing. - Included test manifests for egress scenarios in the task runner tests. - Developed timeline query service tests to verify tenant and event ID handling.
This commit is contained in:
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"_note": "Placeholder Grafana dashboard stub for Export Center. Replace panels when metrics endpoints are available; keep offline-import friendly.",
|
||||
"schemaVersion": 39,
|
||||
"title": "Export Center Observability (stub)",
|
||||
"panels": []
|
||||
}
|
||||
37
docs/modules/export-center/operations/observability.md
Normal file
37
docs/modules/export-center/operations/observability.md
Normal file
@@ -0,0 +1,37 @@
|
||||
# Export Center observability runbook (stub · 2025-11-29 demo)
|
||||
|
||||
## Dashboards (offline import)
|
||||
- Grafana JSON: `docs/modules/export-center/operations/dashboards/export-center-observability.json` (import locally; no external data sources assumed).
|
||||
- Planned panels: export job duration p95/p99, bundle size histogram, registry push latency, provenance/attestation verification failures, queue depth, and error rate per profile.
|
||||
|
||||
## Key metrics
|
||||
- `export_job_duration_seconds_bucket{profile}` — export duration by profile.
|
||||
- `export_bundle_size_bytes_bucket{profile}` — bundle size distribution.
|
||||
- `export_registry_push_latency_seconds_bucket{profile}` — registry push latency.
|
||||
- `export_attestation_failures_total{reason}` — DSSE/provenance verification failures.
|
||||
- `export_queue_depth` — pending export jobs.
|
||||
- `export_manifest_publish_total{result}` — manifest publish successes/failures.
|
||||
|
||||
## Logs & traces
|
||||
- Correlate by `exportId`, `profile`, `tenant`; include `bundleDigest`, `attestationStatus`, `registry`. Traces disabled by default; enable OTLP to on-prem collector when permitted.
|
||||
|
||||
## Health/diagnostics
|
||||
- `/health/liveness` and `/health/readiness` (export service) check storage, registry reachability, and attestation verification path.
|
||||
- `/status` exposes build version, commit, feature flags; verify against offline bundle manifest.
|
||||
- Verification probe: `stella export bundle verify --manifest <path>` once bundle available; validate hashes against manifest.
|
||||
|
||||
## Alert hints
|
||||
- Export job duration p99 > target SLA per profile.
|
||||
- Attestation verification failures > 0 over 10m.
|
||||
- Registry push latency spikes or error rate > threshold.
|
||||
- Queue depth growth without completion.
|
||||
|
||||
## Offline verification steps
|
||||
1) Import Grafana JSON locally; point to Prometheus scrape labeled `export-center`.
|
||||
2) Run `stella export bundle --profile <profile> --manifest out/manifest.json` and verify hashes via `jq -r '.files[].sha256'` against generated bundles.
|
||||
3) Fetch `/status` and compare commit/version to offline bundle manifest.
|
||||
|
||||
## Evidence locations
|
||||
- Sprint tracker: `docs/implplan/SPRINT_0320_0001_0001_docs_modules_export_center.md`.
|
||||
- Module docs: `README.md`, `architecture.md`, `implementation_plan.md`.
|
||||
- Dashboard stub: `operations/dashboards/export-center-observability.json`.
|
||||
@@ -0,0 +1,42 @@
|
||||
# Risk Bundle Provider Matrix & Signing Baseline
|
||||
|
||||
Status: Baseline for Sprint 0164-0001-0001 (RISK-BUNDLE-69/70 chain)
|
||||
|
||||
## Provider catalog (deterministic ordering)
|
||||
| Provider ID | Source feed (offline-ready) | Coverage | Refresh cadence | Signing / integrity | Notes |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| cisa-kev | CISA Known Exploited Vulnerabilities JSON | Exploited CVEs with required/known exploited flag | Daily | DSSE signature using ExportCenter signing key; feed hash recorded in `provider-manifest.json` | Mandatory; fails bundle if feed missing or hash mismatch. |
|
||||
| first-epss | FIRST EPSS CSV snapshot | Probability scores per CVE | Daily | DSSE signature; SHA-256 of snapshot stored in manifest | Optional; omit if snapshot stale >48h unless `allowStale=true`. |
|
||||
| osv | OpenSSF OSV bulk JSON (per-ecosystem shards) | OSS advisories with affected package ranges | Weekly | DSSE signature; per-shard SHA-256 list | Included only when `includeOsv=true` in job options to keep bundle size constrained. |
|
||||
| vendor-csaf | CSAF vendor advisories (Red Hat, SUSE, Debian) mirrored via Offline Kit | Vendor-specific CVEs, remediations | Weekly | Detached signature per CSAF document (vendor-provided where available) plus bundle-level DSSE manifest | Requires Offline Kit mirror; missing vendor feeds logged but bundle continues if `allowPartialVendors=true`. |
|
||||
|
||||
## Manifest baseline
|
||||
- Generate `provider-manifest.json` with sorted provider entries. Fields per provider: `{id, source, snapshotDate, sha256, signaturePath, optional}`.
|
||||
- Store DSSE envelope for `provider-manifest.json` at `signatures/provider-manifest.dsse` (cosign/KMS).
|
||||
- Include provider digests in `manifests/provenance.json` materials array with URI `risk-provider://<id>/<snapshotDate>`.
|
||||
|
||||
## Signing baseline
|
||||
- Use Export Center signing path (cosign + Authority KMS) for:
|
||||
- `provider-manifest.json` (DSSE)
|
||||
- Aggregated `risk-bundle.tar.*` (detached signature `risk-bundle.sig`)
|
||||
- Vendor-provided signatures (when present) are preserved inside `providers/<id>/` and referenced from `provider-manifest.json`.
|
||||
- Rekor publishing remains optional; default **off** for offline kits (`rekor_publish=false`).
|
||||
|
||||
## Validation rules (bundle build)
|
||||
- Fail build if any mandatory provider (currently `cisa-kev`) is missing or hash mismatch.
|
||||
- Warn (non-fatal) when optional providers are stale beyond cadence unless `allowStale=true`.
|
||||
- Deterministic ordering: providers sorted by `id`; files sorted lexicographically inside bundle.
|
||||
- Record bundle-level inputs hash combining provider SHA-256 values (stable ordering) and include in provenance `materials[]`.
|
||||
|
||||
## Verification workflow alignment
|
||||
- CLI `stella risk bundle verify` must validate:
|
||||
- DSSE on `provider-manifest.json`
|
||||
- Hash match for each provider snapshot
|
||||
- Presence (or allowed absence) per `optional` flag
|
||||
- Detached signature on bundle archive (cosign/KMS)
|
||||
- Offline verification uses bundled public key (`signatures/pubkeys/<tenant>.pem`).
|
||||
|
||||
## Next steps / TODOs
|
||||
- Add test fixtures: minimal provider snapshots (kev+epss) with fixed hashes for deterministic regression tests.
|
||||
- Update ExportCenter worker to emit `provider-manifest.json` and DSSE using existing signing pipeline.
|
||||
- Extend CLI verify command to surface per-provider status (missing/stale/hash mismatch) and exit non-zero on mandatory failures.
|
||||
@@ -21,6 +21,7 @@ Related documentation:
|
||||
- `docs/modules/export-center/api.md`
|
||||
- `docs/modules/export-center/cli.md`
|
||||
- `docs/modules/export-center/operations/kms-envelope-pattern.md`
|
||||
- `docs/modules/export-center/operations/risk-bundle-provider-matrix.md`
|
||||
|
||||
## 2. Contacts & tooling
|
||||
|
||||
|
||||
Reference in New Issue
Block a user