feat: Add Scanner CI runner and related artifacts
- Implemented `run-scanner-ci.sh` to build and run tests for the Scanner solution with a warmed NuGet cache. - Created `excititor-vex-traces.json` dashboard for monitoring Excititor VEX observations. - Added Docker Compose configuration for the OTLP span sink in `docker-compose.spansink.yml`. - Configured OpenTelemetry collector in `otel-spansink.yaml` to receive and process traces. - Developed `run-spansink.sh` script to run the OTLP span sink for Excititor traces. - Introduced `FileSystemRiskBundleObjectStore` for storing risk bundle artifacts in the filesystem. - Built `RiskBundleBuilder` for creating risk bundles with associated metadata and providers. - Established `RiskBundleJob` to execute the risk bundle creation and storage process. - Defined models for risk bundle inputs, entries, and manifests in `RiskBundleModels.cs`. - Implemented signing functionality for risk bundle manifests with `HmacRiskBundleManifestSigner`. - Created unit tests for `RiskBundleBuilder`, `RiskBundleJob`, and signing functionality to ensure correctness. - Added filesystem artifact reader tests to validate manifest parsing and artifact listing. - Included test manifests for egress scenarios in the task runner tests. - Developed timeline query service tests to verify tenant and event ID handling.
This commit is contained in:
StellaOps Bot
@@ -8,6 +8,7 @@ Export Center packages reproducible evidence bundles (JSON, Trivy DB, mirror) wi
|
||||
- [Architecture](./architecture.md)
|
||||
- [Implementation plan](./implementation_plan.md)
|
||||
- [Task board](./TASKS.md)
|
||||
- [Observability runbook](./operations/observability.md) (offline import friendly)
|
||||
|
||||
## How to get started
|
||||
1. Open sprint file `/docs/implplan/SPRINT_*.md` and locate the stories referencing this module.
|
||||
|
||||
@@ -2,11 +2,16 @@
|
||||
|
||||
Export Center packages reproducible evidence bundles (JSON, Trivy DB, mirror) with provenance metadata and optional signing for offline or mirrored deployments.
|
||||
|
||||
## Responsibilities
|
||||
- Coordinate export jobs based on profiles and scope selectors.
|
||||
- Assemble manifests, provenance documents, and cosign signatures.
|
||||
- Stream bundles via HTTP/OCI and stage them for Offline Kit uses.
|
||||
- Expose CLI/API surfaces for automation.
|
||||
## Latest updates (2025-11-30)
|
||||
- Sprint tracker `docs/implplan/SPRINT_0320_0001_0001_docs_modules_export_center.md` and module `TASKS.md` added to mirror status.
|
||||
- Observability runbook stub + dashboard placeholder added under `operations/` (offline import).
|
||||
- Bundle/profile/offline manifest guidance reaffirmed (`devportal-offline*.md`, `mirror-bundles.md`, `provenance-and-signing.md`).
|
||||
|
||||
## Responsibilities
|
||||
- Coordinate export jobs based on profiles and scope selectors.
|
||||
- Assemble manifests, provenance documents, and cosign signatures.
|
||||
- Stream bundles via HTTP/OCI and stage them for Offline Kit uses.
|
||||
- Expose CLI/API surfaces for automation.
|
||||
|
||||
## Key components
|
||||
- `StellaOps.ExportCenter.WebService` planner.
|
||||
@@ -24,10 +29,11 @@ Export Center packages reproducible evidence bundles (JSON, Trivy DB, mirror) wi
|
||||
- Signer/Attestor for provenance signing.
|
||||
- CLI for operator-managed exports.
|
||||
|
||||
## Operational notes
|
||||
- Runbooks in ./operations/ for deployment and monitoring.
|
||||
- Mirror bundle instructions and validation notes.
|
||||
- Telemetry dashboards for export latency and retry rates.
|
||||
## Operational notes
|
||||
- Runbooks in ./operations/ for deployment and monitoring.
|
||||
- Observability assets: `operations/observability.md` and `operations/dashboards/export-center-observability.json` (offline import).
|
||||
- Mirror bundle instructions and validation notes.
|
||||
- Telemetry dashboards for export latency and retry rates.
|
||||
|
||||
## Related resources
|
||||
- ./operations/runbook.md
|
||||
|
||||
9
docs/modules/export-center/TASKS.md
Normal file
9
docs/modules/export-center/TASKS.md
Normal file
@@ -0,0 +1,9 @@
|
||||
# Export Center · TASKS (status mirror)
|
||||
|
||||
| Task ID | Status | Owner(s) | Notes / Evidence |
|
||||
| --- | --- | --- | --- |
|
||||
| EXPORT CENTER-DOCS-0001 | DONE (2025-11-30) | Docs Guild | README/architecture/implementation_plan refreshed; bundle/profiles/offline guidance linked; sprint references added. |
|
||||
| EXPORT CENTER-ENG-0001 | DONE (2025-11-30) | Module Team | TASKS board created; statuses mirrored with `docs/implplan/SPRINT_0320_0001_0001_docs_modules_export_center.md`. |
|
||||
| EXPORT CENTER-OPS-0001 | DONE (2025-11-30) | Ops Guild | Observability runbook stub + Grafana placeholder added; devportal/offline manifest links verified. |
|
||||
|
||||
> Keep this table in lockstep with the sprint Delivery Tracker (TODO/DOING/DONE/BLOCKED updates go to both files).
|
||||
@@ -101,11 +101,11 @@ Adapters expose structured telemetry events (`adapter.start`, `adapter.chunk`, `
|
||||
- **Object storage.** Writes to tenant-prefixed paths (`s3://stella-exports/{tenant}/{run-id}/...`) with immutable retention policies. Retention scheduler purges expired runs based on profile configuration.
|
||||
- **Offline Kit seeding.** Mirror bundles optionally staged into Offline Kit assembly pipelines, inheriting the same manifests and signatures.
|
||||
|
||||
## Observability
|
||||
- **Metrics.** Emits `exporter_run_duration_seconds`, `exporter_run_bytes_total{profile}`, `exporter_run_failures_total{error_code}`, `exporter_active_runs{tenant}`, `exporter_distribution_push_seconds{type}`.
|
||||
- **Logs.** Structured logs with fields `run_id`, `tenant`, `profile_kind`, `adapter`, `phase`, `correlation_id`, `error_code`. Phases include `plan`, `resolve`, `adapter`, `manifest`, `sign`, `distribute`.
|
||||
- **Traces.** Optional OpenTelemetry spans (`export.plan`, `export.fetch`, `export.write`, `export.sign`, `export.distribute`) for cross-service correlation.
|
||||
- **Dashboards & alerts.** DevOps pipeline seeds Grafana dashboards summarising throughput, size, failure ratios, and distribution latency. Alert thresholds: failure rate >5% per profile, median run duration >p95 baseline, signature verification failures >0.
|
||||
## Observability
|
||||
- **Metrics.** Emits `exporter_run_duration_seconds`, `exporter_run_bytes_total{profile}`, `exporter_run_failures_total{error_code}`, `exporter_active_runs{tenant}`, `exporter_distribution_push_seconds{type}`.
|
||||
- **Logs.** Structured logs with fields `run_id`, `tenant`, `profile_kind`, `adapter`, `phase`, `correlation_id`, `error_code`. Phases include `plan`, `resolve`, `adapter`, `manifest`, `sign`, `distribute`.
|
||||
- **Traces.** Optional OpenTelemetry spans (`export.plan`, `export.fetch`, `export.write`, `export.sign`, `export.distribute`) for cross-service correlation.
|
||||
- **Dashboards & alerts.** DevOps pipeline seeds Grafana dashboards summarising throughput, size, failure ratios, and distribution latency. Alert thresholds: failure rate >5% per profile, median run duration >p95 baseline, signature verification failures >0. Runbook + dashboard stub for offline import: `operations/observability.md`, `operations/dashboards/export-center-observability.json`.
|
||||
|
||||
## Security posture
|
||||
- Tenant claim enforced at every query and distribution path; cross-tenant selectors rejected unless explicit cross-tenant mirror feature toggled with signed approval.
|
||||
|
||||
@@ -58,9 +58,14 @@
|
||||
- **Security:** tenant fuzzing, RBAC coverage, redaction/PII filters, key rotation.
|
||||
- **Performance & chaos:** stress exports with large datasets, simulate worker/API failures mid-run, confirm deterministic recovery.
|
||||
|
||||
## Definition of done
|
||||
- Service, worker, and adapters deployed with telemetry & alerting.
|
||||
- CLI & Console workflows published, Offline Kit instructions updated.
|
||||
- Documentation set listed above refreshed; imposed rule statements appended where required.
|
||||
- CI pipelines include schema validation, profile verification, and determinism checks.
|
||||
- ./TASKS.md + ../../TASKS.md reflect current status for in-flight stories.
|
||||
## Definition of done
|
||||
- Service, worker, and adapters deployed with telemetry & alerting.
|
||||
- CLI & Console workflows published, Offline Kit instructions updated.
|
||||
- Documentation set listed above refreshed; imposed rule statements appended where required.
|
||||
- CI pipelines include schema validation, profile verification, and determinism checks.
|
||||
- ./TASKS.md + ../../TASKS.md reflect current status for in-flight stories.
|
||||
|
||||
## Sprint alignment (2025-11-30)
|
||||
- Docs sprint: `docs/implplan/SPRINT_0320_0001_0001_docs_modules_export_center.md`; statuses mirrored in `docs/modules/export-center/TASKS.md`.
|
||||
- Observability evidence stub lives in `operations/observability.md` with Grafana placeholder under `operations/dashboards/`.
|
||||
- Bundle/profile/offline manifest guidance maintained in `devportal-offline*.md`, `mirror-bundles.md`, and `provenance-and-signing.md`; update sprint/TASKS if these change.
|
||||
|
||||
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"_note": "Placeholder Grafana dashboard stub for Export Center. Replace panels when metrics endpoints are available; keep offline-import friendly.",
|
||||
"schemaVersion": 39,
|
||||
"title": "Export Center Observability (stub)",
|
||||
"panels": []
|
||||
}
|
||||
37
docs/modules/export-center/operations/observability.md
Normal file
37
docs/modules/export-center/operations/observability.md
Normal file
@@ -0,0 +1,37 @@
|
||||
# Export Center observability runbook (stub · 2025-11-29 demo)
|
||||
|
||||
## Dashboards (offline import)
|
||||
- Grafana JSON: `docs/modules/export-center/operations/dashboards/export-center-observability.json` (import locally; no external data sources assumed).
|
||||
- Planned panels: export job duration p95/p99, bundle size histogram, registry push latency, provenance/attestation verification failures, queue depth, and error rate per profile.
|
||||
|
||||
## Key metrics
|
||||
- `export_job_duration_seconds_bucket{profile}` — export duration by profile.
|
||||
- `export_bundle_size_bytes_bucket{profile}` — bundle size distribution.
|
||||
- `export_registry_push_latency_seconds_bucket{profile}` — registry push latency.
|
||||
- `export_attestation_failures_total{reason}` — DSSE/provenance verification failures.
|
||||
- `export_queue_depth` — pending export jobs.
|
||||
- `export_manifest_publish_total{result}` — manifest publish successes/failures.
|
||||
|
||||
## Logs & traces
|
||||
- Correlate by `exportId`, `profile`, `tenant`; include `bundleDigest`, `attestationStatus`, `registry`. Traces disabled by default; enable OTLP to on-prem collector when permitted.
|
||||
|
||||
## Health/diagnostics
|
||||
- `/health/liveness` and `/health/readiness` (export service) check storage, registry reachability, and attestation verification path.
|
||||
- `/status` exposes build version, commit, feature flags; verify against offline bundle manifest.
|
||||
- Verification probe: `stella export bundle verify --manifest <path>` once bundle available; validate hashes against manifest.
|
||||
|
||||
## Alert hints
|
||||
- Export job duration p99 > target SLA per profile.
|
||||
- Attestation verification failures > 0 over 10m.
|
||||
- Registry push latency spikes or error rate > threshold.
|
||||
- Queue depth growth without completion.
|
||||
|
||||
## Offline verification steps
|
||||
1) Import Grafana JSON locally; point to Prometheus scrape labeled `export-center`.
|
||||
2) Run `stella export bundle --profile <profile> --manifest out/manifest.json` and verify hashes via `jq -r '.files[].sha256'` against generated bundles.
|
||||
3) Fetch `/status` and compare commit/version to offline bundle manifest.
|
||||
|
||||
## Evidence locations
|
||||
- Sprint tracker: `docs/implplan/SPRINT_0320_0001_0001_docs_modules_export_center.md`.
|
||||
- Module docs: `README.md`, `architecture.md`, `implementation_plan.md`.
|
||||
- Dashboard stub: `operations/dashboards/export-center-observability.json`.
|
||||
@@ -0,0 +1,42 @@
|
||||
# Risk Bundle Provider Matrix & Signing Baseline
|
||||
|
||||
Status: Baseline for Sprint 0164-0001-0001 (RISK-BUNDLE-69/70 chain)
|
||||
|
||||
## Provider catalog (deterministic ordering)
|
||||
| Provider ID | Source feed (offline-ready) | Coverage | Refresh cadence | Signing / integrity | Notes |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| cisa-kev | CISA Known Exploited Vulnerabilities JSON | Exploited CVEs with required/known exploited flag | Daily | DSSE signature using ExportCenter signing key; feed hash recorded in `provider-manifest.json` | Mandatory; fails bundle if feed missing or hash mismatch. |
|
||||
| first-epss | FIRST EPSS CSV snapshot | Probability scores per CVE | Daily | DSSE signature; SHA-256 of snapshot stored in manifest | Optional; omit if snapshot stale >48h unless `allowStale=true`. |
|
||||
| osv | OpenSSF OSV bulk JSON (per-ecosystem shards) | OSS advisories with affected package ranges | Weekly | DSSE signature; per-shard SHA-256 list | Included only when `includeOsv=true` in job options to keep bundle size constrained. |
|
||||
| vendor-csaf | CSAF vendor advisories (Red Hat, SUSE, Debian) mirrored via Offline Kit | Vendor-specific CVEs, remediations | Weekly | Detached signature per CSAF document (vendor-provided where available) plus bundle-level DSSE manifest | Requires Offline Kit mirror; missing vendor feeds logged but bundle continues if `allowPartialVendors=true`. |
|
||||
|
||||
## Manifest baseline
|
||||
- Generate `provider-manifest.json` with sorted provider entries. Fields per provider: `{id, source, snapshotDate, sha256, signaturePath, optional}`.
|
||||
- Store DSSE envelope for `provider-manifest.json` at `signatures/provider-manifest.dsse` (cosign/KMS).
|
||||
- Include provider digests in `manifests/provenance.json` materials array with URI `risk-provider://<id>/<snapshotDate>`.
|
||||
|
||||
## Signing baseline
|
||||
- Use Export Center signing path (cosign + Authority KMS) for:
|
||||
- `provider-manifest.json` (DSSE)
|
||||
- Aggregated `risk-bundle.tar.*` (detached signature `risk-bundle.sig`)
|
||||
- Vendor-provided signatures (when present) are preserved inside `providers/<id>/` and referenced from `provider-manifest.json`.
|
||||
- Rekor publishing remains optional; default **off** for offline kits (`rekor_publish=false`).
|
||||
|
||||
## Validation rules (bundle build)
|
||||
- Fail build if any mandatory provider (currently `cisa-kev`) is missing or hash mismatch.
|
||||
- Warn (non-fatal) when optional providers are stale beyond cadence unless `allowStale=true`.
|
||||
- Deterministic ordering: providers sorted by `id`; files sorted lexicographically inside bundle.
|
||||
- Record bundle-level inputs hash combining provider SHA-256 values (stable ordering) and include in provenance `materials[]`.
|
||||
|
||||
## Verification workflow alignment
|
||||
- CLI `stella risk bundle verify` must validate:
|
||||
- DSSE on `provider-manifest.json`
|
||||
- Hash match for each provider snapshot
|
||||
- Presence (or allowed absence) per `optional` flag
|
||||
- Detached signature on bundle archive (cosign/KMS)
|
||||
- Offline verification uses bundled public key (`signatures/pubkeys/<tenant>.pem`).
|
||||
|
||||
## Next steps / TODOs
|
||||
- Add test fixtures: minimal provider snapshots (kev+epss) with fixed hashes for deterministic regression tests.
|
||||
- Update ExportCenter worker to emit `provider-manifest.json` and DSSE using existing signing pipeline.
|
||||
- Extend CLI verify command to surface per-provider status (missing/stale/hash mismatch) and exit non-zero on mandatory failures.
|
||||
@@ -21,6 +21,7 @@ Related documentation:
|
||||
- `docs/modules/export-center/api.md`
|
||||
- `docs/modules/export-center/cli.md`
|
||||
- `docs/modules/export-center/operations/kms-envelope-pattern.md`
|
||||
- `docs/modules/export-center/operations/risk-bundle-provider-matrix.md`
|
||||
|
||||
## 2. Contacts & tooling
|
||||
|
||||
|
||||
Reference in New Issue
Block a user