feat: Add Scanner CI runner and related artifacts
Some checks failed
Docs CI / lint-and-preview (push) Has been cancelled
Airgap Sealed CI Smoke / sealed-smoke (push) Has been cancelled
Export Center CI / export-ci (push) Has been cancelled
Signals CI & Image / signals-ci (push) Has been cancelled

- Implemented `run-scanner-ci.sh` to build and run tests for the Scanner solution with a warmed NuGet cache.
- Created `excititor-vex-traces.json` dashboard for monitoring Excititor VEX observations.
- Added Docker Compose configuration for the OTLP span sink in `docker-compose.spansink.yml`.
- Configured OpenTelemetry collector in `otel-spansink.yaml` to receive and process traces.
- Developed `run-spansink.sh` script to run the OTLP span sink for Excititor traces.
- Introduced `FileSystemRiskBundleObjectStore` for storing risk bundle artifacts in the filesystem.
- Built `RiskBundleBuilder` for creating risk bundles with associated metadata and providers.
- Established `RiskBundleJob` to execute the risk bundle creation and storage process.
- Defined models for risk bundle inputs, entries, and manifests in `RiskBundleModels.cs`.
- Implemented signing functionality for risk bundle manifests with `HmacRiskBundleManifestSigner`.
- Created unit tests for `RiskBundleBuilder`, `RiskBundleJob`, and signing functionality to ensure correctness.
- Added filesystem artifact reader tests to validate manifest parsing and artifact listing.
- Included test manifests for egress scenarios in the task runner tests.
- Developed timeline query service tests to verify tenant and event ID handling.
This commit is contained in:
StellaOps Bot
2025-11-30 19:12:35 +02:00
parent 17d45a6d30
commit 71e9a56cfd
92 changed files with 2596 additions and 387 deletions

View File

@@ -8,6 +8,7 @@ Attestor moves signed evidence through the trust chain by accepting DSSE bundles
- [Architecture](./architecture.md)
- [Implementation plan](./implementation_plan.md)
- [Task board](./TASKS.md)
- [Observability runbook](./operations/observability.md) (offline import friendly)
## How to get started
1. Open sprint file `/docs/implplan/SPRINT_*.md` and locate the stories referencing this module.

View File

@@ -477,11 +477,11 @@ sequenceDiagram
---
## 11) Failure modes & responses
| Condition | Return | Details | | |
| ------------------------------------- | ----------------------- | --------------------------------------------------------- | -------- | ------------ |
| mTLS/OpTok invalid | `401 invalid_token` | Include `WWW-Authenticate` DPoP challenge when applicable | | |
## 11) Failure modes & responses
| Condition | Return | Details | | |
| ------------------------------------- | ----------------------- | --------------------------------------------------------- | -------- | ------------ |
| mTLS/OpTok invalid | `401 invalid_token` | Include `WWW-Authenticate` DPoP challenge when applicable | | |
| Bundle not signed by trusted identity | `403 chain_untrusted` | DSSE accepted only from Signer identities | | |
| Duplicate bundle | `409 duplicate_bundle` | Return existing `uuid` (idempotent) | | |
| Rekor unreachable/timeout | `502 rekor_unavailable` | Retry with backoff; surface `Retry-After` | | |
@@ -529,5 +529,14 @@ sequenceDiagram
* **Duallog** write (primary + mirror) and **crosslog proof** packaging.
* **Cloud endorsement**: send `{uuid, artifactSha256}` to StellaOps cloud; store returned endorsement id for marketing/chainofcustody.
* **Checkpoint pinning**: periodically pin latest Rekor checkpoints to an external audit store for independent monitoring.
* **Checkpoint pinning**: periodically pin latest Rekor checkpoints to an external audit store for independent monitoring.
---
## 16) Observability (stub)
- Runbook + dashboard placeholder for offline import: `operations/observability.md`, `operations/dashboards/attestor-observability.json`.
- Metrics to surface: signing latency p95/p99, verification failure rate, transparency log submission lag, key rotation age, queue backlog, attestation bundle size histogram.
- Health endpoints: `/health/liveness`, `/health/readiness`, `/status`; verification probe `/api/attestations/verify` once demo bundle is available (see runbook).
- Alert hints: signing latency > 1s p99, verification failure spikes, tlog submission lag >10s, key rotation age over policy threshold, backlog above configured threshold.

View File

@@ -67,11 +67,16 @@
- **Performance:** throughput benchmarks, cache hit-rate monitoring, large batch verification.
- **Chaos:** inject Rekor outages, network failures, corrupt bundles; ensure graceful degradation and auditable alerts.
## Definition of done
- Phased milestones delivered with telemetry, documentation, and runbooks in place.
- CLI/Console parity verified; Offline Kit procedures validated in sealed environment.
- Cross-module dependencies acknowledged in ./TASKS.md and ../../TASKS.md.
- Documentation set refreshed (overview, architecture, key management, transparency, CLI/UI) with imposed rule statement.
## Definition of done
- Phased milestones delivered with telemetry, documentation, and runbooks in place.
- CLI/Console parity verified; Offline Kit procedures validated in sealed environment.
- Cross-module dependencies acknowledged in ./TASKS.md and ../../TASKS.md.
- Documentation set refreshed (overview, architecture, key management, transparency, CLI/UI) with imposed rule statement.
## Sprint alignment (2025-11-30)
- Docs sprint: `docs/implplan/SPRINT_0313_0001_0001_docs_modules_attestor.md`; statuses mirrored in `docs/modules/attestor/TASKS.md`.
- Observability evidence stub lives in `operations/observability.md` with Grafana placeholder under `operations/dashboards/`; finalize after next demo outputs.
- ATTESTOR-OPS-0001 remains BLOCKED until next demo provides observability data; update sprint/TASKS when available.
---

View File

@@ -8,6 +8,7 @@ Export Center packages reproducible evidence bundles (JSON, Trivy DB, mirror) wi
- [Architecture](./architecture.md)
- [Implementation plan](./implementation_plan.md)
- [Task board](./TASKS.md)
- [Observability runbook](./operations/observability.md) (offline import friendly)
## How to get started
1. Open sprint file `/docs/implplan/SPRINT_*.md` and locate the stories referencing this module.

View File

@@ -2,11 +2,16 @@
Export Center packages reproducible evidence bundles (JSON, Trivy DB, mirror) with provenance metadata and optional signing for offline or mirrored deployments.
## Responsibilities
- Coordinate export jobs based on profiles and scope selectors.
- Assemble manifests, provenance documents, and cosign signatures.
- Stream bundles via HTTP/OCI and stage them for Offline Kit uses.
- Expose CLI/API surfaces for automation.
## Latest updates (2025-11-30)
- Sprint tracker `docs/implplan/SPRINT_0320_0001_0001_docs_modules_export_center.md` and module `TASKS.md` added to mirror status.
- Observability runbook stub + dashboard placeholder added under `operations/` (offline import).
- Bundle/profile/offline manifest guidance reaffirmed (`devportal-offline*.md`, `mirror-bundles.md`, `provenance-and-signing.md`).
## Responsibilities
- Coordinate export jobs based on profiles and scope selectors.
- Assemble manifests, provenance documents, and cosign signatures.
- Stream bundles via HTTP/OCI and stage them for Offline Kit uses.
- Expose CLI/API surfaces for automation.
## Key components
- `StellaOps.ExportCenter.WebService` planner.
@@ -24,10 +29,11 @@ Export Center packages reproducible evidence bundles (JSON, Trivy DB, mirror) wi
- Signer/Attestor for provenance signing.
- CLI for operator-managed exports.
## Operational notes
- Runbooks in ./operations/ for deployment and monitoring.
- Mirror bundle instructions and validation notes.
- Telemetry dashboards for export latency and retry rates.
## Operational notes
- Runbooks in ./operations/ for deployment and monitoring.
- Observability assets: `operations/observability.md` and `operations/dashboards/export-center-observability.json` (offline import).
- Mirror bundle instructions and validation notes.
- Telemetry dashboards for export latency and retry rates.
## Related resources
- ./operations/runbook.md

View File

@@ -0,0 +1,9 @@
# Export Center · TASKS (status mirror)
| Task ID | Status | Owner(s) | Notes / Evidence |
| --- | --- | --- | --- |
| EXPORT CENTER-DOCS-0001 | DONE (2025-11-30) | Docs Guild | README/architecture/implementation_plan refreshed; bundle/profiles/offline guidance linked; sprint references added. |
| EXPORT CENTER-ENG-0001 | DONE (2025-11-30) | Module Team | TASKS board created; statuses mirrored with `docs/implplan/SPRINT_0320_0001_0001_docs_modules_export_center.md`. |
| EXPORT CENTER-OPS-0001 | DONE (2025-11-30) | Ops Guild | Observability runbook stub + Grafana placeholder added; devportal/offline manifest links verified. |
> Keep this table in lockstep with the sprint Delivery Tracker (TODO/DOING/DONE/BLOCKED updates go to both files).

View File

@@ -101,11 +101,11 @@ Adapters expose structured telemetry events (`adapter.start`, `adapter.chunk`, `
- **Object storage.** Writes to tenant-prefixed paths (`s3://stella-exports/{tenant}/{run-id}/...`) with immutable retention policies. Retention scheduler purges expired runs based on profile configuration.
- **Offline Kit seeding.** Mirror bundles optionally staged into Offline Kit assembly pipelines, inheriting the same manifests and signatures.
## Observability
- **Metrics.** Emits `exporter_run_duration_seconds`, `exporter_run_bytes_total{profile}`, `exporter_run_failures_total{error_code}`, `exporter_active_runs{tenant}`, `exporter_distribution_push_seconds{type}`.
- **Logs.** Structured logs with fields `run_id`, `tenant`, `profile_kind`, `adapter`, `phase`, `correlation_id`, `error_code`. Phases include `plan`, `resolve`, `adapter`, `manifest`, `sign`, `distribute`.
- **Traces.** Optional OpenTelemetry spans (`export.plan`, `export.fetch`, `export.write`, `export.sign`, `export.distribute`) for cross-service correlation.
- **Dashboards & alerts.** DevOps pipeline seeds Grafana dashboards summarising throughput, size, failure ratios, and distribution latency. Alert thresholds: failure rate >5% per profile, median run duration >p95 baseline, signature verification failures >0.
## Observability
- **Metrics.** Emits `exporter_run_duration_seconds`, `exporter_run_bytes_total{profile}`, `exporter_run_failures_total{error_code}`, `exporter_active_runs{tenant}`, `exporter_distribution_push_seconds{type}`.
- **Logs.** Structured logs with fields `run_id`, `tenant`, `profile_kind`, `adapter`, `phase`, `correlation_id`, `error_code`. Phases include `plan`, `resolve`, `adapter`, `manifest`, `sign`, `distribute`.
- **Traces.** Optional OpenTelemetry spans (`export.plan`, `export.fetch`, `export.write`, `export.sign`, `export.distribute`) for cross-service correlation.
- **Dashboards & alerts.** DevOps pipeline seeds Grafana dashboards summarising throughput, size, failure ratios, and distribution latency. Alert thresholds: failure rate >5% per profile, median run duration >p95 baseline, signature verification failures >0. Runbook + dashboard stub for offline import: `operations/observability.md`, `operations/dashboards/export-center-observability.json`.
## Security posture
- Tenant claim enforced at every query and distribution path; cross-tenant selectors rejected unless explicit cross-tenant mirror feature toggled with signed approval.

View File

@@ -58,9 +58,14 @@
- **Security:** tenant fuzzing, RBAC coverage, redaction/PII filters, key rotation.
- **Performance & chaos:** stress exports with large datasets, simulate worker/API failures mid-run, confirm deterministic recovery.
## Definition of done
- Service, worker, and adapters deployed with telemetry & alerting.
- CLI & Console workflows published, Offline Kit instructions updated.
- Documentation set listed above refreshed; imposed rule statements appended where required.
- CI pipelines include schema validation, profile verification, and determinism checks.
- ./TASKS.md + ../../TASKS.md reflect current status for in-flight stories.
## Definition of done
- Service, worker, and adapters deployed with telemetry & alerting.
- CLI & Console workflows published, Offline Kit instructions updated.
- Documentation set listed above refreshed; imposed rule statements appended where required.
- CI pipelines include schema validation, profile verification, and determinism checks.
- ./TASKS.md + ../../TASKS.md reflect current status for in-flight stories.
## Sprint alignment (2025-11-30)
- Docs sprint: `docs/implplan/SPRINT_0320_0001_0001_docs_modules_export_center.md`; statuses mirrored in `docs/modules/export-center/TASKS.md`.
- Observability evidence stub lives in `operations/observability.md` with Grafana placeholder under `operations/dashboards/`.
- Bundle/profile/offline manifest guidance maintained in `devportal-offline*.md`, `mirror-bundles.md`, and `provenance-and-signing.md`; update sprint/TASKS if these change.

View File

@@ -0,0 +1,6 @@
{
"_note": "Placeholder Grafana dashboard stub for Export Center. Replace panels when metrics endpoints are available; keep offline-import friendly.",
"schemaVersion": 39,
"title": "Export Center Observability (stub)",
"panels": []
}

View File

@@ -0,0 +1,37 @@
# Export Center observability runbook (stub · 2025-11-29 demo)
## Dashboards (offline import)
- Grafana JSON: `docs/modules/export-center/operations/dashboards/export-center-observability.json` (import locally; no external data sources assumed).
- Planned panels: export job duration p95/p99, bundle size histogram, registry push latency, provenance/attestation verification failures, queue depth, and error rate per profile.
## Key metrics
- `export_job_duration_seconds_bucket{profile}` — export duration by profile.
- `export_bundle_size_bytes_bucket{profile}` — bundle size distribution.
- `export_registry_push_latency_seconds_bucket{profile}` — registry push latency.
- `export_attestation_failures_total{reason}` — DSSE/provenance verification failures.
- `export_queue_depth` — pending export jobs.
- `export_manifest_publish_total{result}` — manifest publish successes/failures.
## Logs & traces
- Correlate by `exportId`, `profile`, `tenant`; include `bundleDigest`, `attestationStatus`, `registry`. Traces disabled by default; enable OTLP to on-prem collector when permitted.
## Health/diagnostics
- `/health/liveness` and `/health/readiness` (export service) check storage, registry reachability, and attestation verification path.
- `/status` exposes build version, commit, feature flags; verify against offline bundle manifest.
- Verification probe: `stella export bundle verify --manifest <path>` once bundle available; validate hashes against manifest.
## Alert hints
- Export job duration p99 > target SLA per profile.
- Attestation verification failures > 0 over 10m.
- Registry push latency spikes or error rate > threshold.
- Queue depth growth without completion.
## Offline verification steps
1) Import Grafana JSON locally; point to Prometheus scrape labeled `export-center`.
2) Run `stella export bundle --profile <profile> --manifest out/manifest.json` and verify hashes via `jq -r '.files[].sha256'` against generated bundles.
3) Fetch `/status` and compare commit/version to offline bundle manifest.
## Evidence locations
- Sprint tracker: `docs/implplan/SPRINT_0320_0001_0001_docs_modules_export_center.md`.
- Module docs: `README.md`, `architecture.md`, `implementation_plan.md`.
- Dashboard stub: `operations/dashboards/export-center-observability.json`.

View File

@@ -0,0 +1,42 @@
# Risk Bundle Provider Matrix & Signing Baseline
Status: Baseline for Sprint 0164-0001-0001 (RISK-BUNDLE-69/70 chain)
## Provider catalog (deterministic ordering)
| Provider ID | Source feed (offline-ready) | Coverage | Refresh cadence | Signing / integrity | Notes |
| --- | --- | --- | --- | --- | --- |
| cisa-kev | CISA Known Exploited Vulnerabilities JSON | Exploited CVEs with required/known exploited flag | Daily | DSSE signature using ExportCenter signing key; feed hash recorded in `provider-manifest.json` | Mandatory; fails bundle if feed missing or hash mismatch. |
| first-epss | FIRST EPSS CSV snapshot | Probability scores per CVE | Daily | DSSE signature; SHA-256 of snapshot stored in manifest | Optional; omit if snapshot stale >48h unless `allowStale=true`. |
| osv | OpenSSF OSV bulk JSON (per-ecosystem shards) | OSS advisories with affected package ranges | Weekly | DSSE signature; per-shard SHA-256 list | Included only when `includeOsv=true` in job options to keep bundle size constrained. |
| vendor-csaf | CSAF vendor advisories (Red Hat, SUSE, Debian) mirrored via Offline Kit | Vendor-specific CVEs, remediations | Weekly | Detached signature per CSAF document (vendor-provided where available) plus bundle-level DSSE manifest | Requires Offline Kit mirror; missing vendor feeds logged but bundle continues if `allowPartialVendors=true`. |
## Manifest baseline
- Generate `provider-manifest.json` with sorted provider entries. Fields per provider: `{id, source, snapshotDate, sha256, signaturePath, optional}`.
- Store DSSE envelope for `provider-manifest.json` at `signatures/provider-manifest.dsse` (cosign/KMS).
- Include provider digests in `manifests/provenance.json` materials array with URI `risk-provider://<id>/<snapshotDate>`.
## Signing baseline
- Use Export Center signing path (cosign + Authority KMS) for:
- `provider-manifest.json` (DSSE)
- Aggregated `risk-bundle.tar.*` (detached signature `risk-bundle.sig`)
- Vendor-provided signatures (when present) are preserved inside `providers/<id>/` and referenced from `provider-manifest.json`.
- Rekor publishing remains optional; default **off** for offline kits (`rekor_publish=false`).
## Validation rules (bundle build)
- Fail build if any mandatory provider (currently `cisa-kev`) is missing or hash mismatch.
- Warn (non-fatal) when optional providers are stale beyond cadence unless `allowStale=true`.
- Deterministic ordering: providers sorted by `id`; files sorted lexicographically inside bundle.
- Record bundle-level inputs hash combining provider SHA-256 values (stable ordering) and include in provenance `materials[]`.
## Verification workflow alignment
- CLI `stella risk bundle verify` must validate:
- DSSE on `provider-manifest.json`
- Hash match for each provider snapshot
- Presence (or allowed absence) per `optional` flag
- Detached signature on bundle archive (cosign/KMS)
- Offline verification uses bundled public key (`signatures/pubkeys/<tenant>.pem`).
## Next steps / TODOs
- Add test fixtures: minimal provider snapshots (kev+epss) with fixed hashes for deterministic regression tests.
- Update ExportCenter worker to emit `provider-manifest.json` and DSSE using existing signing pipeline.
- Extend CLI verify command to surface per-provider status (missing/stale/hash mismatch) and exit non-zero on mandatory failures.

View File

@@ -21,6 +21,7 @@ Related documentation:
- `docs/modules/export-center/api.md`
- `docs/modules/export-center/cli.md`
- `docs/modules/export-center/operations/kms-envelope-pattern.md`
- `docs/modules/export-center/operations/risk-bundle-provider-matrix.md`
## 2. Contacts & tooling

View File

@@ -8,6 +8,7 @@ Platform module describes cross-cutting architecture, contracts, and guardrails
- [Architecture](./architecture.md)
- [Implementation plan](./implementation_plan.md)
- [Task board](./TASKS.md)
- [Architecture overview](./architecture-overview.md)
## How to get started
1. Open sprint file `/docs/implplan/SPRINT_*.md` and locate the stories referencing this module.

View File

@@ -2,24 +2,31 @@
Platform module describes cross-cutting architecture, contracts, and guardrails that bind the services together.
## Responsibilities
- Maintain the system-wide architecture overview and integration diagrams.
- Capture Aggregation-Only Contract guidance and migration playbooks.
- Document shared services such as API gateway, tenancy, quotas, and offline posture.
- Coordinate platform-wide epics and compliance checklists.
## Latest updates (2025-11-30)
- Sprint tracker `docs/implplan/SPRINT_0324_0001_0001_docs_modules_platform.md` and module `TASKS.md` added to mirror status.
- README now points to architecture overview, AOC references, and offline guidance entry points.
- Platform module remains docs-only; no runtime services.
## Responsibilities
- Maintain the system-wide architecture overview and integration diagrams.
- Capture Aggregation-Only Contract guidance and migration playbooks.
- Document shared services such as API gateway, tenancy, quotas, and offline posture.
- Coordinate platform-wide epics and compliance checklists.
## Key components
- Architecture overview in ./architecture-overview.md.
- References to high-level docs (../../07_HIGH_LEVEL_ARCHITECTURE.md).
## Key components
- Architecture overview in `architecture-overview.md`.
- Platform architecture summary in `architecture.md`.
- High-level reference: `../../07_HIGH_LEVEL_ARCHITECTURE.md`.
## Integrations & dependencies
- All StellaOps services via shared contracts (AOC, telemetry, security).
- DevOps for release governance.
- Docs guild for cross-module onboarding.
## Operational notes
- No runtime component; focus is architectural governance.
- Glossaries and guardrails cross-linked across docs.
## Operational notes
- Docs-only module; focus is architectural governance and cross-module guardrails.
- Glossaries and guardrails cross-linked across docs; keep AOC references current.
- Status mirrors: sprint file and `docs/modules/platform/TASKS.md`.
## Backlog references
- DOCS-AOC-19-002/003 in ../../TASKS.md.

View File

@@ -0,0 +1,9 @@
# Platform · TASKS (status mirror)
| Task ID | Status | Owner(s) | Notes / Evidence |
| --- | --- | --- | --- |
| PLATFORM-DOCS-0001 | DONE (2025-11-30) | Docs Guild | README/architecture/implementation_plan refreshed; AOC/offline guardrails linked. |
| PLATFORM-ENG-0001 | DONE (2025-11-30) | Module Team | TASKS board created; statuses mirrored with `docs/implplan/SPRINT_0324_0001_0001_docs_modules_platform.md`. |
| PLATFORM-OPS-0001 | DONE (2025-11-30) | Ops Guild | Cross-links to architecture-overview and 07_HLA verified; offline guidance highlighted. |
> Keep this table in lockstep with the sprint Delivery Tracker (TODO/DOING/DONE/BLOCKED updates go to both files).

View File

@@ -1,7 +1,18 @@
# Platform architecture
> Cross-cutting view anchored in the Authority, Policy, Graph, Vulnerability Explorer, Orchestrator, Export Center, and Notifications module documentation set.
This placeholder summarises the planned architecture for Platform. Consolidate design details from implementation plans and upcoming epics before coding.
Refer to the module README and implementation plan for immediate context, and update this document once component boundaries and data flows are finalised.
# Platform architecture (summary)
This module aggregates cross-cutting contracts and guardrails that every StellaOps service must follow.
## Anchors
- High-level system view: `../../07_HIGH_LEVEL_ARCHITECTURE.md`
- Platform overview: `architecture-overview.md`
- Aggregation-Only Contract: `../ingestion/aggregation-only-contract.md` (referenced across ingestion/observability docs)
## Scope
- **Identity & tenancy**: Authority-issued OpToks, tenant scoping, RBAC, short TTLs; see Authority module docs.
- **AOC & provenance**: services ingest evidence without mutating/merging; provenance preserved; determinism required.
- **Offline posture**: Offline Kit parity, sealed-mode defaults, deterministic bundles.
- **Observability baseline**: metrics/logging/tracing patterns reused across modules; collectors documented under Telemetry module.
- **Determinism**: stable ordering, UTC timestamps, content-addressed artifacts, reproducible exports.
## Coordination
Platform docs are the starting point for new contributors; keep this summary in sync with module-specific dossiers and sprint references.

View File

@@ -16,7 +16,12 @@
- **Epics 611:** ensure cross-cutting contracts (Explorer, Lens, AI, Orchestrator, Notifications) stay aligned.
- Track additional platform updates in ../../TASKS.md and docs/implplan/SPRINTS.md.
## Coordination
- Review ./AGENTS.md before picking up new work.
- Sync with cross-cutting teams noted in `/docs/implplan/SPRINT_*.md`.
- Update this plan whenever scope, dependencies, or guardrails change.
## Coordination
- Review ./AGENTS.md before picking up new work.
- Sync with cross-cutting teams noted in `/docs/implplan/SPRINT_*.md`.
- Update this plan whenever scope, dependencies, or guardrails change.
## Sprint alignment (2025-11-30)
- Docs sprint: `docs/implplan/SPRINT_0324_0001_0001_docs_modules_platform.md`; statuses mirrored in `docs/modules/platform/TASKS.md`.
- Keep links to `architecture-overview.md` and `../../07_HIGH_LEVEL_ARCHITECTURE.md` current; update both sprint and TASKS if platform guardrails change.
- Platform is docs-only; ensure Offline Kit and AOC references remain discoverable from README/architecture.