diff --git a/SPRINTS.md b/SPRINTS.md index 06a60200..38ae9843 100644 --- a/SPRINTS.md +++ b/SPRINTS.md @@ -1,3 +1,5 @@ +This file describe implementation of Stella Ops (docs/README.md). Implementation must respect rules from AGENTS.md (read if you have not). + | Sprint | Theme | Tasks File Path | Status | Type of Specialist | Task ID | Task Description | | --- | --- | --- | --- | --- | --- | --- | | Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Models/TASKS.md | DONE (2025-10-12) | Team Models & Merge Leads | FEEDMODELS-SCHEMA-01-001 | SemVer primitive range-style metadata
Instructions to work:
DONE Read ./AGENTS.md and src/StellaOps.Concelier.Models/AGENTS.md. This task lays the groundwork—complete the SemVer helper updates before teammates pick up FEEDMODELS-SCHEMA-01-002/003 and FEEDMODELS-SCHEMA-02-900. Use ./src/FASTER_MODELING_AND_NORMALIZATION.md for the target rule structure. | @@ -138,19 +140,201 @@ | Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Connectors.MSRC.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – MSRC | EXCITITOR-CONN-MS-01-001 | Deliver AAD onboarding/token cache for MSRC CSAF ingestion. | | Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Connectors.Oracle.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – Oracle | EXCITITOR-CONN-ORACLE-01-001 | Implement Oracle CSAF catalogue discovery with CPU calendar awareness. | | Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Connectors.Ubuntu.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – Ubuntu | EXCITITOR-CONN-UBUNTU-01-001 | Implement Ubuntu CSAF discovery and channel selection for USN ingestion. | -| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/TASKS.md | TODO | Team Excititor Connectors – OCI | EXCITITOR-CONN-OCI-01-001 | Wire OCI discovery/auth to fetch OpenVEX attestations for configured images. | -| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Cli/TASKS.md | TODO | DevEx/CLI | EXCITITOR-CLI-01-001 | Add `excititor` CLI verbs bridging to WebService with consistent auth and offline UX. | +| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/TASKS.md | DONE (2025-10-18) | Team Excititor Connectors – OCI | EXCITITOR-CONN-OCI-01-001 | Wire OCI discovery/auth to fetch OpenVEX attestations for configured images. | +| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/TASKS.md | DONE (2025-10-18) | Team Excititor Connectors – OCI | EXCITITOR-CONN-OCI-01-002 | Attestation fetch & verify loop – download DSSE attestations, trigger verification, handle retries/backoff, persist raw statements. | +| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/TASKS.md | DONE (2025-10-18) | Team Excititor Connectors – OCI | EXCITITOR-CONN-OCI-01-003 | Provenance metadata & policy hooks – emit image, subject digest, issuer, and trust metadata for policy weighting/logging. | +| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Cli/TASKS.md | DONE (2025-10-18) | DevEx/CLI | EXCITITOR-CLI-01-001 | Add `excititor` CLI verbs bridging to WebService with consistent auth and offline UX. | | Sprint 7 | Contextual Truth Foundations | src/StellaOps.Excititor.Core/TASKS.md | TODO | Team Excititor Core & Policy | EXCITITOR-CORE-02-001 | Context signal schema prep – extend consensus models with severity/KEV/EPSS fields and update canonical serializers. | | Sprint 7 | Contextual Truth Foundations | src/StellaOps.Excititor.Policy/TASKS.md | TODO | Team Excititor Policy | EXCITITOR-POLICY-02-001 | Scoring coefficients & weight ceilings – add α/β options, weight boosts, and validation guidance. | | Sprint 7 | Contextual Truth Foundations | src/StellaOps.Excititor.Storage.Mongo/TASKS.md | TODO | Team Excititor Storage | EXCITITOR-STORAGE-02-001 | Statement events & scoring signals – create immutable VEX statement store plus consensus extensions with indexes/migrations. | | Sprint 7 | Contextual Truth Foundations | src/StellaOps.Excititor.WebService/TASKS.md | TODO | Team Excititor WebService | EXCITITOR-WEB-01-004 | Resolve API & signed responses – expose `/excititor/resolve`, return signed consensus/score envelopes, document auth. | +| Sprint 7 | Contextual Truth Foundations | src/StellaOps.Excititor.WebService/TASKS.md | TODO | Team Excititor WebService | EXCITITOR-WEB-01-005 | Mirror distribution endpoints – expose download APIs for downstream Excititor instances. | | Sprint 7 | Contextual Truth Foundations | src/StellaOps.Excititor.Attestation/TASKS.md | DONE (2025-10-16) | Team Excititor Attestation | EXCITITOR-ATTEST-01-002 | Rekor v2 client integration – ship transparency log client with retries and offline queue. | | Sprint 7 | Contextual Truth Foundations | src/StellaOps.Excititor.Worker/TASKS.md | TODO | Team Excititor Worker | EXCITITOR-WORKER-01-004 | TTL refresh & stability damper – schedule re-resolve loops and guard against status flapping. | | Sprint 7 | Contextual Truth Foundations | src/StellaOps.Excititor.Export/TASKS.md | TODO | Team Excititor Export | EXCITITOR-EXPORT-01-005 | Score & resolve envelope surfaces – include signed consensus/score artifacts in exports. | +| Sprint 7 | Contextual Truth Foundations | src/StellaOps.Excititor.Export/TASKS.md | TODO | Team Excititor Export | EXCITITOR-EXPORT-01-006 | Quiet provenance packaging – attach quieted-by statement IDs, signers, justification codes to exports and attestations. | +| Sprint 7 | Contextual Truth Foundations | src/StellaOps.Excititor.Export/TASKS.md | TODO | Team Excititor Export | EXCITITOR-EXPORT-01-007 | Mirror bundle + domain manifest – publish signed consensus bundles for mirrors. | +| Sprint 7 | Contextual Truth Foundations | src/StellaOps.Excititor.Connectors.StellaOpsMirror/TASKS.md | TODO | Excititor Connectors – Stella | EXCITITOR-CONN-STELLA-07-001 | Excititor mirror connector – ingest signed mirror bundles and map to VexClaims with resume handling. | | Sprint 7 | Contextual Truth Foundations | src/StellaOps.Concelier.Core/TASKS.md | TODO | Team Core Engine & Storage Analytics | FEEDCORE-ENGINE-07-001 | Advisory event log & asOf queries – surface immutable statements and replay capability. | | Sprint 7 | Contextual Truth Foundations | src/StellaOps.Concelier.Core/TASKS.md | TODO | Team Core Engine & Data Science | FEEDCORE-ENGINE-07-002 | Noise prior computation service – learn false-positive priors and expose deterministic summaries. | +| Sprint 7 | Contextual Truth Foundations | src/StellaOps.Concelier.Core/TASKS.md | TODO | Team Core Engine & Storage Analytics | FEEDCORE-ENGINE-07-003 | Unknown state ledger & confidence seeding – persist unknown flags, seed confidence bands, expose query surface. | | Sprint 7 | Contextual Truth Foundations | src/StellaOps.Concelier.Storage.Mongo/TASKS.md | TODO | Team Normalization & Storage Backbone | FEEDSTORAGE-DATA-07-001 | Advisory statement & conflict collections – provision Mongo schema/indexes for event-sourced merge. | | Sprint 7 | Contextual Truth Foundations | src/StellaOps.Concelier.Merge/TASKS.md | TODO | BE-Merge | FEEDMERGE-ENGINE-07-001 | Conflict sets & explainers – persist conflict materialization and replay hashes for merge decisions. | | Sprint 8 | Mongo strengthening | src/StellaOps.Concelier.Storage.Mongo/TASKS.md | TODO | Team Normalization & Storage Backbone | FEEDSTORAGE-MONGO-08-001 | Causal-consistent Concelier storage sessions
Ensure `AddMongoStorage` registers a scoped session facilitator (causal consistency + majority concerns), update repositories to accept optional session handles, and add integration coverage proving read-your-write and monotonic reads across a replica set/election scenario. | | Sprint 8 | Mongo strengthening | src/StellaOps.Authority/TASKS.md | TODO | Authority Core & Storage Guild | AUTHSTORAGE-MONGO-08-001 | Harden Authority Mongo usage
Introduce scoped MongoDB sessions with `writeConcern`/`readConcern` majority defaults, flow the session through stores used in mutations + follow-up reads, and document middleware pattern for web/API & GraphQL layers. | | Sprint 8 | Mongo strengthening | src/StellaOps.Excititor.Storage.Mongo/TASKS.md | TODO | Team Excititor Storage | EXCITITOR-STORAGE-MONGO-08-001 | Causal consistency for Excititor repositories
Register Mongo options with majority defaults, push session-aware overloads through raw/export/consensus/cache stores, and extend migration/tests to validate causal reads after writes (including GridFS-backed content) under replica-set failover. | +| Sprint 8 | Mirror Distribution | src/StellaOps.Concelier.Exporter.Json/TASKS.md | TODO | Concelier Export Guild | CONCELIER-EXPORT-08-201 | Mirror bundle + domain manifest – produce signed JSON aggregates for `*.stella-ops.org` mirrors. | +| Sprint 8 | Mirror Distribution | src/StellaOps.Concelier.Exporter.TrivyDb/TASKS.md | TODO | Concelier Export Guild | CONCELIER-EXPORT-08-202 | Mirror-ready Trivy DB bundles – ship domain-specific archives + metadata for downstream sync. | +| Sprint 8 | Mirror Distribution | src/StellaOps.Concelier.WebService/TASKS.md | TODO | Concelier WebService Guild | CONCELIER-WEB-08-201 | Mirror distribution endpoints – expose domain-scoped index/download APIs with auth/quota. | +| Sprint 8 | Mirror Distribution | src/StellaOps.Concelier.Connector.StellaOpsMirror/TASKS.md | TODO | BE-Conn-Stella | FEEDCONN-STELLA-08-001 | Concelier mirror connector – fetch mirror manifest, verify signatures, and hydrate canonical DTOs with resume support. | +| Sprint 8 | Mirror Distribution | ops/devops/TASKS.md | TODO | DevOps Guild | DEVOPS-MIRROR-08-001 | Managed mirror deployments for `*.stella-ops.org` – Helm/Compose overlays, CDN, runbooks. | +| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.Core/TASKS.md | TODO | Team Scanner Core | SCANNER-CORE-09-501 | Define shared DTOs (ScanJob, ProgressEvent), error taxonomy, and deterministic ID/timestamp helpers aligning with `ARCHITECTURE_SCANNER.md` §3–§4. | +| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.Core/TASKS.md | TODO | Team Scanner Core | SCANNER-CORE-09-502 | Observability helpers (correlation IDs, logging scopes, metric namespacing, deterministic hashes) consumed by WebService/Worker. | +| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.Core/TASKS.md | TODO | Team Scanner Core | SCANNER-CORE-09-503 | Security utilities: Authority client factory, OpTok caching, DPoP verifier, restart-time plug-in guardrails for scanner components. | +| Sprint 9 | Scanner Build-time | src/StellaOps.Scanner.Sbomer.BuildXPlugin/TASKS.md | TODO | BuildX Guild | SP9-BLDX-09-001 | Buildx driver scaffold + handshake with Scanner.Emit (local CAS). | +| Sprint 9 | Scanner Build-time | src/StellaOps.Scanner.Sbomer.BuildXPlugin/TASKS.md | TODO | BuildX Guild | SP9-BLDX-09-002 | OCI annotations + provenance hand-off to Attestor. | +| Sprint 9 | Scanner Build-time | src/StellaOps.Scanner.Sbomer.BuildXPlugin/TASKS.md | TODO | BuildX Guild | SP9-BLDX-09-003 | CI demo: minimal SBOM push & backend report wiring. | +| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.WebService/TASKS.md | TODO | Team Scanner WebService | SCANNER-WEB-09-101 | Minimal API host with Authority enforcement, health/ready endpoints, and restart-time plug-in loader per architecture §1, §4. | +| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.WebService/TASKS.md | TODO | Team Scanner WebService | SCANNER-WEB-09-102 | `/api/v1/scans` submission/status endpoints with deterministic IDs, validation, and cancellation support. | +| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.WebService/TASKS.md | TODO | Team Scanner WebService | SCANNER-WEB-09-103 | Progress streaming (SSE/JSONL) with correlation IDs and ISO-8601 UTC timestamps, documented in API reference. | +| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.WebService/TASKS.md | TODO | Team Scanner WebService | SCANNER-WEB-09-104 | Configuration binding for Mongo, MinIO, queue, feature flags; startup diagnostics and fail-fast policy. | +| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.WebService/TASKS.md | TODO | Team Scanner WebService | SCANNER-POLICY-09-105 | Policy snapshot loader + schema + OpenAPI (YAML ignore rules, VEX include/exclude, vendor precedence). | +| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.WebService/TASKS.md | TODO | Team Scanner WebService | SCANNER-POLICY-09-106 | `/reports` verdict assembly (Feedser+Vexer+Policy) + signed response envelope. | +| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.WebService/TASKS.md | TODO | Team Scanner WebService | SCANNER-POLICY-09-107 | Expose score inputs, config version, and quiet provenance in `/reports` JSON and signed payload. | +| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.Worker/TASKS.md | TODO | Team Scanner Worker | SCANNER-WORKER-09-201 | Worker host bootstrap with Authority auth, hosted services, and graceful shutdown semantics. | +| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.Worker/TASKS.md | TODO | Team Scanner Worker | SCANNER-WORKER-09-202 | Lease/heartbeat loop with retry+jitter, poison-job quarantine, structured logging. | +| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.Worker/TASKS.md | TODO | Team Scanner Worker | SCANNER-WORKER-09-203 | Analyzer dispatch skeleton emitting deterministic stage progress and honoring cancellation tokens. | +| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.Worker/TASKS.md | TODO | Team Scanner Worker | SCANNER-WORKER-09-204 | Worker metrics (queue latency, stage duration, failure counts) with OpenTelemetry resource wiring. | +| Sprint 9 | Policy Foundations | src/StellaOps.Policy/TASKS.md | TODO | Policy Guild | POLICY-CORE-09-001 | Policy schema + binder + diagnostics. | +| Sprint 9 | Policy Foundations | src/StellaOps.Policy/TASKS.md | TODO | Policy Guild | POLICY-CORE-09-002 | Policy snapshot store + revision digests. | +| Sprint 9 | Policy Foundations | src/StellaOps.Policy/TASKS.md | TODO | Policy Guild | POLICY-CORE-09-003 | `/policy/preview` API (image digest → projected verdict diff). | +| Sprint 9 | Policy Foundations | src/StellaOps.Policy/TASKS.md | TODO | Policy Guild | POLICY-CORE-09-004 | Versioned scoring config with schema validation, trust table, and golden fixtures. | +| Sprint 9 | Policy Foundations | src/StellaOps.Policy/TASKS.md | TODO | Policy Guild | POLICY-CORE-09-005 | Scoring/quiet engine – compute score, enforce VEX-only quiet rules, emit inputs and provenance. | +| Sprint 9 | Policy Foundations | src/StellaOps.Policy/TASKS.md | TODO | Policy Guild | POLICY-CORE-09-006 | Unknown state & confidence decay – deterministic bands surfaced in policy outputs. | +| Sprint 9 | DevOps Foundations | ops/devops/TASKS.md | TODO | DevOps Guild | DEVOPS-HELM-09-001 | Helm/Compose environment profiles (dev/staging/airgap) with deterministic digests. | +| Sprint 9 | Docs & Governance | docs/TASKS.md | TODO | Docs Guild, DevEx | DOCS-ADR-09-001 | Establish ADR process and template. | +| Sprint 9 | Docs & Governance | docs/TASKS.md | TODO | Docs Guild, Platform Events | DOCS-EVENTS-09-002 | Publish event schema catalog (`docs/events/`) for critical envelopes. | +| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.Storage/TASKS.md | TODO | Team Scanner Storage | SCANNER-STORAGE-09-301 | Mongo catalog schemas/indexes for images, layers, artifacts, jobs, lifecycle rules plus migrations. | +| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.Storage/TASKS.md | TODO | Team Scanner Storage | SCANNER-STORAGE-09-302 | MinIO layout, immutability policies, client abstraction, and configuration binding. | +| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.Storage/TASKS.md | TODO | Team Scanner Storage | SCANNER-STORAGE-09-303 | Repositories/services with dual-write feature flag, deterministic digests, TTL enforcement tests. | +| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.Queue/TASKS.md | TODO | Team Scanner Queue | SCANNER-QUEUE-09-401 | Queue abstraction + Redis Streams adapter with ack/claim APIs and idempotency tokens. | +| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.Queue/TASKS.md | TODO | Team Scanner Queue | SCANNER-QUEUE-09-402 | Pluggable backend support (Redis, NATS) with configuration binding, health probes, failover docs. | +| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.Queue/TASKS.md | TODO | Team Scanner Queue | SCANNER-QUEUE-09-403 | Retry + dead-letter strategy with structured logs/metrics for offline deployments. | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Cache/TASKS.md | TODO | Scanner Cache Guild | SCANNER-CACHE-10-101 | Implement layer cache store keyed by layer digest with metadata retention per architecture §3.3. | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Cache/TASKS.md | TODO | Scanner Cache Guild | SCANNER-CACHE-10-102 | Build file CAS with dedupe, TTL enforcement, and offline import/export hooks. | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Cache/TASKS.md | TODO | Scanner Cache Guild | SCANNER-CACHE-10-103 | Expose cache metrics/logging and configuration toggles for warm/cold thresholds. | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Cache/TASKS.md | TODO | Scanner Cache Guild | SCANNER-CACHE-10-104 | Implement cache invalidation workflows (layer delete, TTL expiry, diff invalidation). | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Analyzers.OS/TASKS.md | TODO | OS Analyzer Guild | SCANNER-ANALYZERS-OS-10-201 | Alpine/apk analyzer emitting deterministic components with provenance. | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Analyzers.OS/TASKS.md | TODO | OS Analyzer Guild | SCANNER-ANALYZERS-OS-10-202 | Debian/dpkg analyzer mapping packages to purl identity with evidence. | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Analyzers.OS/TASKS.md | TODO | OS Analyzer Guild | SCANNER-ANALYZERS-OS-10-203 | RPM analyzer capturing EVR, file listings, provenance. | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Analyzers.OS/TASKS.md | TODO | OS Analyzer Guild | SCANNER-ANALYZERS-OS-10-204 | Shared OS evidence helpers for package identity + provenance. | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Analyzers.OS/TASKS.md | TODO | OS Analyzer Guild | SCANNER-ANALYZERS-OS-10-205 | Vendor metadata enrichment (source packages, license, CVE hints). | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Analyzers.OS/TASKS.md | TODO | OS Analyzer Guild | SCANNER-ANALYZERS-OS-10-206 | Determinism harness + fixtures for OS analyzers. | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Analyzers.OS/TASKS.md | TODO | OS Analyzer Guild | SCANNER-ANALYZERS-OS-10-207 | Package OS analyzers as restart-time plug-ins (manifest + host registration). | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Analyzers.Lang/TASKS.md | TODO | Language Analyzer Guild | SCANNER-ANALYZERS-LANG-10-301 | Java analyzer emitting `pkg:maven` with provenance. | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Analyzers.Lang/TASKS.md | TODO | Language Analyzer Guild | SCANNER-ANALYZERS-LANG-10-302 | Node analyzer handling workspaces/symlinks emitting `pkg:npm`. | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Analyzers.Lang/TASKS.md | TODO | Language Analyzer Guild | SCANNER-ANALYZERS-LANG-10-303 | Python analyzer reading `*.dist-info`, RECORD hashes, entry points. | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Analyzers.Lang/TASKS.md | TODO | Language Analyzer Guild | SCANNER-ANALYZERS-LANG-10-304 | Go analyzer leveraging buildinfo for `pkg:golang` components. | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Analyzers.Lang/TASKS.md | TODO | Language Analyzer Guild | SCANNER-ANALYZERS-LANG-10-305 | .NET analyzer parsing `*.deps.json`, assembly metadata, RID variants. | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Analyzers.Lang/TASKS.md | TODO | Language Analyzer Guild | SCANNER-ANALYZERS-LANG-10-306 | Rust analyzer detecting crates or falling back to `bin:{sha256}`. | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Analyzers.Lang/TASKS.md | TODO | Language Analyzer Guild | SCANNER-ANALYZERS-LANG-10-307 | Shared language evidence helpers + usage flag propagation. | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Analyzers.Lang/TASKS.md | TODO | Language Analyzer Guild | SCANNER-ANALYZERS-LANG-10-308 | Determinism + fixture harness for language analyzers. | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Analyzers.Lang/TASKS.md | TODO | Language Analyzer Guild | SCANNER-ANALYZERS-LANG-10-309 | Package language analyzers as restart-time plug-ins (manifest + host registration). | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.EntryTrace/TASKS.md | TODO | EntryTrace Guild | SCANNER-ENTRYTRACE-10-401 | POSIX shell AST parser with deterministic output. | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.EntryTrace/TASKS.md | TODO | EntryTrace Guild | SCANNER-ENTRYTRACE-10-402 | Command resolution across layered rootfs with evidence attribution. | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.EntryTrace/TASKS.md | TODO | EntryTrace Guild | SCANNER-ENTRYTRACE-10-403 | Interpreter tracing for shell wrappers to Python/Node/Java launchers. | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.EntryTrace/TASKS.md | TODO | EntryTrace Guild | SCANNER-ENTRYTRACE-10-404 | Python entry analyzer (venv shebang, module invocation, usage flag). | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.EntryTrace/TASKS.md | TODO | EntryTrace Guild | SCANNER-ENTRYTRACE-10-405 | Node/Java launcher analyzer capturing script/jar targets. | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.EntryTrace/TASKS.md | TODO | EntryTrace Guild | SCANNER-ENTRYTRACE-10-406 | Explainability + diagnostics for unresolved constructs with metrics. | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.EntryTrace/TASKS.md | TODO | EntryTrace Guild | SCANNER-ENTRYTRACE-10-407 | Package EntryTrace analyzers as restart-time plug-ins (manifest + host registration). | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Diff/TASKS.md | TODO | Diff Guild | SCANNER-DIFF-10-501 | Build component differ tracking add/remove/version changes with deterministic ordering. | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Diff/TASKS.md | TODO | Diff Guild | SCANNER-DIFF-10-502 | Attribute diffs to introducing/removing layers including provenance evidence. | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Diff/TASKS.md | TODO | Diff Guild | SCANNER-DIFF-10-503 | Produce JSON diff output for inventory vs usage views aligned with API contract. | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Emit/TASKS.md | TODO | Emit Guild | SCANNER-EMIT-10-601 | Compose inventory SBOM (CycloneDX JSON/Protobuf) from layer fragments. | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Emit/TASKS.md | TODO | Emit Guild | SCANNER-EMIT-10-602 | Compose usage SBOM leveraging EntryTrace to flag actual usage. | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Emit/TASKS.md | TODO | Emit Guild | SCANNER-EMIT-10-603 | Generate BOM index sidecar (purl table + roaring bitmap + usage flag). | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Emit/TASKS.md | TODO | Emit Guild | SCANNER-EMIT-10-604 | Package artifacts for export + attestation with deterministic manifests. | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Emit/TASKS.md | TODO | Emit Guild | SCANNER-EMIT-10-605 | Emit BOM-Index sidecar schema/fixtures (CRITICAL PATH for SP16). | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Emit/TASKS.md | TODO | Emit Guild | SCANNER-EMIT-10-606 | Usage view bit flags integrated with EntryTrace. | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Emit/TASKS.md | TODO | Emit Guild | SCANNER-EMIT-10-607 | Embed scoring inputs, confidence band, and quiet provenance in CycloneDX/DSSE artifacts. | +| Sprint 10 | Benchmarks | bench/TASKS.md | TODO | Bench Guild, Scanner Team | BENCH-SCANNER-10-001 | Analyzer microbench harness + baseline CSV. | +| Sprint 10 | Samples | samples/TASKS.md | TODO | Samples Guild, Scanner Team | SAMPLES-10-001 | Sample images with SBOM/BOM-Index sidecars. | +| Sprint 10 | DevOps Perf | ops/devops/TASKS.md | TODO | DevOps Guild | DEVOPS-PERF-10-001 | Perf smoke job ensuring <5 s SBOM compose. | +| Sprint 11 | Signing Chain Bring-up | src/StellaOps.Authority/TASKS.md | TODO | Authority Core & Security Guild | AUTH-DPOP-11-001 | Implement DPoP proof validation + nonce handling for high-value audiences per architecture. | +| Sprint 11 | Signing Chain Bring-up | src/StellaOps.Authority/TASKS.md | TODO | Authority Core & Security Guild | AUTH-MTLS-11-002 | Add OAuth mTLS client credential support with certificate-bound tokens and introspection updates. | +| Sprint 11 | Signing Chain Bring-up | src/StellaOps.Signer/TASKS.md | TODO | Signer Guild | SIGNER-API-11-101 | `/sign/dsse` pipeline with Authority auth, PoE introspection, release verification, DSSE signing. | +| Sprint 11 | Signing Chain Bring-up | src/StellaOps.Signer/TASKS.md | TODO | Signer Guild | SIGNER-REF-11-102 | `/verify/referrers` endpoint with OCI lookup, caching, and policy enforcement. | +| Sprint 11 | Signing Chain Bring-up | src/StellaOps.Signer/TASKS.md | TODO | Signer Guild | SIGNER-QUOTA-11-103 | Enforce plan quotas, concurrency/QPS limits, artifact size caps with metrics/audit logs. | +| Sprint 11 | Signing Chain Bring-up | src/StellaOps.Attestor/TASKS.md | TODO | Attestor Guild | ATTESTOR-API-11-201 | `/rekor/entries` submission pipeline with dedupe, proof acquisition, and persistence. | +| Sprint 11 | Signing Chain Bring-up | src/StellaOps.Attestor/TASKS.md | TODO | Attestor Guild | ATTESTOR-VERIFY-11-202 | `/rekor/verify` + retrieval endpoints validating signatures and Merkle proofs. | +| Sprint 11 | Signing Chain Bring-up | src/StellaOps.Attestor/TASKS.md | TODO | Attestor Guild | ATTESTOR-OBS-11-203 | Telemetry, alerting, mTLS hardening, and archive workflow for Attestor. | +| Sprint 11 | UI Integration | src/StellaOps.UI/TASKS.md | TODO | UI Guild | UI-ATTEST-11-005 | Attestation visibility (Rekor id, status) on Scan Detail. | +| Sprint 12 | Runtime Guardrails | src/StellaOps.Zastava.Core/TASKS.md | TODO | Zastava Core Guild | ZASTAVA-CORE-12-201 | Define runtime event/admission DTOs, hashing helpers, and versioning strategy. | +| Sprint 12 | Runtime Guardrails | src/StellaOps.Zastava.Core/TASKS.md | TODO | Zastava Core Guild | ZASTAVA-CORE-12-202 | Provide configuration/logging/metrics utilities shared by Observer/Webhook. | +| Sprint 12 | Runtime Guardrails | src/StellaOps.Zastava.Core/TASKS.md | TODO | Zastava Core Guild | ZASTAVA-CORE-12-203 | Authority client helpers, OpTok caching, and security guardrails for runtime services. | +| Sprint 12 | Runtime Guardrails | src/StellaOps.Zastava.Core/TASKS.md | TODO | Zastava Core Guild | ZASTAVA-OPS-12-204 | Operational runbooks, alert rules, and dashboard exports for runtime plane. | +| Sprint 12 | Runtime Guardrails | src/StellaOps.Zastava.Observer/TASKS.md | TODO | Zastava Observer Guild | ZASTAVA-OBS-12-001 | Container lifecycle watcher emitting deterministic runtime events with buffering. | +| Sprint 12 | Runtime Guardrails | src/StellaOps.Zastava.Observer/TASKS.md | TODO | Zastava Observer Guild | ZASTAVA-OBS-12-002 | Capture entrypoint traces + loaded libraries, hashing binaries and linking to baseline SBOM. | +| Sprint 12 | Runtime Guardrails | src/StellaOps.Zastava.Observer/TASKS.md | TODO | Zastava Observer Guild | ZASTAVA-OBS-12-003 | Posture checks for signatures/SBOM/attestation with offline caching. | +| Sprint 12 | Runtime Guardrails | src/StellaOps.Zastava.Observer/TASKS.md | TODO | Zastava Observer Guild | ZASTAVA-OBS-12-004 | Batch `/runtime/events` submissions with disk-backed buffer and rate limits. | +| Sprint 12 | Runtime Guardrails | src/StellaOps.Zastava.Webhook/TASKS.md | TODO | Zastava Webhook Guild | ZASTAVA-WEBHOOK-12-101 | Admission controller host with TLS bootstrap and Authority auth. | +| Sprint 12 | Runtime Guardrails | src/StellaOps.Zastava.Webhook/TASKS.md | TODO | Zastava Webhook Guild | ZASTAVA-WEBHOOK-12-102 | Query Scanner `/policy/runtime`, resolve digests, enforce verdicts. | +| Sprint 12 | Runtime Guardrails | src/StellaOps.Zastava.Webhook/TASKS.md | TODO | Zastava Webhook Guild | ZASTAVA-WEBHOOK-12-103 | Caching, fail-open/closed toggles, metrics/logging for admission decisions. | +| Sprint 12 | Runtime Guardrails | src/StellaOps.Scanner.WebService/TASKS.md | TODO | Scanner WebService Guild | SCANNER-RUNTIME-12-301 | `/runtime/events` ingestion endpoint with validation, batching, storage hooks. | +| Sprint 12 | Runtime Guardrails | src/StellaOps.Scanner.WebService/TASKS.md | TODO | Scanner WebService Guild | SCANNER-RUNTIME-12-302 | `/policy/runtime` endpoint joining SBOM baseline + policy verdict with TTL guidance. | +| Sprint 13 | UX & CLI Experience | src/StellaOps.UI/TASKS.md | TODO | UI Guild | UI-AUTH-13-001 | Integrate Authority OIDC + DPoP flows with session management. | +| Sprint 13 | UX & CLI Experience | src/StellaOps.UI/TASKS.md | TODO | UI Guild | UI-SCANS-13-002 | Build scans module (list/detail/SBOM/diff/attestation) with performance + accessibility targets. | +| Sprint 13 | UX & CLI Experience | src/StellaOps.UI/TASKS.md | TODO | UI Guild | UI-VEX-13-003 | Implement VEX explorer + policy editor with preview integration. | +| Sprint 13 | UX & CLI Experience | src/StellaOps.UI/TASKS.md | TODO | UI Guild | UI-ADMIN-13-004 | Deliver admin area (tenants/clients/quotas/licensing) with RBAC + audit hooks. | +| Sprint 13 | UX & CLI Experience | src/StellaOps.UI/TASKS.md | TODO | UI Guild | UI-SCHED-13-005 | Scheduler panel: schedules CRUD, run history, dry-run preview. | +| Sprint 13 | UX & CLI Experience | src/StellaOps.UI/TASKS.md | TODO | UI Guild | UI-NOTIFY-13-006 | Notify panel: channels/rules CRUD, deliveries view, test send. | +| Sprint 13 | UX & CLI Experience | src/StellaOps.Cli/TASKS.md | TODO | DevEx/CLI | CLI-RUNTIME-13-005 | Add runtime policy test verbs that consume `/policy/runtime` and display verdicts. | +| Sprint 13 | UX & CLI Experience | src/StellaOps.Cli/TASKS.md | TODO | DevEx/CLI | CLI-OFFLINE-13-006 | Implement offline kit pull/import/status commands with integrity checks. | +| Sprint 13 | UX & CLI Experience | src/StellaOps.Cli/TASKS.md | TODO | DevEx/CLI | CLI-PLUGIN-13-007 | Package non-core CLI verbs as restart-time plug-ins (manifest + loader tests). | +| Sprint 14 | Release & Offline Ops | ops/devops/TASKS.md | TODO | DevOps Guild | DEVOPS-REL-14-001 | Deterministic build/release pipeline with SBOM/provenance, signing, and manifest generation. | +| Sprint 14 | Release & Offline Ops | ops/offline-kit/TASKS.md | TODO | Offline Kit Guild | DEVOPS-OFFLINE-14-002 | Offline kit packaging workflow with integrity verification and documentation. | +| Sprint 14 | Release & Offline Ops | ops/deployment/TASKS.md | TODO | Deployment Guild | DEVOPS-OPS-14-003 | Deployment/update/rollback automation and channel management documentation. | +| Sprint 14 | Release & Offline Ops | ops/licensing/TASKS.md | TODO | Licensing Guild | DEVOPS-LIC-14-004 | Registry token service tied to Authority, plan gating, revocation handling, monitoring. | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Models/TASKS.md | TODO | Notify Models Guild | NOTIFY-MODELS-15-101 | Define core Notify DTOs, validation helpers, canonical serialization. | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Models/TASKS.md | TODO | Notify Models Guild | NOTIFY-MODELS-15-102 | Publish schema docs and sample payloads for Notify. | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Models/TASKS.md | TODO | Notify Models Guild | NOTIFY-MODELS-15-103 | Versioning/migration helpers for rules/templates/deliveries. | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Storage.Mongo/TASKS.md | TODO | Notify Storage Guild | NOTIFY-STORAGE-15-201 | Mongo schemas/indexes for rules, channels, deliveries, digests, locks, audit. | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Storage.Mongo/TASKS.md | TODO | Notify Storage Guild | NOTIFY-STORAGE-15-202 | Repositories with tenant scoping, soft delete, TTL, causal consistency options. | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Storage.Mongo/TASKS.md | TODO | Notify Storage Guild | NOTIFY-STORAGE-15-203 | Delivery history retention and query APIs. | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Queue/TASKS.md | TODO | Notify Queue Guild | NOTIFY-QUEUE-15-401 | Bus abstraction + Redis Streams adapter with ordering/idempotency. | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Queue/TASKS.md | TODO | Notify Queue Guild | NOTIFY-QUEUE-15-402 | NATS JetStream adapter with health probes and failover. | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Queue/TASKS.md | TODO | Notify Queue Guild | NOTIFY-QUEUE-15-403 | Delivery queue with retry/dead-letter + metrics. | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Engine/TASKS.md | TODO | Notify Engine Guild | NOTIFY-ENGINE-15-301 | Rules evaluation core (filters, throttles, idempotency). | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Engine/TASKS.md | TODO | Notify Engine Guild | NOTIFY-ENGINE-15-302 | Action planner + digest coalescer. | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Engine/TASKS.md | TODO | Notify Engine Guild | NOTIFY-ENGINE-15-303 | Template rendering engine (Slack/Teams/Email/Webhook). | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Engine/TASKS.md | TODO | Notify Engine Guild | NOTIFY-ENGINE-15-304 | Test-send sandbox + preview utilities. | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.WebService/TASKS.md | TODO | Notify WebService Guild | NOTIFY-WEB-15-101 | Minimal API host with Authority enforcement and plug-in loading. | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.WebService/TASKS.md | TODO | Notify WebService Guild | NOTIFY-WEB-15-102 | Rules/channel/template CRUD with audit logging. | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.WebService/TASKS.md | TODO | Notify WebService Guild | NOTIFY-WEB-15-103 | Delivery history & test-send endpoints. | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.WebService/TASKS.md | TODO | Notify WebService Guild | NOTIFY-WEB-15-104 | Configuration binding + startup diagnostics. | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Worker/TASKS.md | TODO | Notify Worker Guild | NOTIFY-WORKER-15-201 | Bus subscription + leasing loop with backoff. | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Worker/TASKS.md | TODO | Notify Worker Guild | NOTIFY-WORKER-15-202 | Rules evaluation pipeline integration. | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Worker/TASKS.md | TODO | Notify Worker Guild | NOTIFY-WORKER-15-203 | Channel dispatch orchestration with retries. | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Worker/TASKS.md | TODO | Notify Worker Guild | NOTIFY-WORKER-15-204 | Metrics/telemetry for Notify workers. | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Connectors.Slack/TASKS.md | TODO | Notify Connectors Guild | NOTIFY-CONN-SLACK-15-501 | Slack connector with rate-limit aware delivery. | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Connectors.Slack/TASKS.md | TODO | Notify Connectors Guild | NOTIFY-CONN-SLACK-15-502 | Slack health/test-send support. | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Connectors.Teams/TASKS.md | TODO | Notify Connectors Guild | NOTIFY-CONN-TEAMS-15-601 | Teams connector with Adaptive Cards. | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Connectors.Teams/TASKS.md | TODO | Notify Connectors Guild | NOTIFY-CONN-TEAMS-15-602 | Teams health/test-send support. | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Connectors.Email/TASKS.md | TODO | Notify Connectors Guild | NOTIFY-CONN-EMAIL-15-701 | SMTP connector with TLS + rendering. | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Connectors.Email/TASKS.md | TODO | Notify Connectors Guild | NOTIFY-CONN-EMAIL-15-702 | DKIM + health/test-send flows. | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Connectors.Webhook/TASKS.md | TODO | Notify Connectors Guild | NOTIFY-CONN-WEBHOOK-15-801 | Webhook connector with signing/retries. | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Connectors.Webhook/TASKS.md | TODO | Notify Connectors Guild | NOTIFY-CONN-WEBHOOK-15-802 | Webhook health/test-send support. | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Connectors.Slack/TASKS.md | TODO | Notify Connectors Guild | NOTIFY-CONN-SLACK-15-503 | Package Slack connector as restart-time plug-in (manifest + host registration). | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Connectors.Teams/TASKS.md | TODO | Notify Connectors Guild | NOTIFY-CONN-TEAMS-15-603 | Package Teams connector as restart-time plug-in (manifest + host registration). | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Connectors.Email/TASKS.md | TODO | Notify Connectors Guild | NOTIFY-CONN-EMAIL-15-703 | Package Email connector as restart-time plug-in (manifest + host registration). | +| Sprint 15 | Notify Foundations | src/StellaOps.Scanner.WebService/TASKS.md | TODO | Scanner WebService Guild | SCANNER-EVENTS-15-201 | Emit `scanner.report.ready` + `scanner.scan.completed` events. | +| Sprint 15 | Benchmarks | bench/TASKS.md | TODO | Bench Guild, Notify Team | BENCH-NOTIFY-15-001 | Notify dispatch throughput bench with results CSV. | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Connectors.Webhook/TASKS.md | TODO | Notify Connectors Guild | NOTIFY-CONN-WEBHOOK-15-803 | Package Webhook connector as restart-time plug-in (manifest + host registration). | +| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.Models/TASKS.md | TODO | Scheduler Models Guild | SCHED-MODELS-16-101 | Define Scheduler DTOs & validation. | +| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.Models/TASKS.md | TODO | Scheduler Models Guild | SCHED-MODELS-16-102 | Publish schema docs/sample payloads. | +| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.Models/TASKS.md | TODO | Scheduler Models Guild | SCHED-MODELS-16-103 | Versioning/migration helpers for schedules/runs. | +| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.Storage.Mongo/TASKS.md | TODO | Scheduler Storage Guild | SCHED-STORAGE-16-201 | Mongo schemas/indexes for Scheduler state. | +| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.Storage.Mongo/TASKS.md | TODO | Scheduler Storage Guild | SCHED-STORAGE-16-202 | Repositories with tenant scoping, TTL, causal consistency. | +| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.Storage.Mongo/TASKS.md | TODO | Scheduler Storage Guild | SCHED-STORAGE-16-203 | Audit + stats materialization for UI. | +| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.Queue/TASKS.md | TODO | Scheduler Queue Guild | SCHED-QUEUE-16-401 | Queue abstraction + Redis Streams adapter. | +| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.Queue/TASKS.md | TODO | Scheduler Queue Guild | SCHED-QUEUE-16-402 | NATS JetStream adapter with health probes. | +| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.Queue/TASKS.md | TODO | Scheduler Queue Guild | SCHED-QUEUE-16-403 | Dead-letter handling + metrics. | +| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.ImpactIndex/TASKS.md | TODO | Scheduler ImpactIndex Guild | SCHED-IMPACT-16-301 | Ingest BOM-Index into roaring bitmap store. | +| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.ImpactIndex/TASKS.md | TODO | Scheduler ImpactIndex Guild | SCHED-IMPACT-16-302 | Query APIs for ResolveByPurls/ResolveByVulns/ResolveAll. | +| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.ImpactIndex/TASKS.md | TODO | Scheduler ImpactIndex Guild | SCHED-IMPACT-16-303 | Snapshot/compaction/invalidation workflow. | +| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.ImpactIndex/TASKS.md | DOING | Scheduler ImpactIndex Guild | SCHED-IMPACT-16-300 | **STUB** ImpactIndex ingest/query using fixtures (to be removed by SP16 completion). | +| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.WebService/TASKS.md | TODO | Scheduler WebService Guild | SCHED-WEB-16-101 | Minimal API host with Authority enforcement. | +| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.WebService/TASKS.md | TODO | Scheduler WebService Guild | SCHED-WEB-16-102 | Schedules CRUD (cron validation, pause/resume, audit). | +| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.WebService/TASKS.md | TODO | Scheduler WebService Guild | SCHED-WEB-16-103 | Runs API (list/detail/cancel) + impact previews. | +| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.WebService/TASKS.md | TODO | Scheduler WebService Guild | SCHED-WEB-16-104 | Feedser/Vexer webhook handlers with security enforcement. | +| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.Worker/TASKS.md | TODO | Scheduler Worker Guild | SCHED-WORKER-16-201 | Planner loop (cron/event triggers, leases, fairness). | +| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.Worker/TASKS.md | TODO | Scheduler Worker Guild | SCHED-WORKER-16-202 | ImpactIndex targeting and shard planning. | +| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.Worker/TASKS.md | TODO | Scheduler Worker Guild | SCHED-WORKER-16-203 | Runner execution invoking Scanner analysis/content refresh. | +| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.Worker/TASKS.md | TODO | Scheduler Worker Guild | SCHED-WORKER-16-204 | Emit rescan/report events for Notify/UI. | +| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.Worker/TASKS.md | TODO | Scheduler Worker Guild | SCHED-WORKER-16-205 | Metrics/telemetry for Scheduler planners/runners. | +| Sprint 16 | Benchmarks | bench/TASKS.md | TODO | Bench Guild, Scheduler Team | BENCH-IMPACT-16-001 | ImpactIndex throughput bench + RAM profile. | +| Sprint 17 | Symbol Intelligence & Forensics | src/StellaOps.Scanner.Emit/TASKS.md | TODO | Emit Guild | SCANNER-EMIT-17-701 | Record GNU build-id for ELF components and surface it in SBOM/diff outputs. | +| Sprint 17 | Symbol Intelligence & Forensics | src/StellaOps.Zastava.Observer/TASKS.md | TODO | Zastava Observer Guild | ZASTAVA-OBS-17-005 | Collect GNU build-id during runtime observation and attach it to emitted events. | +| Sprint 17 | Symbol Intelligence & Forensics | src/StellaOps.Scanner.WebService/TASKS.md | TODO | Scanner WebService Guild | SCANNER-RUNTIME-17-401 | Persist runtime build-id observations and expose them for debug-symbol correlation. | +| Sprint 17 | Symbol Intelligence & Forensics | ops/devops/TASKS.md | TODO | DevOps Guild | DEVOPS-REL-17-002 | Ship stripped debug artifacts organised by build-id within release/offline kits. | +| Sprint 17 | Symbol Intelligence & Forensics | docs/TASKS.md | TODO | Docs Guild | DOCS-RUNTIME-17-004 | Document build-id workflows for SBOMs, runtime events, and debug-store usage. | diff --git a/SPRINTS_EXCITITOR.md b/SPRINTS_EXCITITOR.md deleted file mode 100644 index cf099e9f..00000000 --- a/SPRINTS_EXCITITOR.md +++ /dev/null @@ -1,2 +0,0 @@ -| Sprint | Theme | Tasks File Path | Status | Type of Specialist | Task ID | Task Description | -| --- | --- | --- | --- | --- | --- | --- | diff --git a/SPRINTS_IMPLEMENTION_PLAN.md b/SPRINTS_IMPLEMENTION_PLAN.md new file mode 100644 index 00000000..8977c123 --- /dev/null +++ b/SPRINTS_IMPLEMENTION_PLAN.md @@ -0,0 +1,289 @@ +# StellaOps Multi-Sprint Implementation Plan (Agile Track) + +This plan translates the current `SPRINTS.md` (read the file if you have not) backlog into parallel-friendly execution clusters. Each sprint is decomposed into **groups** that can run concurrently without stepping on the same directories. For every group we capture: + +- **Tasks** (ID · est. effort · path) +- **Acceptance metrics** (quantitative targets to reduce rework) +- **Gate** artifacts required before dependent groups can start + +Durations are estimated work sizes (1 d ≈ one focused engineer day). Milestones are gated by artifacts—not calendar dates—to keep us agile and adaptable to competitor pressure. + +--- + +## Sprint 9 – Scanner Core Foundations (ID: SP9, ~3 w) + +### Group SP9-G1 — Core Contracts & Observability (src/StellaOps.Scanner.Core) ~1 w +- Tasks: + - SCANNER-CORE-09-501 · 3 d · `/src/StellaOps.Scanner.Core/TASKS.md` + - SCANNER-CORE-09-502 · 2 d · same path + - SCANNER-CORE-09-503 · 2 d · same path +- Acceptance metrics: DTO round-trip tests stable; middleware adds ≤5 µs per call. +- Gate SP9-G1 → WebService: `scanner-core-contracts.md` snippet plus `ScannerCoreContractsTests` green. + +### Group SP9-G2 — Queue Backbone (src/StellaOps.Scanner.Queue) ~1 w +- Tasks: SCANNER-QUEUE-09-401 (3 d), -402 (2 d), -403 (2 d) · `/src/StellaOps.Scanner.Queue/TASKS.md` +- Acceptance: dequeue latency p95 ≤20 ms at 40 rps; chaos test retains leases. +- Gate: Redis/NATS adapters docs + `QueueLeaseIntegrationTests` passing. + +### Group SP9-G3 — Storage Backbone (src/StellaOps.Scanner.Storage) ~1 w +- Tasks: SCANNER-STORAGE-09-301 (3 d), -302 (2 d), -303 (2 d) +- Acceptance: majority write/read ≤50 ms; TTL verified. +- Gate: migrations checked in; `StorageDualWriteFixture` passes. + +### Group SP9-G4 — WebService Host & Policy Surfacing (src/StellaOps.Scanner.WebService) ~1.2 w +- Tasks: SCANNER-WEB-09-101 (2 d), -102 (3 d), -103 (2 d), -104 (2 d), SCANNER-POLICY-09-105 (3 d), SCANNER-POLICY-09-106 (4 d) +- Acceptance: `/api/v1/scans` enqueue p95 ≤50 ms under synthetic load; policy validation errors actionable; `/reports` response signed. +- Gate SP9-G4 → SP10/SP11: `/reports` OpenAPI frozen; sample signed envelope committed in `samples/api/reports/`. + +### Group SP9-G5 — Worker Host (src/StellaOps.Scanner.Worker) ~1 w +- Tasks: SCANNER-WORKER-09-201 (3 d), -202 (3 d), -203 (2 d), -204 (2 d) +- Acceptance: job lease never drops <3× heartbeat; progress events deterministic. +- Gate: `WorkerBasicScanScenario` integration recorded. + +### Group SP9-G6 — Buildx Plug-in (src/StellaOps.Scanner.Sbomer.BuildXPlugin) ~0.8 w +- Tasks: SP9-BLDX-09-001 (3 d), SP9-BLDX-09-002 (2 d), SP9-BLDX-09-003 (2 d) +- Acceptance: build-time overhead ≤300 ms/layer on 4 vCPU; CAS handshake reliable in CI sample. +- Gate: buildx demo workflow artifact + quickstart doc. + +### Group SP9-G7 — Policy Engine Core (src/StellaOps.Policy) ~1 w +- Tasks: POLICY-CORE-09-001 (2 d), -002 (3 d), -003 (3 d), -004 (3 d), -005 (4 d), -006 (2 d) +- Acceptance: policy parsing ≥200 files/s; preview diff response <200 ms for 500-component SBOM; quieting logic audited. +- Gate: `policy-schema@1` published; revision digests stored; preview API doc updated. + +### Group SP9-G8 — DevOps Early Guardrails (ops/devops) ~0.4 w +- Tasks: DEVOPS-HELM-09-001 (3 d) +- Acceptance: helm/compose profiles for dev/stage/airgap lint + dry-run clean; manifests pinned to digest. +- Gate: profiles merged under `deploy/`; install guide cross-link. + +### Group SP9-G9 — Documentation & Events (docs/) ~0.4 w +- Tasks: DOCS-ADR-09-001 (2 d), DOCS-EVENTS-09-002 (2 d) +- Acceptance: ADR process broadcast; event schemas validated via CI. +- Gate: `docs/adr/index.md` linking template; `docs/events/README.md` referencing schemas. + +--- + +## Sprint 10 – Scanner Analyzers & SBOM (ID: SP10, ~4 w) + +### Group SP10-G1 — OS Analyzer Plug-ins (src/StellaOps.Scanner.Analyzers.OS) ~1 w +- Tasks: SCANNER-ANALYZERS-OS-10-201..207 (durations 2–3 d each) +- Acceptance: analyzer runtime <1.5 s/image; memory <250 MB. +- Gate: plug-ins packaged under `plugins/scanner/analyzers/os/`; determinism CI job green. + +### Group SP10-G2 — Language Analyzer Plug-ins (src/StellaOps.Scanner.Analyzers.Lang) ~1.5 w +- Tasks: SCANNER-ANALYZERS-LANG-10-301..309 +- Acceptance: Node analyzer handles 10 k modules <2 s; Python memory <200 MB. +- Gate: golden outputs stored; plugin manifests present. + +### Group SP10-G3 — EntryTrace Plug-ins (src/StellaOps.Scanner.EntryTrace) ~0.8 w +- Tasks: SCANNER-ENTRYTRACE-10-401..407 +- Acceptance: ≥95 % launcher resolution success on samples; unknown reasons enumerated. +- Gate: entrytrace plug-ins packaged; explainability doc updated. + +### Group SP10-G4 — SBOM Composition & BOM Index (src/StellaOps.Scanner.Diff + Emit) ~1 w +- Tasks: SCANNER-DIFF-10-501..503, SCANNER-EMIT-10-601..606 +- Acceptance: BOM-Index emission <500 ms/image; diff output deterministic across runs. +- Gate SP10-G4 → SP16: `docs/artifacts/bom-index/` schema + fixtures; tests `BOMIndexGoldenIsStable` & `UsageFlagsAreAccurate` green. + +### Group SP10-G5 — Cache Subsystem (src/StellaOps.Scanner.Cache) ~0.6 w +- Tasks: SCANNER-CACHE-10-101..104 +- Acceptance: cache hit instrumentation validated; eviction keeps footprint <5 GB. +- Gate: cache configuration doc; integration test `LayerCacheRoundTrip` green. + +### Group SP10-G6 — Benchmarks & Samples (bench/, samples/, ops/devops) ~0.6 w +- Tasks: BENCH-SCANNER-10-001 (2 d), SAMPLES-10-001 (finish – 3 d), DEVOPS-PERF-10-001 (2 d) +- Acceptance: analyzer benchmark CSV published; perf CI guard ensures SBOM compose <5 s; sample SBOM/BOM-Index committed. +- Gate: bench results stored under `bench/`; `samples/` populated; CI job added. + +--- + +## Sprint 11 – Signing Chain Bring-up (ID: SP11, ~3 w) + +### Group SP11-G1 — Authority Sender Constraints (src/StellaOps.Authority) ~0.8 w +- Tasks: AUTH-DPOP-11-001 (3 d), AUTH-MTLS-11-002 (2 d) +- Acceptance: DPoP nonce dance validated; mTLS tokens issued in ≤40 ms. +- Gate: updated Authority OpenAPI; QA scripts verifying DPoP/mTLS. + +### Group SP11-G2 — Signer Service (src/StellaOps.Signer) ~1.2 w +- Tasks: SIGNER-API-11-101 (4 d), SIGNER-REF-11-102 (2 d), SIGNER-QUOTA-11-103 (2 d) +- Acceptance: signing throughput ≥30 req/min; p95 latency ≤200 ms. +- Gate SP11-G2 → Attestor/UI: `/sign/dsse` OpenAPI frozen; signed DSSE bundle in repo; Rekor interop test passing. + +### Group SP11-G3 — Attestor Service (src/StellaOps.Attestor) ~1 w +- Tasks: ATTESTOR-API-11-201 (3 d), ATTESTOR-VERIFY-11-202 (2 d), ATTESTOR-OBS-11-203 (2 d) +- Acceptance: inclusion proof retrieval <500 ms; audit log coverage 100 %. +- Gate: Attestor API doc + verification script. + +### Group SP11-G4 — UI Attestation Hooks (src/StellaOps.UI) ~0.4 w +- Tasks: UI-ATTEST-11-005 (3 d) +- Acceptance: attestation panel renders within 200 ms; Rekor link verified. +- Gate SP11-G4 → SP13-G1: recorded UX walkthrough. + +--- + +## Sprint 12 – Runtime Guardrails (ID: SP12, ~3 w) + +### Group SP12-G1 — Zastava Core (src/StellaOps.Zastava.Core) ~0.8 w +- Tasks: ZASTAVA-CORE-12-201..204 +- Acceptance: DTO tests stable; configuration docs produced. +- Gate: schema doc + logging helpers integrated. + +### Group SP12-G2 — Zastava Observer (src/StellaOps.Zastava.Observer) ~0.8 w +- Tasks: ZASTAVA-OBS-12-001..004 +- Acceptance: observer memory <200 MB; event flush ≤2 s. +- Gate: sample runtime events stored; offline buffer test passes. + +### Group SP12-G3 — Zastava Webhook (src/StellaOps.Zastava.Webhook) ~0.6 w +- Tasks: ZASTAVA-WEBHOOK-12-101..103 +- Acceptance: admission latency p95 ≤45 ms; cache TTL adhered to. +- Gate: TLS rotation procedure documented; readiness probe script. + +### Group SP12-G4 — Scanner Runtime APIs (src/StellaOps.Scanner.WebService) ~0.8 w +- Tasks: SCANNER-RUNTIME-12-301 (2 d), SCANNER-RUNTIME-12-302 (3 d) +- Acceptance: `/runtime/events` handles 500 events/sec; `/policy/runtime` output matches webhook decisions. +- Gate SP12-G4 → SP13/SP15: API documented, fixtures updated. + +--- + +## Sprint 13 – UX & CLI Experience (ID: SP13, ~2 w) + +### Group SP13-G1 — UI Shell & Panels (src/StellaOps.UI) ~1.6 w +- Tasks: UI-AUTH-13-001 (3 d), UI-SCANS-13-002 (4 d), UI-VEX-13-003 (3 d), UI-ADMIN-13-004 (2 d), UI-SCHED-13-005 (3 d), UI-NOTIFY-13-006 (3 d) +- Acceptance: Lighthouse ≥85; Scheduler/Notify panels function against mocked APIs. +- Gate: UI dev server fixtures committed; QA sign-off captured. + +### Group SP13-G2 — CLI Enhancements (src/StellaOps.Cli) ~0.8 w +- Tasks: CLI-RUNTIME-13-005 (3 d), CLI-OFFLINE-13-006 (3 d), CLI-PLUGIN-13-007 (2 d) +- Acceptance: runtime policy CLI completes <1 s for 10 images; offline kit commands resume downloads. +- Gate: CLI plugin manifest doc; smoke tests covering new verbs. + +--- + +## Sprint 14 – Release & Offline Ops (ID: SP14, ~2 w) + +### Group SP14-G1 — Release Automation (ops/devops) ~0.8 w +- Tasks: DEVOPS-REL-14-001 (4 d) +- Acceptance: reproducible build diff tool shows zero drift across two runs; signing pipeline green. +- Gate: signed manifest + provenance published. + +### Group SP14-G2 — Offline Kit Packaging (ops/offline-kit) ~0.6 w +- Tasks: DEVOPS-OFFLINE-14-002 (3 d) +- Acceptance: kit import <5 min with integrity verification CLI. +- Gate: kit doc updated; import script included. + +### Group SP14-G3 — Deployment Playbooks (ops/deployment) ~0.4 w +- Tasks: DEVOPS-OPS-14-003 (2 d) +- Acceptance: rollback drill recorded; compatibility matrix produced. +- Gate: playbook PR merged with Ops sign-off. + +### Group SP14-G4 — Licensing Token Service (ops/licensing) ~0.4 w +- Tasks: DEVOPS-LIC-14-004 (2 d) +- Acceptance: token service handles 100 req/min; revocation latency <60 s. +- Gate: monitoring dashboard links; failover doc. + +--- + +## Sprint 15 – Notify Foundations (ID: SP15, ~3 w) + +### Group SP15-G1 — Models & Storage (src/StellaOps.Notify.Models + Storage.Mongo) ~0.8 w +- Tasks: NOTIFY-MODELS-15-101 (2 d), -102 (2 d), -103 (1 d); NOTIFY-STORAGE-15-201 (3 d), -202 (2 d), -203 (1 d) +- Acceptance: rule CRUD latency <120 ms; delivery retention job verified. +- Gate: schema docs + fixtures published. + +### Group SP15-G2 — Engine & Queue (src/StellaOps.Notify.Engine + Queue) ~0.8 w +- Tasks: NOTIFY-ENGINE-15-301..304, NOTIFY-QUEUE-15-401..403 +- Acceptance: rules evaluation ≥5k events/min; queue dead-letter <0.5 %. +- Gate: digest outputs committed; queue config doc updated. + +### Group SP15-G3 — WebService & Worker (src/StellaOps.Notify.WebService + Worker) ~0.8 w +- Tasks: NOTIFY-WEB-15-101..104, NOTIFY-WORKER-15-201..204 +- Acceptance: API p95 <120 ms; worker delivery success ≥99 %. +- Gate: end-to-end fixture run producing delivery record. + +### Group SP15-G4 — Channel Plug-ins (src/StellaOps.Notify.Connectors.*) ~0.6 w +- Tasks: NOTIFY-CONN-SLACK-15-501..503, NOTIFY-CONN-TEAMS-15-601..603, NOTIFY-CONN-EMAIL-15-701..703, NOTIFY-CONN-WEBHOOK-15-801..803 +- Acceptance: channel-specific retry policies verified; rate limits respected. +- Gate: plug-in manifests inside `plugins/notify/**`; test-send docs. + +### Group SP15-G5 — Events & Benchmarks (src/StellaOps.Scanner.WebService + bench) ~0.5 w +- Tasks: SCANNER-EVENTS-15-201 (2 d), BENCH-NOTIFY-15-001 (2 d) +- Acceptance: event emission latency <100 ms; throughput bench results stored. +- Gate: `docs/events/samples/` contains sample payloads; bench CSV in repo. + +--- + +## Sprint 16 – Scheduler Intelligence (ID: SP16, ~4 w) + +### Group SP16-G1 — Models & Storage (src/StellaOps.Scheduler.Models + Storage.Mongo) ~1 w +- Tasks: SCHED-MODELS-16-101 (3 d), -102 (2 d), -103 (2 d); SCHED-STORAGE-16-201 (3 d), -202 (2 d), -203 (2 d) +- Acceptance: schedule CRUD latency <120 ms; run retention TTL enforced. +- Gate: schema doc + integration tests passing. + +### Group SP16-G2 — ImpactIndex & Queue (src/StellaOps.Scheduler.ImpactIndex + Queue + Bench) ~1.2 w +- Tasks: SCHED-IMPACT-16-300 (2 d, DOING), SCHED-IMPACT-16-301 (3 d), -302 (3 d), -303 (2 d); SCHED-QUEUE-16-401..403 (each 2 d); BENCH-IMPACT-16-001 (2 d) +- Acceptance: impact resolve 10k productKeys <300 ms hot; stub removed by sprint end. +- Gate: roaring snapshot stored; bench CSV published; removal plan for stub recorded. + +### Group SP16-G3 — Scheduler WebService (src/StellaOps.Scheduler.WebService) ~0.8 w +- Tasks: SCHED-WEB-16-101..104 (each 2 d) +- Acceptance: preview endpoint <250 ms; webhook security enforced. +- Gate: OpenAPI published; dry-run JSON fixtures stored. + +### Group SP16-G4 — Scheduler Worker (src/StellaOps.Scheduler.Worker) ~1 w +- Tasks: SCHED-WORKER-16-201 (3 d), -202 (2 d), -203 (3 d), -204 (2 d), -205 (2 d) +- Acceptance: planner fairness metrics captured; runner success ≥98 % across 1k sims. +- Gate: event emission to Notify verified; metrics dashboards live. + +--- + +## Sprint 17 – Symbol Intelligence & Forensics (ID: SP17, ~2.5 w) + +### Group SP17-G1 — Scanner Forensics (src/StellaOps.Scanner.Emit + WebService) ~1.2 w +- Tasks: SCANNER-EMIT-17-701 (4 d), SCANNER-RUNTIME-17-401 (3 d) +- Acceptance: forensic overlays add ≤150 ms per image; runtime API exposes symbol hints with feature flag. +- Gate: forensic SBOM samples committed; API doc updated. + +### Group SP17-G2 — Zastava Observability (src/StellaOps.Zastava.Observer) ~0.6 w +- Tasks: ZASTAVA-OBS-17-005 (3 d) +- Acceptance: new telemetry surfaces symbol diffs; observer CPU <10 % under load. +- Gate: Grafana dashboard export, alert thresholds defined. + +### Group SP17-G3 — Release Hardening (ops/devops) ~0.4 w +- Tasks: DEVOPS-REL-17-002 (2 d) +- Acceptance: deterministic build verifier job updated to include forensics artifacts. +- Gate: CI pipeline stage `forensics-verify` green. + +### Group SP17-G4 — Documentation (docs/) ~0.3 w +- Tasks: DOCS-RUNTIME-17-004 (2 d) +- Acceptance: runtime forensic guide published with troubleshooting. +- Gate: docs review sign-off; links added to UI help. + +--- + +## Integration Buffers +- **INT-A (0.3 w, after SP10):** Image → SBOM → BOM-Index → Scheduler preview → UI dry-run using fixtures. +- **INT-B (0.3 w, after SP11 & SP15):** SBOM → policy verdict → signed DSSE → Rekor entry → Notify delivery end-to-end. + +## Parallelisation Strategy +- SP9 core modules and SP11 authority upgrades can progress in parallel; scanner clients rely on feature flags while DPoP/mTLS hardening lands. +- SP10 SBOM emission may start alongside Scheduler ImpactIndex using `samples/` fixtures; stub SCHED-IMPACT-16-300 keeps velocity while awaiting roaring index. +- Notify foundations (SP15) can begin once event schemas freeze (delivered in SP9-G9/SP12-G4), consuming canned events until Scanner emits live ones. +- UI (SP13) uses mocked endpoints early, decoupling front-end delivery from backend readiness. + +## Risk Registry + +| Risk ID | Description | Owner | Mitigation | Trigger | +|---------|-------------|-------|-----------|---------| +| R1 | BOM-Index memory blow-up on large fleets | Scheduler ImpactIndex Guild | Shard + mmap plan; monitor BENCH-IMPACT-16-001 | RAM > 8 GB in bench | +| R2 | Buildx plugin latency regression | BuildX Guild | DEVOPS-PERF-10-001 guard; fallback to post-build scan | Buildx job >300 ms/layer | +| R3 | Notify digests flooding Slack | Notify Engine Guild | throttle defaults, BENCH-NOTIFY-15-001 coverage | Dropped messages >1 % | +| R4 | Policy precedence confusion | Policy Guild | ADR, preview API, unit tests | Operator escalation about precedence | +| R5 | ImpactIndex stub lingers | Scheduler ImpactIndex Guild | Track SCHED-IMPACT-16-300 removal in sprint review | Stub present past SP16 | +| R6 | Symbol forensics slows runtime | Scanner Emit Guild | Feature flag; perf tests in SP17-G1 | Forensics adds >150 ms/image | + +## Envelope & ADR Governance +- Event schemas (`docs/events/*.json`) versioned; producers must bump suffix on breaking changes. +- ADR template (`docs/adr/0000-template.md`) mandatory for BOM-Index format, event envelopes, DPoP nonce policy, Rekor migration. + +--- + +**Summary:** The plan keeps high-impact artifacts (policy engine, BOM-Index, signing chain) on the critical path while unlocking parallel tracks (Notify, Scheduler, UI) through early schema freezes and fixtures. Integration buffers ensure cross-team touchpoints are validated continuously, supporting rapid iteration against competitive pressure. diff --git a/bench/TASKS.md b/bench/TASKS.md new file mode 100644 index 00000000..e62bc419 --- /dev/null +++ b/bench/TASKS.md @@ -0,0 +1,7 @@ +# Benchmarks Task Board + +| ID | Status | Owner(s) | Depends on | Description | Exit Criteria | +|----|--------|----------|------------|-------------|---------------| +| BENCH-SCANNER-10-001 | TODO | Bench Guild, Scanner Team | SCANNER-ANALYZERS-LANG-10-303 | Analyzer microbench harness (node_modules, site-packages) + baseline CSV. | Harness committed under `bench/Scanner.Analyzers`; baseline CSV recorded; CI job publishes results. | +| BENCH-IMPACT-16-001 | TODO | Bench Guild, Scheduler Team | SCHED-IMPACT-16-301 | ImpactIndex throughput bench (resolve 10k productKeys) + RAM profile. | Benchmark script ready; baseline metrics recorded; alert thresholds defined. | +| BENCH-NOTIFY-15-001 | TODO | Bench Guild, Notify Team | NOTIFY-ENGINE-15-301 | Notify dispatch throughput bench (vary rule density) with results CSV. | Bench executed; results stored; regression alert configured. | diff --git a/docs/07_HIGH_LEVEL_ARCHITECTURE.md b/docs/07_HIGH_LEVEL_ARCHITECTURE.md index cd6d0b6b..3a11036b 100755 --- a/docs/07_HIGH_LEVEL_ARCHITECTURE.md +++ b/docs/07_HIGH_LEVEL_ARCHITECTURE.md @@ -1,8 +1,3 @@ -Below is the **revised, consolidated** `high_level_architecture.md`. -It **absorbs** all content from `components.md` so you have a single, authoritative file. No separate components doc is required. - ---- - # High‑Level Architecture — **Stella Ops** (Consolidated • 2025Q4) > **Purpose.** A complete, implementation‑ready map of Stella Ops: product vision, all runtime components, trust boundaries, tokens/licensing, control/data flows, storage, APIs, security, scale, DevOps, and verification logic. @@ -30,28 +25,32 @@ It **absorbs** all content from `components.md` so you have a single, authoritat ### 1.1 Runtime inventory (first‑party) -| Service / Tool | Container image | Core role | Scale pattern | -| ------------------------------- | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------- | -| **Scanner.WebService** | `stellaops/scanner-web` | Control plane for scans; catalog; SBOM composition (inventory & usage); diff; exports. | Stateless; N replicas behind LB. | -| **Scanner.Worker** | `stellaops/scanner-worker` | Runs analyzers (OS, Lang: Java/Node/Python/Go/.NET/Rust, Native ELF/PE/Mach‑O, EntryTrace); emits per‑layer SBOMs and composes image SBOMs. | Horizontal; queue‑driven; sharded by layer digest. | -| **Scanner.Sbomer.BuildXPlugin** | `stellaops/sbom-indexer` | BuildKit **generator** for build‑time SBOMs as OCI **referrers**. | CI‑side; ephemeral. | -| **Scanner.Sbomer.DockerImage** | `stellaops/scanner-cli` | CLI‑orchestrated scanner container for post‑build scans. | Local/CI; ephemeral. | -| **Concelier.WebService** | `stellaops/concelier-web` | Vulnerability ingest/normalize/merge/export (JSON + Trivy DB). | HA via Mongo locks. | -| **Excititor.WebService** | `stellaops/excititor-web` | VEX ingest/normalize/consensus; conflict retention; exports. | HA via Mongo locks. | -| **Policy Engine** | (in `scanner-web`) | YAML DSL evaluator (waivers, vendor preferences, KEV/EPSS, license, usage‑gating); produces **policy digest**. | In‑process; cache per digest. | -| **Signer** | `stellaops/signer` | **Hard gate:** validates entitlement + release integrity; mints signing cert (Fulcio keyless) or uses KMS; signs DSSE. | Stateless; HPA by QPS. | -| **Attestor** | `stellaops/attestor` | Posts DSSE bundles to **Rekor v2**; verification endpoints. | Stateless; HPA by QPS. | -| **Authority** | `stellaops/authority` | On‑prem OIDC issuing **short‑lived OpToks** with DPoP/mTLS sender constraint. | HA behind LB. | -| **Zastava** (Runtime) | `stellaops/zastava` | Runtime inspector/enforcer (observer + optional Admission Webhook). | DaemonSet + Webhook. | -| **Web UI** | `stellaops/ui` | Angular app for scans, diffs, policy, VEX, runtime, reports. | Stateless. | -| **StellaOps.Cli** | `stellaops/cli` | CLI for init/scan/export/diff/policy/report/verify; Buildx helper. | Local/CI. | +| Service / Tool | Container image | Core role | Scale pattern | +| ------------------------------- | ---------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------- | +| **Scanner.WebService** | `stellaops/scanner-web` | Control plane for scans; catalog; SBOM composition (inventory & usage); diff; exports; **analysis‑only report runs** for Scheduler. | Stateless; N replicas behind LB. | +| **Scanner.Worker** | `stellaops/scanner-worker` | Runs analyzers (OS, Lang: Java/Node/Python/Go/.NET/Rust, Native ELF/PE/Mach‑O, EntryTrace); emits per‑layer SBOMs and composes image SBOMs. | Horizontal; queue‑driven; sharded by layer digest. | +| **Scanner.Sbomer.BuildXPlugin** | `stellaops/sbom-indexer` | BuildKit **generator** for build‑time SBOMs as OCI **referrers**. | CI‑side; ephemeral. | +| **Scanner.Sbomer.DockerImage** | `stellaops/scanner-cli` | CLI‑orchestrated scanner container for post‑build scans. | Local/CI; ephemeral. | +| **Concelier.WebService** | `stellaops/concelier-web` | Vulnerability ingest/normalize/merge/export (JSON + Trivy DB). | HA via Mongo locks. | +| **Excititor.WebService** | `stellaops/excititor-web` | VEX ingest/normalize/consensus; conflict retention; exports. | HA via Mongo locks. | +| **Policy Engine** | (in `scanner-web`) | YAML DSL evaluator (waivers, vendor preferences, KEV/EPSS, license, usage‑gating); produces **policy digest**. | In‑process; cache per digest. | +| **Scheduler.WebService** | `stellaops/scheduler-web` | Schedules **re‑evaluation** runs; consumes Concelier/Excititor deltas; selects **impacted images** via BOM‑Index; orchestrates analysis‑only reports. | Stateless API. | +| **Scheduler.Worker** | `stellaops/scheduler-worker` | Executes selection and enqueues batches toward Scanner; enforces rate/limits and windows; maintains impact cursors. | Horizontal; queue‑driven. | +| **Notify.WebService** | `stellaops/notify-web` | Rules engine for outbound notifications; manages channels, templates, throttle/digest logic. | Stateless API. | +| **Notify.Worker** | `stellaops/notify-worker` | Delivers to Slack/Teams/Email/Webhooks; idempotent retries; digests. | Horizontal; per‑channel rate limits. | +| **Signer** | `stellaops/signer` | **Hard gate:** validates entitlement + release integrity; mints signing cert (Fulcio keyless) or uses KMS; signs DSSE. | Stateless; HPA by QPS. | +| **Attestor** | `stellaops/attestor` | Posts DSSE bundles to **Rekor v2**; verification endpoints. | Stateless; HPA by QPS. | +| **Authority** | `stellaops/authority` | On‑prem OIDC issuing **short‑lived OpToks** with DPoP/mTLS sender constraint. | HA behind LB. | +| **Zastava** (Runtime) | `stellaops/zastava` | Runtime inspector/enforcer (observer + optional Admission Webhook). | DaemonSet + Webhook. | +| **Web UI** | `stellaops/ui` | Angular app for scans, diffs, policy, VEX, **Scheduler**, **Notify**, runtime, reports. | Stateless. | +| **StellaOps.Cli** | `stellaops/cli` | CLI for init/scan/export/diff/policy/report/verify; Buildx helper; **schedule** and **notify** verbs. | Local/CI. | ### 1.2 Third‑party (self‑hosted) * **Fulcio** (Sigstore CA) — issues short‑lived signing certs (keyless). * **Rekor v2** (tile‑backed transparency log). * **MinIO** — S3‑compatible object store with lifecycle & Object Lock. -* **MongoDB** — catalog, advisories, VEX. +* **MongoDB** — catalog, advisories, VEX, scheduler, notify. * **Queue** — Redis Streams / NATS / RabbitMQ (pluggable). * **OCI Registry** — must support **Referrers API** (discover SBOMs/signatures). @@ -71,8 +70,12 @@ flowchart LR Auth[Authority (OIDC)\nOpTok (DPoP/mTLS)] SW[Scanner.WebService] WK[Scanner.Worker xN] - FEED[Concelier] - VEX[Excititor] + CONC[Concelier] + EXC[Excititor] + SCHW[Scheduler.Web] + SCH[Scheduler.Worker xN] + NOTW[Notify.Web] + NOT[Notify.Worker xN] POL[Policy Engine (in Scanner.Web)] SGN[Signer\n(entitlement + signing)] ATT[Attestor\n(Rekor v2 submit/verify)] @@ -93,11 +96,19 @@ flowchart LR QUE --> WK WK --> MIN SW --> MGO - FEED --> MGO - VEX --> MGO + CONC --> MGO + EXC --> MGO UI --> SW Z --> SW + %% New event-driven loop + CONC -- export.delta --> SCHW + EXC -- export.delta --> SCHW + SCHW --> SCH + SCH --> SW + SW -- report.ready --> NOTW + Z -- admission/observe --> NOTW + SGN <--> Auth SGN --> FUL SGN -->|mTLS| ATT @@ -106,7 +117,7 @@ flowchart LR SGN <-->|verify referrers| REG ``` -**Trust boundaries.** Only **Signer** can sign; only **Attestor** can write to **Rekor v2**. Scanner/UI never sign. +**Trust boundaries.** Only **Signer** can sign; only **Attestor** can write to **Rekor v2**. Scanner/UI/Scheduler/Notify never sign. --- @@ -116,7 +127,7 @@ flowchart LR * **License Token (LT)** — long‑lived JWT from **Licensing Service**; used **once** to enroll the installation; never used in hot path. * **Proof‑of‑Entitlement (PoE)** — bound to the installation key (mTLS client cert **or** DPoP‑bound JWT with `cnf`); medium‑lived; renewable; revocable. -* **Operational token (OpTok)** — 2–5 min OIDC token from **Authority**, **sender‑constrained** (DPoP or mTLS). Used to authenticate to **Signer**/**Scanner.WebService**. +* **Operational token (OpTok)** — 2–5 min OIDC token from **Authority**, **sender‑constrained** (DPoP or mTLS). Used to authenticate to **Signer**/**Scanner.WebService**/**Scheduler.Web**/**Notify.Web**. **Signer enforces both:** PoE proves entitlement; OpTok proves “who is calling now”. It also **independently verifies** the **scanner image digest** is **Stella Ops‑signed** via **Referrers + cosign** before signing anything. @@ -173,6 +184,11 @@ LS --> IA: PoE (mTLS client cert or JWT with cnf=K_inst), CRL/OCSP/introspect * Buildx **generator** runs analyzers during `docker buildx build --attest=type=sbom,generator=stellaops/sbom-indexer`, attaches SBOMs as **OCI referrers**. * Scanner.WebService can trust these (policy‑configurable) and **skip** re‑scan; DSSE + Rekor v2 can be done either at build time or post‑push via Signer/Attestor. +### 3.5 Events / integrations + +* **Out:** `report.ready` (summary + verdict + Rekor UUID) → internal bus for **Notify** & UI. +* **Expose:** image‑level **BOM‑Index** metadata for **Scheduler** impact selection. + --- ## 4) Backend evaluation (decider) @@ -227,6 +243,8 @@ s3://stellaops/ * `artifacts` (type/format/sha/size/rekor/ttl/immutable/refCount/createdAt) * `images`, `layers`, `links`, `lifecycleRules` +* **Scheduler:** `schedules`, `runs`, `locks`, `impact_cursors` +* **Notify:** `rules`, `deliveries`, `channels`, `templates` **Retention** @@ -239,13 +257,13 @@ s3://stellaops/ ### 7.1 Scanner.WebService ``` -POST /api/scans { imageRef|digest, force? } → { scanId } -GET /api/scans/{id} → { status, digests, artifacts[] } -GET /api/sboms/{imageDigest} ?format=cdx-json|cdx-pb|spdx-json&view=inventory|usage +POST /api/scans { imageRef|digest, force? } → { scanId } +GET /api/scans/{id} → { status, digests, artifacts[] } +GET /api/sboms/{imageDigest} ?format=cdx-json|cdx-pb|spdx-json&view=inventory|usage GET /api/diff?old=&new= → { added[], removed[], changed[], byLayer[] } -POST /api/exports { imageDigest, format, view } → { artifactId, rekorUrl } -POST /api/reports { imageDigest, policyRevision? } → { reportId, rekorUrl } -GET /api/catalog/artifacts/{id} → { size, ttl, immutable, rekor, refs } +POST /api/exports { imageDigest, format, view } → { artifactId, rekorUrl } +POST /api/reports { imageDigest, policyRevision?, vexSnapshot? } → { reportId, verdict, rekorUrl } +GET /api/catalog/artifacts/{id} → { size, ttl, immutable, rekor, refs } GET /healthz | /readyz | /metrics ``` @@ -276,6 +294,25 @@ POST /license/introspect { poe } → { active, claims, exp } POST /attest/endorse { bundle } → endorsement bundle (optional) ``` +### 7.6 Scheduler + +``` +POST /api/v1/scheduler/schedules {yaml|json} → { scheduleId } +GET /api/v1/scheduler/schedules → [ { id, nextRun, status, stats } ] +POST /api/v1/scheduler/run { id|selector } → { runId } +GET /api/v1/scheduler/runs/{id} → { status, counts, links } +GET /api/v1/scheduler/cursor → { lastConcelierExportId, lastExcititorExportId } +``` + +### 7.7 Notify + +``` +POST /api/v1/notify/test { channel, target } → { delivered } +POST /api/v1/notify/rules {yaml|json} → { ruleId } +GET /api/v1/notify/rules → [ { id, match, actions, enabled } ] +GET /api/v1/notify/deliveries → [ { id, eventId, channel, status, attempts } ] +``` + --- ## 8) Security & verifiability @@ -283,8 +320,9 @@ POST /attest/endorse { bundle } → endorsement bundle (optio * **Sender‑constrained tokens.** All operational calls use **DPoP** (RFC 9449) or **mTLS‑bound** tokens (RFC 8705). * **Entitlement.** **PoE** is mandatory; revocation honored online. * **Release integrity.** **Signer** independently verifies **scanner image digest** via **Referrers + cosign** before signing. -* **Separation of duties.** Scanner/UI cannot sign; only **Signer** can sign; only **Attestor** can write to **Rekor v2**. +* **Separation of duties.** Scanner/UI/Scheduler/Notify cannot sign; only **Signer** can sign; only **Attestor** can write to **Rekor v2**. * **Verifiers.** Anyone can verify: DSSE signature → certificate chain to **Stella Ops Fulcio/KMS root** → **Rekor v2** inclusion. +* **RBAC.** Roles: `scanner.admin|read`, `scheduler.admin|read`, `notify.admin|read`, `zastava.admin|read`. * **Community vs Authorized.** Free/community runs throttled with no official attestations; authorized runs full speed and produce **Stella Ops‑verified** bundles. **DSSE predicate (SBOM/report)** @@ -321,6 +359,8 @@ Binary header + purl table + roaring bitmaps; optional `usedByEntrypoint` flags * Build‑time path P95 ≤ 3–5 s on warmed bases. * Post‑build delta scan P95 ≤ 10 s for 200 MB images. * Policy + VEX evaluation ≤ 500 ms for 5k components using BOM‑Index. + * **Event → notification** p95 ≤ **30–60 s** under nominal load. + * **Export delta → re‑evaluation verdict** p95 ≤ **5 min** for 10k impacted images. * **Quotas:** license plan enforces QPS/concurrency/size; **Signer** throttles and can deny DSSE. --- @@ -337,32 +377,37 @@ Binary header + purl table + roaring bitmaps; optional `usedByEntrypoint` flags ```yaml services: - authority: { image: stellaops/authority } - fulcio: { image: sigstore/fulcio } - rekor: { image: sigstore/rekor-v2 } - minio: { image: minio/minio, command: server /data --console-address ":9001" } - mongo: { image: mongo:7 } - signer: { image: stellaops/signer, depends_on: [authority, fulcio] } - attestor: { image: stellaops/attestor, depends_on: [rekor, signer] } - scanner-web:{ image: stellaops/scanner-web, depends_on: [mongo, minio, signer, attestor] } - scanner-worker: - image: stellaops/scanner-worker - deploy: { replicas: 4 } - depends_on: [scanner-web] - concelier: { image: stellaops/concelier-web, depends_on: [mongo] } - excititor: { image: stellaops/excititor-web, depends_on: [mongo] } - ui: { image: stellaops/ui, depends_on: [scanner-web, concelier, excititor] } + authority: { image: stellaops/authority } + fulcio: { image: sigstore/fulcio } + rekor: { image: sigstore/rekor-v2 } + minio: { image: minio/minio, command: server /data --console-address ":9001" } + mongo: { image: mongo:7 } + signer: { image: stellaops/signer, depends_on: [authority, fulcio] } + attestor: { image: stellaops/attestor, depends_on: [rekor, signer] } + scanner-web: { image: stellaops/scanner-web, depends_on: [mongo, minio, signer, attestor] } + scanner-worker: { image: stellaops/scanner-worker, deploy: { replicas: 4 }, depends_on: [scanner-web] } + concelier: { image: stellaops/concelier-web, depends_on: [mongo] } + excititor: { image: stellaops/excititor-web, depends_on: [mongo] } + scheduler-web: { image: stellaops/scheduler-web, depends_on: [mongo] } + scheduler-worker:{ image: stellaops/scheduler-worker, deploy: { replicas: 2 }, depends_on: [scheduler-web] } + notify-web: { image: stellaops/notify-web, depends_on: [mongo] } + notify-worker: { image: stellaops/notify-worker, deploy: { replicas: 2 }, depends_on: [notify-web] } + ui: { image: stellaops/ui, depends_on: [scanner-web, concelier, excititor, scheduler-web, notify-web] } ``` * **Backups:** Mongo dumps; MinIO versioned buckets & replication; Rekor v2 DB snapshots; JWKS/Fulcio/KMS key rotation. +* **Ops runbooks:** Scheduler catch‑up after Concelier/Excititor recovery; connector key rotation (Slack/Teams/SMTP). +* **SLOs & alerts:** lag between Concelier/Excititor export and first rescan verdict; delivery failure rates by channel. --- ## 11) Observability & audit * **Metrics:** scan latency, layer cache hit %, artifact bytes, DSSE/Rekor latency, policy evaluation time, queue depth, admission decisions (Zastava). -* **Tracing:** per‑stage spans; correlation IDs across Scanner→Signer→Attestor. -* **Audit logs:** every signing records `license_id`, `image_digest`, `policy_digest`, and Rekor UUID. +* **Scheduler metrics:** `scheduler.impacted_images_total`, `scheduler.jobs_enqueued_total`, `scheduler.selection_ms`, end‑to‑end p95 (event → verdict). +* **Notify metrics:** `notify.sent_total{channel}`, `notify.dropped_total{reason}`, `notify.digest_coalesced_total`, `notify.latency_ms`. +* **Tracing:** per‑stage spans; correlation IDs across Scanner→Signer→Attestor and Concelier/Excititor→Scheduler→Scanner→Notify. +* **Audit logs:** every signing records `license_id`, `image_digest`, `policy_digest`, and Rekor UUID; Scheduler records who scheduled what; Notify records where, when, and why messages were sent or deduped. * **Compliance:** MinIO **Object Lock** for immutable artifacts; reproducible outputs via policy digest + SBOM digest in predicate. --- @@ -373,11 +418,13 @@ services: * M2: Buildx generator certified flows; cross‑registry trust policies. * M3: Patch‑Presence plugin (signature‑based backport detection), opt‑in. * M3: Zastava Admission control GA with policy presets and dry‑run→enforce stages. +* M3: **Scheduler GA** with export‑delta impact routing and capacity‑aware pacing. +* M3: **Notify GA** with digests, Slack/Teams/Email/Webhooks; **M4:** PagerDuty/Opsgenie connectors. * Continuous: Policy UX (waiver TTLs, vendor rules), Excititor connectors expansion. --- -## 13) Canonical sequences (verification & signing) +## 13) Canonical sequences (verification, re‑evaluation & notify) **Sign & log (OpTok + PoE, image verify, DSSE, Rekor).** @@ -409,22 +456,62 @@ sequenceDiagram end ``` -**Verification (third party).** +**Event‑driven re‑evaluation & notify.** -```plantuml -@startuml -actor Verifier -participant "stellaops verify" as Tool -database "Fulcio/KMS root" as Root -participant "Rekor v2" as R2 -Verifier -> Tool: bundle (URL/file) -Tool -> Tool: Verify DSSE signature -Tool -> Root: Verify cert chain to StellaOps root -Tool -> R2: Verify inclusion proof / query by UUID -Tool -> Verifier: OK + claims (license_id, policy_digest, version) -@enduml +```mermaid +sequenceDiagram + participant CONC as Concelier + participant EXC as Excititor + participant SCH as Scheduler + participant SC as Scanner.WebService + participant NO as Notify + + CONC->>SCH: export.delta {changedProductKeys, exportId} + EXC ->>SCH: export.delta {changedProductKeys, exportId} + SCH->>SCH: Impact select via BOM-Index bitmaps + SCH->>SC: Enqueue analysis-only reports (batches) + SC-->>SCH: verdict stream (PASS/FAIL, deltas) + SCH->>NO: rescan.delta {imageDigest, newCriticals, links} + NO-->>Slack/Teams/Email/Webhook: deliver (throttle/digest rules applied) ``` --- -**End of `high_level_architecture.md` (Consolidated).** +## 14) Minimal data shapes (Scheduler & Notify) + +**Scheduler schedule (YAML via UI/CLI)** + +```yaml +name: nightly-eu +when: "0 2 * * * Europe/Sofia" +mode: analysis-only # or content-refresh +selection: + scope: all-images # or tenant/ns/repo label selectors + onlyIf: { lastReportOlderThanDays: 7 } +notify: + onNewFindings: true + minSeverity: high +limits: + maxJobs: 5000 + ratePerSecond: 50 +``` + +**Notify rule (YAML)** + +```yaml +name: high-critical-alerts +match: + eventKinds: ["report.ready","rescan.delta","zastava.admission"] + minSeverity: high + namespaces: ["prod-*"] + vex: { includeAcceptedJustifications: false } +actions: + - channel: slack + target: "#sec-alerts" + template: "concise" + throttle: "5m" + - channel: email + target: "soc@acme.org" + digest: "hourly" +enabled: true +``` diff --git a/docs/ARCHITECTURE_CLI.md b/docs/ARCHITECTURE_CLI.md index ff822d03..0ef8ee48 100644 --- a/docs/ARCHITECTURE_CLI.md +++ b/docs/ARCHITECTURE_CLI.md @@ -37,6 +37,8 @@ src/ **Language/runtime**: .NET 10 **Native AOT** for speed/startup; Linux builds use **musl** static when possible. +**Plug-in verbs.** Non-core verbs (Excititor, runtime helpers, future integrations) ship as restart-time plug-ins under `plugins/cli/**` with manifest descriptors. The launcher loads plug-ins on startup; hot reloading is intentionally unsupported. + **OS targets**: linux‑x64/arm64, windows‑x64/arm64, macOS‑x64/arm64. --- @@ -386,4 +388,3 @@ script: * macOS: 13–15 (x64, arm64). * Windows: 10/11, Server 2019/2022 (x64, arm64). * Docker engines: Docker Desktop, containerd‑based runners. - diff --git a/docs/ARCHITECTURE_NOTIFY.md b/docs/ARCHITECTURE_NOTIFY.md new file mode 100644 index 00000000..829d0960 --- /dev/null +++ b/docs/ARCHITECTURE_NOTIFY.md @@ -0,0 +1,456 @@ +> **Scope.** Implementation‑ready architecture for **Notify**: a rules‑driven, tenant‑aware notification service that consumes platform events (scan completed, report ready, rescan deltas, attestation logged, admission decisions, etc.), evaluates operator‑defined routing rules, renders **channel‑specific messages** (Slack/Teams/Email/Webhook), and delivers them **reliably** with idempotency, throttling, and digests. It is UI‑managed, auditable, and safe by default (no secrets leakage, no spam storms). + +--- + +## 0) Mission & boundaries + +**Mission.** Convert **facts** from Stella Ops into **actionable, noise‑controlled** signals where teams already live (chat/email/webhooks), with **explainable** reasons and deep links to the UI. + +**Boundaries.** + +* Notify **does not make policy decisions** and **does not rescan**; it **consumes** events from Scanner/Scheduler/Vexer/Feedser/Attestor/Zastava and routes them. +* Attachments are **links** (UI/attestation pages); Notify **does not** attach SBOMs or large blobs to messages. +* Secrets for channels (Slack tokens, SMTP creds) are **referenced**, not stored raw in Mongo. + +--- + +## 1) Runtime shape & projects + +``` +src/ + ├─ StellaOps.Notify.WebService/ # REST: rules/channels CRUD, test send, deliveries browse + ├─ StellaOps.Notify.Worker/ # consumers + evaluators + renderers + delivery workers + ├─ StellaOps.Notify.Connectors.* / # channel plug-ins: Slack, Teams, Email, Webhook (v1) + │ └─ *.Tests/ + ├─ StellaOps.Notify.Engine/ # rules engine, templates, idempotency, digests, throttles + ├─ StellaOps.Notify.Models/ # DTOs (Rule, Channel, Event, Delivery, Template) + ├─ StellaOps.Notify.Storage.Mongo/ # rules, channels, deliveries, digests, locks + ├─ StellaOps.Notify.Queue/ # bus client (Redis Streams/NATS JetStream) + └─ StellaOps.Notify.Tests.* # unit/integration/e2e +``` + +**Deployables**: + +* **Notify.WebService** (stateless API) +* **Notify.Worker** (horizontal scale) + +**Dependencies**: Authority (OpToks; DPoP/mTLS), MongoDB, Redis/NATS (bus), HTTP egress to Slack/Teams/Webhooks, SMTP relay for Email. + +--- + +## 2) Responsibilities + +1. **Ingest** platform events from internal bus with strong ordering per key (e.g., image digest). +2. **Evaluate rules** (tenant‑scoped) with matchers: severity changes, namespaces, repos, labels, KEV flags, provider provenance (VEX), component keys, admission decisions, etc. +3. **Control noise**: **throttle**, **coalesce** (digest windows), and **dedupe** via idempotency keys. +4. **Render** channel‑specific messages using safe templates; include **evidence** and **links**. +5. **Deliver** with retries/backoff; record outcome; expose delivery history to UI. +6. **Test** paths (send test to channel targets) without touching live rules. +7. **Audit**: log who configured what, when, and why a message was sent. + +--- + +## 3) Event model (inputs) + +Notify subscribes to the **internal event bus** (produced by services, escaped JSON; gzip allowed with caps): + +* `scanner.scan.completed` — new SBOM(s) composed; artifacts ready +* `scanner.report.ready` — analysis verdict (policy+vex) available; carries deltas summary +* `scheduler.rescan.delta` — new findings after Feedser/Vexer deltas (already summarized) +* `attestor.logged` — Rekor UUID returned (sbom/report/vex export) +* `zastava.admission` — admit/deny with reasons, namespace, image digests +* `feedser.export.completed` — new export ready (rarely notified directly; usually drives Scheduler) +* `vexer.export.completed` — new consensus snapshot (ditto) + +**Canonical envelope (bus → Notify.Engine):** + +```json +{ + "eventId": "uuid", + "kind": "scanner.report.ready", + "tenant": "tenant-01", + "ts": "2025-10-18T05:41:22Z", + "actor": "scanner-webservice", + "scope": { "namespace":"payments", "repo":"ghcr.io/acme/api", "digest":"sha256:..." }, + "payload": { /* kind-specific fields, see below */ } +} +``` + +**Examples (payload cores):** + +* `scanner.report.ready`: + + ```json + { "verdict":"fail|warn|pass", + "delta": { "newCritical":1, "newHigh":2, "kev":["CVE-2025-..."] }, + "topFindings":[{"purl":"pkg:rpm/openssl","vulnId":"CVE-2025-...","severity":"critical"}], + "links":{"ui":"https://ui/...","rekor":"https://rekor/..."} } + ``` + +* `zastava.admission`: + + ```json + { "decision":"deny|allow", "reasons":["unsigned image","missing SBOM"], + "images":[{"digest":"sha256:...","signed":false,"hasSbom":false}] } + ``` + +--- + +## 4) Rules engine — semantics + +**Rule shape (simplified):** + +```yaml +name: "high-critical-alerts-prod" +enabled: true +match: + eventKinds: ["scanner.report.ready","scheduler.rescan.delta","zastava.admission"] + namespaces: ["prod-*"] + repos: ["ghcr.io/acme/*"] + minSeverity: "high" # min of new findings (delta context) + kev: true # require KEV-tagged or allow any if false + verdict: ["fail","deny"] # filter for report/admission + vex: + includeRejectedJustifications: false # notify only on accepted 'affected' +actions: + - channel: "slack:sec-alerts" # reference to Channel object + template: "concise" + throttle: "5m" + - channel: "email:soc" + digest: "hourly" + template: "detailed" +``` + +**Evaluation order** + +1. **Tenant check** → discard if rule tenant ≠ event tenant. +2. **Kind filter** → discard early. +3. **Scope match** (namespace/repo/labels). +4. **Delta/severity gates** (if event carries `delta`). +5. **VEX gate** (drop if event’s finding is not affected under policy consensus unless rule says otherwise). +6. **Throttling/dedup** (idempotency key) — skip if suppressed. +7. **Actions** → enqueue per‑channel job(s). + +**Idempotency key**: `hash(ruleId | actionId | event.kind | scope.digest | delta.hash | day-bucket)`; ensures “same alert” doesn’t fire more than once within throttle window. + +**Digest windows**: maintain per action a **coalescer**: + +* Window: `5m|15m|1h|1d` (configurable); coalesces events by tenant + namespace/repo or by digest group. +* Digest messages summarize top N items and counts, with safe truncation. + +--- + +## 5) Channels & connectors (plug‑ins) + +Channel config is **two‑part**: a **Channel** record (name, type, options) and a Secret **reference** (Vault/K8s Secret). Connectors are **restart-time plug-ins** discovered on service start (same manifest convention as Concelier/Excititor) and live under `plugins/notify//`. + +**Built‑in v1:** + +* **Slack**: Bot token (xoxb‑…), `chat.postMessage` + `blocks`; rate limit aware (HTTP 429). +* **Microsoft Teams**: Incoming Webhook (or Graph card later); adaptive card payloads. +* **Email (SMTP)**: TLS (STARTTLS or implicit), From/To/CC/BCC; HTML+text alt; DKIM optional. +* **Generic Webhook**: POST JSON with HMAC signature (Ed25519 or SHA‑256) in headers. + +**Connector contract:** (implemented by plug-in assemblies) + +```csharp +public interface INotifyConnector { + string Type { get; } // "slack" | "teams" | "email" | "webhook" | ... + Task SendAsync(DeliveryContext ctx, CancellationToken ct); + Task HealthAsync(ChannelConfig cfg, CancellationToken ct); +} +``` + +**DeliveryContext** includes **rendered content** and **raw event** for audit. + +**Secrets**: `ChannelConfig.secretRef` points to Authority‑managed secret handle or K8s Secret path; workers load at send-time; plug-in manifests (`notify-plugin.json`) declare capabilities and version. + +--- + +## 6) Templates & rendering + +**Template engine**: strongly typed, safe Handlebars‑style; no arbitrary code. Partial templates per channel. Deterministic outputs (prop order, no locale drift unless requested). + +**Variables** (examples): + +* `event.kind`, `event.ts`, `scope.namespace`, `scope.repo`, `scope.digest` +* `payload.verdict`, `payload.delta.newCritical`, `payload.links.ui`, `payload.links.rekor` +* `topFindings[]` with `purl`, `vulnId`, `severity` +* `policy.name`, `policy.revision` (if available) + +**Helpers**: + +* `severity_icon(sev)`, `link(text,url)`, `pluralize(n, "finding")`, `truncate(text, n)`, `code(text)`. + +**Channel mapping**: + +* Slack: title + blocks, limited to 50 blocks/3000 chars per section; long lists → link to UI. +* Teams: Adaptive Card schema 1.5; fallback text for older channels. +* Email: HTML + text; inline table of top N findings, rest behind UI link. +* Webhook: JSON with `event`, `ruleId`, `actionId`, `summary`, `links`, and raw `payload` subset. + +**i18n**: template set per locale (English default; Bulgarian built‑in). + +--- + +## 7) Data model (Mongo) + +**Database**: `notify` + +* `rules` + + ``` + { _id, tenantId, name, enabled, match, actions, createdBy, updatedBy, createdAt, updatedAt } + ``` + +* `channels` + + ``` + { _id, tenantId, name:"slack:sec-alerts", type:"slack", + config:{ webhookUrl?:"", channel:"#sec-alerts", workspace?: "...", secretRef:"ref://..." }, + createdAt, updatedAt } + ``` + +* `deliveries` + + ``` + { _id, tenantId, ruleId, actionId, eventId, kind, scope, status:"sent|failed|throttled|digested|dropped", + attempts:[{ts, status, code, reason}], + rendered:{ title, body, target }, // redacted for PII; body hash stored + sentAt, lastError? } + ``` + +* `digests` + + ``` + { _id, tenantId, actionKey, window:"hourly", openedAt, items:[{eventId, scope, delta}], status:"open|flushed" } + ``` + +* `throttles` + + ``` + { key:"idem:", ttlAt } // short-lived, also cached in Redis + ``` + +**Indexes**: rules by `{tenantId, enabled}`, deliveries by `{tenantId, sentAt desc}`, digests by `{tenantId, actionKey}`. + +--- + +## 8) External APIs (WebService) + +Base path: `/api/v1/notify` (Authority OpToks; scopes: `notify.admin` for write, `notify.read` for view). + +* **Channels** + + * `POST /channels` | `GET /channels` | `GET /channels/{id}` | `PATCH /channels/{id}` | `DELETE /channels/{id}` + * `POST /channels/{id}/test` → send sample message (no rule evaluation) + * `GET /channels/{id}/health` → connector self‑check + +* **Rules** + + * `POST /rules` | `GET /rules` | `GET /rules/{id}` | `PATCH /rules/{id}` | `DELETE /rules/{id}` + * `POST /rules/{id}/test` → dry‑run rule against a **sample event** (no delivery unless `--send`) + +* **Deliveries** + + * `GET /deliveries?tenant=...&since=...` → list + * `GET /deliveries/{id}` → detail (redacted body + metadata) + * `POST /deliveries/{id}/retry` → force retry (admin) + +* **Admin** + + * `GET /stats` (per tenant counts, last hour/day) + * `GET /healthz|readyz` (liveness) + +**Ingestion**: workers do **not** expose public ingestion; they **subscribe** to the internal bus. (Optional `/events/test` for integration testing, admin‑only.) + +--- + +## 9) Delivery pipeline (worker) + +``` +[Event bus] → [Ingestor] → [RuleMatcher] → [Throttle/Dedupe] → [DigestCoalescer] → [Renderer] → [Connector] → [Result] + └────────→ [DeliveryStore] +``` + +* **Ingestor**: N consumers with per‑key ordering (key = tenant|digest|namespace). +* **RuleMatcher**: loads active rules snapshot for tenant into memory; vectorized predicate check. +* **Throttle/Dedupe**: consult Redis + Mongo `throttles`; if hit → record `status=throttled`. +* **DigestCoalescer**: append to open digest window or flush when timer expires. +* **Renderer**: select template (channel+locale), inject variables, enforce length limits, compute `bodyHash`. +* **Connector**: send; handle provider‑specific rate limits and backoffs; `maxAttempts` with exponential jitter; overflow → DLQ (dead‑letter topic) + UI surfacing. + +**Idempotency**: per action **idempotency key** stored in Redis (TTL = `throttle window` or `digest window`). Connectors also respect **provider** idempotency where available (e.g., Slack `client_msg_id`). + +--- + +## 10) Reliability & rate controls + +* **Per‑tenant** RPM caps (default 600/min) + **per‑channel** concurrency (Slack 1–4, Teams 1–2, Email 8–32 based on relay). +* **Backoff** map: Slack 429 → respect `Retry‑After`; SMTP 4xx → retry; 5xx → retry with jitter; permanent rejects → drop with status recorded. +* **DLQ**: NATS/Redis stream `notify.dlq` with `{event, rule, action, error}` for operator inspection; UI shows DLQ items. + +--- + +## 11) Security & privacy + +* **AuthZ**: all APIs require **Authority** OpToks; actions scoped by tenant. +* **Secrets**: `secretRef` only; Notify fetches just‑in‑time from Authority Secret proxy or K8s Secret (mounted). No plaintext secrets in Mongo. +* **Egress TLS**: validate SSL; pin domains per channel config; optional CA bundle override for on‑prem SMTP. +* **Webhook signing**: HMAC or Ed25519 signatures in `X-StellaOps-Signature` + replay‑window timestamp; include canonical body hash in header. +* **Redaction**: deliveries store **hashes** of bodies, not full payloads for chat/email to minimize PII retention (configurable). +* **Quiet hours**: per tenant (e.g., 22:00–06:00) route high‑sev only; defer others to digests. +* **Loop prevention**: Webhook target allowlist + event origin tags; do not ingest own webhooks. + +--- + +## 12) Observability (Prometheus + OTEL) + +* `notify.events_consumed_total{kind}` +* `notify.rules_matched_total{ruleId}` +* `notify.throttled_total{reason}` +* `notify.digest_coalesced_total{window}` +* `notify.sent_total{channel}` / `notify.failed_total{channel,code}` +* `notify.delivery_latency_seconds{channel}` (end‑to‑end) +* **Tracing**: spans `ingest`, `match`, `render`, `send`; correlation id = `eventId`. + +**SLO targets** + +* Event→delivery p95 **≤ 30–60 s** under nominal load. +* Failure rate p95 **< 0.5%** per hour (excluding provider outages). +* Duplicate rate **≈ 0** (idempotency working). + +--- + +## 13) Configuration (YAML) + +```yaml +notify: + authority: + issuer: "https://authority.internal" + require: "dpop" # or "mtls" + bus: + kind: "redis" # or "nats" + streams: + - "scanner.events" + - "scheduler.events" + - "attestor.events" + - "zastava.events" + mongo: + uri: "mongodb://mongo/notify" + limits: + perTenantRpm: 600 + perChannel: + slack: { concurrency: 2 } + teams: { concurrency: 1 } + email: { concurrency: 8 } + webhook: { concurrency: 8 } + digests: + defaultWindow: "1h" + maxItems: 100 + quietHours: + enabled: true + window: "22:00-06:00" + minSeverity: "critical" + webhooks: + sign: + method: "ed25519" # or "hmac-sha256" + keyRef: "ref://notify/webhook-sign-key" +``` + +--- + +## 14) UI touch‑points + +* **Notifications → Channels**: add Slack/Teams/Email/Webhook; run **health**; rotate secrets. +* **Notifications → Rules**: create/edit YAML rules with linting; test with sample events; see match rate. +* **Notifications → Deliveries**: timeline with filters (status, channel, rule); inspect last error; retry. +* **Digest preview**: shows current window contents and when it will flush. +* **Quiet hours**: configure per tenant; show overrides. +* **DLQ**: browse dead‑letters; requeue after fix. + +--- + +## 15) Failure modes & responses + +| Condition | Behavior | +| ----------------------------------- | ------------------------------------------------------------------------------------- | +| Slack 429 / Teams 429 | Respect `Retry‑After`, backoff with jitter, reduce concurrency | +| SMTP transient 4xx | Retry up to `maxAttempts`; escalate to DLQ on exhaust | +| Invalid channel secret | Mark channel unhealthy; suppress sends; surface in UI | +| Rule explosion (matches everything) | Safety valve: per‑tenant RPM caps; auto‑pause rule after X drops; UI alert | +| Bus outage | Buffer to local queue (bounded); resume consuming when healthy | +| Mongo slowness | Fall back to Redis throttles; batch write deliveries; shed low‑priority notifications | + +--- + +## 16) Testing matrix + +* **Unit**: matchers, throttle math, digest coalescing, idempotency keys, template rendering edge cases. +* **Connectors**: provider‑level rate limits, payload size truncation, error mapping. +* **Integration**: synthetic event storm (10k/min), ensure p95 latency & duplicate rate. +* **Security**: DPoP/mTLS on APIs; secretRef resolution; webhook signing & replay windows. +* **i18n**: localized templates render deterministically. +* **Chaos**: Slack/Teams API flaps; SMTP greylisting; Redis hiccups; ensure graceful degradation. + +--- + +## 17) Sequences (representative) + +**A) New criticals after Feedser delta (Slack immediate + Email hourly digest)** + +```mermaid +sequenceDiagram + autonumber + participant SCH as Scheduler + participant NO as Notify.Worker + participant SL as Slack + participant SMTP as Email + + SCH->>NO: bus event scheduler.rescan.delta { newCritical:1, digest:sha256:... } + NO->>NO: match rules (Slack immediate; Email hourly digest) + NO->>SL: chat.postMessage (concise) + SL-->>NO: 200 OK + NO->>NO: append to digest window (email:soc) + Note over NO: At window close → render digest email + NO->>SMTP: send email (detailed digest) + SMTP-->>NO: 250 OK +``` + +**B) Admission deny (Teams card + Webhook)** + +```mermaid +sequenceDiagram + autonumber + participant ZA as Zastava + participant NO as Notify.Worker + participant TE as Teams + participant WH as Webhook + + ZA->>NO: bus event zastava.admission { decision: "deny", reasons: [...] } + NO->>TE: POST adaptive card + TE-->>NO: 200 OK + NO->>WH: POST JSON (signed) + WH-->>NO: 2xx +``` + +--- + +## 18) Implementation notes + +* **Language**: .NET 10; minimal API; `System.Text.Json` with canonical writer for body hashing; Channels for pipelines. +* **Bus**: Redis Streams (**XGROUP** consumers) or NATS JetStream for at‑least‑once with ack; per‑tenant consumer groups to localize backpressure. +* **Templates**: compile and cache per rule+channel+locale; version with rule `updatedAt` to invalidate. +* **Rules**: store raw YAML + parsed AST; validate with schema + static checks (e.g., nonsensical combos). +* **Secrets**: pluggable secret resolver (Authority Secret proxy, K8s, Vault). +* **Rate limiting**: `System.Threading.RateLimiting` + per‑connector adapters. + +--- + +## 19) Roadmap (post‑v1) + +* **PagerDuty/Opsgenie** connectors; **Jira** ticket creation. +* **User inbox** (in‑app notifications) + mobile push via webhook relay. +* **Anomaly suppression**: auto‑pause noisy rules with hints (learned thresholds). +* **Graph rules**: “only notify if *not_affected → affected* transition at consensus layer”. +* **Label enrichment**: pluggable taggers (business criticality, data classification) to refine matchers. diff --git a/docs/ARCHITECTURE_SCANNER.md b/docs/ARCHITECTURE_SCANNER.md index 952a0d94..c45f51b6 100644 --- a/docs/ARCHITECTURE_SCANNER.md +++ b/docs/ARCHITECTURE_SCANNER.md @@ -40,6 +40,8 @@ src/ └─ StellaOps.Scanner.Sbomer.DockerImage/ # CLI‑driven scanner container ``` +Analyzer assemblies and buildx generators are packaged as **restart-time plug-ins** under `plugins/scanner/**` with manifests; services must restart to activate new plug-ins. + **Runtime form‑factor:** two deployables * **Scanner.WebService** (stateless REST) @@ -410,4 +412,3 @@ vector purls map components optional map usedByEntrypoint ``` - diff --git a/docs/ARCHITECTURE_SCHEDULER.md b/docs/ARCHITECTURE_SCHEDULER.md new file mode 100644 index 00000000..2f605501 --- /dev/null +++ b/docs/ARCHITECTURE_SCHEDULER.md @@ -0,0 +1,424 @@ +# component_architecture_scheduler.md — **Stella Ops Scheduler** (2025Q4) + +> **Scope.** Implementation‑ready architecture for **Scheduler**: a service that (1) **re‑evaluates** already‑cataloged images when intel changes (Feedser/Vexer/policy), (2) orchestrates **nightly** and **ad‑hoc** runs, (3) targets only the **impacted** images using the BOM‑Index, and (4) emits **report‑ready** events that downstream **Notify** fans out. Default mode is **analysis‑only** (no image pull); optional **content‑refresh** can be enabled per schedule. + +--- + +## 0) Mission & boundaries + +**Mission.** Keep scan results **current** without rescanning the world. When new advisories or VEX claims land, **pinpoint** affected images and ask the backend to recompute **verdicts** against the **existing SBOMs**. Surface only **meaningful deltas** to humans and ticket queues. + +**Boundaries.** + +* Scheduler **does not** compute SBOMs and **does not** sign. It calls Scanner/WebService’s **/reports (analysis‑only)** endpoint and lets the backend (Policy + Vexer + Feedser) decide PASS/FAIL. +* Scheduler **may** ask Scanner to **content‑refresh** selected targets (e.g., mutable tags) but the default is **no** image pull. +* Notifications are **not** sent directly; Scheduler emits events consumed by **Notify**. + +--- + +## 1) Runtime shape & projects + +``` +src/ + ├─ StellaOps.Scheduler.WebService/ # REST (schedules CRUD, runs, admin) + ├─ StellaOps.Scheduler.Worker/ # planners + runners (N replicas) + ├─ StellaOps.Scheduler.ImpactIndex/ # purl→images inverted index (roaring bitmaps) + ├─ StellaOps.Scheduler.Models/ # DTOs (Schedule, Run, ImpactSet, Deltas) + ├─ StellaOps.Scheduler.Storage.Mongo/ # schedules, runs, cursors, locks + ├─ StellaOps.Scheduler.Queue/ # Redis Streams / NATS abstraction + ├─ StellaOps.Scheduler.Tests.* # unit/integration/e2e +``` + +**Deployables**: + +* **Scheduler.WebService** (stateless) +* **Scheduler.Worker** (scale‑out; planners + executors) + +**Dependencies**: Authority (OpTok + DPoP/mTLS), Scanner.WebService, Feedser, Vexer, MongoDB, Redis/NATS, (optional) Notify. + +--- + +## 2) Core responsibilities + +1. **Time‑based** runs: cron windows per tenant/timezone (e.g., “02:00 Europe/Sofia”). +2. **Event‑driven** runs: react to **Feedser export** and **Vexer export** deltas (changed product keys / advisories / claims). +3. **Impact targeting**: map changes to **image sets** using a **global inverted index** built from Scanner’s per‑image **BOM‑Index** sidecars. +4. **Run planning**: shard, pace, and rate‑limit jobs to avoid thundering herds. +5. **Execution**: call Scanner **/reports (analysis‑only)** or **/scans (content‑refresh)**; aggregate **delta** results. +6. **Events**: publish `rescan.delta` and `report.ready` summaries for **Notify** & **UI**. +7. **Control plane**: CRUD schedules, **pause/resume**, dry‑run previews, audit. + +--- + +## 3) Data model (Mongo) + +**Database**: `scheduler` + +* `schedules` + + ``` + { _id, tenantId, name, enabled, whenCron, timezone, + mode: "analysis-only" | "content-refresh", + selection: { scope: "all-images" | "by-namespace" | "by-repo" | "by-digest" | "by-labels", + includeTags?: ["prod-*"], digests?: [sha256...], resolvesTags?: bool }, + onlyIf: { lastReportOlderThanDays?: int, policyRevision?: string }, + notify: { onNewFindings: bool, minSeverity: "low|medium|high|critical", includeKEV: bool }, + limits: { maxJobs?: int, ratePerSecond?: int, parallelism?: int }, + createdAt, updatedAt, createdBy, updatedBy } + ``` + +* `runs` + + ``` + { _id, scheduleId?, tenantId, trigger: "cron|feedser|vexer|manual", + reason?: { feedserExportId?, vexerExportId?, cursor? }, + state: "planning|queued|running|completed|error|cancelled", + stats: { candidates: int, deduped: int, queued: int, completed: int, deltas: int, newCriticals: int }, + startedAt, finishedAt, error? } + ``` + +* `impact_cursors` + + ``` + { _id: tenantId, feedserLastExportId, vexerLastExportId, updatedAt } + ``` + +* `locks` (singleton schedulers, run leases) + +* `audit` (CRUD actions, run outcomes) + +**Indexes**: + +* `schedules` on `{tenantId, enabled}`, `{whenCron}`. +* `runs` on `{tenantId, startedAt desc}`, `{state}`. +* TTL optional for completed runs (e.g., 180 days). + +--- + +## 4) ImpactIndex (global inverted index) + +Goal: translate **change keys** → **image sets** in **milliseconds**. + +**Source**: Scanner produces per‑image **BOM‑Index** sidecars (purls, and `usedByEntrypoint` bitmaps). Scheduler ingests/refreshes them to build a **global** index. + +**Representation**: + +* Assign **image IDs** (dense ints) to catalog images. +* Keep **Roaring Bitmaps**: + + * `Contains[purl] → bitmap(imageIds)` + * `UsedBy[purl] → bitmap(imageIds)` (subset of Contains) +* Optionally keep **Owner maps**: `{imageId → {tenantId, namespaces[], repos[]}}` for selection filters. +* Persist in RocksDB/LMDB or Redis‑modules; cache hot shards in memory; snapshot to Mongo for cold start. + +**Update paths**: + +* On new/updated image SBOM: **merge** per‑image set into global maps. +* On image remove/expiry: **clear** id from bitmaps. + +**API (internal)**: + +```csharp +IImpactIndex { + ImpactSet ResolveByPurls(IEnumerable purls, bool usageOnly, Selector sel); + ImpactSet ResolveByVulns(IEnumerable vulnIds, bool usageOnly, Selector sel); // optional (vuln->purl precomputed by Feedser) + ImpactSet ResolveAll(Selector sel); // for nightly +} +``` + +**Selector filters**: tenant, namespaces, repos, labels, digest allowlists, `includeTags` patterns. + +--- + +## 5) External interfaces (REST) + +Base path: `/api/v1/scheduler` (Authority OpToks; scopes: `scheduler.read`, `scheduler.admin`). + +### 5.1 Schedules CRUD + +* `POST /schedules` → create +* `GET /schedules` → list (filter by tenant) +* `GET /schedules/{id}` → details + next run +* `PATCH /schedules/{id}` → pause/resume/update +* `DELETE /schedules/{id}` → delete (soft delete, optional) + +### 5.2 Run control & introspection + +* `POST /run` — ad‑hoc run + + ```json + { "mode": "analysis-only|content-refresh", "selection": {...}, "reason": "manual" } + ``` +* `GET /runs` — list with paging +* `GET /runs/{id}` — status, stats, links to deltas +* `POST /runs/{id}/cancel` — best‑effort cancel + +### 5.3 Previews (dry‑run) + +* `POST /preview/impact` — returns **candidate count** and a small sample of impacted digests for given change keys or selection. + +### 5.4 Event webhooks (optional push from Feedser/Vexer) + +* `POST /events/feedser-export` + + ```json + { "exportId":"...", "changedProductKeys":["pkg:rpm/openssl", ...], "kev": ["CVE-..."], "window": { "from":"...","to":"..." } } + ``` +* `POST /events/vexer-export` + + ```json + { "exportId":"...", "changedClaims":[ { "productKey":"pkg:deb/...", "vulnId":"CVE-...", "status":"not_affected→affected"} ], ... } + ``` + +**Security**: webhook requires **mTLS** or an **HMAC** `X-Scheduler-Signature` (Ed25519 / SHA‑256) plus Authority token. + +--- + +## 6) Planner → Runner pipeline + +### 6.1 Planning algorithm (event‑driven) + +``` +On Export Event (Feedser/Vexer): + keys = Normalize(change payload) # productKeys or vulnIds→productKeys + usageOnly = schedule/policy hint? # default true + sel = Selector for tenant/scope from schedules subscribed to events + + impacted = ImpactIndex.ResolveByPurls(keys, usageOnly, sel) + impacted = ApplyOwnerFilters(impacted, sel) # namespaces/repos/labels + impacted = DeduplicateByDigest(impacted) + impacted = EnforceLimits(impacted, limits.maxJobs) + shards = Shard(impacted, byHashPrefix, n=limits.parallelism) + + For each shard: + Enqueue RunSegment (runId, shard, rate=limits.ratePerSecond) +``` + +**Fairness & pacing** + +* Use **leaky bucket** per tenant and per registry host. +* Prioritize **KEV‑tagged** and **critical** first if oversubscribed. + +### 6.2 Nightly planning + +``` +At cron tick: + sel = resolve selection + candidates = ImpactIndex.ResolveAll(sel) + if lastReportOlderThanDays present → filter by report age (via Scanner catalog) + shard & enqueue as above +``` + +### 6.3 Execution (Runner) + +* Pop **RunSegment** job → for each image digest: + + * **analysis‑only**: `POST scanner/reports { imageDigest, policyRevision? }` + * **content‑refresh**: resolve tag→digest if needed; `POST scanner/scans { imageRef, attest? false }` then `POST /reports` +* Collect **delta**: `newFindings`, `newCriticals`/`highs`, `links` (UI deep link, Rekor if present). +* Persist per‑image outcome in `runs.{id}.stats` (incremental counters). +* Emit `scheduler.rescan.delta` events to **Notify** only when **delta > 0** and matches severity rule. + +--- + +## 7) Event model (outbound) + +**Topic**: `rescan.delta` (internal bus → Notify; UI subscribes via backend). + +```json +{ + "tenant": "tenant-01", + "runId": "324af…", + "imageDigest": "sha256:…", + "newCriticals": 1, + "newHigh": 2, + "kevHits": ["CVE-2025-..."], + "topFindings": [ + { "purl":"pkg:rpm/openssl@3.0.12-...","vulnId":"CVE-2025-...","severity":"critical","link":"https://ui/scans/..." } + ], + "reportUrl": "https://ui/.../scans/sha256:.../report", + "attestation": { "uuid":"rekor-uuid", "verified": true }, + "ts": "2025-10-18T03:12:45Z" +} +``` + +**Also**: `report.ready` for “no‑change” summaries (digest + zero delta), which Notify can ignore by rule. + +--- + +## 8) Security posture + +* **AuthN/Z**: Authority OpToks with `aud=scheduler`; DPoP (preferred) or mTLS. +* **Multi‑tenant**: every schedule, run, and event carries `tenantId`; ImpactIndex filters by tenant‑visible images. +* **Webhook** callers (Feedser/Vexer) present **mTLS** or **HMAC** and Authority token. +* **Input hardening**: size caps on changed key lists; reject >100k keys per event; compress (zstd/gzip) allowed with limits. +* **No secrets** in logs; redact tokens and signatures. + +--- + +## 9) Observability & SLOs + +**Metrics (Prometheus)** + +* `scheduler.events_total{source, result}` +* `scheduler.impact_resolve_seconds{quantile}` +* `scheduler.images_selected_total{mode}` +* `scheduler.jobs_enqueued_total{mode}` +* `scheduler.run_latency_seconds{quantile}` // event → first verdict +* `scheduler.delta_images_total{severity}` +* `scheduler.rate_limited_total{reason}` + +**Targets** + +* Resolve 10k changed keys → impacted set in **<300 ms** (hot cache). +* Event → first rescan verdict in **≤60 s** (p95). +* Nightly coverage 50k images in **≤10 min** with 10 workers (analysis‑only). + +**Tracing** (OTEL): spans `plan`, `resolve`, `enqueue`, `report_call`, `persist`, `emit`. + +--- + +## 10) Configuration (YAML) + +```yaml +scheduler: + authority: + issuer: "https://authority.internal" + require: "dpop" # or "mtls" + queue: + kind: "redis" # or "nats" + url: "redis://redis:6379/4" + mongo: + uri: "mongodb://mongo/scheduler" + impactIndex: + storage: "rocksdb" # "rocksdb" | "redis" | "memory" + warmOnStart: true + usageOnlyDefault: true + limits: + defaultRatePerSecond: 50 + defaultParallelism: 8 + maxJobsPerRun: 50000 + integrates: + scannerUrl: "https://scanner-web.internal" + feedserWebhook: true + vexerWebhook: true + notifications: + emitBus: "internal" # deliver to Notify via internal bus +``` + +--- + +## 11) UI touch‑points + +* **Schedules** page: CRUD, enable/pause, next run, last run stats, mode (analysis/content), selector preview. +* **Runs** page: timeline; heat‑map of deltas; drill‑down to affected images. +* **Dry‑run preview** modal: “This Feedser export touches ~3,214 images; projected deltas: ~420 (34 KEV).” + +--- + +## 12) Failure modes & degradations + +| Condition | Behavior | +| ------------------------------------ | ---------------------------------------------------------------------------------------- | +| ImpactIndex cold / incomplete | Fall back to **All** selection for nightly; for events, cap to KEV+critical until warmed | +| Feedser/Vexer webhook storm | Coalesce by exportId; debounce 30–60 s; keep last | +| Scanner under load (429) | Backoff with jitter; respect per‑tenant/leaky bucket | +| Oversubscription (too many impacted) | Prioritize KEV/critical first; spillover to next window; UI banner shows backlog | +| Notify down | Buffer outbound events in queue (TTL 24h) | +| Mongo slow | Cut batch sizes; sample‑log; alert ops; don’t drop runs unless critical | + +--- + +## 13) Testing matrix + +* **ImpactIndex**: correctness (purl→image sets), performance, persistence after restart, memory pressure with 1M purls. +* **Planner**: dedupe, shard, fairness, limit enforcement, KEV prioritization. +* **Runner**: parallel report calls, error backoff, partial failures, idempotency. +* **End‑to‑end**: Feedser export → deltas visible in UI in ≤60 s. +* **Security**: webhook auth (mTLS/HMAC), DPoP nonce dance, tenant isolation. +* **Chaos**: drop scanner availability; simulate registry throttles (content‑refresh mode). +* **Nightly**: cron tick correctness across timezones and DST. + +--- + +## 14) Implementation notes + +* **Language**: .NET 10 minimal API; Channels‑based pipeline; `System.Threading.RateLimiting`. +* **Bitmaps**: Roaring via `RoaringBitmap` bindings; memory‑map large shards if RocksDB used. +* **Cron**: Quartz‑style parser with timezone support; clock skew tolerated ±60 s. +* **Dry‑run**: use ImpactIndex only; never call scanner. +* **Idempotency**: run segments carry deterministic keys; retries safe. +* **Backpressure**: per‑tenant buckets; per‑host registry budgets respected when content‑refresh enabled. + +--- + +## 15) Sequences (representative) + +**A) Event‑driven rescan (Feedser delta)** + +```mermaid +sequenceDiagram + autonumber + participant FE as Feedser + participant SCH as Scheduler.Worker + participant IDX as ImpactIndex + participant SC as Scanner.WebService + participant NO as Notify + + FE->>SCH: POST /events/feedser-export {exportId, changedProductKeys} + SCH->>IDX: ResolveByPurls(keys, usageOnly=true, sel) + IDX-->>SCH: bitmap(imageIds) → digests list + SCH->>SC: POST /reports {imageDigest} (batch/sequenced) + SC-->>SCH: report deltas (new criticals/highs) + alt delta>0 + SCH->>NO: rescan.delta {digest, newCriticals, links} + end +``` + +**B) Nightly rescan** + +```mermaid +sequenceDiagram + autonumber + participant CRON as Cron + participant SCH as Scheduler.Worker + participant IDX as ImpactIndex + participant SC as Scanner.WebService + + CRON->>SCH: tick (02:00 Europe/Sofia) + SCH->>IDX: ResolveAll(selector) + IDX-->>SCH: candidates + SCH->>SC: POST /reports {digest} (paced) + SC-->>SCH: results + SCH-->>SCH: aggregate, store run stats +``` + +**C) Content‑refresh (tag followers)** + +```mermaid +sequenceDiagram + autonumber + participant SCH as Scheduler + participant SC as Scanner + SCH->>SC: resolve tag→digest (if changed) + alt digest changed + SCH->>SC: POST /scans {imageRef} # new SBOM + SC-->>SCH: scan complete (artifacts) + SCH->>SC: POST /reports {imageDigest} + else unchanged + SCH->>SC: POST /reports {imageDigest} # analysis-only + end +``` + +--- + +## 16) Roadmap + +* **Vuln‑centric impact**: pre‑join vuln→purl→images to rank by **KEV** and **exploited‑in‑the‑wild** signals. +* **Policy diff preview**: when a staged policy changes, show projected breakage set before promotion. +* **Cross‑cluster federation**: one Scheduler instance driving many Scanner clusters (tenant isolation). +* **Windows containers**: integrate Zastava runtime hints for Usage view tightening. + +--- + +**End — component_architecture_scheduler.md** diff --git a/docs/README.md b/docs/README.md index 0a9767cc..03ec1b65 100755 --- a/docs/README.md +++ b/docs/README.md @@ -31,31 +31,33 @@ Everything here is open‑source and versioned — when you check out a git ta - **03 – [Vision & Road‑map](03_VISION.md)** - **04 – [Feature Matrix](04_FEATURE_MATRIX.md)** -### Reference & concepts -- **05 – [System Requirements Specification](05_SYSTEM_REQUIREMENTS_SPEC.md)** -- **07 – [High‑Level Architecture](07_HIGH_LEVEL_ARCHITECTURE.md)** -- **08 – Module Architecture Dossiers** - - [Scanner](ARCHITECTURE_SCANNER.md) - - [Concelier](ARCHITECTURE_CONCELIER.md) - - [Excititor](ARCHITECTURE_EXCITITOR.md) - - [Signer](ARCHITECTURE_SIGNER.md) - - [Attestor](ARCHITECTURE_ATTESTOR.md) - - [Authority](ARCHITECTURE_AUTHORITY.md) - - [CLI](ARCHITECTURE_CLI.md) - - [Web UI](ARCHITECTURE_UI.md) - - [Zastava Runtime](ARCHITECTURE_ZASTAVA.md) - - [Release & Operations](ARCHITECTURE_DEVOPS.md) -- **09 – [API & CLI Reference](09_API_CLI_REFERENCE.md)** -- **10 – [Plug‑in SDK Guide](10_PLUGIN_SDK_GUIDE.md)** -- **10 – [Concelier CLI Quickstart](10_CONCELIER_CLI_QUICKSTART.md)** -- **30 – [Excititor Connector Packaging Guide](dev/30_EXCITITOR_CONNECTOR_GUIDE.md)** -- **30 – Developer Templates** - - [Excititor Connector Skeleton](dev/templates/excititor-connector/) -- **11 – [Authority Service](11_AUTHORITY.md)** -- **11 – [Data Schemas](11_DATA_SCHEMAS.md)** -- **12 – [Performance Workbook](12_PERFORMANCE_WORKBOOK.md)** -- **13 – [Release‑Engineering Playbook](13_RELEASE_ENGINEERING_PLAYBOOK.md)** -- **30 – [Fixture Maintenance](dev/fixtures.md)** +### Reference & concepts +- **05 – [System Requirements Specification](05_SYSTEM_REQUIREMENTS_SPEC.md)** +- **07 – [High‑Level Architecture](07_HIGH_LEVEL_ARCHITECTURE.md)** +- **08 – Module Architecture Dossiers** + - [Scanner](ARCHITECTURE_SCANNER.md) + - [Concelier](ARCHITECTURE_CONCELIER.md) + - [Excititor](ARCHITECTURE_EXCITITOR.md) + - [Signer](ARCHITECTURE_SIGNER.md) + - [Attestor](ARCHITECTURE_ATTESTOR.md) + - [Authority](ARCHITECTURE_AUTHORITY.md) + - [Notify](ARCHITECTURE_NOTIFY.md) + - [Scheduler](ARCHITECTURE_SCHEDULER.md) + - [CLI](ARCHITECTURE_CLI.md) + - [Web UI](ARCHITECTURE_UI.md) + - [Zastava Runtime](ARCHITECTURE_ZASTAVA.md) + - [Release & Operations](ARCHITECTURE_DEVOPS.md) +- **09 – [API & CLI Reference](09_API_CLI_REFERENCE.md)** +- **10 – [Plug‑in SDK Guide](10_PLUGIN_SDK_GUIDE.md)** +- **10 – [Concelier CLI Quickstart](10_CONCELIER_CLI_QUICKSTART.md)** +- **30 – [Excititor Connector Packaging Guide](dev/30_EXCITITOR_CONNECTOR_GUIDE.md)** +- **30 – Developer Templates** + - [Excititor Connector Skeleton](dev/templates/excititor-connector/) +- **11 – [Authority Service](11_AUTHORITY.md)** +- **11 – [Data Schemas](11_DATA_SCHEMAS.md)** +- **12 – [Performance Workbook](12_PERFORMANCE_WORKBOOK.md)** +- **13 – [Release‑Engineering Playbook](13_RELEASE_ENGINEERING_PLAYBOOK.md)** +- **30 – [Fixture Maintenance](dev/fixtures.md)** ### User & operator guides - **14 – [Glossary](14_GLOSSARY_OF_TERMS.md)** @@ -64,18 +66,18 @@ Everything here is open‑source and versioned — when you check out a git ta - **18 – [Coding Standards](18_CODING_STANDARDS.md)** - **19 – [Test‑Suite Overview](19_TEST_SUITE_OVERVIEW.md)** - **21 – [Install Guide](21_INSTALL_GUIDE.md)** -- **22 – [CI/CD Recipes Library](ci/20_CI_RECIPES.md)** -- **23 – [FAQ](23_FAQ_MATRIX.md)** -- **24 – [Offline Update Kit Admin Guide](24_OFFLINE_KIT.md)** -- **25 – [Concelier Apple Connector Operations](ops/concelier-apple-operations.md)** -- **26 – [Authority Key Rotation Playbook](ops/authority-key-rotation.md)** -- **27 – [Concelier CCCS Connector Operations](ops/concelier-cccs-operations.md)** -- **28 – [Concelier CISA ICS Connector Operations](ops/concelier-icscisa-operations.md)** -- **29 – [Concelier CERT-Bund Connector Operations](ops/concelier-certbund-operations.md)** -- **30 – [Concelier MSRC Connector – AAD Onboarding](ops/concelier-msrc-operations.md)** - -### Legal & licence -- **31 – [Legal & Quota FAQ](29_LEGAL_FAQ_QUOTA.md)** +- **22 – [CI/CD Recipes Library](ci/20_CI_RECIPES.md)** +- **23 – [FAQ](23_FAQ_MATRIX.md)** +- **24 – [Offline Update Kit Admin Guide](24_OFFLINE_KIT.md)** +- **25 – [Concelier Apple Connector Operations](ops/concelier-apple-operations.md)** +- **26 – [Authority Key Rotation Playbook](ops/authority-key-rotation.md)** +- **27 – [Concelier CCCS Connector Operations](ops/concelier-cccs-operations.md)** +- **28 – [Concelier CISA ICS Connector Operations](ops/concelier-icscisa-operations.md)** +- **29 – [Concelier CERT-Bund Connector Operations](ops/concelier-certbund-operations.md)** +- **30 – [Concelier MSRC Connector – AAD Onboarding](ops/concelier-msrc-operations.md)** + +### Legal & licence +- **31 – [Legal & Quota FAQ](29_LEGAL_FAQ_QUOTA.md)** diff --git a/docs/TASKS.md b/docs/TASKS.md index 4763b7b8..9e4a7c74 100644 --- a/docs/TASKS.md +++ b/docs/TASKS.md @@ -9,6 +9,9 @@ | DOC5.Concelier-Runbook | DONE (2025-10-12) | Docs Guild | DOC3.Concelier-Authority | Produce dedicated Concelier authority audit runbook covering log fields, monitoring recommendations, and troubleshooting steps. | ✅ Runbook published; ✅ linked from DOC3/DOC5; ✅ alerting guidance included. | | FEEDDOCS-DOCS-05-001 | DONE (2025-10-11) | Docs Guild | FEEDMERGE-ENGINE-04-001, FEEDMERGE-ENGINE-04-002 | Publish Concelier conflict resolution runbook covering precedence workflow, merge-event auditing, and Sprint 3 metrics. | ✅ `docs/ops/concelier-conflict-resolution.md` committed; ✅ metrics/log tables align with latest merge code; ✅ Ops alert guidance handed to Concelier team. | | FEEDDOCS-DOCS-05-002 | DONE (2025-10-16) | Docs Guild, Concelier Ops | FEEDDOCS-DOCS-05-001 | Ops sign-off captured: conflict runbook circulated, alert thresholds tuned, and rollout decisions documented in change log. | ✅ Ops review recorded; ✅ alert thresholds finalised using `docs/ops/concelier-authority-audit-runbook.md`; ✅ change-log entry linked from runbook once GHSA/NVD/OSV regression fixtures land. | +| DOCS-ADR-09-001 | TODO | Docs Guild, DevEx | — | Establish ADR process (`docs/adr/0000-template.md`) and document usage guidelines. | Template published; README snippet linking ADR process; announcement posted. | +| DOCS-EVENTS-09-002 | TODO | Docs Guild, Platform Events | SCANNER-EVENTS-15-201 | Publish event schema catalog (`docs/events/`) for `scanner.report.ready@1`, `scheduler.rescan.delta@1`, `attestor.logged@1`. | Schemas validated; docs/events/README summarises usage; Notify/Scheduler teams acknowledge. | +| DOCS-RUNTIME-17-004 | TODO | Docs Guild, Runtime Guild | SCANNER-EMIT-17-701, ZASTAVA-OBS-17-005, DEVOPS-REL-17-002 | Document build-id workflows: SBOM exposure, runtime event payloads, debug-store layout, and operator guidance for symbol retrieval. | Architecture + operator docs updated with build-id sections, examples show `readelf` output + debuginfod usage, references linked from Offline Kit/Release guides. | > Update statuses (TODO/DOING/REVIEW/DONE/BLOCKED) as progress changes. Keep guides in sync with configuration samples under `etc/`. diff --git a/docs/adr/0000-template.md b/docs/adr/0000-template.md new file mode 100644 index 00000000..20d091da --- /dev/null +++ b/docs/adr/0000-template.md @@ -0,0 +1,18 @@ +# ADR-0000: Title + +## Status +Proposed + +## Context +- What decision needs to be made? +- What are the forces (requirements, constraints, stakeholders)? + +## Decision +- Summary of the chosen option. + +## Consequences +- Positive/negative consequences. +- Follow-up actions or tasks. + +## References +- Links to related ADRs, issues, documents. diff --git a/docs/events/README.md b/docs/events/README.md new file mode 100644 index 00000000..500077f3 --- /dev/null +++ b/docs/events/README.md @@ -0,0 +1,9 @@ +# Event Envelope Schemas + +Versioned JSON Schemas for platform events consumed by Scheduler, Notify, and UI. + +- `scanner.report.ready@1.json` +- `scheduler.rescan.delta@1.json` +- `attestor.logged@1.json` + +Producers must bump the version suffix when introducing breaking changes; consumers validate incoming payloads against these schemas. diff --git a/docs/events/attestor.logged@1.json b/docs/events/attestor.logged@1.json new file mode 100644 index 00000000..2c1e9475 --- /dev/null +++ b/docs/events/attestor.logged@1.json @@ -0,0 +1,38 @@ +{ + "$id": "https://stella-ops.org/schemas/events/attestor.logged@1.json", + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "required": ["eventId", "kind", "tenant", "ts", "payload"], + "properties": { + "eventId": {"type": "string", "format": "uuid"}, + "kind": {"const": "attestor.logged"}, + "tenant": {"type": "string"}, + "ts": {"type": "string", "format": "date-time"}, + "payload": { + "type": "object", + "required": ["artifactSha256", "rekor", "subject"], + "properties": { + "artifactSha256": {"type": "string"}, + "rekor": { + "type": "object", + "required": ["uuid", "url"], + "properties": { + "uuid": {"type": "string"}, + "url": {"type": "string", "format": "uri"}, + "index": {"type": "integer", "minimum": 0} + } + }, + "subject": { + "type": "object", + "required": ["type", "name"], + "properties": { + "type": {"enum": ["sbom", "report", "vex-export"]}, + "name": {"type": "string"} + } + } + }, + "additionalProperties": true + } + }, + "additionalProperties": false +} diff --git a/docs/events/scanner.report.ready@1.json b/docs/events/scanner.report.ready@1.json new file mode 100644 index 00000000..b3376c3f --- /dev/null +++ b/docs/events/scanner.report.ready@1.json @@ -0,0 +1,46 @@ +{ + "$id": "https://stella-ops.org/schemas/events/scanner.report.ready@1.json", + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "required": ["eventId", "kind", "tenant", "ts", "scope", "payload"], + "properties": { + "eventId": {"type": "string", "format": "uuid"}, + "kind": {"const": "scanner.report.ready"}, + "tenant": {"type": "string"}, + "ts": {"type": "string", "format": "date-time"}, + "scope": { + "type": "object", + "required": ["repo", "digest"], + "properties": { + "namespace": {"type": "string"}, + "repo": {"type": "string"}, + "digest": {"type": "string"} + } + }, + "payload": { + "type": "object", + "required": ["verdict", "delta", "links"], + "properties": { + "verdict": {"enum": ["pass", "warn", "fail"]}, + "delta": { + "type": "object", + "properties": { + "newCritical": {"type": "integer", "minimum": 0}, + "newHigh": {"type": "integer", "minimum": 0}, + "kev": {"type": "array", "items": {"type": "string"}} + } + }, + "links": { + "type": "object", + "properties": { + "ui": {"type": "string", "format": "uri"}, + "rekor": {"type": "string", "format": "uri"} + }, + "additionalProperties": false + } + }, + "additionalProperties": true + } + }, + "additionalProperties": false +} diff --git a/docs/events/scheduler.rescan.delta@1.json b/docs/events/scheduler.rescan.delta@1.json new file mode 100644 index 00000000..3fe7bd65 --- /dev/null +++ b/docs/events/scheduler.rescan.delta@1.json @@ -0,0 +1,33 @@ +{ + "$id": "https://stella-ops.org/schemas/events/scheduler.rescan.delta@1.json", + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "required": ["eventId", "kind", "tenant", "ts", "payload"], + "properties": { + "eventId": {"type": "string", "format": "uuid"}, + "kind": {"const": "scheduler.rescan.delta"}, + "tenant": {"type": "string"}, + "ts": {"type": "string", "format": "date-time"}, + "payload": { + "type": "object", + "required": ["scheduleId", "impactedDigests", "summary"], + "properties": { + "scheduleId": {"type": "string"}, + "impactedDigests": { + "type": "array", + "items": {"type": "string"} + }, + "summary": { + "type": "object", + "properties": { + "newCritical": {"type": "integer", "minimum": 0}, + "newHigh": {"type": "integer", "minimum": 0}, + "total": {"type": "integer", "minimum": 0} + } + } + }, + "additionalProperties": true + } + }, + "additionalProperties": false +} diff --git a/node_modules/.package-lock.json b/node_modules/.package-lock.json new file mode 100644 index 00000000..9337962d --- /dev/null +++ b/node_modules/.package-lock.json @@ -0,0 +1,6 @@ +{ + "name": "git.stella-ops.org", + "lockfileVersion": 3, + "requires": true, + "packages": {} +} diff --git a/ops/deployment/AGENTS.md b/ops/deployment/AGENTS.md new file mode 100644 index 00000000..57491442 --- /dev/null +++ b/ops/deployment/AGENTS.md @@ -0,0 +1,4 @@ +# Deployment & Operations — Agent Charter + +## Mission +Maintain deployment/upgrade/rollback workflows (Helm/Compose) per `docs/ARCHITECTURE_DEVOPS.md` including environment-specific configs. diff --git a/ops/deployment/TASKS.md b/ops/deployment/TASKS.md new file mode 100644 index 00000000..3c157986 --- /dev/null +++ b/ops/deployment/TASKS.md @@ -0,0 +1,5 @@ +# Deployment Task Board + +| ID | Status | Owner(s) | Depends on | Description | Exit Criteria | +|----|--------|----------|------------|-------------|---------------| +| DEVOPS-OPS-14-003 | TODO | Deployment Guild | DEVOPS-REL-14-001 | Document and script upgrade/rollback flows, channel management, and compatibility matrices per architecture. | Helm/Compose guides updated with digest pinning, automated checks committed, rollback drill recorded. | diff --git a/ops/devops/AGENTS.md b/ops/devops/AGENTS.md new file mode 100644 index 00000000..8334244d --- /dev/null +++ b/ops/devops/AGENTS.md @@ -0,0 +1,11 @@ +# DevOps & Release — Agent Charter + +## Mission +Execute deterministic build/release pipeline per `docs/ARCHITECTURE_DEVOPS.md`: +- Reproducible builds with SBOM/provenance, cosign signing, transparency logging. +- Channel manifests (LTS/Stable/Edge) with digests, Helm/Compose profiles. +- Performance guard jobs ensuring budgets. + +## Expectations +- Coordinate with Scanner/Scheduler/Notify teams for artifact availability. +- Maintain CI reliability; update `TASKS.md` as states change. diff --git a/ops/devops/TASKS.md b/ops/devops/TASKS.md new file mode 100644 index 00000000..a4871844 --- /dev/null +++ b/ops/devops/TASKS.md @@ -0,0 +1,9 @@ +# DevOps Task Board + +| ID | Status | Owner(s) | Depends on | Description | Exit Criteria | +|----|--------|----------|------------|-------------|---------------| +| DEVOPS-HELM-09-001 | TODO | DevOps Guild | SCANNER-WEB-09-101 | Create Helm/Compose environment profiles (dev, staging, airgap) with deterministic digests. | Profiles committed under `deploy/`; docs updated; CI smoke deploy passes. | +| DEVOPS-PERF-10-001 | TODO | DevOps Guild | BENCH-SCANNER-10-001 | Add perf smoke job (SBOM compose <5 s target) to CI. | CI job runs sample build verifying <5 s; alerts configured. | +| DEVOPS-REL-14-001 | TODO | DevOps Guild | SIGNER-API-11-101, ATTESTOR-API-11-201 | Deterministic build/release pipeline with SBOM/provenance, signing, manifest generation. | CI pipeline produces signed images + SBOM/attestations, manifests published with verified hashes, docs updated. | +| DEVOPS-REL-17-002 | TODO | DevOps Guild | DEVOPS-REL-14-001, SCANNER-EMIT-17-701 | Persist stripped-debug artifacts organised by GNU build-id and bundle them into release/offline kits with checksum manifests. | CI job writes `.debug` files under `artifacts/debug/.build-id/`, manifest + checksums published, offline kit includes cache, smoke job proves symbol lookup via build-id. | +| DEVOPS-MIRROR-08-001 | TODO | DevOps Guild | DEVOPS-REL-14-001 | Stand up managed mirror profiles for `*.stella-ops.org` (Concelier/Excititor), including Helm/Compose overlays, multi-tenant secrets, CDN caching, and sync documentation. | Infra overlays committed, CI smoke deploy hits mirror endpoints, runbooks published for downstream sync and quota management. | diff --git a/ops/licensing/AGENTS.md b/ops/licensing/AGENTS.md new file mode 100644 index 00000000..d682c625 --- /dev/null +++ b/ops/licensing/AGENTS.md @@ -0,0 +1,4 @@ +# Licensing & Registry Access — Agent Charter + +## Mission +Implement licensing token service and registry access workflows described in `docs/ARCHITECTURE_DEVOPS.md`. diff --git a/ops/licensing/TASKS.md b/ops/licensing/TASKS.md new file mode 100644 index 00000000..8aec44f4 --- /dev/null +++ b/ops/licensing/TASKS.md @@ -0,0 +1,5 @@ +# Licensing Task Board + +| ID | Status | Owner(s) | Depends on | Description | Exit Criteria | +|----|--------|----------|------------|-------------|---------------| +| DEVOPS-LIC-14-004 | TODO | Licensing Guild | AUTH-MTLS-11-002 | Implement registry token service tied to Authority (DPoP/mTLS), plan gating, revocation handling, and monitoring per architecture. | Token service issues scoped tokens, revocation tested, monitoring dashboards in place, docs updated. | diff --git a/ops/offline-kit/AGENTS.md b/ops/offline-kit/AGENTS.md new file mode 100644 index 00000000..02938a1e --- /dev/null +++ b/ops/offline-kit/AGENTS.md @@ -0,0 +1,4 @@ +# Offline Kit — Agent Charter + +## Mission +Package Offline Update Kit per `docs/ARCHITECTURE_DEVOPS.md` and `docs/24_OFFLINE_KIT.md` with deterministic digests and import tooling. diff --git a/ops/offline-kit/TASKS.md b/ops/offline-kit/TASKS.md new file mode 100644 index 00000000..bdc0d926 --- /dev/null +++ b/ops/offline-kit/TASKS.md @@ -0,0 +1,5 @@ +# Offline Kit Task Board + +| ID | Status | Owner(s) | Depends on | Description | Exit Criteria | +|----|--------|----------|------------|-------------|---------------| +| DEVOPS-OFFLINE-14-002 | TODO | Offline Kit Guild | DEVOPS-REL-14-001 | Build offline kit packaging workflow (artifact bundling, manifest generation, signature verification). | Offline tarball generated with manifest + checksums + signatures; import script verifies integrity; docs updated. | diff --git a/package-lock.json b/package-lock.json new file mode 100644 index 00000000..9337962d --- /dev/null +++ b/package-lock.json @@ -0,0 +1,6 @@ +{ + "name": "git.stella-ops.org", + "lockfileVersion": 3, + "requires": true, + "packages": {} +} diff --git a/package.json b/package.json new file mode 100644 index 00000000..0967ef42 --- /dev/null +++ b/package.json @@ -0,0 +1 @@ +{} diff --git a/samples/TASKS.md b/samples/TASKS.md new file mode 100644 index 00000000..699b39ce --- /dev/null +++ b/samples/TASKS.md @@ -0,0 +1,5 @@ +# Samples Task Board + +| ID | Status | Owner(s) | Depends on | Description | Exit Criteria | +|----|--------|----------|------------|-------------|---------------| +| SAMPLES-10-001 | TODO | Samples Guild, Scanner Team | SCANNER-EMIT-10-605 | Curate sample images (nginx, alpine+busybox, distroless+go, .NET AOT, python venv, npm monorepo) with expected SBOM/BOM-Index sidecars. | Samples committed under `samples/`; golden SBOM/BOM-Index files present; documented usage. | diff --git a/src/StellaOps.Cli.Tests/Commands/CommandHandlersTests.cs b/src/StellaOps.Cli.Tests/Commands/CommandHandlersTests.cs index c3ca71be..687efe2a 100644 --- a/src/StellaOps.Cli.Tests/Commands/CommandHandlersTests.cs +++ b/src/StellaOps.Cli.Tests/Commands/CommandHandlersTests.cs @@ -1,6 +1,7 @@ using System; using System.Collections.Generic; using System.IO; +using System.Net.Http; using System.Security.Cryptography; using System.Text; using System.Text.Json; @@ -211,6 +212,113 @@ public sealed class CommandHandlersTests } } + [Fact] + public async Task HandleExcititorInitAsync_CallsBackend() + { + var original = Environment.ExitCode; + try + { + var backend = new StubBackendClient(new JobTriggerResult(true, "accepted", null, null)); + var provider = BuildServiceProvider(backend); + + await CommandHandlers.HandleExcititorInitAsync( + provider, + new[] { "redhat" }, + resume: true, + verbose: false, + cancellationToken: CancellationToken.None); + + Assert.Equal(0, Environment.ExitCode); + Assert.Equal("init", backend.LastExcititorRoute); + Assert.Equal(HttpMethod.Post, backend.LastExcititorMethod); + var payload = Assert.IsAssignableFrom>(backend.LastExcititorPayload); + Assert.Equal(true, payload["resume"]); + var providers = Assert.IsAssignableFrom>(payload["providers"]!); + Assert.Contains("redhat", providers, StringComparer.OrdinalIgnoreCase); + } + finally + { + Environment.ExitCode = original; + } + } + + [Fact] + public async Task HandleExcititorListProvidersAsync_WritesOutput() + { + var original = Environment.ExitCode; + try + { + var backend = new StubBackendClient(new JobTriggerResult(true, "ok", null, null)) + { + ProviderSummaries = new[] + { + new ExcititorProviderSummary("redhat", "distro", "Red Hat", "vendor", true, DateTimeOffset.UtcNow) + } + }; + + var provider = BuildServiceProvider(backend); + await CommandHandlers.HandleExcititorListProvidersAsync(provider, includeDisabled: false, verbose: false, cancellationToken: CancellationToken.None); + + Assert.Equal(0, Environment.ExitCode); + } + finally + { + Environment.ExitCode = original; + } + } + + [Fact] + public async Task HandleExcititorVerifyAsync_FailsWithoutArguments() + { + var original = Environment.ExitCode; + try + { + var backend = new StubBackendClient(new JobTriggerResult(true, "ok", null, null)); + var provider = BuildServiceProvider(backend); + + await CommandHandlers.HandleExcititorVerifyAsync(provider, null, null, null, verbose: false, cancellationToken: CancellationToken.None); + + Assert.Equal(1, Environment.ExitCode); + } + finally + { + Environment.ExitCode = original; + } + } + + [Fact] + public async Task HandleExcititorVerifyAsync_AttachesAttestationFile() + { + var original = Environment.ExitCode; + using var tempFile = new TempFile("attestation.json", Encoding.UTF8.GetBytes("{\"ok\":true}")); + try + { + var backend = new StubBackendClient(new JobTriggerResult(true, "ok", null, null)); + var provider = BuildServiceProvider(backend); + + await CommandHandlers.HandleExcititorVerifyAsync( + provider, + exportId: "export-123", + digest: "sha256:abc", + attestationPath: tempFile.Path, + verbose: false, + cancellationToken: CancellationToken.None); + + Assert.Equal(0, Environment.ExitCode); + Assert.Equal("verify", backend.LastExcititorRoute); + var payload = Assert.IsAssignableFrom>(backend.LastExcititorPayload); + Assert.Equal("export-123", payload["exportId"]); + Assert.Equal("sha256:abc", payload["digest"]); + var attestation = Assert.IsAssignableFrom>(payload["attestation"]!); + Assert.Equal(Path.GetFileName(tempFile.Path), attestation["fileName"]); + Assert.NotNull(attestation["base64"]); + } + finally + { + Environment.ExitCode = original; + } + } + [Theory] [InlineData(null)] [InlineData("default")] @@ -502,33 +610,49 @@ public sealed class CommandHandlersTests return new StubExecutor(new ScannerExecutionResult(0, tempResultsFile, tempMetadataFile)); } - private sealed class StubBackendClient : IBackendOperationsClient - { - private readonly JobTriggerResult _result; - - public StubBackendClient(JobTriggerResult result) - { - _result = result; - } - - public string? LastJobKind { get; private set; } - public string? LastUploadPath { get; private set; } - - public Task DownloadScannerAsync(string channel, string outputPath, bool overwrite, bool verbose, CancellationToken cancellationToken) - => throw new NotImplementedException(); - - public Task UploadScanResultsAsync(string filePath, CancellationToken cancellationToken) - { - LastUploadPath = filePath; - return Task.CompletedTask; - } - - public Task TriggerJobAsync(string jobKind, IDictionary parameters, CancellationToken cancellationToken) - { - LastJobKind = jobKind; - return Task.FromResult(_result); - } - } + private sealed class StubBackendClient : IBackendOperationsClient + { + private readonly JobTriggerResult _jobResult; + + public StubBackendClient(JobTriggerResult result) + { + _jobResult = result; + } + + public string? LastJobKind { get; private set; } + public string? LastUploadPath { get; private set; } + public string? LastExcititorRoute { get; private set; } + public HttpMethod? LastExcititorMethod { get; private set; } + public object? LastExcititorPayload { get; private set; } + public ExcititorOperationResult? ExcititorResult { get; set; } = new ExcititorOperationResult(true, "ok", null, null); + public IReadOnlyList ProviderSummaries { get; set; } = Array.Empty(); + + public Task DownloadScannerAsync(string channel, string outputPath, bool overwrite, bool verbose, CancellationToken cancellationToken) + => throw new NotImplementedException(); + + public Task UploadScanResultsAsync(string filePath, CancellationToken cancellationToken) + { + LastUploadPath = filePath; + return Task.CompletedTask; + } + + public Task TriggerJobAsync(string jobKind, IDictionary parameters, CancellationToken cancellationToken) + { + LastJobKind = jobKind; + return Task.FromResult(_jobResult); + } + + public Task ExecuteExcititorOperationAsync(string route, HttpMethod method, object? payload, CancellationToken cancellationToken) + { + LastExcititorRoute = route; + LastExcititorMethod = method; + LastExcititorPayload = payload; + return Task.FromResult(ExcititorResult ?? new ExcititorOperationResult(true, "ok", null, null)); + } + + public Task> GetExcititorProvidersAsync(bool includeDisabled, CancellationToken cancellationToken) + => Task.FromResult(ProviderSummaries); + } private sealed class StubExecutor : IScannerExecutor { diff --git a/src/StellaOps.Cli.Tests/Testing/TestHelpers.cs b/src/StellaOps.Cli.Tests/Testing/TestHelpers.cs index 561e27da..28c35051 100644 --- a/src/StellaOps.Cli.Tests/Testing/TestHelpers.cs +++ b/src/StellaOps.Cli.Tests/Testing/TestHelpers.cs @@ -1,14 +1,15 @@ using System; -using System.Collections.Generic; -using System.IO; -using System.Net.Http; -using System.Threading; -using System.Threading.Tasks; +using System.Collections.Generic; +using System.IO; +using System.Net.Http; +using System.Threading; +using System.Threading.Tasks; +using System.Text; namespace StellaOps.Cli.Tests.Testing; -internal sealed class TempDirectory : IDisposable -{ +internal sealed class TempDirectory : IDisposable +{ public TempDirectory() { Path = System.IO.Path.Combine(System.IO.Path.GetTempPath(), $"stellaops-cli-tests-{Guid.NewGuid():N}"); @@ -31,7 +32,41 @@ internal sealed class TempDirectory : IDisposable // ignored } } -} +} + +internal sealed class TempFile : IDisposable +{ + public TempFile(string fileName, byte[] contents) + { + var directory = System.IO.Path.Combine(System.IO.Path.GetTempPath(), $"stellaops-cli-file-{Guid.NewGuid():N}"); + Directory.CreateDirectory(directory); + Path = System.IO.Path.Combine(directory, fileName); + File.WriteAllBytes(Path, contents); + } + + public string Path { get; } + + public void Dispose() + { + try + { + if (File.Exists(Path)) + { + File.Delete(Path); + } + + var directory = System.IO.Path.GetDirectoryName(Path); + if (!string.IsNullOrEmpty(directory) && Directory.Exists(directory)) + { + Directory.Delete(directory, recursive: true); + } + } + catch + { + // ignored intentionally + } + } +} internal sealed class StubHttpMessageHandler : HttpMessageHandler { diff --git a/src/StellaOps.Cli/Commands/CommandFactory.cs b/src/StellaOps.Cli/Commands/CommandFactory.cs index 576d3835..7f7566fb 100644 --- a/src/StellaOps.Cli/Commands/CommandFactory.cs +++ b/src/StellaOps.Cli/Commands/CommandFactory.cs @@ -24,6 +24,7 @@ internal static class CommandFactory root.Add(BuildScannerCommand(services, verboseOption, cancellationToken)); root.Add(BuildScanCommand(services, options, verboseOption, cancellationToken)); root.Add(BuildDatabaseCommand(services, verboseOption, cancellationToken)); + root.Add(BuildExcititorCommand(services, verboseOption, cancellationToken)); root.Add(BuildAuthCommand(services, options, verboseOption, cancellationToken)); root.Add(BuildConfigCommand(options)); @@ -220,10 +221,191 @@ internal static class CommandFactory db.Add(fetch); db.Add(merge); - db.Add(export); + db.Add(export); return db; } + private static Command BuildExcititorCommand(IServiceProvider services, Option verboseOption, CancellationToken cancellationToken) + { + var excititor = new Command("excititor", "Manage Excititor ingest, exports, and reconciliation workflows."); + + var init = new Command("init", "Initialize Excititor ingest state."); + var initProviders = new Option("--provider", new[] { "-p" }) + { + Description = "Optional provider identifier(s) to initialize.", + Arity = ArgumentArity.ZeroOrMore + }; + var resumeOption = new Option("--resume") + { + Description = "Resume ingest from the last persisted checkpoint instead of starting fresh." + }; + init.Add(initProviders); + init.Add(resumeOption); + init.SetAction((parseResult, _) => + { + var providers = parseResult.GetValue(initProviders) ?? Array.Empty(); + var resume = parseResult.GetValue(resumeOption); + var verbose = parseResult.GetValue(verboseOption); + return CommandHandlers.HandleExcititorInitAsync(services, providers, resume, verbose, cancellationToken); + }); + + var pull = new Command("pull", "Trigger Excititor ingest for configured providers."); + var pullProviders = new Option("--provider", new[] { "-p" }) + { + Description = "Optional provider identifier(s) to ingest.", + Arity = ArgumentArity.ZeroOrMore + }; + var sinceOption = new Option("--since") + { + Description = "Optional ISO-8601 timestamp to begin the ingest window." + }; + var windowOption = new Option("--window") + { + Description = "Optional window duration (e.g. 24:00:00)." + }; + var forceOption = new Option("--force") + { + Description = "Force ingestion even if the backend reports no pending work." + }; + pull.Add(pullProviders); + pull.Add(sinceOption); + pull.Add(windowOption); + pull.Add(forceOption); + pull.SetAction((parseResult, _) => + { + var providers = parseResult.GetValue(pullProviders) ?? Array.Empty(); + var since = parseResult.GetValue(sinceOption); + var window = parseResult.GetValue(windowOption); + var force = parseResult.GetValue(forceOption); + var verbose = parseResult.GetValue(verboseOption); + return CommandHandlers.HandleExcititorPullAsync(services, providers, since, window, force, verbose, cancellationToken); + }); + + var resume = new Command("resume", "Resume Excititor ingest using a checkpoint token."); + var resumeProviders = new Option("--provider", new[] { "-p" }) + { + Description = "Optional provider identifier(s) to resume.", + Arity = ArgumentArity.ZeroOrMore + }; + var checkpointOption = new Option("--checkpoint") + { + Description = "Optional checkpoint identifier to resume from." + }; + resume.Add(resumeProviders); + resume.Add(checkpointOption); + resume.SetAction((parseResult, _) => + { + var providers = parseResult.GetValue(resumeProviders) ?? Array.Empty(); + var checkpoint = parseResult.GetValue(checkpointOption); + var verbose = parseResult.GetValue(verboseOption); + return CommandHandlers.HandleExcititorResumeAsync(services, providers, checkpoint, verbose, cancellationToken); + }); + + var list = new Command("list-providers", "List Excititor providers and their ingest status."); + var includeDisabledOption = new Option("--include-disabled") + { + Description = "Include disabled providers in the listing." + }; + list.Add(includeDisabledOption); + list.SetAction((parseResult, _) => + { + var includeDisabled = parseResult.GetValue(includeDisabledOption); + var verbose = parseResult.GetValue(verboseOption); + return CommandHandlers.HandleExcititorListProvidersAsync(services, includeDisabled, verbose, cancellationToken); + }); + + var export = new Command("export", "Trigger Excititor export generation."); + var formatOption = new Option("--format") + { + Description = "Export format (e.g. openvex, json)." + }; + var exportDeltaOption = new Option("--delta") + { + Description = "Request a delta export when supported." + }; + var exportScopeOption = new Option("--scope") + { + Description = "Optional policy scope or tenant identifier." + }; + var exportSinceOption = new Option("--since") + { + Description = "Optional ISO-8601 timestamp to restrict export contents." + }; + var exportProviderOption = new Option("--provider") + { + Description = "Optional provider identifier when requesting targeted exports." + }; + export.Add(formatOption); + export.Add(exportDeltaOption); + export.Add(exportScopeOption); + export.Add(exportSinceOption); + export.Add(exportProviderOption); + export.SetAction((parseResult, _) => + { + var format = parseResult.GetValue(formatOption) ?? "openvex"; + var delta = parseResult.GetValue(exportDeltaOption); + var scope = parseResult.GetValue(exportScopeOption); + var since = parseResult.GetValue(exportSinceOption); + var provider = parseResult.GetValue(exportProviderOption); + var verbose = parseResult.GetValue(verboseOption); + return CommandHandlers.HandleExcititorExportAsync(services, format, delta, scope, since, provider, verbose, cancellationToken); + }); + + var verify = new Command("verify", "Verify Excititor exports or attestations."); + var exportIdOption = new Option("--export-id") + { + Description = "Export identifier to verify." + }; + var digestOption = new Option("--digest") + { + Description = "Expected digest for the export or attestation." + }; + var attestationOption = new Option("--attestation") + { + Description = "Path to a local attestation file to verify (base64 content will be uploaded)." + }; + verify.Add(exportIdOption); + verify.Add(digestOption); + verify.Add(attestationOption); + verify.SetAction((parseResult, _) => + { + var exportId = parseResult.GetValue(exportIdOption); + var digest = parseResult.GetValue(digestOption); + var attestation = parseResult.GetValue(attestationOption); + var verbose = parseResult.GetValue(verboseOption); + return CommandHandlers.HandleExcititorVerifyAsync(services, exportId, digest, attestation, verbose, cancellationToken); + }); + + var reconcile = new Command("reconcile", "Trigger Excititor reconciliation against canonical advisories."); + var reconcileProviders = new Option("--provider", new[] { "-p" }) + { + Description = "Optional provider identifier(s) to reconcile.", + Arity = ArgumentArity.ZeroOrMore + }; + var maxAgeOption = new Option("--max-age") + { + Description = "Optional maximum age window (e.g. 7.00:00:00)." + }; + reconcile.Add(reconcileProviders); + reconcile.Add(maxAgeOption); + reconcile.SetAction((parseResult, _) => + { + var providers = parseResult.GetValue(reconcileProviders) ?? Array.Empty(); + var maxAge = parseResult.GetValue(maxAgeOption); + var verbose = parseResult.GetValue(verboseOption); + return CommandHandlers.HandleExcititorReconcileAsync(services, providers, maxAge, verbose, cancellationToken); + }); + + excititor.Add(init); + excititor.Add(pull); + excititor.Add(resume); + excititor.Add(list); + excititor.Add(export); + excititor.Add(verify); + excititor.Add(reconcile); + return excititor; + } + private static Command BuildAuthCommand(IServiceProvider services, StellaOpsCliOptions options, Option verboseOption, CancellationToken cancellationToken) { var auth = new Command("auth", "Manage authentication with StellaOps Authority."); diff --git a/src/StellaOps.Cli/Commands/CommandHandlers.cs b/src/StellaOps.Cli/Commands/CommandHandlers.cs index 3fcb77a8..f932fba7 100644 --- a/src/StellaOps.Cli/Commands/CommandHandlers.cs +++ b/src/StellaOps.Cli/Commands/CommandHandlers.cs @@ -4,6 +4,8 @@ using System.Collections.Generic; using System.Diagnostics; using System.Globalization; using System.IO; +using System.Linq; +using System.Net.Http; using System.Security.Cryptography; using System.Text.Json; using System.Text; @@ -340,6 +342,310 @@ internal static class CommandHandlers } } + public static Task HandleExcititorInitAsync( + IServiceProvider services, + IReadOnlyList providers, + bool resume, + bool verbose, + CancellationToken cancellationToken) + { + var normalizedProviders = NormalizeProviders(providers); + var payload = new Dictionary(StringComparer.Ordinal); + if (normalizedProviders.Count > 0) + { + payload["providers"] = normalizedProviders; + } + if (resume) + { + payload["resume"] = true; + } + + return ExecuteExcititorCommandAsync( + services, + commandName: "excititor init", + verbose, + new Dictionary + { + ["providers"] = normalizedProviders.Count, + ["resume"] = resume + }, + client => client.ExecuteExcititorOperationAsync("init", HttpMethod.Post, RemoveNullValues(payload), cancellationToken), + cancellationToken); + } + + public static Task HandleExcititorPullAsync( + IServiceProvider services, + IReadOnlyList providers, + DateTimeOffset? since, + TimeSpan? window, + bool force, + bool verbose, + CancellationToken cancellationToken) + { + var normalizedProviders = NormalizeProviders(providers); + var payload = new Dictionary(StringComparer.Ordinal); + if (normalizedProviders.Count > 0) + { + payload["providers"] = normalizedProviders; + } + if (since.HasValue) + { + payload["since"] = since.Value.ToUniversalTime().ToString("O", CultureInfo.InvariantCulture); + } + if (window.HasValue) + { + payload["window"] = window.Value.ToString("c", CultureInfo.InvariantCulture); + } + if (force) + { + payload["force"] = true; + } + + return ExecuteExcititorCommandAsync( + services, + commandName: "excititor pull", + verbose, + new Dictionary + { + ["providers"] = normalizedProviders.Count, + ["force"] = force, + ["since"] = since?.ToUniversalTime().ToString("O", CultureInfo.InvariantCulture), + ["window"] = window?.ToString("c", CultureInfo.InvariantCulture) + }, + client => client.ExecuteExcititorOperationAsync("ingest/run", HttpMethod.Post, RemoveNullValues(payload), cancellationToken), + cancellationToken); + } + + public static Task HandleExcititorResumeAsync( + IServiceProvider services, + IReadOnlyList providers, + string? checkpoint, + bool verbose, + CancellationToken cancellationToken) + { + var normalizedProviders = NormalizeProviders(providers); + var payload = new Dictionary(StringComparer.Ordinal); + if (normalizedProviders.Count > 0) + { + payload["providers"] = normalizedProviders; + } + if (!string.IsNullOrWhiteSpace(checkpoint)) + { + payload["checkpoint"] = checkpoint.Trim(); + } + + return ExecuteExcititorCommandAsync( + services, + commandName: "excititor resume", + verbose, + new Dictionary + { + ["providers"] = normalizedProviders.Count, + ["checkpoint"] = checkpoint + }, + client => client.ExecuteExcititorOperationAsync("ingest/resume", HttpMethod.Post, RemoveNullValues(payload), cancellationToken), + cancellationToken); + } + + public static async Task HandleExcititorListProvidersAsync( + IServiceProvider services, + bool includeDisabled, + bool verbose, + CancellationToken cancellationToken) + { + await using var scope = services.CreateAsyncScope(); + var client = scope.ServiceProvider.GetRequiredService(); + var logger = scope.ServiceProvider.GetRequiredService().CreateLogger("excititor-list-providers"); + var verbosity = scope.ServiceProvider.GetRequiredService(); + var previousLevel = verbosity.MinimumLevel; + verbosity.MinimumLevel = verbose ? LogLevel.Debug : LogLevel.Information; + using var activity = CliActivitySource.Instance.StartActivity("cli.excititor.list-providers", ActivityKind.Client); + activity?.SetTag("stellaops.cli.command", "excititor list-providers"); + activity?.SetTag("stellaops.cli.include_disabled", includeDisabled); + using var duration = CliMetrics.MeasureCommandDuration("excititor list-providers"); + + try + { + var providers = await client.GetExcititorProvidersAsync(includeDisabled, cancellationToken).ConfigureAwait(false); + Environment.ExitCode = 0; + logger.LogInformation("Providers returned: {Count}", providers.Count); + + if (providers.Count > 0) + { + if (AnsiConsole.Profile.Capabilities.Interactive) + { + var table = new Table().Border(TableBorder.Rounded).AddColumns("Provider", "Kind", "Trust", "Enabled", "Last Ingested"); + foreach (var provider in providers) + { + table.AddRow( + provider.Id, + provider.Kind, + string.IsNullOrWhiteSpace(provider.TrustTier) ? "-" : provider.TrustTier, + provider.Enabled ? "yes" : "no", + provider.LastIngestedAt?.ToString("yyyy-MM-dd HH:mm:ss 'UTC'", CultureInfo.InvariantCulture) ?? "unknown"); + } + + AnsiConsole.Write(table); + } + else + { + foreach (var provider in providers) + { + logger.LogInformation("{ProviderId} [{Kind}] Enabled={Enabled} Trust={Trust} LastIngested={LastIngested}", + provider.Id, + provider.Kind, + provider.Enabled ? "yes" : "no", + string.IsNullOrWhiteSpace(provider.TrustTier) ? "-" : provider.TrustTier, + provider.LastIngestedAt?.ToString("O", CultureInfo.InvariantCulture) ?? "unknown"); + } + } + } + } + catch (Exception ex) + { + logger.LogError(ex, "Failed to list Excititor providers."); + Environment.ExitCode = 1; + } + finally + { + verbosity.MinimumLevel = previousLevel; + } + } + + public static Task HandleExcititorExportAsync( + IServiceProvider services, + string format, + bool delta, + string? scope, + DateTimeOffset? since, + string? provider, + bool verbose, + CancellationToken cancellationToken) + { + var payload = new Dictionary(StringComparer.Ordinal) + { + ["format"] = string.IsNullOrWhiteSpace(format) ? "openvex" : format.Trim(), + ["delta"] = delta + }; + + if (!string.IsNullOrWhiteSpace(scope)) + { + payload["scope"] = scope.Trim(); + } + if (since.HasValue) + { + payload["since"] = since.Value.ToUniversalTime().ToString("O", CultureInfo.InvariantCulture); + } + if (!string.IsNullOrWhiteSpace(provider)) + { + payload["provider"] = provider.Trim(); + } + + return ExecuteExcititorCommandAsync( + services, + commandName: "excititor export", + verbose, + new Dictionary + { + ["format"] = payload["format"], + ["delta"] = delta, + ["scope"] = scope, + ["since"] = since?.ToUniversalTime().ToString("O", CultureInfo.InvariantCulture), + ["provider"] = provider + }, + client => client.ExecuteExcititorOperationAsync("export", HttpMethod.Post, RemoveNullValues(payload), cancellationToken), + cancellationToken); + } + + public static Task HandleExcititorVerifyAsync( + IServiceProvider services, + string? exportId, + string? digest, + string? attestationPath, + bool verbose, + CancellationToken cancellationToken) + { + if (string.IsNullOrWhiteSpace(exportId) && string.IsNullOrWhiteSpace(digest) && string.IsNullOrWhiteSpace(attestationPath)) + { + var logger = services.GetRequiredService().CreateLogger("excititor-verify"); + logger.LogError("At least one of --export-id, --digest, or --attestation must be provided."); + Environment.ExitCode = 1; + return Task.CompletedTask; + } + + var payload = new Dictionary(StringComparer.Ordinal); + if (!string.IsNullOrWhiteSpace(exportId)) + { + payload["exportId"] = exportId.Trim(); + } + if (!string.IsNullOrWhiteSpace(digest)) + { + payload["digest"] = digest.Trim(); + } + if (!string.IsNullOrWhiteSpace(attestationPath)) + { + var fullPath = Path.GetFullPath(attestationPath); + if (!File.Exists(fullPath)) + { + var logger = services.GetRequiredService().CreateLogger("excititor-verify"); + logger.LogError("Attestation file not found at {Path}.", fullPath); + Environment.ExitCode = 1; + return Task.CompletedTask; + } + + var bytes = File.ReadAllBytes(fullPath); + payload["attestation"] = new Dictionary(StringComparer.Ordinal) + { + ["fileName"] = Path.GetFileName(fullPath), + ["base64"] = Convert.ToBase64String(bytes) + }; + } + + return ExecuteExcititorCommandAsync( + services, + commandName: "excititor verify", + verbose, + new Dictionary + { + ["export_id"] = exportId, + ["digest"] = digest, + ["attestation_path"] = attestationPath + }, + client => client.ExecuteExcititorOperationAsync("verify", HttpMethod.Post, RemoveNullValues(payload), cancellationToken), + cancellationToken); + } + + public static Task HandleExcititorReconcileAsync( + IServiceProvider services, + IReadOnlyList providers, + TimeSpan? maxAge, + bool verbose, + CancellationToken cancellationToken) + { + var normalizedProviders = NormalizeProviders(providers); + var payload = new Dictionary(StringComparer.Ordinal); + if (normalizedProviders.Count > 0) + { + payload["providers"] = normalizedProviders; + } + if (maxAge.HasValue) + { + payload["maxAge"] = maxAge.Value.ToString("c", CultureInfo.InvariantCulture); + } + + return ExecuteExcititorCommandAsync( + services, + commandName: "excititor reconcile", + verbose, + new Dictionary + { + ["providers"] = normalizedProviders.Count, + ["max_age"] = maxAge?.ToString("c", CultureInfo.InvariantCulture) + }, + client => client.ExecuteExcititorOperationAsync("reconcile", HttpMethod.Post, RemoveNullValues(payload), cancellationToken), + cancellationToken); + } + public static async Task HandleAuthLoginAsync( IServiceProvider services, StellaOpsCliOptions options, @@ -1111,12 +1417,109 @@ internal static class CommandHandlers "jti" }; + private static async Task ExecuteExcititorCommandAsync( + IServiceProvider services, + string commandName, + bool verbose, + IDictionary? activityTags, + Func> operation, + CancellationToken cancellationToken) + { + await using var scope = services.CreateAsyncScope(); + var client = scope.ServiceProvider.GetRequiredService(); + var logger = scope.ServiceProvider.GetRequiredService().CreateLogger(commandName.Replace(' ', '-')); + var verbosity = scope.ServiceProvider.GetRequiredService(); + var previousLevel = verbosity.MinimumLevel; + verbosity.MinimumLevel = verbose ? LogLevel.Debug : LogLevel.Information; + using var activity = CliActivitySource.Instance.StartActivity($"cli.{commandName.Replace(' ', '.')}" , ActivityKind.Client); + activity?.SetTag("stellaops.cli.command", commandName); + if (activityTags is not null) + { + foreach (var tag in activityTags) + { + activity?.SetTag(tag.Key, tag.Value); + } + } + using var duration = CliMetrics.MeasureCommandDuration(commandName); + + try + { + var result = await operation(client).ConfigureAwait(false); + if (result.Success) + { + if (!string.IsNullOrWhiteSpace(result.Message)) + { + logger.LogInformation(result.Message); + } + else + { + logger.LogInformation("Operation completed successfully."); + } + + if (!string.IsNullOrWhiteSpace(result.Location)) + { + logger.LogInformation("Location: {Location}", result.Location); + } + + if (result.Payload is JsonElement payload && payload.ValueKind is not JsonValueKind.Undefined and not JsonValueKind.Null) + { + logger.LogDebug("Response payload: {Payload}", payload.ToString()); + } + + Environment.ExitCode = 0; + } + else + { + logger.LogError(string.IsNullOrWhiteSpace(result.Message) ? "Operation failed." : result.Message); + Environment.ExitCode = 1; + } + } + catch (Exception ex) + { + logger.LogError(ex, "Excititor operation failed."); + Environment.ExitCode = 1; + } + finally + { + verbosity.MinimumLevel = previousLevel; + } + } + + private static IReadOnlyList NormalizeProviders(IReadOnlyList providers) + { + if (providers is null || providers.Count == 0) + { + return Array.Empty(); + } + + var list = new List(); + foreach (var provider in providers) + { + if (!string.IsNullOrWhiteSpace(provider)) + { + list.Add(provider.Trim()); + } + } + + return list.Count == 0 ? Array.Empty() : list; + } + + private static IDictionary RemoveNullValues(Dictionary source) + { + foreach (var key in source.Where(kvp => kvp.Value is null).Select(kvp => kvp.Key).ToList()) + { + source.Remove(key); + } + + return source; + } + private static async Task TriggerJobAsync( IBackendOperationsClient client, ILogger logger, string jobKind, - IDictionary parameters, - CancellationToken cancellationToken) + IDictionary parameters, + CancellationToken cancellationToken) { JobTriggerResult result = await client.TriggerJobAsync(jobKind, parameters, cancellationToken).ConfigureAwait(false); if (result.Success) diff --git a/src/StellaOps.Cli/Services/BackendOperationsClient.cs b/src/StellaOps.Cli/Services/BackendOperationsClient.cs index 6e4df175..97c56d7f 100644 --- a/src/StellaOps.Cli/Services/BackendOperationsClient.cs +++ b/src/StellaOps.Cli/Services/BackendOperationsClient.cs @@ -231,9 +231,99 @@ internal sealed class BackendOperationsClient : IBackendOperationsClient return new JobTriggerResult(true, "Accepted", location, run); } - var failureMessage = await CreateFailureMessageAsync(response, cancellationToken).ConfigureAwait(false); - return new JobTriggerResult(false, failureMessage, null, null); - } + var failureMessage = await CreateFailureMessageAsync(response, cancellationToken).ConfigureAwait(false); + return new JobTriggerResult(false, failureMessage, null, null); + } + + public async Task ExecuteExcititorOperationAsync(string route, HttpMethod method, object? payload, CancellationToken cancellationToken) + { + EnsureBackendConfigured(); + + if (string.IsNullOrWhiteSpace(route)) + { + throw new ArgumentException("Route must be provided.", nameof(route)); + } + + var relative = route.TrimStart('/'); + using var request = CreateRequest(method, $"excititor/{relative}"); + + if (payload is not null && method != HttpMethod.Get && method != HttpMethod.Delete) + { + request.Content = JsonContent.Create(payload, options: SerializerOptions); + } + + await AuthorizeRequestAsync(request, cancellationToken).ConfigureAwait(false); + using var response = await _httpClient.SendAsync(request, cancellationToken).ConfigureAwait(false); + + if (response.IsSuccessStatusCode) + { + var (message, payloadElement) = await ExtractExcititorResponseAsync(response, cancellationToken).ConfigureAwait(false); + var location = response.Headers.Location?.ToString(); + return new ExcititorOperationResult(true, message, location, payloadElement); + } + + var failure = await CreateFailureMessageAsync(response, cancellationToken).ConfigureAwait(false); + return new ExcititorOperationResult(false, failure, null, null); + } + + public async Task> GetExcititorProvidersAsync(bool includeDisabled, CancellationToken cancellationToken) + { + EnsureBackendConfigured(); + + var query = includeDisabled ? "?includeDisabled=true" : string.Empty; + using var request = CreateRequest(HttpMethod.Get, $"excititor/providers{query}"); + await AuthorizeRequestAsync(request, cancellationToken).ConfigureAwait(false); + using var response = await _httpClient.SendAsync(request, cancellationToken).ConfigureAwait(false); + + if (!response.IsSuccessStatusCode) + { + var failure = await CreateFailureMessageAsync(response, cancellationToken).ConfigureAwait(false); + throw new InvalidOperationException(failure); + } + + if (response.Content is null || response.Content.Headers.ContentLength is 0) + { + return Array.Empty(); + } + + await using var stream = await response.Content.ReadAsStreamAsync(cancellationToken).ConfigureAwait(false); + if (stream is null || stream.Length == 0) + { + return Array.Empty(); + } + + using var document = await JsonDocument.ParseAsync(stream, cancellationToken: cancellationToken).ConfigureAwait(false); + var root = document.RootElement; + if (root.ValueKind == JsonValueKind.Object && root.TryGetProperty("providers", out var providersProperty)) + { + root = providersProperty; + } + + if (root.ValueKind != JsonValueKind.Array) + { + return Array.Empty(); + } + + var list = new List(); + foreach (var item in root.EnumerateArray()) + { + var id = GetStringProperty(item, "id") ?? string.Empty; + if (string.IsNullOrWhiteSpace(id)) + { + continue; + } + + var kind = GetStringProperty(item, "kind") ?? "unknown"; + var displayName = GetStringProperty(item, "displayName") ?? id; + var trustTier = GetStringProperty(item, "trustTier") ?? string.Empty; + var enabled = GetBooleanProperty(item, "enabled", defaultValue: true); + var lastIngested = GetDateTimeOffsetProperty(item, "lastIngestedAt"); + + list.Add(new ExcititorProviderSummary(id, kind, displayName, trustTier, enabled, lastIngested)); + } + + return list; + } private HttpRequestMessage CreateRequest(HttpMethod method, string relativeUri) { @@ -328,10 +418,114 @@ internal sealed class BackendOperationsClient : IBackendOperationsClient } } + private async Task<(string Message, JsonElement? Payload)> ExtractExcititorResponseAsync(HttpResponseMessage response, CancellationToken cancellationToken) + { + if (response.Content is null || response.Content.Headers.ContentLength is 0) + { + return ($"HTTP {(int)response.StatusCode}", null); + } + + try + { + await using var stream = await response.Content.ReadAsStreamAsync(cancellationToken).ConfigureAwait(false); + if (stream is null || stream.Length == 0) + { + return ($"HTTP {(int)response.StatusCode}", null); + } + + using var document = await JsonDocument.ParseAsync(stream, cancellationToken: cancellationToken).ConfigureAwait(false); + var root = document.RootElement.Clone(); + string? message = null; + if (root.ValueKind == JsonValueKind.Object) + { + message = GetStringProperty(root, "message") ?? GetStringProperty(root, "status"); + } + + if (string.IsNullOrWhiteSpace(message)) + { + message = root.ValueKind == JsonValueKind.Object || root.ValueKind == JsonValueKind.Array + ? root.ToString() + : root.GetRawText(); + } + + return (message ?? $"HTTP {(int)response.StatusCode}", root); + } + catch (JsonException) + { + var text = await response.Content.ReadAsStringAsync(cancellationToken).ConfigureAwait(false); + return (string.IsNullOrWhiteSpace(text) ? $"HTTP {(int)response.StatusCode}" : text.Trim(), null); + } + } + + private static bool TryGetPropertyCaseInsensitive(JsonElement element, string propertyName, out JsonElement property) + { + if (element.ValueKind == JsonValueKind.Object && element.TryGetProperty(propertyName, out property)) + { + return true; + } + + if (element.ValueKind == JsonValueKind.Object) + { + foreach (var candidate in element.EnumerateObject()) + { + if (string.Equals(candidate.Name, propertyName, StringComparison.OrdinalIgnoreCase)) + { + property = candidate.Value; + return true; + } + } + } + + property = default; + return false; + } + + private static string? GetStringProperty(JsonElement element, string propertyName) + { + if (TryGetPropertyCaseInsensitive(element, propertyName, out var property)) + { + if (property.ValueKind == JsonValueKind.String) + { + return property.GetString(); + } + } + + return null; + } + + private static bool GetBooleanProperty(JsonElement element, string propertyName, bool defaultValue) + { + if (TryGetPropertyCaseInsensitive(element, propertyName, out var property)) + { + return property.ValueKind switch + { + JsonValueKind.True => true, + JsonValueKind.False => false, + JsonValueKind.String when bool.TryParse(property.GetString(), out var parsed) => parsed, + _ => defaultValue + }; + } + + return defaultValue; + } + + private static DateTimeOffset? GetDateTimeOffsetProperty(JsonElement element, string propertyName) + { + if (TryGetPropertyCaseInsensitive(element, propertyName, out var property) && property.ValueKind == JsonValueKind.String) + { + if (DateTimeOffset.TryParse(property.GetString(), CultureInfo.InvariantCulture, DateTimeStyles.AssumeUniversal, out var parsed)) + { + return parsed.ToUniversalTime(); + } + } + + return null; + } + private void EnsureBackendConfigured() { if (_httpClient.BaseAddress is null) - { + { throw new InvalidOperationException("Backend URL is not configured. Provide STELLAOPS_BACKEND_URL or configure appsettings."); } } diff --git a/src/StellaOps.Cli/Services/IBackendOperationsClient.cs b/src/StellaOps.Cli/Services/IBackendOperationsClient.cs index b593524b..d3601761 100644 --- a/src/StellaOps.Cli/Services/IBackendOperationsClient.cs +++ b/src/StellaOps.Cli/Services/IBackendOperationsClient.cs @@ -1,16 +1,21 @@ -using System.Collections.Generic; -using System.Threading; -using System.Threading.Tasks; -using StellaOps.Cli.Configuration; -using StellaOps.Cli.Services.Models; +using System.Collections.Generic; +using System.Net.Http; +using System.Threading; +using System.Threading.Tasks; +using StellaOps.Cli.Configuration; +using StellaOps.Cli.Services.Models; namespace StellaOps.Cli.Services; internal interface IBackendOperationsClient { - Task DownloadScannerAsync(string channel, string outputPath, bool overwrite, bool verbose, CancellationToken cancellationToken); - - Task UploadScanResultsAsync(string filePath, CancellationToken cancellationToken); - - Task TriggerJobAsync(string jobKind, IDictionary parameters, CancellationToken cancellationToken); -} + Task DownloadScannerAsync(string channel, string outputPath, bool overwrite, bool verbose, CancellationToken cancellationToken); + + Task UploadScanResultsAsync(string filePath, CancellationToken cancellationToken); + + Task TriggerJobAsync(string jobKind, IDictionary parameters, CancellationToken cancellationToken); + + Task ExecuteExcititorOperationAsync(string route, HttpMethod method, object? payload, CancellationToken cancellationToken); + + Task> GetExcititorProvidersAsync(bool includeDisabled, CancellationToken cancellationToken); +} diff --git a/src/StellaOps.Cli/Services/Models/ExcititorOperationResult.cs b/src/StellaOps.Cli/Services/Models/ExcititorOperationResult.cs new file mode 100644 index 00000000..2cb702d7 --- /dev/null +++ b/src/StellaOps.Cli/Services/Models/ExcititorOperationResult.cs @@ -0,0 +1,9 @@ +using System.Text.Json; + +namespace StellaOps.Cli.Services.Models; + +internal sealed record ExcititorOperationResult( + bool Success, + string Message, + string? Location, + JsonElement? Payload); diff --git a/src/StellaOps.Cli/Services/Models/ExcititorProviderSummary.cs b/src/StellaOps.Cli/Services/Models/ExcititorProviderSummary.cs new file mode 100644 index 00000000..496e15b1 --- /dev/null +++ b/src/StellaOps.Cli/Services/Models/ExcititorProviderSummary.cs @@ -0,0 +1,11 @@ +using System; + +namespace StellaOps.Cli.Services.Models; + +internal sealed record ExcititorProviderSummary( + string Id, + string Kind, + string DisplayName, + string TrustTier, + bool Enabled, + DateTimeOffset? LastIngestedAt); diff --git a/src/StellaOps.Cli/TASKS.md b/src/StellaOps.Cli/TASKS.md index 38391e6e..964f6278 100644 --- a/src/StellaOps.Cli/TASKS.md +++ b/src/StellaOps.Cli/TASKS.md @@ -14,6 +14,9 @@ If you are working on this file you need to read docs/ARCHITECTURE_EXCITITOR.md |Expose auth client resilience settings|DevEx/CLI|Auth libraries LIB5|**DONE (2025-10-10)** – CLI options now bind resilience knobs, `AddStellaOpsAuthClient` honours them, and tests cover env overrides.| |Document advanced Authority tuning|Docs/CLI|Expose auth client resilience settings|**DONE (2025-10-10)** – docs/09 and docs/10 describe retry/offline settings with env examples and point to the integration guide.| |Surface password policy diagnostics in CLI output|DevEx/CLI, Security Guild|AUTHSEC-CRYPTO-02-004|**DONE (2025-10-15)** – CLI startup runs the Authority plug-in analyzer, logs weakened password policy warnings with manifest paths, added unit tests (`dotnet test src/StellaOps.Cli.Tests`) and updated docs/09 with remediation guidance.| -|EXCITITOR-CLI-01-001 – Add `excititor` command group|DevEx/CLI|EXCITITOR-WEB-01-001|TODO – Introduce `excititor` verb hierarchy (init/pull/resume/list-providers/export/verify/reconcile) forwarding to WebService with token auth and consistent exit codes.| +|EXCITITOR-CLI-01-001 – Add `excititor` command group|DevEx/CLI|EXCITITOR-WEB-01-001|DONE (2025-10-18) – Introduced `excititor` verbs (init/pull/resume/list-providers/export/verify/reconcile) with token-auth backend calls, provenance-friendly logging, and regression coverage.| |EXCITITOR-CLI-01-002 – Export download & attestation UX|DevEx/CLI|EXCITITOR-CLI-01-001, EXCITITOR-EXPORT-01-001|TODO – Display export metadata (sha256, size, Rekor link), support optional artifact download path, and handle cache hits gracefully.| |EXCITITOR-CLI-01-003 – CLI docs & examples for Excititor|Docs/CLI|EXCITITOR-CLI-01-001|TODO – Update docs/09_API_CLI_REFERENCE.md and quickstart snippets to cover Excititor verbs, offline guidance, and attestation verification workflow.| +|CLI-RUNTIME-13-005 – Runtime policy test verbs|DevEx/CLI|SCANNER-RUNTIME-12-302, ZASTAVA-WEBHOOK-12-102|TODO – Add `runtime policy test` and related verbs to query `/policy/runtime`, display verdicts/TTL/reasons, and support batch inputs.| +|CLI-OFFLINE-13-006 – Offline kit workflows|DevEx/CLI|DEVOPS-OFFLINE-14-002|TODO – Implement `offline kit pull/import/status` commands with integrity checks, resumable downloads, and doc updates.| +|CLI-PLUGIN-13-007 – Plugin packaging|DevEx/CLI|CLI-RUNTIME-13-005, CLI-OFFLINE-13-006|TODO – Package non-core verbs as restart-time plug-ins (manifest + loader updates, tests ensuring no hot reload).| diff --git a/src/StellaOps.Concelier.Connector.StellaOpsMirror/TASKS.md b/src/StellaOps.Concelier.Connector.StellaOpsMirror/TASKS.md new file mode 100644 index 00000000..3d8feffc --- /dev/null +++ b/src/StellaOps.Concelier.Connector.StellaOpsMirror/TASKS.md @@ -0,0 +1,7 @@ +# StellaOps Mirror Connector Task Board (Sprint 8) + +| ID | Status | Owner(s) | Depends on | Description | Exit Criteria | +|----|--------|----------|------------|-------------|---------------| +| FEEDCONN-STELLA-08-001 | TODO | BE-Conn-Stella | CONCELIER-EXPORT-08-201 | Implement Concelier mirror fetcher hitting `https://.stella-ops.org/concelier/exports/index.json`, verify signatures/digests, and persist raw documents with provenance. | Fetch job downloads mirror manifest, verifies digest/signature, stores raw docs with tests covering happy-path + tampered manifest. | +| FEEDCONN-STELLA-08-002 | TODO | BE-Conn-Stella | FEEDCONN-STELLA-08-001 | Map mirror payloads into canonical advisory DTOs with provenance referencing mirror domain + original source metadata. | Mapper produces advisories/aliases/affected with mirror provenance; fixtures assert canonical parity with upstream JSON exporters. | +| FEEDCONN-STELLA-08-003 | TODO | BE-Conn-Stella | FEEDCONN-STELLA-08-002 | Add incremental cursor + resume support (per-export fingerprint) and document configuration for downstream Concelier instances. | Connector resumes from last export, handles deletion/delta cases, docs updated with config sample; integration test covers resume + new export scenario. | diff --git a/src/StellaOps.Concelier.Core/TASKS.md b/src/StellaOps.Concelier.Core/TASKS.md index 66843401..8c5ce549 100644 --- a/src/StellaOps.Concelier.Core/TASKS.md +++ b/src/StellaOps.Concelier.Core/TASKS.md @@ -18,3 +18,4 @@ |Reference normalization & freshness instrumentation cleanup|BE-Core, QA|Models|DONE (2025-10-15) – reference keys normalized, freshness overrides applied to union fields, and new tests assert decision logging.| |FEEDCORE-ENGINE-07-001 – Advisory event log & asOf queries|Team Core Engine & Storage Analytics|FEEDSTORAGE-DATA-07-001|TODO – Introduce immutable advisory statement events, expose `asOf` query surface for merge/export pipelines, and document determinism guarantees for replay.| |FEEDCORE-ENGINE-07-002 – Noise prior computation service|Team Core Engine & Data Science|FEEDCORE-ENGINE-07-001|TODO – Build rule-based learner capturing false-positive priors per package/env, persist summaries, and expose APIs for Excititor/scan suppressors with reproducible statistics.| +|FEEDCORE-ENGINE-07-003 – Unknown state ledger & confidence seeding|Team Core Engine & Storage Analytics|FEEDCORE-ENGINE-07-001|TODO – Persist `unknown_vuln_range/unknown_origin/ambiguous_fix` markers with initial confidence bands, expose query surface for Policy, and add fixtures validating canonical serialization.| diff --git a/src/StellaOps.Concelier.Exporter.Json/TASKS.md b/src/StellaOps.Concelier.Exporter.Json/TASKS.md index 23285a11..b00a2741 100644 --- a/src/StellaOps.Concelier.Exporter.Json/TASKS.md +++ b/src/StellaOps.Concelier.Exporter.Json/TASKS.md @@ -10,3 +10,4 @@ |Stream advisories during export|BE-Export|Storage.Mongo|DONE – exporter + streaming-only test ensures single enumeration and per-file digest capture.| |Emit export manifest with digest metadata|BE-Export|Exporters|DONE – manifest now includes per-file digests/sizes alongside tree digest.| |Surface new advisory fields (description/CWEs/canonical metric)|BE-Export|Models, Core|DONE (2025-10-15) – JSON exporter validated with new fixtures ensuring description/CWEs/canonical metric are preserved in outputs; `dotnet test src/StellaOps.Concelier.Exporter.Json.Tests` run 2025-10-15 for regression coverage.| +|CONCELIER-EXPORT-08-201 – Mirror bundle + domain manifest|Team Concelier Export|FEEDCORE-ENGINE-07-001|TODO – Produce per-domain aggregate bundles (JSON + manifest) with deterministic digests, include upstream source metadata, and publish index consumed by mirror endpoints/tests.| diff --git a/src/StellaOps.Concelier.Exporter.TrivyDb/TASKS.md b/src/StellaOps.Concelier.Exporter.TrivyDb/TASKS.md index 078ec861..a2a860b8 100644 --- a/src/StellaOps.Concelier.Exporter.TrivyDb/TASKS.md +++ b/src/StellaOps.Concelier.Exporter.TrivyDb/TASKS.md @@ -12,3 +12,4 @@ |Streamed package building to avoid large copies|BE-Export|Exporters|DONE – metadata/config now reuse backing arrays and OCI writer streams directly without double buffering.| |Plan incremental/delta exports|BE-Export|Exporters|DONE – state captures per-file manifests, planner schedules delta vs full resets, layer reuse smoke test verifies OCI reuse, and operator guide documents the validation flow.| |Advisory schema parity export (description/CWEs/canonical metric)|BE-Export|Models, Core|DONE (2025-10-15) – exporter/test fixtures updated to handle description/CWEs/canonical metric fields during Trivy DB packaging; `dotnet test src/StellaOps.Concelier.Exporter.TrivyDb.Tests` re-run 2025-10-15 to confirm coverage.| +|CONCELIER-EXPORT-08-202 – Mirror-ready Trivy DB bundles|Team Concelier Export|CONCELIER-EXPORT-08-201|TODO – Generate domain-specific Trivy DB archives + metadata manifest, ensure deterministic digests, and document sync process for downstream Concelier nodes.| diff --git a/src/StellaOps.Concelier.WebService/TASKS.md b/src/StellaOps.Concelier.WebService/TASKS.md index 1404da04..8d4e5c6f 100644 --- a/src/StellaOps.Concelier.WebService/TASKS.md +++ b/src/StellaOps.Concelier.WebService/TASKS.md @@ -22,3 +22,4 @@ |Update Concelier operator guide for enforcement cutoff|Docs/Concelier|FSR1 rollout|**DONE (2025-10-12)** – Installation guide emphasises disabling `allowAnonymousFallback` before 2025-12-31 UTC and connects audit signals to the rollout checklist.| |Rename plugin drop directory to namespaced path|BE-Base|Plugins|**TODO** – Point Concelier source/exporter build outputs to `StellaOps.Concelier.PluginBinaries`, update PluginHost defaults/search patterns to match, ensure Offline Kit packaging/tests expect the new folder, and document migration guidance for operators.| |Authority resilience adoption|Concelier WebService, Docs|Plumb Authority client resilience options|**BLOCKED (2025-10-10)** – Roll out retry/offline knobs to deployment docs and confirm CLI parity once LIB5 lands; unblock after resilience options wired and tested.| +|CONCELIER-WEB-08-201 – Mirror distribution endpoints|Concelier WebService Guild|CONCELIER-EXPORT-08-201, DEVOPS-MIRROR-08-001|TODO – Add domain-scoped mirror configuration (`*.stella-ops.org`), expose signed export index/download APIs with quota and auth, and document sync workflow for downstream Concelier instances.| diff --git a/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest.Tests/Configuration/OciOpenVexAttestationConnectorOptionsValidatorTests.cs b/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest.Tests/Configuration/OciOpenVexAttestationConnectorOptionsValidatorTests.cs new file mode 100644 index 00000000..bbb3717d --- /dev/null +++ b/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest.Tests/Configuration/OciOpenVexAttestationConnectorOptionsValidatorTests.cs @@ -0,0 +1,76 @@ +using FluentAssertions; +using Microsoft.Extensions.Logging.Abstractions; +using StellaOps.Excititor.Connectors.Abstractions; +using StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest.Configuration; +using StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest.Discovery; +using StellaOps.Excititor.Core; +using System.Collections.Generic; +using System.IO.Abstractions.TestingHelpers; +using Xunit; + +namespace StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest.Tests.Configuration; + +public sealed class OciOpenVexAttestationConnectorOptionsValidatorTests +{ + [Fact] + public void Validate_WithValidConfiguration_Succeeds() + { + var fileSystem = new MockFileSystem(new Dictionary + { + ["/offline/registry.example.com/repo/latest/openvex-attestations.tgz"] = new MockFileData(string.Empty), + }); + + var validator = new OciOpenVexAttestationConnectorOptionsValidator(fileSystem); + var options = new OciOpenVexAttestationConnectorOptions + { + AllowHttpRegistries = true, + }; + + options.Images.Add(new OciImageSubscriptionOptions + { + Reference = "registry.example.com/repo/image:latest", + OfflineBundlePath = "/offline/registry.example.com/repo/latest/openvex-attestations.tgz", + }); + + options.Registry.Username = "user"; + options.Registry.Password = "pass"; + + options.Cosign.Mode = CosignCredentialMode.None; + + var errors = new List(); + + validator.Validate(new VexConnectorDescriptor("id", VexProviderKind.Attestation, "display"), options, errors); + + errors.Should().BeEmpty(); + } + + [Fact] + public void Validate_WhenImagesMissing_AddsError() + { + var validator = new OciOpenVexAttestationConnectorOptionsValidator(new MockFileSystem()); + var options = new OciOpenVexAttestationConnectorOptions(); + + var errors = new List(); + + validator.Validate(new VexConnectorDescriptor("id", VexProviderKind.Attestation, "display"), options, errors); + + errors.Should().ContainSingle().Which.Should().Contain("At least one OCI image reference must be configured."); + } + + [Fact] + public void Validate_WhenDigestMalformed_AddsError() + { + var validator = new OciOpenVexAttestationConnectorOptionsValidator(new MockFileSystem()); + var options = new OciOpenVexAttestationConnectorOptions(); + options.Images.Add(new OciImageSubscriptionOptions + { + Reference = "registry.test/repo/image@sha256:not-a-digest", + }); + + var errors = new List(); + + validator.Validate(new VexConnectorDescriptor("id", VexProviderKind.Attestation, "display"), options, errors); + + errors.Should().ContainSingle(); + } +} diff --git a/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest.Tests/Connector/OciOpenVexAttestationConnectorTests.cs b/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest.Tests/Connector/OciOpenVexAttestationConnectorTests.cs new file mode 100644 index 00000000..3fef0843 --- /dev/null +++ b/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest.Tests/Connector/OciOpenVexAttestationConnectorTests.cs @@ -0,0 +1,213 @@ +using System; +using System.Collections.Generic; +using System.Collections.Immutable; +using System.Net; +using System.Net.Http; +using System.Threading; +using System.Threading.Tasks; +using FluentAssertions; +using Microsoft.Extensions.Caching.Memory; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Logging.Abstractions; +using StellaOps.Excititor.Connectors.Abstractions; +using StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest; +using StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest.Configuration; +using StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest.DependencyInjection; +using StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest.Discovery; +using StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest.Fetch; +using StellaOps.Excititor.Core; +using System.IO.Abstractions.TestingHelpers; +using Xunit; + +namespace StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest.Tests.Connector; + +public sealed class OciOpenVexAttestationConnectorTests +{ + [Fact] + public async Task FetchAsync_WithOfflineBundle_EmitsRawDocument() + { + var fileSystem = new MockFileSystem(new Dictionary + { + ["/bundles/attestation.json"] = new MockFileData("{\"payload\":\"\",\"payloadType\":\"application/vnd.in-toto+json\",\"signatures\":[{\"sig\":\"\"}]}"), + }); + + using var cache = new MemoryCache(new MemoryCacheOptions()); + var httpClient = new HttpClient(new StubHttpMessageHandler()) + { + BaseAddress = new System.Uri("https://registry.example.com/") + }; + + var httpFactory = new SingleClientHttpClientFactory(httpClient); + var discovery = new OciAttestationDiscoveryService(cache, fileSystem, NullLogger.Instance); + var fetcher = new OciAttestationFetcher(httpFactory, fileSystem, NullLogger.Instance); + + var connector = new OciOpenVexAttestationConnector( + discovery, + fetcher, + NullLogger.Instance, + TimeProvider.System); + + var settingsValues = ImmutableDictionary.Empty + .Add("Images:0:Reference", "registry.example.com/repo/image:latest") + .Add("Images:0:OfflineBundlePath", "/bundles/attestation.json") + .Add("Offline:PreferOffline", "true") + .Add("Offline:AllowNetworkFallback", "false") + .Add("Cosign:Mode", "None"); + + var settings = new VexConnectorSettings(settingsValues); + await connector.ValidateAsync(settings, CancellationToken.None); + + var sink = new CapturingRawSink(); + var verifier = new CapturingSignatureVerifier(); + var context = new VexConnectorContext( + Since: null, + Settings: VexConnectorSettings.Empty, + RawSink: sink, + SignatureVerifier: verifier, + Normalizers: new NoopNormalizerRouter(), + Services: new Microsoft.Extensions.DependencyInjection.ServiceCollection().BuildServiceProvider()); + + var documents = new List(); + await foreach (var document in connector.FetchAsync(context, CancellationToken.None)) + { + documents.Add(document); + } + + documents.Should().HaveCount(1); + sink.Documents.Should().HaveCount(1); + documents[0].Format.Should().Be(VexDocumentFormat.OciAttestation); + documents[0].Metadata.Should().ContainKey("oci.attestation.sourceKind").WhoseValue.Should().Be("offline"); + documents[0].Metadata.Should().ContainKey("vex.provenance.sourceKind").WhoseValue.Should().Be("offline"); + documents[0].Metadata.Should().ContainKey("vex.provenance.registryAuthMode").WhoseValue.Should().Be("Anonymous"); + verifier.VerifyCalls.Should().Be(1); + } + + [Fact] + public async Task FetchAsync_WithSignatureMetadata_EnrichesProvenance() + { + var fileSystem = new MockFileSystem(new Dictionary + { + ["/bundles/attestation.json"] = new MockFileData("{\"payload\":\"\",\"payloadType\":\"application/vnd.in-toto+json\",\"signatures\":[{\"sig\":\"\"}]}"), + }); + + using var cache = new MemoryCache(new MemoryCacheOptions()); + var httpClient = new HttpClient(new StubHttpMessageHandler()) + { + BaseAddress = new System.Uri("https://registry.example.com/") + }; + + var httpFactory = new SingleClientHttpClientFactory(httpClient); + var discovery = new OciAttestationDiscoveryService(cache, fileSystem, NullLogger.Instance); + var fetcher = new OciAttestationFetcher(httpFactory, fileSystem, NullLogger.Instance); + + var connector = new OciOpenVexAttestationConnector( + discovery, + fetcher, + NullLogger.Instance, + TimeProvider.System); + + var settingsValues = ImmutableDictionary.Empty + .Add("Images:0:Reference", "registry.example.com/repo/image:latest") + .Add("Images:0:OfflineBundlePath", "/bundles/attestation.json") + .Add("Offline:PreferOffline", "true") + .Add("Offline:AllowNetworkFallback", "false") + .Add("Cosign:Mode", "Keyless") + .Add("Cosign:Keyless:Issuer", "https://issuer.example.com") + .Add("Cosign:Keyless:Subject", "subject@example.com"); + + var settings = new VexConnectorSettings(settingsValues); + await connector.ValidateAsync(settings, CancellationToken.None); + + var sink = new CapturingRawSink(); + var verifier = new CapturingSignatureVerifier + { + Result = new VexSignatureMetadata( + type: "cosign", + subject: "sig-subject", + issuer: "sig-issuer", + keyId: "key-id", + verifiedAt: DateTimeOffset.UtcNow, + transparencyLogReference: "rekor://entry/123") + }; + + var context = new VexConnectorContext( + Since: null, + Settings: VexConnectorSettings.Empty, + RawSink: sink, + SignatureVerifier: verifier, + Normalizers: new NoopNormalizerRouter(), + Services: new ServiceCollection().BuildServiceProvider()); + + var documents = new List(); + await foreach (var document in connector.FetchAsync(context, CancellationToken.None)) + { + documents.Add(document); + } + + documents.Should().HaveCount(1); + var metadata = documents[0].Metadata; + metadata.Should().Contain("vex.signature.type", "cosign"); + metadata.Should().Contain("vex.signature.subject", "sig-subject"); + metadata.Should().Contain("vex.signature.issuer", "sig-issuer"); + metadata.Should().Contain("vex.signature.keyId", "key-id"); + metadata.Should().ContainKey("vex.signature.verifiedAt"); + metadata.Should().Contain("vex.signature.transparencyLogReference", "rekor://entry/123"); + metadata.Should().Contain("vex.provenance.cosign.mode", "Keyless"); + metadata.Should().Contain("vex.provenance.cosign.issuer", "https://issuer.example.com"); + metadata.Should().Contain("vex.provenance.cosign.subject", "subject@example.com"); + verifier.VerifyCalls.Should().Be(1); + } + + private sealed class CapturingRawSink : IVexRawDocumentSink + { + public List Documents { get; } = new(); + + public ValueTask StoreAsync(VexRawDocument document, CancellationToken cancellationToken) + { + Documents.Add(document); + return ValueTask.CompletedTask; + } + } + + private sealed class CapturingSignatureVerifier : IVexSignatureVerifier + { + public int VerifyCalls { get; private set; } + + public VexSignatureMetadata? Result { get; set; } + + public ValueTask VerifyAsync(VexRawDocument document, CancellationToken cancellationToken) + { + VerifyCalls++; + return ValueTask.FromResult(Result); + } + } + + private sealed class NoopNormalizerRouter : IVexNormalizerRouter + { + public ValueTask NormalizeAsync(VexRawDocument document, CancellationToken cancellationToken) + => ValueTask.FromResult(new VexClaimBatch(document, ImmutableArray.Empty, ImmutableDictionary.Empty)); + } + + private sealed class SingleClientHttpClientFactory : IHttpClientFactory + { + private readonly HttpClient _client; + + public SingleClientHttpClientFactory(HttpClient client) + { + _client = client; + } + + public HttpClient CreateClient(string name) => _client; + } + + private sealed class StubHttpMessageHandler : HttpMessageHandler + { + protected override Task SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) + { + return Task.FromResult(new HttpResponseMessage(HttpStatusCode.NotFound) + { + RequestMessage = request + }); + } + } +} diff --git a/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest.Tests/Discovery/OciAttestationDiscoveryServiceTests.cs b/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest.Tests/Discovery/OciAttestationDiscoveryServiceTests.cs new file mode 100644 index 00000000..e993fb73 --- /dev/null +++ b/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest.Tests/Discovery/OciAttestationDiscoveryServiceTests.cs @@ -0,0 +1,83 @@ +using FluentAssertions; +using Microsoft.Extensions.Caching.Memory; +using Microsoft.Extensions.Logging.Abstractions; +using StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest.Configuration; +using StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest.Discovery; +using System.Collections.Generic; +using System.IO.Abstractions.TestingHelpers; +using Xunit; + +namespace StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest.Tests.Discovery; + +public sealed class OciAttestationDiscoveryServiceTests +{ + [Fact] + public async Task LoadAsync_ResolvesOfflinePaths() + { + var fileSystem = new MockFileSystem(new Dictionary + { + ["/bundles/registry.example.com/repo/image/latest/openvex-attestations.tgz"] = new MockFileData(string.Empty), + }); + + using var cache = new MemoryCache(new MemoryCacheOptions()); + var service = new OciAttestationDiscoveryService(cache, fileSystem, NullLogger.Instance); + + var options = new OciOpenVexAttestationConnectorOptions + { + AllowHttpRegistries = true, + }; + + options.Images.Add(new OciImageSubscriptionOptions + { + Reference = "registry.example.com/repo/image:latest", + }); + + options.Offline.RootDirectory = "/bundles"; + options.Cosign.Mode = CosignCredentialMode.None; + + var result = await service.LoadAsync(options, CancellationToken.None); + + result.Targets.Should().ContainSingle(); + result.Targets[0].OfflineBundle.Should().NotBeNull(); + var offline = result.Targets[0].OfflineBundle!; + offline.Exists.Should().BeTrue(); + var expectedPath = fileSystem.Path.Combine( + fileSystem.Path.GetFullPath("/bundles"), + "registry.example.com", + "repo", + "image", + "latest", + "openvex-attestations.tgz"); + offline.Path.Should().Be(expectedPath); + } + + [Fact] + public async Task LoadAsync_CachesResults() + { + var fileSystem = new MockFileSystem(new Dictionary + { + ["/bundles/registry.example.com/repo/image/latest/openvex-attestations.tgz"] = new MockFileData(string.Empty), + }); + + using var cache = new MemoryCache(new MemoryCacheOptions()); + var service = new OciAttestationDiscoveryService(cache, fileSystem, NullLogger.Instance); + + var options = new OciOpenVexAttestationConnectorOptions + { + AllowHttpRegistries = true, + }; + + options.Images.Add(new OciImageSubscriptionOptions + { + Reference = "registry.example.com/repo/image:latest", + }); + + options.Offline.RootDirectory = "/bundles"; + options.Cosign.Mode = CosignCredentialMode.None; + + var first = await service.LoadAsync(options, CancellationToken.None); + var second = await service.LoadAsync(options, CancellationToken.None); + + ReferenceEquals(first, second).Should().BeTrue(); + } +} diff --git a/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest.Tests/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest.Tests.csproj b/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest.Tests/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest.Tests.csproj new file mode 100644 index 00000000..38a25aaf --- /dev/null +++ b/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest.Tests/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest.Tests.csproj @@ -0,0 +1,18 @@ + + + net10.0 + preview + enable + enable + true + NU1903 + + + + + + + + + + diff --git a/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/Authentication/OciCosignAuthority.cs b/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/Authentication/OciCosignAuthority.cs new file mode 100644 index 00000000..79bd67d3 --- /dev/null +++ b/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/Authentication/OciCosignAuthority.cs @@ -0,0 +1,110 @@ +using System; +using System.IO.Abstractions; +using StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest.Configuration; + +namespace StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest.Authentication; + +public sealed record CosignKeylessIdentity( + string Issuer, + string Subject, + Uri? FulcioUrl, + Uri? RekorUrl, + string? ClientId, + string? ClientSecret, + string? Audience, + string? IdentityToken); + +public sealed record CosignKeyPairIdentity( + string PrivateKeyPath, + string? Password, + string? CertificatePath, + Uri? RekorUrl, + string? FulcioRootPath); + +public sealed record OciCosignAuthority( + CosignCredentialMode Mode, + CosignKeylessIdentity? Keyless, + CosignKeyPairIdentity? KeyPair, + bool RequireSignature, + TimeSpan VerifyTimeout); + +public static class OciCosignAuthorityFactory +{ + public static OciCosignAuthority Create(OciCosignVerificationOptions options, IFileSystem? fileSystem = null) + { + ArgumentNullException.ThrowIfNull(options); + + CosignKeylessIdentity? keyless = null; + CosignKeyPairIdentity? keyPair = null; + + switch (options.Mode) + { + case CosignCredentialMode.None: + break; + + case CosignCredentialMode.Keyless: + keyless = CreateKeyless(options.Keyless); + break; + + case CosignCredentialMode.KeyPair: + keyPair = CreateKeyPair(options.KeyPair, fileSystem); + break; + + default: + throw new InvalidOperationException($"Unsupported Cosign credential mode '{options.Mode}'."); + } + + return new OciCosignAuthority( + Mode: options.Mode, + Keyless: keyless, + KeyPair: keyPair, + RequireSignature: options.RequireSignature, + VerifyTimeout: options.VerifyTimeout); + } + + private static CosignKeylessIdentity CreateKeyless(CosignKeylessOptions options) + { + ArgumentNullException.ThrowIfNull(options); + + Uri? fulcio = null; + Uri? rekor = null; + + if (!string.IsNullOrWhiteSpace(options.FulcioUrl)) + { + fulcio = new Uri(options.FulcioUrl, UriKind.Absolute); + } + + if (!string.IsNullOrWhiteSpace(options.RekorUrl)) + { + rekor = new Uri(options.RekorUrl, UriKind.Absolute); + } + + return new CosignKeylessIdentity( + Issuer: options.Issuer!, + Subject: options.Subject!, + FulcioUrl: fulcio, + RekorUrl: rekor, + ClientId: options.ClientId, + ClientSecret: options.ClientSecret, + Audience: options.Audience, + IdentityToken: options.IdentityToken); + } + + private static CosignKeyPairIdentity CreateKeyPair(CosignKeyPairOptions options, IFileSystem? fileSystem) + { + ArgumentNullException.ThrowIfNull(options); + + Uri? rekor = null; + if (!string.IsNullOrWhiteSpace(options.RekorUrl)) + { + rekor = new Uri(options.RekorUrl, UriKind.Absolute); + } + + return new CosignKeyPairIdentity( + PrivateKeyPath: options.PrivateKeyPath!, + Password: options.Password, + CertificatePath: options.CertificatePath, + RekorUrl: rekor, + FulcioRootPath: options.FulcioRootPath); + } +} diff --git a/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/Authentication/OciRegistryAuthorization.cs b/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/Authentication/OciRegistryAuthorization.cs new file mode 100644 index 00000000..4579c884 --- /dev/null +++ b/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/Authentication/OciRegistryAuthorization.cs @@ -0,0 +1,59 @@ +using System; +using StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest.Configuration; + +namespace StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest.Authentication; + +public enum OciRegistryAuthMode +{ + Anonymous = 0, + Basic = 1, + IdentityToken = 2, + RefreshToken = 3, +} + +public sealed record OciRegistryAuthorization( + string? RegistryAuthority, + OciRegistryAuthMode Mode, + string? Username, + string? Password, + string? IdentityToken, + string? RefreshToken, + bool AllowAnonymousFallback) +{ + public static OciRegistryAuthorization Create(OciRegistryAuthenticationOptions options) + { + ArgumentNullException.ThrowIfNull(options); + + var mode = OciRegistryAuthMode.Anonymous; + string? username = null; + string? password = null; + string? identityToken = null; + string? refreshToken = null; + + if (!string.IsNullOrWhiteSpace(options.IdentityToken)) + { + mode = OciRegistryAuthMode.IdentityToken; + identityToken = options.IdentityToken; + } + else if (!string.IsNullOrWhiteSpace(options.RefreshToken)) + { + mode = OciRegistryAuthMode.RefreshToken; + refreshToken = options.RefreshToken; + } + else if (!string.IsNullOrWhiteSpace(options.Username)) + { + mode = OciRegistryAuthMode.Basic; + username = options.Username; + password = options.Password; + } + + return new OciRegistryAuthorization( + RegistryAuthority: options.RegistryAuthority, + Mode: mode, + Username: username, + Password: password, + IdentityToken: identityToken, + RefreshToken: refreshToken, + AllowAnonymousFallback: options.AllowAnonymousFallback); + } +} diff --git a/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/Configuration/OciOpenVexAttestationConnectorOptions.cs b/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/Configuration/OciOpenVexAttestationConnectorOptions.cs new file mode 100644 index 00000000..9b102f67 --- /dev/null +++ b/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/Configuration/OciOpenVexAttestationConnectorOptions.cs @@ -0,0 +1,321 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.IO.Abstractions; +using System.Linq; +using StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest.Discovery; + +namespace StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest.Configuration; + +public sealed class OciOpenVexAttestationConnectorOptions +{ + public const string HttpClientName = "excititor.connector.oci.openvex.attest"; + + public IList Images { get; } = new List(); + + public OciRegistryAuthenticationOptions Registry { get; } = new(); + + public OciCosignVerificationOptions Cosign { get; } = new(); + + public OciOfflineBundleOptions Offline { get; } = new(); + + public TimeSpan DiscoveryCacheDuration { get; set; } = TimeSpan.FromMinutes(15); + + public int MaxParallelResolutions { get; set; } = 4; + + public bool AllowHttpRegistries { get; set; } + + public void Validate(IFileSystem? fileSystem = null) + { + if (Images.Count == 0) + { + throw new InvalidOperationException("At least one OCI image reference must be configured."); + } + + foreach (var image in Images) + { + image.Validate(); + } + + if (MaxParallelResolutions <= 0 || MaxParallelResolutions > 32) + { + throw new InvalidOperationException("MaxParallelResolutions must be between 1 and 32."); + } + + if (DiscoveryCacheDuration <= TimeSpan.Zero) + { + throw new InvalidOperationException("DiscoveryCacheDuration must be a positive time span."); + } + + Registry.Validate(); + Cosign.Validate(fileSystem); + Offline.Validate(fileSystem); + + if (!AllowHttpRegistries && Images.Any(i => i.Reference is not null && i.Reference.StartsWith("http://", StringComparison.OrdinalIgnoreCase))) + { + throw new InvalidOperationException("HTTP (non-TLS) registries are disabled. Enable AllowHttpRegistries to permit them."); + } + } +} + +public sealed class OciImageSubscriptionOptions +{ + private OciImageReference? _parsedReference; + + /// + /// Gets or sets the OCI reference (e.g. registry.example.com/repository:tag or registry.example.com/repository@sha256:abcdef). + /// + public string? Reference { get; set; } + + /// + /// Optional friendly name used in logs when referencing this subscription. + /// + public string? DisplayName { get; set; } + + /// + /// Optional file path for an offline attestation bundle associated with this image. + /// + public string? OfflineBundlePath { get; set; } + + /// + /// Optional override for the expected subject digest. When provided, discovery will verify resolved digests match. + /// + public string? ExpectedSubjectDigest { get; set; } + + internal OciImageReference? ParsedReference => _parsedReference; + + public void Validate() + { + if (string.IsNullOrWhiteSpace(Reference)) + { + throw new InvalidOperationException("Image Reference is required for OCI OpenVEX attestation connector."); + } + + _parsedReference = OciImageReferenceParser.Parse(Reference); + + if (!string.IsNullOrWhiteSpace(ExpectedSubjectDigest)) + { + if (!ExpectedSubjectDigest.StartsWith("sha256:", StringComparison.OrdinalIgnoreCase)) + { + throw new InvalidOperationException("ExpectedSubjectDigest must start with 'sha256:'."); + } + + if (ExpectedSubjectDigest.Length != "sha256:".Length + 64) + { + throw new InvalidOperationException("ExpectedSubjectDigest must contain a 64-character hexadecimal hash."); + } + } + } +} + +public sealed class OciRegistryAuthenticationOptions +{ + /// + /// Optional registry authority filter (e.g. registry.example.com:5000). When set it must match image references. + /// + public string? RegistryAuthority { get; set; } + + public string? Username { get; set; } + + public string? Password { get; set; } + + public string? IdentityToken { get; set; } + + public string? RefreshToken { get; set; } + + public bool AllowAnonymousFallback { get; set; } = true; + + public void Validate() + { + var hasUser = !string.IsNullOrWhiteSpace(Username); + var hasPassword = !string.IsNullOrWhiteSpace(Password); + var hasIdentityToken = !string.IsNullOrWhiteSpace(IdentityToken); + var hasRefreshToken = !string.IsNullOrWhiteSpace(RefreshToken); + + if (hasIdentityToken && (hasUser || hasPassword)) + { + throw new InvalidOperationException("IdentityToken cannot be combined with Username/Password for OCI registry authentication."); + } + + if (hasRefreshToken && (hasUser || hasPassword)) + { + throw new InvalidOperationException("RefreshToken cannot be combined with Username/Password for OCI registry authentication."); + } + + if (hasUser != hasPassword) + { + throw new InvalidOperationException("Username and Password must be provided together for OCI registry authentication."); + } + + if (!string.IsNullOrWhiteSpace(RegistryAuthority) && RegistryAuthority.Contains('/', StringComparison.Ordinal)) + { + throw new InvalidOperationException("RegistryAuthority must not contain path segments."); + } + } +} + +public sealed class OciCosignVerificationOptions +{ + public CosignCredentialMode Mode { get; set; } = CosignCredentialMode.Keyless; + + public CosignKeylessOptions Keyless { get; } = new(); + + public CosignKeyPairOptions KeyPair { get; } = new(); + + public bool RequireSignature { get; set; } = true; + + public TimeSpan VerifyTimeout { get; set; } = TimeSpan.FromSeconds(30); + + public void Validate(IFileSystem? fileSystem = null) + { + if (VerifyTimeout <= TimeSpan.Zero) + { + throw new InvalidOperationException("VerifyTimeout must be a positive time span."); + } + + switch (Mode) + { + case CosignCredentialMode.None: + break; + + case CosignCredentialMode.Keyless: + Keyless.Validate(); + break; + + case CosignCredentialMode.KeyPair: + KeyPair.Validate(fileSystem); + break; + + default: + throw new InvalidOperationException($"Unsupported Cosign credential mode '{Mode}'."); + } + } +} + +public enum CosignCredentialMode +{ + None = 0, + Keyless = 1, + KeyPair = 2, +} + +public sealed class CosignKeylessOptions +{ + public string? Issuer { get; set; } + + public string? Subject { get; set; } + + public string? FulcioUrl { get; set; } = "https://fulcio.sigstore.dev"; + + public string? RekorUrl { get; set; } = "https://rekor.sigstore.dev"; + + public string? ClientId { get; set; } + + public string? ClientSecret { get; set; } + + public string? Audience { get; set; } + + public string? IdentityToken { get; set; } + + public void Validate() + { + if (string.IsNullOrWhiteSpace(Issuer)) + { + throw new InvalidOperationException("Cosign keyless Issuer must be provided."); + } + + if (string.IsNullOrWhiteSpace(Subject)) + { + throw new InvalidOperationException("Cosign keyless Subject must be provided."); + } + + if (!string.IsNullOrWhiteSpace(FulcioUrl) && !Uri.TryCreate(FulcioUrl, UriKind.Absolute, out var fulcio)) + { + throw new InvalidOperationException("FulcioUrl must be an absolute URI when provided."); + } + + if (!string.IsNullOrWhiteSpace(RekorUrl) && !Uri.TryCreate(RekorUrl, UriKind.Absolute, out var rekor)) + { + throw new InvalidOperationException("RekorUrl must be an absolute URI when provided."); + } + + if (!string.IsNullOrWhiteSpace(ClientSecret) && string.IsNullOrWhiteSpace(ClientId)) + { + throw new InvalidOperationException("Cosign keyless ClientId must be provided when ClientSecret is specified."); + } + } +} + +public sealed class CosignKeyPairOptions +{ + public string? PrivateKeyPath { get; set; } + + public string? Password { get; set; } + + public string? CertificatePath { get; set; } + + public string? RekorUrl { get; set; } + + public string? FulcioRootPath { get; set; } + + public void Validate(IFileSystem? fileSystem = null) + { + if (string.IsNullOrWhiteSpace(PrivateKeyPath)) + { + throw new InvalidOperationException("PrivateKeyPath must be provided for Cosign key pair mode."); + } + + var fs = fileSystem ?? new FileSystem(); + if (!fs.File.Exists(PrivateKeyPath)) + { + throw new InvalidOperationException($"Cosign private key file not found: {PrivateKeyPath}"); + } + + if (!string.IsNullOrWhiteSpace(CertificatePath) && !fs.File.Exists(CertificatePath)) + { + throw new InvalidOperationException($"Cosign certificate file not found: {CertificatePath}"); + } + + if (!string.IsNullOrWhiteSpace(FulcioRootPath) && !fs.File.Exists(FulcioRootPath)) + { + throw new InvalidOperationException($"Cosign Fulcio root file not found: {FulcioRootPath}"); + } + + if (!string.IsNullOrWhiteSpace(RekorUrl) && !Uri.TryCreate(RekorUrl, UriKind.Absolute, out _)) + { + throw new InvalidOperationException("RekorUrl must be an absolute URI when provided for Cosign key pair mode."); + } + } +} + +public sealed class OciOfflineBundleOptions +{ + public string? RootDirectory { get; set; } + + public bool PreferOffline { get; set; } + + public bool AllowNetworkFallback { get; set; } = true; + + public string? DefaultBundleFileName { get; set; } = "openvex-attestations.tgz"; + + public bool RequireBundles { get; set; } + + public void Validate(IFileSystem? fileSystem = null) + { + if (string.IsNullOrWhiteSpace(RootDirectory)) + { + return; + } + + var fs = fileSystem ?? new FileSystem(); + if (!fs.Directory.Exists(RootDirectory)) + { + if (PreferOffline || RequireBundles) + { + throw new InvalidOperationException($"Offline bundle root directory '{RootDirectory}' does not exist."); + } + + fs.Directory.CreateDirectory(RootDirectory); + } + } +} diff --git a/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/Configuration/OciOpenVexAttestationConnectorOptionsValidator.cs b/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/Configuration/OciOpenVexAttestationConnectorOptionsValidator.cs new file mode 100644 index 00000000..120cd0c0 --- /dev/null +++ b/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/Configuration/OciOpenVexAttestationConnectorOptionsValidator.cs @@ -0,0 +1,35 @@ +using System; +using System.Collections.Generic; +using System.IO.Abstractions; +using StellaOps.Excititor.Connectors.Abstractions; + +namespace StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest.Configuration; + +public sealed class OciOpenVexAttestationConnectorOptionsValidator : IVexConnectorOptionsValidator +{ + private readonly IFileSystem _fileSystem; + + public OciOpenVexAttestationConnectorOptionsValidator(IFileSystem fileSystem) + { + _fileSystem = fileSystem ?? throw new ArgumentNullException(nameof(fileSystem)); + } + + public void Validate( + VexConnectorDescriptor descriptor, + OciOpenVexAttestationConnectorOptions options, + IList errors) + { + ArgumentNullException.ThrowIfNull(descriptor); + ArgumentNullException.ThrowIfNull(options); + ArgumentNullException.ThrowIfNull(errors); + + try + { + options.Validate(_fileSystem); + } + catch (Exception ex) + { + errors.Add(ex.Message); + } + } +} diff --git a/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/DependencyInjection/OciOpenVexAttestationConnectorServiceCollectionExtensions.cs b/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/DependencyInjection/OciOpenVexAttestationConnectorServiceCollectionExtensions.cs new file mode 100644 index 00000000..4dbbccb9 --- /dev/null +++ b/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/DependencyInjection/OciOpenVexAttestationConnectorServiceCollectionExtensions.cs @@ -0,0 +1,52 @@ +using System; +using System.Net; +using System.Net.Http; +using Microsoft.Extensions.Caching.Memory; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.DependencyInjection.Extensions; +using StellaOps.Excititor.Connectors.Abstractions; +using StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest.Configuration; +using StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest.Fetch; +using StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest.Discovery; +using StellaOps.Excititor.Core; +using System.IO.Abstractions; + +namespace StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest.DependencyInjection; + +public static class OciOpenVexAttestationConnectorServiceCollectionExtensions +{ + public static IServiceCollection AddOciOpenVexAttestationConnector( + this IServiceCollection services, + Action? configure = null) + { + ArgumentNullException.ThrowIfNull(services); + + services.TryAddSingleton(); + services.TryAddSingleton(); + + services.AddOptions() + .Configure(options => + { + configure?.Invoke(options); + }); + + services.AddSingleton, OciOpenVexAttestationConnectorOptionsValidator>(); + services.AddSingleton(); + services.AddSingleton(); + services.AddSingleton(); + + services.AddHttpClient(OciOpenVexAttestationConnectorOptions.HttpClientName, client => + { + client.Timeout = TimeSpan.FromSeconds(30); + client.DefaultRequestHeaders.UserAgent.ParseAdd("StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/1.0"); + client.DefaultRequestHeaders.Accept.ParseAdd("application/vnd.cncf.openvex.v1+json"); + client.DefaultRequestHeaders.Accept.ParseAdd("application/json"); + }) + .ConfigurePrimaryHttpMessageHandler(() => new HttpClientHandler + { + AutomaticDecompression = DecompressionMethods.All, + }); + + return services; + } +} diff --git a/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/Discovery/OciAttestationDiscoveryResult.cs b/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/Discovery/OciAttestationDiscoveryResult.cs new file mode 100644 index 00000000..84bf373d --- /dev/null +++ b/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/Discovery/OciAttestationDiscoveryResult.cs @@ -0,0 +1,11 @@ +using System.Collections.Immutable; +using StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest.Authentication; + +namespace StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest.Discovery; + +public sealed record OciAttestationDiscoveryResult( + ImmutableArray Targets, + OciRegistryAuthorization RegistryAuthorization, + OciCosignAuthority CosignAuthority, + bool PreferOffline, + bool AllowNetworkFallback); diff --git a/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/Discovery/OciAttestationDiscoveryService.cs b/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/Discovery/OciAttestationDiscoveryService.cs new file mode 100644 index 00000000..8d11d57e --- /dev/null +++ b/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/Discovery/OciAttestationDiscoveryService.cs @@ -0,0 +1,188 @@ +using System; +using System.Collections.Generic; +using System.Collections.Immutable; +using System.IO.Abstractions; +using System.Linq; +using System.Security.Cryptography; +using System.Text; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Caching.Memory; +using Microsoft.Extensions.Logging; +using StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest.Authentication; +using StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest.Configuration; + +namespace StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest.Discovery; + +public sealed class OciAttestationDiscoveryService +{ + private readonly IMemoryCache _memoryCache; + private readonly IFileSystem _fileSystem; + private readonly ILogger _logger; + + public OciAttestationDiscoveryService( + IMemoryCache memoryCache, + IFileSystem fileSystem, + ILogger logger) + { + _memoryCache = memoryCache ?? throw new ArgumentNullException(nameof(memoryCache)); + _fileSystem = fileSystem ?? throw new ArgumentNullException(nameof(fileSystem)); + _logger = logger ?? throw new ArgumentNullException(nameof(logger)); + } + + public Task LoadAsync( + OciOpenVexAttestationConnectorOptions options, + CancellationToken cancellationToken) + { + ArgumentNullException.ThrowIfNull(options); + + cancellationToken.ThrowIfCancellationRequested(); + + var cacheKey = CreateCacheKey(options); + if (_memoryCache.TryGetValue(cacheKey, out OciAttestationDiscoveryResult? cached) && cached is not null) + { + _logger.LogDebug("Using cached OCI attestation discovery result for {ImageCount} images.", cached.Targets.Length); + return Task.FromResult(cached); + } + + var targets = new List(options.Images.Count); + foreach (var image in options.Images) + { + cancellationToken.ThrowIfCancellationRequested(); + + var parsed = image.ParsedReference ?? OciImageReferenceParser.Parse(image.Reference!); + var offlinePath = ResolveOfflinePath(options, image, parsed); + + OciOfflineBundleReference? offline = null; + if (!string.IsNullOrWhiteSpace(offlinePath)) + { + var fullPath = _fileSystem.Path.GetFullPath(offlinePath!); + var exists = _fileSystem.File.Exists(fullPath) || _fileSystem.Directory.Exists(fullPath); + + if (!exists && options.Offline.RequireBundles) + { + throw new InvalidOperationException($"Required offline bundle '{fullPath}' for reference '{parsed.Canonical}' was not found."); + } + + offline = new OciOfflineBundleReference(fullPath, exists, image.ExpectedSubjectDigest); + } + + targets.Add(new OciAttestationTarget(parsed, image.ExpectedSubjectDigest, offline)); + } + + var authorization = OciRegistryAuthorization.Create(options.Registry); + var cosignAuthority = OciCosignAuthorityFactory.Create(options.Cosign, _fileSystem); + + var result = new OciAttestationDiscoveryResult( + targets.ToImmutableArray(), + authorization, + cosignAuthority, + options.Offline.PreferOffline, + options.Offline.AllowNetworkFallback); + + _memoryCache.Set(cacheKey, result, options.DiscoveryCacheDuration); + + return Task.FromResult(result); + } + + private string? ResolveOfflinePath( + OciOpenVexAttestationConnectorOptions options, + OciImageSubscriptionOptions image, + OciImageReference parsed) + { + if (!string.IsNullOrWhiteSpace(image.OfflineBundlePath)) + { + return image.OfflineBundlePath; + } + + if (string.IsNullOrWhiteSpace(options.Offline.RootDirectory)) + { + return null; + } + + var root = options.Offline.RootDirectory!; + var segments = new List { SanitizeSegment(parsed.Registry) }; + + var repositoryParts = parsed.Repository.Split('/', StringSplitOptions.RemoveEmptyEntries); + if (repositoryParts.Length == 0) + { + segments.Add(SanitizeSegment(parsed.Repository)); + } + else + { + foreach (var part in repositoryParts) + { + segments.Add(SanitizeSegment(part)); + } + } + + var versionSegment = parsed.Digest is not null + ? SanitizeSegment(parsed.Digest) + : SanitizeSegment(parsed.Tag ?? "latest"); + + segments.Add(versionSegment); + + var combined = _fileSystem.Path.Combine(new[] { root }.Concat(segments).ToArray()); + + if (!string.IsNullOrWhiteSpace(options.Offline.DefaultBundleFileName)) + { + combined = _fileSystem.Path.Combine(combined, options.Offline.DefaultBundleFileName!); + } + + return combined; + } + + private static string SanitizeSegment(string value) + { + if (string.IsNullOrEmpty(value)) + { + return "_"; + } + + var builder = new StringBuilder(value.Length); + foreach (var ch in value) + { + if (char.IsLetterOrDigit(ch) || ch is '-' or '_' or '.') + { + builder.Append(ch); + } + else + { + builder.Append('_'); + } + } + + return builder.Length == 0 ? "_" : builder.ToString(); + } + + private static string CreateCacheKey(OciOpenVexAttestationConnectorOptions options) + { + using var sha = SHA256.Create(); + var builder = new StringBuilder(); + builder.AppendLine("oci-openvex-attest"); + builder.AppendLine(options.MaxParallelResolutions.ToString()); + builder.AppendLine(options.AllowHttpRegistries.ToString()); + builder.AppendLine(options.Offline.PreferOffline.ToString()); + builder.AppendLine(options.Offline.AllowNetworkFallback.ToString()); + + foreach (var image in options.Images) + { + builder.AppendLine(image.Reference ?? string.Empty); + builder.AppendLine(image.ExpectedSubjectDigest ?? string.Empty); + builder.AppendLine(image.OfflineBundlePath ?? string.Empty); + } + + if (!string.IsNullOrWhiteSpace(options.Offline.RootDirectory)) + { + builder.AppendLine(options.Offline.RootDirectory); + builder.AppendLine(options.Offline.DefaultBundleFileName ?? string.Empty); + } + + builder.AppendLine(options.Registry.RegistryAuthority ?? string.Empty); + builder.AppendLine(options.Registry.AllowAnonymousFallback.ToString()); + + var bytes = Encoding.UTF8.GetBytes(builder.ToString()); + var hashBytes = sha.ComputeHash(bytes); + return Convert.ToHexString(hashBytes); + } +} diff --git a/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/Discovery/OciAttestationTarget.cs b/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/Discovery/OciAttestationTarget.cs new file mode 100644 index 00000000..6001fb31 --- /dev/null +++ b/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/Discovery/OciAttestationTarget.cs @@ -0,0 +1,6 @@ +namespace StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest.Discovery; + +public sealed record OciAttestationTarget( + OciImageReference Image, + string? ExpectedSubjectDigest, + OciOfflineBundleReference? OfflineBundle); diff --git a/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/Discovery/OciImageReference.cs b/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/Discovery/OciImageReference.cs new file mode 100644 index 00000000..bf53d2dc --- /dev/null +++ b/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/Discovery/OciImageReference.cs @@ -0,0 +1,27 @@ +using System; + +namespace StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest.Discovery; + +public sealed record OciImageReference(string Registry, string Repository, string? Tag, string? Digest, string OriginalReference, string Scheme = "https") +{ + public string Canonical => + Digest is not null + ? $"{Registry}/{Repository}@{Digest}" + : Tag is not null + ? $"{Registry}/{Repository}:{Tag}" + : $"{Registry}/{Repository}"; + + public bool HasDigest => !string.IsNullOrWhiteSpace(Digest); + + public bool HasTag => !string.IsNullOrWhiteSpace(Tag); + + public OciImageReference WithDigest(string digest) + { + if (string.IsNullOrWhiteSpace(digest)) + { + throw new ArgumentException("Digest must be provided.", nameof(digest)); + } + + return this with { Digest = digest }; + } +} diff --git a/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/Discovery/OciImageReferenceParser.cs b/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/Discovery/OciImageReferenceParser.cs new file mode 100644 index 00000000..c6fe4f3a --- /dev/null +++ b/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/Discovery/OciImageReferenceParser.cs @@ -0,0 +1,129 @@ +using System; +using System.Text.RegularExpressions; + +namespace StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest.Discovery; + +internal static class OciImageReferenceParser +{ + private static readonly Regex DigestRegex = new(@"^(?[A-Za-z0-9+._-]+):(?[A-Fa-f0-9]{32,})$", RegexOptions.Compiled | RegexOptions.CultureInvariant); + private static readonly Regex RepositoryRegex = new(@"^[a-z0-9]+(?:(?:[._]|__|[-]*)[a-z0-9]+)*(?:/[a-z0-9]+(?:(?:[._]|__|[-]*)[a-z0-9]+)*)*$", RegexOptions.Compiled | RegexOptions.CultureInvariant); + + public static OciImageReference Parse(string reference) + { + if (string.IsNullOrWhiteSpace(reference)) + { + throw new InvalidOperationException("OCI reference cannot be empty."); + } + + var trimmed = reference.Trim(); + string original = trimmed; + + var scheme = "https"; + if (trimmed.StartsWith("oci://", StringComparison.OrdinalIgnoreCase)) + { + trimmed = trimmed.Substring("oci://".Length); + } + + if (trimmed.StartsWith("https://", StringComparison.OrdinalIgnoreCase)) + { + trimmed = trimmed.Substring("https://".Length); + scheme = "https"; + } + else if (trimmed.StartsWith("http://", StringComparison.OrdinalIgnoreCase)) + { + trimmed = trimmed.Substring("http://".Length); + scheme = "http"; + } + + var firstSlash = trimmed.IndexOf('/'); + if (firstSlash <= 0) + { + throw new InvalidOperationException($"OCI reference '{reference}' must include a registry and repository component."); + } + + var registry = trimmed[..firstSlash]; + var remainder = trimmed[(firstSlash + 1)..]; + + if (!LooksLikeRegistry(registry)) + { + throw new InvalidOperationException($"OCI reference '{reference}' is missing an explicit registry component."); + } + + string? digest = null; + string? tag = null; + + var digestIndex = remainder.IndexOf('@'); + if (digestIndex >= 0) + { + digest = remainder[(digestIndex + 1)..]; + remainder = remainder[..digestIndex]; + + if (!DigestRegex.IsMatch(digest)) + { + throw new InvalidOperationException($"Digest segment '{digest}' is not a valid OCI digest."); + } + } + + var tagIndex = remainder.LastIndexOf(':'); + if (tagIndex >= 0) + { + tag = remainder[(tagIndex + 1)..]; + remainder = remainder[..tagIndex]; + + if (string.IsNullOrWhiteSpace(tag)) + { + throw new InvalidOperationException("OCI tag segment cannot be empty."); + } + + if (tag.Contains('/', StringComparison.Ordinal)) + { + throw new InvalidOperationException("OCI tag segment cannot contain '/'."); + } + } + + var repository = remainder; + if (string.IsNullOrWhiteSpace(repository)) + { + throw new InvalidOperationException("OCI repository segment cannot be empty."); + } + + if (!RepositoryRegex.IsMatch(repository)) + { + throw new InvalidOperationException($"Repository segment '{repository}' is not valid per OCI distribution rules."); + } + + return new OciImageReference( + Registry: registry, + Repository: repository, + Tag: tag, + Digest: digest, + OriginalReference: original, + Scheme: scheme); + } + + private static bool LooksLikeRegistry(string value) + { + if (string.IsNullOrWhiteSpace(value)) + { + return false; + } + + if (value.Equals("localhost", StringComparison.OrdinalIgnoreCase)) + { + return true; + } + + if (value.Contains('.', StringComparison.Ordinal) || value.Contains(':', StringComparison.Ordinal)) + { + return true; + } + + // IPv4/IPv6 simplified check + if (value.Length >= 3 && char.IsDigit(value[0])) + { + return true; + } + + return false; + } +} diff --git a/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/Discovery/OciOfflineBundleReference.cs b/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/Discovery/OciOfflineBundleReference.cs new file mode 100644 index 00000000..407222de --- /dev/null +++ b/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/Discovery/OciOfflineBundleReference.cs @@ -0,0 +1,5 @@ +using System; + +namespace StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest.Discovery; + +public sealed record OciOfflineBundleReference(string Path, bool Exists, string? ExpectedSubjectDigest); diff --git a/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/Fetch/OciArtifactDescriptor.cs b/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/Fetch/OciArtifactDescriptor.cs new file mode 100644 index 00000000..5dfd6566 --- /dev/null +++ b/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/Fetch/OciArtifactDescriptor.cs @@ -0,0 +1,14 @@ +using System.Collections.Generic; +using System.Text.Json.Serialization; + +namespace StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest.Fetch; + +internal sealed record OciArtifactDescriptor( + [property: JsonPropertyName("digest")] string Digest, + [property: JsonPropertyName("mediaType")] string MediaType, + [property: JsonPropertyName("artifactType")] string? ArtifactType, + [property: JsonPropertyName("size")] long Size, + [property: JsonPropertyName("annotations")] IReadOnlyDictionary? Annotations); + +internal sealed record OciReferrerIndex( + [property: JsonPropertyName("referrers")] IReadOnlyList Referrers); diff --git a/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/Fetch/OciAttestationDocument.cs b/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/Fetch/OciAttestationDocument.cs new file mode 100644 index 00000000..4e7bebb1 --- /dev/null +++ b/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/Fetch/OciAttestationDocument.cs @@ -0,0 +1,13 @@ +using System; +using System.Collections.Immutable; + +namespace StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest.Fetch; + +public sealed record OciAttestationDocument( + Uri SourceUri, + ReadOnlyMemory Content, + ImmutableDictionary Metadata, + string? SubjectDigest, + string? ArtifactDigest, + string? ArtifactType, + string SourceKind); diff --git a/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/Fetch/OciAttestationFetcher.cs b/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/Fetch/OciAttestationFetcher.cs new file mode 100644 index 00000000..4f594e07 --- /dev/null +++ b/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/Fetch/OciAttestationFetcher.cs @@ -0,0 +1,258 @@ +using System; +using System.Collections.Generic; +using System.Collections.Immutable; +using System.IO; +using System.IO.Abstractions; +using System.IO.Compression; +using System.Linq; +using System.Text; +using System.Threading; +using System.Threading.Tasks; +using System.Net.Http; +using Microsoft.Extensions.Logging; +using StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest.Authentication; +using StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest.Configuration; +using StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest.Discovery; +using System.Formats.Tar; + +namespace StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest.Fetch; + +public sealed class OciAttestationFetcher +{ + private readonly IHttpClientFactory _httpClientFactory; + private readonly IFileSystem _fileSystem; + private readonly ILogger _logger; + + public OciAttestationFetcher( + IHttpClientFactory httpClientFactory, + IFileSystem fileSystem, + ILogger logger) + { + _httpClientFactory = httpClientFactory ?? throw new ArgumentNullException(nameof(httpClientFactory)); + _fileSystem = fileSystem ?? throw new ArgumentNullException(nameof(fileSystem)); + _logger = logger ?? throw new ArgumentNullException(nameof(logger)); + } + + public async IAsyncEnumerable FetchAsync( + OciAttestationDiscoveryResult discovery, + OciOpenVexAttestationConnectorOptions options, + [System.Runtime.CompilerServices.EnumeratorCancellation] CancellationToken cancellationToken) + { + ArgumentNullException.ThrowIfNull(discovery); + ArgumentNullException.ThrowIfNull(options); + + foreach (var target in discovery.Targets) + { + cancellationToken.ThrowIfCancellationRequested(); + + bool yieldedOffline = false; + if (target.OfflineBundle is not null && target.OfflineBundle.Exists) + { + await foreach (var offlineDocument in ReadOfflineAsync(target, cancellationToken)) + { + yieldedOffline = true; + yield return offlineDocument; + } + + if (!discovery.AllowNetworkFallback) + { + continue; + } + } + + if (discovery.PreferOffline && yieldedOffline && !discovery.AllowNetworkFallback) + { + continue; + } + + if (!discovery.PreferOffline || discovery.AllowNetworkFallback || !yieldedOffline) + { + await foreach (var registryDocument in FetchFromRegistryAsync(discovery, options, target, cancellationToken)) + { + yield return registryDocument; + } + } + } + } + + private async IAsyncEnumerable ReadOfflineAsync( + OciAttestationTarget target, + [System.Runtime.CompilerServices.EnumeratorCancellation] CancellationToken cancellationToken) + { + var offline = target.OfflineBundle!; + var path = _fileSystem.Path.GetFullPath(offline.Path); + + if (!_fileSystem.File.Exists(path)) + { + if (offline.Exists) + { + _logger.LogWarning("Offline bundle {Path} disappeared before processing.", path); + } + yield break; + } + + var extension = _fileSystem.Path.GetExtension(path).ToLowerInvariant(); + var subjectDigest = target.Image.Digest ?? target.ExpectedSubjectDigest; + + if (string.Equals(extension, ".json", StringComparison.OrdinalIgnoreCase) || + string.Equals(extension, ".dsse", StringComparison.OrdinalIgnoreCase)) + { + var bytes = await _fileSystem.File.ReadAllBytesAsync(path, cancellationToken).ConfigureAwait(false); + var metadata = BuildOfflineMetadata(target, path, entryName: null, subjectDigest); + yield return new OciAttestationDocument( + new Uri(path, UriKind.Absolute), + bytes, + metadata, + subjectDigest, + null, + null, + "offline"); + yield break; + } + + if (string.Equals(extension, ".tgz", StringComparison.OrdinalIgnoreCase) || + string.Equals(extension, ".gz", StringComparison.OrdinalIgnoreCase) || + string.Equals(extension, ".tar", StringComparison.OrdinalIgnoreCase)) + { + await foreach (var document in ReadTarArchiveAsync(target, path, subjectDigest, cancellationToken)) + { + yield return document; + } + yield break; + } + + // Default: treat as binary blob. + var fallbackBytes = await _fileSystem.File.ReadAllBytesAsync(path, cancellationToken).ConfigureAwait(false); + var fallbackMetadata = BuildOfflineMetadata(target, path, entryName: null, subjectDigest); + yield return new OciAttestationDocument( + new Uri(path, UriKind.Absolute), + fallbackBytes, + fallbackMetadata, + subjectDigest, + null, + null, + "offline"); + } + + private async IAsyncEnumerable ReadTarArchiveAsync( + OciAttestationTarget target, + string path, + string? subjectDigest, + [System.Runtime.CompilerServices.EnumeratorCancellation] CancellationToken cancellationToken) + { + await using var fileStream = _fileSystem.File.OpenRead(path); + Stream archiveStream = fileStream; + + if (path.EndsWith(".gz", StringComparison.OrdinalIgnoreCase) || + path.EndsWith(".tgz", StringComparison.OrdinalIgnoreCase)) + { + archiveStream = new GZipStream(fileStream, CompressionMode.Decompress, leaveOpen: false); + } + + using var tarReader = new TarReader(archiveStream, leaveOpen: false); + TarEntry? entry; + + while ((entry = await tarReader.GetNextEntryAsync(copyData: false, cancellationToken).ConfigureAwait(false)) is not null) + { + if (entry.EntryType is not TarEntryType.RegularFile || entry.DataStream is null) + { + continue; + } + + await using var entryStream = entry.DataStream; + using var buffer = new MemoryStream(); + await entryStream.CopyToAsync(buffer, cancellationToken).ConfigureAwait(false); + + var metadata = BuildOfflineMetadata(target, path, entry.Name, subjectDigest); + var sourceUri = new Uri($"{_fileSystem.Path.GetFullPath(path)}#{entry.Name}", UriKind.Absolute); + yield return new OciAttestationDocument( + sourceUri, + buffer.ToArray(), + metadata, + subjectDigest, + null, + null, + "offline"); + } + } + + private async IAsyncEnumerable FetchFromRegistryAsync( + OciAttestationDiscoveryResult discovery, + OciOpenVexAttestationConnectorOptions options, + OciAttestationTarget target, + [System.Runtime.CompilerServices.EnumeratorCancellation] CancellationToken cancellationToken) + { + var registryClient = new OciRegistryClient( + _httpClientFactory, + _logger, + discovery.RegistryAuthorization, + options); + + var subjectDigest = target.Image.Digest ?? target.ExpectedSubjectDigest; + if (string.IsNullOrWhiteSpace(subjectDigest)) + { + subjectDigest = await registryClient.ResolveDigestAsync(target.Image, cancellationToken).ConfigureAwait(false); + } + + if (string.IsNullOrWhiteSpace(subjectDigest)) + { + _logger.LogWarning("Unable to resolve subject digest for {Reference}; skipping registry fetch.", target.Image.Canonical); + yield break; + } + + if (!string.IsNullOrWhiteSpace(target.ExpectedSubjectDigest) && + !string.Equals(target.ExpectedSubjectDigest, subjectDigest, StringComparison.OrdinalIgnoreCase)) + { + _logger.LogWarning( + "Resolved digest {Resolved} does not match expected digest {Expected} for {Reference}.", + subjectDigest, + target.ExpectedSubjectDigest, + target.Image.Canonical); + } + + var descriptors = await registryClient.ListReferrersAsync(target.Image, subjectDigest, cancellationToken).ConfigureAwait(false); + if (descriptors.Count == 0) + { + yield break; + } + + foreach (var descriptor in descriptors) + { + cancellationToken.ThrowIfCancellationRequested(); + var document = await registryClient.DownloadAttestationAsync(target.Image, descriptor, subjectDigest, cancellationToken).ConfigureAwait(false); + if (document is not null) + { + yield return document; + } + } + } + + private static ImmutableDictionary BuildOfflineMetadata( + OciAttestationTarget target, + string bundlePath, + string? entryName, + string? subjectDigest) + { + var builder = ImmutableDictionary.CreateBuilder(StringComparer.Ordinal); + builder["oci.image.registry"] = target.Image.Registry; + builder["oci.image.repository"] = target.Image.Repository; + builder["oci.image.reference"] = target.Image.Canonical; + if (!string.IsNullOrWhiteSpace(subjectDigest)) + { + builder["oci.image.subjectDigest"] = subjectDigest; + } + if (!string.IsNullOrWhiteSpace(target.ExpectedSubjectDigest)) + { + builder["oci.image.expectedSubjectDigest"] = target.ExpectedSubjectDigest!; + } + + builder["oci.attestation.sourceKind"] = "offline"; + builder["oci.attestation.source"] = bundlePath; + if (!string.IsNullOrWhiteSpace(entryName)) + { + builder["oci.attestation.bundleEntry"] = entryName!; + } + + return builder.ToImmutable(); + } +} diff --git a/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/Fetch/OciRegistryClient.cs b/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/Fetch/OciRegistryClient.cs new file mode 100644 index 00000000..ca102e69 --- /dev/null +++ b/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/Fetch/OciRegistryClient.cs @@ -0,0 +1,362 @@ +using System; +using System.Collections.Generic; +using System.Collections.Immutable; +using System.Globalization; +using System.Linq; +using System.Net; +using System.Net.Http; +using System.Net.Http.Headers; +using System.Text; +using System.Text.Json; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging; +using StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest.Authentication; +using StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest.Configuration; +using StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest.Discovery; + +namespace StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest.Fetch; + +internal sealed class OciRegistryClient +{ + private const string ManifestMediaType = "application/vnd.oci.image.manifest.v1+json"; + private const string ReferrersArtifactType = "application/vnd.dsse.envelope.v1+json"; + private const string DsseMediaType = "application/vnd.dsse.envelope.v1+json"; + private const string OpenVexMediaType = "application/vnd.cncf.openvex.v1+json"; + + private static readonly JsonSerializerOptions SerializerOptions = new(JsonSerializerDefaults.Web); + + private readonly IHttpClientFactory _httpClientFactory; + private readonly ILogger _logger; + private readonly OciRegistryAuthorization _authorization; + private readonly OciOpenVexAttestationConnectorOptions _options; + + public OciRegistryClient( + IHttpClientFactory httpClientFactory, + ILogger logger, + OciRegistryAuthorization authorization, + OciOpenVexAttestationConnectorOptions options) + { + _httpClientFactory = httpClientFactory ?? throw new ArgumentNullException(nameof(httpClientFactory)); + _logger = logger ?? throw new ArgumentNullException(nameof(logger)); + _authorization = authorization ?? throw new ArgumentNullException(nameof(authorization)); + _options = options ?? throw new ArgumentNullException(nameof(options)); + } + + public async Task ResolveDigestAsync(OciImageReference image, CancellationToken cancellationToken) + { + ArgumentNullException.ThrowIfNull(image); + + if (image.HasDigest) + { + return image.Digest; + } + + var requestUri = BuildRegistryUri(image, $"manifests/{EscapeReference(image.Tag ?? "latest")}"); + + async Task RequestFactory() + { + var request = new HttpRequestMessage(HttpMethod.Head, requestUri); + request.Headers.Accept.ParseAdd(ManifestMediaType); + ApplyAuthentication(request); + return await Task.FromResult(request).ConfigureAwait(false); + } + + using var response = await SendAsync(RequestFactory, cancellationToken).ConfigureAwait(false); + + if (!response.IsSuccessStatusCode) + { + if (response.StatusCode == HttpStatusCode.NotFound) + { + _logger.LogWarning("Failed to resolve digest for {Reference}; registry returned 404.", image.Canonical); + return null; + } + + response.EnsureSuccessStatusCode(); + } + + if (response.Headers.TryGetValues("Docker-Content-Digest", out var values)) + { + var digest = values.FirstOrDefault(); + if (!string.IsNullOrWhiteSpace(digest)) + { + return digest; + } + } + + // Manifest may have been returned without digest header; fall back to GET. + async Task ManifestRequestFactory() + { + var request = new HttpRequestMessage(HttpMethod.Get, requestUri); + request.Headers.Accept.ParseAdd(ManifestMediaType); + ApplyAuthentication(request); + return await Task.FromResult(request).ConfigureAwait(false); + } + + using var manifestResponse = await SendAsync(ManifestRequestFactory, cancellationToken).ConfigureAwait(false); + manifestResponse.EnsureSuccessStatusCode(); + + if (manifestResponse.Headers.TryGetValues("Docker-Content-Digest", out var manifestValues)) + { + return manifestValues.FirstOrDefault(); + } + + _logger.LogWarning("Registry {Registry} did not provide Docker-Content-Digest header for {Reference}.", image.Registry, image.Canonical); + return null; + } + + public async Task> ListReferrersAsync( + OciImageReference image, + string subjectDigest, + CancellationToken cancellationToken) + { + ArgumentNullException.ThrowIfNull(image); + ArgumentNullException.ThrowIfNull(subjectDigest); + + var query = $"artifactType={Uri.EscapeDataString(ReferrersArtifactType)}"; + var requestUri = BuildRegistryUri(image, $"referrers/{subjectDigest}", query); + + async Task RequestFactory() + { + var request = new HttpRequestMessage(HttpMethod.Get, requestUri); + ApplyAuthentication(request); + request.Headers.Accept.ParseAdd("application/json"); + return await Task.FromResult(request).ConfigureAwait(false); + } + + using var response = await SendAsync(RequestFactory, cancellationToken).ConfigureAwait(false); + if (!response.IsSuccessStatusCode) + { + if (response.StatusCode == HttpStatusCode.NotFound) + { + _logger.LogDebug("Registry returned 404 for referrers on {Subject}.", subjectDigest); + return Array.Empty(); + } + + response.EnsureSuccessStatusCode(); + } + + await using var stream = await response.Content.ReadAsStreamAsync(cancellationToken).ConfigureAwait(false); + var index = await JsonSerializer.DeserializeAsync(stream, SerializerOptions, cancellationToken).ConfigureAwait(false); + return index?.Referrers ?? Array.Empty(); + } + + public async Task DownloadAttestationAsync( + OciImageReference image, + OciArtifactDescriptor descriptor, + string subjectDigest, + CancellationToken cancellationToken) + { + ArgumentNullException.ThrowIfNull(image); + ArgumentNullException.ThrowIfNull(descriptor); + + if (!IsSupportedDescriptor(descriptor)) + { + return null; + } + + var requestUri = BuildRegistryUri(image, $"blobs/{descriptor.Digest}"); + + async Task RequestFactory() + { + var request = new HttpRequestMessage(HttpMethod.Get, requestUri); + ApplyAuthentication(request); + request.Headers.Accept.ParseAdd(descriptor.MediaType ?? "application/octet-stream"); + return await Task.FromResult(request).ConfigureAwait(false); + } + + using var response = await SendAsync(RequestFactory, cancellationToken).ConfigureAwait(false); + if (!response.IsSuccessStatusCode) + { + if (response.StatusCode == HttpStatusCode.NotFound) + { + _logger.LogWarning("Registry returned 404 while downloading attestation {Digest} for {Subject}.", descriptor.Digest, subjectDigest); + return null; + } + + response.EnsureSuccessStatusCode(); + } + + var buffer = await response.Content.ReadAsByteArrayAsync(cancellationToken).ConfigureAwait(false); + var metadata = BuildMetadata(image, descriptor, "registry", requestUri.ToString(), subjectDigest); + return new OciAttestationDocument( + requestUri, + buffer, + metadata, + subjectDigest, + descriptor.Digest, + descriptor.ArtifactType, + "registry"); + } + + private static bool IsSupportedDescriptor(OciArtifactDescriptor descriptor) + { + if (descriptor is null) + { + return false; + } + + if (!string.IsNullOrWhiteSpace(descriptor.ArtifactType) && + descriptor.ArtifactType.Equals(OpenVexMediaType, StringComparison.OrdinalIgnoreCase)) + { + return true; + } + + if (!string.IsNullOrWhiteSpace(descriptor.MediaType) && + (descriptor.MediaType.Equals(DsseMediaType, StringComparison.OrdinalIgnoreCase) || + descriptor.MediaType.Equals(OpenVexMediaType, StringComparison.OrdinalIgnoreCase))) + { + return true; + } + + return false; + } + + private async Task SendAsync( + Func> requestFactory, + CancellationToken cancellationToken) + { + const int maxAttempts = 3; + TimeSpan delay = TimeSpan.FromSeconds(1); + Exception? lastError = null; + + for (var attempt = 1; attempt <= maxAttempts; attempt++) + { + cancellationToken.ThrowIfCancellationRequested(); + + using var request = await requestFactory().ConfigureAwait(false); + var client = _httpClientFactory.CreateClient(OciOpenVexAttestationConnectorOptions.HttpClientName); + + try + { + var response = await client.SendAsync(request, cancellationToken).ConfigureAwait(false); + + if (response.IsSuccessStatusCode) + { + return response; + } + + if (response.StatusCode == HttpStatusCode.Unauthorized) + { + if (_authorization.Mode == OciRegistryAuthMode.Anonymous && !_authorization.AllowAnonymousFallback) + { + var message = $"Registry request to {request.RequestUri} was unauthorized and anonymous fallback is disabled."; + response.Dispose(); + throw new HttpRequestException(message); + } + + lastError = new HttpRequestException($"Registry returned 401 Unauthorized for {request.RequestUri}."); + } + else if ((int)response.StatusCode >= 500 || response.StatusCode == (HttpStatusCode)429) + { + lastError = new HttpRequestException($"Registry returned status {(int)response.StatusCode} ({response.ReasonPhrase}) for {request.RequestUri}."); + } + else + { + response.EnsureSuccessStatusCode(); + } + + response.Dispose(); + } + catch (Exception ex) when (ex is HttpRequestException or TaskCanceledException) + { + lastError = ex; + } + + if (attempt < maxAttempts) + { + await Task.Delay(delay, cancellationToken).ConfigureAwait(false); + delay = TimeSpan.FromSeconds(Math.Min(delay.TotalSeconds * 2, 10)); + } + } + + throw new HttpRequestException("Failed to execute OCI registry request after multiple attempts.", lastError); + } + + private void ApplyAuthentication(HttpRequestMessage request) + { + switch (_authorization.Mode) + { + case OciRegistryAuthMode.Basic when + !string.IsNullOrEmpty(_authorization.Username) && + !string.IsNullOrEmpty(_authorization.Password): + var credentials = Convert.ToBase64String(Encoding.UTF8.GetBytes($"{_authorization.Username}:{_authorization.Password}")); + request.Headers.Authorization = new AuthenticationHeaderValue("Basic", credentials); + break; + case OciRegistryAuthMode.IdentityToken when !string.IsNullOrWhiteSpace(_authorization.IdentityToken): + request.Headers.Authorization = new AuthenticationHeaderValue("Bearer", _authorization.IdentityToken); + break; + case OciRegistryAuthMode.RefreshToken when !string.IsNullOrWhiteSpace(_authorization.RefreshToken): + request.Headers.Authorization = new AuthenticationHeaderValue("Bearer", _authorization.RefreshToken); + break; + default: + if (_authorization.Mode != OciRegistryAuthMode.Anonymous && !_authorization.AllowAnonymousFallback) + { + _logger.LogDebug("No authentication header applied for request to {Uri} (mode {Mode}).", request.RequestUri, _authorization.Mode); + } + break; + } + } + + private Uri BuildRegistryUri(OciImageReference image, string relativePath, string? query = null) + { + var scheme = image.Scheme; + if (!string.Equals(scheme, "https", StringComparison.OrdinalIgnoreCase) && !_options.AllowHttpRegistries) + { + throw new InvalidOperationException($"HTTP access to registry '{image.Registry}' is disabled. Set AllowHttpRegistries to true to enable."); + } + + var builder = new UriBuilder($"{scheme}://{image.Registry}") + { + Path = $"v2/{BuildRepositoryPath(image.Repository)}/{relativePath}" + }; + + if (!string.IsNullOrWhiteSpace(query)) + { + builder.Query = query; + } + + return builder.Uri; + } + + private static string BuildRepositoryPath(string repository) + { + var segments = repository.Split('/', StringSplitOptions.RemoveEmptyEntries); + return string.Join('/', segments.Select(Uri.EscapeDataString)); + } + + private static string EscapeReference(string reference) + { + return Uri.EscapeDataString(reference); + } + + private static ImmutableDictionary BuildMetadata( + OciImageReference image, + OciArtifactDescriptor descriptor, + string sourceKind, + string sourcePath, + string subjectDigest) + { + var builder = ImmutableDictionary.CreateBuilder(StringComparer.Ordinal); + builder["oci.image.registry"] = image.Registry; + builder["oci.image.repository"] = image.Repository; + builder["oci.image.reference"] = image.Canonical; + builder["oci.image.subjectDigest"] = subjectDigest; + builder["oci.attestation.sourceKind"] = sourceKind; + builder["oci.attestation.source"] = sourcePath; + builder["oci.attestation.artifactDigest"] = descriptor.Digest; + builder["oci.attestation.mediaType"] = descriptor.MediaType ?? string.Empty; + builder["oci.attestation.artifactType"] = descriptor.ArtifactType ?? string.Empty; + builder["oci.attestation.size"] = descriptor.Size.ToString(CultureInfo.InvariantCulture); + + if (descriptor.Annotations is not null) + { + foreach (var annotation in descriptor.Annotations) + { + builder[$"oci.attestation.annotations.{annotation.Key}"] = annotation.Value; + } + } + + return builder.ToImmutable(); + } +} diff --git a/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/OciOpenVexAttestationConnector.cs b/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/OciOpenVexAttestationConnector.cs new file mode 100644 index 00000000..d18cf9eb --- /dev/null +++ b/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/OciOpenVexAttestationConnector.cs @@ -0,0 +1,221 @@ +using System.Collections.Generic; +using System.Collections.Immutable; +using System.Runtime.CompilerServices; +using Microsoft.Extensions.Logging; +using StellaOps.Excititor.Connectors.Abstractions; +using StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest.Configuration; +using StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest.Discovery; +using StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest.Fetch; +using StellaOps.Excititor.Core; + +namespace StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest; + +public sealed class OciOpenVexAttestationConnector : VexConnectorBase +{ + private static readonly VexConnectorDescriptor StaticDescriptor = new( + id: "excititor:oci.openvex.attest", + kind: VexProviderKind.Attestation, + displayName: "OCI OpenVEX Attestations") + { + Tags = ImmutableArray.Create("oci", "openvex", "attestation", "cosign", "offline"), + }; + + private readonly OciAttestationDiscoveryService _discoveryService; + private readonly OciAttestationFetcher _fetcher; + private readonly IEnumerable> _validators; + + private OciOpenVexAttestationConnectorOptions? _options; + private OciAttestationDiscoveryResult? _discovery; + + public OciOpenVexAttestationConnector( + OciAttestationDiscoveryService discoveryService, + OciAttestationFetcher fetcher, + ILogger logger, + TimeProvider timeProvider, + IEnumerable>? validators = null) + : base(StaticDescriptor, logger, timeProvider) + { + _discoveryService = discoveryService ?? throw new ArgumentNullException(nameof(discoveryService)); + _fetcher = fetcher ?? throw new ArgumentNullException(nameof(fetcher)); + _validators = validators ?? Array.Empty>(); + } + + public override async ValueTask ValidateAsync(VexConnectorSettings settings, CancellationToken cancellationToken) + { + _options = VexConnectorOptionsBinder.Bind( + Descriptor, + settings, + validators: _validators); + + _discovery = await _discoveryService.LoadAsync(_options, cancellationToken).ConfigureAwait(false); + + LogConnectorEvent(LogLevel.Information, "validate", "Resolved OCI attestation targets.", new Dictionary + { + ["targets"] = _discovery.Targets.Length, + ["offlinePreferred"] = _discovery.PreferOffline, + ["allowNetworkFallback"] = _discovery.AllowNetworkFallback, + ["authMode"] = _discovery.RegistryAuthorization.Mode.ToString(), + ["cosignMode"] = _discovery.CosignAuthority.Mode.ToString(), + }); + } + + public override async IAsyncEnumerable FetchAsync(VexConnectorContext context, [EnumeratorCancellation] CancellationToken cancellationToken) + { + ArgumentNullException.ThrowIfNull(context); + + if (_options is null) + { + throw new InvalidOperationException("Connector must be validated before fetch operations."); + } + + if (_discovery is null) + { + _discovery = await _discoveryService.LoadAsync(_options, cancellationToken).ConfigureAwait(false); + } + + var documentCount = 0; + await foreach (var document in _fetcher.FetchAsync(_discovery, _options, cancellationToken)) + { + cancellationToken.ThrowIfCancellationRequested(); + + var verificationDocument = CreateRawDocument( + VexDocumentFormat.OciAttestation, + document.SourceUri, + document.Content, + document.Metadata); + + var signatureMetadata = await context.SignatureVerifier.VerifyAsync(verificationDocument, cancellationToken).ConfigureAwait(false); + if (signatureMetadata is not null) + { + LogConnectorEvent(LogLevel.Debug, "signature", "Signature metadata captured for attestation.", new Dictionary + { + ["subject"] = signatureMetadata.Subject, + ["type"] = signatureMetadata.Type, + }); + } + + var enrichedMetadata = BuildProvenanceMetadata(document, signatureMetadata); + var rawDocument = CreateRawDocument( + VexDocumentFormat.OciAttestation, + document.SourceUri, + document.Content, + enrichedMetadata); + + await context.RawSink.StoreAsync(rawDocument, cancellationToken).ConfigureAwait(false); + documentCount++; + yield return rawDocument; + } + + LogConnectorEvent(LogLevel.Information, "fetch", "OCI attestation fetch completed.", new Dictionary + { + ["documents"] = documentCount, + ["since"] = context.Since?.ToString("O"), + }); + } + + public override ValueTask NormalizeAsync(VexRawDocument document, CancellationToken cancellationToken) + => throw new NotSupportedException("Attestation documents rely on dedicated normalizers, to be wired in EXCITITOR-CONN-OCI-01-002."); + + public OciAttestationDiscoveryResult? GetCachedDiscovery() => _discovery; + + private ImmutableDictionary BuildProvenanceMetadata(OciAttestationDocument document, VexSignatureMetadata? signature) + { + var builder = document.Metadata.ToBuilder(); + + if (!string.IsNullOrWhiteSpace(document.SourceKind)) + { + builder["vex.provenance.sourceKind"] = document.SourceKind; + } + + if (!string.IsNullOrWhiteSpace(document.SubjectDigest)) + { + builder["vex.provenance.subjectDigest"] = document.SubjectDigest!; + } + + if (!string.IsNullOrWhiteSpace(document.ArtifactDigest)) + { + builder["vex.provenance.artifactDigest"] = document.ArtifactDigest!; + } + + if (!string.IsNullOrWhiteSpace(document.ArtifactType)) + { + builder["vex.provenance.artifactType"] = document.ArtifactType!; + } + + if (_discovery is not null) + { + builder["vex.provenance.registryAuthMode"] = _discovery.RegistryAuthorization.Mode.ToString(); + var registryAuthority = _discovery.RegistryAuthorization.RegistryAuthority; + if (string.IsNullOrWhiteSpace(registryAuthority)) + { + if (builder.TryGetValue("oci.image.registry", out var metadataRegistry) && !string.IsNullOrWhiteSpace(metadataRegistry)) + { + registryAuthority = metadataRegistry; + } + } + + if (!string.IsNullOrWhiteSpace(registryAuthority)) + { + builder["vex.provenance.registryAuthority"] = registryAuthority!; + } + + builder["vex.provenance.cosign.mode"] = _discovery.CosignAuthority.Mode.ToString(); + + if (_discovery.CosignAuthority.Keyless is not null) + { + var keyless = _discovery.CosignAuthority.Keyless; + builder["vex.provenance.cosign.issuer"] = keyless!.Issuer; + builder["vex.provenance.cosign.subject"] = keyless.Subject; + if (keyless.FulcioUrl is not null) + { + builder["vex.provenance.cosign.fulcioUrl"] = keyless.FulcioUrl!.ToString(); + } + + if (keyless.RekorUrl is not null) + { + builder["vex.provenance.cosign.rekorUrl"] = keyless.RekorUrl!.ToString(); + } + } + else if (_discovery.CosignAuthority.KeyPair is not null) + { + var keyPair = _discovery.CosignAuthority.KeyPair; + builder["vex.provenance.cosign.keyPair"] = "true"; + if (keyPair!.RekorUrl is not null) + { + builder["vex.provenance.cosign.rekorUrl"] = keyPair.RekorUrl!.ToString(); + } + } + } + + if (signature is not null) + { + builder["vex.signature.type"] = signature.Type; + if (!string.IsNullOrWhiteSpace(signature.Subject)) + { + builder["vex.signature.subject"] = signature.Subject!; + } + + if (!string.IsNullOrWhiteSpace(signature.Issuer)) + { + builder["vex.signature.issuer"] = signature.Issuer!; + } + + if (!string.IsNullOrWhiteSpace(signature.KeyId)) + { + builder["vex.signature.keyId"] = signature.KeyId!; + } + + if (signature.VerifiedAt is not null) + { + builder["vex.signature.verifiedAt"] = signature.VerifiedAt.Value.ToString("O"); + } + + if (!string.IsNullOrWhiteSpace(signature.TransparencyLogReference)) + { + builder["vex.signature.transparencyLogReference"] = signature.TransparencyLogReference!; + } + } + + return builder.ToImmutable(); + } +} diff --git a/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest.csproj b/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest.csproj new file mode 100644 index 00000000..947627bf --- /dev/null +++ b/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest.csproj @@ -0,0 +1,20 @@ + + + net10.0 + preview + enable + enable + true + NU1903 + + + + + + + + + + + + diff --git a/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/TASKS.md b/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/TASKS.md index 7ddece4c..28d39565 100644 --- a/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/TASKS.md +++ b/src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/TASKS.md @@ -2,6 +2,6 @@ If you are working on this file you need to read docs/ARCHITECTURE_EXCITITOR.md # TASKS | Task | Owner(s) | Depends on | Notes | |---|---|---|---| -|EXCITITOR-CONN-OCI-01-001 – OCI discovery & auth plumbing|Team Excititor Connectors – OCI|EXCITITOR-CONN-ABS-01-001|TODO – Resolve OCI references, configure cosign auth (keyless/keyed), and support offline attestation bundles.| -|EXCITITOR-CONN-OCI-01-002 – Attestation fetch & verify loop|Team Excititor Connectors – OCI|EXCITITOR-CONN-OCI-01-001, EXCITITOR-ATTEST-01-002|TODO – Download DSSE attestations, trigger verification, handle retries/backoff, and persist raw statements with metadata.| -|EXCITITOR-CONN-OCI-01-003 – Provenance metadata & policy hooks|Team Excititor Connectors – OCI|EXCITITOR-CONN-OCI-01-002, EXCITITOR-POLICY-01-001|TODO – Emit provenance hints (image, subject digest, issuer) and trust metadata for policy weighting/logging.| +|EXCITITOR-CONN-OCI-01-001 – OCI discovery & auth plumbing|Team Excititor Connectors – OCI|EXCITITOR-CONN-ABS-01-001|DONE (2025-10-18) – Added connector skeleton, options/validators, discovery caching, cosign/auth descriptors, offline bundle resolution, DI wiring, and regression tests.| +|EXCITITOR-CONN-OCI-01-002 – Attestation fetch & verify loop|Team Excititor Connectors – OCI|EXCITITOR-CONN-OCI-01-001, EXCITITOR-ATTEST-01-002|DONE (2025-10-18) – Added offline/registry fetch services, DSSE retrieval with retries, signature verification callout, and raw persistence coverage.| +|EXCITITOR-CONN-OCI-01-003 – Provenance metadata & policy hooks|Team Excititor Connectors – OCI|EXCITITOR-CONN-OCI-01-002, EXCITITOR-POLICY-01-001|DONE (2025-10-18) – Enriched attestation metadata with provenance hints, cosign expectations, registry auth context, and signature diagnostics for policy consumption.| diff --git a/src/StellaOps.Excititor.Connectors.StellaOpsMirror/TASKS.md b/src/StellaOps.Excititor.Connectors.StellaOpsMirror/TASKS.md new file mode 100644 index 00000000..f5c47b1b --- /dev/null +++ b/src/StellaOps.Excititor.Connectors.StellaOpsMirror/TASKS.md @@ -0,0 +1,7 @@ +# StellaOps Mirror VEX Connector Task Board (Sprint 7) + +| ID | Status | Owner(s) | Depends on | Description | Exit Criteria | +|----|--------|----------|------------|-------------|---------------| +| EXCITITOR-CONN-STELLA-07-001 | TODO | Excititor Connectors – Stella | EXCITITOR-EXPORT-01-007 | Implement mirror fetch client consuming `https://.stella-ops.org/excititor/exports/index.json`, validating signatures/digests, storing raw consensus bundles with provenance. | Fetch job downloads mirror manifest, verifies DSSE/signature, stores raw documents + provenance; unit tests cover happy path and tampered manifest failure. | +| EXCITITOR-CONN-STELLA-07-002 | TODO | Excititor Connectors – Stella | EXCITITOR-CONN-STELLA-07-001 | Normalize mirror bundles into VexClaim sets referencing original provider metadata and mirror provenance. | Normalizer emits VexClaims with mirror provenance + policy metadata, fixtures assert deterministic output parity vs local exports. | +| EXCITITOR-CONN-STELLA-07-003 | TODO | Excititor Connectors – Stella | EXCITITOR-CONN-STELLA-07-002 | Implement incremental cursor handling per-export digest, support resume, and document configuration for downstream Excititor mirrors. | Connector resumes from last export digest, handles delta/export rotation, docs show configuration; integration test covers resume + new export ingest. | diff --git a/src/StellaOps.Excititor.Export/TASKS.md b/src/StellaOps.Excititor.Export/TASKS.md index 570b34f3..acb2dc03 100644 --- a/src/StellaOps.Excititor.Export/TASKS.md +++ b/src/StellaOps.Excititor.Export/TASKS.md @@ -7,3 +7,5 @@ If you are working on this file you need to read docs/ARCHITECTURE_EXCITITOR.md |EXCITITOR-EXPORT-01-003 – Artifact store adapters|Team Excititor Export|EXCITITOR-EXPORT-01-001|**DONE (2025-10-16)** – Implemented multi-store pipeline with filesystem, S3-compatible, and offline bundle adapters (hash verification + manifest/zip output) plus unit coverage and DI hooks.| |EXCITITOR-EXPORT-01-004 – Attestation handoff integration|Team Excititor Export|EXCITITOR-EXPORT-01-001, EXCITITOR-ATTEST-01-001|**DONE (2025-10-17)** – Export engine now invokes attestation client, logs diagnostics, and persists Rekor/envelope metadata on manifests; regression coverage added in `ExportEngineTests.ExportAsync_AttachesAttestationMetadata`.| |EXCITITOR-EXPORT-01-005 – Score & resolve envelope surfaces|Team Excititor Export|EXCITITOR-EXPORT-01-004, EXCITITOR-CORE-02-001|TODO – Emit consensus+score envelopes in export manifests, include policy/scoring digests, and update offline bundle/ORAS layouts to carry signed VEX responses.| +|EXCITITOR-EXPORT-01-006 – Quiet provenance packaging|Team Excititor Export|EXCITITOR-EXPORT-01-005, POLICY-CORE-09-005|TODO – Attach `quietedBy` statement IDs, signers, and justification codes to exports/offline bundles, mirror metadata into attested manifest, and add regression fixtures.| +|EXCITITOR-EXPORT-01-007 – Mirror bundle + domain manifest|Team Excititor Export|EXCITITOR-EXPORT-01-006|TODO – Create per-domain mirror bundles with consensus/score artifacts, publish signed index for downstream Excititor sync, and ensure deterministic digests + fixtures.| diff --git a/src/StellaOps.Excititor.WebService/TASKS.md b/src/StellaOps.Excititor.WebService/TASKS.md index 2b5d3486..86ed4955 100644 --- a/src/StellaOps.Excititor.WebService/TASKS.md +++ b/src/StellaOps.Excititor.WebService/TASKS.md @@ -6,3 +6,4 @@ If you are working on this file you need to read docs/ARCHITECTURE_EXCITITOR.md |EXCITITOR-WEB-01-002 – Ingest & reconcile endpoints|Team Excititor WebService|EXCITITOR-WEB-01-001|TODO – Implement `/excititor/init`, `/excititor/ingest/run`, `/excititor/ingest/resume`, `/excititor/reconcile` with token scope enforcement and structured run telemetry.| |EXCITITOR-WEB-01-003 – Export & verify endpoints|Team Excititor WebService|EXCITITOR-WEB-01-001, EXCITITOR-EXPORT-01-001, EXCITITOR-ATTEST-01-001|TODO – Add `/excititor/export`, `/excititor/export/{id}`, `/excititor/export/{id}/download`, `/excititor/verify`, returning artifact + attestation metadata with cache awareness.| |EXCITITOR-WEB-01-004 – Resolve API & signed responses|Team Excititor WebService|EXCITITOR-WEB-01-001, EXCITITOR-ATTEST-01-002|TODO – Deliver `/excititor/resolve` (subject/context), return consensus + score envelopes, attach cosign/Rekor metadata, and document auth + rate guardrails.| +|EXCITITOR-WEB-01-005 – Mirror distribution endpoints|Team Excititor WebService|EXCITITOR-EXPORT-01-007, DEVOPS-MIRROR-08-001|TODO – Provide domain-scoped mirror index/download APIs for consensus exports, enforce quota/auth, and document sync workflow for downstream Excititor deployments.| diff --git a/src/StellaOps.Notify.Connectors.Email/AGENTS.md b/src/StellaOps.Notify.Connectors.Email/AGENTS.md new file mode 100644 index 00000000..cddac426 --- /dev/null +++ b/src/StellaOps.Notify.Connectors.Email/AGENTS.md @@ -0,0 +1,4 @@ +# StellaOps.Notify.Connectors.Email — Agent Charter + +## Mission +Implement SMTP connector plug-in per `docs/ARCHITECTURE_NOTIFY.md`. diff --git a/src/StellaOps.Notify.Connectors.Email/StellaOps.Notify.Connectors.Email.csproj b/src/StellaOps.Notify.Connectors.Email/StellaOps.Notify.Connectors.Email.csproj new file mode 100644 index 00000000..6c3a8871 --- /dev/null +++ b/src/StellaOps.Notify.Connectors.Email/StellaOps.Notify.Connectors.Email.csproj @@ -0,0 +1,7 @@ + + + net10.0 + enable + enable + + diff --git a/src/StellaOps.Notify.Connectors.Email/TASKS.md b/src/StellaOps.Notify.Connectors.Email/TASKS.md new file mode 100644 index 00000000..012d4f94 --- /dev/null +++ b/src/StellaOps.Notify.Connectors.Email/TASKS.md @@ -0,0 +1,7 @@ +# Notify Email Connector Task Board (Sprint 15) + +| ID | Status | Owner(s) | Depends on | Description | Exit Criteria | +|----|--------|----------|------------|-------------|---------------| +| NOTIFY-CONN-EMAIL-15-701 | TODO | Notify Connectors Guild | NOTIFY-ENGINE-15-303 | Implement SMTP connector with STARTTLS/implicit TLS support, HTML+text rendering, attachment policy enforcement. | Integration tests with SMTP stub pass; TLS enforced; attachments blocked per policy. | +| NOTIFY-CONN-EMAIL-15-702 | TODO | Notify Connectors Guild | NOTIFY-CONN-EMAIL-15-701 | Add DKIM signing optional support and health/test-send flows. | DKIM optional config verified; test-send passes; secrets handled securely. | +| NOTIFY-CONN-EMAIL-15-703 | TODO | Notify Connectors Guild | NOTIFY-CONN-EMAIL-15-702 | Package Email connector as restart-time plug-in (manifest + host registration). | Plugin manifest added; host loads connector from `plugins/notify/email/`; restart validation passes. | diff --git a/src/StellaOps.Notify.Connectors.Slack/AGENTS.md b/src/StellaOps.Notify.Connectors.Slack/AGENTS.md new file mode 100644 index 00000000..f5811215 --- /dev/null +++ b/src/StellaOps.Notify.Connectors.Slack/AGENTS.md @@ -0,0 +1,4 @@ +# StellaOps.Notify.Connectors.Slack — Agent Charter + +## Mission +Deliver Slack connector plug-in per `docs/ARCHITECTURE_NOTIFY.md`. diff --git a/src/StellaOps.Notify.Connectors.Slack/StellaOps.Notify.Connectors.Slack.csproj b/src/StellaOps.Notify.Connectors.Slack/StellaOps.Notify.Connectors.Slack.csproj new file mode 100644 index 00000000..6c3a8871 --- /dev/null +++ b/src/StellaOps.Notify.Connectors.Slack/StellaOps.Notify.Connectors.Slack.csproj @@ -0,0 +1,7 @@ + + + net10.0 + enable + enable + + diff --git a/src/StellaOps.Notify.Connectors.Slack/TASKS.md b/src/StellaOps.Notify.Connectors.Slack/TASKS.md new file mode 100644 index 00000000..08753de5 --- /dev/null +++ b/src/StellaOps.Notify.Connectors.Slack/TASKS.md @@ -0,0 +1,7 @@ +# Notify Slack Connector Task Board (Sprint 15) + +| ID | Status | Owner(s) | Depends on | Description | Exit Criteria | +|----|--------|----------|------------|-------------|---------------| +| NOTIFY-CONN-SLACK-15-501 | TODO | Notify Connectors Guild | NOTIFY-ENGINE-15-303 | Implement Slack connector with bot token auth, message rendering (blocks), rate limit handling, retries/backoff. | Integration tests stub Slack API; retries/jitter validated; 429 handling documented. | +| NOTIFY-CONN-SLACK-15-502 | TODO | Notify Connectors Guild | NOTIFY-CONN-SLACK-15-501 | Health check & test-send support with minimal scopes and redacted tokens. | `/channels/{id}/test` hitting Slack stub passes; secrets never logged; health endpoint returns diagnostics. | +| NOTIFY-CONN-SLACK-15-503 | TODO | Notify Connectors Guild | NOTIFY-CONN-SLACK-15-502 | Package Slack connector as restart-time plug-in (manifest + host registration). | Plugin manifest added; host loads connector from `plugins/notify/slack/`; restart validation passes. | diff --git a/src/StellaOps.Notify.Connectors.Teams/AGENTS.md b/src/StellaOps.Notify.Connectors.Teams/AGENTS.md new file mode 100644 index 00000000..9e9b4dfb --- /dev/null +++ b/src/StellaOps.Notify.Connectors.Teams/AGENTS.md @@ -0,0 +1,4 @@ +# StellaOps.Notify.Connectors.Teams — Agent Charter + +## Mission +Implement Microsoft Teams connector plug-in per `docs/ARCHITECTURE_NOTIFY.md`. diff --git a/src/StellaOps.Notify.Connectors.Teams/StellaOps.Notify.Connectors.Teams.csproj b/src/StellaOps.Notify.Connectors.Teams/StellaOps.Notify.Connectors.Teams.csproj new file mode 100644 index 00000000..6c3a8871 --- /dev/null +++ b/src/StellaOps.Notify.Connectors.Teams/StellaOps.Notify.Connectors.Teams.csproj @@ -0,0 +1,7 @@ + + + net10.0 + enable + enable + + diff --git a/src/StellaOps.Notify.Connectors.Teams/TASKS.md b/src/StellaOps.Notify.Connectors.Teams/TASKS.md new file mode 100644 index 00000000..5acb4d8d --- /dev/null +++ b/src/StellaOps.Notify.Connectors.Teams/TASKS.md @@ -0,0 +1,7 @@ +# Notify Teams Connector Task Board (Sprint 15) + +| ID | Status | Owner(s) | Depends on | Description | Exit Criteria | +|----|--------|----------|------------|-------------|---------------| +| NOTIFY-CONN-TEAMS-15-601 | TODO | Notify Connectors Guild | NOTIFY-ENGINE-15-303 | Implement Teams connector using Adaptive Cards 1.5, handle webhook auth, size limits, retries. | Adaptive card payloads validated; 413/429 handling implemented; integration tests cover success/fail. | +| NOTIFY-CONN-TEAMS-15-602 | TODO | Notify Connectors Guild | NOTIFY-CONN-TEAMS-15-601 | Provide health/test-send support with fallback text for legacy clients. | Test-send returns card preview; fallback text logged; docs updated. | +| NOTIFY-CONN-TEAMS-15-603 | TODO | Notify Connectors Guild | NOTIFY-CONN-TEAMS-15-602 | Package Teams connector as restart-time plug-in (manifest + host registration). | Plugin manifest added; host loads connector from `plugins/notify/teams/`; restart validation passes. | diff --git a/src/StellaOps.Notify.Connectors.Webhook/AGENTS.md b/src/StellaOps.Notify.Connectors.Webhook/AGENTS.md new file mode 100644 index 00000000..f9040433 --- /dev/null +++ b/src/StellaOps.Notify.Connectors.Webhook/AGENTS.md @@ -0,0 +1,4 @@ +# StellaOps.Notify.Connectors.Webhook — Agent Charter + +## Mission +Implement generic webhook connector plug-in per `docs/ARCHITECTURE_NOTIFY.md`. diff --git a/src/StellaOps.Notify.Connectors.Webhook/StellaOps.Notify.Connectors.Webhook.csproj b/src/StellaOps.Notify.Connectors.Webhook/StellaOps.Notify.Connectors.Webhook.csproj new file mode 100644 index 00000000..6c3a8871 --- /dev/null +++ b/src/StellaOps.Notify.Connectors.Webhook/StellaOps.Notify.Connectors.Webhook.csproj @@ -0,0 +1,7 @@ + + + net10.0 + enable + enable + + diff --git a/src/StellaOps.Notify.Connectors.Webhook/TASKS.md b/src/StellaOps.Notify.Connectors.Webhook/TASKS.md new file mode 100644 index 00000000..a0cc45b5 --- /dev/null +++ b/src/StellaOps.Notify.Connectors.Webhook/TASKS.md @@ -0,0 +1,7 @@ +# Notify Webhook Connector Task Board (Sprint 15) + +| ID | Status | Owner(s) | Depends on | Description | Exit Criteria | +|----|--------|----------|------------|-------------|---------------| +| NOTIFY-CONN-WEBHOOK-15-801 | TODO | Notify Connectors Guild | NOTIFY-ENGINE-15-303 | Implement webhook connector: JSON payload, signature (HMAC/Ed25519), retries/backoff, status code handling. | Integration tests with webhook stub validate signatures, retries, error handling; payload schema documented. | +| NOTIFY-CONN-WEBHOOK-15-802 | TODO | Notify Connectors Guild | NOTIFY-CONN-WEBHOOK-15-801 | Health/test-send support with signature validation hints and secret management. | Test-send returns success with sample payload; docs include verification guide; secrets never logged. | +| NOTIFY-CONN-WEBHOOK-15-803 | TODO | Notify Connectors Guild | NOTIFY-CONN-WEBHOOK-15-802 | Package Webhook connector as restart-time plug-in (manifest + host registration). | Plugin manifest added; host loads connector from `plugins/notify/webhook/`; restart validation passes. | diff --git a/src/StellaOps.Notify.Engine/AGENTS.md b/src/StellaOps.Notify.Engine/AGENTS.md new file mode 100644 index 00000000..dcbf9b1a --- /dev/null +++ b/src/StellaOps.Notify.Engine/AGENTS.md @@ -0,0 +1,4 @@ +# StellaOps.Notify.Engine — Agent Charter + +## Mission +Deliver rule evaluation, digest, and rendering logic per `docs/ARCHITECTURE_NOTIFY.md`. diff --git a/src/StellaOps.Notify.Engine/StellaOps.Notify.Engine.csproj b/src/StellaOps.Notify.Engine/StellaOps.Notify.Engine.csproj new file mode 100644 index 00000000..6c3a8871 --- /dev/null +++ b/src/StellaOps.Notify.Engine/StellaOps.Notify.Engine.csproj @@ -0,0 +1,7 @@ + + + net10.0 + enable + enable + + diff --git a/src/StellaOps.Notify.Engine/TASKS.md b/src/StellaOps.Notify.Engine/TASKS.md new file mode 100644 index 00000000..1b361f37 --- /dev/null +++ b/src/StellaOps.Notify.Engine/TASKS.md @@ -0,0 +1,8 @@ +# Notify Engine Task Board (Sprint 15) + +| ID | Status | Owner(s) | Depends on | Description | Exit Criteria | +|----|--------|----------|------------|-------------|---------------| +| NOTIFY-ENGINE-15-301 | TODO | Notify Engine Guild | NOTIFY-MODELS-15-101 | Rules evaluation core: tenant/kind filters, severity/delta gates, VEX gating, throttling, idempotency key generation. | Unit tests cover rule permutations; idempotency keys deterministic; documentation updated. | +| NOTIFY-ENGINE-15-302 | TODO | Notify Engine Guild | NOTIFY-ENGINE-15-301 | Action planner + digest coalescer with window management and dedupe per architecture §4. | Digest windows tested; throttles and digests recorded; metrics counters exposed. | +| NOTIFY-ENGINE-15-303 | TODO | Notify Engine Guild | NOTIFY-ENGINE-15-302 | Template rendering engine (Slack, Teams, Email, Webhook) with helpers and i18n support. | Rendering fixtures validated; helpers documented; deterministic output proven via golden tests. | +| NOTIFY-ENGINE-15-304 | TODO | Notify Engine Guild | NOTIFY-ENGINE-15-303 | Test-send sandbox + preview utilities for WebService. | Preview/test functions validated; sample outputs returned; no state persisted. | diff --git a/src/StellaOps.Notify.Models/AGENTS.md b/src/StellaOps.Notify.Models/AGENTS.md new file mode 100644 index 00000000..66d688c6 --- /dev/null +++ b/src/StellaOps.Notify.Models/AGENTS.md @@ -0,0 +1,4 @@ +# StellaOps.Notify.Models — Agent Charter + +## Mission +Define Notify DTOs and contracts per `docs/ARCHITECTURE_NOTIFY.md`. diff --git a/src/StellaOps.Notify.Models/StellaOps.Notify.Models.csproj b/src/StellaOps.Notify.Models/StellaOps.Notify.Models.csproj new file mode 100644 index 00000000..6c3a8871 --- /dev/null +++ b/src/StellaOps.Notify.Models/StellaOps.Notify.Models.csproj @@ -0,0 +1,7 @@ + + + net10.0 + enable + enable + + diff --git a/src/StellaOps.Notify.Models/TASKS.md b/src/StellaOps.Notify.Models/TASKS.md new file mode 100644 index 00000000..885d48a6 --- /dev/null +++ b/src/StellaOps.Notify.Models/TASKS.md @@ -0,0 +1,7 @@ +# Notify Models Task Board (Sprint 15) + +| ID | Status | Owner(s) | Depends on | Description | Exit Criteria | +|----|--------|----------|------------|-------------|---------------| +| NOTIFY-MODELS-15-101 | TODO | Notify Models Guild | — | Define core DTOs (Rule, Channel, Template, Event envelope, Delivery) with validation helpers and canonical JSON serialization. | DTOs merged with tests; documented; serialization deterministic. | +| NOTIFY-MODELS-15-102 | TODO | Notify Models Guild | NOTIFY-MODELS-15-101 | Publish schema docs + sample payloads for channels, rules, events (used by UI + connectors). | Markdown/JSON schema generated; linked in docs; integration tests reference samples. | +| NOTIFY-MODELS-15-103 | TODO | Notify Models Guild | NOTIFY-MODELS-15-101 | Provide versioning and migration helpers (e.g., rule evolution, template revisions). | Migration helpers implemented; tests cover upgrade/downgrade; guidance captured in docs. | diff --git a/src/StellaOps.Notify.Queue/AGENTS.md b/src/StellaOps.Notify.Queue/AGENTS.md new file mode 100644 index 00000000..36c84448 --- /dev/null +++ b/src/StellaOps.Notify.Queue/AGENTS.md @@ -0,0 +1,4 @@ +# StellaOps.Notify.Queue — Agent Charter + +## Mission +Provide event & delivery queues for Notify per `docs/ARCHITECTURE_NOTIFY.md`. diff --git a/src/StellaOps.Notify.Queue/StellaOps.Notify.Queue.csproj b/src/StellaOps.Notify.Queue/StellaOps.Notify.Queue.csproj new file mode 100644 index 00000000..6c3a8871 --- /dev/null +++ b/src/StellaOps.Notify.Queue/StellaOps.Notify.Queue.csproj @@ -0,0 +1,7 @@ + + + net10.0 + enable + enable + + diff --git a/src/StellaOps.Notify.Queue/TASKS.md b/src/StellaOps.Notify.Queue/TASKS.md new file mode 100644 index 00000000..fbd971f9 --- /dev/null +++ b/src/StellaOps.Notify.Queue/TASKS.md @@ -0,0 +1,7 @@ +# Notify Queue Task Board (Sprint 15) + +| ID | Status | Owner(s) | Depends on | Description | Exit Criteria | +|----|--------|----------|------------|-------------|---------------| +| NOTIFY-QUEUE-15-401 | TODO | Notify Queue Guild | NOTIFY-MODELS-15-101 | Build queue abstraction + Redis Streams adapter with ack/claim APIs, idempotency tokens, serialization contracts. | Adapter integration tests cover enqueue/dequeue/ack; ordering preserved; idempotency tokens supported. | +| NOTIFY-QUEUE-15-402 | TODO | Notify Queue Guild | NOTIFY-QUEUE-15-401 | Add NATS JetStream adapter with configuration binding, health probes, failover. | Health endpoints verified; failover documented; integration tests exercise both adapters. | +| NOTIFY-QUEUE-15-403 | TODO | Notify Queue Guild | NOTIFY-QUEUE-15-401 | Delivery queue for channel actions with retry schedules, poison queues, and metrics instrumentation. | Delivery queue integration tests cover retries/dead-letter; metrics/logging emitted per spec. | diff --git a/src/StellaOps.Notify.Storage.Mongo/AGENTS.md b/src/StellaOps.Notify.Storage.Mongo/AGENTS.md new file mode 100644 index 00000000..734ffa53 --- /dev/null +++ b/src/StellaOps.Notify.Storage.Mongo/AGENTS.md @@ -0,0 +1,4 @@ +# StellaOps.Notify.Storage.Mongo — Agent Charter + +## Mission +Implement Mongo persistence (rules, channels, deliveries, digests, locks, audit) per `docs/ARCHITECTURE_NOTIFY.md`. diff --git a/src/StellaOps.Notify.Storage.Mongo/StellaOps.Notify.Storage.Mongo.csproj b/src/StellaOps.Notify.Storage.Mongo/StellaOps.Notify.Storage.Mongo.csproj new file mode 100644 index 00000000..6c3a8871 --- /dev/null +++ b/src/StellaOps.Notify.Storage.Mongo/StellaOps.Notify.Storage.Mongo.csproj @@ -0,0 +1,7 @@ + + + net10.0 + enable + enable + + diff --git a/src/StellaOps.Notify.Storage.Mongo/TASKS.md b/src/StellaOps.Notify.Storage.Mongo/TASKS.md new file mode 100644 index 00000000..a65bbdc6 --- /dev/null +++ b/src/StellaOps.Notify.Storage.Mongo/TASKS.md @@ -0,0 +1,7 @@ +# Notify Storage Task Board (Sprint 15) + +| ID | Status | Owner(s) | Depends on | Description | Exit Criteria | +|----|--------|----------|------------|-------------|---------------| +| NOTIFY-STORAGE-15-201 | TODO | Notify Storage Guild | NOTIFY-MODELS-15-101 | Create Mongo schemas/collections (rules, channels, deliveries, digests, locks, audit) with indexes per architecture §7. | Migration scripts authored; indexes tested; integration tests cover CRUD/read paths. | +| NOTIFY-STORAGE-15-202 | TODO | Notify Storage Guild | NOTIFY-STORAGE-15-201 | Implement repositories/services with tenant scoping, soft deletes, TTL, causal consistency (majority) options. | Repositories unit-tested; soft delete + TTL validated; majority read/write configuration documented. | +| NOTIFY-STORAGE-15-203 | TODO | Notify Storage Guild | NOTIFY-STORAGE-15-201 | Delivery history retention + query APIs (paging, filters). | History queries return expected data; paging verified; docs updated. | diff --git a/src/StellaOps.Notify.WebService/AGENTS.md b/src/StellaOps.Notify.WebService/AGENTS.md new file mode 100644 index 00000000..bbdf5c00 --- /dev/null +++ b/src/StellaOps.Notify.WebService/AGENTS.md @@ -0,0 +1,4 @@ +# StellaOps.Notify.WebService — Agent Charter + +## Mission +Implement Notify control plane per `docs/ARCHITECTURE_NOTIFY.md`. diff --git a/src/StellaOps.Notify.WebService/StellaOps.Notify.WebService.csproj b/src/StellaOps.Notify.WebService/StellaOps.Notify.WebService.csproj new file mode 100644 index 00000000..4c69387c --- /dev/null +++ b/src/StellaOps.Notify.WebService/StellaOps.Notify.WebService.csproj @@ -0,0 +1,7 @@ + + + net10.0 + enable + enable + + diff --git a/src/StellaOps.Notify.WebService/TASKS.md b/src/StellaOps.Notify.WebService/TASKS.md new file mode 100644 index 00000000..9c9a189f --- /dev/null +++ b/src/StellaOps.Notify.WebService/TASKS.md @@ -0,0 +1,8 @@ +# Notify WebService Task Board (Sprint 15) + +| ID | Status | Owner(s) | Depends on | Description | Exit Criteria | +|----|--------|----------|------------|-------------|---------------| +| NOTIFY-WEB-15-101 | TODO | Notify WebService Guild | NOTIFY-MODELS-15-101 | Bootstrap minimal API host with Authority auth, health endpoints, and plug-in discovery per architecture. | Service starts with config validation, `/healthz`/`/readyz` pass, plug-ins loaded at restart. | +| NOTIFY-WEB-15-102 | TODO | Notify WebService Guild | NOTIFY-WEB-15-101 | Rules/channel/template CRUD endpoints with tenant scoping, validation, audit logging. | CRUD endpoints tested; invalid inputs rejected; audit entries persisted. | +| NOTIFY-WEB-15-103 | TODO | Notify WebService Guild | NOTIFY-WEB-15-102 | Delivery history + test-send endpoints with rate limits. | `/deliveries` and `/channels/{id}/test` tested; rate limits enforced. | +| NOTIFY-WEB-15-104 | TODO | Notify WebService Guild | NOTIFY-STORAGE-15-201, NOTIFY-QUEUE-15-401 | Configuration binding for Mongo/queue/secrets; startup diagnostics. | Misconfiguration fails fast; diagnostics logged; integration tests cover env overrides. | diff --git a/src/StellaOps.Notify.Worker/AGENTS.md b/src/StellaOps.Notify.Worker/AGENTS.md new file mode 100644 index 00000000..aba55e4c --- /dev/null +++ b/src/StellaOps.Notify.Worker/AGENTS.md @@ -0,0 +1,4 @@ +# StellaOps.Notify.Worker — Agent Charter + +## Mission +Consume events, evaluate rules, and dispatch deliveries per `docs/ARCHITECTURE_NOTIFY.md`. diff --git a/src/StellaOps.Notify.Worker/StellaOps.Notify.Worker.csproj b/src/StellaOps.Notify.Worker/StellaOps.Notify.Worker.csproj new file mode 100644 index 00000000..1094c46d --- /dev/null +++ b/src/StellaOps.Notify.Worker/StellaOps.Notify.Worker.csproj @@ -0,0 +1,8 @@ + + + net10.0 + enable + enable + Exe + + diff --git a/src/StellaOps.Notify.Worker/TASKS.md b/src/StellaOps.Notify.Worker/TASKS.md new file mode 100644 index 00000000..8a6fc64d --- /dev/null +++ b/src/StellaOps.Notify.Worker/TASKS.md @@ -0,0 +1,8 @@ +# Notify Worker Task Board (Sprint 15) + +| ID | Status | Owner(s) | Depends on | Description | Exit Criteria | +|----|--------|----------|------------|-------------|---------------| +| NOTIFY-WORKER-15-201 | TODO | Notify Worker Guild | NOTIFY-QUEUE-15-401 | Implement bus subscription + leasing loop with correlation IDs, backoff, dead-letter handling (§1–§5). | Worker consumes events from queue, ack/retry behaviour proven in integration tests; logs include correlation IDs. | +| NOTIFY-WORKER-15-202 | TODO | Notify Worker Guild | NOTIFY-ENGINE-15-301 | Wire rules evaluation pipeline (tenant scoping, filters, throttles, digests, idempotency) with deterministic decisions. | Evaluation unit tests cover rule combinations; throttles/digests produce expected suppression; idempotency keys validated. | +| NOTIFY-WORKER-15-203 | TODO | Notify Worker Guild | NOTIFY-ENGINE-15-302 | Channel dispatch orchestration: invoke connectors, manage retries/jitter, record delivery outcomes. | Connector mocks show retries/backoff; delivery results stored; metrics incremented per outcome. | +| NOTIFY-WORKER-15-204 | TODO | Notify Worker Guild | NOTIFY-WORKER-15-203 | Metrics/telemetry: `notify.sent_total`, `notify.dropped_total`, latency histograms, tracing integration. | Metrics emitted per spec; OTLP spans annotated; dashboards documented. | diff --git a/src/StellaOps.Policy/AGENTS.md b/src/StellaOps.Policy/AGENTS.md new file mode 100644 index 00000000..05098272 --- /dev/null +++ b/src/StellaOps.Policy/AGENTS.md @@ -0,0 +1,12 @@ +# StellaOps.Policy — Agent Charter + +## Mission +Deliver the policy engine outlined in `docs/ARCHITECTURE_SCANNER.md` and related prose: +- Define YAML schema (ignore rules, VEX inclusion/exclusion, vendor precedence, license gates). +- Provide policy snapshot storage with revision digests and diagnostics. +- Offer preview APIs to compare policy impacts on existing reports. + +## Expectations +- Coordinate with Scanner.WebService, Feedser, Vexer, UI, Notify. +- Maintain deterministic serialization and unit tests for precedence rules. +- Update `TASKS.md` and broadcast contract changes. diff --git a/src/StellaOps.Policy/StellaOps.Policy.csproj b/src/StellaOps.Policy/StellaOps.Policy.csproj new file mode 100644 index 00000000..6c3a8871 --- /dev/null +++ b/src/StellaOps.Policy/StellaOps.Policy.csproj @@ -0,0 +1,7 @@ + + + net10.0 + enable + enable + + diff --git a/src/StellaOps.Policy/TASKS.md b/src/StellaOps.Policy/TASKS.md new file mode 100644 index 00000000..22f396a8 --- /dev/null +++ b/src/StellaOps.Policy/TASKS.md @@ -0,0 +1,13 @@ +# Policy Engine Task Board (Sprint 9) + +| ID | Status | Owner(s) | Depends on | Description | Exit Criteria | +|----|--------|----------|------------|-------------|---------------| +| POLICY-CORE-09-001 | TODO | Policy Guild | SCANNER-WEB-09-101 | Define YAML schema/binder, diagnostics, CLI validation for policy files. | Schema doc published; binder loads sample policy; validation errors actionable. | +| POLICY-CORE-09-002 | TODO | Policy Guild | POLICY-CORE-09-001 | Implement policy snapshot store + revision digests + audit logging. | Snapshots persisted with digest; tests compare revisions; audit entries created. | +| POLICY-CORE-09-003 | TODO | Policy Guild | POLICY-CORE-09-002 | `/policy/preview` API (image digest → projected verdict delta). | Preview returns diff JSON; integration tests with mocked report; docs updated. | +| POLICY-CORE-09-004 | TODO | Policy Guild | POLICY-CORE-09-001 | Versioned scoring config with schema validation, trust table, and golden fixtures. | Scoring config documented; fixtures stored; validation CLI passes. | +| POLICY-CORE-09-005 | TODO | Policy Guild | POLICY-CORE-09-004 | Scoring/quiet engine – compute score, enforce VEX-only quiet rules, emit inputs and provenance. | Engine unit tests cover severity weighting; outputs include provenance data. | +| POLICY-CORE-09-006 | TODO | Policy Guild | POLICY-CORE-09-005 | Unknown state & confidence decay – deterministic bands surfaced in policy outputs. | Confidence decay tests pass; docs updated; preview endpoint displays banding. | +| POLICY-CORE-09-004 | TODO | Policy Guild | POLICY-CORE-09-001 | Versioned scoring config (weights, trust table, reachability buckets) with schema validation, binder, and golden fixtures. | Config serialized with semantic version, binder loads defaults, fixtures assert deterministic hash. | +| POLICY-CORE-09-005 | TODO | Policy Guild | POLICY-CORE-09-004, POLICY-CORE-09-002 | Implement scoring/quiet engine: compute score from config, enforce VEX-only quiet rules, emit inputs + `quietedBy` metadata in policy verdicts. | `/reports` policy result includes score, inputs, configVersion, quiet provenance; unit/integration tests prove reproducibility. | +| POLICY-CORE-09-006 | TODO | Policy Guild | POLICY-CORE-09-005, FEEDCORE-ENGINE-07-003 | Track unknown states with deterministic confidence bands that decay over time; expose state in policy outputs and docs. | Unknown flags + confidence band persisted, decay job deterministic, preview/report APIs show state with tests covering decay math. | diff --git a/src/StellaOps.Scanner.Emit/TASKS.md b/src/StellaOps.Scanner.Emit/TASKS.md new file mode 100644 index 00000000..c79fefef --- /dev/null +++ b/src/StellaOps.Scanner.Emit/TASKS.md @@ -0,0 +1,12 @@ +# Scanner Emit Task Board (Sprint 10) + +| ID | Status | Owner(s) | Depends on | Description | Exit Criteria | +|----|--------|----------|------------|-------------|---------------| +| SCANNER-EMIT-10-601 | TODO | Emit Guild | SCANNER-CACHE-10-101 | Compose inventory SBOM (CycloneDX JSON/Protobuf) from layer fragments with deterministic ordering. | Inventory SBOM validated against schema; fixtures confirm deterministic output. | +| SCANNER-EMIT-10-602 | TODO | Emit Guild | SCANNER-EMIT-10-601 | Compose usage SBOM leveraging EntryTrace to flag actual usage; ensure separate view toggles. | Usage SBOM tests confirm correct subset; API contract documented. | +| SCANNER-EMIT-10-603 | TODO | Emit Guild | SCANNER-EMIT-10-601 | Generate BOM index sidecar (purl table + roaring bitmap + usedByEntrypoint flag). | Index format validated; query helpers proven; stored artifacts hashed deterministically. | +| SCANNER-EMIT-10-604 | TODO | Emit Guild | SCANNER-EMIT-10-602 | Package artifacts for export + attestation (naming, compression, manifests). | Export pipeline produces deterministic file paths/hashes; integration test with storage passes. | +| SCANNER-EMIT-10-605 | TODO | Emit Guild | SCANNER-EMIT-10-603 | Emit BOM-Index sidecar schema/fixtures (`bom-index@1`) and note CRITICAL PATH for Scheduler. | Schema + fixtures in docs/artifacts/bom-index; tests `BOMIndexGoldenIsStable` green. | +| SCANNER-EMIT-10-606 | TODO | Emit Guild | SCANNER-EMIT-10-605 | Integrate EntryTrace usage flags into BOM-Index; document semantics. | Usage bits present in sidecar; integration tests with EntryTrace fixtures pass. | +| SCANNER-EMIT-17-701 | TODO | Emit Guild, Native Analyzer Guild | SCANNER-EMIT-10-602 | Record GNU build-id for ELF components and surface it in inventory/usage SBOM plus diff payloads with deterministic ordering. | Native analyzer emits buildId for every ELF executable/library, SBOM/diff fixtures updated with canonical `buildId` field, regression tests prove stability, docs call out debug-symbol lookup flow. | +| SCANNER-EMIT-10-607 | TODO | Emit Guild | SCANNER-EMIT-10-604, POLICY-CORE-09-005 | Embed scoring inputs, confidence band, and `quietedBy` provenance into CycloneDX 1.6 and DSSE predicates; verify deterministic serialization. | SBOM/attestation fixtures include score, inputs, configVersion, quiet metadata; golden tests confirm canonical output. | diff --git a/src/StellaOps.Scanner.Sbomer.BuildXPlugin/AGENTS.md b/src/StellaOps.Scanner.Sbomer.BuildXPlugin/AGENTS.md new file mode 100644 index 00000000..820580b4 --- /dev/null +++ b/src/StellaOps.Scanner.Sbomer.BuildXPlugin/AGENTS.md @@ -0,0 +1,12 @@ +# StellaOps.Scanner.Sbomer.BuildXPlugin — Agent Charter + +## Mission +Implement the build-time SBOM generator described in `docs/ARCHITECTURE_SCANNER.md` and new buildx dossier requirements: +- Provide a deterministic BuildKit/Buildx generator that produces layer SBOM fragments and uploads them to local CAS. +- Emit OCI annotations (+provenance) compatible with Scanner.Emit and Attestor hand-offs. +- Respect restart-time plug-in policy (`plugins/scanner/buildx/` manifests) and keep CI overhead ≤300 ms per layer. + +## Expectations +- Read architecture + upcoming Buildx addendum before coding. +- Ensure graceful fallback to post-build scan when generator unavailable. +- Provide integration tests with mock BuildKit, and update `TASKS.md` as states change. diff --git a/src/StellaOps.Scanner.Sbomer.BuildXPlugin/StellaOps.Scanner.Sbomer.BuildXPlugin.csproj b/src/StellaOps.Scanner.Sbomer.BuildXPlugin/StellaOps.Scanner.Sbomer.BuildXPlugin.csproj new file mode 100644 index 00000000..1094c46d --- /dev/null +++ b/src/StellaOps.Scanner.Sbomer.BuildXPlugin/StellaOps.Scanner.Sbomer.BuildXPlugin.csproj @@ -0,0 +1,8 @@ + + + net10.0 + enable + enable + Exe + + diff --git a/src/StellaOps.Scanner.Sbomer.BuildXPlugin/TASKS.md b/src/StellaOps.Scanner.Sbomer.BuildXPlugin/TASKS.md new file mode 100644 index 00000000..8cd30819 --- /dev/null +++ b/src/StellaOps.Scanner.Sbomer.BuildXPlugin/TASKS.md @@ -0,0 +1,7 @@ +# BuildX Plugin Task Board (Sprint 9) + +| ID | Status | Owner(s) | Depends on | Description | Exit Criteria | +|----|--------|----------|------------|-------------|---------------| +| SP9-BLDX-09-001 | TODO | BuildX Guild | SCANNER-EMIT-10-601 (awareness) | Scaffold buildx driver, manifest, local CAS handshake; ensure plugin loads from `plugins/scanner/buildx/`. | Plugin manifest + loader tests; local CAS writes succeed; restart required to activate. | +| SP9-BLDX-09-002 | TODO | BuildX Guild | SP9-BLDX-09-001 | Emit OCI annotations + provenance metadata for Attestor handoff (image + SBOM). | OCI descriptors include DSSE/provenance placeholders; Attestor mock accepts payload. | +| SP9-BLDX-09-003 | TODO | BuildX Guild | SP9-BLDX-09-002 | CI demo pipeline: build sample image, produce SBOM, verify backend report wiring. | GitHub/CI job runs sample build within 5 s overhead; artifacts saved; documentation updated. | diff --git a/src/StellaOps.Scanner.WebService/TASKS.md b/src/StellaOps.Scanner.WebService/TASKS.md new file mode 100644 index 00000000..4c3463e8 --- /dev/null +++ b/src/StellaOps.Scanner.WebService/TASKS.md @@ -0,0 +1,15 @@ +# Scanner WebService Task Board + +| ID | Status | Owner(s) | Depends on | Description | Exit Criteria | +|----|--------|----------|------------|-------------|---------------| +| SCANNER-WEB-09-101 | TODO | Scanner WebService Guild | SCANNER-CORE-09-501 | Stand up minimal API host with Authority OpTok + DPoP enforcement, health/ready endpoints, and restart-time plug-in loader per architecture §1, §4. | Host boots with configuration validation, `/healthz` and `/readyz` return 200, Authority middleware enforced in integration tests. | +| SCANNER-WEB-09-102 | TODO | Scanner WebService Guild | SCANNER-WEB-09-101, SCANNER-QUEUE-09-401 | Implement `/api/v1/scans` submission/status endpoints with deterministic IDs, validation, and cancellation tokens. | Contract documented, e2e test posts scan request and retrieves status, cancellation token honoured. | +| SCANNER-WEB-09-103 | TODO | Scanner WebService Guild | SCANNER-WEB-09-102, SCANNER-CORE-09-502 | Emit scan progress via SSE/JSONL with correlation IDs and deterministic timestamps; document API reference. | Streaming endpoint verified in tests, timestamps formatted ISO-8601 UTC, docs updated in `docs/09_API_CLI_REFERENCE.md`. | +| SCANNER-WEB-09-104 | TODO | Scanner WebService Guild | SCANNER-STORAGE-09-301, SCANNER-QUEUE-09-401 | Bind configuration for Mongo, MinIO, queue, feature flags; add startup diagnostics and fail-fast policy for missing deps. | Misconfiguration fails fast with actionable errors, configuration bound tests pass, diagnostics logged with correlation IDs. | +| SCANNER-POLICY-09-105 | TODO | Scanner WebService Guild | POLICY-CORE-09-001 | Integrate policy schema loader + diagnostics + OpenAPI (YAML ignore rules, VEX include/exclude, vendor precedence). | Policy endpoints documented; validation surfaces actionable errors; OpenAPI schema published. | +| SCANNER-POLICY-09-106 | TODO | Scanner WebService Guild | POLICY-CORE-09-002, SCANNER-POLICY-09-105 | `/reports` verdict assembly (Feedser/Vexer/Policy merge) + signed response envelope. | Aggregated report includes policy metadata; integration test verifies signed response; docs updated. | +| SCANNER-POLICY-09-107 | TODO | Scanner WebService Guild | POLICY-CORE-09-005, SCANNER-POLICY-09-106 | Surface score inputs, config version, and `quietedBy` provenance in `/reports` response and signed payload; document schema changes. | `/reports` JSON + DSSE contain score, reachability, sourceTrust, confidenceBand, quiet provenance; contract tests updated; docs refreshed. | +| SCANNER-RUNTIME-12-301 | TODO | Scanner WebService Guild | ZASTAVA-CORE-12-201 | Implement `/runtime/events` ingestion endpoint with validation, batching, and storage hooks per Zastava contract. | Observer fixtures POST events, data persisted and acked; invalid payloads rejected with deterministic errors. | +| SCANNER-RUNTIME-12-302 | TODO | Scanner WebService Guild | SCANNER-RUNTIME-12-301, ZASTAVA-CORE-12-201 | Implement `/policy/runtime` endpoint joining SBOM baseline + policy verdict, returning admission guidance. | Webhook integration test passes; responses include verdict, TTL, reasons; metrics/logging added. | +| SCANNER-EVENTS-15-201 | TODO | Scanner WebService Guild | NOTIFY-QUEUE-15-401 | Emit `scanner.report.ready` and `scanner.scan.completed` events (bus adapters + tests). | Event envelopes published to queue with schemas; fixtures committed; Notify consumption test passes. | +| SCANNER-RUNTIME-17-401 | TODO | Scanner WebService Guild | SCANNER-RUNTIME-12-301, ZASTAVA-OBS-17-005, SCANNER-EMIT-17-701 | Persist runtime build-id observations and expose them via `/runtime/events` + policy joins for debug-symbol correlation. | Mongo schema stores optional `buildId`, API/SDK responses document field, integration test resolves debug-store path using stored build-id, docs updated accordingly. | diff --git a/src/StellaOps.Scheduler.ImpactIndex/AGENTS.md b/src/StellaOps.Scheduler.ImpactIndex/AGENTS.md new file mode 100644 index 00000000..c876ef3a --- /dev/null +++ b/src/StellaOps.Scheduler.ImpactIndex/AGENTS.md @@ -0,0 +1,4 @@ +# StellaOps.Scheduler.ImpactIndex — Agent Charter + +## Mission +Build the global impact index per `docs/ARCHITECTURE_SCHEDULER.md` (roaring bitmaps, selectors, snapshotting). diff --git a/src/StellaOps.Scheduler.ImpactIndex/StellaOps.Scheduler.ImpactIndex.csproj b/src/StellaOps.Scheduler.ImpactIndex/StellaOps.Scheduler.ImpactIndex.csproj new file mode 100644 index 00000000..6c3a8871 --- /dev/null +++ b/src/StellaOps.Scheduler.ImpactIndex/StellaOps.Scheduler.ImpactIndex.csproj @@ -0,0 +1,7 @@ + + + net10.0 + enable + enable + + diff --git a/src/StellaOps.Scheduler.ImpactIndex/TASKS.md b/src/StellaOps.Scheduler.ImpactIndex/TASKS.md new file mode 100644 index 00000000..4f614aa0 --- /dev/null +++ b/src/StellaOps.Scheduler.ImpactIndex/TASKS.md @@ -0,0 +1,8 @@ +# Scheduler ImpactIndex Task Board (Sprint 16) + +| ID | Status | Owner(s) | Depends on | Description | Exit Criteria | +|----|--------|----------|------------|-------------|---------------| +| SCHED-IMPACT-16-300 | DOING | Scheduler ImpactIndex Guild | SAMPLES-10-001 | **STUB** ingest/query using fixtures to unblock Scheduler planning (remove by SP16 end). | Stub merges fixture BOM-Index, query API returns deterministic results, removal note tracked. | +| SCHED-IMPACT-16-301 | TODO | Scheduler ImpactIndex Guild | SCANNER-EMIT-10-605 | Implement ingestion of per-image BOM-Index sidecars into roaring bitmap store (contains/usedBy). | Ingestion tests process sample SBOM index; bitmaps persisted; deterministic IDs assigned. | +| SCHED-IMPACT-16-302 | TODO | Scheduler ImpactIndex Guild | SCHED-IMPACT-16-301 | Provide query APIs (ResolveByPurls, ResolveByVulns, ResolveAll, selectors) with tenant/namespace filters. | Query functions tested; performance benchmarks documented; selectors enforce filters. | +| SCHED-IMPACT-16-303 | TODO | Scheduler ImpactIndex Guild | SCHED-IMPACT-16-301 | Snapshot/compaction + invalidation for removed images; persistence to RocksDB/Redis per architecture. | Snapshot routine implemented; invalidation tests pass; docs describe recovery. | diff --git a/src/StellaOps.Scheduler.Models/AGENTS.md b/src/StellaOps.Scheduler.Models/AGENTS.md new file mode 100644 index 00000000..132b4943 --- /dev/null +++ b/src/StellaOps.Scheduler.Models/AGENTS.md @@ -0,0 +1,4 @@ +# StellaOps.Scheduler.Models — Agent Charter + +## Mission +Define Scheduler DTOs (Schedule, Run, ImpactSet, Selector, DeltaSummary) per `docs/ARCHITECTURE_SCHEDULER.md`. diff --git a/src/StellaOps.Scheduler.Models/StellaOps.Scheduler.Models.csproj b/src/StellaOps.Scheduler.Models/StellaOps.Scheduler.Models.csproj new file mode 100644 index 00000000..6c3a8871 --- /dev/null +++ b/src/StellaOps.Scheduler.Models/StellaOps.Scheduler.Models.csproj @@ -0,0 +1,7 @@ + + + net10.0 + enable + enable + + diff --git a/src/StellaOps.Scheduler.Models/TASKS.md b/src/StellaOps.Scheduler.Models/TASKS.md new file mode 100644 index 00000000..062722fb --- /dev/null +++ b/src/StellaOps.Scheduler.Models/TASKS.md @@ -0,0 +1,7 @@ +# Scheduler Models Task Board (Sprint 16) + +| ID | Status | Owner(s) | Depends on | Description | Exit Criteria | +|----|--------|----------|------------|-------------|---------------| +| SCHED-MODELS-16-101 | TODO | Scheduler Models Guild | — | Define DTOs (Schedule, Run, ImpactSet, Selector, DeltaSummary, AuditRecord) with validation + canonical JSON. | DTOs merged with tests; documentation snippet added; serialization deterministic. | +| SCHED-MODELS-16-102 | TODO | Scheduler Models Guild | SCHED-MODELS-16-101 | Publish schema docs & sample payloads for UI/Notify integration. | Samples committed; docs referenced; contract tests pass. | +| SCHED-MODELS-16-103 | TODO | Scheduler Models Guild | SCHED-MODELS-16-101 | Versioning/migration helpers (schedule evolution, run state transitions). | Migration helpers implemented; tests cover upgrade/downgrade; guidelines documented. | diff --git a/src/StellaOps.Scheduler.Queue/AGENTS.md b/src/StellaOps.Scheduler.Queue/AGENTS.md new file mode 100644 index 00000000..ae7ee032 --- /dev/null +++ b/src/StellaOps.Scheduler.Queue/AGENTS.md @@ -0,0 +1,4 @@ +# StellaOps.Scheduler.Queue — Agent Charter + +## Mission +Provide queue abstraction (Redis Streams / NATS JetStream) for planner inputs and runner segments per `docs/ARCHITECTURE_SCHEDULER.md`. diff --git a/src/StellaOps.Scheduler.Queue/StellaOps.Scheduler.Queue.csproj b/src/StellaOps.Scheduler.Queue/StellaOps.Scheduler.Queue.csproj new file mode 100644 index 00000000..6c3a8871 --- /dev/null +++ b/src/StellaOps.Scheduler.Queue/StellaOps.Scheduler.Queue.csproj @@ -0,0 +1,7 @@ + + + net10.0 + enable + enable + + diff --git a/src/StellaOps.Scheduler.Queue/TASKS.md b/src/StellaOps.Scheduler.Queue/TASKS.md new file mode 100644 index 00000000..b529a2ba --- /dev/null +++ b/src/StellaOps.Scheduler.Queue/TASKS.md @@ -0,0 +1,7 @@ +# Scheduler Queue Task Board (Sprint 16) + +| ID | Status | Owner(s) | Depends on | Description | Exit Criteria | +|----|--------|----------|------------|-------------|---------------| +| SCHED-QUEUE-16-401 | TODO | Scheduler Queue Guild | SCHED-MODELS-16-101 | Implement queue abstraction + Redis Streams adapter (planner inputs, runner segments) with ack/lease semantics. | Integration tests cover enqueue/dequeue/ack; lease renewal implemented; ordering preserved. | +| SCHED-QUEUE-16-402 | TODO | Scheduler Queue Guild | SCHED-QUEUE-16-401 | Add NATS JetStream adapter with configuration binding, health probes, failover. | Health endpoints verified; failover documented; adapter tested. | +| SCHED-QUEUE-16-403 | TODO | Scheduler Queue Guild | SCHED-QUEUE-16-401 | Dead-letter handling + metrics (queue depth, retry counts), configuration toggles. | Dead-letter policy tested; metrics exported; docs updated. | diff --git a/src/StellaOps.Scheduler.Storage.Mongo/AGENTS.md b/src/StellaOps.Scheduler.Storage.Mongo/AGENTS.md new file mode 100644 index 00000000..0c196b49 --- /dev/null +++ b/src/StellaOps.Scheduler.Storage.Mongo/AGENTS.md @@ -0,0 +1,4 @@ +# StellaOps.Scheduler.Storage.Mongo — Agent Charter + +## Mission +Implement Mongo persistence (schedules, runs, impact cursors, locks, audit) per `docs/ARCHITECTURE_SCHEDULER.md`. diff --git a/src/StellaOps.Scheduler.Storage.Mongo/StellaOps.Scheduler.Storage.Mongo.csproj b/src/StellaOps.Scheduler.Storage.Mongo/StellaOps.Scheduler.Storage.Mongo.csproj new file mode 100644 index 00000000..6c3a8871 --- /dev/null +++ b/src/StellaOps.Scheduler.Storage.Mongo/StellaOps.Scheduler.Storage.Mongo.csproj @@ -0,0 +1,7 @@ + + + net10.0 + enable + enable + + diff --git a/src/StellaOps.Scheduler.Storage.Mongo/TASKS.md b/src/StellaOps.Scheduler.Storage.Mongo/TASKS.md new file mode 100644 index 00000000..2173bcfe --- /dev/null +++ b/src/StellaOps.Scheduler.Storage.Mongo/TASKS.md @@ -0,0 +1,7 @@ +# Scheduler Storage Task Board (Sprint 16) + +| ID | Status | Owner(s) | Depends on | Description | Exit Criteria | +|----|--------|----------|------------|-------------|---------------| +| SCHED-STORAGE-16-201 | TODO | Scheduler Storage Guild | SCHED-MODELS-16-101 | Create Mongo collections (schedules, runs, impact_cursors, locks, audit) with indexes/migrations per architecture. | Migration scripts and indexes implemented; integration tests cover CRUD paths. | +| SCHED-STORAGE-16-202 | TODO | Scheduler Storage Guild | SCHED-STORAGE-16-201 | Implement repositories/services with tenant scoping, soft delete, TTL for completed runs, and causal consistency options. | Unit tests pass; TTL/soft delete validated; documentation updated. | +| SCHED-STORAGE-16-203 | TODO | Scheduler Storage Guild | SCHED-STORAGE-16-201 | Audit/logging pipeline + run stats materialized views for UI. | Audit entries persisted; stats queries efficient; docs capture usage. | diff --git a/src/StellaOps.Scheduler.WebService/AGENTS.md b/src/StellaOps.Scheduler.WebService/AGENTS.md new file mode 100644 index 00000000..d45d6b13 --- /dev/null +++ b/src/StellaOps.Scheduler.WebService/AGENTS.md @@ -0,0 +1,4 @@ +# StellaOps.Scheduler.WebService — Agent Charter + +## Mission +Implement Scheduler control plane per `docs/ARCHITECTURE_SCHEDULER.md`. diff --git a/src/StellaOps.Scheduler.WebService/StellaOps.Scheduler.WebService.csproj b/src/StellaOps.Scheduler.WebService/StellaOps.Scheduler.WebService.csproj new file mode 100644 index 00000000..4c69387c --- /dev/null +++ b/src/StellaOps.Scheduler.WebService/StellaOps.Scheduler.WebService.csproj @@ -0,0 +1,7 @@ + + + net10.0 + enable + enable + + diff --git a/src/StellaOps.Scheduler.WebService/TASKS.md b/src/StellaOps.Scheduler.WebService/TASKS.md new file mode 100644 index 00000000..0b65569d --- /dev/null +++ b/src/StellaOps.Scheduler.WebService/TASKS.md @@ -0,0 +1,8 @@ +# Scheduler WebService Task Board (Sprint 16) + +| ID | Status | Owner(s) | Depends on | Description | Exit Criteria | +|----|--------|----------|------------|-------------|---------------| +| SCHED-WEB-16-101 | TODO | Scheduler WebService Guild | SCHED-MODELS-16-101 | Bootstrap Minimal API host with Authority OpTok + DPoP, health endpoints, plug-in discovery per architecture §§1–2. | Service boots with config validation; `/healthz`/`/readyz` pass; restart-only plug-ins enforced. | +| SCHED-WEB-16-102 | TODO | Scheduler WebService Guild | SCHED-WEB-16-101 | Implement schedules CRUD (tenant-scoped) with cron validation, pause/resume, audit logging. | CRUD operations tested; invalid cron inputs rejected; audit entries persisted. | +| SCHED-WEB-16-103 | TODO | Scheduler WebService Guild | SCHED-WEB-16-102 | Runs API (list/detail/cancel), ad-hoc run POST, and impact preview endpoints. | Integration tests cover run lifecycle; preview returns counts/sample; cancellation honoured. | +| SCHED-WEB-16-104 | TODO | Scheduler WebService Guild | SCHED-QUEUE-16-401, SCHED-STORAGE-16-201 | Webhook endpoints for Feedser/Vexer exports with mTLS/HMAC validation and rate limiting. | Webhooks validated via tests; invalid signatures rejected; rate limits documented. | diff --git a/src/StellaOps.Scheduler.Worker/AGENTS.md b/src/StellaOps.Scheduler.Worker/AGENTS.md new file mode 100644 index 00000000..8077e87d --- /dev/null +++ b/src/StellaOps.Scheduler.Worker/AGENTS.md @@ -0,0 +1,4 @@ +# StellaOps.Scheduler.Worker — Agent Charter + +## Mission +Implement Scheduler planners/runners per `docs/ARCHITECTURE_SCHEDULER.md`. diff --git a/src/StellaOps.Scheduler.Worker/StellaOps.Scheduler.Worker.csproj b/src/StellaOps.Scheduler.Worker/StellaOps.Scheduler.Worker.csproj new file mode 100644 index 00000000..1094c46d --- /dev/null +++ b/src/StellaOps.Scheduler.Worker/StellaOps.Scheduler.Worker.csproj @@ -0,0 +1,8 @@ + + + net10.0 + enable + enable + Exe + + diff --git a/src/StellaOps.Scheduler.Worker/TASKS.md b/src/StellaOps.Scheduler.Worker/TASKS.md new file mode 100644 index 00000000..3596ac4c --- /dev/null +++ b/src/StellaOps.Scheduler.Worker/TASKS.md @@ -0,0 +1,9 @@ +# Scheduler Worker Task Board (Sprint 16) + +| ID | Status | Owner(s) | Depends on | Description | Exit Criteria | +|----|--------|----------|------------|-------------|---------------| +| SCHED-WORKER-16-201 | TODO | Scheduler Worker Guild | SCHED-QUEUE-16-401 | Planner loop (cron + event triggers) with lease management, fairness, and rate limiting (§6). | Planner integration tests cover cron/event triggers; rate limits enforced; logs include run IDs. | +| SCHED-WORKER-16-202 | TODO | Scheduler Worker Guild | SCHED-IMPACT-16-301 | Wire ImpactIndex targeting (ResolveByPurls/vulns), dedupe, shard planning. | Targeting tests confirm correct image selection; dedupe documented; shards evenly distributed. | +| SCHED-WORKER-16-203 | TODO | Scheduler Worker Guild | SCHED-WORKER-16-202 | Runner execution: call Scanner `/reports` (analysis-only) or `/scans` when configured; collect deltas; handle retries. | Runner tests stub Scanner; retries/backoff validated; deltas aggregated deterministically. | +| SCHED-WORKER-16-204 | TODO | Scheduler Worker Guild | SCHED-WORKER-16-203 | Emit events (`scheduler.rescan.delta`, `scanner.report.ready`) for Notify/UI with summaries. | Events published to queue; payload schema documented; integration tests verify consumption. | +| SCHED-WORKER-16-205 | TODO | Scheduler Worker Guild | SCHED-WORKER-16-201 | Metrics/telemetry: run stats, queue depth, planner latency, delta counts. | Metrics exported per spec; dashboards updated; alerts configured. | diff --git a/src/StellaOps.UI/TASKS.md b/src/StellaOps.UI/TASKS.md new file mode 100644 index 00000000..96f7f3fd --- /dev/null +++ b/src/StellaOps.UI/TASKS.md @@ -0,0 +1,11 @@ +# UI Task Board (Sprints 11 & 13) + +| ID | Status | Owner(s) | Depends on | Description | Exit Criteria | +|----|--------|----------|------------|-------------|---------------| +| UI-AUTH-13-001 | TODO | UI Guild | AUTH-DPOP-11-001, AUTH-MTLS-11-002 | Integrate Authority OIDC + DPoP flows with session management. | Login/logout flows pass e2e tests; tokens refreshed; DPoP nonce handling validated. | +| UI-SCANS-13-002 | TODO | UI Guild | SCANNER-WEB-09-102, SIGNER-API-11-101 | Build scans module (list/detail/SBOM/diff/attestation) with performance + accessibility targets. | Cypress tests cover SBOM/diff; performance budgets met; accessibility checks pass. | +| UI-VEX-13-003 | TODO | UI Guild | EXCITITOR-CORE-02-001, EXCITITOR-EXPORT-01-005 | Implement VEX explorer + policy editor with preview integration. | VEX views render consensus/conflicts; staged policy preview works; accessibility checks pass. | +| UI-ADMIN-13-004 | TODO | UI Guild | AUTH-MTLS-11-002 | Deliver admin area (tenants/clients/quotas/licensing) with RBAC + audit hooks. | Admin e2e tests pass; unauthorized access blocked; telemetry wired. | +| UI-ATTEST-11-005 | TODO | UI Guild | SIGNER-API-11-101, ATTESTOR-API-11-201 | Attestation visibility (Rekor id, status) on Scan Detail. | UI shows Rekor UUID/status; mock attestation fixtures displayed; tests cover success/failure. | +| UI-SCHED-13-005 | TODO | UI Guild | SCHED-WEB-16-101 | Scheduler panel: schedules CRUD, run history, dry-run preview using API/mocks. | Panel functional with mocked endpoints; UX signoff; integration tests added. | +| UI-NOTIFY-13-006 | TODO | UI Guild | NOTIFY-WEB-15-101 | Notify panel: channels/rules CRUD, deliveries view, test send integration. | Panel interacts with mocked Notify API; tests cover rule lifecycle; docs updated. | diff --git a/src/StellaOps.Zastava.Observer/TASKS.md b/src/StellaOps.Zastava.Observer/TASKS.md new file mode 100644 index 00000000..1a780b74 --- /dev/null +++ b/src/StellaOps.Zastava.Observer/TASKS.md @@ -0,0 +1,9 @@ +# Zastava Observer Task Board + +| ID | Status | Owner(s) | Depends on | Description | Exit Criteria | +|----|--------|----------|------------|-------------|---------------| +| ZASTAVA-OBS-12-001 | TODO | Zastava Observer Guild | ZASTAVA-CORE-12-201 | Build container lifecycle watcher that tails CRI (containerd/cri-o/docker) events and emits deterministic runtime records with buffering + backoff. | Fixture cluster produces start/stop events with stable ordering, jitter/backoff tested, metrics/logging wired. | +| ZASTAVA-OBS-12-002 | TODO | Zastava Observer Guild | ZASTAVA-OBS-12-001 | Capture entrypoint traces and loaded libraries, hashing binaries and correlating to SBOM baseline per architecture sections 2.1 and 10. | EntryTrace parser covers shell/python/node launchers, loaded library hashes recorded, fixtures assert linkage to SBOM usage view. | +| ZASTAVA-OBS-12-003 | TODO | Zastava Observer Guild | ZASTAVA-OBS-12-002 | Implement runtime posture checks (signature/SBOM/attestation presence) with offline caching and warning surfaces. | Observer marks posture status, caches refresh across restarts, integration tests prove offline tolerance. | +| ZASTAVA-OBS-12-004 | TODO | Zastava Observer Guild | ZASTAVA-OBS-12-002 | Batch `/runtime/events` submissions with disk-backed buffer, rate limits, and deterministic envelopes. | Buffered submissions survive restart, rate-limits enforced in tests, JSON envelopes match schema in docs/events. | +| ZASTAVA-OBS-17-005 | TODO | Zastava Observer Guild | ZASTAVA-OBS-12-002 | Collect GNU build-id for ELF processes and attach it to emitted runtime events to enable symbol lookup + debug-store correlation. | Observer reads build-id via `/proc//exe`/notes without pausing workloads, runtime events include `buildId` field, fixtures cover glibc/musl images, docs updated with retrieval notes. |